WEBDIENSTE/theHarvester
1
0
Fork 0
mirror of https://github.com/laramies/theHarvester.git synced 2026-08-17 19:35:40 +02:00
Code Issues Packages Projects Releases Wiki Activity

refactor: centralize source execution lifecycle

Browse Source
This commit is contained in:
NotoriousRebel
2026-08-16 00:32:22 -04:00
parent f3dc2eb7ab
commit 2ac59f3a38
76 changed files with 1208 additions and 1605 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+1
View File
@@ -36,6 +36,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added root contributor and security policies, structured issue forms, repository agent guidance, discovery terminology, and an operator-focused documentation wiki ([d090a29a](https://github.com/laramies/theHarvester/commit/d090a29a), [7c491ef5](https://github.com/laramies/theHarvester/commit/7c491ef5), [8b9d420b](https://github.com/laramies/theHarvester/commit/8b9d420b)).
### Changed
- Centralized passive-source completion reporting in an immutable `SourceExecutionReport` returned by adapters, with the source runner alone deriving partial and no-result outcomes from retained evidence; removed mutable per-adapter `execution_status` and `stop_reason` state.
- Updated BeVigil, Dymo, FOFA, FullHunt, Hunter.how, Netlas, ONYPHE, SecurityScorecard, SecurityTrails, SherlockEye, SubdomainFinder C99, VirusTotal, WhoisXML, and ZoomEye provider contracts to retain scoped partial evidence and report authentication, quota, transport, HTTP, and malformed-response outcomes truthfully. FOFA and ONYPHE now honor the operator result limit across documented pagination, while ZoomEye uses the current `POST /v2/search` API and no longer stops after five empty result pages.
- Replaced runtime takeover fingerprint downloads and global body-substring matches with pinned provider-gated DNS, wildcard controls, and compound HTTP rules. Every checked hostname is now stored as an indicator, no-indicator, or inconclusive outcome with typed DNS, HTTP, rule, and error details in JSONL, SQLite, the API, and HarvestView. Direct checks share one cookie-free HTTP session, keep bounded response bodies, and rely on the whole-run deadline instead of silently inheriting aiohttp's default timeout.
- HarvestView now summarizes retained evidence and producer health at a glance, links directly to execution outcomes that need attention, and keeps evidence values ahead of optional actions on mobile. Its source picker reports credential readiness without exposing values, prevents unavailable source selections, and replaces the mobile nested-scroll catalog with collapsible activity groups.
docs/wiki/How-to-add-a-new-module.md
+5 -2
View File
@@ -20,11 +20,13 @@ Create the adapter under [`theHarvester/discovery/`](https://github.com/laramies
An adapter normally provides:
- an initializer for the target and local result sets;
- an asynchronous `process()` method;
- an asynchronous `process()` method returning `SourceExecutionReport | None`;
- only the getters it actually supports, such as `get_hostnames()`, `get_emails()`, `get_ips()`, `get_asns()`, `get_urls()`, or `get_results()`.
Do not return fields the provider did not supply. Normalize and deduplicate before returning results.
Return `None` when the provider conversation completed normally, including a valid zero-result response. Return an immutable `SourceExecutionReport` with a stable provider-specific reason for another terminal condition: `completed` for a successful early stop such as reaching the requested result limit, `failed` for provider or transport failure, `rate-limited` for a terminal rate limit, or `partial` when the provider itself confirms incomplete coverage. Do not add mutable `execution_status` or `stop_reason` fields to an adapter. The source runner owns finalization: it promotes any incomplete report with retained normalized evidence to `partial`, and records a normal zero-result completion as `completed` with `no-results`.
### Own the provider conversation
A provider conversation is the related request sequence for one source execution: initial request, pagination, retries or polling, and final response handling. Give that sequence one explicit owner.
@@ -33,7 +35,7 @@ A provider conversation is the related request sequence for one source execution
- Keep the default cookie jar when later provider requests may depend on earlier responses. Use `aiohttp.DummyCookieJar()` for deliberately independent probes, such as takeover candidates, so one target cannot influence another.
- Scope a session to one provider and authorized target. Never share cookies, authentication state, or proxy identity across source executions or unrelated targets.
- Preserve cancellation while closing every owned session, response, task, and connector. Cover both successful completion and interruption in focused tests.
- Treat session construction and teardown as adapter lifecycle stages. Preserve the existing TLS and timeout policy unless the source contract explicitly changes, classify ordinary lifecycle failures through the adapter status fields, and let native cancellation propagate.
- Treat session construction and teardown as adapter lifecycle stages. Preserve the existing TLS and timeout policy unless the source contract explicitly changes, return a `SourceExecutionReport` for ordinary lifecycle failures, and let native cancellation propagate.
- Before extending a shared fetcher interface, audit positional callers and every owned-versus-borrowed branch. New optional parameters must not reinterpret existing calls.
The completion check is an offline test in which a later page depends on state established by an earlier page, plus a cleanup assertion proving the provider session closes.
@@ -65,6 +67,7 @@ Useful cases include:
- missing required credentials;
- non-success, timeout, empty, or malformed responses;
- pagination and termination;
- the returned execution report for incomplete work and `None` for normal completion;
- normalized and deduplicated results.
Tests must not require external network access or real provider credentials.
tests/discovery/test_apisguru.py
+61 -74
View File
@@ -68,7 +68,7 @@ async def test_preferred_openapi_three_spec_returns_only_target_scoped_evidence(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_hostnames() == {
'api.example.com',
@@ -90,8 +90,7 @@ async def test_preferred_openapi_three_spec_returns_only_target_scoped_evidence(
assert all(call['proxy'] is True for call in calls)
assert all(call['request_timeout'] == 60 for call in calls)
assert all(set(call) == {'url', 'proxy', 'request_timeout'} for call in calls)
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -126,12 +125,11 @@ async def test_templated_server_path_retains_its_concrete_hostname(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=500)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert await search.get_urls() == set()
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -179,7 +177,7 @@ async def test_unwrapped_directory_and_openapi_two_spec_are_supported(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('EXAMPLE.com.', limit=5)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com', 'support.example.com'}
assert await search.get_emails() == {'ops@example.com'}
@@ -188,8 +186,7 @@ async def test_unwrapped_directory_and_openapi_two_spec_are_supported(
'https://api.example.com/v2',
'https://support.example.com/help',
}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -239,12 +236,12 @@ async def test_external_spec_url_is_not_fetched_and_valid_results_remain_partial
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process()
report = await search.process()
assert requested_urls == ['https://api.apis.guru/v2/example.com.json', valid_spec_url]
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -276,12 +273,11 @@ async def test_preferred_spec_requests_are_not_artificially_capped_at_five(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=500)
await search.process()
report = await search.process()
assert requested_urls == ['https://api.apis.guru/v2/example.com.json', *spec_urls]
assert await search.get_hostnames() == {f'api-{index}.example.com' for index in range(6)}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -317,12 +313,12 @@ async def test_oversized_preferred_spec_does_not_discard_later_results(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=500)
await search.process()
report = await search.process()
assert requested_urls == ['https://api.apis.guru/v2/example.com.json', *spec_urls]
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'response-limit'
assert report.status == 'failed'
assert report.stop_reason == 'response-limit'
@pytest.mark.asyncio
@@ -358,12 +354,12 @@ async def test_missing_preferred_spec_does_not_discard_later_results(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=500)
await search.process()
report = await search.process()
assert requested_urls == ['https://api.apis.guru/v2/example.com.json', *spec_urls]
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'http-404'
assert report.status == 'failed'
assert report.stop_reason == 'http-404'
@pytest.mark.asyncio
@@ -393,11 +389,11 @@ async def test_access_denied_preferred_spec_stops_before_later_requests(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=500)
await search.process()
report = await search.process()
assert requested_urls == ['https://api.apis.guru/v2/example.com.json', spec_urls[0]]
assert search.execution_status == 'failed'
assert search.stop_reason == 'access-denied'
assert report.status == 'failed'
assert report.stop_reason == 'access-denied'
@pytest.mark.asyncio
@@ -442,13 +438,13 @@ async def test_result_limit_does_not_truncate_preferred_spec_traversal(monkeypat
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=2)
await search.process()
report = await search.process()
assert requested_urls == ['https://api.apis.guru/v2/example.com.json', *spec_urls]
assert await search.get_hostnames() == {'one.example.com', 'two.example.com'}
assert await search.get_urls() == {'https://one.example.com', 'https://two.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'result-limit'
assert report.status == 'completed'
assert report.stop_reason == 'result-limit'
@pytest.mark.asyncio
@@ -487,11 +483,11 @@ async def test_malformed_preferred_spec_does_not_discard_later_valid_results(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -532,12 +528,12 @@ async def test_malformed_spec_fields_preserve_scoped_results_as_partial(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert await search.get_emails() == set()
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -559,11 +555,11 @@ async def test_malformed_matching_directory_entry_is_failed(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -599,12 +595,12 @@ async def test_directory_entry_scan_is_bounded(monkeypatch: pytest.MonkeyPatch)
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process()
report = await search.process()
assert requested_urls == ['https://api.apis.guru/v2/example.com.json', spec_urls[0]]
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'directory-entry-limit'
assert report.status == 'failed'
assert report.stop_reason == 'directory-entry-limit'
@pytest.mark.asyncio
@@ -641,12 +637,11 @@ async def test_openapi_two_host_is_retained_without_inventing_a_server_scheme(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert await search.get_urls() == set()
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.parametrize(
@@ -674,13 +669,11 @@ async def test_directory_failures_are_attributed(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == execution_status
assert search.stop_reason == stop_reason
assert report.status == execution_status
assert report.stop_reason == stop_reason
@pytest.mark.asyncio
@@ -691,10 +684,9 @@ async def test_missing_provider_is_completed_with_no_results(monkeypatch: pytest
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process()
report = await search.process()
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.parametrize('target', ['example.com/path', 'localhost', '192.0.2.1', 'straße.de'])
@@ -709,10 +701,10 @@ async def test_invalid_target_is_rejected_before_provider_request(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru(target, limit=5)
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-target'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-target'
@pytest.mark.asyncio
@@ -747,11 +739,10 @@ async def test_malformed_contact_email_is_not_retained(monkeypatch: pytest.Monke
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process()
report = await search.process()
assert await search.get_emails() == set()
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.parametrize(
@@ -810,10 +801,10 @@ async def test_directory_response_byte_limit_is_failed(monkeypatch: pytest.Monke
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'response-limit'
assert report.status == 'failed'
assert report.stop_reason == 'response-limit'
@pytest.mark.asyncio
@@ -842,11 +833,10 @@ async def test_apex_only_hostname_is_not_counted_as_a_retained_result(monkeypatc
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.asyncio
@@ -886,12 +876,12 @@ async def test_scalar_results_are_hard_bounded(monkeypatch: pytest.MonkeyPatch)
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'one.example.com'}
assert await search.get_urls() == {'https://one.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'result-limit'
assert report.status == 'failed'
assert report.stop_reason == 'result-cap'
@pytest.mark.asyncio
@@ -905,9 +895,6 @@ async def test_cancellation_is_attributed_and_propagated(monkeypatch: pytest.Mon
with pytest.raises(asyncio.CancelledError):
await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'cancelled'
@pytest.mark.asyncio
async def test_runtime_limit_is_attributed(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -919,10 +906,10 @@ async def test_runtime_limit_is_attributed(monkeypatch: pytest.MonkeyPatch) -> N
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=500)
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'runtime-limit'
assert report.status == 'failed'
assert report.stop_reason == 'runtime-limit'
@pytest.mark.parametrize(
@@ -969,10 +956,10 @@ async def test_preferred_spec_failures_are_attributed(
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
await search.process()
report = await search.process()
assert search.execution_status == execution_status
assert search.stop_reason == stop_reason
assert report.status == execution_status
assert report.stop_reason == stop_reason
pytestmark = pytest.mark.provider_contract('apis-guru')
tests/discovery/test_baidusearch.py
+22 -20
View File
@@ -57,8 +57,9 @@ class TestBaiduSearch:
monkeypatch.setattr(baidusearch.Core, 'get_browser_user_agent', staticmethod(lambda: 'UA'))
search = baidusearch.SearchBaidu(word='example.com', limit=21)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert report is None
assert [call['url'] for call in calls] == [
'https://www.baidu.com/',
'https://www.baidu.com/s?wd=site%3Aexample.com&pn=0',
@@ -87,13 +88,13 @@ class TestBaiduSearch:
)
search = baidusearch.SearchBaidu(word='example.com', limit=20)
await search.process()
report = await search.process()
assert len(calls) == 2
assert calls[-1]['follow_redirects'] is False
assert session.delays == [1.0]
assert search.execution_status == 'failed'
assert search.stop_reason == 'security-verification'
assert report.status == 'failed'
assert report.stop_reason == 'security-verification'
assert session.closed is True
@pytest.mark.asyncio
@@ -104,12 +105,12 @@ class TestBaiduSearch:
)
search = baidusearch.SearchBaidu(word='example.com', limit=10)
await search.process()
report = await search.process()
assert len(calls) == 1
assert calls[0]['follow_redirects'] is False
assert search.execution_status == 'failed'
assert search.stop_reason == 'security-verification'
assert report.status == 'failed'
assert report.stop_reason == 'security-verification'
@pytest.mark.asyncio
async def test_later_captcha_preserves_partial_results(self, monkeypatch: pytest.MonkeyPatch) -> None:
@@ -124,42 +125,42 @@ class TestBaiduSearch:
)
search = baidusearch.SearchBaidu(word='example.com', limit=30)
await search.process()
report = await search.process()
assert len(calls) == 3
assert await search.get_hostnames() == ['api.example.com']
assert search.execution_status == 'partial'
assert search.stop_reason == 'security-verification'
assert report.status == 'failed'
assert report.stop_reason == 'security-verification'
@pytest.mark.asyncio
async def test_transport_failure_is_reported(self, monkeypatch: pytest.MonkeyPatch) -> None:
patch_requests(monkeypatch, [response('<html>homepage</html>'), None])
search = baidusearch.SearchBaidu(word='example.com', limit=10)
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'transport-error'
assert report.status == 'failed'
assert report.stop_reason == 'transport-error'
@pytest.mark.asyncio
async def test_empty_response_is_reported(self, monkeypatch: pytest.MonkeyPatch) -> None:
patch_requests(monkeypatch, [response('<html>homepage</html>'), response('')])
search = baidusearch.SearchBaidu(word='example.com', limit=10)
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'no-response'
assert report.status == 'failed'
assert report.stop_reason == 'no-response'
@pytest.mark.asyncio
async def test_http_429_is_reported(self, monkeypatch: pytest.MonkeyPatch) -> None:
patch_requests(monkeypatch, [response('<html>homepage</html>'), response('', status=429)])
search = baidusearch.SearchBaidu(word='example.com', limit=10)
await search.process()
report = await search.process()
assert search.execution_status == 'rate-limited'
assert search.stop_reason == 'http-429'
assert report.status == 'rate-limited'
assert report.stop_reason == 'http-429'
@pytest.mark.asyncio
async def test_pagination_limit_is_exclusive(self, monkeypatch: pytest.MonkeyPatch) -> None:
@@ -169,8 +170,9 @@ class TestBaiduSearch:
)
search = baidusearch.SearchBaidu(word='example.com', limit=20)
await search.process()
report = await search.process()
assert report is None
assert [call['url'] for call in calls[1:]] == [
'https://www.baidu.com/s?wd=site%3Aexample.com&pn=0',
'https://www.baidu.com/s?wd=site%3Aexample.com&pn=10',
tests/discovery/test_bevigil.py
+11 -12
View File
@@ -48,7 +48,7 @@ async def test_process_collects_scoped_hostnames_and_urls(monkeypatch: pytest.Mo
monkeypatch.setattr(bevigil.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = bevigil.SearchBeVigil('example.com')
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_hostnames() == {'api.example.com'}
assert await search.get_urls() == {'https://portal.example.com/path'}
@@ -69,8 +69,7 @@ async def test_process_collects_scoped_hostnames_and_urls(monkeypatch: pytest.Mo
}
]
assert session_exited is True
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.parametrize('key', [None, '', ' '])
@@ -107,12 +106,12 @@ async def test_failed_first_response_is_attributed(
monkeypatch.setattr(bevigil.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = bevigil.SearchBeVigil('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_urls() == set()
assert search.execution_status == execution_status
assert search.stop_reason == stop_reason
assert report.status == execution_status
assert report.stop_reason == stop_reason
@pytest.mark.asyncio
@@ -129,12 +128,12 @@ async def test_later_malformed_response_preserves_partial_results(monkeypatch: p
monkeypatch.setattr(bevigil.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = bevigil.SearchBeVigil('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert await search.get_urls() == set()
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -175,8 +174,8 @@ async def test_early_malformed_rows_and_later_valid_evidence_are_partial(monkeyp
monkeypatch.setattr(bevigil.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = bevigil.SearchBeVigil('example.com')
await search.process()
report = await search.process()
assert await search.get_urls() == {'https://portal.example.com/path'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
tests/discovery/test_bufferoverun.py
+14 -17
View File
@@ -30,15 +30,13 @@ async def test_process_parses_historical_four_column_rows(monkeypatch: pytest.Mo
monkeypatch.setattr(bufferoverun.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = bufferoverun.SearchBufferover('example.com')
await search.process(proxy=True)
report = await search.process(proxy=True)
assert captured['proxy'] is True
assert await search.get_hostnames() == {'api.example.com', 'ipv6.example.com'}
assert await search.get_ips() == {'192.0.2.10', '2001:db8::10'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -52,12 +50,11 @@ async def test_valid_empty_response_is_completed(monkeypatch: pytest.MonkeyPatch
monkeypatch.setattr(bufferoverun.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = bufferoverun.SearchBufferover('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.parametrize(
@@ -85,12 +82,12 @@ async def test_failed_responses_are_attributed(
monkeypatch.setattr(bufferoverun.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = bufferoverun.SearchBufferover('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
assert search.execution_status == execution_status
assert search.stop_reason == stop_reason
assert report.status == execution_status
assert report.stop_reason == stop_reason
@pytest.mark.asyncio
@@ -109,12 +106,12 @@ async def test_domain_first_five_column_row_is_rejected_without_crashing(monkeyp
monkeypatch.setattr(bufferoverun.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = bufferoverun.SearchBufferover('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -133,12 +130,12 @@ async def test_non_string_row_preserves_valid_partial_results(monkeypatch: pytes
monkeypatch.setattr(bufferoverun.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = bufferoverun.SearchBufferover('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert await search.get_ips() == {'192.0.2.10'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
pytestmark = pytest.mark.provider_contract('bufferoverun')
tests/discovery/test_builtwith.py
+16 -18
View File
@@ -76,7 +76,7 @@ async def test_process_uses_v23_privacy_controls_and_parses_nested_results(monke
monkeypatch.setattr(builtwith.AsyncFetcher, 'fetch_json', fake_fetch_json)
search = builtwith.SearchBuiltWith('example.com')
await search.process(proxy=True)
report = await search.process(proxy=True)
assert captured == {
'url': 'https://api.builtwith.com/v23/api.json',
@@ -101,8 +101,7 @@ async def test_process_uses_v23_privacy_controls_and_parses_nested_results(monke
assert await search.get_servers() == {'nginx'}
assert await search.get_cms() == {'WordPress'}
assert await search.get_analytics() == {'Google Analytics'}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -129,13 +128,12 @@ async def test_www_target_does_not_accept_sibling_subdomains(monkeypatch) -> Non
monkeypatch.setattr(builtwith.AsyncFetcher, 'fetch_json', fake_fetch_json)
search = builtwith.SearchBuiltWith('www.example.com')
await search.process()
report = await search.process()
assert captured['params']['LOOKUP'] == 'example.com'
assert await search.get_hostnames() == {'www.example.com'}
assert await search.get_urls() == set()
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -164,13 +162,13 @@ async def test_named_technology_without_a_usable_category_is_malformed(monkeypat
monkeypatch.setattr(builtwith.AsyncFetcher, 'fetch_json', fake_fetch_json)
search = builtwith.SearchBuiltWith('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'www.example.com'}
assert await search.get_urls() == set()
assert await search.get_frameworks() == set()
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -199,12 +197,12 @@ async def test_process_reports_failed_responses_truthfully(
monkeypatch.setattr(builtwith.AsyncFetcher, 'fetch_json', fake_fetch_json)
search = builtwith.SearchBuiltWith('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_tech_stack() == {}
assert search.execution_status == expected_status
assert search.stop_reason == expected_reason
assert report.status == expected_status
assert report.stop_reason == expected_reason
@pytest.mark.asyncio
@@ -240,14 +238,14 @@ async def test_malformed_nested_containers_retain_accepted_evidence(monkeypatch)
monkeypatch.setattr(builtwith.AsyncFetcher, 'fetch_json', fake_fetch_json)
search = builtwith.SearchBuiltWith('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert await search.get_urls() == set()
assert await search.get_frameworks() == {'Django'}
assert await search.get_servers() == {'nginx'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -261,10 +259,10 @@ async def test_bounded_response_failures_are_attributed(monkeypatch, reason: str
monkeypatch.setattr(builtwith.AsyncFetcher, 'fetch_json', fake_fetch_json)
search = builtwith.SearchBuiltWith('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == reason
assert report.status == 'failed'
assert report.stop_reason == reason
@pytest.mark.asyncio
tests/discovery/test_censys.py
+18 -18
View File
@@ -104,7 +104,7 @@ async def test_search_calls_platform_api_directly_and_follows_page_tokens(monkey
monkeypatch.setattr(censysearch.AsyncFetcher, 'open_session', fake_open_session)
search = censysearch.SearchCensys('example.com', limit=250)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert [call['url'] for call in calls] == [
'https://api.platform.censys.io/v3/global/search/query',
@@ -137,7 +137,7 @@ async def test_search_calls_platform_api_directly_and_follows_page_tokens(monkey
assert all(call['session'] is session for call in calls)
assert await search.get_hostnames() == {'a.example.com', 'b.example.com'}
assert await search.get_emails() == {'admin@example.com', 'ops@example.com'}
assert search.execution_status == 'completed'
assert report is None
@pytest.mark.asyncio
@@ -152,10 +152,10 @@ async def test_session_setup_failure_reports_transport_error(monkeypatch: pytest
monkeypatch.setattr(censysearch.AsyncFetcher, 'open_session', failed_open_session)
search = censysearch.SearchCensys('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'transport-error'
assert report.status == 'failed'
assert report.stop_reason == 'transport-error'
@pytest.mark.asyncio
@@ -193,8 +193,9 @@ async def test_search_uses_free_wallet_and_respects_result_limit(monkeypatch) ->
monkeypatch.setattr(censysearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = censysearch.SearchCensys('example.com', limit=3)
await search.process()
report = await search.process()
assert report is None
assert calls[0]['params'] == ''
assert calls[0]['json_body']['page_size'] == 3
assert await search.get_hostnames() == {'1.example.com', '2.example.com', '3.example.com'}
@@ -211,10 +212,9 @@ async def test_search_accepts_missing_terminal_page_token(monkeypatch) -> None:
monkeypatch.setattr(censysearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = censysearch.SearchCensys('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.asyncio
@@ -236,11 +236,11 @@ async def test_malformed_limit_hit_is_partial_when_it_contains_evidence(monkeypa
monkeypatch.setattr(censysearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = censysearch.SearchCensys('example.com', limit=1)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'a.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.parametrize(
@@ -264,10 +264,10 @@ async def test_search_reports_provider_failures_truthfully(
monkeypatch.setattr(censysearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = censysearch.SearchCensys('example.com')
await search.process()
report = await search.process()
assert search.execution_status == expected_status
assert search.stop_reason == expected_reason
assert report.status == expected_status
assert report.stop_reason == expected_reason
@pytest.mark.asyncio
@@ -302,10 +302,10 @@ async def test_search_classifies_transport_exceptions(monkeypatch) -> None:
monkeypatch.setattr(censysearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = censysearch.SearchCensys('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'transport-error'
assert report.status == 'failed'
assert report.stop_reason == 'transport-error'
def test_deprecated_censys_sdk_is_not_a_runtime_dependency() -> None:
tests/discovery/test_certspotter.py
+38 -31
View File
@@ -1,5 +1,4 @@
#!/usr/bin/env python3
# coding=utf-8
import logging
from typing import Any
from urllib.parse import parse_qs, urlparse
@@ -11,13 +10,13 @@ from theHarvester.discovery import certspottersearch
from theHarvester.lib.core import Core
class TestCertspotter(object):
class TestCertspotter:
@staticmethod
def domain() -> str:
return 'example.com'
class TestCertspotterSearch(object):
class TestCertspotterSearch:
@pytest.mark.live_network
def test_api(self, live_test_domain: str) -> None:
base_url = f'https://api.certspotter.com/v1/issuances?domain={live_test_domain}&expand=dns_names'
@@ -35,7 +34,10 @@ class TestCertspotterSearch(object):
monkeypatch.setattr(certspottersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = certspottersearch.SearchCertspoter(TestCertspotter.domain())
await search.process()
report = await search.process()
assert report is not None
assert report.status == 'partial'
assert report.stop_reason == 'invalid-cursor'
assert await search.get_hostnames() == {'api.example.com', 'www.example.com'}
@pytest.mark.asyncio
@@ -53,8 +55,9 @@ class TestCertspotterSearch(object):
monkeypatch.setattr(certspottersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = certspottersearch.SearchCertspoter(TestCertspotter.domain())
await search.process()
report = await search.process()
assert report is None
assert await search.get_hostnames() == {'first.example.com', 'second.example.com'}
assert [parse_qs(urlparse(url).query) for url in requested_urls] == [
{'domain': ['example.com'], 'include_subdomains': ['true'], 'expand': ['dns_names']},
@@ -92,12 +95,12 @@ class TestCertspotterSearch(object):
search = certspottersearch.SearchCertspoter(TestCertspotter.domain())
with caplog.at_level(logging.WARNING, logger=certspottersearch.__name__):
await search.process()
report = await search.process()
assert requests == 2
assert await search.get_hostnames() == {'host-1.example.com', 'host-2.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'page-limit'
assert report.status == 'partial'
assert report.stop_reason == 'page-limit'
assert 'page limit reached; results may be incomplete' in caplog.text
@pytest.mark.asyncio
@@ -127,8 +130,11 @@ class TestCertspotterSearch(object):
monkeypatch.setattr(certspottersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = certspottersearch.SearchCertspoter(' Example.COM. ')
await search.process()
report = await search.process()
assert report is not None
assert report.status == 'partial'
assert report.stop_reason == 'malformed-issuance'
assert await search.get_hostnames() == {'api.example.com', 'example.com', 'www.example.com'}
@pytest.mark.asyncio
@@ -146,11 +152,11 @@ class TestCertspotterSearch(object):
monkeypatch.setattr(certspottersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = certspottersearch.SearchCertspoter(TestCertspotter.domain())
with caplog.at_level(logging.WARNING, logger=certspottersearch.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'first.example.com'}
assert search.execution_status == 'rate-limited'
assert search.stop_reason == 'rate_limited'
assert report.status == 'rate-limited'
assert report.stop_reason == 'rate_limited'
assert 'rate_limited' in caplog.text
assert 'results may be incomplete' in caplog.text
assert 'provider details must not be logged' not in caplog.text
@@ -170,11 +176,11 @@ class TestCertspotterSearch(object):
monkeypatch.setattr(certspottersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = certspottersearch.SearchCertspoter(TestCertspotter.domain())
with caplog.at_level(logging.WARNING, logger=certspottersearch.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'first.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'partial'
assert report.stop_reason == 'invalid-response'
assert 'results may be incomplete' in caplog.text
@pytest.mark.asyncio
@@ -192,11 +198,11 @@ class TestCertspotterSearch(object):
monkeypatch.setattr(certspottersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = certspottersearch.SearchCertspoter(TestCertspotter.domain())
with caplog.at_level(logging.WARNING, logger=certspottersearch.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == 'partial'
assert search.stop_reason == stop_reason
assert report.status == 'partial'
assert report.stop_reason == stop_reason
assert 'results may be incomplete' in caplog.text
@pytest.mark.asyncio
@@ -214,11 +220,11 @@ class TestCertspotterSearch(object):
monkeypatch.setattr(certspottersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = certspottersearch.SearchCertspoter(TestCertspotter.domain())
with caplog.at_level(logging.WARNING, logger=certspottersearch.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'first.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'malformed-issuance'
assert report.status == 'partial'
assert report.stop_reason == 'malformed-issuance'
assert 'results may be incomplete' in caplog.text
@pytest.mark.asyncio
@@ -250,11 +256,11 @@ class TestCertspotterSearch(object):
monkeypatch.setattr(certspottersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = certspottersearch.SearchCertspoter(TestCertspotter.domain())
with caplog.at_level(logging.WARNING, logger=certspottersearch.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'first.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == stop_reason
assert report.status == 'partial'
assert report.stop_reason == stop_reason
assert 'results may be incomplete' in caplog.text
assert str(error) not in caplog.text
@@ -276,12 +282,12 @@ class TestCertspotterSearch(object):
monkeypatch.setattr(certspottersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = certspottersearch.SearchCertspoter(TestCertspotter.domain())
with caplog.at_level(logging.WARNING, logger=certspottersearch.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'first.example.com', 'second.example.com'}
assert calls == 2
assert search.execution_status == 'partial'
assert search.stop_reason == 'repeated-cursor'
assert report.status == 'partial'
assert report.stop_reason == 'repeated-cursor'
assert 'results may be incomplete' in caplog.text
@pytest.mark.asyncio
@@ -297,8 +303,9 @@ class TestCertspotterSearch(object):
monkeypatch.setattr(certspottersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = certspottersearch.SearchCertspoter(TestCertspotter.domain())
await search.process()
report = await search.process()
assert report is None
assert await search.get_hostnames() == {f'page-{page_number}.example.com' for page_number in range(1, 12)}
assert calls == 12
@@ -326,12 +333,12 @@ class TestCertspotterSearch(object):
monkeypatch.setattr(certspottersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = certspottersearch.SearchCertspoter(TestCertspotter.domain())
with caplog.at_level(logging.WARNING, logger=certspottersearch.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'first.example.com'}
assert calls == 1
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-cursor'
assert report.status == 'partial'
assert report.stop_reason == 'invalid-cursor'
assert 'results may be incomplete' in caplog.text
tests/discovery/test_commoncrawl.py
+36 -26
View File
@@ -40,8 +40,9 @@ async def test_process_exhausts_current_catalog_index_pages(monkeypatch: pytest.
monkeypatch.setattr(commoncrawl.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = commoncrawl.SearchCommoncrawl('example.com')
await search.process()
report = await search.process()
assert report is None
assert await search.get_hostnames() == {
'api.example.com',
'example.com',
@@ -89,8 +90,9 @@ async def test_process_uses_unique_indexes_from_latest_catalog_year_window_and_s
monkeypatch.setattr(commoncrawl.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = commoncrawl.SearchCommoncrawl('Example.COM.')
await search.process()
report = await search.process()
assert report is None
assert await search.get_hostnames() == {'api.example.com', 'dev.example.com', 'example.com'}
assert sum(current_endpoint in url for url in requested_urls) == 4
assert sum(recent_endpoint in url for url in requested_urls) == 4
@@ -186,12 +188,12 @@ async def test_process_caps_provider_page_counts_and_reports_truncation(
search = commoncrawl.SearchCommoncrawl('example.com', limit=50)
with caplog.at_level(logging.WARNING, logger=commoncrawl.__name__):
await search.process()
report = await search.process()
assert requested_pages == [0, 1, 0, 1]
assert 'Common Crawl page limit reached for index CC-MAIN-2026-30; results may be incomplete' in caplog.text
assert search.execution_status == 'partial'
assert search.stop_reason == 'page-limit'
assert report.status == 'partial'
assert report.stop_reason == 'page-limit'
@pytest.mark.asyncio
@@ -224,8 +226,9 @@ async def test_process_respects_the_result_limit_across_page_requests(monkeypatc
monkeypatch.setattr(commoncrawl.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = commoncrawl.SearchCommoncrawl('example.com', limit=51)
await search.process()
report = await search.process()
assert report is None
assert requested_limits == [50, 1]
assert len(await search.get_hostnames()) == 51
@@ -269,8 +272,9 @@ async def test_process_counts_only_unique_in_scope_hosts_toward_the_result_limit
monkeypatch.setattr(commoncrawl.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = commoncrawl.SearchCommoncrawl('example.com', limit=3)
await search.process()
report = await search.process()
assert report is None
assert await search.get_hostnames() == {
'duplicate.example.com',
'host-3.example.com',
@@ -312,8 +316,11 @@ async def test_process_reports_failed_or_malformed_index_without_discarding_othe
search = commoncrawl.SearchCommoncrawl('example.com')
with caplog.at_level(logging.WARNING, logger=commoncrawl.__name__):
await search.process()
report = await search.process()
assert report is not None
assert report.status == 'partial'
assert report.stop_reason == 'query-errors'
assert await search.get_hostnames() == {'survivor.example.com'}
assert 'CC-MAIN-BROKEN' in caplog.text
assert 'CC-MAIN-2026-30' in caplog.text
@@ -342,11 +349,11 @@ async def test_process_reports_non_json_upstream_response(
search = commoncrawl.SearchCommoncrawl('example.com')
with caplog.at_level(logging.WARNING, logger=commoncrawl.__name__):
await search.process()
report = await search.process()
assert 'unexpected non-JSON response' in caplog.text
assert search.execution_status == 'failed'
assert search.stop_reason == 'all-queries-failed'
assert report.status == 'failed'
assert report.stop_reason == 'all-queries-failed'
@pytest.mark.asyncio
@@ -377,13 +384,13 @@ async def test_process_stops_a_query_after_three_entirely_unusable_pages(
search = commoncrawl.SearchCommoncrawl('example.com')
with caplog.at_level(logging.INFO, logger=commoncrawl.__name__):
await search.process()
report = await search.process()
assert page_batches == 6
assert 'Common Crawl selected 1 index and 2 queries' in caplog.text
assert 'example.com' not in caplog.text
assert search.execution_status == 'failed'
assert search.stop_reason == 'all-queries-failed'
assert report.status == 'failed'
assert report.stop_reason == 'all-queries-failed'
@pytest.mark.asyncio
@@ -413,13 +420,12 @@ async def test_process_retains_partial_results_at_the_runtime_limit(monkeypatch:
monkeypatch.setattr(commoncrawl.SearchCommoncrawl, 'RUNTIME_SECONDS', 0.01)
search = commoncrawl.SearchCommoncrawl('example.com')
await asyncio.wait_for(search.process(), timeout=0.1)
report = await asyncio.wait_for(search.process(), timeout=0.1)
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'runtime-limit'
assert report.status == 'failed'
assert report.stop_reason == 'runtime-limit'
@pytest.mark.asyncio
@@ -431,11 +437,11 @@ async def test_process_reports_a_runtime_limit_before_collecting_results(monkeyp
monkeypatch.setattr(commoncrawl.SearchCommoncrawl, 'RUNTIME_SECONDS', 0.01)
search = commoncrawl.SearchCommoncrawl('example.com')
await asyncio.wait_for(search.process(), timeout=0.1)
report = await asyncio.wait_for(search.process(), timeout=0.1)
assert await search.get_hostnames() == set()
assert search.execution_status == 'failed'
assert search.stop_reason == 'runtime-limit'
assert report.status == 'failed'
assert report.stop_reason == 'runtime-limit'
@pytest.mark.asyncio
@@ -488,12 +494,12 @@ async def test_process_keeps_later_valid_page_after_malformed_page(
search = commoncrawl.SearchCommoncrawl('example.com')
with caplog.at_level(logging.WARNING, logger=commoncrawl.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert 'malformed JSON line' in caplog.text
assert search.execution_status == 'partial'
assert search.stop_reason == 'query-errors'
assert report.status == 'partial'
assert report.stop_reason == 'query-errors'
@pytest.mark.asyncio
@@ -519,8 +525,9 @@ async def test_process_isolates_malformed_urls_within_a_page(monkeypatch: pytest
monkeypatch.setattr(commoncrawl.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = commoncrawl.SearchCommoncrawl('example.com')
await search.process()
report = await search.process()
assert report is None
assert await search.get_hostnames() == {'api.example.com', 'mail.example.com'}
@@ -547,8 +554,11 @@ async def test_process_keeps_results_when_another_query_fails(monkeypatch: pytes
monkeypatch.setattr(commoncrawl.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = commoncrawl.SearchCommoncrawl('example.com')
await search.process()
report = await search.process()
assert report is not None
assert report.status == 'partial'
assert report.stop_reason == 'query-errors'
assert await search.get_hostnames() == {'api.example.com'}
tests/discovery/test_crtname.py
+22 -27
View File
@@ -58,7 +58,7 @@ async def test_crt_name_streams_one_provider_response_and_retains_scoped_descend
monkeypatch.setattr(crtname.AsyncFetcher, 'stream_records', stream_records)
search = crtname.SearchCrtName(' Example.COM. ')
await search.process(proxy=True)
report = await search.process(proxy=True)
assert calls == [
{
@@ -72,8 +72,8 @@ async def test_crt_name_streams_one_provider_response_and_retains_scoped_descend
}
]
assert await search.get_hostnames() == {'api.example.com', 'wild.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -90,12 +90,11 @@ async def test_crt_name_queries_and_retains_only_the_requested_scope(
monkeypatch.setattr(crtname.AsyncFetcher, 'stream_records', stream_records)
search = crtname.SearchCrtName('www.example.com')
await search.process()
report = await search.process()
assert calls[0]['params'] == {'apex': 'www.example.com'}
assert await search.get_hostnames() == {'deep.www.example.com'}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -140,11 +139,11 @@ async def test_crt_name_attributes_provider_status_without_parsing_error_bodies(
monkeypatch.setattr(crtname.AsyncFetcher, 'stream_records', stream_records)
search = crtname.SearchCrtName('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == execution_status
assert search.stop_reason == stop_reason
assert report.status == execution_status
assert report.stop_reason == stop_reason
@pytest.mark.asyncio
@@ -156,11 +155,11 @@ async def test_crt_name_preserves_valid_prefix_when_stream_fails(monkeypatch: py
monkeypatch.setattr(crtname.AsyncFetcher, 'stream_records', stream_records)
search = crtname.SearchCrtName('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'response-limit'
assert report.status == 'failed'
assert report.stop_reason == 'response-limit'
@pytest.mark.asyncio
@@ -181,11 +180,11 @@ async def test_crt_name_preserves_valid_prefix_at_runtime_limit(monkeypatch: pyt
monkeypatch.setattr(crtname.AsyncFetcher, 'stream_records', stream_records)
search = crtname.SearchCrtName('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'runtime-limit'
assert report.status == 'failed'
assert report.stop_reason == 'runtime-limit'
@pytest.mark.asyncio
@@ -197,11 +196,10 @@ async def test_crt_name_empty_response_completes_without_results(monkeypatch: py
monkeypatch.setattr(crtname.AsyncFetcher, 'stream_records', stream_records)
search = crtname.SearchCrtName('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.asyncio
@@ -228,10 +226,10 @@ async def test_crt_name_rejects_invalid_targets_without_requesting(
monkeypatch.setattr(crtname.AsyncFetcher, 'stream_records', unexpected_request)
search = crtname.SearchCrtName(target)
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-target'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-target'
@pytest.mark.asyncio
@@ -243,11 +241,11 @@ async def test_crt_name_rejects_non_ascii_provider_records(monkeypatch: pytest.M
monkeypatch.setattr(crtname.AsyncFetcher, 'stream_records', stream_records)
search = crtname.SearchCrtName('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -284,9 +282,6 @@ async def test_crt_name_and_crtsh_share_one_result_with_both_sources(
completed_results.append(result)
class FakeCrtsh:
execution_status = 'completed'
stop_reason = None
def __init__(self, _word: str) -> None:
pass
tests/discovery/test_crtsh.py
+22 -15
View File
@@ -41,7 +41,8 @@ class TestCrtshSearch:
async def test_process_collects_hostnames(self, monkeypatch):
_patch_fetch(monkeypatch, [{'name_value': 'www.example.com'}, {'name_value': 'mail.example.com'}])
search = crtsh.SearchCrtsh('example.com')
await search.process(proxy=True)
report = await search.process(proxy=True)
assert report is None
assert search.proxy is True
assert set(await search.get_hostnames()) == {'www.example.com', 'mail.example.com'}
@@ -49,7 +50,8 @@ class TestCrtshSearch:
async def test_wildcard_prefix_is_stripped(self, monkeypatch):
_patch_fetch(monkeypatch, [{'name_value': '*.example.com'}])
search = crtsh.SearchCrtsh('example.com')
await search.process()
report = await search.process()
assert report is None
assert set(await search.get_hostnames()) == {'example.com'}
@pytest.mark.asyncio
@@ -57,14 +59,16 @@ class TestCrtshSearch:
# crt.sh packs several names into one name_value separated by newlines.
_patch_fetch(monkeypatch, [{'name_value': 'a.example.com\nb.example.com'}])
search = crtsh.SearchCrtsh('example.com')
await search.process()
report = await search.process()
assert report is None
assert set(await search.get_hostnames()) == {'a.example.com', 'b.example.com'}
@pytest.mark.asyncio
async def test_numeric_prefixed_entries_are_preserved(self, monkeypatch):
_patch_fetch(monkeypatch, [{'name_value': '1234.example.com'}, {'name_value': 'good.example.com'}])
search = crtsh.SearchCrtsh('example.com')
await search.process()
report = await search.process()
assert report is None
assert set(await search.get_hostnames()) == {'1234.example.com', 'good.example.com'}
@pytest.mark.asyncio
@@ -72,13 +76,12 @@ class TestCrtshSearch:
fetches = _patch_fetch(monkeypatch, [])
delays = _patch_sleep(monkeypatch)
search = crtsh.SearchCrtsh('example.com')
await search.process()
report = await search.process()
assert report is None
assert len(fetches) == 1
assert delays == []
assert await search.get_hostnames() == []
assert search.execution_status is None
assert search.stop_reason is None
@pytest.mark.asyncio
async def test_failed_fetches_are_retried_with_delay(self, monkeypatch):
@@ -97,11 +100,12 @@ class TestCrtshSearch:
delays = _patch_sleep(monkeypatch)
search = crtsh.SearchCrtsh('example.com')
await search.process()
report = await search.process()
assert fetch_count == 3
assert delays == [2, 2]
assert await search.get_hostnames() == ['api.example.com']
assert report is None
@pytest.mark.asyncio
async def test_exhausted_http_failures_are_reported(self, monkeypatch):
@@ -118,13 +122,13 @@ class TestCrtshSearch:
delays = _patch_sleep(monkeypatch)
search = crtsh.SearchCrtsh('example.com')
await search.process()
report = await search.process()
assert fetch_count == 3
assert delays == [2, 2]
assert await search.get_hostnames() == []
assert search.execution_status == 'failed'
assert search.stop_reason == 'http-502'
assert report.status == 'failed'
assert report.stop_reason == 'http-502'
@pytest.mark.asyncio
async def test_cancellation_propagates(self, monkeypatch):
@@ -146,19 +150,22 @@ class TestCrtshSearch:
monkeypatch.setattr(crtsh.SearchCrtsh, 'RUNTIME_SECONDS', 0.01)
search = crtsh.SearchCrtsh('example.com')
await asyncio.wait_for(search.process(), timeout=1.0)
report = await asyncio.wait_for(search.process(), timeout=1.0)
assert await search.get_hostnames() == []
assert search.execution_status == 'failed'
assert search.stop_reason == 'runtime-limit'
assert report.status == 'failed'
assert report.stop_reason == 'runtime-limit'
@pytest.mark.asyncio
async def test_missing_name_value_key_is_handled(self, monkeypatch):
fetches = _patch_fetch(monkeypatch, [{'issuer_ca_id': 1}])
delays = _patch_sleep(monkeypatch)
search = crtsh.SearchCrtsh('example.com')
await search.process()
report = await search.process()
assert report is not None
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
assert len(fetches) == 1
assert delays == []
assert await search.get_hostnames() == []
tests/discovery/test_dnsdumpster.py
+13 -15
View File
@@ -30,14 +30,13 @@ async def test_process_keeps_only_label_scoped_records_and_preserves_proxy(monke
monkeypatch.setattr(search_dnsdumpster.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = search_dnsdumpster.SearchDNSDumpster('example.com')
await search.process(proxy=True)
report = await search.process(proxy=True)
assert captured['proxy'] is True
assert captured['include_metadata'] is True
assert await search.get_hostnames() == {'api.example.com'}
assert await search.get_ips() == {'192.0.2.10'}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -50,12 +49,11 @@ async def test_valid_empty_response_is_completed(monkeypatch: pytest.MonkeyPatch
monkeypatch.setattr(search_dnsdumpster.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = search_dnsdumpster.SearchDNSDumpster('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.parametrize(
@@ -87,12 +85,12 @@ async def test_failed_responses_are_attributed(
monkeypatch.setattr(search_dnsdumpster.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = search_dnsdumpster.SearchDNSDumpster('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
assert search.execution_status == execution_status
assert search.stop_reason == stop_reason
assert report.status == execution_status
assert report.stop_reason == stop_reason
@pytest.mark.asyncio
@@ -118,12 +116,12 @@ async def test_valid_and_malformed_records_preserve_partial_results(monkeypatch:
monkeypatch.setattr(search_dnsdumpster.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = search_dnsdumpster.SearchDNSDumpster('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert await search.get_ips() == {'2001:db8::1'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -136,10 +134,10 @@ async def test_fetch_exception_is_transport_failure(monkeypatch: pytest.MonkeyPa
monkeypatch.setattr(search_dnsdumpster.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = search_dnsdumpster.SearchDNSDumpster('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'transport-error'
assert report.status == 'failed'
assert report.stop_reason == 'transport-error'
pytestmark = pytest.mark.provider_contract('dnsdumpster')
tests/discovery/test_dymosearch.py
+7 -9
View File
@@ -27,7 +27,7 @@ async def test_process_extracts_scoped_canonical_and_suggested_domains(monkeypat
monkeypatch.setattr(dymosearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = dymosearch.SearchDymo('example.com')
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_hostnames() == {'example.com', 'www.example.com'}
assert (await search.get_results())['domain']['domain'] == 'example.com'
@@ -35,8 +35,7 @@ async def test_process_extracts_scoped_canonical_and_suggested_domains(monkeypat
assert captured['json_body'] == {'domain': 'example.com', 'url': 'https://example.com'}
assert captured['include_metadata'] is True
assert captured['proxy'] is True
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.parametrize('key', [None, '', ' '])
@@ -71,10 +70,10 @@ async def test_failures_are_structured(
monkeypatch.setattr(dymosearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = dymosearch.SearchDymo('example.com')
await search.process()
report = await search.process()
assert search.execution_status == status
assert search.stop_reason == reason
assert report.status == status
assert report.stop_reason == reason
@pytest.mark.asyncio
@@ -86,10 +85,9 @@ async def test_empty_object_is_completed_without_evidence(monkeypatch: pytest.Mo
monkeypatch.setattr(dymosearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = dymosearch.SearchDymo('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.asyncio
tests/discovery/test_fofa.py
+9 -12
View File
@@ -9,6 +9,7 @@ import pytest
from theHarvester.discovery import fofa
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
@pytest.mark.provider_contract('fofa')
@@ -51,12 +52,11 @@ async def test_process_uses_cursor_api_to_limit_and_retains_scoped_results(monke
monkeypatch.setattr(fofa.AsyncFetcher, 'fetch', fake_fetch)
search = fofa.SearchFofa('example.com', limit=3)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_hostnames() == {'api.example.com', 'mail.example.com'}
assert await search.get_ips() == {'192.0.2.10', '2001:db8::10'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report == SourceExecutionReport('failed', 'invalid-response')
assert [call['params']['size'] for call in calls] == [3, 2]
assert [call['params'].get('next') for call in calls] == [None, 'cursor-2']
assert all(call['url'] == 'https://fofa.info/api/v1/search/next' for call in calls)
@@ -117,10 +117,9 @@ async def test_failures_are_structured(
monkeypatch.setattr(fofa.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(fofa.AsyncFetcher, 'fetch', fake_fetch)
search = fofa.SearchFofa('example.com', 10)
await search.process()
report = await search.process()
assert search.execution_status == status
assert search.stop_reason == reason
assert report == SourceExecutionReport(status, reason)
@pytest.mark.asyncio
@@ -141,10 +140,9 @@ async def test_repeated_cursor_stops_without_a_third_request(monkeypatch: pytest
monkeypatch.setattr(fofa.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(fofa.AsyncFetcher, 'fetch', fake_fetch)
search = fofa.SearchFofa('example.com', 10)
await search.process()
report = await search.process()
assert search.execution_status == 'partial'
assert search.stop_reason == 'repeated-cursor'
assert report == SourceExecutionReport('failed', 'repeated-cursor')
assert responses == []
@@ -173,12 +171,11 @@ async def test_malformed_url_does_not_discard_later_valid_rows(monkeypatch: pyte
monkeypatch.setattr(fofa.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(fofa.AsyncFetcher, 'fetch', fake_fetch)
search = fofa.SearchFofa('example.com', 2)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert await search.get_ips() == {'192.0.2.10'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report == SourceExecutionReport('failed', 'invalid-response')
@pytest.mark.asyncio
tests/discovery/test_fullhuntsearch.py
+22 -24
View File
@@ -40,7 +40,7 @@ async def test_process_reuses_one_session_for_fallback_requests(monkeypatch: pyt
monkeypatch.setattr(fullhuntsearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = fullhuntsearch.SearchFullHunt('example.com')
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_hostnames() == ['api.example.com']
assert [urls for urls, _kwargs in calls] == [
@@ -56,8 +56,7 @@ async def test_process_reuses_one_session_for_fallback_requests(monkeypatch: pyt
}
]
assert session_exited is True
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -78,13 +77,13 @@ async def test_http_failure_is_reported_without_results(
search = fullhuntsearch.SearchFullHunt('example.com')
with caplog.at_level(logging.INFO, logger=fullhuntsearch.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == []
assert await search.get_ips() == []
assert requests == ['https://fullhunt.io/api/v1/domain/example.com/details']
assert search.execution_status == 'failed'
assert search.stop_reason == 'access-denied'
assert report.status == 'failed'
assert report.stop_reason == 'access-denied'
@pytest.mark.parametrize('key', ['', ' '])
@@ -121,12 +120,12 @@ async def test_malformed_domain_details_are_reported_without_fallback(
search = fullhuntsearch.SearchFullHunt('example.com')
with caplog.at_level(logging.INFO, logger=fullhuntsearch.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == []
assert requests == ['https://fullhunt.io/api/v1/domain/example.com/details']
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -157,13 +156,13 @@ async def test_malformed_host_does_not_hide_later_valid_results(
search = fullhuntsearch.SearchFullHunt('example.com')
with caplog.at_level(logging.INFO, logger=fullhuntsearch.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == ['api.example.com']
assert await search.get_ips() == ['192.0.2.20']
assert caplog.text.count('FullHunt ignored a malformed host item') == 4
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -184,11 +183,11 @@ async def test_malformed_subdomain_fallback_is_reported(
search = fullhuntsearch.SearchFullHunt('example.com')
with caplog.at_level(logging.INFO, logger=fullhuntsearch.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == []
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -209,12 +208,12 @@ async def test_fallback_ignores_malformed_and_out_of_scope_hosts(
search = fullhuntsearch.SearchFullHunt('example.com')
with caplog.at_level(logging.INFO, logger=fullhuntsearch.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == ['api.example.com']
assert caplog.text.count('FullHunt ignored a malformed subdomain item') == 2
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -244,15 +243,14 @@ async def test_nested_results_use_normalized_hostname(monkeypatch: pytest.Monkey
monkeypatch.setattr(fullhuntsearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = fullhuntsearch.SearchFullHunt('example.com')
await search.process()
report = await search.process()
assert await search.get_dns_records() == {'api.example.com': {'a': ['192.0.2.20']}}
assert await search.get_http_info() == {'api.example.com': {'status': 200}}
assert await search.get_geo_info() == {'api.example.com': {'country': 'US'}}
assert await search.get_cloud_info() == {'api.example.com': {'provider': 'example'}}
assert await search.get_certificate_info() == [{'issuer': 'Example CA', 'hostname': 'api.example.com'}]
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.parametrize(
@@ -278,10 +276,10 @@ async def test_failures_are_structured(
monkeypatch.setattr(fullhuntsearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = fullhuntsearch.SearchFullHunt('example.com')
await search.process()
report = await search.process()
assert search.execution_status == status
assert search.stop_reason == reason
assert report.status == status
assert report.stop_reason == reason
@pytest.mark.asyncio
tests/discovery/test_haveibeenpwned.py
+15 -14
View File
@@ -43,8 +43,9 @@ async def test_public_breach_catalog_preserves_metadata(monkeypatch: pytest.Monk
monkeypatch.setattr(haveibeenpwned.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = haveibeenpwned.SearchHaveIBeenPwned('example.com')
await search.process()
report = await search.process()
assert report is None
assert await search.get_hostnames() == {'example.com'}
assert await search.get_breach_names() == {'ExampleBreach'}
assert await search.get_breach_dates() == {'2024-01-02'}
@@ -70,8 +71,9 @@ async def test_public_breach_catalog_ignores_blank_names(monkeypatch: pytest.Mon
monkeypatch.setattr(haveibeenpwned.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = haveibeenpwned.SearchHaveIBeenPwned('example.com')
await search.process()
report = await search.process()
assert report is None
assert await search.get_breach_names() == {'ExampleBreach'}
@@ -83,14 +85,13 @@ async def test_public_breach_catalog_valid_empty_is_completed(monkeypatch: pytes
monkeypatch.setattr(haveibeenpwned.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = haveibeenpwned.SearchHaveIBeenPwned('example.com')
await search.process()
report = await search.process()
assert await search.get_breaches() == []
assert await search.get_hostnames() == set()
assert await search.get_breach_dates() == set()
assert await search.get_affected_data() == set()
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.asyncio
@@ -111,12 +112,12 @@ async def test_public_breach_catalog_attributes_http_failures(
search = haveibeenpwned.SearchHaveIBeenPwned('example.com')
with caplog.at_level(logging.INFO, logger=haveibeenpwned.__name__):
await search.process()
report = await search.process()
assert await search.get_breaches() == []
assert f'HaveIBeenPwned request failed with HTTP {http_status}' in caplog.text
assert search.execution_status == execution_status
assert search.stop_reason == f'http-{http_status}'
assert report.status == execution_status
assert report.stop_reason == f'http-{http_status}'
@pytest.mark.asyncio
@@ -131,10 +132,10 @@ async def test_public_breach_catalog_malformed_json_fails(
monkeypatch.setattr(haveibeenpwned.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = haveibeenpwned.SearchHaveIBeenPwned('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
assert await search.get_breaches() == []
@@ -146,10 +147,10 @@ async def test_public_breach_catalog_missing_metadata_is_transport_failure(monke
monkeypatch.setattr(haveibeenpwned.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = haveibeenpwned.SearchHaveIBeenPwned('example.com')
await search.process(proxy=True)
report = await search.process(proxy=True)
assert search.execution_status == 'failed'
assert search.stop_reason == 'transport-error'
assert report.status == 'failed'
assert report.stop_reason == 'transport-error'
@pytest.mark.asyncio
tests/discovery/test_hunterhow.py
+8 -9
View File
@@ -60,7 +60,7 @@ async def test_process_paginates_to_limit_and_keeps_scoped_hostnames(monkeypatch
monkeypatch.setattr(searchhunterhow.asyncio, 'sleep', fake_sleep)
search = searchhunterhow.SearchHunterHow('example.com', limit=3)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_hostnames() == {'api.example.com', 'www.example.com'}
assert session_options == [{'headers': {'User-Agent': searchhunterhow.Core.get_user_agent()}, 'proxy': True}]
@@ -72,8 +72,7 @@ async def test_process_paginates_to_limit_and_keeps_scoped_hostnames(monkeypatch
assert all(call['url'] == 'https://api.hunter.how/search' for call in calls)
assert all(call['include_metadata'] is True for call in calls)
assert delays == [2.0]
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
assert session_exited is True
@@ -117,11 +116,11 @@ async def test_failed_response_is_attributed(
monkeypatch.setattr(searchhunterhow.AsyncFetcher, 'fetch', fake_fetch)
search = searchhunterhow.SearchHunterHow('example.com', limit=10)
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == execution_status
assert search.stop_reason == stop_reason
assert report.status == execution_status
assert report.stop_reason == stop_reason
def test_limit_must_be_positive(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -154,11 +153,11 @@ async def test_early_malformed_rows_and_later_valid_evidence_are_partial(monkeyp
monkeypatch.setattr(searchhunterhow.asyncio, 'sleep', fake_sleep)
search = searchhunterhow.SearchHunterHow('example.com', limit=2)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
tests/discovery/test_netlas.py
+8 -9
View File
@@ -51,11 +51,10 @@ async def test_process_uses_current_api_contract_and_keeps_scoped_results(monkey
monkeypatch.setattr(netlas.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = netlas.SearchNetlas('example.com', limit=2)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
assert session_options == [{'headers': {'Authorization': 'Bearer test-key'}, 'proxy': True}]
assert calls[0]['url'] == 'https://app.netlas.io/api/domains/download/'
assert calls[0]['session'] is session
@@ -114,10 +113,10 @@ async def test_download_failures_are_truthful(
monkeypatch.setattr(netlas.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = netlas.SearchNetlas('example.com', limit=10)
await search.process()
report = await search.process()
assert search.execution_status == status
assert search.stop_reason == reason
assert report.status == status
assert report.stop_reason == reason
@pytest.mark.asyncio
@@ -138,11 +137,11 @@ async def test_malformed_download_rows_preserve_valid_partial_results(monkeypatc
monkeypatch.setattr(netlas.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = netlas.SearchNetlas('example.com', limit=10)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'ok.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
tests/discovery/test_onyphe.py
+16 -18
View File
@@ -67,7 +67,7 @@ async def test_process_paginates_to_limit_and_preserves_all_routes(monkeypatch:
monkeypatch.setattr(onyphe.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(onyphe.AsyncFetcher, 'fetch', fake_fetch)
search = onyphe.SearchOnyphe('example.com', limit=3)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_ips() == {'192.0.2.10', '2001:db8::10'}
assert await search.get_hostnames() == {'api.example.com', 'geo.example.com', 'mail.example.com', 'www.example.com'}
@@ -81,8 +81,7 @@ async def test_process_paginates_to_limit_and_preserves_all_routes(monkeypatch:
assert [call['params']['page'] for call in calls] == [1, 2]
assert [call['params']['size'] for call in calls] == [3, 3]
assert all(call['session'] is session for call in calls)
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
assert session_exited is True
@@ -131,10 +130,10 @@ async def test_failures_are_structured(
monkeypatch.setattr(onyphe.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(onyphe.AsyncFetcher, 'fetch', fake_fetch)
search = onyphe.SearchOnyphe('example.com', 10)
await search.process()
report = await search.process()
assert search.execution_status == status
assert search.stop_reason == reason
assert report.status == status
assert report.stop_reason == reason
@pytest.mark.asyncio
@@ -159,11 +158,11 @@ async def test_later_page_failure_preserves_partial_evidence(monkeypatch: pytest
monkeypatch.setattr(onyphe.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(onyphe.AsyncFetcher, 'fetch', fake_fetch)
search = onyphe.SearchOnyphe('example.com', 2)
await search.process()
report = await search.process()
assert await search.get_ips() == {'192.0.2.10'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'http-429'
assert report.status == 'rate-limited'
assert report.stop_reason == 'http-429'
@pytest.mark.asyncio
@@ -190,11 +189,10 @@ async def test_operator_limit_is_not_reported_as_a_provider_limit(monkeypatch: p
monkeypatch.setattr(onyphe.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(onyphe.AsyncFetcher, 'fetch', fake_fetch)
search = onyphe.SearchOnyphe('example.com', 2)
await search.process()
report = await search.process()
assert await search.get_ips() == {'192.0.2.10', '192.0.2.11'}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -223,13 +221,13 @@ async def test_search_api_reports_its_documented_total_boundary(monkeypatch: pyt
monkeypatch.setattr(onyphe.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(onyphe.AsyncFetcher, 'fetch', fake_fetch)
search = onyphe.SearchOnyphe('example.com', 10_001)
await search.process()
report = await search.process()
assert [call['params']['page'] for call in calls] == [1]
assert [call['params']['size'] for call in calls] == [10_000]
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'provider-limit'
assert report.status == 'failed'
assert report.stop_reason == 'provider-limit'
@pytest.mark.asyncio
@@ -254,11 +252,11 @@ async def test_malformed_items_preserve_valid_partial_results(monkeypatch: pytes
monkeypatch.setattr(onyphe.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(onyphe.AsyncFetcher, 'fetch', fake_fetch)
search = onyphe.SearchOnyphe('example.com', 10)
await search.process()
report = await search.process()
assert await search.get_ips() == {'192.0.2.10'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
tests/discovery/test_otx.py
+21 -20
View File
@@ -44,7 +44,8 @@ class TestOtx:
monkeypatch.setattr(otxsearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = otxsearch.SearchOtx(TestOtx.domain())
await search.process()
report = await search.process()
assert report is None
assert await search.get_hostnames() == {'api.example.com', 'www.example.com'}
assert await search.get_ips() == {'192.0.2.1', '2001:db8::1'}
@@ -61,12 +62,12 @@ class TestOtx:
monkeypatch.setattr(otxsearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = otxsearch.SearchOtx(TestOtx.domain())
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
async def test_empty_passive_dns_is_a_valid_zero_yield(self, monkeypatch: pytest.MonkeyPatch) -> None:
@@ -76,12 +77,11 @@ class TestOtx:
monkeypatch.setattr(otxsearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = otxsearch.SearchOtx(TestOtx.domain())
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
assert search.execution_status is None
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
async def test_provider_failures_are_attributed(
@@ -96,12 +96,12 @@ class TestOtx:
search = otxsearch.SearchOtx(TestOtx.domain())
with caplog.at_level(logging.INFO, logger=otxsearch.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
assert search.execution_status == 'failed'
assert search.stop_reason == 'transport-error'
assert report.status == 'failed'
assert report.stop_reason == 'transport-error'
assert 'OTX request failed' in caplog.text
@pytest.mark.asyncio
@@ -112,12 +112,12 @@ class TestOtx:
monkeypatch.setattr(otxsearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = otxsearch.SearchOtx(TestOtx.domain())
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
assert search.execution_status == 'failed'
assert search.stop_reason == 'transport-error'
assert report.status == 'failed'
assert report.stop_reason == 'transport-error'
@pytest.mark.asyncio
async def test_http_failure_is_attributed(self, monkeypatch: pytest.MonkeyPatch) -> None:
@@ -127,12 +127,12 @@ class TestOtx:
monkeypatch.setattr(otxsearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = otxsearch.SearchOtx(TestOtx.domain())
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
assert search.execution_status == 'failed'
assert search.stop_reason == 'http-503'
assert report.status == 'failed'
assert report.stop_reason == 'http-503'
@pytest.mark.asyncio
async def test_cancellation_propagates(self, monkeypatch: pytest.MonkeyPatch) -> None:
@@ -168,8 +168,9 @@ class TestOtx:
monkeypatch.setattr(otxsearch.asyncio, 'sleep', fake_sleep)
search = otxsearch.SearchOtx(TestOtx.domain())
await search.process()
report = await search.process()
assert report is None
assert waits == [2]
assert await search.get_hostnames() == {'api.example.com'}
assert await search.get_ips() == {'192.0.2.1'}
@@ -206,14 +207,14 @@ class TestOtx:
search = otxsearch.SearchOtx(TestOtx.domain())
with caplog.at_level(logging.INFO, logger=otxsearch.__name__):
await search.process()
report = await search.process()
assert responses == []
assert waits == expected_waits
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
assert search.execution_status == 'rate-limited'
assert search.stop_reason == 'http-429'
assert report.status == 'rate-limited'
assert report.stop_reason == 'http-429'
assert 'OTX request failed with HTTP 429' in caplog.text
tests/discovery/test_projectdiscovery.py
+24 -24
View File
@@ -30,11 +30,11 @@ async def test_http_failure_is_reported_without_results(
search = projectdiscovery.SearchDiscovery('example.com')
with caplog.at_level(logging.INFO, logger=projectdiscovery.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == 'failed'
assert search.stop_reason == 'access-denied'
assert report.status == 'failed'
assert report.stop_reason == 'access-denied'
assert 'ProjectDiscovery request failed with HTTP 403' in caplog.text
@@ -48,11 +48,11 @@ async def test_http_429_is_rate_limited(monkeypatch: pytest.MonkeyPatch) -> None
monkeypatch.setattr(projectdiscovery.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = projectdiscovery.SearchDiscovery('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == 'rate-limited'
assert search.stop_reason == 'http-429'
assert report.status == 'rate-limited'
assert report.stop_reason == 'http-429'
@pytest.mark.parametrize('key', ['', ' '])
@@ -73,11 +73,11 @@ async def test_provider_unauthorized_payload_is_access_denied(monkeypatch: pytes
monkeypatch.setattr(projectdiscovery.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = projectdiscovery.SearchDiscovery('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == 'failed'
assert search.stop_reason == 'access-denied'
assert report.status == 'failed'
assert report.stop_reason == 'access-denied'
@pytest.mark.asyncio
@@ -94,11 +94,11 @@ async def test_malformed_response_is_reported(
search = projectdiscovery.SearchDiscovery('example.com')
with caplog.at_level(logging.INFO, logger=projectdiscovery.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
assert 'ProjectDiscovery returned malformed data' in caplog.text
@@ -122,11 +122,11 @@ async def test_success_preserves_supported_subdomain_shapes(
search = projectdiscovery.SearchDiscovery('example.com')
with caplog.at_level(logging.INFO, logger=projectdiscovery.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com', 'www.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
assert caplog.text.count('ProjectDiscovery ignored a malformed subdomain item') == 2
@@ -144,11 +144,11 @@ async def test_malformed_subdomain_collection_is_reported(
search = projectdiscovery.SearchDiscovery('example.com')
with caplog.at_level(logging.INFO, logger=projectdiscovery.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
assert 'ProjectDiscovery returned malformed subdomain data' in caplog.text
@@ -166,11 +166,11 @@ async def test_fetch_exception_is_transport_failure(
search = projectdiscovery.SearchDiscovery('example.com')
with caplog.at_level(logging.INFO, logger=projectdiscovery.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == 'failed'
assert search.stop_reason == 'transport-error'
assert report.status == 'failed'
assert report.stop_reason == 'transport-error'
assert 'private transport details' not in caplog.text
@@ -199,11 +199,11 @@ async def test_parser_exception_preserves_valid_partial_results(
search = projectdiscovery.SearchDiscovery('example.com')
with caplog.at_level(logging.INFO, logger=projectdiscovery.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
assert 'private provider payload' not in caplog.text
tests/discovery/test_provider_session_lifecycle.py
+2 -2
View File
@@ -48,13 +48,13 @@ async def test_censys_pagination_preserves_provider_cookies(
try:
source = censysearch.SearchCensys('example.com', limit=2)
await source.process()
report = await source.process()
finally:
await runner.cleanup()
assert requests == [(None, None), ('page-two', 'ready')]
assert await source.get_hostnames() == {'one.example.com', 'two.example.com'}
assert source.execution_status == 'completed'
assert report is None
@pytest.mark.asyncio
tests/discovery/test_robtex.py
+23 -15
View File
@@ -29,8 +29,9 @@ async def test_robtex_does_not_send_domain_to_reverse_ip_endpoint(monkeypatch: p
monkeypatch.setattr(robtex.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = robtex.SearchRobtex('example.com')
await search.process()
report = await search.process()
assert report is None
assert requested_urls == ['https://freeapi.robtex.com/pdns/forward/example.com']
assert await search.get_ips() == {'192.0.2.1'}
@@ -52,8 +53,9 @@ async def test_robtex_collects_ipv4_and_ipv6_addresses(monkeypatch: pytest.Monke
monkeypatch.setattr(robtex.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = robtex.SearchRobtex('example.com')
await search.process()
report = await search.process()
assert report is None
assert await search.get_ips() == {'192.0.2.1', '2001:db8::1'}
@@ -69,8 +71,14 @@ async def test_robtex_handles_empty_responses(
monkeypatch.setattr(robtex.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = robtex.SearchRobtex('example.com')
await search.process()
report = await search.process()
if response:
assert report is None
else:
assert report is not None
assert report.status == 'failed'
assert report.stop_reason == 'transport-error'
assert await search.get_ips() == set()
@@ -82,10 +90,10 @@ async def test_robtex_reports_malformed_response(monkeypatch: pytest.MonkeyPatch
monkeypatch.setattr(robtex.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = robtex.SearchRobtex('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -100,12 +108,12 @@ async def test_robtex_attributes_provider_failures(
search = robtex.SearchRobtex('example.com')
with caplog.at_level(logging.INFO, logger=robtex.__name__):
await search.process()
report = await search.process()
assert await search.get_ips() == set()
assert 'Robtex API error' in caplog.text
assert search.execution_status == 'failed'
assert search.stop_reason == 'transport-error'
assert report.status == 'failed'
assert report.stop_reason == 'transport-error'
@pytest.mark.asyncio
@@ -121,12 +129,12 @@ async def test_robtex_attributes_http_failures(
search = robtex.SearchRobtex('example.com')
with caplog.at_level(logging.INFO, logger=robtex.__name__):
await search.process()
report = await search.process()
assert await search.get_ips() == set()
assert 'Robtex request failed with HTTP 503' in caplog.text
assert search.execution_status == 'failed'
assert search.stop_reason == 'http-503'
assert report.status == 'failed'
assert report.stop_reason == 'http-503'
@pytest.mark.asyncio
@@ -137,10 +145,10 @@ async def test_robtex_attributes_rate_limits(monkeypatch: pytest.MonkeyPatch) ->
monkeypatch.setattr(robtex.AsyncFetcher, 'fetch_all', rate_limited)
search = robtex.SearchRobtex('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'rate-limited'
assert search.stop_reason == 'http-429'
assert report.status == 'rate-limited'
assert report.stop_reason == 'http-429'
@pytest.mark.asyncio
tests/discovery/test_securityscorecard.py
+12 -11
View File
@@ -2,8 +2,7 @@ from __future__ import annotations
import asyncio
import contextlib
from collections.abc import AsyncIterator
from typing import Any
from typing import TYPE_CHECKING, Any
import pytest
@@ -11,6 +10,9 @@ from theHarvester.discovery import securityscorecard
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import FetcherResponse
if TYPE_CHECKING:
from collections.abc import AsyncIterator
@pytest.mark.provider_contract('securityscorecard')
@pytest.mark.asyncio
@@ -52,14 +54,13 @@ async def test_process_paginates_documented_domain_and_ip_asset_routes(monkeypat
monkeypatch.setattr(securityscorecard.AsyncFetcher, 'fetch', fake_fetch)
monkeypatch.setattr(securityscorecard.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = securityscorecard.SearchSecurityScorecard('example.com', 3)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_hostnames() == {'api.example.com', 'www.example.com'}
assert await search.get_ips() == {'192.0.2.1', '198.51.100.2', '2001:db8::1'}
assert await search.get_score() == 92
assert await search.get_grades() == {'overall': 'A', 'network_security': 'A'}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
assert calls[0] == {
'session': session,
'url': 'https://api.securityscorecard.io/companies/example.com',
@@ -120,10 +121,10 @@ async def test_provider_failures_are_truthful(
monkeypatch.setattr(securityscorecard.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(securityscorecard.AsyncFetcher, 'fetch', fake_fetch)
search = securityscorecard.SearchSecurityScorecard('example.com', 10)
await search.process()
report = await search.process()
assert search.execution_status == status
assert search.stop_reason == reason
assert report.status == status
assert report.stop_reason == reason
@pytest.mark.asyncio
@@ -148,12 +149,12 @@ async def test_malformed_asset_rows_preserve_valid_partial_results(monkeypatch:
monkeypatch.setattr(securityscorecard.AsyncFetcher, 'fetch', fake_fetch)
monkeypatch.setattr(securityscorecard.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = securityscorecard.SearchSecurityScorecard('example.com', 2)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert await search.get_ips() == {'192.0.2.1'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
tests/discovery/test_securitytrails.py
+6 -6
View File
@@ -52,12 +52,12 @@ async def test_process_reuses_session_and_parses_scoped_evidence(monkeypatch: py
monkeypatch.setattr(securitytrailssearch.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(securitytrailssearch.AsyncFetcher, 'fetch', fake_fetch)
search = securitytrailssearch.SearchSecuritytrail('example.com')
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_hostnames() == {'api.example.com', 'www.example.com'}
assert await search.get_ips() == {'192.0.2.1', '2001:db8::1'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
assert [call['url'] for call in calls] == [
'https://api.securitytrails.com/v1/domain/example.com',
'https://api.securitytrails.com/v1/domain/example.com/subdomains',
@@ -102,10 +102,10 @@ async def test_first_request_failures_are_truthful(
monkeypatch.setattr(securitytrailssearch.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(securitytrailssearch.AsyncFetcher, 'fetch', fake_fetch)
search = securitytrailssearch.SearchSecuritytrail('example.com')
await search.process()
report = await search.process()
assert search.execution_status == status
assert search.stop_reason == reason
assert report.status == status
assert report.stop_reason == reason
@pytest.mark.asyncio
tests/discovery/test_sherlockeye.py
+22 -24
View File
@@ -73,7 +73,7 @@ async def test_process_uses_one_shared_provider_session(monkeypatch: pytest.Monk
monkeypatch.setattr(sherlockeye.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = sherlockeye.SearchSherlockeye('example.com')
await search.process(proxy=True)
report = await search.process(proxy=True)
assert calls == [
{
@@ -89,8 +89,7 @@ async def test_process_uses_one_shared_provider_session(monkeypatch: pytest.Monk
}
]
assert exited is True
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.asyncio
@@ -138,13 +137,12 @@ async def test_process_extracts_domain_intelligence(monkeypatch) -> None:
monkeypatch.setattr(sherlockeye.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = sherlockeye.SearchSherlockeye('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'sub.example.com', 'www.example.com', 'api.example.com'}
assert await search.get_emails() == {'user@example.com'}
assert await search.get_ips() == {'203.0.113.10'}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -158,15 +156,15 @@ async def test_process_handles_api_error(monkeypatch, caplog) -> None:
caplog.set_level(logging.INFO, logger=sherlockeye.__name__)
search = sherlockeye.SearchSherlockeye('example.com')
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_emails() == set()
assert await search.get_ips() == set()
assert 'provider-secret-payload' not in caplog.text
assert '401' in caplog.text
assert search.execution_status == 'failed'
assert search.stop_reason == 'access-denied'
assert report.status == 'failed'
assert report.stop_reason == 'access-denied'
@pytest.mark.asyncio
@@ -180,12 +178,12 @@ async def test_process_does_not_log_provider_error_message(monkeypatch, caplog)
caplog.set_level(logging.INFO, logger=sherlockeye.__name__)
search = sherlockeye.SearchSherlockeye('example.com')
await search.process()
report = await search.process()
assert 'provider-secret-payload' not in caplog.text
assert 'API error' in caplog.text
assert search.execution_status == 'failed'
assert search.stop_reason == 'provider-error'
assert report.status == 'failed'
assert report.stop_reason == 'provider-error'
@pytest.mark.parametrize(
@@ -206,10 +204,10 @@ async def test_http_failures_are_structured(
monkeypatch.setattr(sherlockeye.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = sherlockeye.SearchSherlockeye('example.com')
await search.process()
report = await search.process()
assert search.execution_status == execution_status
assert search.stop_reason == stop_reason
assert report.status == execution_status
assert report.stop_reason == stop_reason
@pytest.mark.asyncio
@@ -221,17 +219,17 @@ async def test_malformed_response_is_structured(monkeypatch: pytest.MonkeyPatch)
monkeypatch.setattr(sherlockeye.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = sherlockeye.SearchSherlockeye('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
def test_malformed_link_does_not_discard_later_valid_results(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(sherlockeye.Core, 'sherlockeye_key', lambda: 'dummy-key')
search = sherlockeye.SearchSherlockeye('example.com')
search._extract_response(
report = search._extract_response(
{
'success': True,
'data': {
@@ -244,8 +242,8 @@ def test_malformed_link_does_not_discard_later_valid_results(monkeypatch: pytest
)
assert search.totalhosts == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -268,9 +266,9 @@ async def test_transport_failure_and_cancellation_are_distinct(monkeypatch: pyte
monkeypatch.setattr(sherlockeye.AsyncFetcher, 'post_fetch', failed_post_fetch)
search = sherlockeye.SearchSherlockeye('example.com')
await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'transport-error'
report = await search.process()
assert report.status == 'failed'
assert report.stop_reason == 'transport-error'
assert session_exit_count == 1
async def cancelled_post_fetch(*_args: Any, **_kwargs: Any) -> FetcherResponse:
tests/discovery/test_shodan_engine.py
+16 -18
View File
@@ -259,7 +259,7 @@ class TestShodanEngine:
targets = patch_resolution(monkeypatch, shodansearch, ('203.0.113.10', '203.0.113.11'))
search = shodansearch.SearchShodan('WWW.Example.TEST.')
await search.process(proxy=True)
report = await search.process(proxy=True)
assert targets == [('www.example.test', socket.AF_INET)]
assert queried_urls == [
@@ -267,8 +267,8 @@ class TestShodanEngine:
'https://api.shodan.io/shodan/host/203.0.113.11',
]
assert await search.get_hostnames() == {'cdn.example.test'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'provider-errors'
assert report.status == 'failed'
assert report.stop_reason == 'provider-errors'
@pytest.mark.asyncio
async def test_shodan_discovery_paginates_hostname_and_tls_searches_with_scoped_certificate_names(self, monkeypatch):
@@ -369,7 +369,7 @@ class TestShodanEngine:
patch_resolution(monkeypatch, shodansearch)
search = shodansearch.SearchShodan('example.test')
await search.process(proxy=True)
report = await search.process(proxy=True)
assert [(call['query'], call['page']) for call in search_calls] == [
('hostname:example.test', 1),
@@ -391,8 +391,7 @@ class TestShodanEngine:
'subject_cn': '*.example.test',
}
assert hosts['198.51.100.21']['services'][0]['tls'] == {'subject_cn': 'cert.example.test'}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
async def test_shodan_discovery_counts_service_only_evidence_as_a_result(self, monkeypatch):
@@ -413,10 +412,9 @@ class TestShodanEngine:
patch_resolution(monkeypatch, shodansearch)
search = shodansearch.SearchShodan('example.test')
await search.process()
report = await search.process()
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
assert [host.ip for host in await search.get_shodan_hosts()] == ['203.0.113.10']
@pytest.mark.asyncio
@@ -442,10 +440,10 @@ class TestShodanEngine:
patch_resolution(monkeypatch, shodansearch)
search = shodansearch.SearchShodan('example.test')
await search.process()
report = await search.process()
assert search.execution_status == 'partial'
assert search.stop_reason == 'provider-error'
assert report.status == 'failed'
assert report.stop_reason == 'provider-error'
assert search.error_type == 'InvalidResponseError'
assert (await search.get_shodan_hosts())[0].to_details() == {'services': [{'port': 53, 'transport': 'udp'}]}
@@ -466,10 +464,10 @@ class TestShodanEngine:
patch_resolution(monkeypatch, shodansearch)
search = shodansearch.SearchShodan('example.test')
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == expected_reason
assert report.status == ('rate-limited' if status == 429 else 'failed')
assert report.stop_reason == expected_reason
def test_shodan_discovery_requires_a_configured_key(self, monkeypatch):
from theHarvester.discovery import shodansearch
@@ -509,12 +507,12 @@ class TestShodanEngine:
monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json)
search = shodansearch.SearchShodan('example.test')
await search.process()
report = await search.process()
assert not await search.get_hostnames()
assert provider_queries == ['hostname:example.test', 'ssl:example.test']
assert search.execution_status == 'failed'
assert search.stop_reason == 'dns-resolution-failed'
assert report.status == 'failed'
assert report.stop_reason == 'dns-resolution-failed'
@pytest.mark.asyncio
async def test_shodan_direct_request_cancellation_propagates(self, monkeypatch):
tests/discovery/test_sourcegraph.py
+43 -45
View File
@@ -97,7 +97,7 @@ async def test_sourcegraph_uses_fixed_chunk_query_and_collects_descendants(
calls = install_stream(monkeypatch, records)
search = sourcegraph.SearchSourcegraph(' Scope.TEST. ', limit=1)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert calls == [
{
@@ -118,8 +118,7 @@ async def test_sourcegraph_uses_fixed_chunk_query_and_collects_descendants(
}
]
assert await search.get_hostnames() == ['api.scope.test', 'deep.api.scope.test']
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -150,11 +149,11 @@ async def test_sourcegraph_rejects_unsafe_targets_before_request(
calls = install_stream(monkeypatch, allow_reserved_target=False)
search = sourcegraph.SearchSourcegraph(target, limit=500)
await search.process()
report = await search.process()
assert calls == []
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-target'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-target'
assert await search.get_hostnames() == []
@@ -177,10 +176,10 @@ async def test_sourcegraph_attributes_http_failures(
install_stream(monkeypatch, status=http_status)
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
await search.process()
report = await search.process()
assert search.execution_status == execution_status
assert search.stop_reason == stop_reason
assert report.status == execution_status
assert report.stop_reason == stop_reason
@pytest.mark.asyncio
@@ -192,10 +191,10 @@ async def test_sourcegraph_attributes_stream_failures(
install_stream(monkeypatch, error=sourcegraph.ResponseStreamError(reason))
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == reason
assert report.status == 'failed'
assert report.stop_reason == reason
@pytest.mark.asyncio
@@ -213,11 +212,11 @@ async def test_sourcegraph_preserves_hosts_before_stream_failure(monkeypatch: py
)
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
await search.process()
report = await search.process()
assert await search.get_hostnames() == ['api.scope.test']
assert search.execution_status == 'partial'
assert search.stop_reason == 'response-limit'
assert report.status == 'failed'
assert report.stop_reason == 'response-limit'
@pytest.mark.asyncio
@@ -240,10 +239,10 @@ async def test_sourcegraph_rejects_malformed_provider_events(
install_stream(monkeypatch, (bad_record, event('done', {})))
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -263,10 +262,10 @@ async def test_sourcegraph_requires_terminal_progress_and_done(
install_stream(monkeypatch, records)
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -287,10 +286,10 @@ async def test_sourcegraph_attributes_provider_error(
install_stream(monkeypatch, tuple(records))
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
await search.process()
report = await search.process()
assert search.execution_status == ('partial' if with_host else 'failed')
assert search.stop_reason == 'provider-error'
assert report.status == 'failed'
assert report.stop_reason == 'provider-error'
@pytest.mark.asyncio
@@ -308,11 +307,11 @@ async def test_sourcegraph_final_skipped_progress_marks_provider_limited(
install_stream(monkeypatch, records)
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
await search.process()
report = await search.process()
assert await search.get_hostnames() == ['api.scope.test']
assert search.execution_status == 'partial'
assert search.stop_reason == 'provider-limited'
assert report.status == 'partial'
assert report.stop_reason == 'provider-limited'
@pytest.mark.asyncio
@@ -330,10 +329,9 @@ async def test_sourcegraph_uses_only_the_final_progress_skipped_value(
)
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
await search.process()
report = await search.process()
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.asyncio
@@ -348,10 +346,10 @@ async def test_sourcegraph_rejects_match_after_terminal_progress(monkeypatch: py
)
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -359,10 +357,10 @@ async def test_sourcegraph_rejects_events_after_done(monkeypatch: pytest.MonkeyP
install_stream(monkeypatch, (event('done', {}), event('progress', {'done': True})))
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -386,11 +384,11 @@ async def test_sourcegraph_preserves_prefix_at_hostname_limit(monkeypatch: pytes
monkeypatch.setattr(sourcegraph.SearchSourcegraph, 'MAX_HOSTNAMES', 1)
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
await search.process()
report = await search.process()
assert await search.get_hostnames() == ['one.scope.test']
assert search.execution_status == 'partial'
assert search.stop_reason == 'response-limit'
assert report.status == 'failed'
assert report.stop_reason == 'response-limit'
@pytest.mark.asyncio
@@ -408,11 +406,11 @@ async def test_sourcegraph_preserves_prefix_at_event_limit(monkeypatch: pytest.M
monkeypatch.setattr(sourcegraph.SearchSourcegraph, 'MAX_EVENTS', 1)
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
await search.process()
report = await search.process()
assert await search.get_hostnames() == ['api.scope.test']
assert search.execution_status == 'partial'
assert search.stop_reason == 'response-limit'
assert report.status == 'failed'
assert report.stop_reason == 'response-limit'
@pytest.mark.asyncio
@@ -421,10 +419,10 @@ async def test_sourcegraph_treats_deep_json_as_invalid(monkeypatch: pytest.Monke
install_stream(monkeypatch, (f'event: alert\ndata: {nested}',))
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
tests/discovery/test_subdomainfinderc99.py
+11 -12
View File
@@ -40,11 +40,10 @@ async def test_successful_response_returns_only_scoped_hostnames(monkeypatch) ->
monkeypatch.setattr(subdomainfinderc99.asyncio, 'sleep', no_sleep)
search = subdomainfinderc99.SearchSubdomainfinderc99('example.test')
await search.process()
report = await search.process()
assert set(await search.get_hostnames()) == {'api.example.test'}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
assert calls == [('get', session), ('post', session)]
assert session_exited is True
@@ -62,11 +61,11 @@ async def test_empty_initial_response_is_transport_failure(monkeypatch) -> None:
monkeypatch.setattr(subdomainfinderc99.AsyncFetcher, 'fetch', fake_fetch)
search = subdomainfinderc99.SearchSubdomainfinderc99('example.test')
await search.process()
report = await search.process()
assert not await search.get_hostnames()
assert search.execution_status == 'failed'
assert search.stop_reason == 'transport-error'
assert report.status == 'failed'
assert report.stop_reason == 'transport-error'
@pytest.mark.parametrize(
@@ -102,11 +101,11 @@ async def test_scan_failures_are_structured(
monkeypatch.setattr(subdomainfinderc99.asyncio, 'sleep', no_sleep)
search = subdomainfinderc99.SearchSubdomainfinderc99('example.test')
await search.process()
report = await search.process()
assert not await search.get_hostnames()
assert search.execution_status == status
assert search.stop_reason == reason
assert report.status == status
assert report.stop_reason == reason
@pytest.mark.parametrize(
@@ -136,10 +135,10 @@ async def test_initial_failures_are_structured(
monkeypatch.setattr(subdomainfinderc99.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(subdomainfinderc99.AsyncFetcher, 'fetch', fake_fetch)
search = subdomainfinderc99.SearchSubdomainfinderc99('example.test')
await search.process()
report = await search.process()
assert search.execution_status == status
assert search.stop_reason == reason
assert report.status == status
assert report.stop_reason == reason
@pytest.mark.asyncio
tests/discovery/test_urlscan.py
+34 -43
View File
@@ -8,6 +8,7 @@ import pytest
from theHarvester.discovery import urlscan
from theHarvester.lib.core import FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
class ProviderSession:
@@ -82,7 +83,7 @@ async def test_process_collects_sequential_pages_and_preserves_all_routes(
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com', 2)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_hostnames() == {'first.example.com', 'second.example.com'}
assert await search.get_ips() == {'192.0.2.10', '2001:db8::10'}
@@ -115,8 +116,7 @@ async def test_process_collects_sequential_pages_and_preserves_all_routes(
assert all(call['include_metadata'] is True for call in calls)
assert all('request_timeout' not in call for call in calls)
assert provider_session.exited is True
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -149,9 +149,10 @@ async def test_repeated_asn_relationship_is_retained_once_per_source_run(monkeyp
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com', 10)
await search.process()
report = await search.process()
assert len(await search.get_asn_attributions()) == 2
assert report is None
@pytest.mark.asyncio
@@ -162,14 +163,13 @@ async def test_valid_empty_response_is_completed(monkeypatch: pytest.MonkeyPatch
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com', 10)
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
assert await search.get_urls() == set()
assert await search.get_asns() == set()
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.asyncio
@@ -189,11 +189,10 @@ async def test_missing_optional_fields_are_skipped(monkeypatch: pytest.MonkeyPat
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com', 10)
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.asyncio
@@ -218,12 +217,11 @@ async def test_malformed_nested_fields_preserve_valid_partial_results(monkeypatc
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com', 10)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'valid.example.com'}
assert await search.get_ips() == set()
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report == SourceExecutionReport('failed', 'invalid-response')
@pytest.mark.asyncio
@@ -264,14 +262,13 @@ async def test_results_are_typed_and_scoped_before_insertion(monkeypatch: pytest
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com', 10)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert await search.get_ips() == {'2001:db8::10'}
assert await search.get_urls() == {'https://portal.example.com/path'}
assert await search.get_asns() == {'AS64496'}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.parametrize(
@@ -299,26 +296,26 @@ async def test_failed_first_page_is_attributed(
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com', 10)
await search.process()
report = await search.process()
assert search.execution_status == execution_status
assert search.stop_reason == stop_reason
assert report == SourceExecutionReport(execution_status, stop_reason)
@pytest.mark.parametrize(
('second_response', 'stop_reason'),
('second_response', 'execution_status', 'stop_reason'),
[
(None, 'transport-error'),
(FetcherResponse(body={}, status=403, headers={}), 'access-denied'),
(FetcherResponse(body={}, status=429, headers={}), 'http-429'),
(FetcherResponse(body={}, status=503, headers={}), 'http-503'),
(FetcherResponse(body={'results': {}}, status=200, headers={}), 'invalid-response'),
(None, 'failed', 'transport-error'),
(FetcherResponse(body={}, status=403, headers={}), 'failed', 'access-denied'),
(FetcherResponse(body={}, status=429, headers={}), 'rate-limited', 'http-429'),
(FetcherResponse(body={}, status=503, headers={}), 'failed', 'http-503'),
(FetcherResponse(body={'results': {}}, status=200, headers={}), 'failed', 'invalid-response'),
],
)
@pytest.mark.asyncio
async def test_later_failure_preserves_partial_results(
monkeypatch: pytest.MonkeyPatch,
second_response: FetcherResponse | None,
execution_status: str,
stop_reason: str,
) -> None:
responses = [
@@ -336,11 +333,10 @@ async def test_later_failure_preserves_partial_results(
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com', 10)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'first.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == stop_reason
assert report == SourceExecutionReport(execution_status, stop_reason)
@pytest.mark.asyncio
@@ -351,10 +347,9 @@ async def test_fetch_exception_is_transport_failure(monkeypatch: pytest.MonkeyPa
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com', 10)
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'transport-error'
assert report == SourceExecutionReport('failed', 'transport-error')
@pytest.mark.asyncio
@@ -373,12 +368,11 @@ async def test_missing_cursor_stops_after_first_page(monkeypatch: pytest.MonkeyP
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com', 10)
await search.process()
report = await search.process()
assert calls == 1
assert await search.get_hostnames() == {'first.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-cursor'
assert report == SourceExecutionReport('failed', 'invalid-cursor')
@pytest.mark.asyncio
@@ -405,12 +399,11 @@ async def test_repeated_cursor_stops_without_a_third_request(monkeypatch: pytest
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com', 10)
await search.process()
report = await search.process()
assert calls == 2
assert await search.get_hostnames() == {'first.example.com', 'second.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'repeated-cursor'
assert report == SourceExecutionReport('failed', 'repeated-cursor')
@pytest.mark.asyncio
@@ -443,15 +436,14 @@ async def test_pagination_continues_beyond_the_removed_local_page_ceiling(monkey
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com', 10_001)
await search.process()
report = await search.process()
assert calls == [
{'q': 'domain:example.com', 'size': 10_000},
{'q': 'domain:example.com', 'size': 1, 'search_after': '1,cursor-10000'},
]
assert await search.get_hostnames() == {f'page-{page}.example.com' for page in range(1, 10_002)}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@@ -470,12 +462,11 @@ async def test_operator_limit_sets_page_size_and_stops_without_an_extra_request(
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com', 10)
await search.process()
report = await search.process()
assert calls == [{'q': 'domain:example.com', 'size': 10}]
assert len(await search.get_hostnames()) == 10
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
@pytest.mark.parametrize('limit', [0, -1, True, 1.5])
tests/discovery/test_virustotal.py
+11 -12
View File
@@ -71,7 +71,7 @@ async def test_process_paginates_without_fixed_sleeps_and_keeps_scoped_evidence(
monkeypatch.setattr(virustotal.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(virustotal.AsyncFetcher, 'fetch', fake_fetch)
search = virustotal.SearchVirustotal('example.com', limit=10)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_hostnames() == {
'api.example.com',
@@ -79,8 +79,7 @@ async def test_process_paginates_without_fixed_sleeps_and_keeps_scoped_evidence(
'mail.example.com',
'tls.example.com',
}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
assert [call['params'] for call in calls] == [{'limit': 10}, {'limit': 9, 'cursor': 'next-page'}]
assert all(call['session'] is session for call in calls)
assert session_exited is True
@@ -146,10 +145,10 @@ async def test_provider_failures_are_truthful(
monkeypatch.setattr(virustotal.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(virustotal.AsyncFetcher, 'fetch', fake_fetch)
search = virustotal.SearchVirustotal('example.com', limit=10)
await search.process()
report = await search.process()
assert search.execution_status == status
assert search.stop_reason == reason
assert report.status == status
assert report.stop_reason == reason
@pytest.mark.asyncio
@@ -174,11 +173,11 @@ async def test_later_rate_limit_preserves_partial_results(monkeypatch: pytest.Mo
monkeypatch.setattr(virustotal.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(virustotal.AsyncFetcher, 'fetch', fake_fetch)
search = virustotal.SearchVirustotal('example.com', limit=10)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'http-429'
assert report.status == 'rate-limited'
assert report.stop_reason == 'http-429'
@pytest.mark.asyncio
@@ -200,11 +199,11 @@ async def test_repeated_cursor_stops_without_spending_more_quota(monkeypatch: py
monkeypatch.setattr(virustotal.AsyncFetcher, 'fetch', fake_fetch)
search = virustotal.SearchVirustotal('example.com', limit=10)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'repeated-cursor'
assert report.status == 'failed'
assert report.stop_reason == 'repeated-cursor'
assert responses == []
tests/discovery/test_waybackarchive.py
+32 -19
View File
@@ -26,8 +26,9 @@ async def test_process_collects_more_than_one_cdx_page(monkeypatch: pytest.Monke
monkeypatch.setattr(waybackarchive.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = waybackarchive.SearchWaybackarchive('example.com')
await search.process()
report = await search.process()
assert report is None
assert await search.get_hostnames() == {f'host-{index}.example.com' for index in range(125)}
assert [query.get('resumeKey') for query in requests if query['url'] == ['*.example.com']] == [
None,
@@ -51,8 +52,9 @@ async def test_process_stops_when_a_resume_key_repeats(monkeypatch: pytest.Monke
monkeypatch.setattr(waybackarchive.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = waybackarchive.SearchWaybackarchive('example.com')
await search.process()
report = await search.process()
assert report is None
wildcard_requests = [query for query in requests if query['url'] == ['*.example.com']]
assert len(wildcard_requests) == 2
assert await search.get_hostnames() == {'api.example.com'}
@@ -78,8 +80,9 @@ async def test_process_normalizes_and_scope_checks_each_page(monkeypatch: pytest
monkeypatch.setattr(waybackarchive.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = waybackarchive.SearchWaybackarchive('example.com')
await search.process()
report = await search.process()
assert report is None
assert await search.get_hostnames() == {'api.example.com', 'example.com'}
@@ -95,8 +98,9 @@ async def test_process_normalizes_the_requested_domain(monkeypatch: pytest.Monke
monkeypatch.setattr(waybackarchive.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = waybackarchive.SearchWaybackarchive('Example.COM.')
await search.process()
report = await search.process()
assert report is None
assert await search.get_hostnames() == {'api.example.com'}
assert requests[0]['url'] == ['*.example.com']
@@ -118,12 +122,12 @@ async def test_process_keeps_partial_results_when_a_later_page_times_out(
search = waybackarchive.SearchWaybackarchive('example.com')
with caplog.at_level(logging.INFO, logger=waybackarchive.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com', 'example.com'}
assert 'Wayback Archive API error for pattern *.example.com' in caplog.text
assert search.execution_status == 'partial'
assert search.stop_reason == 'request-error'
assert report.status == 'failed'
assert report.stop_reason == 'request-error'
@pytest.mark.asyncio
@@ -150,12 +154,16 @@ async def test_process_ignores_empty_html_and_non_text_responses(
search = waybackarchive.SearchWaybackarchive('example.com')
with caplog.at_level(logging.INFO, logger=waybackarchive.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == set()
assert expected_log in caplog.text
assert search.execution_status == expected_status
assert search.stop_reason == expected_stop_reason
if expected_status is None:
assert report is None
else:
assert report is not None
assert report.status == expected_status
assert report.stop_reason == expected_stop_reason
@pytest.mark.asyncio
@@ -178,11 +186,14 @@ async def test_process_respects_the_per_query_page_bound(
search = waybackarchive.SearchWaybackarchive('example.com')
with caplog.at_level(logging.INFO, logger=waybackarchive.__name__):
await search.process()
report = await search.process()
assert wildcard_requests == 2
assert await search.get_hostnames() == {'host-1.example.com', 'host-2.example.com'}
assert 'Wayback Archive page limit reached for pattern *.example.com; results may be incomplete' in caplog.text
assert report is not None
assert report.status == 'partial'
assert report.stop_reason == 'page-limit'
@pytest.mark.asyncio
@@ -205,11 +216,11 @@ async def test_process_retains_partial_results_at_the_runtime_limit(
search = waybackarchive.SearchWaybackarchive('example.com')
with caplog.at_level(logging.INFO, logger=waybackarchive.__name__):
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'runtime-limit'
assert report.status == 'failed'
assert report.stop_reason == 'runtime-limit'
assert 'Wayback Archive page 1: hosts=1' in caplog.text
assert 'example.com' not in caplog.text
@@ -225,10 +236,12 @@ async def test_process_stops_at_the_requested_result_limit(monkeypatch: pytest.M
monkeypatch.setattr(waybackarchive.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = waybackarchive.SearchWaybackarchive('example.com', limit=2)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'one.example.com', 'two.example.com'}
assert search.stop_reason == 'result-limit'
assert report is not None
assert report.status == 'completed'
assert report.stop_reason == 'result-limit'
assert len(requested_urls) == 1
@@ -245,11 +258,11 @@ async def test_process_keeps_an_earlier_failure_when_a_later_pattern_reaches_the
monkeypatch.setattr(waybackarchive.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = waybackarchive.SearchWaybackarchive('example.com', limit=1)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
pytestmark = pytest.mark.provider_contract('waybackarchive')
tests/discovery/test_whoisxml.py
+11 -12
View File
@@ -77,11 +77,10 @@ async def test_response_body_is_not_logged_and_scoped_records_are_returned(
monkeypatch.setattr(whoisxml.AsyncFetcher, 'fetch', fake_fetch)
search = whoisxml.SearchWhoisXML('example.com', 3)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_hostnames() == {'api.example.com', 'www.example.com'}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
assert [call['url'] for call in calls] == ['https://subdomains.whoisxmlapi.com/api/v2'] * 2
assert all(call['session'] is provider_session for call in calls)
assert [call['params'] for call in calls] == [
@@ -131,10 +130,10 @@ async def test_provider_failures_are_truthful(
monkeypatch.setattr(whoisxml.AsyncFetcher, 'fetch', fake_fetch)
search = whoisxml.SearchWhoisXML('example.com', 10)
await search.process()
report = await search.process()
assert search.execution_status == status
assert search.stop_reason == reason
assert report.status == status
assert report.stop_reason == reason
@pytest.mark.asyncio
@@ -156,11 +155,11 @@ async def test_malformed_rows_preserve_valid_partial_results(monkeypatch: pytest
monkeypatch.setattr(whoisxml.AsyncFetcher, 'fetch', fake_fetch)
search = whoisxml.SearchWhoisXML('example.com', 10)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'ok.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
@@ -186,11 +185,11 @@ async def test_later_page_failure_preserves_partial_results(monkeypatch: pytest.
monkeypatch.setattr(whoisxml.AsyncFetcher, 'fetch', fake_fetch)
search = whoisxml.SearchWhoisXML('example.com', 10)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'http-429'
assert report.status == 'rate-limited'
assert report.stop_reason == 'http-429'
@pytest.mark.asyncio
tests/discovery/test_zoomeyesearch.py
+12 -12
View File
@@ -58,15 +58,14 @@ async def test_process_reuses_session_and_collects_all_capabilities(monkeypatch:
monkeypatch.setattr(zoomeyesearch.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(zoomeyesearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = zoomeyesearch.SearchZoomEye('example.com', 2)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert await search.get_hostnames() == {'api.example.com', 'host.example.com', 'portal.example.com'}
assert await search.get_ips() == {'192.0.2.1'}
assert await search.get_asns() == {'AS64500'}
assert await search.get_urls() == {'https://portal.example.com/v1'}
assert await search.get_emails() == {'admin@example.com'}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
assert [call['url'] for call in calls] == [search.baseurl]
assert all(call['session'] is session for call in calls)
assert session_exited is True
@@ -119,10 +118,10 @@ async def test_provider_failures_are_truthful(
monkeypatch.setattr(zoomeyesearch.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(zoomeyesearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = zoomeyesearch.SearchZoomEye('example.com', 1)
await search.process()
report = await search.process()
assert search.execution_status == status
assert search.stop_reason == reason
assert report.status == status
assert report.stop_reason == reason
@pytest.mark.asyncio
@@ -151,11 +150,11 @@ async def test_empty_pages_do_not_hide_later_provider_results(monkeypatch: pytes
monkeypatch.setattr(zoomeyesearch.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(zoomeyesearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = zoomeyesearch.SearchZoomEye('example.com', 70_000)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'late.example.com'}
assert responses == []
assert search.execution_status == 'completed'
assert report is None
@pytest.mark.asyncio
@@ -175,8 +174,9 @@ async def test_numbered_pages_keep_a_stable_size_and_slice_the_final_page(monkey
monkeypatch.setattr(zoomeyesearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = zoomeyesearch.SearchZoomEye('example.com', 10_005)
await search.process()
report = await search.process()
assert report is None
assert [(call['page'], call['pagesize']) for call in calls] == [(1, 10_000), (2, 10_000)]
@@ -200,11 +200,11 @@ async def test_early_malformed_rows_and_later_valid_evidence_are_partial(monkeyp
monkeypatch.setattr(zoomeyesearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = zoomeyesearch.SearchZoomEye('example.com', 2)
await search.process()
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.asyncio
tests/lib/test_source_runner.py
+54 -41
View File
@@ -12,6 +12,7 @@ from theHarvester.discovery.constants import MissingKeyError
from theHarvester.lib.asn_attribution import AsnAttributionObservation
from theHarvester.lib.completed_result import ResultObservation, SourceExecution
from theHarvester.lib.source_catalog import SOURCE_SPECS
from theHarvester.lib.source_execution import SourceExecutionReport
from theHarvester.lib.source_runner import (
SOURCE_FACTORIES,
SourceJob,
@@ -34,6 +35,7 @@ def test_source_contracts_are_immutable() -> None:
request = SourceRequest('APIS-GURU', 'example.test', 25, 5, True, True)
job = SourceJob(request)
outcome = SourceOutcome(SourceExecution('apis-guru', 'completed', 0, 0))
report = SourceExecutionReport('failed', 'provider-failure')
with pytest.raises(FrozenInstanceError):
request.target = 'changed.test' # type: ignore[misc]
@@ -41,9 +43,29 @@ def test_source_contracts_are_immutable() -> None:
job.request = request # type: ignore[misc]
with pytest.raises(FrozenInstanceError):
outcome.observations = () # type: ignore[misc]
with pytest.raises(FrozenInstanceError):
report.stop_reason = 'changed' # type: ignore[misc]
assert request.source == 'apis-guru'
@pytest.mark.parametrize(
('status', 'stop_reason', 'message'),
[
('skipped', 'missing-key', 'cannot report execution status'),
('unknown', 'provider-failure', 'cannot report execution status'),
('failed', '', 'stop reason must not be empty'),
('failed', ' ', 'stop reason must not be empty'),
],
)
def test_source_execution_report_rejects_invalid_contract_values(
status: str,
stop_reason: str,
message: str,
) -> None:
with pytest.raises(ValueError, match=message):
SourceExecutionReport(status, stop_reason) # type: ignore[arg-type]
def test_source_factories_match_the_catalog() -> None:
assert set(SOURCE_FACTORIES) == set(SOURCE_SPECS)
@@ -199,9 +221,6 @@ def test_factory_constructor_shapes(
@pytest.mark.asyncio
async def test_runner_normalizes_only_declared_apis_guru_routes(monkeypatch: pytest.MonkeyPatch) -> None:
class FakeApisGuru:
execution_status = 'completed'
stop_reason = None
def __init__(self, target: str, limit: int) -> None:
assert (target, limit) == ('example.test', 25)
@@ -406,31 +425,28 @@ async def test_runner_reports_normal_zero_yield_as_completed_no_results(monkeypa
assert outcome.execution.result_count == 0
@pytest.mark.parametrize('source', ['builtwith', 'hudsonrock', 'shodan'])
@pytest.mark.parametrize(
('reported_status', 'reported_reason', 'has_results', 'expected_count'),
('report', 'has_results', 'expected_status', 'expected_reason'),
[
('completed', None, False, 0),
('partial', 'provider-partial', True, 1),
('failed', 'provider-failure', False, 0),
('rate-limited', 'http-429', False, 0),
(None, False, 'completed', 'no-results'),
(SourceExecutionReport('completed', 'result-limit'), True, 'completed', 'result-limit'),
(SourceExecutionReport('completed', 'result-limit'), False, 'completed', 'result-limit'),
(SourceExecutionReport('failed', 'provider-failure'), True, 'partial', 'provider-failure'),
(SourceExecutionReport('failed', 'provider-failure'), False, 'failed', 'provider-failure'),
(SourceExecutionReport('rate-limited', 'http-429'), False, 'rate-limited', 'http-429'),
],
)
@pytest.mark.asyncio
async def test_special_sources_share_runner_outcome_semantics(
async def test_runner_combines_adapter_report_with_normalized_evidence(
monkeypatch: pytest.MonkeyPatch,
source: str,
reported_status: str,
reported_reason: str | None,
report: SourceExecutionReport | None,
has_results: bool,
expected_count: int,
expected_status: str,
expected_reason: str | None,
) -> None:
class FakeSpecialSource:
execution_status = reported_status
stop_reason = reported_reason
async def process(self, _proxy: bool) -> None:
return None
class FakeSource:
async def process(self, _proxy: bool) -> SourceExecutionReport | None:
return report
async def get_hostnames(self) -> set[str]:
return {'partial.example.test'} if has_results else set()
@@ -444,35 +460,32 @@ async def test_special_sources_share_runner_outcome_semantics(
async def get_urls(self) -> set[str]:
return set()
async def get_frameworks(self) -> set[str]:
return set()
monkeypatch.setitem(SOURCE_FACTORIES, 'apis-guru', lambda _request: FakeSource())
async def get_languages(self) -> set[str]:
return set()
outcome = await run_source(SourceRequest('apis-guru', 'example.test', 25, 0, False, True))
async def get_servers(self) -> set[str]:
return set()
assert outcome.execution.status == expected_status
assert outcome.execution.stop_reason == expected_reason
assert outcome.execution.result_count == int(has_results)
async def get_cms(self) -> set[str]:
return set()
async def get_analytics(self) -> set[str]:
return set()
@pytest.mark.asyncio
async def test_runner_rejects_legacy_or_untyped_execution_reports(monkeypatch: pytest.MonkeyPatch) -> None:
class InvalidSource:
async def process(self, _proxy: bool) -> object:
return {'status': 'failed', 'stop_reason': 'provider-failure'}
async def get_infostealers(self) -> list[dict[str, object]]:
return []
async def get_hostnames(self) -> set[str]:
raise AssertionError('result getters must not run after a contract violation')
async def get_shodan_hosts(self) -> tuple[()]:
return ()
monkeypatch.setitem(SOURCE_FACTORIES, 'apis-guru', lambda _request: InvalidSource())
monkeypatch.setitem(SOURCE_FACTORIES, source, lambda _request: FakeSpecialSource())
outcome = await run_source(SourceRequest('apis-guru', 'example.test', 25, 0, False, True))
outcome = await run_source(SourceRequest(source, 'example.test', 25, 0, False, True))
assert outcome.execution.status == reported_status
assert outcome.execution.stop_reason == (reported_reason or 'no-results')
assert outcome.execution.result_count == expected_count
assert {observation.source for observation in outcome.observations} == ({source} if has_results else set())
assert outcome.execution.status == 'failed'
assert outcome.execution.error_type == 'ValueError'
assert outcome.execution.result_count == 0
assert outcome.observations == ()
@pytest.mark.asyncio
tests/test_hackertarget_apikey.py
+28 -29
View File
@@ -36,8 +36,9 @@ class TestHackerTargetApiKey:
monkeypatch.setattr(ht_mod.AsyncFetcher, 'fetch_all', fake_fetch_all)
s = ht_mod.SearchHackerTarget('example.com')
await s.process(proxy=True)
report = await s.process(proxy=True)
assert report is None
assert requested_urls == [
'https://api.hackertarget.com/hostsearch/?q=example.com&apikey=TESTKEY',
'https://api.hackertarget.com/reversedns/?q=example.com&apikey=TESTKEY',
@@ -66,8 +67,9 @@ class TestHackerTargetApiKey:
monkeypatch.setattr(ht_mod.AsyncFetcher, 'fetch_all', fake_fetch_all)
s = ht_mod.SearchHackerTarget('example.com')
await s.process()
report = await s.process()
assert report is None
assert requested_urls == [
'https://api.hackertarget.com/hostsearch/?q=example.com',
'https://api.hackertarget.com/reversedns/?q=example.com',
@@ -92,10 +94,10 @@ class TestHackerTargetApiKey:
monkeypatch.setattr(ht_mod.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = ht_mod.SearchHackerTarget('example.com')
await search.process()
report = await search.process()
assert search.execution_status == execution_status
assert search.stop_reason == stop_reason
assert report.status == execution_status
assert report.stop_reason == stop_reason
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
@@ -113,10 +115,10 @@ class TestHackerTargetApiKey:
monkeypatch.setattr(ht_mod.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = ht_mod.SearchHackerTarget('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
@@ -133,10 +135,10 @@ class TestHackerTargetApiKey:
monkeypatch.setattr(ht_mod.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = ht_mod.SearchHackerTarget('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == 'invalid-response'
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
@@ -153,10 +155,9 @@ class TestHackerTargetApiKey:
monkeypatch.setattr(ht_mod.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = ht_mod.SearchHackerTarget('example.com')
await search.process()
report = await search.process()
assert search.execution_status is None
assert search.stop_reason is None
assert report is None
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
@@ -180,10 +181,10 @@ class TestHackerTargetApiKey:
monkeypatch.setattr(ht_mod.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = ht_mod.SearchHackerTarget('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'partial'
assert search.stop_reason == stop_reason
assert report.status == 'partial'
assert report.stop_reason == stop_reason
assert await search.get_hostnames() == {'www.example.com'}
assert await search.get_ips() == {'192.0.2.2'}
@@ -200,10 +201,10 @@ class TestHackerTargetApiKey:
monkeypatch.setattr(ht_mod.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = ht_mod.SearchHackerTarget('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'partial'
assert search.stop_reason == 'http-503'
assert report.status == 'partial'
assert report.stop_reason == 'http-503'
assert await search.get_hostnames() == set()
assert await search.get_ips() == set()
@@ -220,10 +221,10 @@ class TestHackerTargetApiKey:
monkeypatch.setattr(ht_mod.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = ht_mod.SearchHackerTarget('example.com')
await search.process()
report = await search.process()
assert search.execution_status == 'partial'
assert search.stop_reason == 'invalid-response'
assert report.status == 'partial'
assert report.stop_reason == 'invalid-response'
assert await search.get_hostnames() == {'www.example.com', 'ptr.example.com'}
assert await search.get_ips() == {'192.0.2.2', '192.0.2.3'}
@@ -245,10 +246,9 @@ class TestHackerTargetApiKey:
monkeypatch.setattr(ht_mod.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = ht_mod.SearchHackerTarget(target)
await search.process()
report = await search.process()
assert search.execution_status is None
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
@pytest.mark.parametrize(
@@ -274,10 +274,9 @@ class TestHackerTargetApiKey:
monkeypatch.setattr(ht_mod.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = ht_mod.SearchHackerTarget(target)
await search.process()
report = await search.process()
assert search.execution_status is None
assert search.stop_reason is None
assert report is None
@pytest.mark.asyncio
async def test_cancellation_propagates(self, monkeypatch):
tests/test_main.py
+9 -17
View File
@@ -23,6 +23,7 @@ from theHarvester.lib.hostchecker import HostDnsRecords
from theHarvester.lib.network_evidence import PrefixOriginObservation, RpkiValidationObservation
from theHarvester.lib.recursive_dns import RecursiveDNSClassification, RecursiveDNSFinding, RecursiveDNSResult
from theHarvester.lib.routeviews import RouteViewsCancelled, RouteViewsResult
from theHarvester.lib.source_execution import SourceExecutionReport
from theHarvester.lib.takeover_evidence import TakeoverCandidateOutcome
from theHarvester.lib.virtual_host import (
HarvestedVirtualHostResult,
@@ -602,14 +603,11 @@ async def test_rapiddns_hostnames_honor_explicit_dns_resolution(monkeypatch: pyt
return {'192.0.2.20'}
class FakeCrtsh:
execution_status = 'partial'
stop_reason = 'invalid-response'
def __init__(self, _word: str) -> None:
pass
async def process(self, _proxy: bool) -> None:
return None
async def process(self, _proxy: bool) -> SourceExecutionReport:
return SourceExecutionReport('failed', 'invalid-response')
async def get_hostnames(self) -> set[str]:
return {'crt.example.com'}
@@ -1995,11 +1993,9 @@ async def test_limited_source_orchestration_uses_immutable_runner_request(
processed_with_proxy: list[bool] = []
class FakeAdapter:
execution_status = 'partial'
stop_reason = 'provider-boundary'
async def process(self, proxy: bool) -> None:
async def process(self, proxy: bool) -> SourceExecutionReport:
processed_with_proxy.append(proxy)
return SourceExecutionReport('partial', 'provider-boundary')
async def get_hostnames(self) -> set[str]:
return {'sub.example.test'}
@@ -2102,11 +2098,9 @@ async def test_target_only_source_orchestration_uses_immutable_runner_request(
processed_with_proxy: list[bool] = []
class FakeAdapter:
execution_status = 'partial'
stop_reason = 'provider-boundary'
async def process(self, proxy: bool) -> None:
async def process(self, proxy: bool) -> SourceExecutionReport:
processed_with_proxy.append(proxy)
return SourceExecutionReport('partial', 'provider-boundary')
async def get_hostnames(self) -> set[str]:
return {'sub.example.test'}
@@ -2174,13 +2168,11 @@ async def test_invalid_source_outcome_is_not_recorded_as_completed(monkeypatch:
completed.append(result)
class InvalidOutcomeCrtsh:
execution_status = 'typo'
def __init__(self, _word: str) -> None:
pass
async def process(self, _proxy: bool) -> None:
return None
async def process(self, _proxy: bool) -> object:
return {'status': 'typo'}
async def get_hostnames(self) -> set[str]:
return set()
tests/test_mojeek.py
+13 -15
View File
@@ -43,7 +43,7 @@ class TestMojeekSearch:
monkeypatch.setattr(mojeek.asyncio, 'sleep', fake_sleep)
search = mojeek.SearchMojeek(word='example.com', limit=30)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert [call['url'] for call in calls] == [
'https://www.mojeek.com/search?q=example.com&s=0',
@@ -56,8 +56,7 @@ class TestMojeekSearch:
assert delays == [1.0]
assert await search.get_hostnames() == ['docs.example.com', 'example.com']
assert await search.get_emails() == {'admin@example.com'}
assert search.execution_status == 'completed'
assert search.stop_reason == 'no-results'
assert report is None
@pytest.mark.parametrize(
('http_status', 'execution_status', 'stop_reason'),
@@ -86,13 +85,13 @@ class TestMojeekSearch:
monkeypatch.setattr(mojeek.asyncio, 'sleep', fake_sleep)
search = mojeek.SearchMojeek(word='example.com', limit=30)
await search.process()
report = await search.process()
assert len(calls) == 1
assert calls[0]['follow_redirects'] is False
assert delays == []
assert search.execution_status == execution_status
assert search.stop_reason == stop_reason
assert report.status == execution_status
assert report.stop_reason == stop_reason
assert await search.get_hostnames() == []
@pytest.mark.parametrize(
@@ -118,10 +117,10 @@ class TestMojeekSearch:
monkeypatch.setattr(mojeek.AsyncFetcher, 'fetch', fake_fetch)
search = mojeek.SearchMojeek(word='example.com', limit=10)
await search.process()
report = await search.process()
assert search.execution_status == execution_status
assert search.stop_reason == stop_reason
assert report.status == execution_status
assert report.stop_reason == stop_reason
assert await search.get_hostnames() == []
@pytest.mark.asyncio
@@ -140,13 +139,13 @@ class TestMojeekSearch:
monkeypatch.setattr(mojeek.AsyncFetcher, 'fetch', reject_scrape)
search = mojeek.SearchMojeek(word='example.com', limit=10)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert len(calls) == 1
assert calls[0]['include_metadata'] is True
assert calls[0]['proxy'] is True
assert search.execution_status == 'failed'
assert search.stop_reason == 'access-denied'
assert report.status == 'failed'
assert report.stop_reason == 'access-denied'
assert await search.get_hostnames() == []
@pytest.mark.asyncio
@@ -186,7 +185,7 @@ class TestMojeekSearch:
monkeypatch.setattr(mojeek.AsyncFetcher, 'fetch', reject_scrape)
search = mojeek.SearchMojeek(word='example.com', limit=20)
await search.process(proxy=True)
report = await search.process(proxy=True)
assert requests == [
{
@@ -205,8 +204,7 @@ class TestMojeekSearch:
'api.example.com',
'blog.example.com',
}
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert report is None
pytestmark = pytest.mark.provider_contract('mojeek')
theHarvester/discovery/apisguru.py
+25 -25
View File
@@ -8,6 +8,7 @@ from urllib.parse import unquote, urlsplit, urlunsplit
from theHarvester.lib.core import AsyncFetcher, FetcherResponse, ResponseStreamError
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus
class SearchApisGuru:
@@ -31,14 +32,16 @@ class SearchApisGuru:
def __init__(self, word: str, limit: int) -> None:
self.word = self._domain(word)
self.result_limit = max(0, min(limit, self.MAX_RESULTS_PER_ROUTE))
requested_limit = max(0, limit)
self.result_limit = min(requested_limit, self.MAX_RESULTS_PER_ROUTE)
self.result_limit_is_protective = requested_limit > self.MAX_RESULTS_PER_ROUTE
self.totalhosts: set[str] = set()
self.totalemails: set[str] = set()
self.urls: set[str] = set()
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
self._report: SourceExecutionReport | None = None
self.result_limit_reached = False
self.protective_limit_reached = False
@staticmethod
def _domain(value: str) -> str:
@@ -68,12 +71,8 @@ class SearchApisGuru:
return ''
return candidate
def _has_results(self) -> bool:
return any(host != self.word for host in self.totalhosts) or bool(self.totalemails or self.urls)
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self._has_results() else status
self.stop_reason = reason
def _stop(self, status: SourceReportStatus, reason: str) -> None:
self._report = SourceExecutionReport(status, reason)
async def _fetch(self, url: str) -> FetcherResponse | None:
try:
@@ -97,7 +96,10 @@ class SearchApisGuru:
def _retain(self, values: set[str], value: str) -> None:
if value not in values and len(values) >= self.result_limit:
self.result_limit_reached = True
if self.result_limit_is_protective:
self.protective_limit_reached = True
else:
self.result_limit_reached = True
else:
values.add(value)
@@ -174,7 +176,7 @@ class SearchApisGuru:
if isinstance(host, str) and isinstance(schemes, list):
path = base_path if isinstance(base_path, str) and base_path.startswith('/') else ''
if len(schemes) > self.MAX_SPEC_ITEMS:
self.result_limit_reached = True
self.protective_limit_reached = True
for scheme in islice(schemes, self.MAX_SPEC_ITEMS):
if isinstance(scheme, str) and scheme.lower() in {'http', 'https'}:
self._add_url(f'{scheme.lower()}://{host}{path}')
@@ -184,7 +186,7 @@ class SearchApisGuru:
servers = spec.get('servers')
if isinstance(servers, list):
if len(servers) > self.MAX_SPEC_ITEMS:
self.result_limit_reached = True
self.protective_limit_reached = True
for server in islice(servers, self.MAX_SPEC_ITEMS):
if not isinstance(server, dict):
malformed = True
@@ -258,12 +260,10 @@ class SearchApisGuru:
return
directory_response = await self._fetch(f'{self.DIRECTORY_ROOT}/{self.word}.json')
if directory_response is None:
if self.execution_status is None:
if self._report is None:
self._stop('failed', 'transport-error')
return
if directory_response.status == 404:
self.execution_status = 'completed'
self.stop_reason = 'no-results'
return
if directory_response.status == 429:
self._stop('rate-limited', 'http-429')
@@ -324,10 +324,10 @@ class SearchApisGuru:
for spec_url in spec_urls:
spec_response = await self._fetch(spec_url)
if spec_response is None:
if self.execution_status is None:
if self._report is None:
self._stop('failed', 'transport-error')
elif self.stop_reason in {'invalid-response', 'response-limit'}:
spec_failure = spec_failure or self.stop_reason
elif self._report.stop_reason in {'invalid-response', 'response-limit'}:
spec_failure = spec_failure or self._report.stop_reason
continue
return
if spec_response.status == 429:
@@ -354,13 +354,12 @@ class SearchApisGuru:
self._stop('failed', 'invalid-response')
elif spec_failure is not None:
self._stop('failed', spec_failure)
elif self.protective_limit_reached:
self._stop('failed', 'result-cap')
elif self.result_limit_reached:
self._stop('failed', 'result-limit')
self._stop('completed', 'result-limit')
elif directory_limit_reached:
self._stop('failed', 'directory-entry-limit')
else:
self.execution_status = 'completed'
self.stop_reason = None if self._has_results() else 'no-results'
async def get_hostnames(self) -> set[str]:
return self.totalhosts
@@ -371,13 +370,14 @@ class SearchApisGuru:
async def get_urls(self) -> set[str]:
return self.urls
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
self._report = None
self.result_limit_reached = False
self.protective_limit_reached = False
try:
async with asyncio.timeout(self.MAX_RUNTIME_SECONDS):
await self.do_search()
except TimeoutError:
self._stop('failed', 'runtime-limit')
return self._report
theHarvester/discovery/baidusearch.py
+10 -22
View File
@@ -2,6 +2,7 @@ import asyncio
from urllib.parse import urlencode
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
from theHarvester.parsers import myparser
@@ -15,12 +16,8 @@ class SearchBaidu:
self.hostname = 'www.baidu.com'
self.limit = limit
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
async def do_search(self) -> None:
self.execution_status = None
self.stop_reason = None
async def do_search(self) -> SourceExecutionReport | None:
headers = {'Host': self.hostname, 'User-Agent': Core.get_browser_user_agent()}
base_url = f'https://{self.server}/s'
urls = [
@@ -49,36 +46,27 @@ class SearchBaidu:
include_metadata=True,
)
if not isinstance(response, FetcherResponse):
self.execution_status = 'partial' if self.total_results else 'failed'
self.stop_reason = 'transport-error'
return
return SourceExecutionReport('failed', 'transport-error')
location = response.headers.get('location', '')
body = response.body if isinstance(response.body, str) else ''
if response.status == 429:
self.execution_status = 'partial' if self.total_results else 'rate-limited'
self.stop_reason = 'http-429'
return
return SourceExecutionReport('rate-limited', 'http-429')
if '百度安全验证' in body or 'wappass.baidu.com/static/captcha' in f'{location} {body}':
self.execution_status = 'partial' if self.total_results else 'failed'
self.stop_reason = 'security-verification'
return
return SourceExecutionReport('failed', 'security-verification')
if response.status >= 300:
self.execution_status = 'partial' if self.total_results else 'failed'
self.stop_reason = f'http-{response.status}'
return
return SourceExecutionReport('failed', f'http-{response.status}')
if not body:
self.execution_status = 'partial' if self.total_results else 'failed'
self.stop_reason = 'no-response'
return
return SourceExecutionReport('failed', 'no-response')
if url != homepage_url:
self.total_results += f' {body}'
finally:
await session.close()
return None
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
return await self.do_search()
async def get_emails(self):
rawres = myparser.Parser(self.total_results, self.word)
theHarvester/discovery/bevigil.py
+10 -27
View File
@@ -4,6 +4,7 @@ from theHarvester.discovery.constants import MissingKey
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
class SearchBeVigil:
@@ -15,15 +16,6 @@ class SearchBeVigil:
if not isinstance(self.key, str) or not self.key.strip():
raise MissingKey('bevigil')
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _has_results(self) -> bool:
return bool(self.totalhosts or self.urls)
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self._has_results() else status
self.stop_reason = reason
def _scoped_url(self, value: object) -> str | None:
if not isinstance(value, str):
@@ -39,9 +31,7 @@ class SearchBeVigil:
netloc = f'{hostname}:{port}' if port is not None else hostname
return urlunsplit((parsed.scheme.casefold(), netloc, parsed.path, parsed.query, ''))
async def do_search(self) -> None:
self.execution_status = None
self.stop_reason = None
async def do_search(self) -> SourceExecutionReport | None:
subdomain_endpoint = f'https://osint.bevigil.com/api/{self.word}/subdomains/'
url_endpoint = f'https://osint.bevigil.com/api/{self.word}/urls/'
headers = {'X-Access-Token': self.key}
@@ -49,6 +39,7 @@ class SearchBeVigil:
(subdomain_endpoint, 'subdomains'),
(url_endpoint, 'urls'),
)
report = None
try:
async with AsyncFetcher.open_session(
@@ -67,12 +58,10 @@ class SearchBeVigil:
)
response = responses[0] if responses else None
if error := provider_http_error(response):
self._stop(*error)
return
return SourceExecutionReport(*error)
assert isinstance(response, FetcherResponse)
if not isinstance(response.body, dict) or not isinstance(response.body.get(field), list):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
malformed = False
for value in response.body[field]:
@@ -86,16 +75,10 @@ class SearchBeVigil:
elif not isinstance(value, str):
malformed = True
if malformed:
self._stop('failed', 'invalid-response')
report = SourceExecutionReport('failed', 'invalid-response')
except Exception:
self._stop('failed', 'transport-error')
return
if self.execution_status is not None and self._has_results():
self.execution_status = 'partial'
elif self.execution_status is None:
self.execution_status = 'completed'
self.stop_reason = None if self._has_results() else 'no-results'
return SourceExecutionReport('failed', 'transport-error')
return report
async def get_hostnames(self) -> set[str]:
return self.totalhosts
@@ -103,6 +86,6 @@ class SearchBeVigil:
async def get_urls(self) -> set[str]:
return self.urls
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
return await self.do_search()
theHarvester/discovery/bufferoverun.py
+11 -25
View File
@@ -4,6 +4,7 @@ from ipaddress import ip_address
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
class SearchBufferover:
@@ -15,10 +16,8 @@ class SearchBufferover:
if self.key is None:
raise MissingKey('bufferoverun')
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
url = f'https://tls.bufferover.run/dns?q={self.word}'
response = await AsyncFetcher.fetch_all(
[url],
@@ -29,25 +28,15 @@ class SearchBufferover:
)
metadata = response[0] if response and isinstance(response[0], FetcherResponse) else None
if metadata is None:
self.execution_status = 'failed'
self.stop_reason = 'transport-error'
return
return SourceExecutionReport('failed', 'transport-error')
if metadata.status == 429:
self.execution_status = 'rate-limited'
self.stop_reason = 'http-429'
return
return SourceExecutionReport('rate-limited', 'http-429')
if metadata.status in {401, 403}:
self.execution_status = 'failed'
self.stop_reason = 'access-denied'
return
return SourceExecutionReport('failed', 'access-denied')
if not 200 <= metadata.status < 300:
self.execution_status = 'failed'
self.stop_reason = f'http-{metadata.status}'
return
return SourceExecutionReport('failed', f'http-{metadata.status}')
if not isinstance(metadata.body, dict) or not isinstance(metadata.body.get('Results'), list):
self.execution_status = 'failed'
self.stop_reason = 'invalid-response'
return
return SourceExecutionReport('failed', 'invalid-response')
results = metadata.body['Results']
malformed = False
@@ -73,11 +62,8 @@ class SearchBufferover:
if normalized_hostname := normalize_scoped_hostname(hostname, self.word):
self.totalhosts.add(normalized_hostname)
if malformed:
self.execution_status = 'partial' if self.totalhosts or self.totalips else 'failed'
self.stop_reason = 'invalid-response'
else:
self.execution_status = 'completed'
self.stop_reason = None if results else 'no-results'
return SourceExecutionReport('failed', 'invalid-response')
return None
async def get_hostnames(self) -> set:
return self.totalhosts
@@ -85,6 +71,6 @@ class SearchBufferover:
async def get_ips(self) -> set:
return self.totalips
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
return await self.do_search()
theHarvester/discovery/builtwith.py
+11 -28
View File
@@ -6,6 +6,7 @@ from urllib.parse import urlsplit, urlunsplit
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse, ResponseStreamError
from theHarvester.lib.hostnames import normalize_hostname, normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
class SearchBuiltWith:
@@ -28,15 +29,6 @@ class SearchBuiltWith:
self.servers: set[str] = set()
self.cms: set[str] = set()
self.analytics: set[str] = set()
self.execution_status = 'completed'
self.stop_reason: str | None = None
def _has_results(self) -> bool:
return bool(self.hosts or self.urls or self.frameworks or self.languages or self.servers or self.cms or self.analytics)
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self._has_results() else status
self.stop_reason = reason
def _path_hostname(self, path: dict[str, object]) -> tuple[str | None, bool]:
domain = path.get('Domain')
@@ -166,7 +158,7 @@ class SearchBuiltWith:
malformed = self._extract_technology(technology) or malformed
return malformed
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
headers = {
'Accept': 'application/json',
'Authorization': f'API {self.api_key}',
@@ -187,34 +179,25 @@ class SearchBuiltWith:
headers=headers,
)
except ResponseStreamError as error:
self._stop('failed', error.reason)
return
return SourceExecutionReport('failed', error.reason)
except Exception:
self._stop('failed', 'transport-error')
return
return SourceExecutionReport('failed', 'transport-error')
if not isinstance(response, FetcherResponse):
self._stop('failed', 'transport-error')
return
return SourceExecutionReport('failed', 'transport-error')
if response.status in {401, 403}:
self._stop('failed', 'access-denied')
return
return SourceExecutionReport('failed', 'access-denied')
if response.status == 429:
self._stop('rate-limited', 'http-429')
return
return SourceExecutionReport('rate-limited', 'http-429')
if not 200 <= response.status < 300 or not isinstance(response.body, dict):
reason = f'http-{response.status}' if not 200 <= response.status < 300 else 'invalid-response'
self._stop('failed', reason)
return
return SourceExecutionReport('failed', reason)
if not isinstance(response.body.get('Results'), list):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
self.tech_stack = response.body
if self._extract_data():
self._stop('failed', 'invalid-response')
else:
self.execution_status = 'completed'
self.stop_reason = None if self._has_results() else 'no-results'
return SourceExecutionReport('failed', 'invalid-response')
return None
async def get_hostnames(self) -> set[str]:
return self.hosts
theHarvester/discovery/censysearch.py
+19 -41
View File
@@ -6,6 +6,7 @@ import aiohttp
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
class SearchCensys:
@@ -22,15 +23,6 @@ class SearchCensys:
self.emails: set[str] = set()
self.limit = limit
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _has_results(self) -> bool:
return bool(self.totalhosts or self.emails)
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self._has_results() else status
self.stop_reason = reason
@staticmethod
def _normalize_emails(email_address: object) -> set[str]:
@@ -62,11 +54,9 @@ class SearchCensys:
self.emails.update(self._normalize_emails(subject.get('email_address')))
return False
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
if self.limit <= 0:
self.execution_status = 'completed'
self.stop_reason = 'no-results'
return
return None
headers = {'Accept': 'application/json', 'Authorization': f'Bearer {self.token}'}
params = (
@@ -100,29 +90,22 @@ class SearchCensys:
json_body=body,
)
except Exception:
self._stop('failed', 'transport-error')
return
return SourceExecutionReport('failed', 'transport-error')
if not isinstance(response, FetcherResponse):
self._stop('failed', 'transport-error')
return
return SourceExecutionReport('failed', 'transport-error')
if response.status == 429:
self._stop('rate-limited', 'http-429')
return
return SourceExecutionReport('rate-limited', 'http-429')
if response.status in {401, 403}:
self._stop('failed', 'access-denied')
return
return SourceExecutionReport('failed', 'access-denied')
if not 200 <= response.status < 300:
self._stop('failed', f'http-{response.status}')
return
return SourceExecutionReport('failed', f'http-{response.status}')
if not isinstance(response.body, dict) or not isinstance(response.body.get('result'), dict):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
result = response.body['result']
hits = result.get('hits')
next_page_token = result.get('next_page_token')
if not isinstance(hits, list) or (next_page_token is not None and not isinstance(next_page_token, str)):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
for hit in hits:
if records_seen >= self.limit:
@@ -131,22 +114,17 @@ class SearchCensys:
records_seen += 1
if records_seen >= self.limit:
if malformed:
self._stop('failed', 'invalid-response')
else:
self.execution_status = 'completed'
return
return SourceExecutionReport('failed', 'invalid-response')
return None
if not next_page_token:
if malformed:
self._stop('failed', 'invalid-response')
else:
self.execution_status = 'completed'
self.stop_reason = None if self._has_results() else 'no-results'
return
return SourceExecutionReport('failed', 'invalid-response')
return None
if next_page_token in seen_tokens:
self._stop('failed', 'repeated-cursor')
return
return SourceExecutionReport('failed', 'repeated-cursor')
seen_tokens.add(next_page_token)
page_token = next_page_token
return None
async def get_hostnames(self) -> set[str]:
return self.totalhosts
@@ -154,11 +132,11 @@ class SearchCensys:
async def get_emails(self) -> set[str]:
return self.emails
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
try:
await self.do_search()
return await self.do_search()
except asyncio.CancelledError:
raise
except (aiohttp.ClientError, TimeoutError, OSError, ssl.SSLError, ValueError):
self._stop('failed', 'transport-error')
return SourceExecutionReport('failed', 'transport-error')
theHarvester/discovery/certspottersearch.py
+7 -5
View File
@@ -2,6 +2,7 @@ import logging
from urllib.parse import urlencode
from theHarvester.lib.core import AsyncFetcher
from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus
logger = logging.getLogger(__name__)
@@ -19,12 +20,11 @@ class SearchCertspoter:
self.word = word.strip().lower().rstrip('.')
self.totalhosts: set = set()
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
self._report: SourceExecutionReport | None = None
def _mark_incomplete(self, reason: str, *, rate_limited: bool = False) -> None:
self.execution_status = 'rate-limited' if rate_limited else 'partial'
self.stop_reason = reason
status: SourceReportStatus = 'rate-limited' if rate_limited else 'partial'
self._report = SourceExecutionReport(status, reason)
async def do_search(self) -> None:
base_url = 'https://api.certspotter.com/v1/issuances'
@@ -131,7 +131,9 @@ class SearchCertspoter:
async def get_hostnames(self) -> set:
return self.totalhosts
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self._report = None
await self.do_search()
logger.info('\tSearching results.')
return self._report
theHarvester/discovery/commoncrawl.py
+17 -29
View File
@@ -6,6 +6,7 @@ from types import ModuleType
from urllib.parse import urlencode, urlsplit
from theHarvester.lib.core import AsyncFetcher, Core
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -41,8 +42,6 @@ class SearchCommoncrawl:
self.totalhosts: set[str] = set()
self.proxy = False
self.hostname = 'https://index.commoncrawl.org'
self.execution_status: str | None = None
self.stop_reason: str | None = None
@staticmethod
def _safe_parse_json_lines(payload: str) -> list:
@@ -120,29 +119,23 @@ class SearchCommoncrawl:
selected.append(entry)
return selected
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
try:
self.execution_status = None
self.stop_reason = None
if self.limit == 0:
return
return None
headers = {'User-agent': Core.get_user_agent()}
catalog_response = await AsyncFetcher.fetch_all(
[f'{self.hostname}/collinfo.json'], headers=headers, proxy=self.proxy, json=True
)
if not catalog_response or not isinstance(catalog_response[0], list) or not catalog_response[0]:
self.execution_status = 'failed'
self.stop_reason = 'invalid-catalog'
logger.error('Common Crawl API error: invalid index catalog')
return
return SourceExecutionReport('failed', 'invalid-catalog')
indexes = self._select_indexes(catalog_response[0])
if not indexes:
self.execution_status = 'failed'
self.stop_reason = 'no-usable-indexes'
logger.error('Common Crawl API error: index catalog contains no usable entries')
return
return SourceExecutionReport('failed', 'no-usable-indexes')
query_total = len(indexes) * 2
logger.info(
@@ -182,7 +175,7 @@ class SearchCommoncrawl:
while first_page < page_limit:
remaining = self.limit - len(self.totalhosts)
if remaining == 0:
return
return None
page_url = f'{endpoint}?{urlencode({"url": query, "output": "json", "pageSize": self.PAGE_SIZE, "page": first_page, "limit": min(remaining, self.MAX_RECORDS_PER_REQUEST)})}'
first_page += 1
responses = await AsyncFetcher.fetch_all([page_url], headers=headers, proxy=self.proxy)
@@ -198,7 +191,7 @@ class SearchCommoncrawl:
if domain.endswith(f'.{self.word}') or domain == self.word:
self.totalhosts.add(domain)
if len(self.totalhosts) >= self.limit:
return
return None
except ValueError as error:
message = str(error)
except Exception:
@@ -228,32 +221,27 @@ class SearchCommoncrawl:
if failed_queries:
if successful_queries or self.totalhosts:
self.execution_status = 'partial'
self.stop_reason = 'query-errors'
else:
self.execution_status = 'failed'
self.stop_reason = 'all-queries-failed'
logger.warning(f'Common Crawl failed all {query_total} queries')
elif page_limit_reached:
self.execution_status = 'partial'
self.stop_reason = 'page-limit'
return SourceExecutionReport('partial', 'query-errors')
logger.warning(f'Common Crawl failed all {query_total} queries')
return SourceExecutionReport('failed', 'all-queries-failed')
if page_limit_reached:
return SourceExecutionReport('partial', 'page-limit')
except Exception as error:
self.execution_status = 'partial' if self.totalhosts else 'failed'
self.stop_reason = 'unexpected-error'
logger.error(f'Common Crawl API error: {error}')
return SourceExecutionReport('failed', 'unexpected-error')
return None
async def get_hostnames(self) -> set:
return self.totalhosts
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
try:
async with asyncio.timeout(self.RUNTIME_SECONDS):
await self.do_search()
return await self.do_search()
except TimeoutError:
self.execution_status = 'partial' if self.totalhosts else 'failed'
self.stop_reason = 'runtime-limit'
logger.info(
f'Common Crawl runtime limit reached after {self.RUNTIME_SECONDS:g}s; preserved {len(self.totalhosts)} hosts'
)
return SourceExecutionReport('failed', 'runtime-limit')
theHarvester/discovery/crtname.py
+12 -25
View File
@@ -2,6 +2,7 @@ import asyncio
from theHarvester.lib.core import AsyncFetcher, ResponseStreamError
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
class SearchCrtName:
@@ -13,8 +14,6 @@ class SearchCrtName:
def __init__(self, word: str) -> None:
self.word = self._normalize_scope(word)
self.hostnames: set[str] = set()
self.execution_status: str | None = None
self.stop_reason: str | None = None
@staticmethod
def _valid_hostname(value: str) -> bool:
@@ -43,11 +42,7 @@ class SearchCrtName:
return ''
return normalized
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self.hostnames else status
self.stop_reason = reason
async def _collect(self, proxy: bool) -> None:
async def _collect(self, proxy: bool) -> SourceExecutionReport | None:
async with AsyncFetcher.stream_records(
self.ENDPOINT,
framing='ndjson',
@@ -58,14 +53,11 @@ class SearchCrtName:
request_timeout=self.RUNTIME_SECONDS,
) as response:
if response.status == 429:
self._stop('rate-limited', 'http-429')
return
return SourceExecutionReport('rate-limited', 'http-429')
if response.status in {401, 403}:
self._stop('failed', 'access-denied')
return
return SourceExecutionReport('failed', 'access-denied')
if not 200 <= response.status < 300:
self._stop('failed', f'http-{response.status}')
return
return SourceExecutionReport('failed', f'http-{response.status}')
malformed = False
async for record in response:
@@ -86,24 +78,19 @@ class SearchCrtName:
self.hostnames.add(normalized)
if malformed:
self._stop('failed', 'invalid-response')
else:
self.execution_status = 'completed'
self.stop_reason = None if self.hostnames else 'no-results'
return SourceExecutionReport('failed', 'invalid-response')
return None
async def process(self, proxy: bool = False) -> None:
self.execution_status = None
self.stop_reason = None
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
if not self.word:
self._stop('failed', 'invalid-target')
return
return SourceExecutionReport('failed', 'invalid-target')
try:
async with asyncio.timeout(self.RUNTIME_SECONDS):
await self._collect(proxy)
return await self._collect(proxy)
except ResponseStreamError as error:
self._stop('failed', error.reason)
return SourceExecutionReport('failed', error.reason)
except TimeoutError:
self._stop('failed', 'runtime-limit')
return SourceExecutionReport('failed', 'runtime-limit')
async def get_hostnames(self) -> set[str]:
return self.hostnames
theHarvester/discovery/crtsh.py
+11 -20
View File
@@ -2,6 +2,7 @@ import asyncio
import logging
from theHarvester.lib.core import AsyncFetcher, FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -13,10 +14,8 @@ class SearchCrtsh:
self.word = word
self.data: list = []
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
async def do_search(self) -> list:
async def do_search(self) -> tuple[list, SourceExecutionReport | None]:
data: set = set()
url = f'https://crt.sh/?q=%25.{self.word}&exclude=expired&deduplicate=Y&output=json'
response = None
@@ -37,47 +36,39 @@ class SearchCrtsh:
break
failure_reason = f'http-{result.status}' if not 200 <= result.status < 300 else 'invalid-response'
if result.status == 429:
self.execution_status = 'rate-limited'
self.stop_reason = failure_reason
return []
return [], SourceExecutionReport('rate-limited', failure_reason)
if attempt < max_attempts - 1:
await asyncio.sleep(2)
if response is None:
self.execution_status = 'failed'
self.stop_reason = failure_reason
logger.info(f'No valid response from crt.sh after {max_attempts} attempts.')
return []
return [], SourceExecutionReport('failed', failure_reason)
data = set([(dct['name_value'][2:] if dct['name_value'][:2] == '*.' else dct['name_value']) for dct in response])
data = {domain for domain in data if domain[0] != '*'}
except KeyError as ke:
self.execution_status = 'failed'
self.stop_reason = 'invalid-response'
logger.info(f'Missing expected key in response: {ke}')
return [], SourceExecutionReport('failed', 'invalid-response')
except Exception as e:
self.execution_status = 'failed'
self.stop_reason = 'unexpected-error'
logger.info(f'Unexpected error: {e}')
return [], SourceExecutionReport('failed', 'unexpected-error')
clean: list = []
for x in data:
pre = x.split()
for y in pre:
clean.append(y)
return clean
return clean, None
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
try:
async with asyncio.timeout(self.RUNTIME_SECONDS):
data = await self.do_search()
data, report = await self.do_search()
except TimeoutError:
self.execution_status = 'failed'
self.stop_reason = 'runtime-limit'
data = []
report = SourceExecutionReport('failed', 'runtime-limit')
self.data = data
return report
async def get_hostnames(self) -> list:
return self.data
theHarvester/discovery/dymosearch.py
+9 -20
View File
@@ -4,6 +4,7 @@ from theHarvester.discovery.constants import MissingKey
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
class SearchDymo:
@@ -29,12 +30,6 @@ class SearchDymo:
if not isinstance(self.key, str) or not self.key.strip():
raise MissingKey('dymo')
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self.totalhosts else status
self.stop_reason = reason
def _headers(self) -> dict[str, str]:
return {
@@ -43,7 +38,7 @@ class SearchDymo:
'Content-Type': 'application/json',
}
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
payload = {
'domain': self.word,
'url': f'https://{self.word}',
@@ -57,12 +52,10 @@ class SearchDymo:
include_metadata=True,
)
if error := provider_http_error(response):
self._stop(*error)
return
return SourceExecutionReport(*error)
assert isinstance(response, FetcherResponse)
if not isinstance(response.body, dict):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
self.results = response.body
@@ -86,10 +79,8 @@ class SearchDymo:
malformed = True
if malformed:
self._stop('failed', 'invalid-response')
else:
self.execution_status = 'completed'
self.stop_reason = None if self.totalhosts else 'no-results'
return SourceExecutionReport('failed', 'invalid-response')
return None
async def get_hostnames(self) -> set:
return self.totalhosts
@@ -97,11 +88,9 @@ class SearchDymo:
async def get_results(self) -> dict[str, Any]:
return self.results
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
try:
await self.do_search()
return await self.do_search()
except Exception:
self._stop('failed', 'transport-error')
return SourceExecutionReport('failed', 'transport-error')
theHarvester/discovery/fofa.py
+18 -39
View File
@@ -7,6 +7,7 @@ from theHarvester.discovery.constants import MissingKey
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
class SearchFofa:
@@ -24,8 +25,6 @@ class SearchFofa:
self.proxy = False
self.hostname = 'https://fofa.info'
self.api_key, self.email = self._get_api_credentials()
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _get_api_credentials(self) -> tuple[str, str]:
try:
@@ -36,13 +35,6 @@ class SearchFofa:
raise MissingKey('Fofa API (key and email required)')
return api_key, email
def _has_results(self) -> bool:
return bool(self.totalhosts or self.totalips)
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self._has_results() else status
self.stop_reason = reason
def _store_results(self, results: list[Any]) -> bool:
malformed = False
for result in results:
@@ -69,21 +61,21 @@ class SearchFofa:
malformed = True
return malformed
def _provider_error(self, body: dict[str, Any]) -> None:
def _provider_error(self, body: dict[str, Any]) -> SourceExecutionReport:
message = body.get('errmsg')
normalized = message.casefold() if isinstance(message, str) else ''
if 'invalid' in normalized or '账号无效' in normalized:
self._stop('failed', 'access-denied')
elif any(term in normalized for term in ('quota', 'limit', 'plan')):
self._stop('failed', 'quota-exhausted')
else:
self._stop('failed', 'provider-error')
return SourceExecutionReport('failed', 'access-denied')
if any(term in normalized for term in ('quota', 'limit', 'plan')):
return SourceExecutionReport('failed', 'quota-exhausted')
return SourceExecutionReport('failed', 'provider-error')
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
query = base64.b64encode(f'domain="{self.word}"'.encode()).decode()
cursor: str | None = None
seen_cursors: set[str] = set()
records_seen = 0
report = None
try:
async with AsyncFetcher.open_session(
headers={'User-Agent': Core.get_user_agent()},
@@ -108,41 +100,30 @@ class SearchFofa:
include_metadata=True,
)
if error := provider_http_error(response):
self._stop(*error)
return
return SourceExecutionReport(*error)
assert isinstance(response, FetcherResponse)
if not isinstance(response.body, dict):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
if response.body.get('error') is True:
self._provider_error(response.body)
return
return self._provider_error(response.body)
results = response.body.get('results')
if not isinstance(results, list):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
page_results = results[:remaining]
records_seen += len(page_results)
if self._store_results(page_results):
self._stop('failed', 'invalid-response')
report = SourceExecutionReport('failed', 'invalid-response')
next_cursor = response.body.get('next')
if not results or not isinstance(next_cursor, str) or not next_cursor:
break
if next_cursor in seen_cursors:
self._stop('failed', 'repeated-cursor')
break
return SourceExecutionReport('failed', 'repeated-cursor')
seen_cursors.add(next_cursor)
cursor = next_cursor
except Exception:
self._stop('failed', 'transport-error')
return
if self.execution_status is not None and self._has_results():
self.execution_status = 'partial'
elif self.execution_status is None:
self.execution_status = 'completed'
self.stop_reason = None if self._has_results() else 'no-results'
return SourceExecutionReport('failed', 'transport-error')
return report
async def get_hostnames(self) -> set[str]:
return self.totalhosts
@@ -150,8 +131,6 @@ class SearchFofa:
async def get_ips(self) -> set[str]:
return self.totalips
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
await self.do_search()
return await self.do_search()
theHarvester/discovery/fullhuntsearch.py
+12 -18
View File
@@ -7,6 +7,7 @@ from theHarvester.discovery.constants import MissingKey
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus
logger = logging.getLogger(__name__)
@@ -141,15 +142,10 @@ class SearchFullHunt:
}
self.proxy = False
self.filters: dict[str, str] = {} # Store filters for advanced searches
self.execution_status: str | None = None
self.stop_reason: str | None = None
self._report: SourceExecutionReport | None = None
def _has_results(self) -> bool:
return bool(self.total_results['hosts'] or self.total_results['ips'])
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self._has_results() else status
self.stop_reason = reason
def _stop(self, status: SourceReportStatus, reason: str) -> None:
self._report = SourceExecutionReport(status, reason)
def _get_headers(self) -> dict[str, str]:
"""Returns the headers needed for API requests"""
@@ -463,18 +459,12 @@ class SearchFullHunt:
await self.extract_data_from_search_results(search_results)
except Exception as error:
if self.execution_status is None:
if self._report is None:
reason = 'invalid-response' if isinstance(error, ValueError) else 'transport-error'
self._stop('failed', reason)
logger.info('Error during FullHunt search: %s', type(error).__name__)
return
if self.execution_status is not None and self._has_results():
self.execution_status = 'partial'
elif self.execution_status is None:
self.execution_status = 'completed'
self.stop_reason = None if self._has_results() else 'no-results'
async def get_hostnames(self) -> list[str]:
"""Return list of discovered subdomains"""
return self.total_results['hosts']
@@ -519,7 +509,11 @@ class SearchFullHunt:
"""Return all collected results"""
return self.total_results
async def process(self, proxy: bool = False, filters: dict[str, str] | None = None) -> None:
async def process(
self,
proxy: bool = False,
filters: dict[str, str] | None = None,
) -> SourceExecutionReport | None:
"""Main processing method
Args:
@@ -528,11 +522,11 @@ class SearchFullHunt:
"""
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
self._report = None
# Apply filters if provided
if filters:
self.add_filters(filters)
await self.do_search()
return self._report
theHarvester/discovery/hackertarget.py
+10 -16
View File
@@ -3,6 +3,7 @@ from ipaddress import ip_address, ip_network
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus
class SearchHackerTarget:
@@ -19,10 +20,8 @@ class SearchHackerTarget:
self.hostname = 'https://api.hackertarget.com'
self.proxy = False
self.key = Core.hackertarget_key()
self.execution_status: str | None = None
self.stop_reason: str | None = None
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
headers = {'User-agent': Core.get_user_agent()}
urls = [f'{self.hostname}/hostsearch/?q={self.word}']
@@ -56,7 +55,7 @@ class SearchHackerTarget:
proxy=self.proxy,
include_metadata=True,
)
failures: list[tuple[str, str]] = []
failures: list[tuple[SourceReportStatus, str]] = []
successful_endpoints = 0
for index, parser in enumerate(parsers):
response = responses[index] if index < len(responses) else None
@@ -89,14 +88,11 @@ class SearchHackerTarget:
if failures:
if successful_endpoints:
self.execution_status = 'partial'
self.stop_reason = failures[0][1]
elif all(status == 'rate-limited' for status, _reason in failures):
self.execution_status = 'rate-limited'
self.stop_reason = failures[0][1]
else:
self.execution_status = 'failed'
self.stop_reason = next(reason for status, reason in failures if status == 'failed')
return SourceExecutionReport('partial', failures[0][1])
if all(status == 'rate-limited' for status, _reason in failures):
return SourceExecutionReport('rate-limited', failures[0][1])
return SourceExecutionReport('failed', next(reason for status, reason in failures if status == 'failed'))
return None
def _parse_hostsearch(self, body: str) -> tuple[int, bool]:
parsed_rows = 0
@@ -142,11 +138,9 @@ class SearchHackerTarget:
parsed_rows += 1
return parsed_rows, malformed
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
await self.do_search()
return await self.do_search()
async def get_hostnames(self) -> set[str]:
return self.totalhosts
theHarvester/discovery/haveibeenpwned.py
+8 -22
View File
@@ -1,6 +1,7 @@
import logging
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -18,13 +19,9 @@ class SearchHaveIBeenPwned:
self.breach_dates: set[str] = set()
self.breach_types: set[str] = set()
self.affected_data: set[str] = set()
self.execution_status: str | None = None
self.stop_reason: str | None = None
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
"""Search for breaches associated with a domain or email."""
self.execution_status = None
self.stop_reason = None
try:
responses = await AsyncFetcher.fetch_all(
[f'{self.base_url}/breaches?domain={self.word}'],
@@ -35,36 +32,25 @@ class SearchHaveIBeenPwned:
)
except (OSError, RuntimeError, ValueError):
logger.info('HaveIBeenPwned request failed')
self.execution_status = 'failed'
self.stop_reason = 'transport-error'
return
return SourceExecutionReport('failed', 'transport-error')
response = responses[0] if responses and isinstance(responses[0], FetcherResponse) else None
if response is None:
logger.info('HaveIBeenPwned request failed')
self.execution_status = 'failed'
self.stop_reason = 'transport-error'
return
return SourceExecutionReport('failed', 'transport-error')
if response.status == 429:
logger.info('HaveIBeenPwned request failed with HTTP 429')
self.execution_status = 'rate-limited'
self.stop_reason = 'http-429'
return
return SourceExecutionReport('rate-limited', 'http-429')
if not 200 <= response.status < 300:
logger.info(f'HaveIBeenPwned request failed with HTTP {response.status}')
self.execution_status = 'failed'
self.stop_reason = f'http-{response.status}'
return
return SourceExecutionReport('failed', f'http-{response.status}')
if not isinstance(response.body, list) or not all(isinstance(breach, dict) for breach in response.body):
logger.info('HaveIBeenPwned returned malformed breach data')
self.execution_status = 'failed'
self.stop_reason = 'invalid-response'
return
return SourceExecutionReport('failed', 'invalid-response')
self.breaches = response.body
self._extract_data()
self.execution_status = 'completed'
self.stop_reason = None if self.breaches else 'no-results'
return None
def _extract_data(self) -> None:
"""Extract and categorize breach information."""
theHarvester/discovery/mojeek.py
+9 -16
View File
@@ -2,6 +2,7 @@ import asyncio
import logging
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus
from theHarvester.parsers import myparser
logger = logging.getLogger(__name__)
@@ -17,8 +18,7 @@ class SearchMojeek:
self.proxy = False
self.server = 'www.mojeek.com'
self.api_server = 'api.mojeek.com'
self.execution_status: str | None = None
self.stop_reason: str | None = None
self._report: SourceExecutionReport | None = None
try:
self.api_key = Core.mojeek_key()
@@ -30,9 +30,8 @@ class SearchMojeek:
else:
logger.info('[*] Mojeek: No API key found, using default scraping mode.')
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self.total_results else status
self.stop_reason = reason
def _stop(self, status: SourceReportStatus, reason: str) -> None:
self._report = SourceExecutionReport(status, reason)
async def _search_api(self, headers: dict[str, str]) -> None:
urls = [
@@ -90,8 +89,6 @@ class SearchMojeek:
return
self.total_results += f' {url} {title} {description} '
self.execution_status = 'completed'
self.stop_reason = None if self.total_results else 'no-results'
logger.info('[*] Mojeek: API search completed successfully.')
async def _search_keyless(self, headers: dict[str, str]) -> None:
@@ -130,29 +127,25 @@ class SearchMojeek:
self._stop('failed', 'access-denied')
return
if 'no results' in normalized_body or 'no-results' in normalized_body:
self.execution_status = 'completed'
self.stop_reason = 'no-results'
return
if 'results-standard' not in normalized_body:
self._stop('failed', 'invalid-response')
return
self.total_results += f' {response.body}'
self.execution_status = 'completed'
async def do_search(self) -> None:
self.execution_status = None
self.stop_reason = None
async def do_search(self) -> SourceExecutionReport | None:
self._report = None
user_agent = Core.get_user_agent() if self.api_key else Core.get_browser_user_agent()
headers = {'User-Agent': user_agent}
if self.api_key:
await self._search_api(headers)
else:
await self._search_keyless(headers)
return self._report
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
return await self.do_search()
async def get_emails(self):
rawres = myparser.Parser(self.total_results, self.word)
theHarvester/discovery/netlas.py
+17 -30
View File
@@ -6,6 +6,7 @@ from theHarvester.discovery.constants import MissingKey
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
class SearchNetlas:
@@ -19,27 +20,19 @@ class SearchNetlas:
if not isinstance(self.key, str) or not self.key.strip():
raise MissingKey('netlas')
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self.totalhosts else status
self.stop_reason = reason
def _response_body(self, response: Any) -> Any | None:
@staticmethod
def _response_body(response: Any) -> tuple[Any | None, SourceExecutionReport | None]:
if isinstance(response, FetcherResponse) and response.status == 402:
self._stop('failed', 'quota-exhausted')
return None
return None, SourceExecutionReport('failed', 'quota-exhausted')
if error := provider_http_error(response):
self._stop(*error)
return None
return None, SourceExecutionReport(*error)
assert isinstance(response, FetcherResponse)
if response.body is None:
self._stop('failed', 'invalid-response')
return None
return response.body
return None, SourceExecutionReport('failed', 'invalid-response')
return response.body, None
async def do_search(self, session: Any, size: int) -> None:
async def do_search(self, session: Any, size: int) -> SourceExecutionReport | None:
response = await AsyncFetcher.post_fetch(
'https://app.netlas.io/api/domains/download/',
session=session,
@@ -52,12 +45,11 @@ class SearchNetlas:
'source_type': 'include',
},
)
body = self._response_body(response)
if body is None:
return
body, report = self._response_body(response)
if report is not None:
return report
if not isinstance(body, list):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
malformed = False
for row in body[:size]:
@@ -71,24 +63,19 @@ class SearchNetlas:
if hostname := normalize_scoped_hostname(domain, self.word):
self.totalhosts.add(hostname)
if malformed:
self._stop('failed', 'invalid-response')
return SourceExecutionReport('failed', 'invalid-response')
return None
async def get_hostnames(self) -> set[str]:
return self.totalhosts
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
try:
async with AsyncFetcher.open_session(
headers={'Authorization': f'Bearer {self.key}'},
proxy=proxy,
) as session:
await self.do_search(session, self.limit)
return await self.do_search(session, self.limit)
except Exception:
self._stop('failed', 'transport-error')
return
if self.execution_status is None:
self.execution_status = 'completed'
self.stop_reason = None if self.totalhosts else 'no-results'
return SourceExecutionReport('failed', 'transport-error')
theHarvester/discovery/onyphe.py
+15 -33
View File
@@ -9,6 +9,7 @@ from theHarvester.lib.asn_attribution import AsnAttributionObservation, SubjectK
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.result_values import normalize_asn
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -39,17 +40,8 @@ class SearchOnyphe:
if not isinstance(self.key, str) or not self.key.strip():
raise MissingKey('onyphe')
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _has_results(self) -> bool:
return bool(self.totalhosts or self.totalips or self.asns)
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self._has_results() else status
self.stop_reason = reason
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
base_url = 'https://www.onyphe.io/api/v2/search/'
headers = {
'User-Agent': Core.get_user_agent(),
@@ -61,6 +53,7 @@ class SearchOnyphe:
result_limit = min(self.limit, self.MAX_RESULTS)
page_size = min(result_limit, self.MAX_RESULTS)
last_total = 0
report = None
try:
async with AsyncFetcher.open_session(headers=headers, proxy=self.proxy) as session:
while records_seen < result_limit:
@@ -73,16 +66,14 @@ class SearchOnyphe:
include_metadata=True,
)
if error := provider_http_error(metadata):
self._stop(*error)
return
return SourceExecutionReport(*error)
assert isinstance(metadata, FetcherResponse)
if not isinstance(metadata.body, dict):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
response_text = metadata.body.get('text')
if response_text != 'Success':
self._stop('failed', 'provider-error' if isinstance(response_text, str) else 'invalid-response')
return
reason = 'provider-error' if isinstance(response_text, str) else 'invalid-response'
return SourceExecutionReport('failed', reason)
results = metadata.body.get('results')
max_page = metadata.body.get('max_page', 1)
total = metadata.body.get('total', len(results) if isinstance(results, list) else None)
@@ -95,34 +86,27 @@ class SearchOnyphe:
or not isinstance(total, int)
or total < 0
):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
page_results = results[:remaining]
records_seen += len(page_results)
last_total = total
self.response = {**metadata.body, 'results': page_results}
if await self.parse_onyphe_resp_json():
self._stop('failed', 'invalid-response')
report = SourceExecutionReport('failed', 'invalid-response')
expected_records = min(total, result_limit)
if (not results or page >= max_page) and records_seen < expected_records:
self._stop('failed', 'invalid-response')
break
return SourceExecutionReport('failed', 'invalid-response')
if not results or page >= max_page or records_seen >= expected_records:
break
page += 1
except Exception as error:
self._stop('failed', 'transport-error')
logger.info('Onyphe request failed: %s', type(error).__name__)
return
return SourceExecutionReport('failed', 'transport-error')
if self.limit > self.MAX_RESULTS and last_total > self.MAX_RESULTS and records_seen >= self.MAX_RESULTS:
self._stop('failed', 'provider-limit')
if self.execution_status is not None and self._has_results():
self.execution_status = 'partial'
elif self.execution_status is None:
self.execution_status = 'completed'
self.stop_reason = None if self._has_results() else 'no-results'
return SourceExecutionReport('failed', 'provider-limit')
return report
async def parse_onyphe_resp_json(self) -> bool:
if not isinstance(self.response, dict):
@@ -259,8 +243,6 @@ class SearchOnyphe:
async def get_ips(self) -> set:
return self.totalips
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
await self.do_search()
return await self.do_search()
theHarvester/discovery/otxsearch.py
+13 -26
View File
@@ -4,6 +4,7 @@ from ipaddress import ip_address
from typing import Any
from theHarvester.lib.core import AsyncFetcher, FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -17,12 +18,8 @@ class SearchOtx:
self.totalhosts: set = set()
self.totalips: set = set()
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
async def do_search(self) -> None:
self.execution_status = None
self.stop_reason = None
async def do_search(self) -> SourceExecutionReport | None:
url = f'https://otx.alienvault.com/api/v1/indicators/domain/{self.word}/passive_dns'
try:
response_list = await AsyncFetcher.fetch_all(
@@ -51,42 +48,32 @@ class SearchOtx:
except (OSError, RuntimeError, ValueError):
self.totalhosts = set()
self.totalips = set()
self.execution_status = 'failed'
self.stop_reason = 'transport-error'
logger.info('OTX request failed')
return
return SourceExecutionReport('failed', 'transport-error')
if response is None:
self.execution_status = 'failed'
self.stop_reason = 'transport-error'
logger.info('OTX request failed')
return
return SourceExecutionReport('failed', 'transport-error')
if not 200 <= response.status < 300:
if response.status == 429:
self.execution_status = 'rate-limited'
self.stop_reason = 'http-429'
report = SourceExecutionReport('rate-limited', 'http-429')
else:
self.execution_status = 'failed'
self.stop_reason = f'http-{response.status}'
report = SourceExecutionReport('failed', f'http-{response.status}')
logger.info(f'OTX request failed with HTTP {response.status}')
return
return report
# Expect a list with one JSON-decoded dict
dct: Any = response.body
if not isinstance(dct, dict):
self.totalhosts = set()
self.totalips = set()
self.execution_status = 'failed'
self.stop_reason = 'invalid-response'
return
return SourceExecutionReport('failed', 'invalid-response')
passive = dct.get('passive_dns')
if not isinstance(passive, list):
self.totalhosts = set()
self.totalips = set()
self.execution_status = 'failed'
self.stop_reason = 'invalid-response'
return
return SourceExecutionReport('failed', 'invalid-response')
try:
self.totalhosts = {host['hostname'] for host in passive if isinstance(host, dict) and 'hostname' in host}
@@ -101,8 +88,8 @@ class SearchOtx:
except (KeyError, TypeError, ValueError):
self.totalhosts = set()
self.totalips = set()
self.execution_status = 'failed'
self.stop_reason = 'invalid-response'
return SourceExecutionReport('failed', 'invalid-response')
return None
async def get_hostnames(self) -> set:
return self.totalhosts
@@ -110,6 +97,6 @@ class SearchOtx:
async def get_ips(self) -> set:
return self.totalips
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
return await self.do_search()
theHarvester/discovery/projectdiscovery.py
+16 -31
View File
@@ -2,6 +2,7 @@ import logging
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus
logger = logging.getLogger(__name__)
@@ -15,8 +16,6 @@ class SearchDiscovery:
self.proxy = False
self.hostname = 'https://dns.projectdiscovery.io'
self.key = self._get_api_key()
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _get_api_key(self) -> str:
try:
@@ -27,7 +26,7 @@ class SearchDiscovery:
raise MissingKey('ProjectDiscovery')
return key
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
try:
url = f'{self.hostname}/dns/{self.word}/subdomains'
response = await AsyncFetcher.fetch_all(
@@ -40,42 +39,33 @@ class SearchDiscovery:
metadata = response[0] if response and isinstance(response[0], FetcherResponse) else None
if metadata is None:
self.execution_status = 'failed'
self.stop_reason = 'transport-error'
logger.info('No response from ProjectDiscovery for: %s', url)
return
return SourceExecutionReport('failed', 'transport-error')
if not 200 <= metadata.status < 300:
self.execution_status = 'rate-limited' if metadata.status == 429 else 'failed'
self.stop_reason = 'access-denied' if metadata.status in {401, 403} else f'http-{metadata.status}'
logger.info('ProjectDiscovery request failed with HTTP %s', metadata.status)
return
status: SourceReportStatus = 'rate-limited' if metadata.status == 429 else 'failed'
reason = 'access-denied' if metadata.status in {401, 403} else f'http-{metadata.status}'
return SourceExecutionReport(status, reason)
try:
data = metadata.body
if not isinstance(data, (dict, list)):
self.execution_status = 'failed'
self.stop_reason = 'invalid-response'
logger.info('ProjectDiscovery returned malformed data')
return
return SourceExecutionReport('failed', 'invalid-response')
if isinstance(data, dict):
if 'error' in data:
error_message = data.get('message', data.get('error', 'Unknown error'))
self.execution_status = 'failed'
self.stop_reason = (
'access-denied' if 'unauthorized' in str(error_message).casefold() else 'provider-error'
)
reason = 'access-denied' if 'unauthorized' in str(error_message).casefold() else 'provider-error'
logger.info('ProjectDiscovery returned an error')
return
return SourceExecutionReport('failed', reason)
subdomains = data.get('subdomains', []) or data.get('data', []) or data.get('results', [])
else:
subdomains = data
if not isinstance(subdomains, list):
self.execution_status = 'failed'
self.stop_reason = 'invalid-response'
logger.info('ProjectDiscovery returned malformed subdomain data')
return
return SourceExecutionReport('failed', 'invalid-response')
malformed_items = False
for subdomain in subdomains:
@@ -94,25 +84,20 @@ class SearchDiscovery:
self.totalhosts.add(f'{label}.{self.word}'.lower() if label else self.word.lower())
if malformed_items:
self.execution_status = 'partial' if self.totalhosts else 'failed'
self.stop_reason = 'invalid-response'
else:
self.execution_status = 'completed'
self.stop_reason = None if subdomains else 'no-results'
return SourceExecutionReport('failed', 'invalid-response')
return None
except Exception as error:
self.execution_status = 'partial' if self.totalhosts else 'failed'
self.stop_reason = 'invalid-response'
logger.info('Failed to parse ProjectDiscovery response: %s', type(error).__name__)
return SourceExecutionReport('failed', 'invalid-response')
except MissingKey:
raise
except Exception as error:
self.execution_status = 'failed'
self.stop_reason = 'transport-error'
logger.info('ProjectDiscovery API error: %s', type(error).__name__)
return SourceExecutionReport('failed', 'transport-error')
async def get_hostnames(self) -> set[str]:
return self.totalhosts
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
return await self.do_search()
theHarvester/discovery/provider_response.py
+2 -1
View File
@@ -1,7 +1,8 @@
from theHarvester.lib.core import FetcherResponse
from theHarvester.lib.source_execution import SourceReportStatus
def provider_http_error(response: object) -> tuple[str, str] | None:
def provider_http_error(response: object) -> tuple[SourceReportStatus, str] | None:
"""Classify transport and HTTP failures shared by provider adapters."""
if not isinstance(response, FetcherResponse):
return 'failed', 'transport-error'
theHarvester/discovery/robtex.py
+14 -28
View File
@@ -6,6 +6,7 @@ from types import ModuleType
import aiohttp
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -28,8 +29,6 @@ class SearchRobtex:
self.totalips: set = set()
self.proxy = False
self.hostname = 'https://freeapi.robtex.com'
self.execution_status: str | None = None
self.stop_reason: str | None = None
@staticmethod
def _safe_parse_json_lines(payload: str) -> list:
@@ -46,7 +45,7 @@ class SearchRobtex:
continue
return results
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
try:
headers = {'User-agent': Core.get_user_agent()}
@@ -59,43 +58,33 @@ class SearchRobtex:
)
response = responses[0] if responses else None
if response is None:
self.execution_status = 'failed'
self.stop_reason = 'transport-error'
logger.info(f'No response from Robtex API for: {url}')
return
return SourceExecutionReport('failed', 'transport-error')
if response.status == 429:
self.execution_status = 'rate-limited'
self.stop_reason = 'http-429'
logger.info('Robtex request was rate limited')
return
return SourceExecutionReport('rate-limited', 'http-429')
if not 200 <= response.status < 300:
self.execution_status = 'failed'
self.stop_reason = f'http-{response.status}'
logger.info(f'Robtex request failed with HTTP {response.status}')
return
return SourceExecutionReport('failed', f'http-{response.status}')
if not isinstance(response.body, str):
self.execution_status = 'failed'
self.stop_reason = 'invalid-response'
logger.info(f'No response from Robtex API for: {url}')
return
return SourceExecutionReport('failed', 'invalid-response')
if not response.body:
return
return None
try:
data = self._safe_parse_json_lines(response.body)
except (TypeError, ValueError) as e:
logger.info(f'Failed to parse JSON lines from Robtex response: {e}')
return
return SourceExecutionReport('failed', 'invalid-response')
records = [
record
for record in data
if isinstance(record, dict) and isinstance(record.get('rrtype'), str) and isinstance(record.get('rrdata'), str)
]
if not records:
self.execution_status = 'failed'
self.stop_reason = 'invalid-response'
logger.info('Robtex returned no valid DNS records')
return
return SourceExecutionReport('failed', 'invalid-response')
for record in records:
rrdata = record['rrdata']
@@ -108,19 +97,16 @@ class SearchRobtex:
pass
except (aiohttp.ClientError, TimeoutError, OSError) as e:
self.execution_status = 'failed'
self.stop_reason = 'transport-error'
logger.info(f'Robtex API error: {e}')
return SourceExecutionReport('failed', 'transport-error')
except (TypeError, ValueError) as e:
self.execution_status = 'failed'
self.stop_reason = 'invalid-response'
logger.info(f'Robtex API error: {e}')
return SourceExecutionReport('failed', 'invalid-response')
return None
async def get_ips(self) -> set:
return self.totalips
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
await self.do_search()
return await self.do_search()
theHarvester/discovery/search_dnsdumpster.py
+14 -34
View File
@@ -5,6 +5,7 @@ from ipaddress import ip_address
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -19,10 +20,8 @@ class SearchDNSDumpster:
self.ips: set = set()
self.base_url = 'https://api.dnsdumpster.com'
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
url = f'{self.base_url}/domain/{self.word}'
headers = {'User-Agent': Core.get_user_agent(), 'X-API-Key': self.key}
try:
@@ -34,40 +33,24 @@ class SearchDNSDumpster:
include_metadata=True,
)
except Exception as error:
self.execution_status = 'failed'
self.stop_reason = 'transport-error'
logger.info('DNSDumpster request failed: %s', type(error).__name__)
return
return SourceExecutionReport('failed', 'transport-error')
metadata = response[0] if response and isinstance(response[0], FetcherResponse) else None
if metadata is None:
self.execution_status = 'failed'
self.stop_reason = 'transport-error'
return
return SourceExecutionReport('failed', 'transport-error')
if metadata.status == 429:
self.execution_status = 'rate-limited'
self.stop_reason = 'http-429'
return
return SourceExecutionReport('rate-limited', 'http-429')
if metadata.status in {401, 403}:
self.execution_status = 'failed'
self.stop_reason = 'access-denied'
return
return SourceExecutionReport('failed', 'access-denied')
if not 200 <= metadata.status < 300:
self.execution_status = 'failed'
self.stop_reason = f'http-{metadata.status}'
return
return SourceExecutionReport('failed', f'http-{metadata.status}')
if not isinstance(metadata.body, dict):
self.execution_status = 'failed'
self.stop_reason = 'invalid-response'
return
return SourceExecutionReport('failed', 'invalid-response')
if 'error' in metadata.body:
self.execution_status = 'failed'
self.stop_reason = 'provider-error'
return
return SourceExecutionReport('failed', 'provider-error')
if not any(record_type in metadata.body for record_type in ('a', 'cname', 'mx', 'ns', 'txt')):
self.execution_status = 'failed'
self.stop_reason = 'invalid-response'
return
return SourceExecutionReport('failed', 'invalid-response')
malformed = False
records = []
@@ -108,15 +91,12 @@ class SearchDNSDumpster:
malformed = True
if malformed:
self.execution_status = 'partial' if self.hosts or self.ips else 'failed'
self.stop_reason = 'invalid-response'
else:
self.execution_status = 'completed'
self.stop_reason = None if self.hosts or self.ips else 'no-results'
return SourceExecutionReport('failed', 'invalid-response')
return None
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
return await self.do_search()
async def get_hostnames(self) -> set:
return self.hosts
theHarvester/discovery/searchhunterhow.py
+14 -32
View File
@@ -9,6 +9,7 @@ from theHarvester.discovery.constants import MissingKey
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
class SearchHunterHow:
@@ -24,12 +25,6 @@ class SearchHunterHow:
if not isinstance(self.key, str) or not self.key.strip():
raise MissingKey('hunterhow')
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self.total_hostnames else status
self.stop_reason = reason
@staticmethod
def _page_size(remaining: int) -> int:
@@ -38,9 +33,7 @@ class SearchHunterHow:
return size
return 1000
async def do_search(self) -> None:
self.execution_status = None
self.stop_reason = None
async def do_search(self) -> SourceExecutionReport | None:
query = base64.urlsafe_b64encode(f'domain.suffix="{self.word}"'.encode()).decode('ascii')
end = datetime.now(UTC).date()
start = end - relativedelta(days=364)
@@ -53,6 +46,7 @@ class SearchHunterHow:
'end_time': end.isoformat(),
'fields': 'domain',
}
report = None
try:
async with AsyncFetcher.open_session(
headers={'User-Agent': Core.get_user_agent()},
@@ -71,28 +65,22 @@ class SearchHunterHow:
include_metadata=True,
)
if error := provider_http_error(response):
self._stop(*error)
return
return SourceExecutionReport(*error)
assert isinstance(response, FetcherResponse)
if not isinstance(response.body, dict):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
code = response.body.get('code')
if code == 40001:
self._stop('failed', 'access-denied')
return
return SourceExecutionReport('failed', 'access-denied')
if code != 200:
self._stop('failed', 'provider-error')
return
return SourceExecutionReport('failed', 'provider-error')
data = response.body.get('data')
if not isinstance(data, dict):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
total = data.get('total')
rows = data.get('list')
if isinstance(total, bool) or not isinstance(total, int) or total < 0 or not isinstance(rows, list):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
remaining = self.limit - returned
malformed = False
@@ -103,7 +91,7 @@ class SearchHunterHow:
if hostname := normalize_scoped_hostname(row['domain'], self.word):
self.total_hostnames.add(hostname)
if malformed:
self._stop('failed', 'invalid-response')
report = SourceExecutionReport('failed', 'invalid-response')
returned += len(rows)
if not rows or returned >= min(total, self.limit):
@@ -111,18 +99,12 @@ class SearchHunterHow:
page += 1
await asyncio.sleep(self.REQUEST_DELAY_SECONDS)
except Exception:
self._stop('failed', 'transport-error')
return
if self.execution_status is not None and self.total_hostnames:
self.execution_status = 'partial'
elif self.execution_status is None:
self.execution_status = 'completed'
self.stop_reason = None if self.total_hostnames else 'no-results'
return SourceExecutionReport('failed', 'transport-error')
return report
async def get_hostnames(self) -> set[str]:
return self.total_hostnames
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
return await self.do_search()
theHarvester/discovery/securityscorecard.py
+11 -19
View File
@@ -7,6 +7,7 @@ from theHarvester.discovery.constants import MissingKey
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus
class SearchSecurityScorecard:
@@ -33,12 +34,10 @@ class SearchSecurityScorecard:
self.recommendations: list[dict[str, Any]] = []
self.history: list[dict[str, Any]] = []
self.ips: set[str] = set()
self.execution_status: str | None = None
self.stop_reason: str | None = None
self._report: SourceExecutionReport | None = None
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self.hosts or self.ips else status
self.stop_reason = reason
def _stop(self, status: SourceReportStatus, reason: str) -> None:
self._report = SourceExecutionReport(status, reason)
def _response_body(self, response: Any) -> dict[str, Any] | None:
if error := provider_http_error(response):
@@ -118,9 +117,8 @@ class SearchSecurityScorecard:
page += 1
return True
async def process(self, proxy: bool = False) -> None:
self.execution_status = None
self.stop_reason = None
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self._report = None
try:
async with AsyncFetcher.open_session(headers=self.headers, proxy=proxy) as session:
response = await AsyncFetcher.fetch(
@@ -131,22 +129,16 @@ class SearchSecurityScorecard:
)
body = self._response_body(response)
if body is None:
return
return self._report
if self._extract_summary(body):
self._stop('failed', 'invalid-response')
if not await self._collect_assets(session, 'domains', 'domain'):
return
return self._report
if not await self._collect_assets(session, 'ips', 'ip'):
return
return self._report
except Exception:
self._stop('failed', 'transport-error')
return
if self.execution_status is not None and (self.hosts or self.ips):
self.execution_status = 'partial'
elif self.execution_status is None:
self.execution_status = 'completed'
self.stop_reason = None if self.hosts or self.ips else 'no-results'
return SourceExecutionReport('failed', 'transport-error')
return self._report
async def get_hostnames(self) -> set[str]:
return self.hosts
theHarvester/discovery/securitytrailssearch.py
+11 -19
View File
@@ -7,6 +7,7 @@ from theHarvester.discovery.constants import MissingKey
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
class SearchSecuritytrail:
@@ -20,20 +21,15 @@ class SearchSecuritytrail:
self.proxy = False
self.domain_data: dict[str, Any] = {}
self.subdomains_data: dict[str, Any] = {}
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self.info[0] or self.info[1] else status
self.stop_reason = reason
self._report: SourceExecutionReport | None = None
def _body(self, response: Any) -> dict[str, Any] | None:
if error := provider_http_error(response):
self._stop(*error)
self._report = SourceExecutionReport(*error)
return None
assert isinstance(response, FetcherResponse)
if not isinstance(response.body, dict):
self._stop('failed', 'invalid-response')
self._report = SourceExecutionReport('failed', 'invalid-response')
return None
return response.body
@@ -72,10 +68,9 @@ class SearchSecuritytrail:
hostnames.add(hostname)
return malformed
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
self._report = None
headers = {'APIKEY': self.key, 'Accept': 'application/json'}
try:
async with AsyncFetcher.open_session(headers=headers, proxy=proxy) as session:
@@ -87,7 +82,7 @@ class SearchSecuritytrail:
)
domain_body = self._body(domain_response)
if domain_body is None:
return
return self._report
self.domain_data = domain_body
malformed = self._parse_domain(domain_body)
@@ -99,17 +94,14 @@ class SearchSecuritytrail:
)
subdomain_body = self._body(subdomain_response)
if subdomain_body is None:
return
return self._report
self.subdomains_data = subdomain_body
malformed = self._parse_subdomains(subdomain_body) or malformed
except Exception:
self._stop('failed', 'transport-error')
return
return SourceExecutionReport('failed', 'transport-error')
if malformed:
self._stop('failed', 'invalid-response')
if self.execution_status is None:
self.execution_status = 'completed'
self.stop_reason = None if self.info[0] or self.info[1] else 'no-results'
return SourceExecutionReport('failed', 'invalid-response')
return None
async def get_ips(self) -> set[str]:
return self.info[0]
theHarvester/discovery/sherlockeye.py
+15 -33
View File
@@ -7,6 +7,7 @@ from theHarvester.discovery.constants import MissingKey
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -33,15 +34,6 @@ class SearchSherlockeye:
self.totalips: set[str] = set()
self.results: list[dict[str, Any]] = []
self.proxy: bool | str = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _has_results(self) -> bool:
return bool(self.totalhosts or self.totalemails or self.totalips)
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self._has_results() else status
self.stop_reason = reason
def _headers(self) -> dict[str, str]:
return {
@@ -107,21 +99,18 @@ class SearchSherlockeye:
malformed = True
return malformed
def _extract_response(self, response: dict[str, Any]) -> None:
def _extract_response(self, response: dict[str, Any]) -> SourceExecutionReport | None:
if response.get('success') is False:
logger.info('Sherlockeye API error')
self._stop('failed', 'provider-error')
return
return SourceExecutionReport('failed', 'provider-error')
data = response.get('data')
if not isinstance(data, dict):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
search_results = data.get('results')
if not isinstance(search_results, list):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
self.results = search_results
malformed = False
@@ -131,12 +120,10 @@ class SearchSherlockeye:
else:
malformed = True
if malformed:
self._stop('failed', 'invalid-response')
elif self.execution_status is None:
self.execution_status = 'completed'
self.stop_reason = None if self._has_results() else 'no-results'
return SourceExecutionReport('failed', 'invalid-response')
return None
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
payload = {
'type': 'domain',
'value': self.word,
@@ -156,22 +143,19 @@ class SearchSherlockeye:
json_body=payload,
)
if error := provider_http_error(response):
self._stop(*error)
status = response.status if isinstance(response, FetcherResponse) else 'transport'
logger.info('Sherlockeye API request failed with status %s: %s', status, error[1])
return
return SourceExecutionReport(*error)
assert isinstance(response, FetcherResponse)
if response.status != 200:
self._stop('failed', f'http-{response.status}')
logger.info('Sherlockeye API request failed with status %s', response.status)
return
return SourceExecutionReport('failed', f'http-{response.status}')
if isinstance(response.body, dict):
self._extract_response(response.body)
else:
self._stop('failed', 'invalid-response')
return self._extract_response(response.body)
return SourceExecutionReport('failed', 'invalid-response')
except Exception as error:
self._stop('failed', 'transport-error')
logger.info('Sherlockeye API error: %s', type(error).__name__)
return SourceExecutionReport('failed', 'transport-error')
async def get_hostnames(self) -> set[str]:
return self.totalhosts
@@ -185,8 +169,6 @@ class SearchSherlockeye:
async def get_results(self) -> list[dict[str, Any]]:
return self.results
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
await self.do_search()
return await self.do_search()
theHarvester/discovery/shodansearch.py
+9 -21
View File
@@ -15,6 +15,7 @@ from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse, ResponseS
from theHarvester.lib.hostchecker import resolve_ip_addresses
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.shodan_evidence import ShodanHostObservation, canonical_shodan_hosts
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -77,8 +78,6 @@ class SearchShodan:
self.error_type: str | None = None
self.asn_attributions: set[AsnAttributionObservation] = set()
self.totalhosts: set[str] = set()
self.execution_status: str | None = None
self.stop_reason: str | None = None
self._next_request_at = 0.0
async def _fetch_json(self, url: str, params: dict[str, object], proxy: bool) -> FetcherResponse:
@@ -488,14 +487,12 @@ class SearchShodan:
async def get_shodan_hosts(self) -> tuple[ShodanHostObservation, ...]:
return canonical_shodan_hosts(list(self.shodan_hosts.values()))
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
if self.word is None:
raise ValueError('A discovery target is required')
assert self.scope is not None
self.totalhosts.clear()
self.execution_status = None
self.stop_reason = None
dns_stop_reason: str | None = None
try:
resolved_ips = await resolve_ip_addresses(self.word, family=socket.AF_INET)
@@ -520,22 +517,13 @@ class SearchShodan:
provider_error_types.add(self.error_type)
self.error_type = next(iter(sorted(provider_error_types)), None)
retained_evidence = bool(self.totalhosts or self.shodan_hosts)
if dns_stop_reason is not None:
self.execution_status = 'partial' if retained_evidence else 'failed'
self.stop_reason = dns_stop_reason
return
return SourceExecutionReport('failed', dns_stop_reason)
if provider_error_types:
self.execution_status = 'partial' if retained_evidence else 'failed'
if provider_error_types <= {'HTTP401Error', 'HTTP403Error'}:
self.stop_reason = 'access-denied'
elif provider_error_types == {'HTTP429Error'}:
self.stop_reason = 'rate-limited'
elif len(resolved_ips) == 1:
self.stop_reason = 'provider-error'
else:
self.stop_reason = 'provider-errors'
return
self.execution_status = 'completed'
self.stop_reason = None if retained_evidence else 'no-results'
return SourceExecutionReport('failed', 'access-denied')
if provider_error_types == {'HTTP429Error'}:
return SourceExecutionReport('rate-limited', 'rate-limited')
reason = 'provider-error' if len(resolved_ips) == 1 else 'provider-errors'
return SourceExecutionReport('failed', reason)
return None
theHarvester/discovery/sourcegraph.py
+7 -9
View File
@@ -5,6 +5,7 @@ import re
from typing import Any
from theHarvester.lib.core import AsyncFetcher, ResponseStreamError
from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus
_HOST_TOKEN = re.compile(
r'(?<![\w*.-])(?:\*|[a-z0-9-]+)(?:\.(?:\*|[a-z0-9-]+))+\.?(?![\w*.-])',
@@ -92,15 +93,13 @@ class SearchSourcegraph:
self.word = ''
self.totalhosts: set[str] = set()
self.proxy: bool | str = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
self._report: SourceExecutionReport | None = None
self._saw_done = False
self._saw_terminal_progress = False
self._final_progress_skipped = False
def _stop(self, reason: str, status: str | None = None) -> None:
self.execution_status = status or ('partial' if self.totalhosts else 'failed')
self.stop_reason = reason
def _stop(self, reason: str, status: SourceReportStatus = 'failed') -> None:
self._report = SourceExecutionReport(status, reason)
def _add_content(self, content: str) -> None:
for match in _HOST_TOKEN.finditer(content):
@@ -215,13 +214,12 @@ class SearchSourcegraph:
self._stop('invalid-response')
elif self._final_progress_skipped:
self._stop('provider-limited', 'partial')
else:
self.execution_status = 'completed'
self.stop_reason = None if self.totalhosts else 'no-results'
async def get_hostnames(self) -> list[str]:
return sorted(self.totalhosts)
async def process(self, proxy: bool | str = False) -> None:
async def process(self, proxy: bool | str = False) -> SourceExecutionReport | None:
self.proxy = proxy
self._report = None
await self.do_search()
return self._report
theHarvester/discovery/subdomainfinderc99.py
+11 -24
View File
@@ -7,6 +7,7 @@ from bs4.element import Tag
from theHarvester.discovery.constants import get_delay
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
from theHarvester.parsers import myparser
@@ -18,14 +19,8 @@ class SearchSubdomainfinderc99:
# TODO add api support
self.server = 'https://subdomainfinder.c99.nl/'
self.totalresults = ''
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _stop(self, status: str, reason: str) -> None:
self.execution_status = status
self.stop_reason = reason
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
# Based on https://gist.github.com/th3gundy/bc83580cbe04031e9164362b33600962
headers = {'User-Agent': Core.get_browser_user_agent()}
async with AsyncFetcher.open_session(headers=headers, proxy=self.proxy) as session:
@@ -35,16 +30,13 @@ class SearchSubdomainfinderc99:
include_metadata=True,
)
if error := provider_http_error(metadata):
self._stop(*error)
return
return SourceExecutionReport(*error)
assert isinstance(metadata, FetcherResponse)
if not isinstance(metadata.body, str):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
data = await self.get_csrf_params(metadata.body)
if not data:
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
data['scan_subdomains'] = ''
data['domain'] = self.word
@@ -57,28 +49,23 @@ class SearchSubdomainfinderc99:
include_metadata=True,
)
if error := provider_http_error(second_resp):
self._stop(*error)
return
return SourceExecutionReport(*error)
assert isinstance(second_resp, FetcherResponse)
if not isinstance(second_resp.body, str):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
self.totalresults += second_resp.body
self.execution_status = 'completed'
self.stop_reason = None if await self.get_hostnames() else 'no-results'
return None
async def get_hostnames(self):
rawres = myparser.Parser(self.totalresults, self.word)
return await rawres.hostnames()
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
try:
await self.do_search()
return await self.do_search()
except Exception:
self._stop('failed', 'transport-error')
return SourceExecutionReport('failed', 'transport-error')
@staticmethod
async def get_csrf_params(data):
theHarvester/discovery/urlscan.py
+11 -34
View File
@@ -8,6 +8,7 @@ from theHarvester.lib.asn_attribution import AsnAttributionObservation, SubjectK
from theHarvester.lib.core import AsyncFetcher, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.result_values import normalize_asn
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -26,15 +27,6 @@ class SearchUrlscan:
self.totalasns: set = set()
self.asn_attributions: set[AsnAttributionObservation] = set()
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _has_results(self) -> bool:
return bool(self.totalhosts or self.totalips or self.urls or self.totalasns)
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self._has_results() else status
self.stop_reason = reason
@staticmethod
def _cursor(result: object) -> str | None:
@@ -142,7 +134,7 @@ class SearchUrlscan:
malformed = True
return malformed
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
url = 'https://urlscan.io/api/v1/search/'
collected_at = datetime.now(UTC)
cursor = None
@@ -168,21 +160,14 @@ class SearchUrlscan:
)
if error := provider_http_error(response):
self._stop(*error)
return
return SourceExecutionReport(*error)
assert isinstance(response, FetcherResponse)
if not isinstance(response.body, dict) or not isinstance(response.body.get('results'), list):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
results = response.body['results']
if not results:
if malformed:
self._stop('failed', 'invalid-response')
else:
self.execution_status = 'completed'
self.stop_reason = None if self._has_results() else 'no-results'
return
return SourceExecutionReport('failed', 'invalid-response') if malformed else None
page_results = results[:remaining]
records_seen += len(page_results)
@@ -191,23 +176,15 @@ class SearchUrlscan:
break
next_cursor = self._cursor(page_results[-1])
if next_cursor is None:
self._stop('failed', 'invalid-cursor')
return
return SourceExecutionReport('failed', 'invalid-cursor')
if next_cursor in seen_cursors:
self._stop('failed', 'repeated-cursor')
return
return SourceExecutionReport('failed', 'repeated-cursor')
seen_cursors.add(next_cursor)
cursor = next_cursor
except Exception as error:
self._stop('failed', 'transport-error')
logger.info('URLScan request failed: %s', type(error).__name__)
return
if self.execution_status is not None and self._has_results():
self.execution_status = 'partial'
elif self.execution_status is None:
self.execution_status = 'completed'
self.stop_reason = None if self._has_results() else 'no-results'
return SourceExecutionReport('failed', 'transport-error')
return SourceExecutionReport('failed', 'invalid-response') if malformed else None
async def get_hostnames(self) -> set:
return self.totalhosts
@@ -224,6 +201,6 @@ class SearchUrlscan:
async def get_asn_attributions(self) -> set[AsnAttributionObservation]:
return self.asn_attributions
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
return await self.do_search()
theHarvester/discovery/virustotal.py
+12 -28
View File
@@ -6,6 +6,7 @@ from theHarvester.discovery.constants import MissingKey
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
class SearchVirustotal:
@@ -19,18 +20,13 @@ class SearchVirustotal:
self.limit = limit
self.proxy = False
self.hostnames: set[str] = set()
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self.hostnames else status
self.stop_reason = reason
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
headers = {'Accept': 'application/json', 'x-apikey': self.key}
cursor: str | None = None
seen_cursors: set[str] = set()
records_seen = 0
report = None
try:
async with AsyncFetcher.open_session(headers=headers, proxy=self.proxy) as session:
while records_seen < self.limit:
@@ -46,17 +42,14 @@ class SearchVirustotal:
include_metadata=True,
)
if error := provider_http_error(response):
self._stop(*error)
return
return SourceExecutionReport(*error)
assert isinstance(response, FetcherResponse)
if not isinstance(response.body, dict):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
data = response.body.get('data')
meta = response.body.get('meta', {})
if not isinstance(data, list) or not isinstance(meta, dict):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
page_data = data[:remaining]
records_seen += len(page_data)
hostnames, malformed = self.parse_hostnames(page_data, self.word)
@@ -65,24 +58,17 @@ class SearchVirustotal:
break
self.hostnames.add(hostname)
if malformed:
self._stop('failed', 'invalid-response')
report = SourceExecutionReport('failed', 'invalid-response')
next_cursor = meta.get('cursor')
if not data or not isinstance(next_cursor, str) or not next_cursor:
break
if next_cursor in seen_cursors:
self._stop('failed', 'repeated-cursor')
break
return SourceExecutionReport('failed', 'repeated-cursor')
seen_cursors.add(next_cursor)
cursor = next_cursor
except Exception:
self._stop('failed', 'transport-error')
return
if self.execution_status is not None and self.hostnames:
self.execution_status = 'partial'
elif self.execution_status is None:
self.execution_status = 'completed'
self.stop_reason = None if self.hostnames else 'no-results'
return SourceExecutionReport('failed', 'transport-error')
return report
async def get_hostnames(self) -> set[str]:
return self.hostnames
@@ -132,8 +118,6 @@ class SearchVirustotal:
add(name)
return hostnames, malformed
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
await self.do_search()
return await self.do_search()
theHarvester/discovery/waybackarchive.py
+11 -16
View File
@@ -3,6 +3,7 @@ import logging
from urllib.parse import unquote_plus, urlencode, urlsplit
from theHarvester.lib.core import AsyncFetcher, Core
from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus
logger = logging.getLogger(__name__)
@@ -24,8 +25,6 @@ class SearchWaybackarchive:
self.totalhosts: set = set()
self.proxy = False
self.hostname = 'https://web.archive.org'
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _extract_domain_from_url(self, url: str) -> str:
"""Extract domain from URL"""
@@ -106,11 +105,9 @@ class SearchWaybackarchive:
logger.info(f'Wayback Archive page limit reached for pattern {pattern}; results may be incomplete')
return 'page-limit'
async def do_search(self) -> None:
self.execution_status = None
self.stop_reason = None
async def do_search(self) -> SourceExecutionReport | None:
if self.limit == 0:
return
return None
try:
headers = {'User-agent': Core.get_user_agent()}
degraded_reason: str | None = None
@@ -125,7 +122,7 @@ class SearchWaybackarchive:
continue
if outcome == 'result-limit':
if degraded_reason is None:
self.stop_reason = 'result-limit'
return SourceExecutionReport('completed', 'result-limit')
break
if outcome == 'page-limit':
degraded_reason = degraded_reason or outcome
@@ -133,24 +130,22 @@ class SearchWaybackarchive:
if outcome is not None:
degraded_reason = degraded_reason or outcome
except TimeoutError:
self.execution_status = 'partial' if self.totalhosts else 'failed'
self.stop_reason = 'runtime-limit'
logger.info(
f'Wayback Archive runtime limit reached after {self.RUNTIME_SECONDS:g}s; '
f'preserved {len(self.totalhosts)} hosts'
)
return
return SourceExecutionReport('failed', 'runtime-limit')
if degraded_reason is not None:
self.execution_status = 'partial' if self.totalhosts or degraded_reason == 'page-limit' else 'failed'
self.stop_reason = degraded_reason
status: SourceReportStatus = 'partial' if degraded_reason == 'page-limit' else 'failed'
return SourceExecutionReport(status, degraded_reason)
except Exception as e:
self.execution_status = 'partial' if self.totalhosts else 'failed'
self.stop_reason = 'unexpected-error'
logger.info(f'Wayback Archive API error: {e}')
return SourceExecutionReport('failed', 'unexpected-error')
return None
async def get_hostnames(self) -> set:
return self.totalhosts
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
return await self.do_search()
theHarvester/discovery/whoisxml.py
+13 -26
View File
@@ -4,6 +4,7 @@ from theHarvester.discovery.constants import MissingKey
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
class SearchWhoisXML:
@@ -17,17 +18,12 @@ class SearchWhoisXML:
raise MissingKey('whoisxml')
self.total_results: set[str] = set()
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
def _stop(self, status: str, reason: str) -> None:
self.execution_status = 'partial' if self.total_results else status
self.stop_reason = reason
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
cursor: str | None = None
seen_cursors: set[str] = set()
records_seen = 0
report = None
async with AsyncFetcher.open_session(proxy=self.proxy) as session:
while records_seen < self.limit:
params = {'apiKey': self.key, 'domainName': self.word}
@@ -41,20 +37,16 @@ class SearchWhoisXML:
include_metadata=True,
)
if error := provider_http_error(response):
self._stop(*error)
return
return SourceExecutionReport(*error)
assert isinstance(response, FetcherResponse)
if not isinstance(response.body, dict):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
result = response.body.get('result')
if not isinstance(result, dict) or not isinstance(result.get('records'), list):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
next_cursor = result.get('nextPageSearchAfter')
if not isinstance(next_cursor, str):
self._stop('failed', 'invalid-response')
return
return SourceExecutionReport('failed', 'invalid-response')
remaining = self.limit - records_seen
records = result['records'][:remaining]
@@ -67,26 +59,21 @@ class SearchWhoisXML:
if hostname := normalize_scoped_hostname(record['domain'], self.word):
self.total_results.add(hostname)
if malformed:
self._stop('failed', 'invalid-response')
report = SourceExecutionReport('failed', 'invalid-response')
if records_seen >= self.limit or not next_cursor:
break
if next_cursor in seen_cursors:
self._stop('failed', 'repeated-cursor')
break
return SourceExecutionReport('failed', 'repeated-cursor')
seen_cursors.add(next_cursor)
cursor = next_cursor
if self.execution_status is None:
self.execution_status = 'completed'
self.stop_reason = None if self.total_results else 'no-results'
return report
async def get_hostnames(self) -> set[str]:
return self.total_results
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
try:
await self.do_search()
return await self.do_search()
except Exception:
self._stop('failed', 'transport-error')
return SourceExecutionReport('failed', 'transport-error')
theHarvester/discovery/zoomeyesearch.py
+8 -17
View File
@@ -11,6 +11,7 @@ from theHarvester.discovery.constants import MissingKey
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus
from theHarvester.parsers import myparser
@@ -49,13 +50,10 @@ class SearchZoomEye:
self.urls: set[str] = set()
self.totalips: set[str] = set()
self.totalemails: set[str] = set()
self.execution_status: str | None = None
self.stop_reason: str | None = None
self._report: SourceExecutionReport | None = None
def _stop(self, status: str, reason: str) -> None:
has_results = any((self.totalhosts, self.totalemails, self.totalips, self.totalasns, self.urls))
self.execution_status = 'partial' if has_results else status
self.stop_reason = reason
def _stop(self, status: SourceReportStatus, reason: str) -> None:
self._report = SourceExecutionReport(status, reason)
def _normalize_url(self, value: Any) -> str | None:
if not isinstance(value, str):
@@ -200,10 +198,9 @@ class SearchZoomEye:
return hostnames, emails, ips, asns, urls, malformed
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
self.execution_status = None
self.stop_reason = None
self._report = None
try:
async with AsyncFetcher.open_session(
headers={'API-KEY': self.key, 'Content-Type': 'application/json'},
@@ -211,14 +208,8 @@ class SearchZoomEye:
) as session:
await self.do_search(session)
except Exception:
self._stop('failed', 'transport-error')
return
has_results = any((self.totalhosts, self.totalemails, self.totalips, self.totalasns, self.urls))
if self.execution_status is not None and has_results:
self.execution_status = 'partial'
elif self.execution_status is None:
self.execution_status = 'completed'
self.stop_reason = None if has_results else 'no-results'
return SourceExecutionReport('failed', 'transport-error')
return self._report
async def get_hostnames(self) -> set[str]:
return self.totalhosts
theHarvester/lib/source_execution.py
+18
View File
@@ -0,0 +1,18 @@
from dataclasses import dataclass
from typing import Literal
SourceReportStatus = Literal['completed', 'partial', 'failed', 'rate-limited']
@dataclass(frozen=True, slots=True)
class SourceExecutionReport:
"""Provider-specific terminal details for one source execution."""
status: SourceReportStatus
stop_reason: str
def __post_init__(self) -> None:
if self.status not in {'completed', 'partial', 'failed', 'rate-limited'}:
raise ValueError(f'adapter cannot report execution status {self.status!r}')
if not isinstance(self.stop_reason, str) or not self.stop_reason.strip():
raise ValueError('adapter stop reason must not be empty')
theHarvester/lib/source_runner.py
+19 -19
View File
@@ -7,7 +7,7 @@ import time
from collections.abc import Awaitable, Callable, Iterable
from dataclasses import dataclass
from ipaddress import ip_address
from typing import Any, cast
from typing import TYPE_CHECKING, Any
from theHarvester.discovery import (
apisguru,
@@ -71,17 +71,15 @@ from theHarvester.discovery import (
)
from theHarvester.discovery.constants import MissingKeyError
from theHarvester.lib.asn_attribution import AsnAttributionObservation, canonical_asn_attributions
from theHarvester.lib.completed_result import (
EXECUTION_STATUSES,
ExecutionStatus,
ResultKind,
ResultObservation,
SourceExecution,
)
from theHarvester.lib.completed_result import ResultKind, ResultObservation, SourceExecution
from theHarvester.lib.enumeration import DEFAULT_SOURCE_WORKERS
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.shodan_evidence import ShodanHostObservation, canonical_shodan_hosts
from theHarvester.lib.source_catalog import ResultRoute, get_source_spec
from theHarvester.lib.source_execution import SourceExecutionReport
if TYPE_CHECKING:
from theHarvester.lib.evidence_types import ExecutionStatus
logger = logging.getLogger(__name__)
@@ -324,8 +322,16 @@ async def run_source(
except Exception as error:
logger.warning('Source start reporter failed for %s: %s', request.source, type(error).__name__)
adapter = created_adapter
await adapter.process(request.proxy)
report = await adapter.process(request.proxy)
process_completed = True
if report is None:
status: ExecutionStatus = 'completed'
stop_reason = None
elif isinstance(report, SourceExecutionReport):
status = report.status
stop_reason = report.stop_reason
else:
raise ValueError(f'Source {source_spec.name} returned invalid execution report: {report!r}')
await _collect_observations(
request,
adapter,
@@ -334,23 +340,17 @@ async def run_source(
shodan_hosts,
reported_host_ip_pairs,
)
reported_status = getattr(adapter, 'execution_status', None)
if reported_status is None:
status: ExecutionStatus = 'completed'
elif isinstance(reported_status, str) and reported_status in EXECUTION_STATUSES:
status = cast('ExecutionStatus', reported_status)
else:
raise ValueError(f'Source {source_spec.name} reported invalid execution status: {reported_status!r}')
stop_reason = getattr(adapter, 'stop_reason', None)
result_count = len(observations)
if not result_count and status == 'completed' and not isinstance(stop_reason, str):
if result_count and status != 'completed':
status = 'partial'
elif not result_count and status == 'completed' and stop_reason is None:
stop_reason = 'no-results'
execution = SourceExecution(
source_spec.name,
status,
(time.perf_counter() - started) * 1000,
result_count,
stop_reason=stop_reason if isinstance(stop_reason, str) else None,
stop_reason=stop_reason,
)
except MissingKeyError:
execution = SourceExecution(