diff --git a/CHANGELOG.md b/CHANGELOG.md index 30ac0195..b3841b22 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Added root contributor and security policies, structured issue forms, repository agent guidance, discovery terminology, and an operator-focused documentation wiki ([d090a29a](https://github.com/laramies/theHarvester/commit/d090a29a), [7c491ef5](https://github.com/laramies/theHarvester/commit/7c491ef5), [8b9d420b](https://github.com/laramies/theHarvester/commit/8b9d420b)). ### Changed +- Replaced Shodan's synchronous Python SDK with cancellable async Host API requests that honor configured proxies, query every unique resolved IPv4, paginate target-bound hostname and TLS-certificate searches without an adapter-specific result cap, retain successful partial results, and add no source-local deadline. Shodan now stores one canonical `shodan-host` result per IP with every normalized TCP or UDP service and scoped certificate CN/SAN metadata in native JSONL, SQLite, API, and HarvestView details instead of an escaped JSON value. - Removed the transport-wide delay before reading ready HTTP responses, bounded Wayback Archive to 30 seconds and Common Crawl to 120 seconds, kept both sources within the requested result limit, and made long-source progress visible in verbose mode. Common Crawl now requests one 50-record page at a time instead of bursting page batches. - Made Baidu, crt.sh, HackerTarget, Have I Been Pwned, Mojeek, OTX, and Robtex report blocked, malformed, or transport failures truthfully. Also fixed HackerTarget CSV parsing and Robtex AAAA results. - Hardened BufferOver, ProjectDiscovery, DNSDumpster, ONYPHE, and URLScan parsing and result attribution, including scoped typed results and bounded URLScan pagination. diff --git a/README.md b/README.md index 06714d24..7beb060e 100644 --- a/README.md +++ b/README.md @@ -247,6 +247,7 @@ On first use, theHarvester creates default configuration files under `~/.theHarv - `api-keys.yaml` stores provider credentials. - `proxies.yaml` configures HTTP and SOCKS5 proxies used with `-p`. +- The `shodan` source and `-s` / `--shodan` enrichment call Shodan's Host REST API without the Python SDK. When `-p` is enabled, both send those requests through `proxies.yaml`. Never commit populated configuration files, API keys, account details, or provider responses. @@ -273,7 +274,7 @@ The JSON report is a single object. Host entries remain plain hostnames or `host | --- | --- | --- | | `cmd` | Always | Command-line arguments used for the run. | | `hosts` | Always | Discovered hosts; an empty array when none are found. | -| `shodan` | Always | Shodan enrichment rows; an empty array when Shodan is not used. | +| `shodan` | Always | Shodan host objects with canonical IP `value` and structured `details`; an empty array when Shodan is not used. | | `ips`, `emails`, `vhosts`, `asns`, `prefixes` | When non-empty | Network and contact findings. RouteViews prefixes are external routing relationships, not claimed target scope. | | `urls` | When non-empty | Discovered URLs from every URL-producing source or action. | | `people`, `twitter_people`, `linkedin_people` | When non-empty | People and profile findings. | @@ -288,7 +289,9 @@ The JSONL report is finalized after the selected one-shot actions finish. The fi {"sources":[],"type":"hostname","value":"api.example.com"} ``` -JSONL is easy to stream one record at a time. The summary preserves the evidence status, source and action outcomes, and screenshot artifact metadata. Finding lines carry `sources` and, when applicable, `actions`; they inherit their run ID and target from the preceding summary. Hostnames, IP addresses, and URLs use the same `hostname`, `ip`, and `url` result kinds in JSONL, SQLite, the API, and HarvestView. Provenance identifies which source or action produced each finding. Structured result types, including recursive DNS records plus `person`, `infostealer`, `shodan`, and `takeover`, store a JSON object inside the string `value`. Parse those values a second time with `fromjson`. +JSONL is easy to stream one record at a time. The summary preserves the evidence status, source and action outcomes, and screenshot artifact metadata. Finding lines carry `sources` and, when applicable, `actions`; they inherit their run ID and target from the preceding summary. Hostnames, IP addresses, and URLs use the same `hostname`, `ip`, and `url` result kinds in JSONL, SQLite, the API, and HarvestView. Provenance identifies which source or action produced each finding. Recursive DNS records plus `person`, `infostealer`, and `takeover` store a JSON object inside the string `value`; parse those values a second time with `fromjson`. + +Shodan host findings instead use the canonical IP as `value` and place normalized host and per-service evidence in a native `details` object. Shodan discovery paginates both hostname and TLS-certificate searches for the target domain without an adapter-specific result cap, merges duplicate services by IP, and rejects names outside the requested domain. Host metadata appears once, while each service retains its port, TCP or UDP transport, product, version, observation time, CPEs, and available HTTP or TLS summary, including scoped certificate CNs and SANs. Raw banners, response bodies, certificate chains, and Shodan crawler metadata are not retained. Virtual-host observations do not use that string encoding. Each confirmed name remains one `hostname` finding with `actions: ["vhost"]` and a native `observations` array. Several endpoint observations can enrich the same hostname without creating another result kind or count. @@ -302,6 +305,12 @@ Parse recursive DNS findings as JSON objects: jq -c 'select(.type == "dns-recursive-finding") | .value | fromjson' report.jsonl ``` +List Shodan services by host: + +```bash +jq -c 'select(.type == "shodan-host") | {ip: .value, services: .details.services}' report.jsonl +``` + List the endpoint observations for each confirmed virtual host: ```bash diff --git a/pyproject.toml b/pyproject.toml index 8f7892b3..198fb3a7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -37,7 +37,6 @@ dependencies = [ "python-dateutil==2.9.0.post0", "httpx==0.28.1", "retrying==1.4.2", - "shodan==1.31.0", "sqlalchemy==2.0.51", "ujson==5.13.0", "uvicorn==0.52.1", diff --git a/tests/discovery/test_shodan_engine.py b/tests/discovery/test_shodan_engine.py index 6659b203..5178b29b 100644 --- a/tests/discovery/test_shodan_engine.py +++ b/tests/discovery/test_shodan_engine.py @@ -1,35 +1,172 @@ +import asyncio import logging import socket import sys from collections import OrderedDict from datetime import UTC +from types import SimpleNamespace +from uuid import UUID import pytest +def patch_resolution(monkeypatch, module, addresses=('203.0.113.10',)): + requested_targets = [] + + async def resolve_ip_addresses(target, *, family=socket.AF_UNSPEC): + requested_targets.append((target, family)) + return tuple(addresses) + + monkeypatch.setattr(module, 'resolve_ip_addresses', resolve_ip_addresses, raising=True) + return requested_targets + + +@pytest.fixture(autouse=True) +def disable_shodan_pacing(monkeypatch): + from theHarvester.discovery import shodansearch + + monkeypatch.setattr(shodansearch.SearchShodan, 'REQUEST_INTERVAL_SECONDS', 0.0) + + class TestShodanEngine: @pytest.mark.asyncio - async def test_shodan_retains_sourced_asn_organization_attribution(self, monkeypatch): - from theHarvester.discovery import shodansearch + async def test_shared_ipv4_resolution_returns_every_unique_address_and_closes(self, monkeypatch): + from theHarvester.lib import hostchecker - class SuccessfulShodan: - def host(self, _ip): - return { + closed = False + + class Resolver: + async def getaddrinfo(self, target, *, family): + assert target == 'example.test' + assert family == socket.AF_INET + return SimpleNamespace( + nodes=[ + SimpleNamespace(addr=(b'203.0.113.11', 0)), + SimpleNamespace(addr=('invalid', 0)), + SimpleNamespace(addr=('203.0.113.10', 0)), + SimpleNamespace(addr=('203.0.113.11', 0)), + ] + ) + + async def close(self): + nonlocal closed + closed = True + + monkeypatch.setattr(hostchecker.aiodns, 'DNSResolver', Resolver) + + assert await hostchecker.resolve_ip_addresses('example.test', family=socket.AF_INET) == ( + '203.0.113.10', + '203.0.113.11', + ) + assert closed is True + + @pytest.mark.asyncio + async def test_shodan_calls_direct_host_api_through_proxy_and_retains_attribution(self, monkeypatch): + from theHarvester.discovery import shodansearch + from theHarvester.lib.core import FetcherResponse + + async def fetch_json(url, *, params, proxy, request_timeout): + assert url == 'https://api.shodan.io/shodan/host/192.0.2.10' + assert 'test-key' not in url + assert params == {'key': 'test-key'} + assert proxy is True + assert request_timeout is None + return FetcherResponse( + body={ 'asn': 'AS64496', - 'org': 'Example Transit', + 'domains': ['Example.TEST.'], + 'hostnames': ['API.Example.TEST.'], + 'ip_str': '192.0.2.10', 'isp': 'Example ISP Label', - 'data': [{'ip_str': '198.51.100.20'}], - } + 'org': 'Example Transit', + 'data': [ + { + 'port': 53, + 'transport': 'udp', + 'product': 'dnsmasq', + 'version': '2.90', + 'timestamp': '2026-08-14T11:58:00Z', + 'cpe': ['cpe:/a:thekelleys:dnsmasq:2.90'], + 'data': 'raw provider banner must not be retained', + }, + { + 'port': 443, + 'transport': 'tcp', + 'product': 'nginx', + 'version': '1.24.0', + 'timestamp': '2026-08-14T12:01:00Z', + 'http': { + 'components': {'nginx': {}, 'Python': {}}, + 'server': 'nginx', + 'title': 'Example', + 'html': 'raw response body must not be retained', + }, + 'ssl': { + 'jarm': 'example-jarm', + 'cert': { + 'subject': {'CN': 'api.example.test'}, + 'issuer': {'CN': 'Example CA'}, + 'expires': '2027-08-14T00:00:00Z', + 'fingerprint': {'sha256': '0123456789abcdef'}, + 'extensions': [ + { + 'name': 'subjectAltName', + 'data': r'0\x82\x10api.example.test\x82\x10www.example.test', + } + ], + }, + }, + }, + ], + }, + status=200, + headers={}, + ) monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') - monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: SuccessfulShodan()) + monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json) search = shodansearch.SearchShodan() - await search.search_ip('192.0.2.10') + result = await search.search_ip('192.0.2.10', proxy=True) - attributions = await search.get_asn_attributions() - assert len(attributions) == 1 - attribution = next(iter(attributions)) + assert result['192.0.2.10'] == { + 'asn': 'AS64496', + 'domains': ['example.test'], + 'hostnames': ['api.example.test'], + 'isp': 'Example ISP Label', + 'organization': 'Example Transit', + 'services': [ + { + 'cpes': ['cpe:/a:thekelleys:dnsmasq:2.90'], + 'observed_at': '2026-08-14T11:58:00Z', + 'port': 53, + 'product': 'dnsmasq', + 'transport': 'udp', + 'version': '2.90', + }, + { + 'http': { + 'components': ['Python', 'nginx'], + 'server': 'nginx', + 'title': 'Example', + }, + 'observed_at': '2026-08-14T12:01:00Z', + 'port': 443, + 'product': 'nginx', + 'tls': { + 'expires_at': '2027-08-14T00:00:00Z', + 'issuer_cn': 'Example CA', + 'jarm': 'example-jarm', + 'sha256': '0123456789abcdef', + 'subject_alt_names': ['www.example.test'], + 'subject_cn': 'api.example.test', + }, + 'transport': 'tcp', + 'version': '1.24.0', + }, + ], + } + attribution = next(iter(await search.get_asn_attributions())) assert attribution.producer_kind == 'action' assert attribution.producer == 'shodan' assert attribution.asn == 'AS64496' @@ -39,15 +176,15 @@ class TestShodanEngine: assert attribution.collected_at.tzinfo is UTC @pytest.mark.asyncio - async def test_shodan_provider_failure_returns_attributed_empty_evidence(self, monkeypatch, caplog): + async def test_shodan_maps_404_to_a_successful_zero_yield(self, monkeypatch, caplog): from theHarvester.discovery import shodansearch + from theHarvester.lib.core import FetcherResponse - class FailingShodan: - def host(self, _ip): - raise shodansearch.exception.APIError('No information available for that IP.') + async def fetch_json(*_args, **_kwargs): + return FetcherResponse(body=None, status=404, headers={}) monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') - monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: FailingShodan()) + monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json) caplog.set_level(logging.INFO, logger=shodansearch.__name__) search = shodansearch.SearchShodan() @@ -58,67 +195,385 @@ class TestShodanEngine: assert '203.0.113.1: Not in Shodan' in caplog.text @pytest.mark.asyncio - async def test_shodan_api_failure_exposes_only_its_error_type(self, monkeypatch, caplog): + @pytest.mark.parametrize('status', [401, 403, 429, 500]) + async def test_shodan_classifies_http_failures_without_provider_payload(self, monkeypatch, caplog, status): from theHarvester.discovery import shodansearch + from theHarvester.lib.core import FetcherResponse - class FailingShodan: - def host(self, _ip): - raise shodansearch.exception.APIError('provider-secret-payload') + async def fetch_json(*_args, **_kwargs): + return FetcherResponse(body=None, status=status, headers={'x-provider-secret': 'secret'}) monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') - monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: FailingShodan()) + monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json) caplog.set_level(logging.INFO, logger=shodansearch.__name__) search = shodansearch.SearchShodan() result = await search.search_ip('203.0.113.1') assert result == OrderedDict({'203.0.113.1': 'Shodan request failed'}) - assert search.error_type == 'APIError' - assert 'provider-secret-payload' not in caplog.text + assert search.error_type == f'HTTP{status}Error' + assert 'secret' not in caplog.text @pytest.mark.asyncio - async def test_shodan_unexpected_failure_exposes_only_its_error_type(self, monkeypatch): + async def test_shodan_classifies_bounded_transport_failure(self, monkeypatch): from theHarvester.discovery import shodansearch + from theHarvester.lib.core import ResponseStreamError - class FailingShodan: - def host(self, _ip): - raise RuntimeError('provider-secret-payload') + async def fetch_json(*_args, **_kwargs): + raise ResponseStreamError('response-limit') monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') - monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: FailingShodan()) + monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json) search = shodansearch.SearchShodan() result = await search.search_ip('203.0.113.1') assert result == OrderedDict({'203.0.113.1': 'Shodan request failed'}) - assert search.error_type == 'RuntimeError' + assert search.error_type == 'ResponseLimitError' @pytest.mark.asyncio - async def test_shodan_engine_processes_without_work_item_error_and_yields_hostnames(self, monkeypatch, capsys): - # Import inside the test so monkeypatching affects the already-imported module namespace. + async def test_shodan_discovery_queries_every_resolved_ipv4_and_keeps_partial_results(self, monkeypatch): + from theHarvester.discovery import shodansearch + from theHarvester.lib.core import FetcherResponse + + queried_urls = [] + + async def fetch_json(url, **_kwargs): + if url.endswith('/search'): + return FetcherResponse(body={'matches': [], 'total': 0}, status=200, headers={}) + queried_urls.append(url) + ip = url.rsplit('/', 1)[-1] + if ip == '203.0.113.10': + return FetcherResponse(body=None, status=500, headers={}) + return FetcherResponse( + body={ + 'data': [{'ip_str': ip, 'port': 443, 'transport': 'tcp'}], + 'hostnames': ['CDN.Example.TEST.', 'outside.test'], + }, + status=200, + headers={}, + ) + + monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') + monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json) + targets = patch_resolution(monkeypatch, shodansearch, ('203.0.113.10', '203.0.113.11')) + + search = shodansearch.SearchShodan('WWW.Example.TEST.') + await search.process(proxy=True) + + assert targets == [('www.example.test', socket.AF_INET)] + assert queried_urls == [ + 'https://api.shodan.io/shodan/host/203.0.113.10', + 'https://api.shodan.io/shodan/host/203.0.113.11', + ] + assert await search.get_hostnames() == {'cdn.example.test'} + assert search.execution_status == 'partial' + assert search.stop_reason == 'provider-errors' + + @pytest.mark.asyncio + async def test_shodan_discovery_paginates_hostname_and_tls_searches_with_scoped_certificate_names(self, monkeypatch): + from theHarvester.discovery import shodansearch + from theHarvester.lib.core import FetcherResponse + + search_calls = [] + + def banner( + ip, + port, + *, + hostnames=(), + subject_cn=None, + subject_alt_name='', + ): + cert = { + 'extensions': [ + {'name': 'subjectAltName', 'data': subject_alt_name}, + {'name': 'keyUsage', 'data': 'Digital Signature'}, + ] + } + if subject_cn is not None: + cert['subject'] = {'CN': subject_cn} + return { + 'ip_str': ip, + 'hostnames': list(hostnames), + 'port': port, + 'transport': 'tcp', + 'ssl': {'cert': cert}, + } + + hostname_pages = { + 1: { + 'total': 2, + 'matches': [ + banner( + '198.51.100.20', + 443, + hostnames=('API.Example.TEST.',), + subject_cn='*.example.test', + subject_alt_name=(r'0\x82\x10api.example.test\x82\x12outside.invalid\x82\x13vpn.example.test'), + ) + ], + }, + 2: { + 'total': 2, + 'matches': [banner('198.51.100.20', 8443, hostnames=('api.example.test',))], + }, + } + ssl_page = { + 'total': 3, + 'matches': [ + banner( + '198.51.100.20', + 443, + hostnames=('api.example.test',), + subject_cn='*.example.test', + subject_alt_name=r'0\x82\x10api.example.test\x82\x13vpn.example.test', + ), + banner( + '198.51.100.21', + 443, + subject_cn='cert.example.test', + subject_alt_name=r'0\x82\x13cert.example.test\x82\x15outside.invalid', + ), + banner( + '198.51.100.22', + 443, + hostnames=('outside.invalid',), + subject_cn='outside.invalid', + subject_alt_name=r'0\x82\x15outside.invalid', + ), + ], + } + + async def fetch_json(url, *, params, proxy, request_timeout): + assert proxy is True + assert request_timeout is None + assert params['key'] == 'test-key' + if url.endswith('/search'): + search_calls.append(dict(params)) + query = params['query'] + if query == 'hostname:example.test': + return FetcherResponse(body=hostname_pages[params['page']], status=200, headers={}) + assert query == 'ssl:example.test' + assert params['page'] == 1 + return FetcherResponse(body=ssl_page, status=200, headers={}) + assert url == 'https://api.shodan.io/shodan/host/203.0.113.10' + return FetcherResponse( + body={'data': [{'port': 80, 'transport': 'tcp'}], 'hostnames': ['www.example.test']}, + status=200, + headers={}, + ) + + monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') + monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json) + patch_resolution(monkeypatch, shodansearch) + + search = shodansearch.SearchShodan('example.test') + await search.process(proxy=True) + + assert [(call['query'], call['page']) for call in search_calls] == [ + ('hostname:example.test', 1), + ('hostname:example.test', 2), + ('ssl:example.test', 1), + ] + assert all(call['minify'] == 'false' and call['fields'] == search.SEARCH_FIELDS for call in search_calls) + assert await search.get_hostnames() == { + 'api.example.test', + 'cert.example.test', + 'vpn.example.test', + 'www.example.test', + } + hosts = {host.ip: host.to_details() for host in await search.get_shodan_hosts()} + assert set(hosts) == {'198.51.100.20', '198.51.100.21', '203.0.113.10'} + assert [service['port'] for service in hosts['198.51.100.20']['services']] == [443, 8443] + assert hosts['198.51.100.20']['services'][0]['tls'] == { + 'subject_alt_names': ['api.example.test', 'vpn.example.test'], + 'subject_cn': '*.example.test', + } + assert hosts['198.51.100.21']['services'][0]['tls'] == {'subject_cn': 'cert.example.test'} + assert search.execution_status == 'completed' + assert search.stop_reason is None + + @pytest.mark.asyncio + async def test_shodan_discovery_counts_service_only_evidence_as_a_result(self, monkeypatch): + from theHarvester.discovery import shodansearch + from theHarvester.lib.core import FetcherResponse + + async def fetch_json(url, **_kwargs): + if url.endswith('/search'): + return FetcherResponse(body={'matches': [], 'total': 0}, status=200, headers={}) + return FetcherResponse( + body={'data': [{'port': 53, 'transport': 'udp'}], 'hostnames': []}, + status=200, + headers={}, + ) + + monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') + monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json) + patch_resolution(monkeypatch, shodansearch) + + search = shodansearch.SearchShodan('example.test') + await search.process() + + assert search.execution_status == 'completed' + assert search.stop_reason is None + assert [host.ip for host in await search.get_shodan_hosts()] == ['203.0.113.10'] + + @pytest.mark.asyncio + async def test_shodan_discovery_retains_valid_services_and_reports_malformed_provider_data(self, monkeypatch): + from theHarvester.discovery import shodansearch + from theHarvester.lib.core import FetcherResponse + + async def fetch_json(*_args, **_kwargs): + return FetcherResponse( + body={ + 'data': [ + {'port': 53, 'transport': 'udp', 'product': {'raw': 'provider object'}}, + {'transport': 'tcp'}, + ], + 'hostnames': [], + }, + status=200, + headers={}, + ) + + monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') + monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json) + patch_resolution(monkeypatch, shodansearch) + + search = shodansearch.SearchShodan('example.test') + await search.process() + + assert search.execution_status == 'partial' + assert search.stop_reason == 'provider-error' + assert search.error_type == 'InvalidResponseError' + assert (await search.get_shodan_hosts())[0].to_details() == {'services': [{'port': 53, 'transport': 'udp'}]} + + @pytest.mark.asyncio + @pytest.mark.parametrize( + ('status', 'expected_reason'), + [(401, 'access-denied'), (403, 'access-denied'), (429, 'rate-limited')], + ) + async def test_shodan_discovery_reports_specific_provider_stop_reason(self, monkeypatch, status, expected_reason): + from theHarvester.discovery import shodansearch + from theHarvester.lib.core import FetcherResponse + + async def fetch_json(*_args, **_kwargs): + return FetcherResponse(body=None, status=status, headers={}) + + monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') + monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json) + patch_resolution(monkeypatch, shodansearch) + + search = shodansearch.SearchShodan('example.test') + await search.process() + + assert search.execution_status == 'failed' + assert search.stop_reason == expected_reason + + def test_shodan_discovery_requires_a_configured_key(self, monkeypatch): + from theHarvester.discovery import shodansearch + from theHarvester.discovery.constants import MissingKey + + monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: None) + + with pytest.raises(MissingKey): + shodansearch.SearchShodan('example.test') + + def test_shodan_discovery_rejects_query_filter_injection(self, monkeypatch): + from theHarvester.discovery import shodansearch + + monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') + + with pytest.raises(ValueError, match='must be a hostname'): + shodansearch.SearchShodan('example.test ssl:true') + + @pytest.mark.asyncio + async def test_shodan_discovery_reports_dns_failure_after_searching_provider(self, monkeypatch): + from theHarvester.discovery import shodansearch + from theHarvester.lib.core import FetcherResponse + + provider_queries = [] + + async def fail_resolution(_target, *, family): + assert family == socket.AF_INET + raise shodansearch.aiodns.error.DNSError(shodansearch.aiodns.error.ARES_ENOTFOUND, 'not found') + + async def fetch_json(url, *, params, **_kwargs): + assert url.endswith('/search') + provider_queries.append(params['query']) + return FetcherResponse(body={'matches': [], 'total': 0}, status=200, headers={}) + + monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') + monkeypatch.setattr(shodansearch, 'resolve_ip_addresses', fail_resolution) + monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json) + + search = shodansearch.SearchShodan('example.test') + await search.process() + + assert not await search.get_hostnames() + assert provider_queries == ['hostname:example.test', 'ssl:example.test'] + assert search.execution_status == 'failed' + assert search.stop_reason == 'dns-resolution-failed' + + @pytest.mark.asyncio + async def test_shodan_direct_request_cancellation_propagates(self, monkeypatch): + from theHarvester.discovery import shodansearch + + entered = asyncio.Event() + release = asyncio.Event() + + async def fetch_json(*_args, **_kwargs): + entered.set() + await release.wait() + raise AssertionError('cancelled request resumed') + + monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') + monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json) + patch_resolution(monkeypatch, shodansearch) + + task = asyncio.create_task(shodansearch.SearchShodan('example.test').process()) + await entered.wait() + task.cancel('operator-stop') + with pytest.raises(asyncio.CancelledError, match='operator-stop'): + await task + release.set() + + @pytest.mark.asyncio + async def test_shodan_engine_processes_and_persists_sourced_hostnames(self, monkeypatch, capsys, tmp_path): import theHarvester.__main__ as main_module + from theHarvester.lib.completed_result import ResultObservation, SourceExecution + from theHarvester.lib.database import ResultStore + from theHarvester.lib.shodan_evidence import ShodanHostObservation - # Make DNS resolution deterministic and offline. - monkeypatch.setattr(socket, 'gethostbyname', lambda _domain: '1.2.3.4', raising=True) + database = tmp_path / 'stash.sqlite' + monkeypatch.setattr(main_module, 'ResultStore', lambda: ResultStore(database), raising=True) - # Avoid filesystem/sqlite side effects. - class DummyResultStore: - async def initialize(self) -> None: - return None - - async def record_observations(self, domain, all, res_type, source) -> None: - return None - - monkeypatch.setattr(main_module, 'ResultStore', DummyResultStore, raising=True) - - # Stub Shodan search to avoid network and API key requirements. class DummySearchShodan: - async def search_ip(self, ip): - return OrderedDict({ip: {'hostnames': ['a.example.com', 'b.example.com']}}) + def __init__(self, domain): + assert domain == 'example.com' + self.shodan_host = ShodanHostObservation.from_record( + '192.0.2.10', + { + 'organization': 'Example Transit', + 'hostnames': ['a.example.com', 'b.example.com'], + 'services': [ + {'port': 53, 'transport': 'udp'}, + {'port': 443, 'transport': 'tcp', 'product': 'nginx'}, + ], + }, + ) + + async def process(self, proxy=False): + return None + + async def get_hostnames(self): + return {'a.example.com', 'b.example.com'} + + async def get_shodan_hosts(self): + return (self.shodan_host,) monkeypatch.setattr(main_module.shodansearch, 'SearchShodan', DummySearchShodan, raising=True) - - # Run the CLI path that uses the engine queue/worker (`-b shodan`). monkeypatch.setattr(sys, 'argv', ['theHarvester', '-d', 'example.com', '-b', 'shodan'], raising=True) with pytest.raises(SystemExit) as excinfo: @@ -127,54 +582,42 @@ class TestShodanEngine: out = capsys.readouterr().out assert 'An error occurred while processing a "work item"' not in out - output_tokens = set(out.split()) - assert {'a.example.com', 'b.example.com'} <= output_tokens + assert {'a.example.com', 'b.example.com'} <= set(out.split()) + assert '"type": "shodan-host"' in out + assert '"value": "192.0.2.10"' in out + assert '"transport": "udp"' in out + assert '"transport": "tcp"' in out + + store = ResultStore(database) + try: + runs = await store.list_runs() + completed = await store.load_run(UUID(str(runs[0]['run_id']))) + assert len(completed.source_executions) == 1 + execution = completed.source_executions[0] + assert execution == SourceExecution('shodan', 'completed', execution.duration_ms, 3) + assert completed.observations == ( + ResultObservation('shodan', 'hostname', 'a.example.com'), + ResultObservation('shodan', 'hostname', 'b.example.com'), + ResultObservation('shodan', 'shodan-host', '192.0.2.10'), + ) + assert completed.shodan_hosts == (DummySearchShodan('example.com').shodan_host,) + finally: + await store.dispose() @pytest.mark.asyncio - async def test_shodan_internetdb_ignores_non_string_resolved_addresses(self, monkeypatch): + async def test_shodan_internetdb_uses_shared_resolution_and_normalizes_scope(self, monkeypatch): from theHarvester.discovery import shodan_internetdb - monkeypatch.setattr( - shodan_internetdb.socket, - 'getaddrinfo', - lambda *_args: [ - (socket.AF_INET, socket.SOCK_STREAM, 0, '', ('203.0.113.1', 0)), - (socket.AF_INET, socket.SOCK_STREAM, 0, '', (12345, 0)), - ], - raising=True, - ) + targets = patch_resolution(monkeypatch, shodan_internetdb, ('203.0.113.1',)) async def fake_fetch_all(urls, json=False, proxy=False): assert urls == ['https://internetdb.shodan.io/203.0.113.1'] - return [{'ip': '203.0.113.1', 'hostnames': ['www.example.com']}] - - monkeypatch.setattr(shodan_internetdb.AsyncFetcher, 'fetch_all', fake_fetch_all, raising=True) - - search = shodan_internetdb.SearchShodanInternetDB('example.com') - await search.process() - - assert await search.get_hostnames() == {'www.example.com'} - assert await search.get_ips() == {'203.0.113.1'} - - @pytest.mark.asyncio - async def test_shodan_internetdb_normalizes_scoped_provider_evidence(self, monkeypatch): - from theHarvester.discovery import shodan_internetdb - - monkeypatch.setattr( - shodan_internetdb.socket, - 'getaddrinfo', - lambda *_args: [(socket.AF_INET, socket.SOCK_STREAM, 0, '', ('203.0.113.1', 0))], - raising=True, - ) - - async def fake_fetch_all(_urls, json=False, proxy=False): return [ { 'ip': '203.0.113.1', 'hostnames': ['API.Example.TEST.', 'www.notexample.test', None], 'ports': [443], - }, - {'ip': '198.51.100.2', 'ports': [80]}, + } ] monkeypatch.setattr(shodan_internetdb.AsyncFetcher, 'fetch_all', fake_fetch_all, raising=True) @@ -182,6 +625,7 @@ class TestShodanEngine: search = shodan_internetdb.SearchShodanInternetDB(' Example.TEST. ') await search.process() + assert targets == [('example.test', socket.AF_UNSPEC)] assert await search.get_hostnames() == {'api.example.test'} assert await search.get_ips() == {'203.0.113.1'} @@ -189,12 +633,7 @@ class TestShodanEngine: async def test_shodan_internetdb_rejects_evidence_for_an_unrequested_ip(self, monkeypatch): from theHarvester.discovery import shodan_internetdb - monkeypatch.setattr( - shodan_internetdb.socket, - 'getaddrinfo', - lambda *_args: [(socket.AF_INET, socket.SOCK_STREAM, 0, '', ('203.0.113.1', 0))], - raising=True, - ) + patch_resolution(monkeypatch, shodan_internetdb, ('203.0.113.1',)) async def fake_fetch_all(_urls, json=False, proxy=False): return [ @@ -220,38 +659,17 @@ class TestShodanEngine: assert not await search.get_tags() assert not await search.get_cpes() - @pytest.mark.asyncio - async def test_shodan_internetdb_retains_a_matching_ip_with_empty_details(self, monkeypatch): - from theHarvester.discovery import shodan_internetdb - - monkeypatch.setattr( - shodan_internetdb.socket, - 'getaddrinfo', - lambda *_args: [(socket.AF_INET, socket.SOCK_STREAM, 0, '', ('203.0.113.1', 0))], - raising=True, - ) - - async def fake_fetch_all(_urls, json=False, proxy=False): - return [{'ip': '203.0.113.1', 'hostnames': [], 'ports': [], 'vulns': [], 'tags': [], 'cpes': []}] - - monkeypatch.setattr(shodan_internetdb.AsyncFetcher, 'fetch_all', fake_fetch_all, raising=True) - - search = shodan_internetdb.SearchShodanInternetDB('example.test') - await search.process() - - assert await search.get_ips() == {'203.0.113.1'} - @pytest.mark.asyncio async def test_shodan_internetdb_resolution_failure_skips_provider_request(self, monkeypatch): from theHarvester.discovery import shodan_internetdb - def fail_resolution(*_args): - raise socket.gaierror + async def fail_resolution(_target): + raise shodan_internetdb.aiodns.error.DNSError(shodan_internetdb.aiodns.error.ARES_ENOTFOUND, 'not found') async def fail_fetch(*_args, **_kwargs): raise AssertionError('provider request must not run') - monkeypatch.setattr(shodan_internetdb.socket, 'getaddrinfo', fail_resolution, raising=True) + monkeypatch.setattr(shodan_internetdb, 'resolve_ip_addresses', fail_resolution) monkeypatch.setattr(shodan_internetdb.AsyncFetcher, 'fetch_all', fail_fetch, raising=True) search = shodan_internetdb.SearchShodanInternetDB('example.test') @@ -264,12 +682,7 @@ class TestShodanEngine: async def test_shodan_internetdb_provider_failure_completes_without_evidence(self, monkeypatch, caplog): from theHarvester.discovery import shodan_internetdb - monkeypatch.setattr( - shodan_internetdb.socket, - 'getaddrinfo', - lambda *_args: [(socket.AF_INET, socket.SOCK_STREAM, 0, '', ('203.0.113.1', 0))], - raising=True, - ) + patch_resolution(monkeypatch, shodan_internetdb, ('203.0.113.1',)) async def fail_fetch(*_args, **_kwargs): raise RuntimeError('provider-secret-payload') diff --git a/tests/lib/test_api_v1.py b/tests/lib/test_api_v1.py index ef3e64f2..562e1d0b 100644 --- a/tests/lib/test_api_v1.py +++ b/tests/lib/test_api_v1.py @@ -436,6 +436,7 @@ def test_openapi_explains_scope_and_execution_controls(tmp_path, monkeypatch) -> 'actions', 'scope', 'observations', + 'details', } assert 'VirtualHostResult' not in schema['components']['schemas'] assert export_content['application/x-ndjson']['schema']['description'] == ( @@ -1198,6 +1199,111 @@ def test_api_database_upload_preserves_grouped_virtual_host_observations(tmp_pat assert json.loads(exported.text.splitlines()[1]) == detail.json()['results'][0] +def test_api_import_preserves_structured_shodan_host_details(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + details = { + 'organization': 'Example Transit', + 'services': [ + {'port': 53, 'transport': 'udp'}, + { + 'port': 443, + 'transport': 'tcp', + 'product': 'nginx', + 'tls': { + 'subject_cn': '*.example.test', + 'subject_alt_names': ['api.example.test'], + 'sha256': '0123456789abcdef', + }, + }, + ], + } + payload = _jsonl_result( + finding_type='shodan-host', + value='192.0.2.10', + finding_fields={'actions': ['shodan'], 'details': details}, + summary_fields={ + 'action_executions': [ + { + 'action': 'shodan', + 'status': 'completed', + 'duration_ms': 1, + 'result_count': 1, + 'error_type': None, + 'stop_reason': None, + } + ] + }, + ) + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + + with TestClient(api.app) as client: + imported = client.post( + '/api/v1/runs/import', + params={'filename': 'shodan.jsonl'}, + headers={'X-API-Key': 'test-key'}, + content=payload, + ) + assert imported.status_code == 201, imported.text + detail = client.get(f'/api/v1/runs/{imported.json()["run_id"]}', headers={'X-API-Key': 'test-key'}) + + assert detail.status_code == 200 + assert detail.json()['results'] == [ + { + 'type': 'shodan-host', + 'value': '192.0.2.10', + 'sources': [], + 'actions': ['shodan'], + 'details': details, + } + ] + + +def test_api_schema_exposes_typed_shodan_host_details() -> None: + from theHarvester.lib.api.run_models import NormalizedResult + + schema = NormalizedResult.model_json_schema() + + assert schema['properties']['details']['anyOf'][0] == {'$ref': '#/$defs/ShodanHostDetailsResponse'} + service = schema['$defs']['ShodanServiceResponse'] + assert service['properties']['port']['minimum'] == 1 + assert service['properties']['port']['maximum'] == 65535 + assert service['properties']['transport']['enum'] == ['tcp', 'udp'] + tls = schema['$defs']['ShodanTlsDetailsResponse'] + assert tls['properties']['subject_alt_names']['items'] == {'type': 'string'} + + +def test_api_evidence_rejects_redundant_shodan_host_fields() -> None: + from fastapi import HTTPException + + from theHarvester.lib.api.run_evidence import validate_evidence + + evidence = { + 'target': 'example.test', + 'status': 'complete', + 'results': [ + { + 'type': 'shodan-host', + 'value': '192.0.2.10', + 'sources': [], + 'actions': ['shodan'], + 'details': { + 'ip': '192.0.2.10', + 'services': [{'port': 443, 'transport': 'tcp'}], + }, + } + ], + 'source_executions': [], + 'action_executions': [], + 'artifacts': [], + } + + with pytest.raises(HTTPException, match='unsupported fields'): + validate_evidence(evidence) + + def test_api_database_upload_rejects_vhost_evidence_outside_its_stored_target(tmp_path, monkeypatch) -> None: from theHarvester.lib.api import api from theHarvester.lib.api.run_evidence import parse_jsonl_import diff --git a/tests/lib/test_completed_persistence.py b/tests/lib/test_completed_persistence.py index 4068333b..610e8450 100644 --- a/tests/lib/test_completed_persistence.py +++ b/tests/lib/test_completed_persistence.py @@ -475,6 +475,62 @@ async def test_structured_network_evidence_round_trips_in_the_results_table(tmp_ assert json.loads(stored_details) == network_observation_details(network_observations) +@pytest.mark.asyncio +async def test_shodan_host_evidence_round_trips_without_a_json_string_value(tmp_path) -> None: + from theHarvester.lib.shodan_evidence import ShodanHostObservation + + database = tmp_path / 'stash.sqlite' + store = ResultStore(database) + await store.initialize() + collected_at = datetime(2026, 8, 14, 12, 2, tzinfo=UTC) + shodan_host = ShodanHostObservation.from_record( + '192.0.2.10', + { + 'asn': 'AS64496', + 'organization': 'Example Transit', + 'services': [ + {'port': 53, 'transport': 'udp', 'product': 'dnsmasq'}, + {'port': 443, 'transport': 'tcp', 'product': 'nginx'}, + ], + }, + ) + result = CompletedResult.finish( + target='example.com', + started_at=collected_at, + completed_at=collected_at, + groups={}, + source_executions=(SourceExecution('shodan', 'completed', 1, 1),), + observations=(ResultObservation('shodan', 'shodan-host', '192.0.2.10'),), + active_evidence=ActiveEvidence( + executions=( + ActionExecution.finish( + action='shodan', + status='completed', + duration_ms=1, + groups={'shodan-host': ['192.0.2.10']}, + ), + ) + ), + shodan_hosts=(shodan_host,), + ) + + await store.save_run(result) + + assert await store.load_run(result.run_id) == result + with sqlite3.connect(database) as db: + stored = db.execute( + 'SELECT kind, value, details_json FROM results WHERE run_id = ?', + (str(result.run_id),), + ).fetchone() + origins = db.execute( + 'SELECT COUNT(*) FROM result_origins WHERE run_id = ?', + (str(result.run_id),), + ).fetchone()[0] + assert stored[:2] == ('shodan-host', '192.0.2.10') + assert json.loads(stored[2]) == shodan_host.to_details() + assert origins == 2 + + @pytest.mark.asyncio async def test_asn_organization_attribution_round_trips_in_a_normalized_table(tmp_path) -> None: database = tmp_path / 'stash.sqlite' diff --git a/tests/lib/test_completed_result.py b/tests/lib/test_completed_result.py index 1a9a5443..07df3234 100644 --- a/tests/lib/test_completed_result.py +++ b/tests/lib/test_completed_result.py @@ -1040,6 +1040,125 @@ def test_jsonl_rejects_legacy_vhost_result_kind() -> None: parse_result_jsonl(payload) +def test_shodan_host_jsonl_uses_the_ip_value_and_nonredundant_details() -> None: + from theHarvester.lib.shodan_evidence import ShodanHostObservation + + completed_at = datetime(2026, 8, 14, 12, 2, tzinfo=UTC) + shodan_host = ShodanHostObservation.from_record( + '192.0.2.10', + { + 'asn': 'AS64496', + 'organization': 'Example Transit', + 'isp': 'Example ISP', + 'hostnames': ['api.example.test'], + 'domains': ['example.test'], + 'services': [ + { + 'port': 53, + 'transport': 'udp', + 'product': 'dnsmasq', + 'version': '2.90', + 'observed_at': '2026-08-14T11:58:00Z', + }, + { + 'port': 443, + 'transport': 'tcp', + 'product': 'nginx', + 'version': '1.24.0', + 'observed_at': '2026-08-14T12:01:00Z', + 'http': {'title': 'Example', 'server': 'nginx', 'components': ['nginx']}, + }, + ], + }, + ) + result = CompletedResult.finish( + target='example.com', + started_at=completed_at, + completed_at=completed_at, + groups={}, + active_evidence=ActiveEvidence( + executions=( + ActionExecution.finish( + action='shodan', + status='completed', + duration_ms=1, + groups={'shodan-host': ['192.0.2.10']}, + ), + ) + ), + shodan_hosts=(shodan_host,), + ) + + records = [json.loads(line) for line in result.jsonl().splitlines()] + + assert records[1] == { + 'actions': ['shodan'], + 'details': { + 'asn': 'AS64496', + 'domains': ['example.test'], + 'hostnames': ['api.example.test'], + 'isp': 'Example ISP', + 'organization': 'Example Transit', + 'services': [ + { + 'observed_at': '2026-08-14T11:58:00Z', + 'port': 53, + 'product': 'dnsmasq', + 'transport': 'udp', + 'version': '2.90', + }, + { + 'http': {'components': ['nginx'], 'server': 'nginx', 'title': 'Example'}, + 'observed_at': '2026-08-14T12:01:00Z', + 'port': 443, + 'product': 'nginx', + 'transport': 'tcp', + 'version': '1.24.0', + }, + ], + }, + 'sources': [], + 'type': 'shodan-host', + 'value': '192.0.2.10', + } + summary, findings = parse_result_jsonl(result.jsonl()) + assert summary['result_count'] == 1 + assert findings == records[1:] + + +def test_completed_result_rejects_conflicting_shodan_hosts_for_one_ip() -> None: + from theHarvester.lib.shodan_evidence import ShodanHostObservation + + completed_at = datetime(2026, 8, 14, 12, 2, tzinfo=UTC) + first = ShodanHostObservation.from_record( + '192.0.2.10', + {'services': [{'port': 53, 'transport': 'udp'}]}, + ) + second = ShodanHostObservation.from_record( + '192.0.2.10', + {'services': [{'port': 443, 'transport': 'tcp'}]}, + ) + + with pytest.raises(ValueError, match='conflicting evidence'): + CompletedResult.finish( + target='example.com', + started_at=completed_at, + completed_at=completed_at, + groups={}, + active_evidence=ActiveEvidence( + executions=( + ActionExecution.finish( + action='shodan', + status='completed', + duration_ms=1, + groups={'shodan-host': ['192.0.2.10']}, + ), + ) + ), + shodan_hosts=(first, second), + ) + + def test_completed_result_rejects_artifact_without_a_real_subject_result() -> None: completed_at = datetime(2026, 8, 5, 12, 1, tzinfo=UTC) artifact = ArtifactReference( diff --git a/tests/lib/test_harvestview_ui.py b/tests/lib/test_harvestview_ui.py index 5a8fee13..efdea7d7 100644 --- a/tests/lib/test_harvestview_ui.py +++ b/tests/lib/test_harvestview_ui.py @@ -49,6 +49,32 @@ def test_harvestview_assets_load_outside_the_repository_directory(tmp_path, monk assert 'function renderResults' in response.text +def test_harvestview_has_an_operator_readable_shodan_host_route(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + + with TestClient(api.app, base_url='http://127.0.0.1', client=('127.0.0.1', 50000)) as client: + script = client.get('/static/harvestview/app.js') + + assert script.status_code == 200 + assert "'shodan-host'" in script.text + assert "'Shodan hosts'" in script.text + assert 'function shodanNetworkFormatter' in script.text + assert 'function shodanServicesFormatter' in script.text + assert 'details.hostnames' in script.text + assert 'details.domains' in script.text + assert 'service.observed_at' in script.text + assert 'service.cpes' in script.text + assert 'http.components' in script.text + assert 'service.tls' in script.text + assert 'tls.subject_alt_names' in script.text + assert "title: 'Network'" in script.text + assert "title: 'Services'" in script.text + + def test_harvestview_offers_jsonl_and_sqlite_imports_with_jsonl_export(tmp_path, monkeypatch) -> None: from theHarvester.lib.api import api diff --git a/tests/test_all_source_orchestration.py b/tests/test_all_source_orchestration.py index a53cbfa6..5036d373 100644 --- a/tests/test_all_source_orchestration.py +++ b/tests/test_all_source_orchestration.py @@ -1,5 +1,4 @@ import json -import socket import sys import xml.etree.ElementTree as ElementTree from collections import Counter @@ -166,15 +165,7 @@ async def test_explicit_non_passive_source_is_scheduled_once( return set() if source == 'shodan': - - class FakeShodan: - async def search_ip(self, _ip: str) -> dict: - nonlocal executions - executions += 1 - return {} - - monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', FakeShodan) - monkeypatch.setattr(socket, 'gethostbyname', lambda _domain: '203.0.113.1') + monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', lambda *_args: FakeAdapter()) else: module, constructor_name = { 'criminalip': (theharvester_main.criminalip, 'SearchCriminalIP'), diff --git a/tests/test_main.py b/tests/test_main.py index 8647acdc..8863a6fd 100644 --- a/tests/test_main.py +++ b/tests/test_main.py @@ -37,7 +37,8 @@ async def test_cli_help_explains_proxy_and_direct_action_scope( help_text = ' '.join(capsys.readouterr().out.split()) assert exit_info.value.code == 0 - assert 'Use proxies.yaml for supported discovery-source and takeover requests.' in help_text + assert 'Use proxies.yaml for supported discovery-source, Shodan, and takeover requests.' in help_text + assert 'Query the Shodan Host API for discovered IPs, using configured proxies when enabled.' in help_text assert ( 'Enrich discovered IPs with sourced ASN attribution, or an explicitly targeted ASN, IP, or prefix, through ' 'RouteViews.' in help_text @@ -2091,6 +2092,8 @@ async def test_screenshot_capture_failure_cancels_sibling_tasks( @pytest.mark.asyncio async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + from theHarvester.lib.shodan_evidence import ShodanHostObservation + class FakeResultStore: async def initialize(self) -> None: return None @@ -2167,8 +2170,10 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte def __init__(self) -> None: self.attributions: set[AsnAttributionObservation] = set() + self.hosts: dict[str, ShodanHostObservation] = {} - async def search_ip(self, ip: str) -> dict[str, dict[str, list[int]]]: + async def search_ip(self, ip: str, *, proxy: bool = False) -> dict[str, dict[str, object]]: + assert proxy is True self.attributions.add( AsnAttributionObservation( 'action', @@ -2180,11 +2185,22 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte datetime.now(UTC), ) ) - return {ip: {'ports': [443]}} + self.hosts[ip] = ShodanHostObservation.from_record( + ip, + { + 'asn': 'AS64496', + 'organization': 'Example Transit', + 'services': [{'port': 443, 'transport': 'tcp', 'product': 'nginx'}], + }, + ) + return {ip: self.hosts[ip].to_details()} async def get_asn_attributions(self) -> set[AsnAttributionObservation]: return self.attributions + async def get_shodan_hosts(self) -> tuple[ShodanHostObservation, ...]: + return tuple(self.hosts.values()) + class FakeApiScanner: def __init__(self, word: str, wordlist: str, exact_paths: bool = False) -> None: assert word == 'example.com' @@ -2266,7 +2282,12 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte assert isinstance(completed, CompletedResult) assert ('url', 'https://example.com/api/v1') in completed.results assert ('screenshot', 'api.example.com') not in completed.results - assert ('shodan', '{"ip":"192.0.2.10","result":{"ports":[443]}}') in completed.results + assert ('shodan-host', '192.0.2.10') in completed.results + assert completed.shodan_hosts[0].to_details() == { + 'asn': 'AS64496', + 'organization': 'Example Transit', + 'services': [{'port': 443, 'transport': 'tcp', 'product': 'nginx'}], + } assert ('asn', 'AS64496') in completed.results assert completed.asn_attributions[0].organization_label == 'Example Transit' takeover_result = ( @@ -2302,6 +2323,84 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte } +@pytest.mark.asyncio +async def test_shodan_source_evidence_is_not_overwritten_by_conflicting_action_evidence( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from theHarvester.lib.shodan_evidence import ShodanHostObservation + + class ConflictingShodan: + error_type = None + + def __init__(self, word: str | None = None) -> None: + self.word = word + self.host = ( + ShodanHostObservation.from_record( + '192.0.2.10', + {'services': [{'port': 53, 'transport': 'udp'}]}, + ) + if word is not None + else None + ) + + async def process(self, proxy: bool = False) -> None: + return None + + async def get_hostnames(self) -> set[str]: + return {'api.example.com'} if self.word is not None else set() + + async def search_ip(self, ip: str, *, proxy: bool = False) -> dict[str, dict[str, object]]: + self.host = ShodanHostObservation.from_record( + ip, + {'services': [{'port': 443, 'transport': 'tcp'}]}, + ) + return {ip: self.host.to_details()} + + async def get_shodan_hosts(self) -> tuple[ShodanHostObservation, ...]: + return (self.host,) if self.host is not None else () + + async def get_asn_attributions(self) -> set: + if self.word is not None: + return set() + return { + AsnAttributionObservation( + 'action', + 'shodan', + 'AS64496', + 'Example Transit', + 'ip', + '192.0.2.10', + datetime.now(UTC), + ) + } + + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', _ApiHostChecker) + monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', ConflictingShodan) + + result = await theharvester_main.start( + EnumerationOptions( + dns_resolve='192.0.2.53', + domain='example.com', + quiet=True, + shodan=True, + source='shodan', + ), + return_completed_result=True, + ) + + completed = result[-1] + assert completed.shodan_hosts[0].to_details() == {'services': [{'port': 53, 'transport': 'udp'}]} + assert ResultObservation('shodan', 'shodan-host', '192.0.2.10') in completed.observations + execution = next(item for item in completed.active_evidence.executions if item.action == 'shodan') + assert execution.status == 'partial' + assert execution.error_type == 'ValueError' + assert {(observation.kind, observation.value) for observation in execution.observations} == { + ('asn', 'AS64496'), + ('ip', '192.0.2.10'), + } + + @pytest.mark.asyncio @pytest.mark.parametrize( ('request_count', 'request_errors', 'scan_error', 'expected_status', 'expected_error', 'expected_reason'), @@ -2362,17 +2461,14 @@ async def test_shodan_action_records_all_target_errors_as_failed( class FailedShodan: error_type = None - async def search_ip(self, ip: str) -> dict[str, str]: + async def search_ip(self, ip: str, *, proxy: bool = False) -> dict[str, str]: + assert proxy is False raise RuntimeError(f'provider-secret-payload for {ip}') - async def no_sleep(_seconds: float) -> None: - return None - monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', _ApiHostSource) monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', _ApiHostChecker) monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', FailedShodan) - monkeypatch.setattr(theharvester_main.asyncio, 'sleep', no_sleep) result = await theharvester_main.start( EnumerationOptions( @@ -2399,17 +2495,14 @@ async def test_shodan_no_data_is_a_completed_zero_yield_action(monkeypatch: pyte class EmptyShodan: error_type = None - async def search_ip(self, _ip: str) -> dict: + async def search_ip(self, _ip: str, *, proxy: bool = False) -> dict: + assert proxy is False return {} - async def no_sleep(_seconds: float) -> None: - return None - monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', _ApiHostSource) monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', _ApiHostChecker) monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', EmptyShodan) - monkeypatch.setattr(theharvester_main.asyncio, 'sleep', no_sleep) result = await theharvester_main.start( EnumerationOptions(dns_resolve='192.0.2.53', domain='example.com', quiet=True, shodan=True, source='crtsh'), @@ -2983,6 +3076,8 @@ async def test_takeover_failure_persists_and_propagates( @pytest.mark.asyncio async def test_shodan_cancellation_persists_failure_and_propagates(monkeypatch: pytest.MonkeyPatch) -> None: + from theHarvester.lib.shodan_evidence import ShodanHostObservation + saved: list[CompletedResult] = [] class FakeChecker: @@ -2999,17 +3094,28 @@ async def test_shodan_cancellation_persists_failure_and_propagates(monkeypatch: class CancelledShodan: error_type = None - async def search_ip(self, ip: str) -> dict: - return {ip: {'ports': [443]}} + def __init__(self) -> None: + self.calls = 0 + self.hosts: dict[str, ShodanHostObservation] = {} - async def cancel_during_throttle(_seconds: float) -> None: - raise asyncio.CancelledError + async def search_ip(self, ip: str, *, proxy: bool = False) -> dict: + assert proxy is False + self.calls += 1 + if self.calls == 2: + raise asyncio.CancelledError + self.hosts[ip] = ShodanHostObservation.from_record( + ip, + {'services': [{'port': 443, 'transport': 'tcp'}]}, + ) + return {ip: self.hosts[ip].to_details()} + + async def get_shodan_hosts(self) -> tuple[ShodanHostObservation, ...]: + return tuple(self.hosts.values()) monkeypatch.setattr(theharvester_main, 'ResultStore', _recording_result_store(saved)) monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', _ApiHostSource) monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', FakeChecker) monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', CancelledShodan) - monkeypatch.setattr(theharvester_main.asyncio, 'sleep', cancel_during_throttle) with pytest.raises(asyncio.CancelledError): await theharvester_main.start( @@ -3035,6 +3141,8 @@ async def test_shodan_cancellation_persists_failure_and_propagates(monkeypatch: async def test_direct_action_checkpoint_cancellation_persists_and_propagates( monkeypatch: pytest.MonkeyPatch, action: str ) -> None: + from theHarvester.lib.shodan_evidence import ShodanHostObservation + checkpoints: list[CompletedResult] = [] saved: list[CompletedResult] = [] @@ -3057,8 +3165,19 @@ async def test_direct_action_checkpoint_cancellation_persists_and_propagates( class FakeShodan: error_type = None - async def search_ip(self, ip: str) -> dict: - return {ip: {'ports': [443]}} + def __init__(self) -> None: + self.hosts: dict[str, ShodanHostObservation] = {} + + async def search_ip(self, ip: str, *, proxy: bool = False) -> dict: + assert proxy is False + self.hosts[ip] = ShodanHostObservation.from_record( + ip, + {'services': [{'port': 443, 'transport': 'tcp'}]}, + ) + return {ip: self.hosts[ip].to_details()} + + async def get_shodan_hosts(self) -> tuple[ShodanHostObservation, ...]: + return tuple(self.hosts.values()) async def cancel_after_action(result: CompletedResult) -> None: if any(execution.action == action for execution in result.active_evidence.executions): diff --git a/tests/test_readme.py b/tests/test_readme.py index 88b5d0d4..3d462bd7 100644 --- a/tests/test_readme.py +++ b/tests/test_readme.py @@ -155,13 +155,17 @@ def test_readme_preserves_project_social_attribution() -> None: assert f'@{handle}' in readme -def test_readme_explains_jsonl_record_and_structured_value_parsing() -> None: +def test_readme_explains_jsonl_record_and_structured_evidence_parsing() -> None: readme = Path('README.md').read_text() assert '{"sources":[],"type":"hostname","value":"api.example.com"}' in readme assert 'select(.type == "dns-recursive-finding") | .value | fromjson' in readme assert 'JSONL is easy to stream one record at a time.' in readme - assert '`person`, `infostealer`, `shodan`, and `takeover`' in readme + assert '`person`, `infostealer`, and `takeover`' in readme + assert 'select(.type == "shodan-host") | {ip: .value, services: .details.services}' in readme + assert 'paginates both hostname and TLS-certificate searches' in readme + assert 'scoped certificate CNs and SANs' in readme + assert 'Raw banners, response bodies, certificate chains, and Shodan crawler metadata are not retained.' in readme assert 'select(.type == "hostname" and .observations) | {hostname: .value, observations}' in readme assert 'Several endpoint observations can enrich the same hostname' in readme assert 'The summary preserves the evidence status, source and action outcomes' in readme diff --git a/theHarvester/__main__.py b/theHarvester/__main__.py index 74b17554..467bf225 100644 --- a/theHarvester/__main__.py +++ b/theHarvester/__main__.py @@ -118,6 +118,7 @@ from theHarvester.lib.recursive_dns import ( from theHarvester.lib.resolver_selection import DEFAULT_DNS_RESOLVERS, normalize_resolver_addresses from theHarvester.lib.result_values import normalize_asn from theHarvester.lib.routeviews import RouteViewsCancelled, RouteViewsResult, enrich_routeviews +from theHarvester.lib.shodan_evidence import ShodanHostObservation, canonical_shodan_hosts from theHarvester.lib.source_catalog import ( SOURCE_SPECS, ActivityClass, @@ -230,7 +231,7 @@ async def start( parser.add_argument( '-p', '--proxies', - help='Use proxies.yaml for supported discovery-source and takeover requests.', + help='Use proxies.yaml for supported discovery-source, Shodan, and takeover requests.', default=False, action='store_true', ) @@ -243,7 +244,7 @@ async def start( parser.add_argument( '-s', '--shodan', - help='Use Shodan to query discovered hosts.', + help='Query the Shodan Host API for discovered IPs, using configured proxies when enabled.', default=False, action='store_true', ) @@ -578,7 +579,8 @@ async def start( screenshot_artifacts: list[ArtifactReference] = [] screenshot_hostnames: set[str] = set() screenshot_ip_addresses: set[str] = set() - shodan_evidence: list[str] = [] + shodan_hosts: dict[str, ShodanHostObservation] = {} + shodan_action_hosts: set[str] = set() takeover_results: dict[str, list[dict[str, str]]] = {} linkedin_people_list_tracker = [] twitter_people_list_tracker = [] @@ -619,6 +621,26 @@ async def start( ) displayed_asn_attributions.update(pending) + def record_shodan_host_observations( + host_observations: Iterable[ShodanHostObservation], + ) -> tuple[ShodanHostObservation, ...]: + canonical_hosts = canonical_shodan_hosts(list(host_observations)) + for host in canonical_hosts: + existing = shodan_hosts.get(host.ip) + if existing is not None and existing != host: + raise ValueError(f'Shodan host {host.ip} has conflicting evidence') + for host in canonical_hosts: + if host.ip not in shodan_hosts: + output_logger.info( + ujson.dumps( + {'type': 'shodan-host', 'value': host.ip, 'details': host.to_details()}, + indent=4, + sort_keys=True, + ) + ) + shodan_hosts[host.ip] = host + return canonical_hosts + def finish_completed_result( *, extra_hostnames: Iterable[str] = (), @@ -672,6 +694,7 @@ async def start( network_observations=network_observations, asn_attributions=asn_attributions, virtual_hosts=vhost_observations if collect_hosts else (), + shodan_hosts=tuple(shodan_hosts.values()), ) except (ValueError, TypeError) as error: output_logger.info(f'[!] An error occurred while completing the result: {error}') @@ -903,6 +926,9 @@ async def start( 'infostealer', (json.dumps(stealer, ensure_ascii=False, separators=(',', ':'), sort_keys=True) for stealer in infostealers), ) + if source == 'shodan': + sourced_shodan_hosts = record_shodan_host_observations(await search_engine.get_shodan_hosts()) + record_source_observations(source, 'shodan-host', (host.ip for host in sourced_shodan_hosts)) async def store(search_engine: Any, source: str) -> None: source_spec = get_source_spec(source) @@ -1632,41 +1658,7 @@ async def start( elif engineitem == 'shodan': try: - shodan_search = shodansearch.SearchShodan() - - # For normal module usage, we need to create a wrapper that works with the store function - class ShodanWrapper: - def __init__(self, domain, shodan_client): - self.word = domain - self.hosts = set() - self.shodan = shodan_client - - async def process(self, use_proxy: bool = False): - import socket - - try: - # Resolve domain to IP and search in Shodan - ip = socket.gethostbyname(self.word) - output_logger.info(f'\tSearching Shodan for {ip}') - result = await self.shodan.search_ip(ip) - if ip in result and isinstance(result[ip], dict): - # Add the IP as a host for consistency with other modules - self.hosts.add(ip) - - for host in result[ip].get('hostnames', []): - self.hosts.add(host) - - output_logger.info(f'Found Shodan data for {ip}') - elif ip in result and isinstance(result[ip], str): - output_logger.info(f'{ip}: {result[ip]}') - except Exception as e: - output_logger.info(f'Error in Shodan search: {e}') - - async def get_hostnames(self): - return list(self.hosts) - - shodan_wrapper = ShodanWrapper(word, shodan_search) - stor_lst.append(store(shodan_wrapper, engineitem)) + stor_lst.append(store(shodansearch.SearchShodan(word), engineitem)) except Exception as e: if isinstance(e, MissingKey): record_missing_credentials(engineitem) @@ -2808,26 +2800,41 @@ async def start( output_logger.info('[+] Note there may be leftover chrome processes you may have to kill manually\n') # Shodan - shodanres = [] + shodanres: list[dict[str, object]] = [] if shodan is True: shodan_started = time.perf_counter() shodan_error_types: set[str] = set() shodan_asns: set[str] = set() shodan_ips: set[str] = set() + + def has_shodan_action_evidence() -> bool: + return bool(shodan_action_hosts or shodan_asns or shodan_ips) + output_logger.info('[*] Searching Shodan. ') try: - for ip_index, ip in enumerate(host_ip): + shodan_search = None + if host_ip: + try: + shodan_search = shodansearch.SearchShodan() + except Exception as init_error: + shodan_error_types.add(type(init_error).__name__) + output_logger.info(f'[SHODAN-error] Error starting Shodan: {type(init_error).__name__}') + for ip in host_ip: + if shodan_search is None: + break try: output_logger.info('\tSearching for ' + ip) - shodan_search = shodansearch.SearchShodan() - shodandict = await shodan_search.search_ip(ip) + shodandict = await shodan_search.search_ip(ip, proxy=use_proxy) get_asn_attributions = getattr(shodan_search, 'get_asn_attributions', None) if get_asn_attributions is not None: collected_attributions = await get_asn_attributions() - asn_attributions.extend(collected_attributions) - shodan_asns.update(attribution.asn for attribution in collected_attributions) - shodan_ips.update(attribution.subject_value for attribution in collected_attributions) - total_asns.extend(attribution.asn for attribution in collected_attributions) + current_attributions = { + attribution for attribution in collected_attributions if attribution.subject_value == ip + } + asn_attributions.extend(current_attributions) + shodan_asns.update(attribution.asn for attribution in current_attributions) + shodan_ips.update(attribution.subject_value for attribution in current_attributions) + total_asns.extend(attribution.asn for attribution in current_attributions) if shodan_search.error_type: shodan_error_types.add(shodan_search.error_type) @@ -2838,21 +2845,12 @@ async def start( # Process the results if it's a dictionary if isinstance(shodan_result, dict): - rowdata = [] - for _key, value in shodan_result.items(): - if isinstance(value, int): - value = str(value) - if isinstance(value, list): - value = ', '.join(map(str, value)) - rowdata.append(value) - shodanres.append(rowdata) - shodan_evidence.append( - json.dumps({'ip': ip, 'result': shodan_result}, separators=(',', ':'), sort_keys=True) + current_hosts = record_shodan_host_observations( + host for host in await shodan_search.get_shodan_hosts() if host.ip == ip ) - output_logger.info(ujson.dumps(shodan_result, indent=4, sort_keys=True)) + shodan_action_hosts.update(host.ip for host in current_hosts) + shodanres.extend({'value': host.ip, 'details': host.to_details()} for host in current_hosts) output_logger.info('\n') - if ip_index + 1 < len(host_ip): - await asyncio.sleep(5) except Exception as ip_error: shodan_error_types.add(type(ip_error).__name__) output_logger.info(f'[SHODAN-error] Error searching {ip}: {type(ip_error).__name__}') @@ -2861,9 +2859,9 @@ async def start( action_executions.append( ActionExecution.finish( action='shodan', - status='partial' if shodan_evidence else 'failed', + status='partial' if has_shodan_action_evidence() else 'failed', duration_ms=(time.perf_counter() - shodan_started) * 1000, - groups={'shodan': shodan_evidence, 'asn': shodan_asns, 'ip': shodan_ips}, + groups={'shodan-host': shodan_action_hosts, 'asn': shodan_asns, 'ip': shodan_ips}, error_type='CancelledError', stop_reason='cancelled', ) @@ -2876,14 +2874,14 @@ async def start( shodan_status = 'skipped' shodan_stop_reason = 'no-input' elif shodan_error_types: - shodan_status = 'partial' if shodan_evidence else 'failed' + shodan_status = 'partial' if has_shodan_action_evidence() else 'failed' shodan_stop_reason = 'target-errors' action_executions.append( ActionExecution.finish( action='shodan', status=shodan_status, duration_ms=(time.perf_counter() - shodan_started) * 1000, - groups={'shodan': shodan_evidence, 'asn': shodan_asns, 'ip': shodan_ips}, + groups={'shodan-host': shodan_action_hosts, 'asn': shodan_asns, 'ip': shodan_ips}, error_type=next(iter(sorted(shodan_error_types)), None), stop_reason=shodan_stop_reason, ) diff --git a/theHarvester/discovery/shodan_internetdb.py b/theHarvester/discovery/shodan_internetdb.py index fe518e9b..780d8609 100644 --- a/theHarvester/discovery/shodan_internetdb.py +++ b/theHarvester/discovery/shodan_internetdb.py @@ -1,9 +1,10 @@ -import asyncio import logging -import socket from ipaddress import ip_address +import aiodns + from theHarvester.lib.core import AsyncFetcher +from theHarvester.lib.hostchecker import resolve_ip_addresses from theHarvester.lib.hostnames import normalize_scoped_hostname logger = logging.getLogger(__name__) @@ -34,27 +35,17 @@ class SearchShodanInternetDB: async def do_search(self) -> None: # Resolve the domain to IP addresses first try: - addr_infos = await asyncio.to_thread(socket.getaddrinfo, self.word, None, socket.AF_UNSPEC, socket.SOCK_STREAM) - except socket.gaierror: + resolved_ips = await resolve_ip_addresses(self.word) + except aiodns.error.DNSError: logger.info(f'Shodan InternetDB: Could not resolve domain {self.word}') return - # Deduplicate IPs from the resolution results - resolved_ips: set[str] = set() - for _family, _type, _proto, _canonname, sockaddr in addr_infos: - ip = sockaddr[0] - if isinstance(ip, str): - try: - resolved_ips.add(str(ip_address(ip))) - except ValueError: - continue - if not resolved_ips: logger.info(f'Shodan InternetDB: No IPs resolved for {self.word}') return # Query InternetDB for each resolved IP - requested_ips = sorted(resolved_ips) + requested_ips = list(resolved_ips) urls = [f'https://internetdb.shodan.io/{ip}' for ip in requested_ips] try: responses = await AsyncFetcher.fetch_all(urls, json=True, proxy=self.proxy) diff --git a/theHarvester/discovery/shodansearch.py b/theHarvester/discovery/shodansearch.py index a441de8d..1a35114e 100644 --- a/theHarvester/discovery/shodansearch.py +++ b/theHarvester/discovery/shodansearch.py @@ -1,16 +1,29 @@ +import asyncio import logging +import re +import socket from collections import OrderedDict from datetime import UTC, datetime from ipaddress import ip_address +from typing import cast -from shodan import Shodan, exception +import aiodns from theHarvester.discovery.constants import MissingKey from theHarvester.lib.asn_attribution import AsnAttributionObservation -from theHarvester.lib.core import Core +from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse, ResponseStreamError +from theHarvester.lib.hostchecker import resolve_ip_addresses +from theHarvester.lib.hostnames import normalize_scoped_hostname +from theHarvester.lib.shodan_evidence import ShodanHostObservation, canonical_shodan_hosts logger = logging.getLogger(__name__) +HostResult = dict[str, object] +_CERTIFICATE_HOSTNAME = re.compile( + r'(?i)(? None: + API_BASE_URL = 'https://api.shodan.io/shodan/host' + SEARCH_URL = f'{API_BASE_URL}/search' + SEARCH_FIELDS = ','.join( + ( + 'ip_str', + 'asn', + 'domains', + 'hostnames', + 'isp', + 'org', + 'port', + 'transport', + 'product', + 'version', + 'timestamp', + 'cpe', + 'http.title', + 'http.server', + 'http.components', + 'ssl.jarm', + 'ssl.cert.subject', + 'ssl.cert.issuer', + 'ssl.cert.expires', + 'ssl.cert.fingerprint', + 'ssl.cert.extensions', + ) + ) + # Preserve the official SDK's one-request-per-second pacing without retaining its blocking transport. + REQUEST_INTERVAL_SECONDS = 1.0 + REQUEST_TIMEOUT_SECONDS: int | None = None + + def __init__(self, word: str | None = None) -> None: + self.word = word.strip().lower().rstrip('.') if word is not None else None + if self.word is not None and (self.word.startswith('*.') or _CERTIFICATE_HOSTNAME.fullmatch(self.word) is None): + raise ValueError('Shodan discovery target must be a hostname') + self.scope = self.word.removeprefix('www.') if self.word is not None else None self.key = Core.shodan_key() if self.key is None: raise MissingKey('Shodan') - self.api = Shodan(self.key) - self.hostdatarow: list = [] - self.tracker: OrderedDict = OrderedDict() + self.tracker: OrderedDict[str, HostResult | str] = OrderedDict() + self.shodan_hosts: dict[str, ShodanHostObservation] = {} self.error_type: str | None = None self.asn_attributions: set[AsnAttributionObservation] = set() + self.totalhosts: set[str] = set() + self.execution_status: str | None = None + self.stop_reason: str | None = None + self._next_request_at = 0.0 - async def search_ip(self, ip) -> OrderedDict: + async def _fetch_json(self, url: str, params: dict[str, object], proxy: bool) -> FetcherResponse: + loop = asyncio.get_running_loop() + delay = self._next_request_at - loop.time() + if delay > 0: + await asyncio.sleep(delay) + self._next_request_at = loop.time() + self.REQUEST_INTERVAL_SECONDS + return await AsyncFetcher.fetch_json( + url, + params=params, + proxy=proxy, + request_timeout=self.REQUEST_TIMEOUT_SECONDS, + ) + + async def _fetch_host(self, ip: str, proxy: bool) -> FetcherResponse: + return await self._fetch_json(f'{self.API_BASE_URL}/{ip}', {'key': self.key}, proxy) + + def _certificate_name(self, value: object) -> str | None: + if not isinstance(value, str) or not (candidate := value.strip().casefold().rstrip('.')): + return None + if _CERTIFICATE_HOSTNAME.fullmatch(candidate) is None: + return None + wildcard = candidate.startswith('*.') + hostname = candidate.removeprefix('*.') + if self.scope is not None: + hostname = normalize_scoped_hostname(hostname, self.scope) or '' + if not hostname: + return None + else: + try: + ip_address(hostname) + except ValueError: + pass + else: + return None + return f'*.{hostname}' if wildcard else hostname + + def _certificate_names(self, cert: object) -> tuple[set[str], bool]: + if not isinstance(cert, dict): + return set(), cert is not None + invalid_response = False + names: set[str] = set() + subject = cert.get('subject') + if subject is not None and not isinstance(subject, dict): + invalid_response = True + elif isinstance(subject, dict): + if normalized := self._certificate_name(subject.get('CN')): + names.add(normalized) + + extensions = cert.get('extensions') + if extensions is not None and not isinstance(extensions, list): + invalid_response = True + elif isinstance(extensions, list): + for extension in extensions: + if not isinstance(extension, dict): + invalid_response = True + continue + if extension.get('name') != 'subjectAltName': + continue + data = extension.get('data') + if not isinstance(data, str): + invalid_response = True + continue + decoded = re.sub(r'\\x[0-9a-f]{2}', ' ', data, flags=re.IGNORECASE) + for match in _CERTIFICATE_HOSTNAME.finditer(decoded): + if normalized := self._certificate_name(match.group()): + names.add(normalized) + return names, invalid_response + + def _scoped_banner_names(self, banner: dict[str, object]) -> set[str]: + assert self.scope is not None + names: set[str] = set() + for field in ('hostnames', 'domains'): + values = banner.get(field) + if isinstance(values, list): + for value in values: + if normalized := normalize_scoped_hostname(value, self.scope): + names.add(normalized) + ssl = banner.get('ssl') + if isinstance(ssl, dict): + cert_names, _invalid = self._certificate_names(ssl.get('cert')) + names.update(cert_names) + return names + + def _merge_host(self, observation: ShodanHostObservation) -> bool: + existing = self.shodan_hosts.get(observation.ip) + if existing is None: + self.shodan_hosts[observation.ip] = observation + return False + + invalid_response = False + scalars: dict[str, str | None] = {} + for field in ('asn', 'organization', 'isp'): + previous = getattr(existing, field) + current = getattr(observation, field) + if previous is not None and current is not None and previous != current: + invalid_response = True + scalars[field] = previous or current + self.shodan_hosts[observation.ip] = ShodanHostObservation.from_record( + observation.ip, + { + **scalars, + 'hostnames': sorted(set(existing.hostnames) | set(observation.hostnames)), + 'domains': sorted(set(existing.domains) | set(observation.domains)), + 'services': [ + service.to_record() + for service in sorted( + set(existing.services) | set(observation.services), + key=lambda service: service.sort_key(), + ) + ], + }, + ) + return invalid_response + + def _record_host(self, ip: str, results: dict[str, object]) -> bool: + data = results.get('data') + if data == []: + logger.info(f'Shodan: No data found for IP {ip}') + return False + if not isinstance(data, list) or not data: + raise ValueError('invalid Shodan host response') + + invalid_response = False + + def normalized_string(value: object) -> str: + nonlocal invalid_response + if value is None: + return '' + if not isinstance(value, str): + invalid_response = True + return '' + normalized = value.strip() + return '' if normalized == 'None' else normalized + + def normalized_strings(value: object) -> list[str]: + nonlocal invalid_response + if value is None: + return [] + if not isinstance(value, list): + invalid_response = True + return [] + if any(not isinstance(item, str) for item in value): + invalid_response = True + return sorted({normalized_string(item) for item in value if isinstance(item, str)} - {''}) + + services: list[dict[str, object]] = [] + for banner in data: + if not isinstance(banner, dict): + invalid_response = True + continue + port = banner.get('port') + transport = normalized_string(banner.get('transport')).casefold() + if isinstance(port, bool) or not isinstance(port, int) or not 1 <= port <= 65535: + invalid_response = True + continue + if transport not in {'tcp', 'udp'}: + invalid_response = True + continue + + service: dict[str, object] = {'port': port, 'transport': transport} + for provider_field, evidence_field in ( + ('product', 'product'), + ('version', 'version'), + ('timestamp', 'observed_at'), + ): + if value := normalized_string(banner.get(provider_field)): + service[evidence_field] = value + + if cpe_values := normalized_strings(banner.get('cpe')): + service['cpes'] = cpe_values + + http = banner.get('http') + if isinstance(http, dict): + http_details: dict[str, object] = {} + for field in ('title', 'server'): + if value := normalized_string(http.get(field)): + http_details[field] = value + components = http.get('components') + if isinstance(components, dict) and ( + component_values := sorted({normalized_string(component) for component in components} - {''}) + ): + http_details['components'] = component_values + elif components is not None and not isinstance(components, dict): + invalid_response = True + if http_details: + service['http'] = http_details + elif http is not None: + invalid_response = True + + ssl = banner.get('ssl') + if isinstance(ssl, dict): + tls_details: dict[str, object] = {} + if jarm := normalized_string(ssl.get('jarm')): + tls_details['jarm'] = jarm + cert = ssl.get('cert') + if isinstance(cert, dict): + subject = cert.get('subject') + issuer = cert.get('issuer') + fingerprint = cert.get('fingerprint') + if subject is not None and not isinstance(subject, dict): + invalid_response = True + if issuer is not None and not isinstance(issuer, dict): + invalid_response = True + if fingerprint is not None and not isinstance(fingerprint, dict): + invalid_response = True + subject_cn = self._certificate_name(subject.get('CN')) if isinstance(subject, dict) else None + cert_names, invalid_cert = self._certificate_names(cert) + invalid_response = invalid_response or invalid_cert + if subject_cn: + tls_details['subject_cn'] = subject_cn + subject_alt_names = sorted(cert_names - ({subject_cn} if subject_cn else set())) + if subject_alt_names: + tls_details['subject_alt_names'] = subject_alt_names + for field, source in ( + ('issuer_cn', issuer.get('CN') if isinstance(issuer, dict) else None), + ('expires_at', cert.get('expires')), + ('sha256', fingerprint.get('sha256') if isinstance(fingerprint, dict) else None), + ): + if value := normalized_string(source): + tls_details[field] = value + elif cert is not None: + invalid_response = True + if tls_details: + service['tls'] = tls_details + elif ssl is not None: + invalid_response = True + + if service not in services: + services.append(service) + + if not services: + raise ValueError('invalid Shodan host response') + services.sort( + key=lambda service: ( + cast('int', service['port']), + str(service['transport']), + str(service.get('observed_at', '')), + ) + ) + + domain_values = normalized_strings(results.get('domains')) + hostname_values = normalized_strings(results.get('hostnames')) + if self.scope is not None: + domain_values = sorted( + {normalized for value in domain_values if (normalized := normalize_scoped_hostname(value, self.scope))} + ) + hostname_values = sorted( + {normalized for value in hostname_values if (normalized := normalize_scoped_hostname(value, self.scope))} + ) + self.totalhosts.update(value for value in domain_values + hostname_values if value != self.scope) + for service in services: + tls = service.get('tls') + if not isinstance(tls, dict): + continue + for field in ('subject_cn', 'subject_alt_names'): + tls_value = tls.get(field) + values = tls_value if isinstance(tls_value, list) else [tls_value] + self.totalhosts.update( + name for name in values if isinstance(name, str) and not name.startswith('*.') and name != self.scope + ) + asn = normalized_string(results.get('asn')) + organization = normalized_string(results.get('org')) + host_details = { + 'asn': asn, + 'domains': domain_values, + 'hostnames': hostname_values, + 'isp': normalized_string(results.get('isp')), + 'organization': organization, + 'services': services, + } + shodan_host = ShodanHostObservation.from_record(ip, host_details) + invalid_response = self._merge_host(shodan_host) or invalid_response + stored_host = self.shodan_hosts[ip] + self.tracker[ip] = stored_host.to_details() + + if stored_host.asn and stored_host.organization: + self.asn_attributions.add( + AsnAttributionObservation( + 'action', + 'shodan', + stored_host.asn, + stored_host.organization, + 'ip', + ip, + datetime.now(UTC), + ) + ) + return invalid_response + + async def _search_target(self, proxy: bool) -> set[str]: + assert self.scope is not None + error_types: set[str] = set() + for query in (f'hostname:{self.scope}', f'ssl:{self.scope}'): + page = 1 + received = 0 + while True: + try: + response = await self._fetch_json( + self.SEARCH_URL, + { + 'key': self.key, + 'query': query, + 'page': page, + 'minify': 'false', + 'fields': self.SEARCH_FIELDS, + }, + proxy, + ) + except ResponseStreamError as error: + error_types.add( + { + 'invalid-response': 'InvalidResponseError', + 'response-limit': 'ResponseLimitError', + 'transport-error': 'TransportError', + }[error.reason] + ) + break + except Exception as error: + error_types.add(type(error).__name__) + break + if not 200 <= response.status < 300: + error_types.add(f'HTTP{response.status}Error') + break + if not isinstance(response.body, dict): + error_types.add('InvalidResponseError') + break + matches = response.body.get('matches') + total = response.body.get('total') + if not isinstance(matches, list) or isinstance(total, bool) or not isinstance(total, int) or total < 0: + error_types.add('InvalidResponseError') + break + for match in matches: + if not isinstance(match, dict): + error_types.add('InvalidResponseError') + continue + if not self._scoped_banner_names(match): + continue + ip_value = match.get('ip_str') + if not isinstance(ip_value, str): + error_types.add('InvalidResponseError') + continue + try: + ip = str(ip_address(ip_value)) + if self._record_host(ip, {**match, 'data': [match]}): + error_types.add('InvalidResponseError') + except (TypeError, ValueError): + error_types.add('InvalidResponseError') + received += len(matches) + if not matches: + if received < total: + error_types.add('InvalidResponseError') + break + if received >= total: + break + page += 1 + return error_types + + async def search_ip(self, ip: str, *, proxy: bool = False) -> OrderedDict[str, HostResult | str]: self.error_type = None try: - ipaddress = ip - results = self.api.host(ipaddress) - - if not results or 'data' not in results or not results['data']: - logger.info(f'Shodan: No data found for IP {ip}') - return OrderedDict() - asn = '' - domains: list = list() - hostnames: list = list() - ip_str = '' - isp = '' - org = '' - ports: list = list() - title = '' - server = '' - product = '' - technologies: list = list() - - data_first_dict = dict(results['data'][0]) - - if 'ip_str' in data_first_dict: - ip_str += data_first_dict['ip_str'] - - if 'http' in data_first_dict: - http_results_dict = dict(data_first_dict['http']) - if 'title' in http_results_dict: - title_val = str(http_results_dict['title']).strip() - if title_val != 'None': - title += title_val - if 'components' in http_results_dict: - for key in http_results_dict['components'].keys(): - technologies.append(key) - if 'server' in http_results_dict: - server_val = str(http_results_dict['server']).strip() - if server_val != 'None': - server += server_val - - for key, value in results.items(): - if key == 'asn': - if isinstance(value, str): - asn += value - elif value is not None: - asn += str(value) - if key == 'domains': - if isinstance(value, list): - domain_values = [str(domain) for domain in value] - domain_values.sort() - domains.extend(domain_values) - if key == 'hostnames': - if isinstance(value, list): - hostname_values = [str(host).strip() for host in value] - hostname_values.sort() - hostnames.extend(hostname_values) - if key == 'isp': - if isinstance(value, str): - isp += value - elif value is not None: - isp += str(value) - if key == 'org': - org += str(value) - if key == 'ports': - if isinstance(value, list): - port_values = [int(port) for port in value if isinstance(port, int)] - port_values.sort() - ports.extend(port_values) - if key == 'product': - if isinstance(value, str): - product += value - elif isinstance(value, list): - product += ', '.join(str(item) for item in value if item is not None) - - technologies = list(set(technologies)) - - self.tracker[ip] = { - 'asn': asn.strip(), - 'domains': domains, - 'hostnames': hostnames, - 'ip_str': ip_str.strip(), - 'isp': isp.strip(), - 'org': org.strip(), - 'ports': ports, - 'product': product.strip(), - 'server': server.strip(), - 'technologies': technologies, - 'title': title.strip(), - } - organization_label = results.get('org') - if asn.strip() and isinstance(organization_label, str) and organization_label.strip(): - try: - subject_ip = str(ip_address(str(ip).strip())) - self.asn_attributions.add( - AsnAttributionObservation( - 'action', - 'shodan', - asn, - organization_label, - 'ip', - subject_ip, - datetime.now(UTC), - ) - ) - except ValueError: - logger.info('Shodan returned invalid ASN organization attribution') - - return self.tracker - except exception.APIError as error: - if str(error).strip().rstrip('.').casefold() == 'no information available for that ip': - logger.info(f'{ip}: Not in Shodan') - self.tracker[ip] = 'Not in Shodan' - else: - self.error_type = type(error).__name__ - self.tracker[ip] = 'Shodan request failed' - except Exception as e: - self.error_type = type(e).__name__ + normalized_ip = str(ip_address(ip.strip())) + except ValueError: + self.error_type = 'ValueError' self.tracker[ip] = 'Shodan request failed' + return self.tracker + try: + response = await self._fetch_host(normalized_ip, proxy) + if response.status == 404: + logger.info(f'{normalized_ip}: Not in Shodan') + self.tracker[normalized_ip] = 'Not in Shodan' + return self.tracker + if not 200 <= response.status < 300: + self.error_type = f'HTTP{response.status}Error' + self.tracker[normalized_ip] = 'Shodan request failed' + return self.tracker + if not isinstance(response.body, dict): + self.error_type = 'InvalidResponseError' + self.tracker[normalized_ip] = 'Shodan request failed' + return self.tracker + try: + if self._record_host(normalized_ip, response.body): + self.error_type = 'InvalidResponseError' + except (TypeError, ValueError): + self.error_type = 'InvalidResponseError' + self.tracker[normalized_ip] = 'Shodan request failed' + except ResponseStreamError as error: + self.error_type = { + 'invalid-response': 'InvalidResponseError', + 'response-limit': 'ResponseLimitError', + 'transport-error': 'TransportError', + }[error.reason] + self.tracker[normalized_ip] = 'Shodan request failed' + except Exception as error: + self.error_type = type(error).__name__ + self.tracker[normalized_ip] = 'Shodan request failed' return self.tracker async def get_asn_attributions(self) -> set[AsnAttributionObservation]: return self.asn_attributions + + async def get_hostnames(self) -> set[str]: + return self.totalhosts + + async def get_shodan_hosts(self) -> tuple[ShodanHostObservation, ...]: + return canonical_shodan_hosts(list(self.shodan_hosts.values())) + + async def process(self, proxy: bool = False) -> None: + if self.word is None: + raise ValueError('A discovery target is required') + assert self.scope is not None + + self.totalhosts.clear() + self.execution_status = None + self.stop_reason = None + dns_stop_reason: str | None = None + try: + resolved_ips = await resolve_ip_addresses(self.word, family=socket.AF_INET) + if not resolved_ips: + raise ValueError('target has no IPv4 addresses') + except TimeoutError: + resolved_ips = () + dns_stop_reason = 'dns-timeout' + except aiodns.error.DNSError as error: + resolved_ips = () + dns_stop_reason = ( + 'dns-timeout' if error.args and error.args[0] == aiodns.error.ARES_ETIMEOUT else 'dns-resolution-failed' + ) + except ValueError: + resolved_ips = () + dns_stop_reason = 'dns-resolution-failed' + + provider_error_types = await self._search_target(proxy) + for resolved_ip in resolved_ips: + await self.search_ip(resolved_ip, proxy=proxy) + if self.error_type is not None: + provider_error_types.add(self.error_type) + + self.error_type = next(iter(sorted(provider_error_types)), None) + retained_evidence = bool(self.totalhosts or self.shodan_hosts) + if dns_stop_reason is not None: + self.execution_status = 'partial' if retained_evidence else 'failed' + self.stop_reason = dns_stop_reason + return + if provider_error_types: + self.execution_status = 'partial' if retained_evidence else 'failed' + if provider_error_types <= {'HTTP401Error', 'HTTP403Error'}: + self.stop_reason = 'access-denied' + elif provider_error_types == {'HTTP429Error'}: + self.stop_reason = 'rate-limited' + elif len(resolved_ips) == 1: + self.stop_reason = 'provider-error' + else: + self.stop_reason = 'provider-errors' + return + + self.execution_status = 'completed' + self.stop_reason = None if retained_evidence else 'no-results' diff --git a/theHarvester/lib/api/run_evidence.py b/theHarvester/lib/api/run_evidence.py index fdb63e2a..7c1bcf7d 100644 --- a/theHarvester/lib/api/run_evidence.py +++ b/theHarvester/lib/api/run_evidence.py @@ -15,6 +15,7 @@ from theHarvester.lib.network_evidence import ( parse_network_observation_details, ) from theHarvester.lib.result_values import normalize_prefix +from theHarvester.lib.shodan_evidence import ShodanHostObservation from .run_models import _normalize_target @@ -102,6 +103,41 @@ def validate_evidence(evidence: dict[str, Any]) -> dict[str, Any]: status_code=status.HTTP_400_BAD_REQUEST, detail='vhost is not a result type; use hostname with virtual-host observations', ) + if result.get('type') == 'shodan-host': + allowed_keys = {'type', 'value', 'sources', 'actions', 'details'} + if set(result) - allowed_keys: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail='Shodan host evidence contains unsupported fields', + ) + sources = result.get('sources', []) + actions = result.get('actions', []) + if ( + not isinstance(sources, list) + or any(not isinstance(source, str) or not source.strip() for source in sources) + or not isinstance(actions, list) + or any(not isinstance(action, str) or not action.strip() for action in actions) + ): + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail='Shodan host producers must be arrays of non-empty strings', + ) + value = result.get('value') + if not isinstance(value, str): + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail='Shodan host evidence must identify a canonical IP address', + ) + try: + shodan_host = ShodanHostObservation.from_record(value, result.get('details')) + if shodan_host.ip != value or shodan_host.to_details() != result.get('details'): + raise ValueError('Shodan host evidence must use canonical structured details') + except ValueError as error: + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(error)) from error + result['sources'] = sorted(set(sources)) + result['actions'] = sorted(set(actions)) + result['details'] = shodan_host.to_details() + continue if result.get('type') == 'prefix': allowed_keys = {'type', 'value', 'sources', 'actions', 'scope', 'observations'} if set(result) - allowed_keys: diff --git a/theHarvester/lib/api/run_models.py b/theHarvester/lib/api/run_models.py index f208c189..e78c8556 100644 --- a/theHarvester/lib/api/run_models.py +++ b/theHarvester/lib/api/run_models.py @@ -402,6 +402,49 @@ class AsnAttributionObservationResponse(BaseModel): collected_at: str +class ShodanHttpDetailsResponse(BaseModel): + model_config = ConfigDict(extra='forbid') + + title: str | None = None + server: str | None = None + components: list[str] = Field(default_factory=list) + + +class ShodanTlsDetailsResponse(BaseModel): + model_config = ConfigDict(extra='forbid') + + subject_cn: str | None = None + subject_alt_names: list[str] = Field(default_factory=list) + issuer_cn: str | None = None + expires_at: str | None = None + sha256: str | None = None + jarm: str | None = None + + +class ShodanServiceResponse(BaseModel): + model_config = ConfigDict(extra='forbid') + + port: int = Field(ge=1, le=65535) + transport: Literal['tcp', 'udp'] + product: str | None = None + version: str | None = None + observed_at: str | None = None + cpes: list[str] = Field(default_factory=list) + http: ShodanHttpDetailsResponse | None = None + tls: ShodanTlsDetailsResponse | None = None + + +class ShodanHostDetailsResponse(BaseModel): + model_config = ConfigDict(extra='forbid') + + asn: str | None = None + organization: str | None = None + isp: str | None = None + hostnames: list[str] = Field(default_factory=list) + domains: list[str] = Field(default_factory=list) + services: list[ShodanServiceResponse] = Field(min_length=1) + + class NormalizedResult(BaseModel): model_config = ConfigDict(extra='forbid') @@ -410,6 +453,10 @@ class NormalizedResult(BaseModel): sources: list[str] = Field(default_factory=list) actions: list[str] = Field(default_factory=list) scope: Literal['external-relationship'] | None = None + details: ShodanHostDetailsResponse | None = Field( + default=None, + description='Canonical host and service evidence for shodan-host results.', + ) observations: ( list[ VirtualHostObservationResponse @@ -425,7 +472,16 @@ class NormalizedResult(BaseModel): def validate_result(self) -> Self: if self.type == 'vhost': raise ValueError('vhost is not a result type; use hostname with virtual-host observations') - if self.type == 'prefix': + if self.type == 'shodan-host': + from theHarvester.lib.shodan_evidence import ShodanHostObservation + + if self.scope is not None or self.observations is not None or self.details is None: + raise ValueError('Shodan host results require details without scope or observations') + shodan_details = self.details.model_dump(exclude_none=True, exclude_defaults=True) + shodan_host = ShodanHostObservation.from_record(self.value, shodan_details) + if shodan_host.ip != self.value or shodan_host.to_details() != shodan_details: + raise ValueError('Shodan host results must use canonical structured details') + elif self.type == 'prefix': if self.scope != 'external-relationship': raise ValueError('Prefix results must have external-relationship scope') if self.observations is not None: @@ -444,6 +500,8 @@ class NormalizedResult(BaseModel): parse_virtual_host_details(self.value, details) else: raise ValueError('Structured observations belong to ASN, hostname, or prefix results') + elif self.details is not None: + raise ValueError('Structured details belong to Shodan host results') return self diff --git a/theHarvester/lib/api/run_projection.py b/theHarvester/lib/api/run_projection.py index 1c63da6a..5d6c8e70 100644 --- a/theHarvester/lib/api/run_projection.py +++ b/theHarvester/lib/api/run_projection.py @@ -42,6 +42,8 @@ def normalized_results(evidence: dict[str, Any] | None) -> list[dict[str, Any]]: result['observations'] = [ dict(observation) for observation in item.get('observations', []) if isinstance(observation, dict) ] + if item.get('type') == 'shodan-host' and isinstance(item.get('details'), dict): + result['details'] = dict(item['details']) if item.get('type') == 'prefix': result['scope'] = str(item.get('scope', '')) results.append(result) diff --git a/theHarvester/lib/api/run_store.py b/theHarvester/lib/api/run_store.py index 7d4869ab..a2a1c35b 100644 --- a/theHarvester/lib/api/run_store.py +++ b/theHarvester/lib/api/run_store.py @@ -19,6 +19,7 @@ from theHarvester.lib.completed_result import ( from theHarvester.lib.database import DuplicateRunError, ResultStore, ResultStoreError, RunLifecycleStore from theHarvester.lib.evidence_types import EXECUTION_STATUSES, EvidenceStatus, ExecutionStatus, ResultKind from theHarvester.lib.network_evidence import NetworkObservation, parse_network_observation_details +from theHarvester.lib.shodan_evidence import ShodanHostObservation from .run_artifacts import RunPaths, read_child_evidence from .run_models import RunRequest, _normalize_target, utc_now @@ -55,6 +56,7 @@ def _completed_result( virtual_hosts: list[VirtualHostObservation] = [] network_observations: list[NetworkObservation] = [] asn_attributions: list[AsnAttributionObservation] = [] + shodan_hosts: list[ShodanHostObservation] = [] for item in results: kind = cast('ResultKind', str(item['type'])) value = str(item['value']) @@ -65,6 +67,8 @@ def _completed_result( network_observations.extend(parse_network_observation_details(value, item.get('observations'))) elif kind == 'asn' and item.get('observations'): asn_attributions.extend(parse_asn_attribution_details(value, item.get('observations'))) + elif kind == 'shodan-host': + shodan_hosts.append(ShodanHostObservation.from_record(value, item.get('details'))) for source in set(item.get('sources', [])): source_name = str(source) source_origins.add(ResultObservation(source_name, kind, value)) @@ -152,6 +156,7 @@ def _completed_result( virtual_hosts=virtual_hosts, network_observations=network_observations, asn_attributions=asn_attributions, + shodan_hosts=shodan_hosts, evidence_status=( cast('EvidenceStatus', str(evidence['status'])) if evidence.get('status') is not None and not execution_status_is_authoritative diff --git a/theHarvester/lib/api/static/harvestview/app.js b/theHarvester/lib/api/static/harvestview/app.js index c1448a02..70c79f91 100644 --- a/theHarvester/lib/api/static/harvestview/app.js +++ b/theHarvester/lib/api/static/harvestview/app.js @@ -4,12 +4,12 @@ const $ = selector => document.querySelector(selector); const $$ = selector => [...document.querySelectorAll(selector)]; const ROUTE_ORDER = [ - 'hostname', 'ip', 'prefix', 'asn', 'email', 'url', 'person', 'person-link', 'takeover', 'shodan', + 'hostname', 'ip', 'prefix', 'asn', 'shodan-host', 'email', 'url', 'person', 'person-link', 'takeover', 'scope-extension', 'external-relationship', 'other' ]; const ROUTE_LABELS = { hostname: 'Hostnames', ip: 'IP addresses', prefix: 'Network prefixes', asn: 'ASNs', email: 'Emails', url: 'URLs', - person: 'People', 'person-link': 'People links', takeover: 'Takeover evidence', shodan: 'Shodan evidence', + person: 'People', 'person-link': 'People links', takeover: 'Takeover evidence', 'shodan-host': 'Shodan hosts', 'scope-extension': 'Scope extensions', 'external-relationship': 'External relationships', other: 'Other' }; const ACTION_FIELDS = {'dns-recursive': 'dns_recursive_depth'}; @@ -470,6 +470,47 @@ return text.includes(query); } + function shodanNetworkFormatter(cell) { + const details = cell.getValue() || {}; + const network = [details.organization, details.asn, details.isp].filter(Boolean).join(' · ') || 'Not recorded'; + const names = [ + details.hostnames?.length ? `Hosts: ${details.hostnames.join(', ')}` : '', + details.domains?.length ? `Domains: ${details.domains.join(', ')}` : '', + ].filter(Boolean); + return `
${escapeHtml(network)}${names.map(value => `${escapeHtml(value)}`).join('')}
`; + } + + function shodanServicesFormatter(cell) { + const services = Array.isArray(cell.getValue()?.services) ? cell.getValue().services : []; + if (!services.length) return 'No service evidence'; + return `
${services.map(service => { + const identity = `${service.port}/${service.transport}`; + const product = [service.product, service.version].filter(Boolean).join(' '); + const http = service.http || {}; + const tls = service.tls || {}; + const description = [ + product, + service.observed_at ? `Seen ${service.observed_at}` : '', + http.title, + http.server, + http.components?.length ? `HTTP: ${http.components.join(', ')}` : '', + service.cpes?.length ? `CPE: ${service.cpes.join(', ')}` : '', + tls.subject_cn ? `TLS subject: ${tls.subject_cn}` : '', + tls.subject_alt_names?.length ? `TLS SANs: ${tls.subject_alt_names.join(', ')}` : '', + tls.issuer_cn ? `issuer: ${tls.issuer_cn}` : '', + tls.expires_at ? `expires: ${tls.expires_at}` : '', + tls.sha256 ? `SHA-256: ${tls.sha256}` : '', + tls.jarm ? `JARM: ${tls.jarm}` : '', + ].filter(Boolean).join(' · '); + return `${escapeHtml(`${identity}${description ? ` · ${description}` : ''}`)}`; + }).join('')}
`; + } + + function shodanDetailsFilter(headerValue, rowValue) { + const query = String(headerValue || '').trim().toLowerCase(); + return JSON.stringify(rowValue || {}).toLowerCase().includes(query); + } + function provenanceFormatter(cell) { const values = Array.isArray(cell.getValue()) ? cell.getValue() : []; return escapeHtml(values.join(', ') || '-'); @@ -480,9 +521,13 @@ nodes.copySelected.disabled = true; nodes.copySelected.textContent = 'Copy selected'; const columns = [ - {title: 'Value', field: 'value', formatter: cell => `${escapeHtml(cell.getValue())}`, minWidth: 260, widthGrow: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter, headerFilterPlaceholder: 'Filter values'}, - {title: 'DNS', field: 'dns_status', formatter: dnsFormatter, width: 130, responsive: 1, headerFilter: 'input', headerFilterFunc: columnTextFilter, headerFilterPlaceholder: 'Filter DNS'}, + {title: state.route === 'shodan-host' ? 'IP' : 'Value', field: 'value', formatter: cell => `${escapeHtml(cell.getValue())}`, minWidth: 260, widthGrow: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter, headerFilterPlaceholder: 'Filter values'}, ]; + if (state.route !== 'shodan-host') { + columns.push( + {title: 'DNS', field: 'dns_status', formatter: dnsFormatter, width: 130, responsive: 1, headerFilter: 'input', headerFilterFunc: columnTextFilter, headerFilterPlaceholder: 'Filter DNS'}, + ); + } if (state.route === 'hostname' && rows.some(row => Array.isArray(row.observations) && row.observations.length)) { columns.push( {title: 'Virtual-host observations', field: 'observations', formatter: vhostObservationsFormatter, minWidth: 420, widthGrow: 4, variableHeight: true, headerFilter: 'input', headerFilterFunc: vhostObservationsFilter, headerFilterPlaceholder: 'Filter endpoint evidence'}, @@ -503,6 +548,14 @@ {title: 'Produced by', field: 'actions', formatter: provenanceFormatter, minWidth: 130, responsive: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter}, ); } + if (state.route === 'shodan-host') { + columns.push( + {title: 'Network', field: 'details', formatter: shodanNetworkFormatter, minWidth: 220, widthGrow: 2, headerFilter: 'input', headerFilterFunc: shodanDetailsFilter}, + {title: 'Services', field: 'details', formatter: shodanServicesFormatter, minWidth: 360, widthGrow: 4, variableHeight: true, headerFilter: 'input', headerFilterFunc: shodanDetailsFilter}, + {title: 'Sources', field: 'sources', formatter: provenanceFormatter, minWidth: 130, responsive: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter}, + {title: 'Produced by', field: 'actions', formatter: provenanceFormatter, minWidth: 130, responsive: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter}, + ); + } if (state.route === 'hostname') { columns.push({ title: 'Actions', field: 'value', formatter: resultActionFormatter, headerSort: false, diff --git a/theHarvester/lib/completed_result.py b/theHarvester/lib/completed_result.py index 998959e1..0d4dd6e4 100644 --- a/theHarvester/lib/completed_result.py +++ b/theHarvester/lib/completed_result.py @@ -32,6 +32,7 @@ from theHarvester.lib.network_evidence import ( parse_network_observation_details, ) from theHarvester.lib.result_values import normalize_result_value +from theHarvester.lib.shodan_evidence import ShodanHostObservation, canonical_shodan_hosts from theHarvester.lib.virtual_host import VirtualHostObservation, normalize_virtual_host_hostname @@ -88,6 +89,8 @@ def parse_result_jsonl(payload: bytes | str) -> tuple[dict[str, object], list[di allowed_keys = {'type', 'value', 'sources', 'actions'} if result_kind in {'asn', 'hostname', 'prefix'} and 'observations' in record: allowed_keys.add('observations') + if result_kind == 'shodan-host' and 'details' in record: + allowed_keys.add('details') if result_kind == 'prefix': allowed_keys.add('scope') if ( @@ -107,7 +110,7 @@ def parse_result_jsonl(payload: bytes | str) -> tuple[dict[str, object], list[di if result_kind == 'prefix' and normalized_result_value != result_value: raise ValueError('prefix result is not canonical') except ValueError as error: - label = 'ASN' if result_kind == 'asn' else 'prefix' + label = {'asn': 'ASN', 'prefix': 'prefix', 'shodan-host': 'Shodan host'}.get(str(result_kind), 'result') raise ValueError(f'JSONL findings must use a canonical {label} value') from error record['value'] = normalized_result_value if result_kind == 'prefix' and record.get('scope') != 'external-relationship': @@ -132,6 +135,14 @@ def parse_result_jsonl(payload: bytes | str) -> tuple[dict[str, object], list[di except ValueError as error: raise ValueError(f'JSONL ASN has invalid organization attributions: {error}') from error record['observations'] = asn_attribution_details(asn_attributions) + elif result_kind == 'shodan-host': + try: + shodan_host = ShodanHostObservation.from_record(record['value'], record.get('details')) + except ValueError as error: + raise ValueError(f'JSONL Shodan host has invalid details: {error}') from error + if shodan_host.ip != record['value'] or shodan_host.to_details() != record.get('details'): + raise ValueError('JSONL Shodan host must use canonical structured details') + record['details'] = shodan_host.to_details() return summary, findings @@ -215,6 +226,7 @@ class CompletedResult: virtual_hosts: tuple[VirtualHostObservation, ...] = () network_observations: tuple[NetworkObservation, ...] = () asn_attributions: tuple[AsnAttributionObservation, ...] = () + shodan_hosts: tuple[ShodanHostObservation, ...] = () evidence_status: EvidenceStatus | None = None def __post_init__(self) -> None: @@ -321,13 +333,26 @@ class CompletedResult: not in origin_observations ): raise ValueError('BGP route and RPKI observations require matching observed-origin evidence') - sorted_asn_attributions = canonical_asn_attributions(list(self.asn_attributions)) - if self.asn_attributions != sorted_asn_attributions: - raise ValueError('ASN attributions must be deduplicated and sorted') + sorted_shodan_hosts = canonical_shodan_hosts(list(self.shodan_hosts)) + if self.shodan_hosts != sorted_shodan_hosts: + raise ValueError('Shodan host observations must be deduplicated and sorted') + structured_shodan_results = {('shodan-host', observation.ip) for observation in self.shodan_hosts} + if structured_shodan_results != {result for result in result_set if result[0] == 'shodan-host'}: + raise ValueError('Shodan host results must contain canonical structured evidence') source_results = {(observation.source, observation.kind, observation.value) for observation in self.observations} action_results = { (action, observation.kind, observation.value) for action, observation in self.active_evidence.observations } + for shodan_observation in self.shodan_hosts: + if ('shodan', 'shodan-host', shodan_observation.ip) not in source_results and ( + 'shodan', + 'shodan-host', + shodan_observation.ip, + ) not in action_results: + raise ValueError('Shodan host evidence must reference Shodan source or action provenance') + sorted_asn_attributions = canonical_asn_attributions(list(self.asn_attributions)) + if self.asn_attributions != sorted_asn_attributions: + raise ValueError('ASN attributions must be deduplicated and sorted') for attribution in self.asn_attributions: if attribution.collected_at < self.started_at or attribution.collected_at > self.completed_at: raise ValueError('ASN attribution collection time must fall within the completed run') @@ -365,6 +390,7 @@ class CompletedResult: virtual_hosts: Iterable[VirtualHostObservation] = (), network_observations: Iterable[NetworkObservation] = (), asn_attributions: Iterable[AsnAttributionObservation] = (), + shodan_hosts: Iterable[ShodanHostObservation] = (), evidence_status: EvidenceStatus | None = None, ) -> Self: completed_active_evidence = active_evidence if active_evidence is not None else ActiveEvidence() @@ -382,6 +408,9 @@ class CompletedResult: completed_virtual_hosts = tuple(sorted(set(virtual_hosts), key=VirtualHostObservation.sort_key)) for virtual_host in completed_virtual_hosts: results.add(('hostname', virtual_host.hostname)) + completed_shodan_hosts = canonical_shodan_hosts(list(shodan_hosts)) + for shodan_host in completed_shodan_hosts: + results.add(('shodan-host', shodan_host.ip)) return cls( run_id=run_id or uuid4(), target=target.strip(), @@ -394,6 +423,7 @@ class CompletedResult: virtual_hosts=completed_virtual_hosts, network_observations=canonical_network_observations(network_observations), asn_attributions=canonical_asn_attributions(list(asn_attributions)), + shodan_hosts=completed_shodan_hosts, evidence_status=evidence_status, ) @@ -459,6 +489,7 @@ class CompletedResult: attribution_by_asn: dict[str, list[AsnAttributionObservation]] = {} for attribution in self.asn_attributions: attribution_by_asn.setdefault(attribution.asn, []).append(attribution) + shodan_by_ip = {observation.ip: observation for observation in self.shodan_hosts} records: list[dict[str, object]] = [] for kind, value in self.results: record: dict[str, object] = { @@ -476,5 +507,7 @@ class CompletedResult: record['observations'] = network_observation_details(tuple(network_observations)) elif kind == 'asn' and (asn_attributions := attribution_by_asn.get(value)): record['observations'] = asn_attribution_details(tuple(asn_attributions)) + elif kind == 'shodan-host' and (shodan_host := shodan_by_ip.get(value)): + record['details'] = shodan_host.to_details() records.append(record) return records diff --git a/theHarvester/lib/core.py b/theHarvester/lib/core.py index 6cbcdc53..638d9cfd 100644 --- a/theHarvester/lib/core.py +++ b/theHarvester/lib/core.py @@ -774,7 +774,7 @@ class AsyncFetcher: proxy: str | bool | None = '', headers: dict[str, str] | None = None, follow_redirects: bool = False, - request_timeout: int = 60, + request_timeout: int | None = 60, ) -> AsyncIterator[aiohttp.ClientResponse]: try: ssl_arg = cls._ssl_context() @@ -814,7 +814,7 @@ class AsyncFetcher: params: Sized = '', proxy: str | bool | None = '', headers: dict[str, str] | None = None, - request_timeout: int = 60, + request_timeout: int | None = 60, ) -> FetcherResponse: """Fetch one bounded JSON response without following redirects.""" async with cls._open_get_response( diff --git a/theHarvester/lib/database.py b/theHarvester/lib/database.py index 54b5ee45..0a422cd0 100644 --- a/theHarvester/lib/database.py +++ b/theHarvester/lib/database.py @@ -61,6 +61,7 @@ from theHarvester.lib.network_evidence import ( network_observation_sort_key, parse_network_observation_json, ) +from theHarvester.lib.shodan_evidence import ShodanHostObservation, canonical_shodan_hosts from theHarvester.lib.virtual_host import VirtualHostObservation if TYPE_CHECKING: @@ -752,6 +753,7 @@ class ResultStore: network_by_prefix: dict[str, list[NetworkObservation]] = {} for network_observation in result.network_observations: network_by_prefix.setdefault(network_observation.prefix, []).append(network_observation) + shodan_by_ip = {observation.ip: observation for observation in result.shodan_hosts} async with self._session() as session: try: session.add( @@ -785,6 +787,13 @@ class ResultStore: sort_keys=True, ) if kind == 'prefix' and value in network_by_prefix + else json.dumps( + shodan_by_ip[value].to_details(), + ensure_ascii=False, + separators=(',', ':'), + sort_keys=True, + ) + if kind == 'shodan-host' and value in shodan_by_ip else None ), ) @@ -901,6 +910,7 @@ class ResultStore: } virtual_hosts: list[VirtualHostObservation] = [] network_observations: list[NetworkObservation] = [] + shodan_hosts: list[ShodanHostObservation] = [] for result_row in rows: has_vhost_provenance = result_row.position in vhost_result_positions if has_vhost_provenance: @@ -915,6 +925,18 @@ class ResultStore: raise ResultStoreError(f'Persisted virtual-host details are not canonical: {result_row.value}') virtual_hosts.extend(parsed_virtual_hosts) continue + if result_row.kind == 'shodan-host': + if result_row.details_json is None: + raise ResultStoreError(f'Persisted Shodan host details are missing: {result_row.value}') + try: + details = json.loads(result_row.details_json) + shodan_host = ShodanHostObservation.from_record(result_row.value, details) + except (json.JSONDecodeError, ValueError) as error: + raise ResultStoreError(f'Persisted Shodan host details are invalid: {result_row.value}') from error + if shodan_host.to_details() != details: + raise ResultStoreError(f'Persisted Shodan host details are not canonical: {result_row.value}') + shodan_hosts.append(shodan_host) + continue if result_row.details_json is None: continue if result_row.kind == 'prefix': @@ -1042,6 +1064,7 @@ class ResultStore: virtual_hosts=tuple(sorted(set(virtual_hosts), key=VirtualHostObservation.sort_key)), network_observations=tuple(sorted(set(network_observations), key=network_observation_sort_key)), asn_attributions=canonical_attributions, + shodan_hosts=canonical_shodan_hosts(shodan_hosts), evidence_status=cast('EvidenceStatus', parent.evidence_status) if parent.evidence_status is not None else None, ) diff --git a/theHarvester/lib/evidence_types.py b/theHarvester/lib/evidence_types.py index de936899..f7bf1a3a 100644 --- a/theHarvester/lib/evidence_types.py +++ b/theHarvester/lib/evidence_types.py @@ -20,7 +20,7 @@ ResultKind = Literal[ 'prefix', 'server', 'screenshot', - 'shodan', + 'shodan-host', 'takeover', 'twitter-person', 'url', diff --git a/theHarvester/lib/hostchecker.py b/theHarvester/lib/hostchecker.py index d2d7a681..05d26ce1 100644 --- a/theHarvester/lib/hostchecker.py +++ b/theHarvester/lib/hostchecker.py @@ -7,7 +7,9 @@ Revised to use aiodns & asyncio on 2019-09-23 from __future__ import annotations import asyncio +import inspect import ipaddress +import socket from dataclasses import dataclass from typing import TYPE_CHECKING @@ -36,6 +38,35 @@ def is_expected_dns_absence(error: BaseException) -> bool: ) +async def resolve_ip_addresses(hostname: str, *, family: socket.AddressFamily = socket.AF_UNSPEC) -> tuple[str, ...]: + """Resolve every unique IP address for a hostname.""" + resolver = aiodns.DNSResolver() + try: + answer = await resolver.getaddrinfo(hostname, family=family) + finally: + close = getattr(resolver, 'close', None) + if close is not None: + close_result = close() + if inspect.isawaitable(close_result): + await close_result + + addresses: set[str] = set() + for node in answer.nodes: + try: + value = node.addr[0] + if isinstance(value, bytes): + value = value.decode('ascii') + address = ipaddress.ip_address(value) + except (AttributeError, IndexError, TypeError, UnicodeDecodeError, ValueError): + continue + if family == socket.AF_INET and address.version != 4: + continue + if family == socket.AF_INET6 and address.version != 6: + continue + addresses.add(str(address)) + return tuple(sorted(addresses, key=lambda value: (ipaddress.ip_address(value).version, int(ipaddress.ip_address(value))))) + + class Checker: """Resolve hosts while preserving the legacy ``check()`` return tuple. diff --git a/theHarvester/lib/result_values.py b/theHarvester/lib/result_values.py index cc9ea10a..97468a60 100644 --- a/theHarvester/lib/result_values.py +++ b/theHarvester/lib/result_values.py @@ -1,6 +1,6 @@ from __future__ import annotations -from ipaddress import ip_network +from ipaddress import ip_address, ip_network from typing import TYPE_CHECKING if TYPE_CHECKING: @@ -40,4 +40,8 @@ def normalize_result_value(kind: ResultKind | str, value: str) -> str: return normalize_asn(normalized) if kind == 'prefix': return normalize_prefix(normalized) + if kind == 'shodan-host': + if '%' in normalized: + raise ValueError('Shodan host must not contain an IPv6 scope identifier') + return str(ip_address(normalized)) return normalized diff --git a/theHarvester/lib/shodan_evidence.py b/theHarvester/lib/shodan_evidence.py new file mode 100644 index 00000000..dd585c4d --- /dev/null +++ b/theHarvester/lib/shodan_evidence.py @@ -0,0 +1,221 @@ +from __future__ import annotations + +from dataclasses import dataclass +from ipaddress import ip_address +from typing import Self + +from theHarvester.lib.result_values import normalize_asn + + +def _optional_text(value: object, field: str) -> str | None: + if value is None: + return None + if not isinstance(value, str): + raise ValueError(f'Shodan {field} must be a string') + return value.strip() or None + + +def _string_tuple(value: object, field: str) -> tuple[str, ...]: + if value is None: + return () + if not isinstance(value, list) or any(not isinstance(item, str) for item in value): + raise ValueError(f'Shodan {field} must be an array of strings') + return tuple(sorted({item.strip() for item in value if item.strip()})) + + +def _hostname_tuple(value: object, field: str) -> tuple[str, ...]: + return tuple(sorted({item.casefold().rstrip('.') for item in _string_tuple(value, field)} - {''})) + + +@dataclass(frozen=True, slots=True) +class ShodanServiceObservation: + port: int + transport: str + product: str | None = None + version: str | None = None + observed_at: str | None = None + cpes: tuple[str, ...] = () + http_title: str | None = None + http_server: str | None = None + http_components: tuple[str, ...] = () + tls_subject_cn: str | None = None + tls_subject_alt_names: tuple[str, ...] = () + tls_issuer_cn: str | None = None + tls_expires_at: str | None = None + tls_sha256: str | None = None + tls_jarm: str | None = None + + @classmethod + def from_record(cls, record: object) -> Self: + if not isinstance(record, dict): + raise ValueError('Shodan services must be objects') + allowed = {'port', 'transport', 'product', 'version', 'observed_at', 'cpes', 'http', 'tls'} + if set(record) - allowed: + raise ValueError('Shodan service contains unsupported fields') + port = record.get('port') + if isinstance(port, bool) or not isinstance(port, int) or not 1 <= port <= 65535: + raise ValueError('Shodan service port must be between 1 and 65535') + transport = _optional_text(record.get('transport'), 'service transport') + if transport is None or transport.casefold() not in {'tcp', 'udp'}: + raise ValueError('Shodan service transport must be tcp or udp') + + http = record.get('http') + if http is not None and (not isinstance(http, dict) or set(http) - {'title', 'server', 'components'}): + raise ValueError('Shodan HTTP evidence contains unsupported fields') + tls = record.get('tls') + if tls is not None and ( + not isinstance(tls, dict) + or set(tls) - {'subject_cn', 'subject_alt_names', 'issuer_cn', 'expires_at', 'sha256', 'jarm'} + ): + raise ValueError('Shodan TLS evidence contains unsupported fields') + + return cls( + port=port, + transport=transport.casefold(), + product=_optional_text(record.get('product'), 'service product'), + version=_optional_text(record.get('version'), 'service version'), + observed_at=_optional_text(record.get('observed_at'), 'service observation time'), + cpes=_string_tuple(record.get('cpes'), 'service CPEs'), + http_title=_optional_text(http.get('title'), 'HTTP title') if isinstance(http, dict) else None, + http_server=_optional_text(http.get('server'), 'HTTP server') if isinstance(http, dict) else None, + http_components=_string_tuple(http.get('components'), 'HTTP components') if isinstance(http, dict) else (), + tls_subject_cn=_optional_text(tls.get('subject_cn'), 'TLS subject CN') if isinstance(tls, dict) else None, + tls_subject_alt_names=( + _hostname_tuple(tls.get('subject_alt_names'), 'TLS subject alternative names') if isinstance(tls, dict) else () + ), + tls_issuer_cn=_optional_text(tls.get('issuer_cn'), 'TLS issuer CN') if isinstance(tls, dict) else None, + tls_expires_at=_optional_text(tls.get('expires_at'), 'TLS expiry') if isinstance(tls, dict) else None, + tls_sha256=_optional_text(tls.get('sha256'), 'TLS SHA-256') if isinstance(tls, dict) else None, + tls_jarm=_optional_text(tls.get('jarm'), 'TLS JARM') if isinstance(tls, dict) else None, + ) + + def sort_key(self) -> tuple[object, ...]: + return ( + self.port, + self.transport, + self.observed_at or '', + self.product or '', + self.version or '', + self.http_title or '', + self.http_server or '', + self.http_components, + self.tls_subject_cn or '', + self.tls_subject_alt_names, + self.tls_issuer_cn or '', + self.tls_expires_at or '', + self.tls_sha256 or '', + self.tls_jarm or '', + self.cpes, + ) + + def to_record(self) -> dict[str, object]: + record: dict[str, object] = {'port': self.port, 'transport': self.transport} + for field, value in ( + ('product', self.product), + ('version', self.version), + ('observed_at', self.observed_at), + ): + if value is not None: + record[field] = value + if self.cpes: + record['cpes'] = list(self.cpes) + http: dict[str, object] = {} + if self.http_title is not None: + http['title'] = self.http_title + if self.http_server is not None: + http['server'] = self.http_server + if self.http_components: + http['components'] = list(self.http_components) + if http: + record['http'] = http + tls: dict[str, object] = {} + if self.tls_subject_alt_names: + tls['subject_alt_names'] = list(self.tls_subject_alt_names) + for field, value in ( + ('subject_cn', self.tls_subject_cn), + ('issuer_cn', self.tls_issuer_cn), + ('expires_at', self.tls_expires_at), + ('sha256', self.tls_sha256), + ('jarm', self.tls_jarm), + ): + if value is not None: + tls[field] = value + if tls: + record['tls'] = tls + return record + + +@dataclass(frozen=True, slots=True) +class ShodanHostObservation: + ip: str + services: tuple[ShodanServiceObservation, ...] + asn: str | None = None + organization: str | None = None + isp: str | None = None + hostnames: tuple[str, ...] = () + domains: tuple[str, ...] = () + + @classmethod + def from_record(cls, value: str, details: object) -> Self: + if not isinstance(value, str) or '%' in value: + raise ValueError('Shodan host value must be a canonical IP address') + try: + canonical_ip = str(ip_address(value.strip())) + except ValueError as error: + raise ValueError('Shodan host value must be a canonical IP address') from error + if not isinstance(details, dict): + raise ValueError('Shodan host details must be an object') + allowed = {'asn', 'organization', 'isp', 'hostnames', 'domains', 'services'} + if set(details) - allowed: + raise ValueError('Shodan host details contain unsupported fields') + raw_services = details.get('services') + if not isinstance(raw_services, list) or not raw_services: + raise ValueError('Shodan host details require at least one service') + services = tuple( + sorted( + {ShodanServiceObservation.from_record(service) for service in raw_services}, + key=ShodanServiceObservation.sort_key, + ) + ) + raw_asn = _optional_text(details.get('asn'), 'ASN') + return cls( + ip=canonical_ip, + services=services, + asn=normalize_asn(raw_asn) if raw_asn is not None else None, + organization=_optional_text(details.get('organization'), 'organization'), + isp=_optional_text(details.get('isp'), 'ISP'), + hostnames=_hostname_tuple(details.get('hostnames'), 'hostnames'), + domains=_hostname_tuple(details.get('domains'), 'domains'), + ) + + def sort_key(self) -> tuple[int, int]: + address = ip_address(self.ip) + return address.version, int(address) + + def to_details(self) -> dict[str, object]: + details: dict[str, object] = {} + for field, value in ( + ('asn', self.asn), + ('organization', self.organization), + ('isp', self.isp), + ): + if value is not None: + details[field] = value + if self.hostnames: + details['hostnames'] = list(self.hostnames) + if self.domains: + details['domains'] = list(self.domains) + details['services'] = [service.to_record() for service in self.services] + return details + + +def canonical_shodan_hosts( + observations: tuple[ShodanHostObservation, ...] | list[ShodanHostObservation], +) -> tuple[ShodanHostObservation, ...]: + by_ip: dict[str, ShodanHostObservation] = {} + for observation in observations: + existing = by_ip.get(observation.ip) + if existing is not None and existing != observation: + raise ValueError(f'Shodan host {observation.ip} has conflicting evidence') + by_ip[observation.ip] = observation + return tuple(sorted(by_ip.values(), key=ShodanHostObservation.sort_key)) diff --git a/uv.lock b/uv.lock index 85aa7f9a..2466d206 100644 --- a/uv.lock +++ b/uv.lock @@ -466,18 +466,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/c7/0d/67e5b4109ea4a837e80daa87c2c696711955e40449a97e8926672534def2/click-8.4.1-py3-none-any.whl", hash = "sha256:482be17c6991b8c19c5429a1e995d9b0efdbb63172824c41f99965dc0ade8ec2", size = 116639, upload-time = "2026-05-22T04:08:35.26Z" }, ] -[[package]] -name = "click-plugins" -version = "1.1.1.2" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "click" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/c3/a4/34847b59150da33690a36da3681d6bbc2ec14ee9a846bc30a6746e5984e4/click_plugins-1.1.1.2.tar.gz", hash = "sha256:d7af3984a99d243c131aa1a828331e7630f4a88a9741fd05c927b204bcf92261", size = 8343, upload-time = "2025-06-25T00:47:37.555Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/3d/9a/2abecb28ae875e39c8cad711eb1186d8d14eab564705325e77e4e6ab9ae5/click_plugins-1.1.1.2-py2.py3-none-any.whl", hash = "sha256:008d65743833ffc1f5417bf0e78e8d2c23aab04d9745ba817bd3e71b0feb6aa6", size = 11051, upload-time = "2025-06-25T00:47:36.731Z" }, -] - [[package]] name = "colorama" version = "0.4.6" @@ -512,15 +500,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/38/a9/69a6924f645eb4dd8cd625bf255b3625990eb3e14e073438a53c405dcd3e/fastapi-0.138.1-py3-none-any.whl", hash = "sha256:b994cae7ba8b82c976a728b544244de31333fa5f7d261f9a1dffe526444cae23", size = 129182, upload-time = "2026-06-25T15:40:40.771Z" }, ] -[[package]] -name = "filelock" -version = "3.29.4" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/e6/dc/be6cbe99670cd6e4ad387123647cb08e0c32975e223f82551e914c5568a6/filelock-3.29.4.tar.gz", hash = "sha256:10cdb3656fc44541cdf30652a93fb10ec6b05325620eb316bd26893e4201538a", size = 63028, upload-time = "2026-06-13T16:12:00.744Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/13/37/a065dc3bd6e49423a6532c642ca7378d3f467b1ef44c2800c937af7f9739/filelock-3.29.4-py3-none-any.whl", hash = "sha256:dac1648087d5115554850d113e7dd8c83ab2d38e3435dde2d4f163847e57b767", size = 42757, upload-time = "2026-06-13T16:11:59.582Z" }, -] - [[package]] name = "frozenlist" version = "1.8.0" @@ -1524,18 +1503,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" }, ] -[[package]] -name = "requests-file" -version = "3.0.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "requests" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/3c/f8/5dc70102e4d337063452c82e1f0d95e39abfe67aa222ed8a5ddeb9df8de8/requests_file-3.0.1.tar.gz", hash = "sha256:f14243d7796c588f3521bd423c5dea2ee4cc730e54a3cac9574d78aca1272576", size = 6967, upload-time = "2025-10-20T18:56:42.279Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/e1/d5/de8f089119205a09da657ed4784c584ede8381a0ce6821212a6d4ca47054/requests_file-3.0.1-py2.py3-none-any.whl", hash = "sha256:d0f5eb94353986d998f80ac63c7f146a307728be051d4d1cd390dbdb59c10fa2", size = 4514, upload-time = "2025-10-20T18:56:41.184Z" }, -] - [[package]] name = "retrying" version = "1.4.2" @@ -1583,20 +1550,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cb/46/240ea004bf6dc4feb40e9832f2205a476a47dd5b8a3f8211a5fc5f95e20e/ruff-0.16.1-py3-none-win_arm64.whl", hash = "sha256:dbaadaac38c70239f056d306b7476f246b0bf000fa6b3876402acbf5b227eaf8", size = 11309414, upload-time = "2026-07-30T19:36:58.79Z" }, ] -[[package]] -name = "shodan" -version = "1.31.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "click" }, - { name = "click-plugins" }, - { name = "colorama" }, - { name = "requests" }, - { name = "tldextract" }, - { name = "xlsxwriter" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/c5/06/c6dcc975a1e7d89bc764fd271da8138b318e18080b48e7f1acd2ab63df28/shodan-1.31.0.tar.gz", hash = "sha256:c73275386ea02390e196c35c660706a28dd4d537c5a21eb387ab6236fac251f6", size = 57939, upload-time = "2023-12-17T01:42:02.426Z" } - [[package]] name = "six" version = "1.17.0" @@ -1700,7 +1653,6 @@ dependencies = [ { name = "python-dateutil" }, { name = "pyyaml" }, { name = "retrying" }, - { name = "shodan" }, { name = "sqlalchemy" }, { name = "ujson" }, { name = "uvicorn" }, @@ -1745,7 +1697,6 @@ requires-dist = [ { name = "python-dateutil", specifier = "==2.9.0.post0" }, { name = "pyyaml", specifier = "==6.0.3" }, { name = "retrying", specifier = "==1.4.2" }, - { name = "shodan", specifier = "==1.31.0" }, { name = "sqlalchemy", specifier = "==2.0.51" }, { name = "ujson", specifier = "==5.13.0" }, { name = "uvicorn", specifier = "==0.52.1" }, @@ -1770,21 +1721,6 @@ dev = [ { name = "wheel", specifier = "==0.47.0" }, ] -[[package]] -name = "tldextract" -version = "5.3.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "filelock" }, - { name = "idna" }, - { name = "requests" }, - { name = "requests-file" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/65/7b/644fbbb49564a6cb124a8582013315a41148dba2f72209bba14a84242bf0/tldextract-5.3.1.tar.gz", hash = "sha256:a72756ca170b2510315076383ea2993478f7da6f897eef1f4a5400735d5057fb", size = 126105, upload-time = "2025-12-28T23:58:05.532Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/6d/42/0e49d6d0aac449ca71952ec5bae764af009754fcb2e76a5cc097543747b3/tldextract-5.3.1-py3-none-any.whl", hash = "sha256:6bfe36d518de569c572062b788e16a659ccaceffc486d243af0484e8ecf432d9", size = 105886, upload-time = "2025-12-28T23:58:04.071Z" }, -] - [[package]] name = "ty" version = "0.0.69" @@ -2024,15 +1960,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0f/8d/52eaa9187b88596b0a8b646874cfec5a5c3fce8c52b5182be1a0253203a3/winloop-0.6.3-cp314-cp314t-win_arm64.whl", hash = "sha256:447006f38f13827ff4600e7beeda70367370cb8dab8ea84042e8fa1749f32b1c", size = 576206, upload-time = "2026-04-27T16:08:07.007Z" }, ] -[[package]] -name = "xlsxwriter" -version = "3.2.9" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/46/2c/c06ef49dc36e7954e55b802a8b231770d286a9758b3d936bd1e04ce5ba88/xlsxwriter-3.2.9.tar.gz", hash = "sha256:254b1c37a368c444eac6e2f867405cc9e461b0ed97a3233b2ac1e574efb4140c", size = 215940, upload-time = "2025-09-16T00:16:21.63Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/3a/0c/3662f4a66880196a590b202f0db82d919dd2f89e99a27fadef91c4a33d41/xlsxwriter-3.2.9-py3-none-any.whl", hash = "sha256:9a5db42bc5dff014806c58a20b9eae7322a134abb6fce3c92c181bfb275ec5b3", size = 175315, upload-time = "2025-09-16T00:16:20.108Z" }, -] - [[package]] name = "yarl" version = "1.24.2"