WEBDIENSTE/theHarvester
1
0
Fork 0
mirror of https://github.com/laramies/theHarvester.git synced 2026-08-17 19:35:40 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix: move Shodan discovery into adapter (#2535)

Browse Source 
* fix: move Shodan discovery into adapter (#283)

* docs: clarify Shodan transport and persistence

* fix: query every Shodan-resolved IPv4

* fix: call Shodan Host API directly

* fix: serve HarvestView assets locally

* Revert "fix: serve HarvestView assets locally"

This reverts commit 413e8b25ab.

* docs: clarify Shodan proxy transport

* docs: clarify Shodan changelog entry

* feat: persist structured Shodan host evidence

* feat: expand Shodan discovery with TLS search
This commit is contained in:
Matt
2026-08-14 21:11:12 -04:00
committed by GitHub
parent 259455f1c1
commit b8a8f7c7ca
27 changed files with 2054 additions and 443 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+1
View File
@@ -35,6 +35,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added root contributor and security policies, structured issue forms, repository agent guidance, discovery terminology, and an operator-focused documentation wiki ([d090a29a](https://github.com/laramies/theHarvester/commit/d090a29a), [7c491ef5](https://github.com/laramies/theHarvester/commit/7c491ef5), [8b9d420b](https://github.com/laramies/theHarvester/commit/8b9d420b)).
### Changed
- Replaced Shodan's synchronous Python SDK with cancellable async Host API requests that honor configured proxies, query every unique resolved IPv4, paginate target-bound hostname and TLS-certificate searches without an adapter-specific result cap, retain successful partial results, and add no source-local deadline. Shodan now stores one canonical `shodan-host` result per IP with every normalized TCP or UDP service and scoped certificate CN/SAN metadata in native JSONL, SQLite, API, and HarvestView details instead of an escaped JSON value.
- Removed the transport-wide delay before reading ready HTTP responses, bounded Wayback Archive to 30 seconds and Common Crawl to 120 seconds, kept both sources within the requested result limit, and made long-source progress visible in verbose mode. Common Crawl now requests one 50-record page at a time instead of bursting page batches.
- Made Baidu, crt.sh, HackerTarget, Have I Been Pwned, Mojeek, OTX, and Robtex report blocked, malformed, or transport failures truthfully. Also fixed HackerTarget CSV parsing and Robtex AAAA results.
- Hardened BufferOver, ProjectDiscovery, DNSDumpster, ONYPHE, and URLScan parsing and result attribution, including scoped typed results and bounded URLScan pagination.
README.md
+11 -2
View File
@@ -247,6 +247,7 @@ On first use, theHarvester creates default configuration files under `~/.theHarv
- `api-keys.yaml` stores provider credentials.
- `proxies.yaml` configures HTTP and SOCKS5 proxies used with `-p`.
- The `shodan` source and `-s` / `--shodan` enrichment call Shodan's Host REST API without the Python SDK. When `-p` is enabled, both send those requests through `proxies.yaml`.
Never commit populated configuration files, API keys, account details, or provider responses.
@@ -273,7 +274,7 @@ The JSON report is a single object. Host entries remain plain hostnames or `host
| --- | --- | --- |
| `cmd` | Always | Command-line arguments used for the run. |
| `hosts` | Always | Discovered hosts; an empty array when none are found. |
| `shodan` | Always | Shodan enrichment rows; an empty array when Shodan is not used. |
| `shodan` | Always | Shodan host objects with canonical IP `value` and structured `details`; an empty array when Shodan is not used. |
| `ips`, `emails`, `vhosts`, `asns`, `prefixes` | When non-empty | Network and contact findings. RouteViews prefixes are external routing relationships, not claimed target scope. |
| `urls` | When non-empty | Discovered URLs from every URL-producing source or action. |
| `people`, `twitter_people`, `linkedin_people` | When non-empty | People and profile findings. |
@@ -288,7 +289,9 @@ The JSONL report is finalized after the selected one-shot actions finish. The fi
{"sources":[],"type":"hostname","value":"api.example.com"}
```
JSONL is easy to stream one record at a time. The summary preserves the evidence status, source and action outcomes, and screenshot artifact metadata. Finding lines carry `sources` and, when applicable, `actions`; they inherit their run ID and target from the preceding summary. Hostnames, IP addresses, and URLs use the same `hostname`, `ip`, and `url` result kinds in JSONL, SQLite, the API, and HarvestView. Provenance identifies which source or action produced each finding. Structured result types, including recursive DNS records plus `person`, `infostealer`, `shodan`, and `takeover`, store a JSON object inside the string `value`. Parse those values a second time with `fromjson`.
JSONL is easy to stream one record at a time. The summary preserves the evidence status, source and action outcomes, and screenshot artifact metadata. Finding lines carry `sources` and, when applicable, `actions`; they inherit their run ID and target from the preceding summary. Hostnames, IP addresses, and URLs use the same `hostname`, `ip`, and `url` result kinds in JSONL, SQLite, the API, and HarvestView. Provenance identifies which source or action produced each finding. Recursive DNS records plus `person`, `infostealer`, and `takeover` store a JSON object inside the string `value`; parse those values a second time with `fromjson`.
Shodan host findings instead use the canonical IP as `value` and place normalized host and per-service evidence in a native `details` object. Shodan discovery paginates both hostname and TLS-certificate searches for the target domain without an adapter-specific result cap, merges duplicate services by IP, and rejects names outside the requested domain. Host metadata appears once, while each service retains its port, TCP or UDP transport, product, version, observation time, CPEs, and available HTTP or TLS summary, including scoped certificate CNs and SANs. Raw banners, response bodies, certificate chains, and Shodan crawler metadata are not retained.
Virtual-host observations do not use that string encoding. Each confirmed name remains one `hostname` finding with `actions: ["vhost"]` and a native `observations` array. Several endpoint observations can enrich the same hostname without creating another result kind or count.
@@ -302,6 +305,12 @@ Parse recursive DNS findings as JSON objects:
jq -c 'select(.type == "dns-recursive-finding") | .value | fromjson' report.jsonl
```
List Shodan services by host:
```bash
jq -c 'select(.type == "shodan-host") | {ip: .value, services: .details.services}' report.jsonl
```
List the endpoint observations for each confirmed virtual host:
```bash
pyproject.toml
-1
View File
@@ -37,7 +37,6 @@ dependencies = [
"python-dateutil==2.9.0.post0",
"httpx==0.28.1",
"retrying==1.4.2",
"shodan==1.31.0",
"sqlalchemy==2.0.51",
"ujson==5.13.0",
"uvicorn==0.52.1",
tests/discovery/test_shodan_engine.py
+535 -122
View File
@@ -1,35 +1,172 @@
import asyncio
import logging
import socket
import sys
from collections import OrderedDict
from datetime import UTC
from types import SimpleNamespace
from uuid import UUID
import pytest
def patch_resolution(monkeypatch, module, addresses=('203.0.113.10',)):
requested_targets = []
async def resolve_ip_addresses(target, *, family=socket.AF_UNSPEC):
requested_targets.append((target, family))
return tuple(addresses)
monkeypatch.setattr(module, 'resolve_ip_addresses', resolve_ip_addresses, raising=True)
return requested_targets
@pytest.fixture(autouse=True)
def disable_shodan_pacing(monkeypatch):
from theHarvester.discovery import shodansearch
monkeypatch.setattr(shodansearch.SearchShodan, 'REQUEST_INTERVAL_SECONDS', 0.0)
class TestShodanEngine:
@pytest.mark.asyncio
async def test_shodan_retains_sourced_asn_organization_attribution(self, monkeypatch):
from theHarvester.discovery import shodansearch
async def test_shared_ipv4_resolution_returns_every_unique_address_and_closes(self, monkeypatch):
from theHarvester.lib import hostchecker
class SuccessfulShodan:
def host(self, _ip):
return {
closed = False
class Resolver:
async def getaddrinfo(self, target, *, family):
assert target == 'example.test'
assert family == socket.AF_INET
return SimpleNamespace(
nodes=[
SimpleNamespace(addr=(b'203.0.113.11', 0)),
SimpleNamespace(addr=('invalid', 0)),
SimpleNamespace(addr=('203.0.113.10', 0)),
SimpleNamespace(addr=('203.0.113.11', 0)),
]
)
async def close(self):
nonlocal closed
closed = True
monkeypatch.setattr(hostchecker.aiodns, 'DNSResolver', Resolver)
assert await hostchecker.resolve_ip_addresses('example.test', family=socket.AF_INET) == (
'203.0.113.10',
'203.0.113.11',
)
assert closed is True
@pytest.mark.asyncio
async def test_shodan_calls_direct_host_api_through_proxy_and_retains_attribution(self, monkeypatch):
from theHarvester.discovery import shodansearch
from theHarvester.lib.core import FetcherResponse
async def fetch_json(url, *, params, proxy, request_timeout):
assert url == 'https://api.shodan.io/shodan/host/192.0.2.10'
assert 'test-key' not in url
assert params == {'key': 'test-key'}
assert proxy is True
assert request_timeout is None
return FetcherResponse(
body={
'asn': 'AS64496',
'org': 'Example Transit',
'domains': ['Example.TEST.'],
'hostnames': ['API.Example.TEST.'],
'ip_str': '192.0.2.10',
'isp': 'Example ISP Label',
'data': [{'ip_str': '198.51.100.20'}],
}
'org': 'Example Transit',
'data': [
{
'port': 53,
'transport': 'udp',
'product': 'dnsmasq',
'version': '2.90',
'timestamp': '2026-08-14T11:58:00Z',
'cpe': ['cpe:/a:thekelleys:dnsmasq:2.90'],
'data': 'raw provider banner must not be retained',
},
{
'port': 443,
'transport': 'tcp',
'product': 'nginx',
'version': '1.24.0',
'timestamp': '2026-08-14T12:01:00Z',
'http': {
'components': {'nginx': {}, 'Python': {}},
'server': 'nginx',
'title': 'Example',
'html': 'raw response body must not be retained',
},
'ssl': {
'jarm': 'example-jarm',
'cert': {
'subject': {'CN': 'api.example.test'},
'issuer': {'CN': 'Example CA'},
'expires': '2027-08-14T00:00:00Z',
'fingerprint': {'sha256': '0123456789abcdef'},
'extensions': [
{
'name': 'subjectAltName',
'data': r'0\x82\x10api.example.test\x82\x10www.example.test',
}
],
},
},
},
],
},
status=200,
headers={},
)
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: SuccessfulShodan())
monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json)
search = shodansearch.SearchShodan()
await search.search_ip('192.0.2.10')
result = await search.search_ip('192.0.2.10', proxy=True)
attributions = await search.get_asn_attributions()
assert len(attributions) == 1
attribution = next(iter(attributions))
assert result['192.0.2.10'] == {
'asn': 'AS64496',
'domains': ['example.test'],
'hostnames': ['api.example.test'],
'isp': 'Example ISP Label',
'organization': 'Example Transit',
'services': [
{
'cpes': ['cpe:/a:thekelleys:dnsmasq:2.90'],
'observed_at': '2026-08-14T11:58:00Z',
'port': 53,
'product': 'dnsmasq',
'transport': 'udp',
'version': '2.90',
},
{
'http': {
'components': ['Python', 'nginx'],
'server': 'nginx',
'title': 'Example',
},
'observed_at': '2026-08-14T12:01:00Z',
'port': 443,
'product': 'nginx',
'tls': {
'expires_at': '2027-08-14T00:00:00Z',
'issuer_cn': 'Example CA',
'jarm': 'example-jarm',
'sha256': '0123456789abcdef',
'subject_alt_names': ['www.example.test'],
'subject_cn': 'api.example.test',
},
'transport': 'tcp',
'version': '1.24.0',
},
],
}
attribution = next(iter(await search.get_asn_attributions()))
assert attribution.producer_kind == 'action'
assert attribution.producer == 'shodan'
assert attribution.asn == 'AS64496'
@@ -39,15 +176,15 @@ class TestShodanEngine:
assert attribution.collected_at.tzinfo is UTC
@pytest.mark.asyncio
async def test_shodan_provider_failure_returns_attributed_empty_evidence(self, monkeypatch, caplog):
async def test_shodan_maps_404_to_a_successful_zero_yield(self, monkeypatch, caplog):
from theHarvester.discovery import shodansearch
from theHarvester.lib.core import FetcherResponse
class FailingShodan:
def host(self, _ip):
raise shodansearch.exception.APIError('No information available for that IP.')
async def fetch_json(*_args, **_kwargs):
return FetcherResponse(body=None, status=404, headers={})
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: FailingShodan())
monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json)
caplog.set_level(logging.INFO, logger=shodansearch.__name__)
search = shodansearch.SearchShodan()
@@ -58,67 +195,385 @@ class TestShodanEngine:
assert '203.0.113.1: Not in Shodan' in caplog.text
@pytest.mark.asyncio
async def test_shodan_api_failure_exposes_only_its_error_type(self, monkeypatch, caplog):
@pytest.mark.parametrize('status', [401, 403, 429, 500])
async def test_shodan_classifies_http_failures_without_provider_payload(self, monkeypatch, caplog, status):
from theHarvester.discovery import shodansearch
from theHarvester.lib.core import FetcherResponse
class FailingShodan:
def host(self, _ip):
raise shodansearch.exception.APIError('provider-secret-payload')
async def fetch_json(*_args, **_kwargs):
return FetcherResponse(body=None, status=status, headers={'x-provider-secret': 'secret'})
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: FailingShodan())
monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json)
caplog.set_level(logging.INFO, logger=shodansearch.__name__)
search = shodansearch.SearchShodan()
result = await search.search_ip('203.0.113.1')
assert result == OrderedDict({'203.0.113.1': 'Shodan request failed'})
assert search.error_type == 'APIError'
assert 'provider-secret-payload' not in caplog.text
assert search.error_type == f'HTTP{status}Error'
assert 'secret' not in caplog.text
@pytest.mark.asyncio
async def test_shodan_unexpected_failure_exposes_only_its_error_type(self, monkeypatch):
async def test_shodan_classifies_bounded_transport_failure(self, monkeypatch):
from theHarvester.discovery import shodansearch
from theHarvester.lib.core import ResponseStreamError
class FailingShodan:
def host(self, _ip):
raise RuntimeError('provider-secret-payload')
async def fetch_json(*_args, **_kwargs):
raise ResponseStreamError('response-limit')
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: FailingShodan())
monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json)
search = shodansearch.SearchShodan()
result = await search.search_ip('203.0.113.1')
assert result == OrderedDict({'203.0.113.1': 'Shodan request failed'})
assert search.error_type == 'RuntimeError'
assert search.error_type == 'ResponseLimitError'
@pytest.mark.asyncio
async def test_shodan_engine_processes_without_work_item_error_and_yields_hostnames(self, monkeypatch, capsys):
# Import inside the test so monkeypatching affects the already-imported module namespace.
async def test_shodan_discovery_queries_every_resolved_ipv4_and_keeps_partial_results(self, monkeypatch):
from theHarvester.discovery import shodansearch
from theHarvester.lib.core import FetcherResponse
queried_urls = []
async def fetch_json(url, **_kwargs):
if url.endswith('/search'):
return FetcherResponse(body={'matches': [], 'total': 0}, status=200, headers={})
queried_urls.append(url)
ip = url.rsplit('/', 1)[-1]
if ip == '203.0.113.10':
return FetcherResponse(body=None, status=500, headers={})
return FetcherResponse(
body={
'data': [{'ip_str': ip, 'port': 443, 'transport': 'tcp'}],
'hostnames': ['CDN.Example.TEST.', 'outside.test'],
},
status=200,
headers={},
)
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json)
targets = patch_resolution(monkeypatch, shodansearch, ('203.0.113.10', '203.0.113.11'))
search = shodansearch.SearchShodan('WWW.Example.TEST.')
await search.process(proxy=True)
assert targets == [('www.example.test', socket.AF_INET)]
assert queried_urls == [
'https://api.shodan.io/shodan/host/203.0.113.10',
'https://api.shodan.io/shodan/host/203.0.113.11',
]
assert await search.get_hostnames() == {'cdn.example.test'}
assert search.execution_status == 'partial'
assert search.stop_reason == 'provider-errors'
@pytest.mark.asyncio
async def test_shodan_discovery_paginates_hostname_and_tls_searches_with_scoped_certificate_names(self, monkeypatch):
from theHarvester.discovery import shodansearch
from theHarvester.lib.core import FetcherResponse
search_calls = []
def banner(
ip,
port,
*,
hostnames=(),
subject_cn=None,
subject_alt_name='',
):
cert = {
'extensions': [
{'name': 'subjectAltName', 'data': subject_alt_name},
{'name': 'keyUsage', 'data': 'Digital Signature'},
]
}
if subject_cn is not None:
cert['subject'] = {'CN': subject_cn}
return {
'ip_str': ip,
'hostnames': list(hostnames),
'port': port,
'transport': 'tcp',
'ssl': {'cert': cert},
}
hostname_pages = {
1: {
'total': 2,
'matches': [
banner(
'198.51.100.20',
443,
hostnames=('API.Example.TEST.',),
subject_cn='*.example.test',
subject_alt_name=(r'0\x82\x10api.example.test\x82\x12outside.invalid\x82\x13vpn.example.test'),
)
],
},
2: {
'total': 2,
'matches': [banner('198.51.100.20', 8443, hostnames=('api.example.test',))],
},
}
ssl_page = {
'total': 3,
'matches': [
banner(
'198.51.100.20',
443,
hostnames=('api.example.test',),
subject_cn='*.example.test',
subject_alt_name=r'0\x82\x10api.example.test\x82\x13vpn.example.test',
),
banner(
'198.51.100.21',
443,
subject_cn='cert.example.test',
subject_alt_name=r'0\x82\x13cert.example.test\x82\x15outside.invalid',
),
banner(
'198.51.100.22',
443,
hostnames=('outside.invalid',),
subject_cn='outside.invalid',
subject_alt_name=r'0\x82\x15outside.invalid',
),
],
}
async def fetch_json(url, *, params, proxy, request_timeout):
assert proxy is True
assert request_timeout is None
assert params['key'] == 'test-key'
if url.endswith('/search'):
search_calls.append(dict(params))
query = params['query']
if query == 'hostname:example.test':
return FetcherResponse(body=hostname_pages[params['page']], status=200, headers={})
assert query == 'ssl:example.test'
assert params['page'] == 1
return FetcherResponse(body=ssl_page, status=200, headers={})
assert url == 'https://api.shodan.io/shodan/host/203.0.113.10'
return FetcherResponse(
body={'data': [{'port': 80, 'transport': 'tcp'}], 'hostnames': ['www.example.test']},
status=200,
headers={},
)
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json)
patch_resolution(monkeypatch, shodansearch)
search = shodansearch.SearchShodan('example.test')
await search.process(proxy=True)
assert [(call['query'], call['page']) for call in search_calls] == [
('hostname:example.test', 1),
('hostname:example.test', 2),
('ssl:example.test', 1),
]
assert all(call['minify'] == 'false' and call['fields'] == search.SEARCH_FIELDS for call in search_calls)
assert await search.get_hostnames() == {
'api.example.test',
'cert.example.test',
'vpn.example.test',
'www.example.test',
}
hosts = {host.ip: host.to_details() for host in await search.get_shodan_hosts()}
assert set(hosts) == {'198.51.100.20', '198.51.100.21', '203.0.113.10'}
assert [service['port'] for service in hosts['198.51.100.20']['services']] == [443, 8443]
assert hosts['198.51.100.20']['services'][0]['tls'] == {
'subject_alt_names': ['api.example.test', 'vpn.example.test'],
'subject_cn': '*.example.test',
}
assert hosts['198.51.100.21']['services'][0]['tls'] == {'subject_cn': 'cert.example.test'}
assert search.execution_status == 'completed'
assert search.stop_reason is None
@pytest.mark.asyncio
async def test_shodan_discovery_counts_service_only_evidence_as_a_result(self, monkeypatch):
from theHarvester.discovery import shodansearch
from theHarvester.lib.core import FetcherResponse
async def fetch_json(url, **_kwargs):
if url.endswith('/search'):
return FetcherResponse(body={'matches': [], 'total': 0}, status=200, headers={})
return FetcherResponse(
body={'data': [{'port': 53, 'transport': 'udp'}], 'hostnames': []},
status=200,
headers={},
)
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json)
patch_resolution(monkeypatch, shodansearch)
search = shodansearch.SearchShodan('example.test')
await search.process()
assert search.execution_status == 'completed'
assert search.stop_reason is None
assert [host.ip for host in await search.get_shodan_hosts()] == ['203.0.113.10']
@pytest.mark.asyncio
async def test_shodan_discovery_retains_valid_services_and_reports_malformed_provider_data(self, monkeypatch):
from theHarvester.discovery import shodansearch
from theHarvester.lib.core import FetcherResponse
async def fetch_json(*_args, **_kwargs):
return FetcherResponse(
body={
'data': [
{'port': 53, 'transport': 'udp', 'product': {'raw': 'provider object'}},
{'transport': 'tcp'},
],
'hostnames': [],
},
status=200,
headers={},
)
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json)
patch_resolution(monkeypatch, shodansearch)
search = shodansearch.SearchShodan('example.test')
await search.process()
assert search.execution_status == 'partial'
assert search.stop_reason == 'provider-error'
assert search.error_type == 'InvalidResponseError'
assert (await search.get_shodan_hosts())[0].to_details() == {'services': [{'port': 53, 'transport': 'udp'}]}
@pytest.mark.asyncio
@pytest.mark.parametrize(
('status', 'expected_reason'),
[(401, 'access-denied'), (403, 'access-denied'), (429, 'rate-limited')],
)
async def test_shodan_discovery_reports_specific_provider_stop_reason(self, monkeypatch, status, expected_reason):
from theHarvester.discovery import shodansearch
from theHarvester.lib.core import FetcherResponse
async def fetch_json(*_args, **_kwargs):
return FetcherResponse(body=None, status=status, headers={})
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json)
patch_resolution(monkeypatch, shodansearch)
search = shodansearch.SearchShodan('example.test')
await search.process()
assert search.execution_status == 'failed'
assert search.stop_reason == expected_reason
def test_shodan_discovery_requires_a_configured_key(self, monkeypatch):
from theHarvester.discovery import shodansearch
from theHarvester.discovery.constants import MissingKey
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: None)
with pytest.raises(MissingKey):
shodansearch.SearchShodan('example.test')
def test_shodan_discovery_rejects_query_filter_injection(self, monkeypatch):
from theHarvester.discovery import shodansearch
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
with pytest.raises(ValueError, match='must be a hostname'):
shodansearch.SearchShodan('example.test ssl:true')
@pytest.mark.asyncio
async def test_shodan_discovery_reports_dns_failure_after_searching_provider(self, monkeypatch):
from theHarvester.discovery import shodansearch
from theHarvester.lib.core import FetcherResponse
provider_queries = []
async def fail_resolution(_target, *, family):
assert family == socket.AF_INET
raise shodansearch.aiodns.error.DNSError(shodansearch.aiodns.error.ARES_ENOTFOUND, 'not found')
async def fetch_json(url, *, params, **_kwargs):
assert url.endswith('/search')
provider_queries.append(params['query'])
return FetcherResponse(body={'matches': [], 'total': 0}, status=200, headers={})
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch, 'resolve_ip_addresses', fail_resolution)
monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json)
search = shodansearch.SearchShodan('example.test')
await search.process()
assert not await search.get_hostnames()
assert provider_queries == ['hostname:example.test', 'ssl:example.test']
assert search.execution_status == 'failed'
assert search.stop_reason == 'dns-resolution-failed'
@pytest.mark.asyncio
async def test_shodan_direct_request_cancellation_propagates(self, monkeypatch):
from theHarvester.discovery import shodansearch
entered = asyncio.Event()
release = asyncio.Event()
async def fetch_json(*_args, **_kwargs):
entered.set()
await release.wait()
raise AssertionError('cancelled request resumed')
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json)
patch_resolution(monkeypatch, shodansearch)
task = asyncio.create_task(shodansearch.SearchShodan('example.test').process())
await entered.wait()
task.cancel('operator-stop')
with pytest.raises(asyncio.CancelledError, match='operator-stop'):
await task
release.set()
@pytest.mark.asyncio
async def test_shodan_engine_processes_and_persists_sourced_hostnames(self, monkeypatch, capsys, tmp_path):
import theHarvester.__main__ as main_module
from theHarvester.lib.completed_result import ResultObservation, SourceExecution
from theHarvester.lib.database import ResultStore
from theHarvester.lib.shodan_evidence import ShodanHostObservation
# Make DNS resolution deterministic and offline.
monkeypatch.setattr(socket, 'gethostbyname', lambda _domain: '1.2.3.4', raising=True)
database = tmp_path / 'stash.sqlite'
monkeypatch.setattr(main_module, 'ResultStore', lambda: ResultStore(database), raising=True)
# Avoid filesystem/sqlite side effects.
class DummyResultStore:
async def initialize(self) -> None:
return None
async def record_observations(self, domain, all, res_type, source) -> None:
return None
monkeypatch.setattr(main_module, 'ResultStore', DummyResultStore, raising=True)
# Stub Shodan search to avoid network and API key requirements.
class DummySearchShodan:
async def search_ip(self, ip):
return OrderedDict({ip: {'hostnames': ['a.example.com', 'b.example.com']}})
def __init__(self, domain):
assert domain == 'example.com'
self.shodan_host = ShodanHostObservation.from_record(
'192.0.2.10',
{
'organization': 'Example Transit',
'hostnames': ['a.example.com', 'b.example.com'],
'services': [
{'port': 53, 'transport': 'udp'},
{'port': 443, 'transport': 'tcp', 'product': 'nginx'},
],
},
)
async def process(self, proxy=False):
return None
async def get_hostnames(self):
return {'a.example.com', 'b.example.com'}
async def get_shodan_hosts(self):
return (self.shodan_host,)
monkeypatch.setattr(main_module.shodansearch, 'SearchShodan', DummySearchShodan, raising=True)
# Run the CLI path that uses the engine queue/worker (`-b shodan`).
monkeypatch.setattr(sys, 'argv', ['theHarvester', '-d', 'example.com', '-b', 'shodan'], raising=True)
with pytest.raises(SystemExit) as excinfo:
@@ -127,54 +582,42 @@ class TestShodanEngine:
out = capsys.readouterr().out
assert 'An error occurred while processing a "work item"' not in out
output_tokens = set(out.split())
assert {'a.example.com', 'b.example.com'} <= output_tokens
assert {'a.example.com', 'b.example.com'} <= set(out.split())
assert '"type": "shodan-host"' in out
assert '"value": "192.0.2.10"' in out
assert '"transport": "udp"' in out
assert '"transport": "tcp"' in out
store = ResultStore(database)
try:
runs = await store.list_runs()
completed = await store.load_run(UUID(str(runs[0]['run_id'])))
assert len(completed.source_executions) == 1
execution = completed.source_executions[0]
assert execution == SourceExecution('shodan', 'completed', execution.duration_ms, 3)
assert completed.observations == (
ResultObservation('shodan', 'hostname', 'a.example.com'),
ResultObservation('shodan', 'hostname', 'b.example.com'),
ResultObservation('shodan', 'shodan-host', '192.0.2.10'),
)
assert completed.shodan_hosts == (DummySearchShodan('example.com').shodan_host,)
finally:
await store.dispose()
@pytest.mark.asyncio
async def test_shodan_internetdb_ignores_non_string_resolved_addresses(self, monkeypatch):
async def test_shodan_internetdb_uses_shared_resolution_and_normalizes_scope(self, monkeypatch):
from theHarvester.discovery import shodan_internetdb
monkeypatch.setattr(
shodan_internetdb.socket,
'getaddrinfo',
lambda *_args: [
(socket.AF_INET, socket.SOCK_STREAM, 0, '', ('203.0.113.1', 0)),
(socket.AF_INET, socket.SOCK_STREAM, 0, '', (12345, 0)),
],
raising=True,
)
targets = patch_resolution(monkeypatch, shodan_internetdb, ('203.0.113.1',))
async def fake_fetch_all(urls, json=False, proxy=False):
assert urls == ['https://internetdb.shodan.io/203.0.113.1']
return [{'ip': '203.0.113.1', 'hostnames': ['www.example.com']}]
monkeypatch.setattr(shodan_internetdb.AsyncFetcher, 'fetch_all', fake_fetch_all, raising=True)
search = shodan_internetdb.SearchShodanInternetDB('example.com')
await search.process()
assert await search.get_hostnames() == {'www.example.com'}
assert await search.get_ips() == {'203.0.113.1'}
@pytest.mark.asyncio
async def test_shodan_internetdb_normalizes_scoped_provider_evidence(self, monkeypatch):
from theHarvester.discovery import shodan_internetdb
monkeypatch.setattr(
shodan_internetdb.socket,
'getaddrinfo',
lambda *_args: [(socket.AF_INET, socket.SOCK_STREAM, 0, '', ('203.0.113.1', 0))],
raising=True,
)
async def fake_fetch_all(_urls, json=False, proxy=False):
return [
{
'ip': '203.0.113.1',
'hostnames': ['API.Example.TEST.', 'www.notexample.test', None],
'ports': [443],
},
{'ip': '198.51.100.2', 'ports': [80]},
}
]
monkeypatch.setattr(shodan_internetdb.AsyncFetcher, 'fetch_all', fake_fetch_all, raising=True)
@@ -182,6 +625,7 @@ class TestShodanEngine:
search = shodan_internetdb.SearchShodanInternetDB(' Example.TEST. ')
await search.process()
assert targets == [('example.test', socket.AF_UNSPEC)]
assert await search.get_hostnames() == {'api.example.test'}
assert await search.get_ips() == {'203.0.113.1'}
@@ -189,12 +633,7 @@ class TestShodanEngine:
async def test_shodan_internetdb_rejects_evidence_for_an_unrequested_ip(self, monkeypatch):
from theHarvester.discovery import shodan_internetdb
monkeypatch.setattr(
shodan_internetdb.socket,
'getaddrinfo',
lambda *_args: [(socket.AF_INET, socket.SOCK_STREAM, 0, '', ('203.0.113.1', 0))],
raising=True,
)
patch_resolution(monkeypatch, shodan_internetdb, ('203.0.113.1',))
async def fake_fetch_all(_urls, json=False, proxy=False):
return [
@@ -220,38 +659,17 @@ class TestShodanEngine:
assert not await search.get_tags()
assert not await search.get_cpes()
@pytest.mark.asyncio
async def test_shodan_internetdb_retains_a_matching_ip_with_empty_details(self, monkeypatch):
from theHarvester.discovery import shodan_internetdb
monkeypatch.setattr(
shodan_internetdb.socket,
'getaddrinfo',
lambda *_args: [(socket.AF_INET, socket.SOCK_STREAM, 0, '', ('203.0.113.1', 0))],
raising=True,
)
async def fake_fetch_all(_urls, json=False, proxy=False):
return [{'ip': '203.0.113.1', 'hostnames': [], 'ports': [], 'vulns': [], 'tags': [], 'cpes': []}]
monkeypatch.setattr(shodan_internetdb.AsyncFetcher, 'fetch_all', fake_fetch_all, raising=True)
search = shodan_internetdb.SearchShodanInternetDB('example.test')
await search.process()
assert await search.get_ips() == {'203.0.113.1'}
@pytest.mark.asyncio
async def test_shodan_internetdb_resolution_failure_skips_provider_request(self, monkeypatch):
from theHarvester.discovery import shodan_internetdb
def fail_resolution(*_args):
raise socket.gaierror
async def fail_resolution(_target):
raise shodan_internetdb.aiodns.error.DNSError(shodan_internetdb.aiodns.error.ARES_ENOTFOUND, 'not found')
async def fail_fetch(*_args, **_kwargs):
raise AssertionError('provider request must not run')
monkeypatch.setattr(shodan_internetdb.socket, 'getaddrinfo', fail_resolution, raising=True)
monkeypatch.setattr(shodan_internetdb, 'resolve_ip_addresses', fail_resolution)
monkeypatch.setattr(shodan_internetdb.AsyncFetcher, 'fetch_all', fail_fetch, raising=True)
search = shodan_internetdb.SearchShodanInternetDB('example.test')
@@ -264,12 +682,7 @@ class TestShodanEngine:
async def test_shodan_internetdb_provider_failure_completes_without_evidence(self, monkeypatch, caplog):
from theHarvester.discovery import shodan_internetdb
monkeypatch.setattr(
shodan_internetdb.socket,
'getaddrinfo',
lambda *_args: [(socket.AF_INET, socket.SOCK_STREAM, 0, '', ('203.0.113.1', 0))],
raising=True,
)
patch_resolution(monkeypatch, shodan_internetdb, ('203.0.113.1',))
async def fail_fetch(*_args, **_kwargs):
raise RuntimeError('provider-secret-payload')
tests/lib/test_api_v1.py
+106
View File
@@ -436,6 +436,7 @@ def test_openapi_explains_scope_and_execution_controls(tmp_path, monkeypatch) ->
'actions',
'scope',
'observations',
'details',
}
assert 'VirtualHostResult' not in schema['components']['schemas']
assert export_content['application/x-ndjson']['schema']['description'] == (
@@ -1198,6 +1199,111 @@ def test_api_database_upload_preserves_grouped_virtual_host_observations(tmp_pat
assert json.loads(exported.text.splitlines()[1]) == detail.json()['results'][0]
def test_api_import_preserves_structured_shodan_host_details(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
details = {
'organization': 'Example Transit',
'services': [
{'port': 53, 'transport': 'udp'},
{
'port': 443,
'transport': 'tcp',
'product': 'nginx',
'tls': {
'subject_cn': '*.example.test',
'subject_alt_names': ['api.example.test'],
'sha256': '0123456789abcdef',
},
},
],
}
payload = _jsonl_result(
finding_type='shodan-host',
value='192.0.2.10',
finding_fields={'actions': ['shodan'], 'details': details},
summary_fields={
'action_executions': [
{
'action': 'shodan',
'status': 'completed',
'duration_ms': 1,
'result_count': 1,
'error_type': None,
'stop_reason': None,
}
]
},
)
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
with TestClient(api.app) as client:
imported = client.post(
'/api/v1/runs/import',
params={'filename': 'shodan.jsonl'},
headers={'X-API-Key': 'test-key'},
content=payload,
)
assert imported.status_code == 201, imported.text
detail = client.get(f'/api/v1/runs/{imported.json()["run_id"]}', headers={'X-API-Key': 'test-key'})
assert detail.status_code == 200
assert detail.json()['results'] == [
{
'type': 'shodan-host',
'value': '192.0.2.10',
'sources': [],
'actions': ['shodan'],
'details': details,
}
]
def test_api_schema_exposes_typed_shodan_host_details() -> None:
from theHarvester.lib.api.run_models import NormalizedResult
schema = NormalizedResult.model_json_schema()
assert schema['properties']['details']['anyOf'][0] == {'$ref': '#/$defs/ShodanHostDetailsResponse'}
service = schema['$defs']['ShodanServiceResponse']
assert service['properties']['port']['minimum'] == 1
assert service['properties']['port']['maximum'] == 65535
assert service['properties']['transport']['enum'] == ['tcp', 'udp']
tls = schema['$defs']['ShodanTlsDetailsResponse']
assert tls['properties']['subject_alt_names']['items'] == {'type': 'string'}
def test_api_evidence_rejects_redundant_shodan_host_fields() -> None:
from fastapi import HTTPException
from theHarvester.lib.api.run_evidence import validate_evidence
evidence = {
'target': 'example.test',
'status': 'complete',
'results': [
{
'type': 'shodan-host',
'value': '192.0.2.10',
'sources': [],
'actions': ['shodan'],
'details': {
'ip': '192.0.2.10',
'services': [{'port': 443, 'transport': 'tcp'}],
},
}
],
'source_executions': [],
'action_executions': [],
'artifacts': [],
}
with pytest.raises(HTTPException, match='unsupported fields'):
validate_evidence(evidence)
def test_api_database_upload_rejects_vhost_evidence_outside_its_stored_target(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
from theHarvester.lib.api.run_evidence import parse_jsonl_import
tests/lib/test_completed_persistence.py
+56
View File
@@ -475,6 +475,62 @@ async def test_structured_network_evidence_round_trips_in_the_results_table(tmp_
assert json.loads(stored_details) == network_observation_details(network_observations)
@pytest.mark.asyncio
async def test_shodan_host_evidence_round_trips_without_a_json_string_value(tmp_path) -> None:
from theHarvester.lib.shodan_evidence import ShodanHostObservation
database = tmp_path / 'stash.sqlite'
store = ResultStore(database)
await store.initialize()
collected_at = datetime(2026, 8, 14, 12, 2, tzinfo=UTC)
shodan_host = ShodanHostObservation.from_record(
'192.0.2.10',
{
'asn': 'AS64496',
'organization': 'Example Transit',
'services': [
{'port': 53, 'transport': 'udp', 'product': 'dnsmasq'},
{'port': 443, 'transport': 'tcp', 'product': 'nginx'},
],
},
)
result = CompletedResult.finish(
target='example.com',
started_at=collected_at,
completed_at=collected_at,
groups={},
source_executions=(SourceExecution('shodan', 'completed', 1, 1),),
observations=(ResultObservation('shodan', 'shodan-host', '192.0.2.10'),),
active_evidence=ActiveEvidence(
executions=(
ActionExecution.finish(
action='shodan',
status='completed',
duration_ms=1,
groups={'shodan-host': ['192.0.2.10']},
),
)
),
shodan_hosts=(shodan_host,),
)
await store.save_run(result)
assert await store.load_run(result.run_id) == result
with sqlite3.connect(database) as db:
stored = db.execute(
'SELECT kind, value, details_json FROM results WHERE run_id = ?',
(str(result.run_id),),
).fetchone()
origins = db.execute(
'SELECT COUNT(*) FROM result_origins WHERE run_id = ?',
(str(result.run_id),),
).fetchone()[0]
assert stored[:2] == ('shodan-host', '192.0.2.10')
assert json.loads(stored[2]) == shodan_host.to_details()
assert origins == 2
@pytest.mark.asyncio
async def test_asn_organization_attribution_round_trips_in_a_normalized_table(tmp_path) -> None:
database = tmp_path / 'stash.sqlite'
tests/lib/test_completed_result.py
+119
View File
@@ -1040,6 +1040,125 @@ def test_jsonl_rejects_legacy_vhost_result_kind() -> None:
parse_result_jsonl(payload)
def test_shodan_host_jsonl_uses_the_ip_value_and_nonredundant_details() -> None:
from theHarvester.lib.shodan_evidence import ShodanHostObservation
completed_at = datetime(2026, 8, 14, 12, 2, tzinfo=UTC)
shodan_host = ShodanHostObservation.from_record(
'192.0.2.10',
{
'asn': 'AS64496',
'organization': 'Example Transit',
'isp': 'Example ISP',
'hostnames': ['api.example.test'],
'domains': ['example.test'],
'services': [
{
'port': 53,
'transport': 'udp',
'product': 'dnsmasq',
'version': '2.90',
'observed_at': '2026-08-14T11:58:00Z',
},
{
'port': 443,
'transport': 'tcp',
'product': 'nginx',
'version': '1.24.0',
'observed_at': '2026-08-14T12:01:00Z',
'http': {'title': 'Example', 'server': 'nginx', 'components': ['nginx']},
},
],
},
)
result = CompletedResult.finish(
target='example.com',
started_at=completed_at,
completed_at=completed_at,
groups={},
active_evidence=ActiveEvidence(
executions=(
ActionExecution.finish(
action='shodan',
status='completed',
duration_ms=1,
groups={'shodan-host': ['192.0.2.10']},
),
)
),
shodan_hosts=(shodan_host,),
)
records = [json.loads(line) for line in result.jsonl().splitlines()]
assert records[1] == {
'actions': ['shodan'],
'details': {
'asn': 'AS64496',
'domains': ['example.test'],
'hostnames': ['api.example.test'],
'isp': 'Example ISP',
'organization': 'Example Transit',
'services': [
{
'observed_at': '2026-08-14T11:58:00Z',
'port': 53,
'product': 'dnsmasq',
'transport': 'udp',
'version': '2.90',
},
{
'http': {'components': ['nginx'], 'server': 'nginx', 'title': 'Example'},
'observed_at': '2026-08-14T12:01:00Z',
'port': 443,
'product': 'nginx',
'transport': 'tcp',
'version': '1.24.0',
},
],
},
'sources': [],
'type': 'shodan-host',
'value': '192.0.2.10',
}
summary, findings = parse_result_jsonl(result.jsonl())
assert summary['result_count'] == 1
assert findings == records[1:]
def test_completed_result_rejects_conflicting_shodan_hosts_for_one_ip() -> None:
from theHarvester.lib.shodan_evidence import ShodanHostObservation
completed_at = datetime(2026, 8, 14, 12, 2, tzinfo=UTC)
first = ShodanHostObservation.from_record(
'192.0.2.10',
{'services': [{'port': 53, 'transport': 'udp'}]},
)
second = ShodanHostObservation.from_record(
'192.0.2.10',
{'services': [{'port': 443, 'transport': 'tcp'}]},
)
with pytest.raises(ValueError, match='conflicting evidence'):
CompletedResult.finish(
target='example.com',
started_at=completed_at,
completed_at=completed_at,
groups={},
active_evidence=ActiveEvidence(
executions=(
ActionExecution.finish(
action='shodan',
status='completed',
duration_ms=1,
groups={'shodan-host': ['192.0.2.10']},
),
)
),
shodan_hosts=(first, second),
)
def test_completed_result_rejects_artifact_without_a_real_subject_result() -> None:
completed_at = datetime(2026, 8, 5, 12, 1, tzinfo=UTC)
artifact = ArtifactReference(
tests/lib/test_harvestview_ui.py
+26
View File
@@ -49,6 +49,32 @@ def test_harvestview_assets_load_outside_the_repository_directory(tmp_path, monk
assert 'function renderResults' in response.text
def test_harvestview_has_an_operator_readable_shodan_host_route(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
with TestClient(api.app, base_url='http://127.0.0.1', client=('127.0.0.1', 50000)) as client:
script = client.get('/static/harvestview/app.js')
assert script.status_code == 200
assert "'shodan-host'" in script.text
assert "'Shodan hosts'" in script.text
assert 'function shodanNetworkFormatter' in script.text
assert 'function shodanServicesFormatter' in script.text
assert 'details.hostnames' in script.text
assert 'details.domains' in script.text
assert 'service.observed_at' in script.text
assert 'service.cpes' in script.text
assert 'http.components' in script.text
assert 'service.tls' in script.text
assert 'tls.subject_alt_names' in script.text
assert "title: 'Network'" in script.text
assert "title: 'Services'" in script.text
def test_harvestview_offers_jsonl_and_sqlite_imports_with_jsonl_export(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
tests/test_all_source_orchestration.py
+1 -10
View File
@@ -1,5 +1,4 @@
import json
import socket
import sys
import xml.etree.ElementTree as ElementTree
from collections import Counter
@@ -166,15 +165,7 @@ async def test_explicit_non_passive_source_is_scheduled_once(
return set()
if source == 'shodan':
class FakeShodan:
async def search_ip(self, _ip: str) -> dict:
nonlocal executions
executions += 1
return {}
monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', FakeShodan)
monkeypatch.setattr(socket, 'gethostbyname', lambda _domain: '203.0.113.1')
monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', lambda *_args: FakeAdapter())
else:
module, constructor_name = {
'criminalip': (theharvester_main.criminalip, 'SearchCriminalIP'),
tests/test_main.py
+140 -21
View File
@@ -37,7 +37,8 @@ async def test_cli_help_explains_proxy_and_direct_action_scope(
help_text = ' '.join(capsys.readouterr().out.split())
assert exit_info.value.code == 0
assert 'Use proxies.yaml for supported discovery-source and takeover requests.' in help_text
assert 'Use proxies.yaml for supported discovery-source, Shodan, and takeover requests.' in help_text
assert 'Query the Shodan Host API for discovered IPs, using configured proxies when enabled.' in help_text
assert (
'Enrich discovered IPs with sourced ASN attribution, or an explicitly targeted ASN, IP, or prefix, through '
'RouteViews.' in help_text
@@ -2091,6 +2092,8 @@ async def test_screenshot_capture_failure_cancels_sibling_tasks(
@pytest.mark.asyncio
async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
from theHarvester.lib.shodan_evidence import ShodanHostObservation
class FakeResultStore:
async def initialize(self) -> None:
return None
@@ -2167,8 +2170,10 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte
def __init__(self) -> None:
self.attributions: set[AsnAttributionObservation] = set()
self.hosts: dict[str, ShodanHostObservation] = {}
async def search_ip(self, ip: str) -> dict[str, dict[str, list[int]]]:
async def search_ip(self, ip: str, *, proxy: bool = False) -> dict[str, dict[str, object]]:
assert proxy is True
self.attributions.add(
AsnAttributionObservation(
'action',
@@ -2180,11 +2185,22 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte
datetime.now(UTC),
)
)
return {ip: {'ports': [443]}}
self.hosts[ip] = ShodanHostObservation.from_record(
ip,
{
'asn': 'AS64496',
'organization': 'Example Transit',
'services': [{'port': 443, 'transport': 'tcp', 'product': 'nginx'}],
},
)
return {ip: self.hosts[ip].to_details()}
async def get_asn_attributions(self) -> set[AsnAttributionObservation]:
return self.attributions
async def get_shodan_hosts(self) -> tuple[ShodanHostObservation, ...]:
return tuple(self.hosts.values())
class FakeApiScanner:
def __init__(self, word: str, wordlist: str, exact_paths: bool = False) -> None:
assert word == 'example.com'
@@ -2266,7 +2282,12 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte
assert isinstance(completed, CompletedResult)
assert ('url', 'https://example.com/api/v1') in completed.results
assert ('screenshot', 'api.example.com') not in completed.results
assert ('shodan', '{"ip":"192.0.2.10","result":{"ports":[443]}}') in completed.results
assert ('shodan-host', '192.0.2.10') in completed.results
assert completed.shodan_hosts[0].to_details() == {
'asn': 'AS64496',
'organization': 'Example Transit',
'services': [{'port': 443, 'transport': 'tcp', 'product': 'nginx'}],
}
assert ('asn', 'AS64496') in completed.results
assert completed.asn_attributions[0].organization_label == 'Example Transit'
takeover_result = (
@@ -2302,6 +2323,84 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte
}
@pytest.mark.asyncio
async def test_shodan_source_evidence_is_not_overwritten_by_conflicting_action_evidence(
monkeypatch: pytest.MonkeyPatch,
) -> None:
from theHarvester.lib.shodan_evidence import ShodanHostObservation
class ConflictingShodan:
error_type = None
def __init__(self, word: str | None = None) -> None:
self.word = word
self.host = (
ShodanHostObservation.from_record(
'192.0.2.10',
{'services': [{'port': 53, 'transport': 'udp'}]},
)
if word is not None
else None
)
async def process(self, proxy: bool = False) -> None:
return None
async def get_hostnames(self) -> set[str]:
return {'api.example.com'} if self.word is not None else set()
async def search_ip(self, ip: str, *, proxy: bool = False) -> dict[str, dict[str, object]]:
self.host = ShodanHostObservation.from_record(
ip,
{'services': [{'port': 443, 'transport': 'tcp'}]},
)
return {ip: self.host.to_details()}
async def get_shodan_hosts(self) -> tuple[ShodanHostObservation, ...]:
return (self.host,) if self.host is not None else ()
async def get_asn_attributions(self) -> set:
if self.word is not None:
return set()
return {
AsnAttributionObservation(
'action',
'shodan',
'AS64496',
'Example Transit',
'ip',
'192.0.2.10',
datetime.now(UTC),
)
}
monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore)
monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', _ApiHostChecker)
monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', ConflictingShodan)
result = await theharvester_main.start(
EnumerationOptions(
dns_resolve='192.0.2.53',
domain='example.com',
quiet=True,
shodan=True,
source='shodan',
),
return_completed_result=True,
)
completed = result[-1]
assert completed.shodan_hosts[0].to_details() == {'services': [{'port': 53, 'transport': 'udp'}]}
assert ResultObservation('shodan', 'shodan-host', '192.0.2.10') in completed.observations
execution = next(item for item in completed.active_evidence.executions if item.action == 'shodan')
assert execution.status == 'partial'
assert execution.error_type == 'ValueError'
assert {(observation.kind, observation.value) for observation in execution.observations} == {
('asn', 'AS64496'),
('ip', '192.0.2.10'),
}
@pytest.mark.asyncio
@pytest.mark.parametrize(
('request_count', 'request_errors', 'scan_error', 'expected_status', 'expected_error', 'expected_reason'),
@@ -2362,17 +2461,14 @@ async def test_shodan_action_records_all_target_errors_as_failed(
class FailedShodan:
error_type = None
async def search_ip(self, ip: str) -> dict[str, str]:
async def search_ip(self, ip: str, *, proxy: bool = False) -> dict[str, str]:
assert proxy is False
raise RuntimeError(f'provider-secret-payload for {ip}')
async def no_sleep(_seconds: float) -> None:
return None
monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore)
monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', _ApiHostSource)
monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', _ApiHostChecker)
monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', FailedShodan)
monkeypatch.setattr(theharvester_main.asyncio, 'sleep', no_sleep)
result = await theharvester_main.start(
EnumerationOptions(
@@ -2399,17 +2495,14 @@ async def test_shodan_no_data_is_a_completed_zero_yield_action(monkeypatch: pyte
class EmptyShodan:
error_type = None
async def search_ip(self, _ip: str) -> dict:
async def search_ip(self, _ip: str, *, proxy: bool = False) -> dict:
assert proxy is False
return {}
async def no_sleep(_seconds: float) -> None:
return None
monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore)
monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', _ApiHostSource)
monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', _ApiHostChecker)
monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', EmptyShodan)
monkeypatch.setattr(theharvester_main.asyncio, 'sleep', no_sleep)
result = await theharvester_main.start(
EnumerationOptions(dns_resolve='192.0.2.53', domain='example.com', quiet=True, shodan=True, source='crtsh'),
@@ -2983,6 +3076,8 @@ async def test_takeover_failure_persists_and_propagates(
@pytest.mark.asyncio
async def test_shodan_cancellation_persists_failure_and_propagates(monkeypatch: pytest.MonkeyPatch) -> None:
from theHarvester.lib.shodan_evidence import ShodanHostObservation
saved: list[CompletedResult] = []
class FakeChecker:
@@ -2999,17 +3094,28 @@ async def test_shodan_cancellation_persists_failure_and_propagates(monkeypatch:
class CancelledShodan:
error_type = None
async def search_ip(self, ip: str) -> dict:
return {ip: {'ports': [443]}}
def __init__(self) -> None:
self.calls = 0
self.hosts: dict[str, ShodanHostObservation] = {}
async def cancel_during_throttle(_seconds: float) -> None:
raise asyncio.CancelledError
async def search_ip(self, ip: str, *, proxy: bool = False) -> dict:
assert proxy is False
self.calls += 1
if self.calls == 2:
raise asyncio.CancelledError
self.hosts[ip] = ShodanHostObservation.from_record(
ip,
{'services': [{'port': 443, 'transport': 'tcp'}]},
)
return {ip: self.hosts[ip].to_details()}
async def get_shodan_hosts(self) -> tuple[ShodanHostObservation, ...]:
return tuple(self.hosts.values())
monkeypatch.setattr(theharvester_main, 'ResultStore', _recording_result_store(saved))
monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', _ApiHostSource)
monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', FakeChecker)
monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', CancelledShodan)
monkeypatch.setattr(theharvester_main.asyncio, 'sleep', cancel_during_throttle)
with pytest.raises(asyncio.CancelledError):
await theharvester_main.start(
@@ -3035,6 +3141,8 @@ async def test_shodan_cancellation_persists_failure_and_propagates(monkeypatch:
async def test_direct_action_checkpoint_cancellation_persists_and_propagates(
monkeypatch: pytest.MonkeyPatch, action: str
) -> None:
from theHarvester.lib.shodan_evidence import ShodanHostObservation
checkpoints: list[CompletedResult] = []
saved: list[CompletedResult] = []
@@ -3057,8 +3165,19 @@ async def test_direct_action_checkpoint_cancellation_persists_and_propagates(
class FakeShodan:
error_type = None
async def search_ip(self, ip: str) -> dict:
return {ip: {'ports': [443]}}
def __init__(self) -> None:
self.hosts: dict[str, ShodanHostObservation] = {}
async def search_ip(self, ip: str, *, proxy: bool = False) -> dict:
assert proxy is False
self.hosts[ip] = ShodanHostObservation.from_record(
ip,
{'services': [{'port': 443, 'transport': 'tcp'}]},
)
return {ip: self.hosts[ip].to_details()}
async def get_shodan_hosts(self) -> tuple[ShodanHostObservation, ...]:
return tuple(self.hosts.values())
async def cancel_after_action(result: CompletedResult) -> None:
if any(execution.action == action for execution in result.active_evidence.executions):
tests/test_readme.py
+6 -2
View File
@@ -155,13 +155,17 @@ def test_readme_preserves_project_social_attribution() -> None:
assert f'@{handle}' in readme
def test_readme_explains_jsonl_record_and_structured_value_parsing() -> None:
def test_readme_explains_jsonl_record_and_structured_evidence_parsing() -> None:
readme = Path('README.md').read_text()
assert '{"sources":[],"type":"hostname","value":"api.example.com"}' in readme
assert 'select(.type == "dns-recursive-finding") | .value | fromjson' in readme
assert 'JSONL is easy to stream one record at a time.' in readme
assert '`person`, `infostealer`, `shodan`, and `takeover`' in readme
assert '`person`, `infostealer`, and `takeover`' in readme
assert 'select(.type == "shodan-host") | {ip: .value, services: .details.services}' in readme
assert 'paginates both hostname and TLS-certificate searches' in readme
assert 'scoped certificate CNs and SANs' in readme
assert 'Raw banners, response bodies, certificate chains, and Shodan crawler metadata are not retained.' in readme
assert 'select(.type == "hostname" and .observations) | {hostname: .value, observations}' in readme
assert 'Several endpoint observations can enrich the same hostname' in readme
assert 'The summary preserves the evidence status, source and action outcomes' in readme
theHarvester/__main__.py
+61 -63
View File
@@ -118,6 +118,7 @@ from theHarvester.lib.recursive_dns import (
from theHarvester.lib.resolver_selection import DEFAULT_DNS_RESOLVERS, normalize_resolver_addresses
from theHarvester.lib.result_values import normalize_asn
from theHarvester.lib.routeviews import RouteViewsCancelled, RouteViewsResult, enrich_routeviews
from theHarvester.lib.shodan_evidence import ShodanHostObservation, canonical_shodan_hosts
from theHarvester.lib.source_catalog import (
SOURCE_SPECS,
ActivityClass,
@@ -230,7 +231,7 @@ async def start(
parser.add_argument(
'-p',
'--proxies',
help='Use proxies.yaml for supported discovery-source and takeover requests.',
help='Use proxies.yaml for supported discovery-source, Shodan, and takeover requests.',
default=False,
action='store_true',
)
@@ -243,7 +244,7 @@ async def start(
parser.add_argument(
'-s',
'--shodan',
help='Use Shodan to query discovered hosts.',
help='Query the Shodan Host API for discovered IPs, using configured proxies when enabled.',
default=False,
action='store_true',
)
@@ -578,7 +579,8 @@ async def start(
screenshot_artifacts: list[ArtifactReference] = []
screenshot_hostnames: set[str] = set()
screenshot_ip_addresses: set[str] = set()
shodan_evidence: list[str] = []
shodan_hosts: dict[str, ShodanHostObservation] = {}
shodan_action_hosts: set[str] = set()
takeover_results: dict[str, list[dict[str, str]]] = {}
linkedin_people_list_tracker = []
twitter_people_list_tracker = []
@@ -619,6 +621,26 @@ async def start(
)
displayed_asn_attributions.update(pending)
def record_shodan_host_observations(
host_observations: Iterable[ShodanHostObservation],
) -> tuple[ShodanHostObservation, ...]:
canonical_hosts = canonical_shodan_hosts(list(host_observations))
for host in canonical_hosts:
existing = shodan_hosts.get(host.ip)
if existing is not None and existing != host:
raise ValueError(f'Shodan host {host.ip} has conflicting evidence')
for host in canonical_hosts:
if host.ip not in shodan_hosts:
output_logger.info(
ujson.dumps(
{'type': 'shodan-host', 'value': host.ip, 'details': host.to_details()},
indent=4,
sort_keys=True,
)
)
shodan_hosts[host.ip] = host
return canonical_hosts
def finish_completed_result(
*,
extra_hostnames: Iterable[str] = (),
@@ -672,6 +694,7 @@ async def start(
network_observations=network_observations,
asn_attributions=asn_attributions,
virtual_hosts=vhost_observations if collect_hosts else (),
shodan_hosts=tuple(shodan_hosts.values()),
)
except (ValueError, TypeError) as error:
output_logger.info(f'[!] An error occurred while completing the result: {error}')
@@ -903,6 +926,9 @@ async def start(
'infostealer',
(json.dumps(stealer, ensure_ascii=False, separators=(',', ':'), sort_keys=True) for stealer in infostealers),
)
if source == 'shodan':
sourced_shodan_hosts = record_shodan_host_observations(await search_engine.get_shodan_hosts())
record_source_observations(source, 'shodan-host', (host.ip for host in sourced_shodan_hosts))
async def store(search_engine: Any, source: str) -> None:
source_spec = get_source_spec(source)
@@ -1632,41 +1658,7 @@ async def start(
elif engineitem == 'shodan':
try:
shodan_search = shodansearch.SearchShodan()
# For normal module usage, we need to create a wrapper that works with the store function
class ShodanWrapper:
def __init__(self, domain, shodan_client):
self.word = domain
self.hosts = set()
self.shodan = shodan_client
async def process(self, use_proxy: bool = False):
import socket
try:
# Resolve domain to IP and search in Shodan
ip = socket.gethostbyname(self.word)
output_logger.info(f'\tSearching Shodan for {ip}')
result = await self.shodan.search_ip(ip)
if ip in result and isinstance(result[ip], dict):
# Add the IP as a host for consistency with other modules
self.hosts.add(ip)
for host in result[ip].get('hostnames', []):
self.hosts.add(host)
output_logger.info(f'Found Shodan data for {ip}')
elif ip in result and isinstance(result[ip], str):
output_logger.info(f'{ip}: {result[ip]}')
except Exception as e:
output_logger.info(f'Error in Shodan search: {e}')
async def get_hostnames(self):
return list(self.hosts)
shodan_wrapper = ShodanWrapper(word, shodan_search)
stor_lst.append(store(shodan_wrapper, engineitem))
stor_lst.append(store(shodansearch.SearchShodan(word), engineitem))
except Exception as e:
if isinstance(e, MissingKey):
record_missing_credentials(engineitem)
@@ -2808,26 +2800,41 @@ async def start(
output_logger.info('[+] Note there may be leftover chrome processes you may have to kill manually\n')
# Shodan
shodanres = []
shodanres: list[dict[str, object]] = []
if shodan is True:
shodan_started = time.perf_counter()
shodan_error_types: set[str] = set()
shodan_asns: set[str] = set()
shodan_ips: set[str] = set()
def has_shodan_action_evidence() -> bool:
return bool(shodan_action_hosts or shodan_asns or shodan_ips)
output_logger.info('[*] Searching Shodan. ')
try:
for ip_index, ip in enumerate(host_ip):
shodan_search = None
if host_ip:
try:
shodan_search = shodansearch.SearchShodan()
except Exception as init_error:
shodan_error_types.add(type(init_error).__name__)
output_logger.info(f'[SHODAN-error] Error starting Shodan: {type(init_error).__name__}')
for ip in host_ip:
if shodan_search is None:
break
try:
output_logger.info('\tSearching for ' + ip)
shodan_search = shodansearch.SearchShodan()
shodandict = await shodan_search.search_ip(ip)
shodandict = await shodan_search.search_ip(ip, proxy=use_proxy)
get_asn_attributions = getattr(shodan_search, 'get_asn_attributions', None)
if get_asn_attributions is not None:
collected_attributions = await get_asn_attributions()
asn_attributions.extend(collected_attributions)
shodan_asns.update(attribution.asn for attribution in collected_attributions)
shodan_ips.update(attribution.subject_value for attribution in collected_attributions)
total_asns.extend(attribution.asn for attribution in collected_attributions)
current_attributions = {
attribution for attribution in collected_attributions if attribution.subject_value == ip
}
asn_attributions.extend(current_attributions)
shodan_asns.update(attribution.asn for attribution in current_attributions)
shodan_ips.update(attribution.subject_value for attribution in current_attributions)
total_asns.extend(attribution.asn for attribution in current_attributions)
if shodan_search.error_type:
shodan_error_types.add(shodan_search.error_type)
@@ -2838,21 +2845,12 @@ async def start(
# Process the results if it's a dictionary
if isinstance(shodan_result, dict):
rowdata = []
for _key, value in shodan_result.items():
if isinstance(value, int):
value = str(value)
if isinstance(value, list):
value = ', '.join(map(str, value))
rowdata.append(value)
shodanres.append(rowdata)
shodan_evidence.append(
json.dumps({'ip': ip, 'result': shodan_result}, separators=(',', ':'), sort_keys=True)
current_hosts = record_shodan_host_observations(
host for host in await shodan_search.get_shodan_hosts() if host.ip == ip
)
output_logger.info(ujson.dumps(shodan_result, indent=4, sort_keys=True))
shodan_action_hosts.update(host.ip for host in current_hosts)
shodanres.extend({'value': host.ip, 'details': host.to_details()} for host in current_hosts)
output_logger.info('\n')
if ip_index + 1 < len(host_ip):
await asyncio.sleep(5)
except Exception as ip_error:
shodan_error_types.add(type(ip_error).__name__)
output_logger.info(f'[SHODAN-error] Error searching {ip}: {type(ip_error).__name__}')
@@ -2861,9 +2859,9 @@ async def start(
action_executions.append(
ActionExecution.finish(
action='shodan',
status='partial' if shodan_evidence else 'failed',
status='partial' if has_shodan_action_evidence() else 'failed',
duration_ms=(time.perf_counter() - shodan_started) * 1000,
groups={'shodan': shodan_evidence, 'asn': shodan_asns, 'ip': shodan_ips},
groups={'shodan-host': shodan_action_hosts, 'asn': shodan_asns, 'ip': shodan_ips},
error_type='CancelledError',
stop_reason='cancelled',
)
@@ -2876,14 +2874,14 @@ async def start(
shodan_status = 'skipped'
shodan_stop_reason = 'no-input'
elif shodan_error_types:
shodan_status = 'partial' if shodan_evidence else 'failed'
shodan_status = 'partial' if has_shodan_action_evidence() else 'failed'
shodan_stop_reason = 'target-errors'
action_executions.append(
ActionExecution.finish(
action='shodan',
status=shodan_status,
duration_ms=(time.perf_counter() - shodan_started) * 1000,
groups={'shodan': shodan_evidence, 'asn': shodan_asns, 'ip': shodan_ips},
groups={'shodan-host': shodan_action_hosts, 'asn': shodan_asns, 'ip': shodan_ips},
error_type=next(iter(sorted(shodan_error_types)), None),
stop_reason=shodan_stop_reason,
)
theHarvester/discovery/shodan_internetdb.py
+6 -15
View File
@@ -1,9 +1,10 @@
import asyncio
import logging
import socket
from ipaddress import ip_address
import aiodns
from theHarvester.lib.core import AsyncFetcher
from theHarvester.lib.hostchecker import resolve_ip_addresses
from theHarvester.lib.hostnames import normalize_scoped_hostname
logger = logging.getLogger(__name__)
@@ -34,27 +35,17 @@ class SearchShodanInternetDB:
async def do_search(self) -> None:
# Resolve the domain to IP addresses first
try:
addr_infos = await asyncio.to_thread(socket.getaddrinfo, self.word, None, socket.AF_UNSPEC, socket.SOCK_STREAM)
except socket.gaierror:
resolved_ips = await resolve_ip_addresses(self.word)
except aiodns.error.DNSError:
logger.info(f'Shodan InternetDB: Could not resolve domain {self.word}')
return
# Deduplicate IPs from the resolution results
resolved_ips: set[str] = set()
for _family, _type, _proto, _canonname, sockaddr in addr_infos:
ip = sockaddr[0]
if isinstance(ip, str):
try:
resolved_ips.add(str(ip_address(ip)))
except ValueError:
continue
if not resolved_ips:
logger.info(f'Shodan InternetDB: No IPs resolved for {self.word}')
return
# Query InternetDB for each resolved IP
requested_ips = sorted(resolved_ips)
requested_ips = list(resolved_ips)
urls = [f'https://internetdb.shodan.io/{ip}' for ip in requested_ips]
try:
responses = await AsyncFetcher.fetch_all(urls, json=True, proxy=self.proxy)
theHarvester/discovery/shodansearch.py
+507 -121
View File
@@ -1,16 +1,29 @@
import asyncio
import logging
import re
import socket
from collections import OrderedDict
from datetime import UTC, datetime
from ipaddress import ip_address
from typing import cast
from shodan import Shodan, exception
import aiodns
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.asn_attribution import AsnAttributionObservation
from theHarvester.lib.core import Core
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse, ResponseStreamError
from theHarvester.lib.hostchecker import resolve_ip_addresses
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.shodan_evidence import ShodanHostObservation, canonical_shodan_hosts
logger = logging.getLogger(__name__)
HostResult = dict[str, object]
_CERTIFICATE_HOSTNAME = re.compile(
r'(?i)(?<![a-z0-9-])(?:\*\.)?(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+'
r'[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(?![a-z0-9.-])'
)
class SearchShodan:
"""Collect Shodan host data and retain ``asn`` + ``org`` attribution.
@@ -20,136 +33,509 @@ class SearchShodan:
``isp`` remains ordinary Shodan output and is not treated as equivalent.
"""
def __init__(self) -> None:
API_BASE_URL = 'https://api.shodan.io/shodan/host'
SEARCH_URL = f'{API_BASE_URL}/search'
SEARCH_FIELDS = ','.join(
(
'ip_str',
'asn',
'domains',
'hostnames',
'isp',
'org',
'port',
'transport',
'product',
'version',
'timestamp',
'cpe',
'http.title',
'http.server',
'http.components',
'ssl.jarm',
'ssl.cert.subject',
'ssl.cert.issuer',
'ssl.cert.expires',
'ssl.cert.fingerprint',
'ssl.cert.extensions',
)
)
# Preserve the official SDK's one-request-per-second pacing without retaining its blocking transport.
REQUEST_INTERVAL_SECONDS = 1.0
REQUEST_TIMEOUT_SECONDS: int | None = None
def __init__(self, word: str | None = None) -> None:
self.word = word.strip().lower().rstrip('.') if word is not None else None
if self.word is not None and (self.word.startswith('*.') or _CERTIFICATE_HOSTNAME.fullmatch(self.word) is None):
raise ValueError('Shodan discovery target must be a hostname')
self.scope = self.word.removeprefix('www.') if self.word is not None else None
self.key = Core.shodan_key()
if self.key is None:
raise MissingKey('Shodan')
self.api = Shodan(self.key)
self.hostdatarow: list = []
self.tracker: OrderedDict = OrderedDict()
self.tracker: OrderedDict[str, HostResult | str] = OrderedDict()
self.shodan_hosts: dict[str, ShodanHostObservation] = {}
self.error_type: str | None = None
self.asn_attributions: set[AsnAttributionObservation] = set()
self.totalhosts: set[str] = set()
self.execution_status: str | None = None
self.stop_reason: str | None = None
self._next_request_at = 0.0
async def search_ip(self, ip) -> OrderedDict:
async def _fetch_json(self, url: str, params: dict[str, object], proxy: bool) -> FetcherResponse:
loop = asyncio.get_running_loop()
delay = self._next_request_at - loop.time()
if delay > 0:
await asyncio.sleep(delay)
self._next_request_at = loop.time() + self.REQUEST_INTERVAL_SECONDS
return await AsyncFetcher.fetch_json(
url,
params=params,
proxy=proxy,
request_timeout=self.REQUEST_TIMEOUT_SECONDS,
)
async def _fetch_host(self, ip: str, proxy: bool) -> FetcherResponse:
return await self._fetch_json(f'{self.API_BASE_URL}/{ip}', {'key': self.key}, proxy)
def _certificate_name(self, value: object) -> str | None:
if not isinstance(value, str) or not (candidate := value.strip().casefold().rstrip('.')):
return None
if _CERTIFICATE_HOSTNAME.fullmatch(candidate) is None:
return None
wildcard = candidate.startswith('*.')
hostname = candidate.removeprefix('*.')
if self.scope is not None:
hostname = normalize_scoped_hostname(hostname, self.scope) or ''
if not hostname:
return None
else:
try:
ip_address(hostname)
except ValueError:
pass
else:
return None
return f'*.{hostname}' if wildcard else hostname
def _certificate_names(self, cert: object) -> tuple[set[str], bool]:
if not isinstance(cert, dict):
return set(), cert is not None
invalid_response = False
names: set[str] = set()
subject = cert.get('subject')
if subject is not None and not isinstance(subject, dict):
invalid_response = True
elif isinstance(subject, dict):
if normalized := self._certificate_name(subject.get('CN')):
names.add(normalized)
extensions = cert.get('extensions')
if extensions is not None and not isinstance(extensions, list):
invalid_response = True
elif isinstance(extensions, list):
for extension in extensions:
if not isinstance(extension, dict):
invalid_response = True
continue
if extension.get('name') != 'subjectAltName':
continue
data = extension.get('data')
if not isinstance(data, str):
invalid_response = True
continue
decoded = re.sub(r'\\x[0-9a-f]{2}', ' ', data, flags=re.IGNORECASE)
for match in _CERTIFICATE_HOSTNAME.finditer(decoded):
if normalized := self._certificate_name(match.group()):
names.add(normalized)
return names, invalid_response
def _scoped_banner_names(self, banner: dict[str, object]) -> set[str]:
assert self.scope is not None
names: set[str] = set()
for field in ('hostnames', 'domains'):
values = banner.get(field)
if isinstance(values, list):
for value in values:
if normalized := normalize_scoped_hostname(value, self.scope):
names.add(normalized)
ssl = banner.get('ssl')
if isinstance(ssl, dict):
cert_names, _invalid = self._certificate_names(ssl.get('cert'))
names.update(cert_names)
return names
def _merge_host(self, observation: ShodanHostObservation) -> bool:
existing = self.shodan_hosts.get(observation.ip)
if existing is None:
self.shodan_hosts[observation.ip] = observation
return False
invalid_response = False
scalars: dict[str, str | None] = {}
for field in ('asn', 'organization', 'isp'):
previous = getattr(existing, field)
current = getattr(observation, field)
if previous is not None and current is not None and previous != current:
invalid_response = True
scalars[field] = previous or current
self.shodan_hosts[observation.ip] = ShodanHostObservation.from_record(
observation.ip,
{
**scalars,
'hostnames': sorted(set(existing.hostnames) | set(observation.hostnames)),
'domains': sorted(set(existing.domains) | set(observation.domains)),
'services': [
service.to_record()
for service in sorted(
set(existing.services) | set(observation.services),
key=lambda service: service.sort_key(),
)
],
},
)
return invalid_response
def _record_host(self, ip: str, results: dict[str, object]) -> bool:
data = results.get('data')
if data == []:
logger.info(f'Shodan: No data found for IP {ip}')
return False
if not isinstance(data, list) or not data:
raise ValueError('invalid Shodan host response')
invalid_response = False
def normalized_string(value: object) -> str:
nonlocal invalid_response
if value is None:
return ''
if not isinstance(value, str):
invalid_response = True
return ''
normalized = value.strip()
return '' if normalized == 'None' else normalized
def normalized_strings(value: object) -> list[str]:
nonlocal invalid_response
if value is None:
return []
if not isinstance(value, list):
invalid_response = True
return []
if any(not isinstance(item, str) for item in value):
invalid_response = True
return sorted({normalized_string(item) for item in value if isinstance(item, str)} - {''})
services: list[dict[str, object]] = []
for banner in data:
if not isinstance(banner, dict):
invalid_response = True
continue
port = banner.get('port')
transport = normalized_string(banner.get('transport')).casefold()
if isinstance(port, bool) or not isinstance(port, int) or not 1 <= port <= 65535:
invalid_response = True
continue
if transport not in {'tcp', 'udp'}:
invalid_response = True
continue
service: dict[str, object] = {'port': port, 'transport': transport}
for provider_field, evidence_field in (
('product', 'product'),
('version', 'version'),
('timestamp', 'observed_at'),
):
if value := normalized_string(banner.get(provider_field)):
service[evidence_field] = value
if cpe_values := normalized_strings(banner.get('cpe')):
service['cpes'] = cpe_values
http = banner.get('http')
if isinstance(http, dict):
http_details: dict[str, object] = {}
for field in ('title', 'server'):
if value := normalized_string(http.get(field)):
http_details[field] = value
components = http.get('components')
if isinstance(components, dict) and (
component_values := sorted({normalized_string(component) for component in components} - {''})
):
http_details['components'] = component_values
elif components is not None and not isinstance(components, dict):
invalid_response = True
if http_details:
service['http'] = http_details
elif http is not None:
invalid_response = True
ssl = banner.get('ssl')
if isinstance(ssl, dict):
tls_details: dict[str, object] = {}
if jarm := normalized_string(ssl.get('jarm')):
tls_details['jarm'] = jarm
cert = ssl.get('cert')
if isinstance(cert, dict):
subject = cert.get('subject')
issuer = cert.get('issuer')
fingerprint = cert.get('fingerprint')
if subject is not None and not isinstance(subject, dict):
invalid_response = True
if issuer is not None and not isinstance(issuer, dict):
invalid_response = True
if fingerprint is not None and not isinstance(fingerprint, dict):
invalid_response = True
subject_cn = self._certificate_name(subject.get('CN')) if isinstance(subject, dict) else None
cert_names, invalid_cert = self._certificate_names(cert)
invalid_response = invalid_response or invalid_cert
if subject_cn:
tls_details['subject_cn'] = subject_cn
subject_alt_names = sorted(cert_names - ({subject_cn} if subject_cn else set()))
if subject_alt_names:
tls_details['subject_alt_names'] = subject_alt_names
for field, source in (
('issuer_cn', issuer.get('CN') if isinstance(issuer, dict) else None),
('expires_at', cert.get('expires')),
('sha256', fingerprint.get('sha256') if isinstance(fingerprint, dict) else None),
):
if value := normalized_string(source):
tls_details[field] = value
elif cert is not None:
invalid_response = True
if tls_details:
service['tls'] = tls_details
elif ssl is not None:
invalid_response = True
if service not in services:
services.append(service)
if not services:
raise ValueError('invalid Shodan host response')
services.sort(
key=lambda service: (
cast('int', service['port']),
str(service['transport']),
str(service.get('observed_at', '')),
)
)
domain_values = normalized_strings(results.get('domains'))
hostname_values = normalized_strings(results.get('hostnames'))
if self.scope is not None:
domain_values = sorted(
{normalized for value in domain_values if (normalized := normalize_scoped_hostname(value, self.scope))}
)
hostname_values = sorted(
{normalized for value in hostname_values if (normalized := normalize_scoped_hostname(value, self.scope))}
)
self.totalhosts.update(value for value in domain_values + hostname_values if value != self.scope)
for service in services:
tls = service.get('tls')
if not isinstance(tls, dict):
continue
for field in ('subject_cn', 'subject_alt_names'):
tls_value = tls.get(field)
values = tls_value if isinstance(tls_value, list) else [tls_value]
self.totalhosts.update(
name for name in values if isinstance(name, str) and not name.startswith('*.') and name != self.scope
)
asn = normalized_string(results.get('asn'))
organization = normalized_string(results.get('org'))
host_details = {
'asn': asn,
'domains': domain_values,
'hostnames': hostname_values,
'isp': normalized_string(results.get('isp')),
'organization': organization,
'services': services,
}
shodan_host = ShodanHostObservation.from_record(ip, host_details)
invalid_response = self._merge_host(shodan_host) or invalid_response
stored_host = self.shodan_hosts[ip]
self.tracker[ip] = stored_host.to_details()
if stored_host.asn and stored_host.organization:
self.asn_attributions.add(
AsnAttributionObservation(
'action',
'shodan',
stored_host.asn,
stored_host.organization,
'ip',
ip,
datetime.now(UTC),
)
)
return invalid_response
async def _search_target(self, proxy: bool) -> set[str]:
assert self.scope is not None
error_types: set[str] = set()
for query in (f'hostname:{self.scope}', f'ssl:{self.scope}'):
page = 1
received = 0
while True:
try:
response = await self._fetch_json(
self.SEARCH_URL,
{
'key': self.key,
'query': query,
'page': page,
'minify': 'false',
'fields': self.SEARCH_FIELDS,
},
proxy,
)
except ResponseStreamError as error:
error_types.add(
{
'invalid-response': 'InvalidResponseError',
'response-limit': 'ResponseLimitError',
'transport-error': 'TransportError',
}[error.reason]
)
break
except Exception as error:
error_types.add(type(error).__name__)
break
if not 200 <= response.status < 300:
error_types.add(f'HTTP{response.status}Error')
break
if not isinstance(response.body, dict):
error_types.add('InvalidResponseError')
break
matches = response.body.get('matches')
total = response.body.get('total')
if not isinstance(matches, list) or isinstance(total, bool) or not isinstance(total, int) or total < 0:
error_types.add('InvalidResponseError')
break
for match in matches:
if not isinstance(match, dict):
error_types.add('InvalidResponseError')
continue
if not self._scoped_banner_names(match):
continue
ip_value = match.get('ip_str')
if not isinstance(ip_value, str):
error_types.add('InvalidResponseError')
continue
try:
ip = str(ip_address(ip_value))
if self._record_host(ip, {**match, 'data': [match]}):
error_types.add('InvalidResponseError')
except (TypeError, ValueError):
error_types.add('InvalidResponseError')
received += len(matches)
if not matches:
if received < total:
error_types.add('InvalidResponseError')
break
if received >= total:
break
page += 1
return error_types
async def search_ip(self, ip: str, *, proxy: bool = False) -> OrderedDict[str, HostResult | str]:
self.error_type = None
try:
ipaddress = ip
results = self.api.host(ipaddress)
if not results or 'data' not in results or not results['data']:
logger.info(f'Shodan: No data found for IP {ip}')
return OrderedDict()
asn = ''
domains: list = list()
hostnames: list = list()
ip_str = ''
isp = ''
org = ''
ports: list = list()
title = ''
server = ''
product = ''
technologies: list = list()
data_first_dict = dict(results['data'][0])
if 'ip_str' in data_first_dict:
ip_str += data_first_dict['ip_str']
if 'http' in data_first_dict:
http_results_dict = dict(data_first_dict['http'])
if 'title' in http_results_dict:
title_val = str(http_results_dict['title']).strip()
if title_val != 'None':
title += title_val
if 'components' in http_results_dict:
for key in http_results_dict['components'].keys():
technologies.append(key)
if 'server' in http_results_dict:
server_val = str(http_results_dict['server']).strip()
if server_val != 'None':
server += server_val
for key, value in results.items():
if key == 'asn':
if isinstance(value, str):
asn += value
elif value is not None:
asn += str(value)
if key == 'domains':
if isinstance(value, list):
domain_values = [str(domain) for domain in value]
domain_values.sort()
domains.extend(domain_values)
if key == 'hostnames':
if isinstance(value, list):
hostname_values = [str(host).strip() for host in value]
hostname_values.sort()
hostnames.extend(hostname_values)
if key == 'isp':
if isinstance(value, str):
isp += value
elif value is not None:
isp += str(value)
if key == 'org':
org += str(value)
if key == 'ports':
if isinstance(value, list):
port_values = [int(port) for port in value if isinstance(port, int)]
port_values.sort()
ports.extend(port_values)
if key == 'product':
if isinstance(value, str):
product += value
elif isinstance(value, list):
product += ', '.join(str(item) for item in value if item is not None)
technologies = list(set(technologies))
self.tracker[ip] = {
'asn': asn.strip(),
'domains': domains,
'hostnames': hostnames,
'ip_str': ip_str.strip(),
'isp': isp.strip(),
'org': org.strip(),
'ports': ports,
'product': product.strip(),
'server': server.strip(),
'technologies': technologies,
'title': title.strip(),
}
organization_label = results.get('org')
if asn.strip() and isinstance(organization_label, str) and organization_label.strip():
try:
subject_ip = str(ip_address(str(ip).strip()))
self.asn_attributions.add(
AsnAttributionObservation(
'action',
'shodan',
asn,
organization_label,
'ip',
subject_ip,
datetime.now(UTC),
)
)
except ValueError:
logger.info('Shodan returned invalid ASN organization attribution')
return self.tracker
except exception.APIError as error:
if str(error).strip().rstrip('.').casefold() == 'no information available for that ip':
logger.info(f'{ip}: Not in Shodan')
self.tracker[ip] = 'Not in Shodan'
else:
self.error_type = type(error).__name__
self.tracker[ip] = 'Shodan request failed'
except Exception as e:
self.error_type = type(e).__name__
normalized_ip = str(ip_address(ip.strip()))
except ValueError:
self.error_type = 'ValueError'
self.tracker[ip] = 'Shodan request failed'
return self.tracker
try:
response = await self._fetch_host(normalized_ip, proxy)
if response.status == 404:
logger.info(f'{normalized_ip}: Not in Shodan')
self.tracker[normalized_ip] = 'Not in Shodan'
return self.tracker
if not 200 <= response.status < 300:
self.error_type = f'HTTP{response.status}Error'
self.tracker[normalized_ip] = 'Shodan request failed'
return self.tracker
if not isinstance(response.body, dict):
self.error_type = 'InvalidResponseError'
self.tracker[normalized_ip] = 'Shodan request failed'
return self.tracker
try:
if self._record_host(normalized_ip, response.body):
self.error_type = 'InvalidResponseError'
except (TypeError, ValueError):
self.error_type = 'InvalidResponseError'
self.tracker[normalized_ip] = 'Shodan request failed'
except ResponseStreamError as error:
self.error_type = {
'invalid-response': 'InvalidResponseError',
'response-limit': 'ResponseLimitError',
'transport-error': 'TransportError',
}[error.reason]
self.tracker[normalized_ip] = 'Shodan request failed'
except Exception as error:
self.error_type = type(error).__name__
self.tracker[normalized_ip] = 'Shodan request failed'
return self.tracker
async def get_asn_attributions(self) -> set[AsnAttributionObservation]:
return self.asn_attributions
async def get_hostnames(self) -> set[str]:
return self.totalhosts
async def get_shodan_hosts(self) -> tuple[ShodanHostObservation, ...]:
return canonical_shodan_hosts(list(self.shodan_hosts.values()))
async def process(self, proxy: bool = False) -> None:
if self.word is None:
raise ValueError('A discovery target is required')
assert self.scope is not None
self.totalhosts.clear()
self.execution_status = None
self.stop_reason = None
dns_stop_reason: str | None = None
try:
resolved_ips = await resolve_ip_addresses(self.word, family=socket.AF_INET)
if not resolved_ips:
raise ValueError('target has no IPv4 addresses')
except TimeoutError:
resolved_ips = ()
dns_stop_reason = 'dns-timeout'
except aiodns.error.DNSError as error:
resolved_ips = ()
dns_stop_reason = (
'dns-timeout' if error.args and error.args[0] == aiodns.error.ARES_ETIMEOUT else 'dns-resolution-failed'
)
except ValueError:
resolved_ips = ()
dns_stop_reason = 'dns-resolution-failed'
provider_error_types = await self._search_target(proxy)
for resolved_ip in resolved_ips:
await self.search_ip(resolved_ip, proxy=proxy)
if self.error_type is not None:
provider_error_types.add(self.error_type)
self.error_type = next(iter(sorted(provider_error_types)), None)
retained_evidence = bool(self.totalhosts or self.shodan_hosts)
if dns_stop_reason is not None:
self.execution_status = 'partial' if retained_evidence else 'failed'
self.stop_reason = dns_stop_reason
return
if provider_error_types:
self.execution_status = 'partial' if retained_evidence else 'failed'
if provider_error_types <= {'HTTP401Error', 'HTTP403Error'}:
self.stop_reason = 'access-denied'
elif provider_error_types == {'HTTP429Error'}:
self.stop_reason = 'rate-limited'
elif len(resolved_ips) == 1:
self.stop_reason = 'provider-error'
else:
self.stop_reason = 'provider-errors'
return
self.execution_status = 'completed'
self.stop_reason = None if retained_evidence else 'no-results'
theHarvester/lib/api/run_evidence.py
+36
View File
@@ -15,6 +15,7 @@ from theHarvester.lib.network_evidence import (
parse_network_observation_details,
)
from theHarvester.lib.result_values import normalize_prefix
from theHarvester.lib.shodan_evidence import ShodanHostObservation
from .run_models import _normalize_target
@@ -102,6 +103,41 @@ def validate_evidence(evidence: dict[str, Any]) -> dict[str, Any]:
status_code=status.HTTP_400_BAD_REQUEST,
detail='vhost is not a result type; use hostname with virtual-host observations',
)
if result.get('type') == 'shodan-host':
allowed_keys = {'type', 'value', 'sources', 'actions', 'details'}
if set(result) - allowed_keys:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail='Shodan host evidence contains unsupported fields',
)
sources = result.get('sources', [])
actions = result.get('actions', [])
if (
not isinstance(sources, list)
or any(not isinstance(source, str) or not source.strip() for source in sources)
or not isinstance(actions, list)
or any(not isinstance(action, str) or not action.strip() for action in actions)
):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail='Shodan host producers must be arrays of non-empty strings',
)
value = result.get('value')
if not isinstance(value, str):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail='Shodan host evidence must identify a canonical IP address',
)
try:
shodan_host = ShodanHostObservation.from_record(value, result.get('details'))
if shodan_host.ip != value or shodan_host.to_details() != result.get('details'):
raise ValueError('Shodan host evidence must use canonical structured details')
except ValueError as error:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(error)) from error
result['sources'] = sorted(set(sources))
result['actions'] = sorted(set(actions))
result['details'] = shodan_host.to_details()
continue
if result.get('type') == 'prefix':
allowed_keys = {'type', 'value', 'sources', 'actions', 'scope', 'observations'}
if set(result) - allowed_keys:
theHarvester/lib/api/run_models.py
+59 -1
View File
@@ -402,6 +402,49 @@ class AsnAttributionObservationResponse(BaseModel):
collected_at: str
class ShodanHttpDetailsResponse(BaseModel):
model_config = ConfigDict(extra='forbid')
title: str | None = None
server: str | None = None
components: list[str] = Field(default_factory=list)
class ShodanTlsDetailsResponse(BaseModel):
model_config = ConfigDict(extra='forbid')
subject_cn: str | None = None
subject_alt_names: list[str] = Field(default_factory=list)
issuer_cn: str | None = None
expires_at: str | None = None
sha256: str | None = None
jarm: str | None = None
class ShodanServiceResponse(BaseModel):
model_config = ConfigDict(extra='forbid')
port: int = Field(ge=1, le=65535)
transport: Literal['tcp', 'udp']
product: str | None = None
version: str | None = None
observed_at: str | None = None
cpes: list[str] = Field(default_factory=list)
http: ShodanHttpDetailsResponse | None = None
tls: ShodanTlsDetailsResponse | None = None
class ShodanHostDetailsResponse(BaseModel):
model_config = ConfigDict(extra='forbid')
asn: str | None = None
organization: str | None = None
isp: str | None = None
hostnames: list[str] = Field(default_factory=list)
domains: list[str] = Field(default_factory=list)
services: list[ShodanServiceResponse] = Field(min_length=1)
class NormalizedResult(BaseModel):
model_config = ConfigDict(extra='forbid')
@@ -410,6 +453,10 @@ class NormalizedResult(BaseModel):
sources: list[str] = Field(default_factory=list)
actions: list[str] = Field(default_factory=list)
scope: Literal['external-relationship'] | None = None
details: ShodanHostDetailsResponse | None = Field(
default=None,
description='Canonical host and service evidence for shodan-host results.',
)
observations: (
list[
VirtualHostObservationResponse
@@ -425,7 +472,16 @@ class NormalizedResult(BaseModel):
def validate_result(self) -> Self:
if self.type == 'vhost':
raise ValueError('vhost is not a result type; use hostname with virtual-host observations')
if self.type == 'prefix':
if self.type == 'shodan-host':
from theHarvester.lib.shodan_evidence import ShodanHostObservation
if self.scope is not None or self.observations is not None or self.details is None:
raise ValueError('Shodan host results require details without scope or observations')
shodan_details = self.details.model_dump(exclude_none=True, exclude_defaults=True)
shodan_host = ShodanHostObservation.from_record(self.value, shodan_details)
if shodan_host.ip != self.value or shodan_host.to_details() != shodan_details:
raise ValueError('Shodan host results must use canonical structured details')
elif self.type == 'prefix':
if self.scope != 'external-relationship':
raise ValueError('Prefix results must have external-relationship scope')
if self.observations is not None:
@@ -444,6 +500,8 @@ class NormalizedResult(BaseModel):
parse_virtual_host_details(self.value, details)
else:
raise ValueError('Structured observations belong to ASN, hostname, or prefix results')
elif self.details is not None:
raise ValueError('Structured details belong to Shodan host results')
return self
theHarvester/lib/api/run_projection.py
+2
View File
@@ -42,6 +42,8 @@ def normalized_results(evidence: dict[str, Any] | None) -> list[dict[str, Any]]:
result['observations'] = [
dict(observation) for observation in item.get('observations', []) if isinstance(observation, dict)
]
if item.get('type') == 'shodan-host' and isinstance(item.get('details'), dict):
result['details'] = dict(item['details'])
if item.get('type') == 'prefix':
result['scope'] = str(item.get('scope', ''))
results.append(result)
theHarvester/lib/api/run_store.py
+5
View File
@@ -19,6 +19,7 @@ from theHarvester.lib.completed_result import (
from theHarvester.lib.database import DuplicateRunError, ResultStore, ResultStoreError, RunLifecycleStore
from theHarvester.lib.evidence_types import EXECUTION_STATUSES, EvidenceStatus, ExecutionStatus, ResultKind
from theHarvester.lib.network_evidence import NetworkObservation, parse_network_observation_details
from theHarvester.lib.shodan_evidence import ShodanHostObservation
from .run_artifacts import RunPaths, read_child_evidence
from .run_models import RunRequest, _normalize_target, utc_now
@@ -55,6 +56,7 @@ def _completed_result(
virtual_hosts: list[VirtualHostObservation] = []
network_observations: list[NetworkObservation] = []
asn_attributions: list[AsnAttributionObservation] = []
shodan_hosts: list[ShodanHostObservation] = []
for item in results:
kind = cast('ResultKind', str(item['type']))
value = str(item['value'])
@@ -65,6 +67,8 @@ def _completed_result(
network_observations.extend(parse_network_observation_details(value, item.get('observations')))
elif kind == 'asn' and item.get('observations'):
asn_attributions.extend(parse_asn_attribution_details(value, item.get('observations')))
elif kind == 'shodan-host':
shodan_hosts.append(ShodanHostObservation.from_record(value, item.get('details')))
for source in set(item.get('sources', [])):
source_name = str(source)
source_origins.add(ResultObservation(source_name, kind, value))
@@ -152,6 +156,7 @@ def _completed_result(
virtual_hosts=virtual_hosts,
network_observations=network_observations,
asn_attributions=asn_attributions,
shodan_hosts=shodan_hosts,
evidence_status=(
cast('EvidenceStatus', str(evidence['status']))
if evidence.get('status') is not None and not execution_status_is_authoritative
theHarvester/lib/api/static/harvestview/app.js
+57 -4
View File
@@ -4,12 +4,12 @@
const $ = selector => document.querySelector(selector);
const $$ = selector => [...document.querySelectorAll(selector)];
const ROUTE_ORDER = [
'hostname', 'ip', 'prefix', 'asn', 'email', 'url', 'person', 'person-link', 'takeover', 'shodan',
'hostname', 'ip', 'prefix', 'asn', 'shodan-host', 'email', 'url', 'person', 'person-link', 'takeover',
'scope-extension', 'external-relationship', 'other'
];
const ROUTE_LABELS = {
hostname: 'Hostnames', ip: 'IP addresses', prefix: 'Network prefixes', asn: 'ASNs', email: 'Emails', url: 'URLs',
person: 'People', 'person-link': 'People links', takeover: 'Takeover evidence', shodan: 'Shodan evidence',
person: 'People', 'person-link': 'People links', takeover: 'Takeover evidence', 'shodan-host': 'Shodan hosts',
'scope-extension': 'Scope extensions', 'external-relationship': 'External relationships', other: 'Other'
};
const ACTION_FIELDS = {'dns-recursive': 'dns_recursive_depth'};
@@ -470,6 +470,47 @@
return text.includes(query);
}
function shodanNetworkFormatter(cell) {
const details = cell.getValue() || {};
const network = [details.organization, details.asn, details.isp].filter(Boolean).join(' · ') || 'Not recorded';
const names = [
details.hostnames?.length ? `Hosts: ${details.hostnames.join(', ')}` : '',
details.domains?.length ? `Domains: ${details.domains.join(', ')}` : '',
].filter(Boolean);
return `<div class="vhost-observations"><span>${escapeHtml(network)}</span>${names.map(value => `<span>${escapeHtml(value)}</span>`).join('')}</div>`;
}
function shodanServicesFormatter(cell) {
const services = Array.isArray(cell.getValue()?.services) ? cell.getValue().services : [];
if (!services.length) return 'No service evidence';
return `<div class="vhost-observations">${services.map(service => {
const identity = `${service.port}/${service.transport}`;
const product = [service.product, service.version].filter(Boolean).join(' ');
const http = service.http || {};
const tls = service.tls || {};
const description = [
product,
service.observed_at ? `Seen ${service.observed_at}` : '',
http.title,
http.server,
http.components?.length ? `HTTP: ${http.components.join(', ')}` : '',
service.cpes?.length ? `CPE: ${service.cpes.join(', ')}` : '',
tls.subject_cn ? `TLS subject: ${tls.subject_cn}` : '',
tls.subject_alt_names?.length ? `TLS SANs: ${tls.subject_alt_names.join(', ')}` : '',
tls.issuer_cn ? `issuer: ${tls.issuer_cn}` : '',
tls.expires_at ? `expires: ${tls.expires_at}` : '',
tls.sha256 ? `SHA-256: ${tls.sha256}` : '',
tls.jarm ? `JARM: ${tls.jarm}` : '',
].filter(Boolean).join(' · ');
return `<span>${escapeHtml(`${identity}${description ? ` · ${description}` : ''}`)}</span>`;
}).join('')}</div>`;
}
function shodanDetailsFilter(headerValue, rowValue) {
const query = String(headerValue || '').trim().toLowerCase();
return JSON.stringify(rowValue || {}).toLowerCase().includes(query);
}
function provenanceFormatter(cell) {
const values = Array.isArray(cell.getValue()) ? cell.getValue() : [];
return escapeHtml(values.join(', ') || '-');
@@ -480,9 +521,13 @@
nodes.copySelected.disabled = true;
nodes.copySelected.textContent = 'Copy selected';
const columns = [
{title: 'Value', field: 'value', formatter: cell => `<span class="value-cell">${escapeHtml(cell.getValue())}</span>`, minWidth: 260, widthGrow: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter, headerFilterPlaceholder: 'Filter values'},
{title: 'DNS', field: 'dns_status', formatter: dnsFormatter, width: 130, responsive: 1, headerFilter: 'input', headerFilterFunc: columnTextFilter, headerFilterPlaceholder: 'Filter DNS'},
{title: state.route === 'shodan-host' ? 'IP' : 'Value', field: 'value', formatter: cell => `<span class="value-cell">${escapeHtml(cell.getValue())}</span>`, minWidth: 260, widthGrow: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter, headerFilterPlaceholder: 'Filter values'},
];
if (state.route !== 'shodan-host') {
columns.push(
{title: 'DNS', field: 'dns_status', formatter: dnsFormatter, width: 130, responsive: 1, headerFilter: 'input', headerFilterFunc: columnTextFilter, headerFilterPlaceholder: 'Filter DNS'},
);
}
if (state.route === 'hostname' && rows.some(row => Array.isArray(row.observations) && row.observations.length)) {
columns.push(
{title: 'Virtual-host observations', field: 'observations', formatter: vhostObservationsFormatter, minWidth: 420, widthGrow: 4, variableHeight: true, headerFilter: 'input', headerFilterFunc: vhostObservationsFilter, headerFilterPlaceholder: 'Filter endpoint evidence'},
@@ -503,6 +548,14 @@
{title: 'Produced by', field: 'actions', formatter: provenanceFormatter, minWidth: 130, responsive: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter},
);
}
if (state.route === 'shodan-host') {
columns.push(
{title: 'Network', field: 'details', formatter: shodanNetworkFormatter, minWidth: 220, widthGrow: 2, headerFilter: 'input', headerFilterFunc: shodanDetailsFilter},
{title: 'Services', field: 'details', formatter: shodanServicesFormatter, minWidth: 360, widthGrow: 4, variableHeight: true, headerFilter: 'input', headerFilterFunc: shodanDetailsFilter},
{title: 'Sources', field: 'sources', formatter: provenanceFormatter, minWidth: 130, responsive: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter},
{title: 'Produced by', field: 'actions', formatter: provenanceFormatter, minWidth: 130, responsive: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter},
);
}
if (state.route === 'hostname') {
columns.push({
title: 'Actions', field: 'value', formatter: resultActionFormatter, headerSort: false,
theHarvester/lib/completed_result.py
+37 -4
View File
@@ -32,6 +32,7 @@ from theHarvester.lib.network_evidence import (
parse_network_observation_details,
)
from theHarvester.lib.result_values import normalize_result_value
from theHarvester.lib.shodan_evidence import ShodanHostObservation, canonical_shodan_hosts
from theHarvester.lib.virtual_host import VirtualHostObservation, normalize_virtual_host_hostname
@@ -88,6 +89,8 @@ def parse_result_jsonl(payload: bytes | str) -> tuple[dict[str, object], list[di
allowed_keys = {'type', 'value', 'sources', 'actions'}
if result_kind in {'asn', 'hostname', 'prefix'} and 'observations' in record:
allowed_keys.add('observations')
if result_kind == 'shodan-host' and 'details' in record:
allowed_keys.add('details')
if result_kind == 'prefix':
allowed_keys.add('scope')
if (
@@ -107,7 +110,7 @@ def parse_result_jsonl(payload: bytes | str) -> tuple[dict[str, object], list[di
if result_kind == 'prefix' and normalized_result_value != result_value:
raise ValueError('prefix result is not canonical')
except ValueError as error:
label = 'ASN' if result_kind == 'asn' else 'prefix'
label = {'asn': 'ASN', 'prefix': 'prefix', 'shodan-host': 'Shodan host'}.get(str(result_kind), 'result')
raise ValueError(f'JSONL findings must use a canonical {label} value') from error
record['value'] = normalized_result_value
if result_kind == 'prefix' and record.get('scope') != 'external-relationship':
@@ -132,6 +135,14 @@ def parse_result_jsonl(payload: bytes | str) -> tuple[dict[str, object], list[di
except ValueError as error:
raise ValueError(f'JSONL ASN has invalid organization attributions: {error}') from error
record['observations'] = asn_attribution_details(asn_attributions)
elif result_kind == 'shodan-host':
try:
shodan_host = ShodanHostObservation.from_record(record['value'], record.get('details'))
except ValueError as error:
raise ValueError(f'JSONL Shodan host has invalid details: {error}') from error
if shodan_host.ip != record['value'] or shodan_host.to_details() != record.get('details'):
raise ValueError('JSONL Shodan host must use canonical structured details')
record['details'] = shodan_host.to_details()
return summary, findings
@@ -215,6 +226,7 @@ class CompletedResult:
virtual_hosts: tuple[VirtualHostObservation, ...] = ()
network_observations: tuple[NetworkObservation, ...] = ()
asn_attributions: tuple[AsnAttributionObservation, ...] = ()
shodan_hosts: tuple[ShodanHostObservation, ...] = ()
evidence_status: EvidenceStatus | None = None
def __post_init__(self) -> None:
@@ -321,13 +333,26 @@ class CompletedResult:
not in origin_observations
):
raise ValueError('BGP route and RPKI observations require matching observed-origin evidence')
sorted_asn_attributions = canonical_asn_attributions(list(self.asn_attributions))
if self.asn_attributions != sorted_asn_attributions:
raise ValueError('ASN attributions must be deduplicated and sorted')
sorted_shodan_hosts = canonical_shodan_hosts(list(self.shodan_hosts))
if self.shodan_hosts != sorted_shodan_hosts:
raise ValueError('Shodan host observations must be deduplicated and sorted')
structured_shodan_results = {('shodan-host', observation.ip) for observation in self.shodan_hosts}
if structured_shodan_results != {result for result in result_set if result[0] == 'shodan-host'}:
raise ValueError('Shodan host results must contain canonical structured evidence')
source_results = {(observation.source, observation.kind, observation.value) for observation in self.observations}
action_results = {
(action, observation.kind, observation.value) for action, observation in self.active_evidence.observations
}
for shodan_observation in self.shodan_hosts:
if ('shodan', 'shodan-host', shodan_observation.ip) not in source_results and (
'shodan',
'shodan-host',
shodan_observation.ip,
) not in action_results:
raise ValueError('Shodan host evidence must reference Shodan source or action provenance')
sorted_asn_attributions = canonical_asn_attributions(list(self.asn_attributions))
if self.asn_attributions != sorted_asn_attributions:
raise ValueError('ASN attributions must be deduplicated and sorted')
for attribution in self.asn_attributions:
if attribution.collected_at < self.started_at or attribution.collected_at > self.completed_at:
raise ValueError('ASN attribution collection time must fall within the completed run')
@@ -365,6 +390,7 @@ class CompletedResult:
virtual_hosts: Iterable[VirtualHostObservation] = (),
network_observations: Iterable[NetworkObservation] = (),
asn_attributions: Iterable[AsnAttributionObservation] = (),
shodan_hosts: Iterable[ShodanHostObservation] = (),
evidence_status: EvidenceStatus | None = None,
) -> Self:
completed_active_evidence = active_evidence if active_evidence is not None else ActiveEvidence()
@@ -382,6 +408,9 @@ class CompletedResult:
completed_virtual_hosts = tuple(sorted(set(virtual_hosts), key=VirtualHostObservation.sort_key))
for virtual_host in completed_virtual_hosts:
results.add(('hostname', virtual_host.hostname))
completed_shodan_hosts = canonical_shodan_hosts(list(shodan_hosts))
for shodan_host in completed_shodan_hosts:
results.add(('shodan-host', shodan_host.ip))
return cls(
run_id=run_id or uuid4(),
target=target.strip(),
@@ -394,6 +423,7 @@ class CompletedResult:
virtual_hosts=completed_virtual_hosts,
network_observations=canonical_network_observations(network_observations),
asn_attributions=canonical_asn_attributions(list(asn_attributions)),
shodan_hosts=completed_shodan_hosts,
evidence_status=evidence_status,
)
@@ -459,6 +489,7 @@ class CompletedResult:
attribution_by_asn: dict[str, list[AsnAttributionObservation]] = {}
for attribution in self.asn_attributions:
attribution_by_asn.setdefault(attribution.asn, []).append(attribution)
shodan_by_ip = {observation.ip: observation for observation in self.shodan_hosts}
records: list[dict[str, object]] = []
for kind, value in self.results:
record: dict[str, object] = {
@@ -476,5 +507,7 @@ class CompletedResult:
record['observations'] = network_observation_details(tuple(network_observations))
elif kind == 'asn' and (asn_attributions := attribution_by_asn.get(value)):
record['observations'] = asn_attribution_details(tuple(asn_attributions))
elif kind == 'shodan-host' and (shodan_host := shodan_by_ip.get(value)):
record['details'] = shodan_host.to_details()
records.append(record)
return records
theHarvester/lib/core.py
+2 -2
View File
@@ -774,7 +774,7 @@ class AsyncFetcher:
proxy: str | bool | None = '',
headers: dict[str, str] | None = None,
follow_redirects: bool = False,
request_timeout: int = 60,
request_timeout: int | None = 60,
) -> AsyncIterator[aiohttp.ClientResponse]:
try:
ssl_arg = cls._ssl_context()
@@ -814,7 +814,7 @@ class AsyncFetcher:
params: Sized = '',
proxy: str | bool | None = '',
headers: dict[str, str] | None = None,
request_timeout: int = 60,
request_timeout: int | None = 60,
) -> FetcherResponse:
"""Fetch one bounded JSON response without following redirects."""
async with cls._open_get_response(
theHarvester/lib/database.py
+23
View File
@@ -61,6 +61,7 @@ from theHarvester.lib.network_evidence import (
network_observation_sort_key,
parse_network_observation_json,
)
from theHarvester.lib.shodan_evidence import ShodanHostObservation, canonical_shodan_hosts
from theHarvester.lib.virtual_host import VirtualHostObservation
if TYPE_CHECKING:
@@ -752,6 +753,7 @@ class ResultStore:
network_by_prefix: dict[str, list[NetworkObservation]] = {}
for network_observation in result.network_observations:
network_by_prefix.setdefault(network_observation.prefix, []).append(network_observation)
shodan_by_ip = {observation.ip: observation for observation in result.shodan_hosts}
async with self._session() as session:
try:
session.add(
@@ -785,6 +787,13 @@ class ResultStore:
sort_keys=True,
)
if kind == 'prefix' and value in network_by_prefix
else json.dumps(
shodan_by_ip[value].to_details(),
ensure_ascii=False,
separators=(',', ':'),
sort_keys=True,
)
if kind == 'shodan-host' and value in shodan_by_ip
else None
),
)
@@ -901,6 +910,7 @@ class ResultStore:
}
virtual_hosts: list[VirtualHostObservation] = []
network_observations: list[NetworkObservation] = []
shodan_hosts: list[ShodanHostObservation] = []
for result_row in rows:
has_vhost_provenance = result_row.position in vhost_result_positions
if has_vhost_provenance:
@@ -915,6 +925,18 @@ class ResultStore:
raise ResultStoreError(f'Persisted virtual-host details are not canonical: {result_row.value}')
virtual_hosts.extend(parsed_virtual_hosts)
continue
if result_row.kind == 'shodan-host':
if result_row.details_json is None:
raise ResultStoreError(f'Persisted Shodan host details are missing: {result_row.value}')
try:
details = json.loads(result_row.details_json)
shodan_host = ShodanHostObservation.from_record(result_row.value, details)
except (json.JSONDecodeError, ValueError) as error:
raise ResultStoreError(f'Persisted Shodan host details are invalid: {result_row.value}') from error
if shodan_host.to_details() != details:
raise ResultStoreError(f'Persisted Shodan host details are not canonical: {result_row.value}')
shodan_hosts.append(shodan_host)
continue
if result_row.details_json is None:
continue
if result_row.kind == 'prefix':
@@ -1042,6 +1064,7 @@ class ResultStore:
virtual_hosts=tuple(sorted(set(virtual_hosts), key=VirtualHostObservation.sort_key)),
network_observations=tuple(sorted(set(network_observations), key=network_observation_sort_key)),
asn_attributions=canonical_attributions,
shodan_hosts=canonical_shodan_hosts(shodan_hosts),
evidence_status=cast('EvidenceStatus', parent.evidence_status) if parent.evidence_status is not None else None,
)
theHarvester/lib/evidence_types.py
+1 -1
View File
@@ -20,7 +20,7 @@ ResultKind = Literal[
'prefix',
'server',
'screenshot',
'shodan',
'shodan-host',
'takeover',
'twitter-person',
'url',
theHarvester/lib/hostchecker.py
+31
View File
@@ -7,7 +7,9 @@ Revised to use aiodns & asyncio on 2019-09-23
from __future__ import annotations
import asyncio
import inspect
import ipaddress
import socket
from dataclasses import dataclass
from typing import TYPE_CHECKING
@@ -36,6 +38,35 @@ def is_expected_dns_absence(error: BaseException) -> bool:
)
async def resolve_ip_addresses(hostname: str, *, family: socket.AddressFamily = socket.AF_UNSPEC) -> tuple[str, ...]:
"""Resolve every unique IP address for a hostname."""
resolver = aiodns.DNSResolver()
try:
answer = await resolver.getaddrinfo(hostname, family=family)
finally:
close = getattr(resolver, 'close', None)
if close is not None:
close_result = close()
if inspect.isawaitable(close_result):
await close_result
addresses: set[str] = set()
for node in answer.nodes:
try:
value = node.addr[0]
if isinstance(value, bytes):
value = value.decode('ascii')
address = ipaddress.ip_address(value)
except (AttributeError, IndexError, TypeError, UnicodeDecodeError, ValueError):
continue
if family == socket.AF_INET and address.version != 4:
continue
if family == socket.AF_INET6 and address.version != 6:
continue
addresses.add(str(address))
return tuple(sorted(addresses, key=lambda value: (ipaddress.ip_address(value).version, int(ipaddress.ip_address(value)))))
class Checker:
"""Resolve hosts while preserving the legacy ``check()`` return tuple.
theHarvester/lib/result_values.py
+5 -1
View File
@@ -1,6 +1,6 @@
from __future__ import annotations
from ipaddress import ip_network
from ipaddress import ip_address, ip_network
from typing import TYPE_CHECKING
if TYPE_CHECKING:
@@ -40,4 +40,8 @@ def normalize_result_value(kind: ResultKind | str, value: str) -> str:
return normalize_asn(normalized)
if kind == 'prefix':
return normalize_prefix(normalized)
if kind == 'shodan-host':
if '%' in normalized:
raise ValueError('Shodan host must not contain an IPv6 scope identifier')
return str(ip_address(normalized))
return normalized
theHarvester/lib/shodan_evidence.py
+221
View File
@@ -0,0 +1,221 @@
from __future__ import annotations
from dataclasses import dataclass
from ipaddress import ip_address
from typing import Self
from theHarvester.lib.result_values import normalize_asn
def _optional_text(value: object, field: str) -> str | None:
if value is None:
return None
if not isinstance(value, str):
raise ValueError(f'Shodan {field} must be a string')
return value.strip() or None
def _string_tuple(value: object, field: str) -> tuple[str, ...]:
if value is None:
return ()
if not isinstance(value, list) or any(not isinstance(item, str) for item in value):
raise ValueError(f'Shodan {field} must be an array of strings')
return tuple(sorted({item.strip() for item in value if item.strip()}))
def _hostname_tuple(value: object, field: str) -> tuple[str, ...]:
return tuple(sorted({item.casefold().rstrip('.') for item in _string_tuple(value, field)} - {''}))
@dataclass(frozen=True, slots=True)
class ShodanServiceObservation:
port: int
transport: str
product: str | None = None
version: str | None = None
observed_at: str | None = None
cpes: tuple[str, ...] = ()
http_title: str | None = None
http_server: str | None = None
http_components: tuple[str, ...] = ()
tls_subject_cn: str | None = None
tls_subject_alt_names: tuple[str, ...] = ()
tls_issuer_cn: str | None = None
tls_expires_at: str | None = None
tls_sha256: str | None = None
tls_jarm: str | None = None
@classmethod
def from_record(cls, record: object) -> Self:
if not isinstance(record, dict):
raise ValueError('Shodan services must be objects')
allowed = {'port', 'transport', 'product', 'version', 'observed_at', 'cpes', 'http', 'tls'}
if set(record) - allowed:
raise ValueError('Shodan service contains unsupported fields')
port = record.get('port')
if isinstance(port, bool) or not isinstance(port, int) or not 1 <= port <= 65535:
raise ValueError('Shodan service port must be between 1 and 65535')
transport = _optional_text(record.get('transport'), 'service transport')
if transport is None or transport.casefold() not in {'tcp', 'udp'}:
raise ValueError('Shodan service transport must be tcp or udp')
http = record.get('http')
if http is not None and (not isinstance(http, dict) or set(http) - {'title', 'server', 'components'}):
raise ValueError('Shodan HTTP evidence contains unsupported fields')
tls = record.get('tls')
if tls is not None and (
not isinstance(tls, dict)
or set(tls) - {'subject_cn', 'subject_alt_names', 'issuer_cn', 'expires_at', 'sha256', 'jarm'}
):
raise ValueError('Shodan TLS evidence contains unsupported fields')
return cls(
port=port,
transport=transport.casefold(),
product=_optional_text(record.get('product'), 'service product'),
version=_optional_text(record.get('version'), 'service version'),
observed_at=_optional_text(record.get('observed_at'), 'service observation time'),
cpes=_string_tuple(record.get('cpes'), 'service CPEs'),
http_title=_optional_text(http.get('title'), 'HTTP title') if isinstance(http, dict) else None,
http_server=_optional_text(http.get('server'), 'HTTP server') if isinstance(http, dict) else None,
http_components=_string_tuple(http.get('components'), 'HTTP components') if isinstance(http, dict) else (),
tls_subject_cn=_optional_text(tls.get('subject_cn'), 'TLS subject CN') if isinstance(tls, dict) else None,
tls_subject_alt_names=(
_hostname_tuple(tls.get('subject_alt_names'), 'TLS subject alternative names') if isinstance(tls, dict) else ()
),
tls_issuer_cn=_optional_text(tls.get('issuer_cn'), 'TLS issuer CN') if isinstance(tls, dict) else None,
tls_expires_at=_optional_text(tls.get('expires_at'), 'TLS expiry') if isinstance(tls, dict) else None,
tls_sha256=_optional_text(tls.get('sha256'), 'TLS SHA-256') if isinstance(tls, dict) else None,
tls_jarm=_optional_text(tls.get('jarm'), 'TLS JARM') if isinstance(tls, dict) else None,
)
def sort_key(self) -> tuple[object, ...]:
return (
self.port,
self.transport,
self.observed_at or '',
self.product or '',
self.version or '',
self.http_title or '',
self.http_server or '',
self.http_components,
self.tls_subject_cn or '',
self.tls_subject_alt_names,
self.tls_issuer_cn or '',
self.tls_expires_at or '',
self.tls_sha256 or '',
self.tls_jarm or '',
self.cpes,
)
def to_record(self) -> dict[str, object]:
record: dict[str, object] = {'port': self.port, 'transport': self.transport}
for field, value in (
('product', self.product),
('version', self.version),
('observed_at', self.observed_at),
):
if value is not None:
record[field] = value
if self.cpes:
record['cpes'] = list(self.cpes)
http: dict[str, object] = {}
if self.http_title is not None:
http['title'] = self.http_title
if self.http_server is not None:
http['server'] = self.http_server
if self.http_components:
http['components'] = list(self.http_components)
if http:
record['http'] = http
tls: dict[str, object] = {}
if self.tls_subject_alt_names:
tls['subject_alt_names'] = list(self.tls_subject_alt_names)
for field, value in (
('subject_cn', self.tls_subject_cn),
('issuer_cn', self.tls_issuer_cn),
('expires_at', self.tls_expires_at),
('sha256', self.tls_sha256),
('jarm', self.tls_jarm),
):
if value is not None:
tls[field] = value
if tls:
record['tls'] = tls
return record
@dataclass(frozen=True, slots=True)
class ShodanHostObservation:
ip: str
services: tuple[ShodanServiceObservation, ...]
asn: str | None = None
organization: str | None = None
isp: str | None = None
hostnames: tuple[str, ...] = ()
domains: tuple[str, ...] = ()
@classmethod
def from_record(cls, value: str, details: object) -> Self:
if not isinstance(value, str) or '%' in value:
raise ValueError('Shodan host value must be a canonical IP address')
try:
canonical_ip = str(ip_address(value.strip()))
except ValueError as error:
raise ValueError('Shodan host value must be a canonical IP address') from error
if not isinstance(details, dict):
raise ValueError('Shodan host details must be an object')
allowed = {'asn', 'organization', 'isp', 'hostnames', 'domains', 'services'}
if set(details) - allowed:
raise ValueError('Shodan host details contain unsupported fields')
raw_services = details.get('services')
if not isinstance(raw_services, list) or not raw_services:
raise ValueError('Shodan host details require at least one service')
services = tuple(
sorted(
{ShodanServiceObservation.from_record(service) for service in raw_services},
key=ShodanServiceObservation.sort_key,
)
)
raw_asn = _optional_text(details.get('asn'), 'ASN')
return cls(
ip=canonical_ip,
services=services,
asn=normalize_asn(raw_asn) if raw_asn is not None else None,
organization=_optional_text(details.get('organization'), 'organization'),
isp=_optional_text(details.get('isp'), 'ISP'),
hostnames=_hostname_tuple(details.get('hostnames'), 'hostnames'),
domains=_hostname_tuple(details.get('domains'), 'domains'),
)
def sort_key(self) -> tuple[int, int]:
address = ip_address(self.ip)
return address.version, int(address)
def to_details(self) -> dict[str, object]:
details: dict[str, object] = {}
for field, value in (
('asn', self.asn),
('organization', self.organization),
('isp', self.isp),
):
if value is not None:
details[field] = value
if self.hostnames:
details['hostnames'] = list(self.hostnames)
if self.domains:
details['domains'] = list(self.domains)
details['services'] = [service.to_record() for service in self.services]
return details
def canonical_shodan_hosts(
observations: tuple[ShodanHostObservation, ...] | list[ShodanHostObservation],
) -> tuple[ShodanHostObservation, ...]:
by_ip: dict[str, ShodanHostObservation] = {}
for observation in observations:
existing = by_ip.get(observation.ip)
if existing is not None and existing != observation:
raise ValueError(f'Shodan host {observation.ip} has conflicting evidence')
by_ip[observation.ip] = observation
return tuple(sorted(by_ip.values(), key=ShodanHostObservation.sort_key))
uv.lock
Generated
-73
View File
@@ -466,18 +466,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/c7/0d/67e5b4109ea4a837e80daa87c2c696711955e40449a97e8926672534def2/click-8.4.1-py3-none-any.whl", hash = "sha256:482be17c6991b8c19c5429a1e995d9b0efdbb63172824c41f99965dc0ade8ec2", size = 116639, upload-time = "2026-05-22T04:08:35.26Z" },
]
[[package]]
name = "click-plugins"
version = "1.1.1.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "click" },
]
sdist = { url = "https://files.pythonhosted.org/packages/c3/a4/34847b59150da33690a36da3681d6bbc2ec14ee9a846bc30a6746e5984e4/click_plugins-1.1.1.2.tar.gz", hash = "sha256:d7af3984a99d243c131aa1a828331e7630f4a88a9741fd05c927b204bcf92261", size = 8343, upload-time = "2025-06-25T00:47:37.555Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/3d/9a/2abecb28ae875e39c8cad711eb1186d8d14eab564705325e77e4e6ab9ae5/click_plugins-1.1.1.2-py2.py3-none-any.whl", hash = "sha256:008d65743833ffc1f5417bf0e78e8d2c23aab04d9745ba817bd3e71b0feb6aa6", size = 11051, upload-time = "2025-06-25T00:47:36.731Z" },
]
[[package]]
name = "colorama"
version = "0.4.6"
@@ -512,15 +500,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/38/a9/69a6924f645eb4dd8cd625bf255b3625990eb3e14e073438a53c405dcd3e/fastapi-0.138.1-py3-none-any.whl", hash = "sha256:b994cae7ba8b82c976a728b544244de31333fa5f7d261f9a1dffe526444cae23", size = 129182, upload-time = "2026-06-25T15:40:40.771Z" },
]
[[package]]
name = "filelock"
version = "3.29.4"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/e6/dc/be6cbe99670cd6e4ad387123647cb08e0c32975e223f82551e914c5568a6/filelock-3.29.4.tar.gz", hash = "sha256:10cdb3656fc44541cdf30652a93fb10ec6b05325620eb316bd26893e4201538a", size = 63028, upload-time = "2026-06-13T16:12:00.744Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/13/37/a065dc3bd6e49423a6532c642ca7378d3f467b1ef44c2800c937af7f9739/filelock-3.29.4-py3-none-any.whl", hash = "sha256:dac1648087d5115554850d113e7dd8c83ab2d38e3435dde2d4f163847e57b767", size = 42757, upload-time = "2026-06-13T16:11:59.582Z" },
]
[[package]]
name = "frozenlist"
version = "1.8.0"
@@ -1524,18 +1503,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" },
]
[[package]]
name = "requests-file"
version = "3.0.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "requests" },
]
sdist = { url = "https://files.pythonhosted.org/packages/3c/f8/5dc70102e4d337063452c82e1f0d95e39abfe67aa222ed8a5ddeb9df8de8/requests_file-3.0.1.tar.gz", hash = "sha256:f14243d7796c588f3521bd423c5dea2ee4cc730e54a3cac9574d78aca1272576", size = 6967, upload-time = "2025-10-20T18:56:42.279Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/e1/d5/de8f089119205a09da657ed4784c584ede8381a0ce6821212a6d4ca47054/requests_file-3.0.1-py2.py3-none-any.whl", hash = "sha256:d0f5eb94353986d998f80ac63c7f146a307728be051d4d1cd390dbdb59c10fa2", size = 4514, upload-time = "2025-10-20T18:56:41.184Z" },
]
[[package]]
name = "retrying"
version = "1.4.2"
@@ -1583,20 +1550,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/cb/46/240ea004bf6dc4feb40e9832f2205a476a47dd5b8a3f8211a5fc5f95e20e/ruff-0.16.1-py3-none-win_arm64.whl", hash = "sha256:dbaadaac38c70239f056d306b7476f246b0bf000fa6b3876402acbf5b227eaf8", size = 11309414, upload-time = "2026-07-30T19:36:58.79Z" },
]
[[package]]
name = "shodan"
version = "1.31.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "click" },
{ name = "click-plugins" },
{ name = "colorama" },
{ name = "requests" },
{ name = "tldextract" },
{ name = "xlsxwriter" },
]
sdist = { url = "https://files.pythonhosted.org/packages/c5/06/c6dcc975a1e7d89bc764fd271da8138b318e18080b48e7f1acd2ab63df28/shodan-1.31.0.tar.gz", hash = "sha256:c73275386ea02390e196c35c660706a28dd4d537c5a21eb387ab6236fac251f6", size = 57939, upload-time = "2023-12-17T01:42:02.426Z" }
[[package]]
name = "six"
version = "1.17.0"
@@ -1700,7 +1653,6 @@ dependencies = [
{ name = "python-dateutil" },
{ name = "pyyaml" },
{ name = "retrying" },
{ name = "shodan" },
{ name = "sqlalchemy" },
{ name = "ujson" },
{ name = "uvicorn" },
@@ -1745,7 +1697,6 @@ requires-dist = [
{ name = "python-dateutil", specifier = "==2.9.0.post0" },
{ name = "pyyaml", specifier = "==6.0.3" },
{ name = "retrying", specifier = "==1.4.2" },
{ name = "shodan", specifier = "==1.31.0" },
{ name = "sqlalchemy", specifier = "==2.0.51" },
{ name = "ujson", specifier = "==5.13.0" },
{ name = "uvicorn", specifier = "==0.52.1" },
@@ -1770,21 +1721,6 @@ dev = [
{ name = "wheel", specifier = "==0.47.0" },
]
[[package]]
name = "tldextract"
version = "5.3.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "filelock" },
{ name = "idna" },
{ name = "requests" },
{ name = "requests-file" },
]
sdist = { url = "https://files.pythonhosted.org/packages/65/7b/644fbbb49564a6cb124a8582013315a41148dba2f72209bba14a84242bf0/tldextract-5.3.1.tar.gz", hash = "sha256:a72756ca170b2510315076383ea2993478f7da6f897eef1f4a5400735d5057fb", size = 126105, upload-time = "2025-12-28T23:58:05.532Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/6d/42/0e49d6d0aac449ca71952ec5bae764af009754fcb2e76a5cc097543747b3/tldextract-5.3.1-py3-none-any.whl", hash = "sha256:6bfe36d518de569c572062b788e16a659ccaceffc486d243af0484e8ecf432d9", size = 105886, upload-time = "2025-12-28T23:58:04.071Z" },
]
[[package]]
name = "ty"
version = "0.0.69"
@@ -2024,15 +1960,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/0f/8d/52eaa9187b88596b0a8b646874cfec5a5c3fce8c52b5182be1a0253203a3/winloop-0.6.3-cp314-cp314t-win_arm64.whl", hash = "sha256:447006f38f13827ff4600e7beeda70367370cb8dab8ea84042e8fa1749f32b1c", size = 576206, upload-time = "2026-04-27T16:08:07.007Z" },
]
[[package]]
name = "xlsxwriter"
version = "3.2.9"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/46/2c/c06ef49dc36e7954e55b802a8b231770d286a9758b3d936bd1e04ce5ba88/xlsxwriter-3.2.9.tar.gz", hash = "sha256:254b1c37a368c444eac6e2f867405cc9e461b0ed97a3233b2ac1e574efb4140c", size = 215940, upload-time = "2025-09-16T00:16:21.63Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/3a/0c/3662f4a66880196a590b202f0db82d919dd2f89e99a27fadef91c4a33d41/xlsxwriter-3.2.9-py3-none-any.whl", hash = "sha256:9a5db42bc5dff014806c58a20b9eae7322a134abb6fce3c92c181bfb275ec5b3", size = 175315, upload-time = "2025-09-16T00:16:20.108Z" },
]
[[package]]
name = "yarl"
version = "1.24.2"