diff --git a/CHANGELOG.md b/CHANGELOG.md index de92f432..ae0be4a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Added keyless Shodan Certificate Transparency hostname discovery with bounded requests and offline response contracts. - Added transactional SQLite storage and loading for completed full-pipeline runs without changing legacy result rows. - Added deterministic JSONL report companions finalized after selected one-shot actions complete. +- Added normalized BuiltWith framework, language, server, CMS, and analytics findings to JSONL and completed-result SQLite output. - Added DNSDB passive DNS discovery with API key configuration, shared transport handling, result parsing, and offline tests ([9b41b78e](https://github.com/laramies/theHarvester/commit/9b41b78e), [aba9fec6](https://github.com/laramies/theHarvester/commit/aba9fec6)). - Added `--verbose` diagnostic logging while keeping normal operator output available at the default log level ([8a7b8b71](https://github.com/laramies/theHarvester/commit/8a7b8b71)). - Added an opt-in passive-provider smoke workflow and a network guard that keeps routine tests offline by default ([72e5820f](https://github.com/laramies/theHarvester/commit/72e5820f)). @@ -21,6 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed - Included HIBP verified-domain in `all` and matching capability selectors like every other P0 source, with REST operator authentication applied after source expansion when its provider key is configured. - Allowed REST `/query` requests to select discovery sources by result capability, matching the CLI's union semantics while preserving explicit source selection. +- Changed `-b all` to select every cataloged P0 passive source once while leaving P1 DNS and P2 direct sources available through explicit selection. - Expanded Common Crawl discovery to use every unique crawl ending within one year of the newest catalog entry, validate catalog endpoints, batch requests, cap each query at 100 pages, and enforce the CLI result limit across page requests ([249ce64b](https://github.com/laramies/theHarvester/commit/249ce64b), [70470cd8](https://github.com/laramies/theHarvester/commit/70470cd8)). - Completed bounded pagination for Wayback Archive and Cert Spotter, including continuation handling, truncation diagnostics, and preservation of partial results on provider failures ([df6ff2c9](https://github.com/laramies/theHarvester/commit/df6ff2c9), [f85a08ff](https://github.com/laramies/theHarvester/commit/f85a08ff)). - Routed operator messages and diagnostics through logging, preserved host logging policy and existing handlers, and configured logging for the standalone API example ([8a7b8b71](https://github.com/laramies/theHarvester/commit/8a7b8b71)). @@ -39,6 +41,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Removed the nonfunctional ThreatCrowd source because its service hostnames terminate at deleted AWS load balancers and return NXDOMAIN; OTX remains available through its separate adapter. ### Fixed +- Removed BuiltWith's duplicate interesting-URL getter by allowing the shared collector to use either established getter spelling. - Made no-filename REST `/query` executions reach completed-result construction and SQLite persistence without changing the legacy response fields. - Made Chaos reject empty credentials, report HTTP and malformed-response failures, and preserve supported subdomain response shapes. - Made Fofa reject incomplete credentials, report HTTP and malformed-response failures, normalize scoped hosts, and discard invalid IP values. diff --git a/README.md b/README.md index abb16443..f08c0246 100644 --- a/README.md +++ b/README.md @@ -54,7 +54,7 @@ Combine capability selectors, or mix them with explicit source names: uv run theHarvester -d example.com -b emails,urls,certspotter ``` -Capability selectors form a union and choose which sources run. They do not discard other result types returned by those sources. Available selectors are `subdomains`, `emails`, `ips`, `asns`, `urls`, `people`, and `breaches`. `-b all` continues to run every registered source. +Capability selectors form a union and choose which sources run. They do not discard other result types returned by those sources. Available selectors are `subdomains`, `emails`, `ips`, `asns`, `urls`, `people`, and `breaches`. `-b all` runs every cataloged P0 passive source. P1 DNS and P2 direct sources require explicit selection. Save JSON, XML, and JSONL reports: @@ -117,6 +117,7 @@ docker compose up --build The table shows which result types each source can add to consolidated CLI results. Legacy JSON and XML keep their existing schemas; breach names are retained in JSONL and SQLite. Some adapters parse fields that the reports do not store. The report groups findings by result type. It does not record which source found each item. Empty optional fields may be omitted. +BuiltWith's normalized frameworks, languages, servers, CMS products, and analytics products are retained in JSONL and completed-result SQLite rows. A checkmark means the source can add that result type. The **Separate output** column lists REST endpoints and optional actions that return other data. @@ -234,7 +235,7 @@ The JSON report is a single object and is the more complete format for automatio The XML report contains the command, emails, hosts, and virtual hosts. Use JSON when you need the additional result types above. -The JSONL report is finalized after the selected one-shot actions finish. Its first line is a summary with an independent run UUID, the target, UTC timestamps, counts, and schema version. Each remaining line is one deterministic, deduplicated string finding, including stable Have I Been Pwned breach names as `breach` records. The format does not claim provider success or record source attribution. +The JSONL report is finalized after the selected one-shot actions finish. Its first line is a summary with an independent run UUID, the target, UTC timestamps, counts, and schema version. Each remaining line is one deterministic, deduplicated string finding, including stable Have I Been Pwned breach names as `breach` records and normalized BuiltWith findings as `framework`, `language`, `server`, `cms`, or `analytics` records. The format does not claim provider success or record source attribution. List every JSONL finding as tab-separated type and value columns: diff --git a/tests/discovery/test_builtwith.py b/tests/discovery/test_builtwith.py index 685aa779..440c83f6 100644 --- a/tests/discovery/test_builtwith.py +++ b/tests/discovery/test_builtwith.py @@ -1,5 +1,7 @@ +import json import sys import types +from pathlib import Path import pytest @@ -16,6 +18,8 @@ if 'aiohttp_socks' not in sys.modules: from theHarvester.discovery import builtwith from theHarvester.discovery.constants import MissingKey +from theHarvester import __main__ as theharvester_main +from theHarvester.lib.completed_result import CompletedResult @pytest.mark.asyncio @@ -36,11 +40,15 @@ async def test_process_accepts_text_json_content_type(monkeypatch) -> None: 'domains': ['sub.example.com'], 'paths': ['https://example.com/login'], 'technologies': [ + None, {'name': 'Django', 'category': 'framework'}, {'name': 'Python', 'category': 'language'}, {'name': 'nginx', 'category': 'server'}, {'name': 'WordPress', 'category': 'cms'}, {'name': 'Google Analytics', 'category': 'analytics'}, + {'category': 'framework'}, + {'name': ' ', 'category': 'server'}, + {'name': 7, 'category': 'cms'}, ], } @@ -120,3 +128,74 @@ async def test_process_handles_non_200_status(monkeypatch) -> None: assert await search.get_hostnames() == set() assert await search.get_tech_stack() == {} + + +@pytest.mark.asyncio +async def test_normalized_builtwith_results_reach_completed_jsonl( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + completed_results: list[CompletedResult] = [] + + class FakeStash: + async def do_init(self) -> None: + return None + + async def store_all(self, *_args: object) -> None: + return None + + async def store_completed_result(self, result: CompletedResult) -> None: + completed_results.append(result) + + class FakeBuiltWith: + def __init__(self, domain: str) -> None: + assert domain == 'example.com' + + async def process(self, _proxy: bool) -> None: + return None + + async def get_hostnames(self) -> set[str]: + return set() + + async def get_interesting_urls(self) -> set[str]: + return {'https://example.com/login'} + + async def get_frameworks(self) -> set[str]: + return {'Django'} + + async def get_languages(self) -> set[str]: + return {'Python'} + + async def get_servers(self) -> set[str]: + return {'nginx'} + + async def get_cms(self) -> set[str]: + return {'WordPress'} + + async def get_analytics(self) -> set[str]: + return {'Google Analytics'} + + report = tmp_path / 'builtwith-report' + monkeypatch.setattr(theharvester_main.stash, 'StashManager', FakeStash) + monkeypatch.setattr(theharvester_main.builtwith, 'SearchBuiltWith', FakeBuiltWith) + monkeypatch.setattr(sys, 'argv', ['theHarvester', '-d', 'example.com', '-b', 'builtwith', '-f', str(report)]) + + with pytest.raises(SystemExit) as exit_info: + await theharvester_main.start() + + assert exit_info.value.code == 0 + assert completed_results[0].results == ( + ('analytics', 'Google Analytics'), + ('cms', 'WordPress'), + ('framework', 'Django'), + ('interesting-url', 'https://example.com/login'), + ('language', 'Python'), + ('server', 'nginx'), + ) + records = [json.loads(line) for line in report.with_suffix('.jsonl').read_text().splitlines()] + assert {'type': 'interesting-url', 'value': 'https://example.com/login'} in records + assert {'type': 'framework', 'value': 'Django'} in records + assert {'type': 'language', 'value': 'Python'} in records + assert {'type': 'server', 'value': 'nginx'} in records + assert {'type': 'cms', 'value': 'WordPress'} in records + assert {'type': 'analytics', 'value': 'Google Analytics'} in records diff --git a/tests/discovery/test_rapiddns.py b/tests/discovery/test_rapiddns.py index b8dc517c..597bae1e 100644 --- a/tests/discovery/test_rapiddns.py +++ b/tests/discovery/test_rapiddns.py @@ -179,7 +179,6 @@ async def test_rapiddns_evidence_reaches_existing_outputs( created = 0 def __init__(self, _domain: str) -> None: - self.is_late_action = self.created > 0 type(self).created += 1 async def process(self, _proxy: bool) -> None: @@ -189,16 +188,14 @@ async def test_rapiddns_evidence_reaches_existing_outputs( return set() async def get_ips(self) -> set[str]: - if self.is_late_action: - return {'2001:0DB8::1', '198.51.100.9', 'not-an-ip'} - return set() + return {'2001:0DB8::1', '198.51.100.9', 'not-an-ip'} async def fake_reverse_all_ips_in_range( iprange: str, callback: Any, nameservers: list[str] | None = None, ) -> None: - assert iprange in {'192.0.2.0/24', '198.51.100.0/24'} + assert iprange in {'192.0.2.0/24', '198.51.100.0/24', '2001:d00::/24'} assert nameservers is None callback('reverse.example.com') @@ -237,7 +234,7 @@ async def test_rapiddns_evidence_reaches_existing_outputs( report_json = json.loads(report.with_suffix('.json').read_text()) assert report_json['hosts'] == ['alias.example.com', 'api.example.com', 'broken.example.com'] - assert report_json['ips'] == ['192.0.2.1'] + assert report_json['ips'] == ['192.0.2.1', '198.51.100.9', '2001:db8::1'] assert 'interesting_urls' not in report_json jsonl_records = [json.loads(line) for line in report.with_suffix('.jsonl').read_text().splitlines()] @@ -314,7 +311,7 @@ async def test_rapiddns_evidence_reaches_existing_outputs( ('ip', ('192.0.2.1',), 'rapiddns'), ] assert len(completed_results) == 2 - assert FakeSecurityScorecard.created == 2 + assert FakeSecurityScorecard.created == 1 assert completed_results[1].target == 'example.com' assert {'192.0.2.1', '198.51.100.2'} <= {value for kind, value in completed_results[1].results if kind == 'ip-address'} diff --git a/tests/lib/test_core.py b/tests/lib/test_core.py index 534d4b02..7285f034 100644 --- a/tests/lib/test_core.py +++ b/tests/lib/test_core.py @@ -11,6 +11,7 @@ import yaml import theHarvester.lib.core as core_module from theHarvester.lib.core import CONFIG_DIRS, DATA_DIR, AsyncFetcher, Core, FetcherResponse from theHarvester.lib.output import configure_logging +from theHarvester.lib.source_catalog import SOURCE_SPECS, ActivityClass @pytest.fixture(autouse=True) @@ -74,8 +75,22 @@ def test_named_source_can_be_combined_with_a_capability() -> None: assert Core.expand_source_selection('breaches,hibpverified') == ['haveibeenpwned', 'hibpverified'] -def test_all_preserves_every_supported_source() -> None: - assert Core.expand_source_selection("ALL") == Core.get_supportedengines() +def test_all_selects_only_passive_catalog_sources() -> None: + assert Core.expand_source_selection("ALL") == sorted( + spec.name + for spec in SOURCE_SPECS.values() + if spec.activity is ActivityClass.PASSIVE + ) + assert { + name: spec.activity for name, spec in SOURCE_SPECS.items() if spec.activity is not ActivityClass.PASSIVE + } == { + "criminalip": ActivityClass.DIRECT, + "pentesttools": ActivityClass.DNS, + "shodan": ActivityClass.DNS, + "shodanInternetDB": ActivityClass.DNS, + "subdomainfinderc99": ActivityClass.DNS, + "windvane": ActivityClass.DNS, + } def mock_read_text(mocked: dict[Path, str | Exception]): diff --git a/tests/test_all_source_orchestration.py b/tests/test_all_source_orchestration.py new file mode 100644 index 00000000..1a9ef018 --- /dev/null +++ b/tests/test_all_source_orchestration.py @@ -0,0 +1,243 @@ +import json +import sys +import xml.etree.ElementTree as ElementTree +from collections import Counter +from pathlib import Path +from types import ModuleType +from typing import Any +from uuid import UUID + +import pytest + +from theHarvester import __main__ as theharvester_main +from theHarvester.lib.source_catalog import SOURCE_SPECS, ActivityClass + + +@pytest.mark.asyncio +async def test_source_help_uses_the_runtime_catalog( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + monkeypatch.setattr(theharvester_main, 'SOURCE_SPECS', {'catalog-only-source': object()}) + monkeypatch.setattr(sys, 'argv', ['theHarvester', '--help']) + + with pytest.raises(SystemExit) as exit_info: + await theharvester_main.start() + + assert exit_info.value.code == 0 + help_output = capsys.readouterr().out + assert 'catalog-only-source' in help_output + assert 'linkedin_links' not in help_output + + +@pytest.mark.asyncio +async def test_activity_summary_includes_source_and_option_classes( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + class FakeStash: + async def do_init(self) -> None: + return None + + async def store_all(self, *_args: object) -> None: + return None + + async def store_completed_result(self, _result: object) -> None: + return None + + class FakeCriminalIP: + def __init__(self, _domain: str) -> None: + pass + + async def process(self, _proxy: bool) -> None: + return None + + async def get_hostnames(self) -> set[str]: + return set() + + async def get_ips(self) -> set[str]: + return set() + + async def get_asns(self) -> set[str]: + return set() + + monkeypatch.setattr(theharvester_main.criminalip, 'SearchCriminalIP', FakeCriminalIP) + monkeypatch.setattr(theharvester_main.stash, 'StashManager', FakeStash) + monkeypatch.setattr(sys, 'argv', ['theHarvester', '-d', 'example.test', '-b', 'criminalip', '-n', '-s']) + + with pytest.raises(SystemExit) as exit_info: + await theharvester_main.start() + + assert exit_info.value.code == 0 + assert '[*] Activity: P0 passive collection, P1 DNS interaction, P2 direct interaction' in capsys.readouterr().out + + +@pytest.mark.asyncio +async def test_activity_summary_covers_api_scan_without_sources( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + class FakeStash: + async def do_init(self) -> None: + return None + + async def store_all(self, *_args: object) -> None: + return None + + async def store_completed_result(self, _result: object) -> None: + return None + + def stop_api_scan(**_kwargs: object) -> None: + raise RuntimeError('offline test stop') + + monkeypatch.setattr(theharvester_main.api_endpoints, 'SearchApiEndpoints', stop_api_scan) + monkeypatch.setattr(theharvester_main.stash, 'StashManager', FakeStash) + monkeypatch.setattr(sys, 'argv', ['theHarvester', '-d', 'example.test', '-a']) + + with pytest.raises(SystemExit) as exit_info: + await theharvester_main.start() + + assert exit_info.value.code == 0 + assert '[*] Activity: P2 direct interaction' in capsys.readouterr().out + + +@pytest.mark.asyncio +async def test_legacy_handlerless_source_does_not_break_activity_summary( + monkeypatch: pytest.MonkeyPatch, +) -> None: + class FakeStash: + async def do_init(self) -> None: + return None + + async def store_completed_result(self, _result: object) -> None: + return None + + monkeypatch.setattr(theharvester_main.stash, 'StashManager', FakeStash) + monkeypatch.setattr(sys, 'argv', ['theHarvester', '-d', 'example.test', '-b', 'linkedin']) + + with pytest.raises(SystemExit) as exit_info: + await theharvester_main.start() + + assert exit_info.value.code == 0 + + +@pytest.mark.asyncio +async def test_all_schedules_each_passive_catalog_source_once_and_reports_results( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + executions: Counter[str] = Counter() + passive_sources = sorted( + source + for source, spec in SOURCE_SPECS.items() + if spec.activity is ActivityClass.PASSIVE + ) + + class TestStash(theharvester_main.stash.StashManager): + def __init__(self) -> None: + super().__init__() + self.db = str(tmp_path / 'stash.sqlite') + + async def store_all(self, *_args: Any, **_kwargs: Any) -> None: + return None + + async def store(self, *_args: Any, **_kwargs: Any) -> None: + return None + + class FakeAdapter: + def __init__(self, adapter: str) -> None: + self.adapter = adapter + + async def process(self, *_args: object, **_kwargs: object) -> None: + executions[self.adapter] += 1 + + async def get_hostnames(self) -> set[str]: + return {'sub.example.test'} + + async def get_emails(self) -> set[str]: + return {'user@example.test'} + + async def get_ips(self) -> set[str]: + return {'192.0.2.1'} + + async def get_asns(self) -> set[str]: + return {'AS64500'} + + async def get_people(self) -> list[dict[str, str]]: + return [{'name': 'Example Person'}] + + async def get_links(self) -> set[str]: + return {'https://sub.example.test/profile'} + + async def get_interestingurls(self) -> set[str]: + return {'https://sub.example.test/evidence'} + + async def get_interesting_urls(self) -> set[str]: + return await self.get_interestingurls() + + async def get_host_ip_pairs(self) -> set[tuple[str, str]]: + return set() + + async def get_breach_names(self) -> set[str]: + return {'ExampleBreach'} + + async def get_infostealers(self) -> list[dict[str, object]]: + return [] + + def fake_constructor(adapter: str): + def constructor(*_args: object, **_kwargs: object) -> FakeAdapter: + return FakeAdapter(adapter) + + return constructor + + discovery_modules = sorted( + { + value + for value in vars(theharvester_main).values() + if isinstance(value, ModuleType) and value.__name__.startswith('theHarvester.discovery.') + }, + key=lambda module: module.__name__, + ) + patched_classes = 0 + for module in discovery_modules: + for name, value in list(vars(module).items()): + if isinstance(value, type) and value.__module__ == module.__name__: + monkeypatch.setattr(module, name, fake_constructor(f'{module.__name__}.{name}')) + patched_classes += 1 + assert patched_classes + + report = tmp_path / 'all-sources' + monkeypatch.setattr(theharvester_main.stash, 'StashManager', TestStash) + monkeypatch.setattr( + sys, + 'argv', + ['theHarvester', '-d', 'example.test', '-b', 'all', '-f', str(report)], + ) + + with pytest.raises(SystemExit) as exit_info: + await theharvester_main.start() + + assert exit_info.value.code == 0 + assert len(executions) == len(passive_sources) + assert sum(executions.values()) == len(passive_sources) + assert set(executions.values()) == {1} + + json_report = json.loads(report.with_suffix('.json').read_text()) + assert 'sub.example.test' in json_report['hosts'] + assert 'user@example.test' in json_report['emails'] + + jsonl_records = [json.loads(line) for line in report.with_suffix('.jsonl').read_text().splitlines()] + assert {'type': 'hostname', 'value': 'sub.example.test'} in jsonl_records + assert {'type': 'email', 'value': 'user@example.test'} in jsonl_records + + completed = await TestStash().load_completed_result(UUID(jsonl_records[0]['run_id'])) + assert completed.target == 'example.test' + assert ('hostname', 'sub.example.test') in completed.results + assert ('email', 'user@example.test') in completed.results + assert ('ip-address', '192.0.2.1') in completed.results + + xml_hosts = { + (element.findtext('hostname') or (element.text or '').strip()) + for element in ElementTree.parse(report.with_suffix('.xml')).getroot().findall('host') + } + assert 'sub.example.test' in xml_hosts diff --git a/theHarvester/__main__.py b/theHarvester/__main__.py index 96f50898..4fc9b365 100644 --- a/theHarvester/__main__.py +++ b/theHarvester/__main__.py @@ -86,7 +86,7 @@ from theHarvester.lib.completed_result import CompletedResult, ResultKind from theHarvester.lib.core import DATA_DIR, Core, show_default_error_message from theHarvester.lib.hostnames import normalize_scoped_hostname from theHarvester.lib.output import configure_logging, output_logger, print_linkedin_sections, print_section, sorted_unique -from theHarvester.lib.source_catalog import ResultRoute, get_source_spec +from theHarvester.lib.source_catalog import SOURCE_SPECS, ActivityClass, ResultRoute, get_source_spec from theHarvester.screenshot.screenshot import ScreenShotter if TYPE_CHECKING: @@ -236,12 +236,10 @@ async def start( parser.add_argument( '-b', '--source', - help="""Comma-separated sources or capability selectors: subdomains, emails, ips, asns, urls, people, breaches, or all. - Sources: baidu, bevigil, brave, bufferoverun, - builtwith, censys, certspotter, chaos, commoncrawl, criminalip, crtsh, dehashed, dnsdumpster, duckduckgo, dymo, fofa, fullhunt, github-code, - gitlab, hackertarget, haveibeenpwned, hibpverified, hudsonrock, hunter, hunterhow, intelx, leakix, leaklookup, mojeek, netlas, onyphe, otx, pentesttools, - projectdiscovery, rapiddns, robtex, rocketreach, securityscorecard, securityTrails, sherlockeye, shodan, shodanct, shodanInternetDB, subdomaincenter, - subdomainfinderc99, thc, tomba, urlscan, venacus, virustotal, waybackarchive, whoisxml, windvane, yahoo, zoomeye""", + help=( + 'Comma-separated sources or capability selectors: subdomains, emails, ips, asns, urls, people, ' + f'breaches, or all. Sources: {", ".join(sorted(SOURCE_SPECS, key=str.casefold))}' + ), ) # determines if the filename is coming from rest api or user @@ -357,6 +355,11 @@ async def start( interesting_urls: list = [] total_asns: list = [] all_breaches: list[str] = [] + all_frameworks: list[str] = [] + all_languages: list[str] = [] + all_servers: list[str] = [] + all_cms: list[str] = [] + all_analytics: list[str] = [] linkedin_people_list_tracker = [] linkedin_links_tracker = [] @@ -439,7 +442,8 @@ async def start( await db.store_all(word, links, 'linkedinlinks', source) if ResultRoute.INTERESTING_URLS in routes: - iurls = await search_engine.get_interestingurls() + get_interesting_urls = getattr(search_engine, 'get_interesting_urls', None) + iurls = await get_interesting_urls() if get_interesting_urls else await search_engine.get_interestingurls() interesting_urls.extend(iurls) if len(iurls) > 0: await db.store_all(word, iurls, 'interestingurls', source) @@ -452,6 +456,18 @@ async def start( if ResultRoute.BREACHES in routes: all_breaches.extend(await search_engine.get_breach_names()) + if source == 'builtwith': + technology_results = ( + ('get_frameworks', all_frameworks, 'framework'), + ('get_languages', all_languages, 'language'), + ('get_servers', all_servers, 'server'), + ('get_cms', all_cms, 'cms'), + ('get_analytics', all_analytics, 'analytics'), + ) + for getter_name, results, result_type in technology_results: + values = await getattr(search_engine, getter_name)() + results.extend(values) + await db_stash.store_all(word, values, result_type, source) if source == 'hudsonrock': all_infostealers.extend(await search_engine.get_infostealers()) logger.info(f'Source {source} completed') @@ -459,6 +475,22 @@ async def start( stor_lst = [] if args.source is not None: engines = Core.expand_source_selection(args.source) + activities = {get_source_spec(engine).activity for engine in engines if engine in SOURCE_SPECS} + if shodan: + activities.add(ActivityClass.PASSIVE) + if dnslookup or dnsbrute[0] or dnsresolve != '': + activities.add(ActivityClass.DNS) + if takeover_status or getattr(args, 'screenshot', '') or getattr(args, 'api_scan', False): + activities.add(ActivityClass.DIRECT) + if activities: + activity_labels = { + ActivityClass.PASSIVE: 'P0 passive collection', + ActivityClass.DNS: 'P1 DNS interaction', + ActivityClass.DIRECT: 'P2 direct interaction', + } + output_logger.info(f'[*] Activity: {", ".join(activity_labels[item] for item in ActivityClass if item in activities)}') + + if args.source is not None: # Iterate through search engines in order if set(engines).issubset(Core.get_supportedengines()): output_logger.info(f'\n[*] Target: {word} \n') @@ -1328,18 +1360,23 @@ async def start( *, extra_hostnames: Iterable[str] = (), virtual_hosts: Iterable[str] = () ) -> CompletedResult | None: groups: dict[ResultKind, Iterable[str]] = { + 'analytics': map(str, all_analytics), 'asn': map(str, total_asns), 'breach': map(str, all_breaches), + 'cms': map(str, all_cms), 'email': map(str, all_emails), + 'framework': map(str, all_frameworks), 'hostname': _normalize_hosts_for_storage((*all_hosts, *extra_hostnames), word), 'infostealer': ( json.dumps(stealer, ensure_ascii=False, separators=(',', ':'), sort_keys=True) for stealer in all_infostealers ), 'interesting-url': map(str, interesting_urls), 'ip-address': _normalize_ip_addresses(all_ip), + 'language': map(str, all_languages), 'linkedin-link': map(str, linkedin_links_tracker), 'linkedin-person': map(str, linkedin_people_list_tracker), 'person': (json.dumps(person, ensure_ascii=False, separators=(',', ':'), sort_keys=True) for person in all_people), + 'server': map(str, all_servers), 'twitter-person': map(str, twitter_people_list_tracker), 'url': map(str, all_urls), 'vhost': map(str, virtual_hosts), @@ -1859,60 +1896,6 @@ async def start( output_logger.info(' Continuing with the rest of the scan...') traceback.print_exc() # More detailed error information for developers - if 'securityscorecard' in engines: - try: - output_logger.info('\n[*] Performing SecurityScorecard scan...') - securityscorecard_scanner = securityscorecard.SearchSecurityScorecard(word) - await securityscorecard_scanner.process(use_proxy) - - # Use the existing API to get results - hosts = await securityscorecard_scanner.get_hostnames() - if hosts: - output_logger.info(f'\n[*] SecurityScorecard results: {len(hosts)} hosts found') - for host in hosts: - output_logger.info(f' - {host}') - - all_hosts.extend(hosts) - - ips = await securityscorecard_scanner.get_ips() - if ips: - output_logger.info(f'\n[*] SecurityScorecard IPs found: {len(ips)}') - for ip in ips: - output_logger.info(f' - {ip}') - all_ip.extend(ips) - - except Exception as e: - output_logger.info(f'An exception has occurred in SecurityScorecard scanning: {e}') - - if 'builtwith' in engines: - try: - output_logger.info('\n[*] Performing BuiltWith scan...') - builtwith_scanner = builtwith.SearchBuiltWith(word) - await builtwith_scanner.process(use_proxy) - - hosts = await builtwith_scanner.get_hostnames() - if hosts: - output_logger.info(f'\n[*] BuiltWith results: {len(hosts)} hosts found') - for host in hosts: - output_logger.info(f' - {host}') - - # Add results to the main host list - all_hosts.extend(hosts) - - urls = list(await builtwith_scanner.get_interesting_urls()) - if urls: - output_logger.info(f'\n[*] BuiltWith interesting URLs found: {len(urls)}') - for url in urls: - output_logger.info(f' - {url}') - interesting_urls.extend(urls) - - except Exception as e: - if isinstance(e, MissingKey): - if not args.quiet: - output_logger.info(MissingKey('BuiltWith')) - else: - output_logger.info(f'An exception has occurred in BuiltWith scanning: {e}') - completed_result = finish_completed_result(extra_hostnames=dnsrev, virtual_hosts=vhost) if filename and completed_result is not None: diff --git a/theHarvester/discovery/builtwith.py b/theHarvester/discovery/builtwith.py index 967cea9c..9c6d1239 100644 --- a/theHarvester/discovery/builtwith.py +++ b/theHarvester/discovery/builtwith.py @@ -54,8 +54,14 @@ class SearchBuiltWith: self.interesting_urls.update(self.tech_stack['paths']) if 'technologies' in self.tech_stack: for tech in self.tech_stack['technologies']: - category = tech.get('category', '').lower() - name = tech.get('name', '') + if not isinstance(tech, dict): + continue + category = tech.get('category') + name = tech.get('name') + if not isinstance(category, str) or not isinstance(name, str) or not name.strip(): + continue + category = category.lower() + name = name.strip() if 'framework' in category: self.frameworks.add(name) diff --git a/theHarvester/lib/completed_result.py b/theHarvester/lib/completed_result.py index f323121b..5403cf89 100644 --- a/theHarvester/lib/completed_result.py +++ b/theHarvester/lib/completed_result.py @@ -7,16 +7,21 @@ from typing import Literal, Self, get_args from uuid import UUID, uuid4 ResultKind = Literal[ + 'analytics', 'asn', 'breach', + 'cms', 'email', + 'framework', 'hostname', 'infostealer', 'interesting-url', 'ip-address', + 'language', 'linkedin-link', 'linkedin-person', 'person', + 'server', 'twitter-person', 'url', 'vhost', diff --git a/theHarvester/lib/core.py b/theHarvester/lib/core.py index ecfa6e5b..00db024c 100644 --- a/theHarvester/lib/core.py +++ b/theHarvester/lib/core.py @@ -19,7 +19,7 @@ from aiohttp_socks import ProxyConnector from theHarvester import __version__ from theHarvester.lib.output import output_logger -from theHarvester.lib.source_catalog import SOURCE_SPECS +from theHarvester.lib.source_catalog import SOURCE_SPECS, ActivityClass if TYPE_CHECKING: from collections.abc import Sized @@ -358,7 +358,7 @@ class Core: def expand_source_selection(cls, selection: str) -> list[str]: """Expand result capability selectors into source names.""" if selection.lower() == 'all': - return cls.get_supportedengines() + return sorted(spec.name for spec in SOURCE_SPECS.values() if spec.activity is ActivityClass.PASSIVE) capabilities = {capability for spec in SOURCE_SPECS.values() for capability in spec.capabilities} selected: set[str] = set() for token in map(str.strip, selection.split(',')): diff --git a/theHarvester/lib/source_catalog.py b/theHarvester/lib/source_catalog.py index bf6d9c0a..619b84d5 100644 --- a/theHarvester/lib/source_catalog.py +++ b/theHarvester/lib/source_catalog.py @@ -1,5 +1,11 @@ from dataclasses import dataclass -from enum import Enum, auto +from enum import Enum, StrEnum, auto + + +class ActivityClass(StrEnum): + PASSIVE = 'P0' + DNS = 'P1' + DIRECT = 'P2' class ResultRoute(Enum): @@ -36,16 +42,22 @@ _ROUTE_CAPABILITIES = { class SourceSpec: name: str routes: frozenset[ResultRoute] + activity: ActivityClass = ActivityClass.PASSIVE @property def capabilities(self) -> frozenset[str]: return frozenset(_ROUTE_CAPABILITIES[route] for route in self.routes) -def _spec(name: str, *routes: ResultRoute) -> SourceSpec: +def _spec( + name: str, + *routes: ResultRoute, + activity: ActivityClass = ActivityClass.PASSIVE, +) -> SourceSpec: return SourceSpec( name=name, routes=frozenset(routes), + activity=activity, ) @@ -59,7 +71,13 @@ _SPECS = ( _spec('certspotter', ResultRoute.SUBDOMAINS), _spec('chaos', ResultRoute.SUBDOMAINS), _spec('commoncrawl', ResultRoute.SUBDOMAINS), - _spec('criminalip', ResultRoute.SUBDOMAINS, ResultRoute.IPS, ResultRoute.ASNS), + _spec( + 'criminalip', + ResultRoute.SUBDOMAINS, + ResultRoute.IPS, + ResultRoute.ASNS, + activity=ActivityClass.DIRECT, + ), _spec('crtsh', ResultRoute.SUBDOMAINS), _spec('dehashed', ResultRoute.IPS), _spec('dnsdb', ResultRoute.SUBDOMAINS), @@ -83,7 +101,7 @@ _SPECS = ( _spec('netlas', ResultRoute.SUBDOMAINS), _spec('onyphe', ResultRoute.SUBDOMAINS, ResultRoute.IPS, ResultRoute.ASNS), _spec('otx', ResultRoute.SUBDOMAINS, ResultRoute.IPS), - _spec('pentesttools', ResultRoute.SUBDOMAINS), + _spec('pentesttools', ResultRoute.SUBDOMAINS, activity=ActivityClass.DNS), _spec('projectdiscovery', ResultRoute.SUBDOMAINS), _spec('rapiddns', ResultRoute.SUBDOMAINS, ResultRoute.IPS), _spec('robtex', ResultRoute.SUBDOMAINS, ResultRoute.IPS), @@ -91,11 +109,16 @@ _SPECS = ( _spec('securityTrails', ResultRoute.SUBDOMAINS, ResultRoute.IPS), _spec('securityscorecard', ResultRoute.SUBDOMAINS, ResultRoute.IPS), _spec('sherlockeye', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS, ResultRoute.IPS), - _spec('shodan', ResultRoute.SUBDOMAINS), - _spec('shodanInternetDB', ResultRoute.SUBDOMAINS, ResultRoute.IPS), + _spec('shodan', ResultRoute.SUBDOMAINS, activity=ActivityClass.DNS), + _spec( + 'shodanInternetDB', + ResultRoute.SUBDOMAINS, + ResultRoute.IPS, + activity=ActivityClass.DNS, + ), _spec('shodanct', ResultRoute.SUBDOMAINS), _spec('subdomaincenter', ResultRoute.SUBDOMAINS), - _spec('subdomainfinderc99', ResultRoute.SUBDOMAINS), + _spec('subdomainfinderc99', ResultRoute.SUBDOMAINS, activity=ActivityClass.DNS), _spec('thc', ResultRoute.SUBDOMAINS), _spec('tomba', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS), _spec('urlscan', ResultRoute.SUBDOMAINS, ResultRoute.IPS, ResultRoute.ASNS, ResultRoute.INTERESTING_URLS), @@ -103,7 +126,13 @@ _SPECS = ( _spec('virustotal', ResultRoute.SUBDOMAINS), _spec('waybackarchive', ResultRoute.SUBDOMAINS), _spec('whoisxml', ResultRoute.SUBDOMAINS), - _spec('windvane', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS, ResultRoute.IPS), + _spec( + 'windvane', + ResultRoute.SUBDOMAINS, + ResultRoute.EMAILS, + ResultRoute.IPS, + activity=ActivityClass.DNS, + ), _spec('yahoo', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS), _spec( 'zoomeye',