Add RouteViews enrichment and sourced ASN organization attribution (#2528)

* Add RouteViews network enrichment

* Add authenticated RouteViews API access

* Restrict RouteViews to explicit network pivots

* Restore discovered ASN RouteViews enrichment

* Retain sourced ASN organization attribution

* Avoid dangling and repeated ASN attributions

* Keep automatic RouteViews evidence target-relevant

* Complete target-relevant RouteViews enrichment

* Clarify automatic RouteViews IP pivots
This commit is contained in:
Matt
2026-08-13 10:46:54 -04:00
committed by GitHub
parent 1db27309dd
commit 624066a3e8
42 changed files with 2909 additions and 60 deletions
+2
View File
@@ -8,6 +8,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- Added sourced ASN organization attribution from URLScan, ONYPHE, and Shodan, linked to the exact hostname or IP evidence and retained in SQLite, JSONL, the API, CLI output, and HarvestView without claiming ownership or scope.
- Added bounded RouteViews routing enrichment for exact discovered IPs with sourced ASN attribution, or explicit ASN, IP, and CIDR targets, retaining typed origin, BGP-route, and RPKI evidence as external relationships without expanding active scope.
- Added bounded, keyless `crt.name` composite-index discovery as a separate source alongside `crtsh`, retaining only descendant-hostname candidates from its streamed response.
- Added bounded, keyless APIs.guru discovery through exact target-domain directory lookups, retaining only target-scoped hostnames, contact emails, and URLs from preferred OpenAPI specifications.
- Added bounded virtual host discovery over harvested or operator-supplied literal-IP endpoints, with aligned HTTP `Host` and TLS SNI, synthetic unknown-host controls, hard request and runtime limits, and structured observations on canonical hostname results in JSONL, SQLite, the API, and HarvestView.
+9 -1
View File
@@ -66,6 +66,14 @@ _Avoid_: IP-address result, resolved host
One canonical IPv4 or IPv6 CIDR merged result retained as external relationship evidence. It is never promoted into the authorized target scope or used as an active-discovery seed without a separate scope decision.
_Avoid_: Owned netblock, in-scope range, registered network
**ASN organization attribution**:
One source's time-bound organization label for a canonical ASN, tied to the hostname or IP result that supplied the relationship. It is provider evidence rather than a canonical organization identity, ownership claim, or target-scope decision.
_Avoid_: ASN owner, owning organization, organization property
**Explicit network pivot**:
A canonical ASN, IP address, or CIDR supplied by the operator as the run target for passive routing enrichment. It authorizes provider-side lookup of that identifier only; related prefixes and origins remain external relationship evidence and do not expand engagement scope.
_Avoid_: Discovered network scope, owned ASN, target netblock
**Observed route origin**:
A provider's time-bound assertion that one ASN originates one network prefix. It records routing evidence, not registration, ownership, or authorization.
_Avoid_: Owned by, registered to, authorized origin
@@ -91,7 +99,7 @@ One finite execution of theHarvester against an explicit target and selected opt
_Avoid_: Scan, monitoring cycle, session, job
**Action-only run**:
An enumeration run with no discovery sources that performs an explicitly selected DNS or direct action against an explicitly authorized target. It creates its own run record and never mutates the evidence of a parent run.
An enumeration run with no discovery sources that performs an explicitly selected provider, DNS, or direct action against an explicitly authorized target. It creates its own run record and never mutates the evidence of a parent run.
_Avoid_: Result action, parent-run update, inline scan
**Run record**:
+13 -2
View File
@@ -13,7 +13,7 @@ It is built for the early reconnaissance stage of authorized security assessment
- **Broad discovery coverage:** combine many independent sources in one run instead of querying each provider manually.
- **Useful result types:** collect hostnames, email addresses, IP addresses, URLs, ASNs, and people.
- **Enrichment after discovery:** optionally resolve DNS, query Shodan, check for subdomain takeovers, brute-force DNS names, scan common API paths, and capture screenshots.
- **Enrichment after discovery:** optionally enrich routing evidence through RouteViews, resolve DNS, query Shodan, check for subdomain takeovers, brute-force DNS names, scan common API paths, and capture screenshots.
- **CLI and browser-accessible API:** use the command line interactively or run the FastAPI service for automation and interactive Swagger/ReDoc documentation.
- **Repeatable output:** print results, write JSON, XML, and JSONL reports, and retain host, email, and IP findings in a local SQLite database.
- **Operational controls:** select individual sources, set result limits, use HTTP or SOCKS proxies, choose DNS resolvers, and suppress missing-key noise.
@@ -251,6 +251,7 @@ Never commit populated configuration files, API keys, account details, or provid
- Full CLI pipeline runs are also stored transactionally by run UUID with their completed, deduplicated findings.
- API executions use the same SQLite database as CLI results. Durable lifecycle rows stay separate from terminal evidence, while typed results and source or action origins remain queryable. JSONL handles individual run interchange, and the API can import completed runs from another theHarvester SQLite database.
- Bounded [virtual host discovery](docs/wiki/Virtual-Host-Discovery.md) enriches each confirmed `hostname` result with structured endpoint observations and `vhost` action provenance.
- `--routeviews` enriches exact discovered IPs that carry sourced ASN attribution, or an explicitly targeted ASN, IP, or CIDR, with bounded observed-origin, BGP route, and RPKI evidence. For example, `-d example.com -b asns --routeviews` asks RouteViews for the most-specific routes covering attributed IPs; it does not dump every prefix originated by a shared cloud or CDN ASN. `-d AS16509 --routeviews` remains the intentional way to request a complete ASN prefix inventory. Returned prefixes remain external relationships rather than claimed engagement scope. RouteViews is a P0 action, is not selected by `-b all`, and does not use `-l`. A configured `routeviews.key` is used automatically for PeeringDB-verified authenticated access; otherwise the action uses the guest allowance.
Treat collected OSINT as potentially sensitive. Keep report files, screenshots, and the local database out of source control and share them only within the authorized engagement.
@@ -265,7 +266,7 @@ The JSON report is a single object. Host entries remain plain hostnames or `host
| `cmd` | Always | Command-line arguments used for the run. |
| `hosts` | Always | Discovered hosts; an empty array when none are found. |
| `shodan` | Always | Shodan enrichment rows; an empty array when Shodan is not used. |
| `ips`, `emails`, `vhosts`, `asns` | When non-empty | Network and contact findings. |
| `ips`, `emails`, `vhosts`, `asns`, `prefixes` | When non-empty | Network and contact findings. RouteViews prefixes are external routing relationships, not claimed target scope. |
| `urls` | When non-empty | Discovered URLs from every URL-producing source or action. |
| `people`, `twitter_people`, `linkedin_people` | When non-empty | People and profile findings. |
| `takeover_results` | When non-empty | Optional takeover-check results. |
@@ -283,6 +284,10 @@ JSONL is easy to stream one record at a time. The summary preserves the evidence
Virtual-host observations do not use that string encoding. Each confirmed name remains one `hostname` finding with `actions: ["vhost"]` and a native `observations` array. Several endpoint observations can enrich the same hostname without creating another result kind or count.
RouteViews evidence also uses native observations. Each `prefix` finding has `scope: "external-relationship"`, `actions: ["routeviews"]`, and observed-origin, BGP route, or RPKI validation records. These records describe provider-observed routing, never registration, ownership, authorization, reachability, or target scope.
ASN organization labels from URLScan, ONYPHE, and the Shodan host action are also native observations. Each label remains tied to its provider and the exact hostname or IP that supplied the relationship. ONYPHE's physical hosting and logical WHOIS labels remain separate observations. Conflicting labels are retained for review; organization text never becomes an ASN owner field or a pivot filter. Before RouteViews runs, the exact source-attributed IP relationship—not the organization label—selects automatic network pivots.
Parse recursive DNS findings as JSON objects:
```bash
@@ -295,6 +300,12 @@ List the endpoint observations for each confirmed virtual host:
jq -c 'select(.type == "hostname" and .observations) | {hostname: .value, observations}' report.jsonl
```
List sourced organization labels for ASNs:
```bash
jq -c 'select(.type == "asn" and .observations) | {asn: .value, observations}' report.jsonl
```
Stable Have I Been Pwned breach names use `breach` records. Normalized BuiltWith findings use `framework`, `language`, `server`, `cms`, or `analytics` records. Recursive runs also include classifications and one summary containing query cost, reached depth, zero-yield batches, and the stop reason.
List every JSONL finding as tab-separated type and value columns:
@@ -18,6 +18,6 @@ The existing result-details seam already preserves structured virtual-host evide
## Consequences
Provider actions must emit a canonical prefix result, its scalar origin ASN, matching prefix action provenance, and an observed-origin record before adding BGP-route or RPKI observations. Per-peer ASNs and AS-path members remain evidence fields rather than promoted results. A future provider integration can enrich known ASN or prefix evidence without joining passive source selection or expanding active scope.
Provider actions must emit a canonical prefix result, its scalar origin ASN, matching prefix action provenance, and an observed-origin record before adding BGP-route or RPKI observations. Per-peer ASNs and AS-path members remain evidence fields rather than promoted results. When RouteViews is explicitly selected for an ordinary discovery run, exact IP subjects from sourced ASN attribution become network pivots and only their most-specific matching routes are retained. Complete ASN prefix expansion requires an operator-supplied ASN target; operator-supplied IP and CIDR pivots remain supported. The resulting prefix relationships remain external evidence and cannot expand active scope.
One prefix may carry at most 10,000 observations and 8 MiB of serialized details. These persistence limits fit beneath the existing 10 MiB JSONL import envelope and are independent of provider-response transport budgets.
@@ -0,0 +1,15 @@
# Persist ASN organization attribution as sourced evidence
Status: accepted
## Decision
Store each ASN organization attribution as a typed domain observation backed by a normalized SQLite relationship row linking the run, ASN result, related hostname or IP result, and source execution. Keep SQLAlchemy rows private to `ResultStore`, and project the observation through JSONL and the API instead of treating an organization label as a mutable property of an ASN.
## Why
Provider labels can be missing, change over time, or disagree across sources. A normalized relationship preserves those conflicts and supports cross-run queries without overloading `results.details_json`; the current unreleased schema remains version 8 and initialization creates the additional table when absent. Organization attribution is retained for operator review and never establishes ownership, authorization, or target scope. Its exact IP subject can select an automatic RouteViews pivot, but the organization label itself is not a filter.
## Consequences
Every attribution must reference a canonical ASN result, an exact hostname or IP subject, and the source or action that produced both. Import and persistence fail closed on missing, duplicate, or noncanonical relationships. Provider labels remain independently visible when they conflict and cannot authorize target-scope expansion.
+5
View File
@@ -31,6 +31,9 @@ apikeys:
hibpverified:
key: your-hibp-api-key
routeviews:
key: your-routeviews-api-key
tomba:
key: your-tomba-key
secret: your-tomba-secret
@@ -44,6 +47,8 @@ Provider pricing, quotas, and terms change frequently. Check the provider's curr
`hibpverified` queries [HIBP's authenticated verified-domain endpoint](https://haveibeenpwned.com/API/v3#BreachedDomain). It is selected by its name, the `breaches` capability, and `all`. Without a configured HIBP API key it is skipped like other unavailable keyed sources. Live use requires a user-owned paid HIBP API key and a user-owned domain verified in that account. The keyless `haveibeenpwned` source continues to query only the public breach catalogue.
`routeviews.key` is optional. RouteViews provides authenticated API keys to verified PeeringDB users. `--routeviews` uses the authenticated endpoint and documented 10-request-per-second allowance when the key is configured; otherwise it uses guest access at one request per second. If RouteViews rejects a configured key, the action fails without retrying as a guest; remove the key to select guest access. RouteViews does not document this as a paid subscription.
## Proxies
Edit `~/.theHarvester/proxies.yaml` using `host:port` entries:
+3
View File
@@ -19,12 +19,15 @@ The following options require additional care:
| `-c`, `--dns-brute` | Tries candidate subdomains against DNS. |
| `-t`, `--take-over` | Checks discovered hosts for takeover indicators. |
| `-s`, `--shodan` | Enriches discovered hosts through Shodan. |
| `--routeviews` | Sends discovered IPs with sourced ASN attribution, or an explicitly targeted ASN, IP, or CIDR, to RouteViews and records external routing relationships. |
| `--vhost`, `--vhost-*` | Probes literal IP endpoints with candidate SNI and HTTP `Host` values. |
| `--screenshot DIR` | Opens discovered web services in a browser. |
| `-a`, `--api-scan` | Requests common API paths from the target. |
Use `--dns-resolvers IPS_OR_FILE` to select resolver addresses for DNS brute force, reverse lookup, or recursive DNS without also enabling hostname resolution. The compatible `--dns-resolve` value still selects resolvers and enables hostname resolution.
`--routeviews` is a separately selected P0 provider action. It is never enabled by `-b all` and ignores `-l`; one run is internally bounded to 300 sequential requests and 300 seconds. With no configured key it uses the documented guest rate of one request per second. A configured `routeviews.key` selects PeeringDB-verified authenticated access and the documented 10-request-per-second allowance. When selected for a domain run, it automatically queries harvested IPs backed by sourced IP-to-ASN attribution. It also accepts an ASN, IP, or CIDR supplied as the run target. Harvested IPs without that attribution are not sent, and bare ASN findings are not expanded into complete prefix inventories. IP lookups retain only the most-specific returned prefix, including all origins for a multi-origin prefix. The action does not recursively query returned prefixes and never promotes returned CIDRs into DNS or direct-action scope. Cloud and CDN routes can be useful relationship evidence, but route origins and RPKI states do not establish ownership or authorization.
Use an owned or explicitly authorized domain for active examples. Do not substitute universities, public companies, bounty targets, or reserved example domains for recurring active scans.
## Protect collected data
+13 -1
View File
@@ -79,6 +79,16 @@ Run submission is asynchronous. Lifecycle status is `queued`, `running`, `cancel
P1 DNS and P2 direct options are fields on the same run request. The OpenAPI schema shows their current defaults, limits, and descriptions. The server uses the operator-selected target and does not impose a public-only egress policy.
RouteViews is the explicit P0 `routeviews` action. When selected for a domain run, it automatically enriches harvested IPs that have sourced IP-to-ASN attribution. It also accepts an AS-prefixed ASN or IP address supplied as the run target. Harvested IPs without that attribution are not sent, and bare ASN findings are not expanded into complete prefix inventories. For IP pivots, only the most-specific matching prefix is retained, including every origin when that prefix is multi-origin. An explicit ASN target still requests its complete prefix inventory. The fixed internal budget is 300 sequential requests and 300 seconds; `limit` does not change it. A server-side `routeviews.key` is used automatically for PeeringDB-verified authenticated access at the documented 10-request-per-second allowance; otherwise the action uses guest access at one request per second. Provider credentials cannot be supplied in a run request. Returned prefixes remain external relationships and are never scheduled as DNS or P2 targets. The CLI also accepts a literal CIDR target.
```json
{
"target": "AS64500",
"sources": [],
"routeviews": true
}
```
### Run an action against one result
Screenshots and DNS brute force can run directly against an authorized hostname without repeating discovery. Submit an empty `sources` array and select one action:
@@ -141,6 +151,8 @@ curl -s "http://127.0.0.1:5000/api/v1/runs/import?filename=report.jsonl" \
JSONL is a terminal report, so an import is recorded as completed. The summary retains evidence status, source and action outcomes, and screenshot artifact metadata. Each finding's `sources` and `actions` arrays rebuild result attribution and must name an execution in the summary. Result kinds such as `hostname`, `ip`, and `url` use the same names in JSONL, SQLite, and the API.
An `asn` result may include `organization-attribution` observations from URLScan, ONYPHE, or Shodan. Each observation names its producer and related hostname or IP; it is provider attribution rather than ownership or engagement-scope evidence.
To load every completed run from another theHarvester database:
```bash
@@ -162,7 +174,7 @@ curl -s "http://127.0.0.1:5000/api/v1/runs/$run_id/export" \
-o results.jsonl
```
The first line is the `summary` record, including evidence status, source and action outcomes, and artifacts. Each remaining line is one normalized finding with `type`, `value`, `sources`, and optional `actions`. A hostname confirmed by the `vhost` action adds a native `observations` array; it remains one `hostname` finding. This keeps the file easy to stream with `jq -c` and makes API exports importable again without a format conversion. Lifecycle details and the submitted request remain available from `GET /api/v1/runs/{run_id}`.
The first line is the `summary` record, including evidence status, source and action outcomes, and artifacts. Each remaining line is one normalized finding with `type`, `value`, `sources`, and optional `actions`. A hostname confirmed by the `vhost` action adds native endpoint observations; a RouteViews `prefix` adds native origin, route, and RPKI observations with fixed external-relationship scope. This keeps the file easy to stream with `jq -c` and makes API exports importable again without a format conversion. Lifecycle details and the submitted request remain available from `GET /api/v1/runs/{run_id}`.
## Security boundary
+9 -4
View File
@@ -38,23 +38,28 @@ The database persists across runs. Account for it in engagement cleanup and rete
Completed CLI executions store one normalized terminal record keyed by run UUID. API executions use the same database by default and may override its path with `THEHARVESTER_RUN_DB`. Lifecycle rows keep queue, cancellation, and worker state separate from terminal evidence. Imported JSONL is stored without executing discovery, and source attribution is rebuilt from each finding's `sources` array. A SQLite import copies every completed run after validating the database and keeps the original run IDs.
The normalized persistence model can represent active-action provenance and artifact metadata through five core tables:
The normalized persistence model can represent active-action provenance and artifact metadata through six core tables:
- `runs`: one finite enumeration run;
- `executions`: each passive source or active action represented by the model;
- `results`: deduplicated hostnames, IPs, emails, URLs, and structured outputs;
- `result_origins`: which execution produced each result; and
- `result_origins`: which execution produced each result;
- `asn_attributions`: sourced organization labels linking an ASN result to the exact hostname or IP result supplied by the same execution; and
- `artifacts`: files such as screenshots, linked to their creating action and subject result.
Virtual-host evidence stays inside this model. `results` holds one `hostname` row, `result_origins` links it to the `vhost` action execution, and the result's `details_json` contains the canonical endpoint observation array. If one hostname is distinct on several IP endpoints, it remains one result with several observations.
Current runtime collection populates passive source executions plus DNS, takeover, Shodan, and API endpoint scan executions and origins. Screenshot actions attach file metadata to their captured hostname or URL without creating fake screenshot findings.
RouteViews creates `prefix` results with `scope: external-relationship` and `routeviews` action provenance. Native observations distinguish one ASN-prefix origin claim, one collector/peer BGP route, and one RPKI validation state. They are routing evidence, not registration, ownership, authorization, reachability, or expanded target scope.
URLScan, ONYPHE, and Shodan can attach a provider organization label to an ASN. SQLite stores each relationship in `asn_attributions`; JSONL, the API, CLI output, and HarvestView project the same typed observation. Labels remain time-bound provider evidence, so missing or conflicting values are retained rather than replaced by one ASN owner property. Shodan's documented `org` field is used for the organization label; its separate `isp` field remains part of the existing Shodan payload and is not treated as equivalent.
Every discovered URL is stored as the `url` result kind. Its source or action origins identify whether it came from BuiltWith, GitLab, RocketReach, API scanning, or another producer; provider-specific URL kinds are not stored.
Hostname and IP evidence use the `hostname` and `ip` result kinds in SQLite, JSONL, the API, and HarvestView. A hostname may be the authorized target itself or a subordinate name, so the result kind does not claim that every value is a subdomain.
Two operational tables support the API without changing those five evidence concepts: `run_records` stores queue and lifecycle state, and `run_worker_leases` prevents two local workers from claiming the same queue. Older runless rows remain in `legacy_observations`. SQLite upgrades supported schemas automatically during normal initialization.
Two operational tables support the API without changing those six evidence concepts: `run_records` stores queue and lifecycle state, and `run_worker_leases` prevents two local workers from claiming the same queue. Older runless rows remain in `legacy_observations`. SQLite upgrades supported schemas automatically during normal initialization.
## Screenshots
@@ -62,7 +67,7 @@ Two operational tables support the API without changing those five evidence conc
## API results
`GET /api/v1/runs/{run_id}` returns lifecycle state plus a normalized `results` array. Each result has `type`, `value`, `sources`, and `actions`. A `hostname` found through the `vhost` action also has a native `observations` array with its endpoint, literal-IP context, unknown-host control, and optional body-confirmation evidence. Run-level source and action outcomes remain available in `source_executions` and `action_executions`, while file metadata is returned through `artifacts`. JSONL imports or exports one run, and SQLite import loads completed runs in bulk. Treat runtime `/docs`, `/redoc`, and OpenAPI as the exact request and response reference.
`GET /api/v1/runs/{run_id}` returns lifecycle state plus a normalized `results` array. Each result has `type`, `value`, `sources`, and `actions`. A `hostname` found through the `vhost` action has native endpoint observations; a `prefix` found through RouteViews has native origin, route, and RPKI observations with fixed external-relationship scope. Run-level source and action outcomes remain available in `source_executions` and `action_executions`, while file metadata is returned through `artifacts`. JSONL imports or exports one run, and SQLite import loads completed runs in bulk. Treat runtime `/docs`, `/redoc`, and OpenAPI as the exact request and response reference.
## Handling and sharing
+18
View File
@@ -30,8 +30,10 @@ async def test_process_keeps_only_canonical_individual_ips_and_preserves_routes(
'subnet': '192.0.2.0/24',
'url': ['https://www.example.com/path'],
'asn': 'AS64496',
'organization': 'Example Physical Network',
'geolocus': {
'asn': 'AS64497',
'organization': 'Example Logical Network',
'subnet': '198.51.100.0/24',
'domain': ['geo.example.com'],
},
@@ -56,6 +58,22 @@ async def test_process_keeps_only_canonical_individual_ips_and_preserves_routes(
assert await search.get_ips() == {'192.0.2.10', '2001:db8::10'}
assert await search.get_hostnames() == {'api.example.com', 'geo.example.com', 'www.example.com'}
assert await search.get_asns() == {'AS64496', 'AS64497'}
assert {
(
observation.asn,
observation.organization_label,
observation.subject_kind,
observation.subject_value,
)
for observation in await search.get_asn_attributions()
} == {
(asn, organization, subject_kind, subject_value)
for asn, organization in {
('AS64496', 'Example Physical Network'),
('AS64497', 'Example Logical Network'),
}
for subject_kind, subject_value in {('ip', '192.0.2.10')}
}
assert search.execution_status == 'completed'
assert search.stop_reason is None
+31
View File
@@ -2,11 +2,42 @@ import logging
import socket
import sys
from collections import OrderedDict
from datetime import UTC
import pytest
class TestShodanEngine:
@pytest.mark.asyncio
async def test_shodan_retains_sourced_asn_organization_attribution(self, monkeypatch):
from theHarvester.discovery import shodansearch
class SuccessfulShodan:
def host(self, _ip):
return {
'asn': 'AS64496',
'org': 'Example Transit',
'isp': 'Example ISP Label',
'data': [{'ip_str': '198.51.100.20'}],
}
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: SuccessfulShodan())
search = shodansearch.SearchShodan()
await search.search_ip('192.0.2.10')
attributions = await search.get_asn_attributions()
assert len(attributions) == 1
attribution = next(iter(attributions))
assert attribution.producer_kind == 'action'
assert attribution.producer == 'shodan'
assert attribution.asn == 'AS64496'
assert attribution.organization_label == 'Example Transit'
assert attribution.subject_kind == 'ip'
assert attribution.subject_value == '192.0.2.10'
assert attribution.collected_at.tzinfo is UTC
@pytest.mark.asyncio
async def test_shodan_provider_failure_returns_attributed_empty_evidence(self, monkeypatch, caplog):
from theHarvester.discovery import shodansearch
+52
View File
@@ -1,4 +1,5 @@
import asyncio
from datetime import UTC, datetime, timedelta
from typing import Any
import pytest
@@ -21,6 +22,7 @@ async def test_process_collects_sequential_pages_and_preserves_all_routes(
'ip': '192.0.2.10',
'url': 'https://first.example.com/path',
'asn': 'AS64496',
'asnname': 'Example Transit One',
},
'sort': [200, 'first'],
}
@@ -38,6 +40,7 @@ async def test_process_collects_sequential_pages_and_preserves_all_routes(
'ip': '2001:db8::10',
'url': 'https://second.example.com/',
'asn': 'AS64497',
'asnname': 'Example Transit Two',
},
'sort': [100, 'second'],
}
@@ -66,6 +69,20 @@ async def test_process_collects_sequential_pages_and_preserves_all_routes(
'https://second.example.com/',
}
assert await search.get_asns() == {'AS64496', 'AS64497'}
assert {
(
observation.asn,
observation.organization_label,
observation.subject_kind,
observation.subject_value,
)
for observation in await search.get_asn_attributions()
} == {
('AS64496', 'Example Transit One', 'hostname', 'first.example.com'),
('AS64496', 'Example Transit One', 'ip', '192.0.2.10'),
('AS64497', 'Example Transit Two', 'hostname', 'second.example.com'),
('AS64497', 'Example Transit Two', 'ip', '2001:db8::10'),
}
assert [call['params'] for call in calls] == [
{'q': 'domain:example.com'},
{'q': 'domain:example.com', 'search_after': '200,first'},
@@ -80,6 +97,41 @@ async def test_process_collects_sequential_pages_and_preserves_all_routes(
assert search.stop_reason is None
@pytest.mark.asyncio
async def test_repeated_asn_relationship_is_retained_once_per_source_run(monkeypatch: pytest.MonkeyPatch) -> None:
moments = iter((datetime(2026, 8, 12, tzinfo=UTC), datetime(2026, 8, 12, tzinfo=UTC) + timedelta(seconds=1)))
class TickingDateTime:
@classmethod
def now(cls, timezone: object) -> datetime:
assert timezone is UTC
return next(moments)
page = {
'domain': 'api.example.com',
'ip': '192.0.2.10',
'url': 'https://api.example.com/',
'asn': 'AS64496',
'asnname': 'Example Transit',
}
responses = [
FetcherResponse(body={'results': [{'page': page, 'sort': [2, 'first']}]}, status=200, headers={}),
FetcherResponse(body={'results': [{'page': page, 'sort': [1, 'second']}]}, status=200, headers={}),
FetcherResponse(body={'results': []}, status=200, headers={}),
]
async def fake_fetch(**_kwargs: Any) -> FetcherResponse:
return responses.pop(0)
monkeypatch.setattr(urlscan, 'datetime', TickingDateTime)
monkeypatch.setattr(urlscan.AsyncFetcher, 'fetch', fake_fetch)
search = urlscan.SearchUrlscan('example.com')
await search.process()
assert len(await search.get_asn_attributions()) == 2
@pytest.mark.asyncio
async def test_valid_empty_response_is_completed(monkeypatch: pytest.MonkeyPatch) -> None:
async def fake_fetch(**_kwargs: Any) -> FetcherResponse:
+148
View File
@@ -175,6 +175,7 @@ def test_harvestview_can_submit_overridable_execution_controls(
page.locator('#run-deadline').fill('86400')
page.locator('[name="proxies"]').check()
page.locator('[name="shodan"]').check()
page.locator('[name="routeviews"]').check()
page.locator('[name="dns_lookup"]').check()
page.locator('[name="takeover"]').check()
page.locator('[name="api_scan"]').check()
@@ -207,6 +208,7 @@ def test_harvestview_can_submit_overridable_execution_controls(
'dns_recursive_runtime_seconds': 12.5,
'dns_brute': False,
'shodan': True,
'routeviews': True,
'screenshot': False,
'takeover': True,
'api_scan': True,
@@ -405,6 +407,152 @@ def test_harvestview_renders_grouped_virtual_host_observations(
expect(page.get_by_role('button', name='DNS brute force admin.example.com (P1)')).to_be_visible()
def test_harvestview_renders_sourced_asn_organization_attribution(
harvestview_server_url: str,
page: Page,
) -> None:
run = {
'run_id': 'asn-attribution-run',
'target': 'example.test',
'status': 'completed',
'origin': 'local',
'created_at': '2026-08-12T12:00:00+00:00',
'started_at': '2026-08-12T12:00:01+00:00',
'completed_at': '2026-08-12T12:00:05+00:00',
'cancellation_requested_at': None,
'evidence_status': 'complete',
'result_count': 2,
'activities': ['P0'],
'sources': ['urlscan'],
'request': {'sources': ['urlscan'], 'limit': 25, 'deadline_seconds': 300},
'source_executions': [
{'source': 'urlscan', 'status': 'completed', 'result_count': 2, 'duration_ms': 125},
],
'action_executions': [],
'results': [
{
'type': 'asn',
'value': 'AS64500',
'sources': ['urlscan'],
'actions': [],
'observations': [
{
'type': 'organization-attribution',
'producer_kind': 'source',
'producer': 'urlscan',
'organization_label': 'Example Transit',
'subject': {'type': 'hostname', 'value': 'api.example.test'},
'collected_at': '2026-08-12T12:00:03Z',
}
],
},
{
'type': 'hostname',
'value': 'api.example.test',
'sources': ['urlscan'],
'actions': [],
},
],
'screenshots': [],
'log': '',
'error': None,
}
page.route(f'{harvestview_server_url}/api/v1/runs', lambda route: route.fulfill(json=[run]))
page.route(f'{harvestview_server_url}/api/v1/runs/asn-attribution-run', lambda route: route.fulfill(json=run))
page.goto(f'{harvestview_server_url}/')
page.get_by_role('button', name='ASNs 1').click()
result_row = page.locator('.tabulator-row').first
expect(result_row).to_contain_text('AS64500')
expect(result_row).to_contain_text('Example Transit · source:urlscan · hostname:api.example.test')
def test_harvestview_summarizes_routeviews_prefix_evidence(
harvestview_server_url: str,
page: Page,
) -> None:
run = {
'run_id': 'routeviews-run',
'target': 'example.test',
'status': 'completed',
'origin': 'local',
'created_at': '2026-08-12T12:00:00+00:00',
'started_at': '2026-08-12T12:00:01+00:00',
'completed_at': '2026-08-12T12:00:05+00:00',
'cancellation_requested_at': None,
'evidence_status': 'complete',
'result_count': 2,
'activities': ['P0'],
'sources': ['urlscan'],
'request': {'sources': ['urlscan'], 'limit': 25, 'deadline_seconds': 300, 'routeviews': True},
'source_executions': [],
'action_executions': [
{'action': 'routeviews', 'status': 'completed', 'result_count': 2, 'duration_ms': 125},
],
'results': [
{
'type': 'asn',
'value': 'AS64500',
'sources': [],
'actions': ['routeviews'],
},
{
'type': 'prefix',
'value': '192.0.2.0/24',
'scope': 'external-relationship',
'sources': [],
'actions': ['routeviews'],
'observations': [
{
'type': 'observed-origin',
'action': 'routeviews',
'origin_asn': 'AS64500',
'collected_at': '2026-08-12T12:00:03Z',
},
{
'type': 'rpki-validation',
'action': 'routeviews',
'origin_asn': 'AS64500',
'state': 'valid',
'observed_at': '2026-08-12T11:59:00Z',
'collected_at': '2026-08-12T12:00:03Z',
},
{
'type': 'bgp-route',
'action': 'routeviews',
'origin_asn': 'AS64500',
'collector': 'route-views.example',
'peer_asn': 'AS64496',
'peer_address': '198.51.100.1',
'as_path': '64496 64500',
'communities': '64500:1',
'observed_at': '2026-08-12T11:59:00Z',
'collected_at': '2026-08-12T12:00:03Z',
},
],
},
],
'screenshots': [],
'log': '',
'error': None,
}
page.route(f'{harvestview_server_url}/api/v1/runs', lambda route: route.fulfill(json=[run]))
page.route(f'{harvestview_server_url}/api/v1/runs/routeviews-run', lambda route: route.fulfill(json=run))
page.goto(f'{harvestview_server_url}/')
page.get_by_role('button', name='Network prefixes 1').click()
expect(page.locator('#request-options')).to_contain_text('RouteViews enrichmentSelected')
result_row = page.locator('.tabulator-row').first
expect(result_row).to_contain_text('192.0.2.0/24')
expect(result_row).to_contain_text('AS64500 · RPKI valid')
expect(result_row.get_by_text('1 BGP route observation')).to_be_visible()
result_row.get_by_text('1 BGP route observation').click()
expect(result_row).to_contain_text('route-views.example · peer AS64496 (198.51.100.1)')
expect(result_row).to_contain_text('path 64496 64500 · communities 64500:1')
def test_hostname_actions_queue_isolated_runs(
harvestview_server_url: str,
page: Page,
+117
View File
@@ -155,6 +155,57 @@ def _network_jsonl_result() -> str:
)
def _asn_attribution_jsonl_result() -> str:
summary = {
'type': 'summary',
'run_id': 'e149aef3-f4c5-4145-82f3-71d11d51d9cd',
'target': 'example.test',
'started_at': '2026-08-12T12:00:00Z',
'completed_at': '2026-08-12T12:01:00Z',
'evidence_status': 'complete',
'result_count': 3,
'counts': {'asn': 1, 'hostname': 1, 'ip': 1},
'source_executions': [
{
'source': 'urlscan',
'status': 'completed',
'duration_ms': 1,
'result_count': 3,
'error_type': None,
'stop_reason': None,
}
],
}
findings = [
{
'type': 'asn',
'value': 'AS64500',
'sources': ['urlscan'],
'observations': [
{
'type': 'organization-attribution',
'producer_kind': 'source',
'producer': 'urlscan',
'organization_label': 'Example Network',
'subject': {'type': 'hostname', 'value': 'api.example.test'},
'collected_at': '2026-08-12T12:01:00Z',
},
{
'type': 'organization-attribution',
'producer_kind': 'source',
'producer': 'urlscan',
'organization_label': 'Example Network',
'subject': {'type': 'ip', 'value': '192.0.2.10'},
'collected_at': '2026-08-12T12:01:00Z',
},
],
},
{'type': 'hostname', 'value': 'api.example.test', 'sources': ['urlscan']},
{'type': 'ip', 'value': '192.0.2.10', 'sources': ['urlscan']},
]
return '\n'.join((json.dumps(summary), *(json.dumps(finding) for finding in findings), ''))
@pytest.mark.parametrize(
('finding_type', 'finding_fields'),
[
@@ -336,6 +387,7 @@ def test_source_catalog_exposes_shared_action_activities(tmp_path, monkeypatch)
{'name': 'dns-lookup', 'activity': 'P1'},
{'name': 'dns-recursive', 'activity': 'P1'},
{'name': 'dns-resolve', 'activity': 'P1'},
{'name': 'routeviews', 'activity': 'P0'},
{'name': 'screenshot', 'activity': 'P2'},
{'name': 'shodan', 'activity': 'P0'},
{'name': 'takeover', 'activity': 'P2'},
@@ -361,6 +413,12 @@ def test_openapi_explains_scope_and_execution_controls(tmp_path, monkeypatch) ->
assert 'do not filter' in properties['sources']['description']
assert '/24' in properties['dns_lookup']['description']
assert 'whole run' in properties['deadline_seconds']['description']
assert 'not establish ownership' in properties['routeviews']['description']
assert (
'discovered IPs with sourced ASN attribution, or an explicitly targeted ASN or IP address'
in properties['routeviews']['description']
)
assert 'prefix' not in properties['routeviews']['description']
assert 'three resolver' in properties['dns_recursive_query_limit']['description']
assert 'discovery sources' in properties['proxies']['description']
assert 'configured proxies' in properties['takeover']['description']
@@ -431,6 +489,30 @@ def test_api_scan_can_run_without_discovery_sources(tmp_path, monkeypatch) -> No
RunRequest(target='example.test', sources=[], api_scan=True, api_scan_paths=['https://other.example/api'])
def test_routeviews_accepts_only_an_action_only_asn_target() -> None:
from pydantic import ValidationError
from theHarvester.lib.api.run_models import RunRequest
request = RunRequest(target='as64500', sources=[], routeviews=True)
assert request.target == 'AS64500'
with pytest.raises(ValidationError, match='ASN target requires RouteViews as the only selected work'):
RunRequest(target='AS64500', sources=['crtsh'], routeviews=True)
with pytest.raises(ValidationError, match='ASN target requires RouteViews as the only selected work'):
RunRequest(target='AS64500', sources=[], routeviews=True, shodan=True)
def test_routeviews_hostname_target_requires_a_discovery_source() -> None:
from pydantic import ValidationError
from theHarvester.lib.api.run_models import RunRequest
with pytest.raises(ValidationError, match='RouteViews hostname target requires a discovery source'):
RunRequest(target='api.example.com', sources=[], routeviews=True)
assert RunRequest(target='192.0.2.7', sources=[], routeviews=True).routeviews is True
def test_fresh_api_uses_catalog_takeover_name_and_rejects_unknown_fields() -> None:
from pydantic import ValidationError
@@ -918,6 +1000,41 @@ def test_api_jsonl_round_trip_preserves_structured_network_evidence(tmp_path, mo
assert reimported.json()['results'] == expected
def test_api_jsonl_round_trip_preserves_asn_organization_attribution(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
headers = {'X-API-Key': 'test-key'}
expected = [json.loads(line) for line in _asn_attribution_jsonl_result().splitlines()[1:]]
for finding in expected:
finding['actions'] = []
with TestClient(api.app, client=('127.0.0.21', 50000)) as client:
imported = client.post(
'/api/v1/runs/import',
params={'filename': 'asn-attribution.jsonl'},
headers=headers,
content=_asn_attribution_jsonl_result(),
)
assert imported.status_code == 201, imported.text
exported = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/export', headers=headers)
reimported = client.post(
'/api/v1/runs/import',
params={'filename': 'asn-attribution-round-trip.jsonl'},
headers=headers,
content=exported.content,
)
assert imported.json()['results'] == expected
assert [json.loads(line) for line in exported.text.splitlines()[1:]] == [
{key: value for key, value in finding.items() if key != 'actions'} for finding in expected
]
assert reimported.status_code == 201
assert reimported.json()['results'] == expected
@pytest.mark.parametrize(
('field', 'value'),
[
+89
View File
@@ -9,6 +9,7 @@ import pytest
from theHarvester.lib import database as database_module
from theHarvester.lib.active_evidence import ActionExecution, ActiveEvidence, ArtifactReference
from theHarvester.lib.asn_attribution import AsnAttributionObservation
from theHarvester.lib.completed_result import CompletedResult, ResultObservation, SourceExecution
from theHarvester.lib.database import (
DuplicateRunError,
@@ -474,6 +475,93 @@ async def test_structured_network_evidence_round_trips_in_the_results_table(tmp_
assert json.loads(stored_details) == network_observation_details(network_observations)
@pytest.mark.asyncio
async def test_asn_organization_attribution_round_trips_in_a_normalized_table(tmp_path) -> None:
database = tmp_path / 'stash.sqlite'
store = ResultStore(database)
await store.initialize()
collected_at = datetime(2026, 8, 12, 12, 1, tzinfo=UTC)
attribution = AsnAttributionObservation(
'source',
'urlscan',
'AS64500',
'Example Network',
'ip',
'192.0.2.10',
collected_at,
)
result = CompletedResult.finish(
target='example.com',
started_at=collected_at,
completed_at=collected_at,
groups={'asn': ['AS64500'], 'ip': ['192.0.2.10']},
source_executions=(SourceExecution('urlscan', 'completed', 1, 2),),
observations=(
ResultObservation('urlscan', 'asn', 'AS64500'),
ResultObservation('urlscan', 'ip', '192.0.2.10'),
),
asn_attributions=(attribution,),
)
await store.save_run(result)
assert await store.load_run(result.run_id) == result
with sqlite3.connect(database) as db:
row = db.execute(
'SELECT organization_label, collected_at, asn_result_position, subject_result_position, '
'execution_position FROM asn_attributions WHERE run_id = ?',
(str(result.run_id),),
).fetchone()
result_rows = db.execute(
'SELECT position, kind, value FROM results WHERE run_id = ? ORDER BY position',
(str(result.run_id),),
).fetchall()
assert result_rows == [(0, 'asn', 'AS64500'), (1, 'ip', '192.0.2.10')]
assert row == ('Example Network', '2026-08-12T12:01:00+00:00', 0, 1, 0)
@pytest.mark.asyncio
@pytest.mark.parametrize('corruption', ['non-asn-result', 'duplicate-row'])
async def test_loading_corrupt_asn_attribution_fails_closed(tmp_path, corruption: str) -> None:
database = tmp_path / 'stash.sqlite'
store = ResultStore(database)
await store.initialize()
collected_at = datetime(2026, 8, 12, 12, 1, tzinfo=UTC)
attribution = AsnAttributionObservation('source', 'urlscan', 'AS64500', 'Example Network', 'ip', '192.0.2.10', collected_at)
result = CompletedResult.finish(
target='example.com',
started_at=collected_at,
completed_at=collected_at,
groups={'asn': ['AS64500'], 'ip': ['192.0.2.10']},
source_executions=(SourceExecution('urlscan', 'completed', 1, 2),),
observations=(
ResultObservation('urlscan', 'asn', 'AS64500'),
ResultObservation('urlscan', 'ip', '192.0.2.10'),
),
asn_attributions=(attribution,),
)
await store.save_run(result)
with sqlite3.connect(database) as db:
if corruption == 'non-asn-result':
db.execute(
'UPDATE asn_attributions SET asn_result_position = subject_result_position WHERE run_id = ?',
(str(result.run_id),),
)
else:
db.execute(
'INSERT INTO asn_attributions '
'(run_id, position, asn_result_position, subject_result_position, execution_position, '
'organization_label, collected_at) '
'SELECT run_id, position + 1, asn_result_position, subject_result_position, execution_position, '
'organization_label, collected_at FROM asn_attributions WHERE run_id = ?',
(str(result.run_id),),
)
db.commit()
with pytest.raises(ResultStoreError, match='Persisted ASN attribution is invalid'):
await store.load_run(result.run_id)
@pytest.mark.asyncio
async def test_loading_prefix_details_with_vhost_provenance_fails_closed(tmp_path) -> None:
database = tmp_path / 'stash.sqlite'
@@ -853,6 +941,7 @@ async def test_mixed_source_action_artifact_round_trip_uses_unified_tables(tmp_p
'executions',
'results',
'result_origins',
'asn_attributions',
'artifacts',
'legacy_observations',
'run_records',
+130
View File
@@ -6,6 +6,11 @@ from uuid import UUID
import pytest
from theHarvester.lib.active_evidence import ActionExecution, ActionObservation, ActiveEvidence, ArtifactReference
from theHarvester.lib.asn_attribution import (
AsnAttributionObservation,
asn_attribution_details,
parse_asn_attribution_details,
)
from theHarvester.lib.completed_result import CompletedResult, ResultObservation, SourceExecution, parse_result_jsonl
from theHarvester.lib.network_evidence import (
MAX_NETWORK_DETAILS_BYTES,
@@ -15,6 +20,7 @@ from theHarvester.lib.network_evidence import (
NetworkEvidenceLimitError,
PrefixOriginObservation,
RpkiValidationObservation,
canonical_network_observations,
network_observation_details,
parse_network_observation_details,
parse_network_observation_json,
@@ -426,6 +432,95 @@ def test_completed_result_groups_canonical_network_evidence_by_prefix() -> None:
assert parsed_findings == [{**records[1], 'actions': []}, records[2]]
def test_completed_result_groups_sourced_asn_organization_attribution() -> None:
collected_at = datetime(2026, 8, 12, 12, 1, tzinfo=UTC)
attribution = AsnAttributionObservation(
'source',
'urlscan',
'64500',
'Example Network',
'hostname',
'api.example.com',
collected_at,
)
result = CompletedResult.finish(
target='example.com',
started_at=collected_at,
completed_at=collected_at,
groups={'asn': ['AS64500'], 'hostname': ['api.example.com']},
source_executions=(SourceExecution('urlscan', 'completed', 1, 2),),
observations=(
ResultObservation('urlscan', 'asn', 'AS64500'),
ResultObservation('urlscan', 'hostname', 'api.example.com'),
),
asn_attributions=(attribution, attribution),
)
records = [json.loads(line) for line in result.jsonl().splitlines()]
_summary, parsed_findings = parse_result_jsonl(result.jsonl())
assert result.asn_attributions == (attribution,)
assert records[1] == {
'type': 'asn',
'value': 'AS64500',
'sources': ['urlscan'],
'observations': [
{
'type': 'organization-attribution',
'producer_kind': 'source',
'producer': 'urlscan',
'organization_label': 'Example Network',
'subject': {'type': 'hostname', 'value': 'api.example.com'},
'collected_at': '2026-08-12T12:01:00Z',
}
],
}
assert parsed_findings == [{**record, 'actions': []} for record in records[1:]]
@pytest.mark.parametrize(
('field', 'value'),
[
('organization_label', 'Example\x00Network'),
('organization_label', 'Example\u202eNetwork'),
('organization_label', 'x' * 256),
('subject_value', 'not-an-ip'),
],
)
def test_asn_organization_attribution_rejects_unsafe_or_noncanonical_values(field: str, value: str) -> None:
kwargs = {
'producer_kind': 'source',
'producer': 'urlscan',
'asn': 'AS64500',
'organization_label': 'Example Network',
'subject_kind': 'ip',
'subject_value': '192.0.2.10',
'collected_at': datetime(2026, 8, 12, 12, 1, tzinfo=UTC),
}
kwargs[field] = value
with pytest.raises(ValueError):
AsnAttributionObservation(**kwargs)
def test_asn_organization_attribution_does_not_apply_an_arbitrary_result_count_ceiling() -> None:
collected_at = datetime(2026, 8, 12, 12, 1, tzinfo=UTC)
observations = tuple(
AsnAttributionObservation(
'source',
'urlscan',
'AS64500',
'Example Network',
'hostname',
f'host-{index:05d}.example.com',
collected_at,
)
for index in range(10_001)
)
assert parse_asn_attribution_details('AS64500', asn_attribution_details(observations)) == observations
def test_bgp_route_rejects_provider_time_after_collection() -> None:
collected_at = datetime(2026, 8, 11, 12, 0, tzinfo=UTC)
@@ -613,6 +708,41 @@ def test_network_details_reject_conflicting_rpki_states() -> None:
parse_network_observation_details('192.0.2.0/24', details)
def test_rpki_canonicalization_preserves_time_bound_state_changes_deterministically() -> None:
collected_at = datetime(2026, 8, 11, 12, 2, tzinfo=UTC)
observations = (
RpkiValidationObservation(
'routeviews', '192.0.2.0/24', 'AS64500', 'valid', collected_at - timedelta(minutes=2), collected_at
),
RpkiValidationObservation(
'routeviews', '192.0.2.0/24', 'AS64500', 'valid', collected_at - timedelta(minutes=1), collected_at
),
RpkiValidationObservation('routeviews', '192.0.2.0/24', 'AS64500', 'invalid', collected_at, collected_at),
)
canonical = canonical_network_observations(observations)
assert [(item.state, item.observed_at) for item in canonical] == [
('valid', collected_at - timedelta(minutes=2)),
('invalid', collected_at),
]
assert canonical_network_observations(tuple(reversed(observations))) == canonical
def test_rpki_incremental_dedup_still_rejects_a_same_time_conflict() -> None:
collected_at = datetime(2026, 8, 11, 12, 2, tzinfo=UTC)
first_time = collected_at - timedelta(minutes=2)
second_time = collected_at - timedelta(minutes=1)
accumulator = NetworkEvidenceAccumulator()
assert accumulator.add(RpkiValidationObservation('routeviews', '192.0.2.0/24', 'AS64500', 'valid', first_time, collected_at))
assert not accumulator.add(
RpkiValidationObservation('routeviews', '192.0.2.0/24', 'AS64500', 'valid', second_time, collected_at)
)
with pytest.raises(ValueError, match='conflicting RPKI states'):
accumulator.add(RpkiValidationObservation('routeviews', '192.0.2.0/24', 'AS64500', 'invalid', second_time, collected_at))
def test_network_evidence_accumulator_owns_incremental_deduplication_and_limits() -> None:
collected_at = datetime(2026, 8, 11, 12, 1, tzinfo=UTC)
accumulator = NetworkEvidenceAccumulator(max_observations_per_prefix=1)
+16 -1
View File
@@ -318,10 +318,11 @@ def test_user_agent_policy_separates_provider_and_browser_identities() -> None:
("bevigil_key", "bevigil-key"),
("censys_key", ("censys-id", "censys-secret")),
("fofa_key", ("fofa-key", "fofa-email")),
("routeviews_key", "routeviews-key"),
("tomba_key", ("tomba-key", "tomba-secret")),
],
)
def test_api_key_accessors_delegate_to_shared_mapping(monkeypatch, accessor_name: str, expected: Any):
def test_api_key_accessors_read_configured_values(monkeypatch, accessor_name: str, expected: Any):
monkeypatch.setattr(
Core,
'api_keys',
@@ -330,6 +331,7 @@ def test_api_key_accessors_delegate_to_shared_mapping(monkeypatch, accessor_name
'bevigil': {'key': 'bevigil-key'},
'censys': {'id': 'censys-id', 'secret': 'censys-secret'},
'fofa': {'key': 'fofa-key', 'email': 'fofa-email'},
'routeviews': {'key': 'routeviews-key'},
'tomba': {'key': 'tomba-key', 'secret': 'tomba-secret'},
}
),
@@ -339,6 +341,19 @@ def test_api_key_accessors_delegate_to_shared_mapping(monkeypatch, accessor_name
assert accessor() == expected
@pytest.mark.parametrize('configured_value', [None, '', ' ', 10])
def test_routeviews_key_ignores_missing_or_invalid_optional_credentials(monkeypatch, configured_value: object) -> None:
monkeypatch.setattr(Core, 'api_keys', staticmethod(lambda: {'routeviews': {'key': configured_value}}))
assert Core.routeviews_key() is None
def test_routeviews_key_strips_configuration_whitespace(monkeypatch) -> None:
monkeypatch.setattr(Core, 'api_keys', staticmethod(lambda: {'routeviews': {'key': ' routeviews-key '}}))
assert Core.routeviews_key() == 'routeviews-key'
@pytest.mark.asyncio
async def test_fetch_creates_session_with_default_headers(monkeypatch) -> None:
async def fail_if_fetch_sleeps(seconds: float) -> None:
+9
View File
@@ -41,6 +41,15 @@ def test_enumeration_options_preserve_explicit_transport_values() -> None:
assert options.screenshot == '/tmp/managed-screenshots'
def test_routeviews_is_an_explicit_passive_action_independent_of_source_limits() -> None:
options = EnumerationOptions.from_namespace(Namespace(domain='example.com', source=None, limit=25, routeviews=True))
assert options.routeviews is True
assert options.limit == 25
assert selected_action_names({'routeviews': True, 'limit': 1}) == ('routeviews',)
assert selected_action_names({'limit': 10_000}) == ()
def test_enumeration_options_preserve_virtual_host_inputs() -> None:
options = EnumerationOptions.from_namespace(
Namespace(
+602
View File
@@ -0,0 +1,602 @@
from __future__ import annotations
import asyncio
from datetime import UTC, datetime, timedelta
from typing import Any
import pytest
import theHarvester.lib.routeviews as routeviews_module
from theHarvester.lib.core import FetcherResponse
from theHarvester.lib.network_evidence import (
BgpRouteObservation,
PrefixOriginObservation,
RpkiValidationObservation,
)
from theHarvester.lib.routeviews import RouteViewsCancelled, enrich_routeviews
def install_runtime(monkeypatch, responses: list[FetcherResponse | BaseException]):
calls: list[tuple[str, dict[str, Any]]] = []
elapsed = [0.0]
collected_at = datetime(2026, 8, 11, 12, tzinfo=UTC)
async def fetch_json(url: str, **kwargs: Any) -> FetcherResponse:
calls.append((url, kwargs))
response = responses.pop(0)
if isinstance(response, BaseException):
raise response
return response
async def sleep(seconds: float) -> None:
elapsed[0] += seconds
monkeypatch.setattr(routeviews_module, '_fetch_json', fetch_json)
monkeypatch.setattr(routeviews_module, '_sleep', sleep)
monkeypatch.setattr(routeviews_module, '_monotonic', lambda: elapsed[0])
monkeypatch.setattr(routeviews_module, '_now', lambda: collected_at + timedelta(seconds=elapsed[0]))
return calls, elapsed
def response(body: object, status: int = 200, headers: dict[str, str] | None = None) -> FetcherResponse:
return FetcherResponse(body=body, status=status, headers=headers or {})
@pytest.mark.asyncio
async def test_routeviews_collects_asn_prefixes_and_rpki_without_prefix_fanout(monkeypatch) -> None:
calls, elapsed = install_runtime(
monkeypatch,
[
response(['192.0.2.0/24', '2001:db8::/32', '192.0.2.0/24']),
response(
{
'64500': {
'prefix': [
{'192.0.2.0/24': 'valid'},
{'2001:db8::/32': 'notfound'},
],
'timestamp': '2026-08-11T12:00:00+00:00',
}
}
),
],
)
result = await enrich_routeviews(['64500'], [])
assert result.status == 'completed'
assert result.stop_reason is None
assert result.request_count == 2
assert result.prefixes == ('192.0.2.0/24', '2001:db8::/32')
assert result.origin_asns == ('AS64500',)
assert sum(isinstance(item, PrefixOriginObservation) for item in result.observations) == 2
assert {(item.prefix, item.state) for item in result.observations if isinstance(item, RpkiValidationObservation)} == {
('192.0.2.0/24', 'valid'),
('2001:db8::/32', 'not-found'),
}
assert [url for url, _kwargs in calls] == [
'https://api.routeviews.org/guest/asn/64500',
'https://api.routeviews.org/guest/rpki',
]
assert calls[1][1]['params'] == {'asn': '64500'}
assert elapsed[0] == 1.0
@pytest.mark.asyncio
async def test_routeviews_uses_configured_key_for_authenticated_access(monkeypatch) -> None:
calls, elapsed = install_runtime(
monkeypatch,
[
response(['192.0.2.0/24']),
response({'64500': None}),
],
)
result = await enrich_routeviews(['AS64500'], [], api_key='routeviews-key')
assert result.status == 'completed'
assert [url for url, _kwargs in calls] == [
'https://api.routeviews.org/asn/64500',
'https://api.routeviews.org/rpki',
]
assert [kwargs['headers'] for _url, kwargs in calls] == [
{'Api-Key': 'routeviews-key'},
{'Api-Key': 'routeviews-key'},
]
assert elapsed[0] == 0.1
@pytest.mark.asyncio
async def test_routeviews_invalid_configured_key_fails_without_guest_downgrade(monkeypatch) -> None:
calls, _elapsed = install_runtime(monkeypatch, [response(None, status=401)])
result = await enrich_routeviews(['AS64500'], [], api_key='invalid-key')
assert result.status == 'failed'
assert result.stop_reason == 'http-401'
assert calls == [
(
'https://api.routeviews.org/asn/64500',
{
'params': '',
'headers': {'Api-Key': 'invalid-key'},
'request_timeout': 30,
},
)
]
@pytest.mark.asyncio
async def test_routeviews_uses_authenticated_access_for_prefix_seeds(monkeypatch) -> None:
calls, elapsed = install_runtime(monkeypatch, [response([]), response([])])
result = await enrich_routeviews(
[],
['192.0.2.0/24', '198.51.100.0/24'],
api_key='routeviews-key',
)
assert result.status == 'completed'
assert calls == [
(
'https://api.routeviews.org/prefix/192.0.2.0%2F24',
{
'params': {'strict-match': 'yes'},
'headers': {'Api-Key': 'routeviews-key'},
'request_timeout': 30,
},
),
(
'https://api.routeviews.org/prefix/198.51.100.0%2F24',
{
'params': {'strict-match': 'yes'},
'headers': {'Api-Key': 'routeviews-key'},
'request_timeout': 30,
},
),
]
assert elapsed[0] == 0.1
@pytest.mark.asyncio
async def test_routeviews_collects_moas_routes_and_strict_prefix_evidence(monkeypatch) -> None:
calls, _elapsed = install_runtime(
monkeypatch,
[
response(
[
{
'prefix': '192.0.2.0/24',
'origin_asn': 64500,
'rpki_state': 'valid',
'rpki_roas': None,
'reporting_peers': [
{
'peer_asn': 64496,
'peer_addr': '198.51.100.1',
'collector': 'route-views.example',
'as_path': ' 64496 64500 ',
'communities': '',
'timestamp': '2026-08-11T11:59:00Z',
}
],
},
{
'prefix': '192.0.2.0/24',
'origin_asn': 64501,
'rpki_state': 'not-found',
'rpki_roas': None,
'reporting_peers': [],
},
]
)
],
)
result = await enrich_routeviews([], ['192.0.2.7/24'])
assert result.status == 'completed'
assert result.prefixes == ('192.0.2.0/24',)
assert result.origin_asns == ('AS64500', 'AS64501')
route = next(item for item in result.observations if isinstance(item, BgpRouteObservation))
assert route.as_path == ' 64496 64500 '
assert route.communities == ''
assert {item.state for item in result.observations if isinstance(item, RpkiValidationObservation)} == {'valid', 'not-found'}
assert calls == [
(
'https://api.routeviews.org/guest/prefix/192.0.2.0%2F24',
{
'params': {'strict-match': 'yes'},
'request_timeout': 30,
},
)
]
@pytest.mark.asyncio
async def test_routeviews_uses_longest_match_for_known_ip(monkeypatch) -> None:
calls, _elapsed = install_runtime(
monkeypatch,
[
response(
[
{
'prefix': '0.0.0.0/0',
'origin_asn': 64500,
'rpki_state': 'not-found',
'reporting_peers': [],
},
{
'prefix': '192.0.0.0/16',
'origin_asn': 64501,
'rpki_state': 'valid',
'reporting_peers': [],
},
{
'prefix': '192.0.2.0/24',
'origin_asn': 64502,
'rpki_state': 'valid',
'reporting_peers': [],
},
{
'prefix': '192.0.2.0/24',
'origin_asn': 64503,
'rpki_state': 'not-found',
'reporting_peers': [],
},
]
)
],
)
result = await enrich_routeviews([], ['192.0.2.7'])
assert result.status == 'completed'
assert result.stop_reason is None
assert result.prefixes == ('192.0.2.0/24',)
assert result.origin_asns == ('AS64502', 'AS64503')
assert calls[0][0] == 'https://api.routeviews.org/guest/prefix/192.0.2.7%2F32'
assert calls[0][1]['params'] == ''
@pytest.mark.asyncio
async def test_routeviews_deduplicates_shared_prefix_evidence_across_ip_seeds(monkeypatch) -> None:
shared_prefix = [
{
'prefix': '192.0.2.0/24',
'origin_asn': 64500,
'rpki_state': 'valid',
'reporting_peers': [
{
'peer_asn': 64496,
'peer_addr': '198.51.100.1',
'collector': 'route-views.example',
'as_path': '64496 64500',
'communities': '64500:1',
'timestamp': '2026-08-11T11:59:00Z',
}
],
}
]
install_runtime(monkeypatch, [response(shared_prefix), response(shared_prefix)])
result = await enrich_routeviews([], ['192.0.2.7', '192.0.2.8'])
assert result.request_count == 2
assert result.prefixes == ('192.0.2.0/24',)
assert sum(isinstance(item, PrefixOriginObservation) for item in result.observations) == 1
assert sum(isinstance(item, BgpRouteObservation) for item in result.observations) == 1
assert sum(isinstance(item, RpkiValidationObservation) for item in result.observations) == 1
@pytest.mark.asyncio
async def test_routeviews_records_each_response_collection_time(monkeypatch) -> None:
install_runtime(
monkeypatch,
[
response([]),
response(
[
{
'prefix': '192.0.2.0/24',
'origin_asn': 64500,
'rpki_state': 'valid',
'reporting_peers': [
{
'peer_asn': 64496,
'peer_addr': '198.51.100.1',
'collector': 'route-views.example',
'as_path': '64496 64500',
'communities': '',
'timestamp': '2026-08-11T12:00:00.500Z',
}
],
}
]
),
],
)
result = await enrich_routeviews([], ['192.0.2.7', '192.0.2.8'])
route = next(item for item in result.observations if isinstance(item, BgpRouteObservation))
assert route.collected_at == datetime(2026, 8, 11, 12, 0, 1, tzinfo=UTC)
@pytest.mark.asyncio
async def test_routeviews_accepts_null_rpki_as_no_data(monkeypatch) -> None:
install_runtime(monkeypatch, [response([]), response({'64500': None})])
result = await enrich_routeviews(['AS64500'], [])
assert result.status == 'completed'
assert result.stop_reason == 'no-results'
assert result.observations == ()
@pytest.mark.asyncio
async def test_routeviews_reports_rate_limit_without_parsing_error_body(monkeypatch) -> None:
install_runtime(monkeypatch, [response(None, status=429, headers={'retry-after': '60'})])
result = await enrich_routeviews(['AS64500'], [])
assert result.status == 'rate-limited'
assert result.error_type == 'HTTPStatusError'
assert result.stop_reason == 'http-429'
assert result.request_count == 1
@pytest.mark.asyncio
async def test_routeviews_reports_terminal_rate_limit_after_recoverable_error(monkeypatch) -> None:
install_runtime(monkeypatch, [response(None, status=503), response(None, status=429)])
result = await enrich_routeviews(['AS64500'], [])
assert result.status == 'rate-limited'
assert result.error_count == 2
assert result.error_type == 'HTTPStatusError'
assert result.stop_reason == 'http-429'
@pytest.mark.asyncio
async def test_routeviews_preserves_valid_prefixes_before_malformed_response(monkeypatch) -> None:
install_runtime(monkeypatch, [response(['192.0.2.0/24']), response({'64500': {'prefix': 'bad'}})])
result = await enrich_routeviews(['AS64500'], [])
assert result.status == 'partial'
assert result.prefixes == ('192.0.2.0/24',)
assert result.error_type == 'ValueError'
assert result.stop_reason == 'invalid-response'
assert result.error_count == 1
@pytest.mark.asyncio
async def test_routeviews_continues_after_one_seed_fails(monkeypatch) -> None:
install_runtime(
monkeypatch,
[
response(None, status=503),
response({'64500': None}),
response(['198.51.100.0/24']),
response({'64501': None}),
],
)
result = await enrich_routeviews(['AS64500', 'AS64501'], [])
assert result.status == 'partial'
assert result.prefixes == ('198.51.100.0/24',)
assert result.origin_asns == ('AS64501',)
assert result.request_count == 4
assert result.error_count == 1
assert result.error_type == 'HTTPStatusError'
assert result.stop_reason == 'http-503'
@pytest.mark.asyncio
async def test_routeviews_rejects_prefix_evidence_unrelated_to_requested_seed(monkeypatch) -> None:
install_runtime(
monkeypatch,
[
response(
[
{
'prefix': '203.0.113.0/24',
'origin_asn': 64500,
'rpki_state': 'not-found',
'rpki_roas': None,
'reporting_peers': [],
},
{
'prefix': '192.0.2.0/24',
'origin_asn': 64501,
'rpki_state': 'valid',
'rpki_roas': None,
'reporting_peers': [],
},
]
)
],
)
result = await enrich_routeviews([], ['192.0.2.7'])
assert result.status == 'partial'
assert result.prefixes == ('192.0.2.0/24',)
assert result.origin_asns == ('AS64501',)
assert result.error_count == 1
assert result.stop_reason == 'invalid-response'
@pytest.mark.asyncio
async def test_routeviews_keeps_routes_when_rpki_or_a_sibling_peer_is_invalid(monkeypatch) -> None:
install_runtime(
monkeypatch,
[
response(
[
{
'prefix': '192.0.2.0/24',
'origin_asn': 64500,
'rpki_state': None,
'rpki_roas': None,
'reporting_peers': [
{'peer_asn': 'bad'},
{
'peer_asn': 64496,
'peer_addr': '198.51.100.1',
'collector': 'route-views.example',
'as_path': '64496 64500',
'communities': '',
'timestamp': '2026-08-11T11:59:00Z',
},
],
}
]
)
],
)
result = await enrich_routeviews([], ['192.0.2.0/24'])
assert result.status == 'partial'
assert result.prefixes == ('192.0.2.0/24',)
assert sum(isinstance(item, BgpRouteObservation) for item in result.observations) == 1
assert not any(isinstance(item, RpkiValidationObservation) for item in result.observations)
assert result.error_count == 2
assert result.stop_reason == 'invalid-response'
@pytest.mark.asyncio
async def test_routeviews_cancellation_carries_completed_prefix_evidence(monkeypatch) -> None:
cancelled = asyncio.CancelledError()
install_runtime(monkeypatch, [response(['192.0.2.0/24']), cancelled])
with pytest.raises(RouteViewsCancelled) as raised:
await enrich_routeviews(['AS64500'], [])
assert raised.value.result.status == 'partial'
assert raised.value.result.prefixes == ('192.0.2.0/24',)
assert raised.value.result.error_type == 'CancelledError'
assert raised.value.result.stop_reason == 'cancelled'
@pytest.mark.asyncio
async def test_routeviews_cancellation_overrides_an_earlier_recoverable_error(monkeypatch) -> None:
install_runtime(monkeypatch, [response(None, status=503), asyncio.CancelledError()])
with pytest.raises(RouteViewsCancelled) as raised:
await enrich_routeviews(['AS64500'], [])
assert raised.value.result.error_count == 2
assert raised.value.result.error_type == 'CancelledError'
assert raised.value.result.stop_reason == 'cancelled'
@pytest.mark.asyncio
async def test_routeviews_fixed_request_budget_preserves_prefix(monkeypatch) -> None:
install_runtime(monkeypatch, [response(['192.0.2.0/24'])])
monkeypatch.setattr(routeviews_module, 'MAX_ROUTEVIEWS_REQUESTS', 1)
result = await enrich_routeviews(['AS64500'], [])
assert result.status == 'partial'
assert result.prefixes == ('192.0.2.0/24',)
assert result.error_type == 'RouteViewsLimitError'
assert result.stop_reason == 'request-limit'
@pytest.mark.asyncio
async def test_routeviews_runtime_budget_preserves_prefix(monkeypatch) -> None:
install_runtime(monkeypatch, [response(['192.0.2.0/24'])])
monkeypatch.setattr(routeviews_module, 'MAX_ROUTEVIEWS_RUNTIME_SECONDS', 0.5)
result = await enrich_routeviews(['AS64500'], [])
assert result.status == 'partial'
assert result.prefixes == ('192.0.2.0/24',)
assert result.stop_reason == 'runtime-limit'
@pytest.mark.asyncio
async def test_routeviews_per_prefix_evidence_limit_returns_partial(monkeypatch) -> None:
install_runtime(
monkeypatch,
[
response(
[
{
'prefix': '192.0.2.0/24',
'origin_asn': 64500,
'rpki_state': 'valid',
'rpki_roas': None,
'reporting_peers': [],
}
]
)
],
)
monkeypatch.setattr(routeviews_module, 'MAX_NETWORK_OBSERVATIONS_PER_PREFIX', 1)
result = await enrich_routeviews([], ['192.0.2.0/24'])
assert result.status == 'partial'
assert result.prefixes == ('192.0.2.0/24',)
assert len(result.observations) == 1
assert result.stop_reason == 'result-limit'
@pytest.mark.asyncio
async def test_routeviews_cumulative_provider_budget_stops_before_next_request(monkeypatch) -> None:
calls, _elapsed = install_runtime(monkeypatch, [response(['192.0.2.0/24'])])
monkeypatch.setattr(routeviews_module, 'MAX_ROUTEVIEWS_RUN_JSON_BYTES', 1)
result = await enrich_routeviews(['AS64500'], [])
assert result.status == 'failed'
assert result.stop_reason == 'result-limit'
assert result.request_count == 1
assert len(calls) == 1
@pytest.mark.asyncio
async def test_routeviews_rejects_timestamp_that_overflows_utc_conversion(monkeypatch) -> None:
install_runtime(
monkeypatch,
[
response([]),
response({'64500': {'prefix': [], 'timestamp': '0001-01-01T00:00:00+01:00'}}),
],
)
result = await enrich_routeviews(['AS64500'], [])
assert result.status == 'failed'
assert result.error_count == 1
assert result.stop_reason == 'invalid-response'
@pytest.mark.asyncio
async def test_routeviews_skips_when_inputs_are_empty_or_invalid(monkeypatch) -> None:
calls, _elapsed = install_runtime(monkeypatch, [])
result = await enrich_routeviews(['not-an-asn'], ['not-an-ip'])
assert result.status == 'skipped'
assert result.stop_reason == 'no-input'
assert result.request_count == 0
assert calls == []
@pytest.mark.asyncio
async def test_routeviews_bounds_input_before_scheduling_requests(monkeypatch) -> None:
calls, _elapsed = install_runtime(monkeypatch, [response([]), response({'64500': None})])
monkeypatch.setattr(routeviews_module, 'MAX_ROUTEVIEWS_INPUT_ITEMS', 1)
result = await enrich_routeviews(['AS64500', 'AS64501'], [])
assert result.status == 'failed'
assert result.error_count == 1
assert result.stop_reason == 'input-limit'
assert len(calls) == 2
+54
View File
@@ -732,6 +732,31 @@ def test_authenticated_operator_can_queue_screenshot_only_run(tmp_path, monkeypa
assert response.json()['activities'] == ['P2']
def test_authenticated_operator_can_queue_routeviews_only_ip_run(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api, run_worker
async def no_op() -> None:
return None
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setattr(api, 'start_worker', no_op)
monkeypatch.setattr(api, 'stop_worker', no_op)
monkeypatch.setattr(run_worker, 'worker_enabled', lambda: True)
monkeypatch.setattr(run_worker, '_worker_task', type('RunningTask', (), {'done': lambda self: False})())
with TestClient(api.app) as client:
response = client.post(
'/api/v1/runs',
headers={'X-API-Key': 'test-key'},
json={'target': '192.0.2.7', 'sources': [], 'routeviews': True},
)
assert response.status_code == 201
assert response.json()['activities'] == ['P0']
assert response.json()['request']['routeviews'] is True
def test_dns_brute_run_accepts_operator_resolver_list(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api, run_worker
@@ -799,6 +824,35 @@ def test_dns_brute_child_uses_operator_resolver_list(tmp_path, monkeypatch) -> N
assert received_options[0].dns_resolvers == ('192.0.2.53',)
def test_routeviews_child_receives_explicit_action_without_source_limit_controls(tmp_path, monkeypatch) -> None:
from theHarvester import __main__ as main_module
from theHarvester.lib.api import run_worker
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
from theHarvester.lib.completed_result import CompletedResult
received_options = []
async def fake_start(options, **_kwargs):
received_options.append(options)
now = datetime.now(UTC)
return (CompletedResult.finish(target=options.domain, started_at=now, completed_at=now, groups={}),)
monkeypatch.setattr(main_module, 'start', fake_start)
async def scenario() -> None:
store = RunStore(tmp_path / 'runs.sqlite')
created = await store.create(RunRequest(target='192.0.2.7', sources=[], routeviews=True, limit=9_999))
assert await store.claim_next() is not None
await run_worker._child_execute(created['run_id'], store.database)
asyncio.run(scenario())
assert received_options[0].source == ''
assert received_options[0].routeviews is True
assert received_options[0].limit == 9_999
def test_virtual_host_request_normalizes_a_self_contained_target_action() -> None:
from pydantic import ValidationError
+280 -1
View File
@@ -3,6 +3,7 @@ import json
import logging
import sys
import xml.etree.ElementTree as ElementTree
from datetime import UTC, datetime
from pathlib import Path
from types import ModuleType
@@ -10,11 +11,14 @@ import pytest
from theHarvester import __main__ as theharvester_main
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.asn_attribution import AsnAttributionObservation
from theHarvester.lib.completed_result import CompletedResult, ResultObservation
from theHarvester.lib.dns_consensus import Addressability
from theHarvester.lib.enumeration import EnumerationOptions
from theHarvester.lib.hostchecker import HostDnsRecords
from theHarvester.lib.network_evidence import PrefixOriginObservation, RpkiValidationObservation
from theHarvester.lib.recursive_dns import RecursiveDNSClassification, RecursiveDNSFinding, RecursiveDNSResult
from theHarvester.lib.routeviews import RouteViewsCancelled, RouteViewsResult
from theHarvester.lib.virtual_host import (
HarvestedVirtualHostResult,
VirtualHostDiscoveryCancelled,
@@ -34,6 +38,10 @@ async def test_cli_help_explains_proxy_and_direct_action_scope(
help_text = ' '.join(capsys.readouterr().out.split())
assert exit_info.value.code == 0
assert 'Use proxies.yaml for supported discovery-source and takeover requests.' in help_text
assert (
'Enrich discovered IPs with sourced ASN attribution, or an explicitly targeted ASN, IP, or prefix, through '
'RouteViews.' in help_text
)
assert 'Accepted for compatibility but currently unused; use --dns-resolvers to select resolvers.' in help_text
assert 'Select resolver IPs for DNS actions without enabling hostname resolution.' in help_text
assert 'text file with one IP per line' in help_text
@@ -2156,9 +2164,26 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte
class FakeShodan:
error_type = None
def __init__(self) -> None:
self.attributions: set[AsnAttributionObservation] = set()
async def search_ip(self, ip: str) -> dict[str, dict[str, list[int]]]:
self.attributions.add(
AsnAttributionObservation(
'action',
'shodan',
'AS64496',
'Example Transit',
'ip',
ip,
datetime.now(UTC),
)
)
return {ip: {'ports': [443]}}
async def get_asn_attributions(self) -> set[AsnAttributionObservation]:
return self.attributions
class FakeApiScanner:
def __init__(self, word: str, wordlist: str, exact_paths: bool = False) -> None:
assert word == 'example.com'
@@ -2241,6 +2266,8 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte
assert ('url', 'https://example.com/api/v1') in completed.results
assert ('screenshot', 'api.example.com') not in completed.results
assert ('shodan', '{"ip":"192.0.2.10","result":{"ports":[443]}}') in completed.results
assert ('asn', 'AS64496') in completed.results
assert completed.asn_attributions[0].organization_label == 'Example Transit'
takeover_result = (
'takeover',
'{"matches":[{"No such app":"Heroku"}],"url":"https://api.example.com"}',
@@ -2259,7 +2286,7 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte
assert screenshot_execution.artifacts[0].subject_value == 'api.example.com'
shodan_execution = next(execution for execution in completed.active_evidence.executions if execution.action == 'shodan')
assert shodan_execution.status == 'completed'
assert shodan_execution.result_count == 1
assert shodan_execution.result_count == 3
assert shodan_execution.error_type is None
assert shodan_execution.stop_reason is None
api_executions = [execution for execution in completed.active_evidence.executions if execution.action == 'api-scan']
@@ -2515,6 +2542,258 @@ async def test_shodan_without_ips_is_skipped_without_starting(monkeypatch: pytes
assert shodan_execution.stop_reason == 'no-input'
@pytest.mark.asyncio
async def test_routeviews_persists_typed_network_evidence_for_explicit_ip_target(
monkeypatch: pytest.MonkeyPatch,
) -> None:
calls: list[tuple[tuple[object, ...], tuple[str, ...], str | None]] = []
async def fake_routeviews(asns, network_seeds, *, api_key: str | None = None) -> RouteViewsResult:
calls.append((tuple(asns), tuple(network_seeds), api_key))
collected_at = datetime.now(UTC)
origin = PrefixOriginObservation('routeviews', '192.0.2.0/24', 'AS64500', collected_at)
rpki = RpkiValidationObservation(
'routeviews',
'192.0.2.0/24',
'AS64500',
'valid',
collected_at,
collected_at,
)
return RouteViewsResult(
prefixes=('192.0.2.0/24',),
origin_asns=('AS64500',),
observations=(origin, rpki),
request_count=1,
error_count=0,
status='completed',
)
monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore)
monkeypatch.setattr(theharvester_main, 'enrich_routeviews', fake_routeviews)
monkeypatch.setattr(theharvester_main.Core, 'routeviews_key', staticmethod(lambda: 'routeviews-key'))
result = await theharvester_main.start(
EnumerationOptions(domain='192.0.2.7', quiet=True, routeviews=True),
return_completed_result=True,
)
completed = result[-1]
assert calls == [((), ('192.0.2.7',), 'routeviews-key')]
assert {value for kind, value in completed.results if kind == 'prefix'} == {'192.0.2.0/24'}
assert {value for kind, value in completed.results if kind == 'asn'} == {'AS64500'}
assert len(completed.network_observations) == 2
assert isinstance(completed.network_observations[0], PrefixOriginObservation)
assert isinstance(completed.network_observations[1], RpkiValidationObservation)
execution = next(item for item in completed.active_evidence.executions if item.action == 'routeviews')
assert execution.status == 'completed'
assert {(item.kind, item.value) for item in execution.observations} == {
('asn', 'AS64500'),
('prefix', '192.0.2.0/24'),
}
@pytest.mark.asyncio
async def test_routeviews_pivots_from_an_explicit_asn_target(monkeypatch: pytest.MonkeyPatch) -> None:
calls: list[tuple[tuple[object, ...], tuple[str, ...]]] = []
async def fake_routeviews(asns, network_seeds, *, api_key: str | None = None) -> RouteViewsResult:
assert api_key is None
calls.append((tuple(asns), tuple(network_seeds)))
return RouteViewsResult((), (), (), 2, 0, 'completed', stop_reason='no-results')
monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore)
monkeypatch.setattr(theharvester_main, 'enrich_routeviews', fake_routeviews)
monkeypatch.setattr(theharvester_main.Core, 'routeviews_key', staticmethod(lambda: None))
result = await theharvester_main.start(
EnumerationOptions(domain='as64500', quiet=True, routeviews=True),
return_completed_result=True,
)
assert calls == [(('AS64500',), ())]
assert result[-1].target == 'AS64500'
execution = next(item for item in result[-1].active_evidence.executions if item.action == 'routeviews')
assert execution.status == 'completed'
assert execution.stop_reason == 'no-results'
@pytest.mark.asyncio
async def test_explicit_asn_target_rejects_discovery_sources(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore)
with pytest.raises(ValueError, match='ASN target requires --routeviews without discovery sources or other actions'):
await theharvester_main.start(
EnumerationOptions(domain='AS64500', quiet=True, routeviews=True, source='crtsh'),
return_completed_result=True,
)
@pytest.mark.asyncio
async def test_routeviews_hostname_target_requires_a_discovery_source(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore)
with pytest.raises(ValueError, match='RouteViews hostname target requires a discovery source'):
await theharvester_main.start(
EnumerationOptions(domain='api.example.com', quiet=True, routeviews=True),
return_completed_result=True,
)
@pytest.mark.asyncio
async def test_routeviews_pivots_from_attributed_ips_without_expanding_discovered_asns(
monkeypatch: pytest.MonkeyPatch,
) -> None:
calls: list[tuple[tuple[object, ...], tuple[str, ...]]] = []
class FakeUrlscan:
def __init__(self, _word: str) -> None:
pass
async def process(self, _proxy: bool) -> None:
return None
async def get_hostnames(self) -> set[str]:
return {'example.com'}
async def get_ips(self) -> set[str]:
return {'192.0.2.10'}
async def get_asns(self) -> set[str]:
return {'AS64500'}
async def get_urls(self) -> set[str]:
return set()
async def get_asn_attributions(self) -> set[AsnAttributionObservation]:
return {
AsnAttributionObservation(
'source',
'urlscan',
'AS64500',
'Example Transit',
'ip',
'192.0.2.10',
datetime.now(UTC),
),
AsnAttributionObservation(
'source',
'urlscan',
'AS64500',
'Example Transit',
'hostname',
'example.com',
datetime.now(UTC),
),
}
async def fake_routeviews(asns, network_seeds, *, api_key: str | None = None) -> RouteViewsResult:
assert api_key is None
calls.append((tuple(asns), tuple(network_seeds)))
return RouteViewsResult((), (), (), 2, 0, 'completed', stop_reason='no-results')
monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore)
monkeypatch.setattr(theharvester_main.urlscan, 'SearchUrlscan', FakeUrlscan)
monkeypatch.setattr(theharvester_main, 'enrich_routeviews', fake_routeviews)
monkeypatch.setattr(theharvester_main.Core, 'routeviews_key', staticmethod(lambda: None))
result = await theharvester_main.start(
EnumerationOptions(domain='example.com', quiet=True, routeviews=True, source='urlscan', proxies=True),
return_completed_result=True,
)
assert calls == [((), ('192.0.2.10',))]
completed = result[-1]
assert ('asn', 'AS64500') in completed.results
assert ('ip', '192.0.2.10') in completed.results
assert len(completed.asn_attributions) == 1
assert completed.asn_attributions[0].subject_value == '192.0.2.10'
execution = next(item for item in result[-1].active_evidence.executions if item.action == 'routeviews')
assert execution.status == 'completed'
assert execution.stop_reason == 'no-results'
@pytest.mark.asyncio
async def test_routeviews_cancellation_persists_partial_network_evidence(monkeypatch: pytest.MonkeyPatch) -> None:
saved: list[CompletedResult] = []
async def cancel_routeviews(asns, network_seeds, *, api_key: str | None = None) -> RouteViewsResult:
assert tuple(asns) == ()
assert tuple(network_seeds) == ('192.0.2.7',)
assert api_key is None
collected_at = datetime.now(UTC)
origin = PrefixOriginObservation('routeviews', '192.0.2.0/24', 'AS64500', collected_at)
raise RouteViewsCancelled(
RouteViewsResult(
prefixes=('192.0.2.0/24',),
origin_asns=('AS64500',),
observations=(origin,),
request_count=1,
error_count=1,
status='partial',
error_type='CancelledError',
stop_reason='cancelled',
)
)
monkeypatch.setattr(theharvester_main, 'ResultStore', _recording_result_store(saved))
monkeypatch.setattr(theharvester_main, 'enrich_routeviews', cancel_routeviews)
monkeypatch.setattr(theharvester_main.Core, 'routeviews_key', staticmethod(lambda: None))
with pytest.raises(RouteViewsCancelled):
await theharvester_main.start(
EnumerationOptions(domain='192.0.2.7', quiet=True, routeviews=True),
return_completed_result=True,
)
execution = next(item for item in saved[-1].active_evidence.executions if item.action == 'routeviews')
assert execution.status == 'partial'
assert execution.error_type == 'CancelledError'
assert execution.stop_reason == 'cancelled'
assert ('prefix', '192.0.2.0/24') in saved[-1].results
@pytest.mark.asyncio
async def test_routeviews_cancellation_persists_when_the_checkpoint_fails(monkeypatch: pytest.MonkeyPatch) -> None:
saved: list[CompletedResult] = []
async def cancel_routeviews(asns, network_seeds, *, api_key: str | None = None) -> RouteViewsResult:
assert api_key is None
collected_at = datetime.now(UTC)
origin = PrefixOriginObservation('routeviews', '192.0.2.0/24', 'AS64500', collected_at)
raise RouteViewsCancelled(
RouteViewsResult(
prefixes=('192.0.2.0/24',),
origin_asns=('AS64500',),
observations=(origin,),
request_count=1,
error_count=1,
status='partial',
error_type='CancelledError',
stop_reason='cancelled',
)
)
async def failed_checkpoint(result: CompletedResult) -> None:
if any(execution.action == 'routeviews' for execution in result.active_evidence.executions):
raise RuntimeError('checkpoint failed')
monkeypatch.setattr(theharvester_main, 'ResultStore', _recording_result_store(saved))
monkeypatch.setattr(theharvester_main, 'enrich_routeviews', cancel_routeviews)
monkeypatch.setattr(theharvester_main.Core, 'routeviews_key', staticmethod(lambda: None))
with pytest.raises(RouteViewsCancelled):
await theharvester_main.start(
EnumerationOptions(domain='192.0.2.7', quiet=True, routeviews=True),
completed_result_checkpoint=failed_checkpoint,
return_completed_result=True,
)
execution = next(item for item in saved[-1].active_evidence.executions if item.action == 'routeviews')
assert execution.stop_reason == 'cancelled'
assert ('prefix', '192.0.2.0/24') in saved[-1].results
@pytest.mark.asyncio
async def test_api_scan_cancellation_persists_failure_and_propagates(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
saved: list[CompletedResult] = []
+5 -2
View File
@@ -88,10 +88,13 @@ def test_readme_matches_declared_source_contracts() -> None:
def test_readme_api_key_markers_match_configuration() -> None:
requirements = _documented_api_key_requirements(Path('README.md').read_text())
readme = Path('README.md').read_text()
requirements = _documented_api_key_requirements(readme)
configured_source_keys = _configured_api_key_sources() - {'routeviews'}
assert set(requirements.values()) <= {'', 'Optional', 'No'}
assert {source for source, marker in requirements.items() if marker != 'No'} == _configured_api_key_sources()
assert {source for source, marker in requirements.items() if marker != 'No'} == configured_source_keys
assert '`routeviews.key`' in readme
assert {source for source, marker in requirements.items() if marker == 'Optional'} == OPTIONAL_API_KEY_SOURCES
+162 -5
View File
@@ -13,9 +13,9 @@ import time
from collections.abc import Awaitable, Callable, Iterable, Mapping
from contextlib import AsyncExitStack
from datetime import UTC, datetime
from ipaddress import ip_address
from ipaddress import ip_address, ip_network
from pathlib import Path
from typing import Any, cast
from typing import TYPE_CHECKING, Any, cast
from urllib.parse import urlsplit
from uuid import UUID, uuid4
@@ -90,6 +90,7 @@ from theHarvester.discovery import (
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib import hostchecker
from theHarvester.lib.active_evidence import ActionExecution, ActiveEvidence, ArtifactReference
from theHarvester.lib.asn_attribution import AsnAttributionObservation
from theHarvester.lib.completed_result import (
EXECUTION_STATUSES,
CompletedResult,
@@ -115,6 +116,8 @@ from theHarvester.lib.recursive_dns import (
discover_recursive_dns,
)
from theHarvester.lib.resolver_selection import DEFAULT_DNS_RESOLVERS, normalize_resolver_addresses
from theHarvester.lib.result_values import normalize_asn
from theHarvester.lib.routeviews import RouteViewsCancelled, RouteViewsResult, enrich_routeviews
from theHarvester.lib.source_catalog import (
SOURCE_SPECS,
ActivityClass,
@@ -137,6 +140,9 @@ from theHarvester.lib.virtual_host import (
)
from theHarvester.screenshot.screenshot import ScreenShotter
if TYPE_CHECKING:
from theHarvester.lib.network_evidence import NetworkObservation
logger = logging.getLogger(__name__)
@@ -200,7 +206,12 @@ async def start(
parser = argparse.ArgumentParser(
description='theHarvester is used to gather open source intelligence (OSINT) on a company or domain.'
)
parser.add_argument('-d', '--domain', help='Company name or domain to search.', required=True)
parser.add_argument(
'-d',
'--domain',
help='Company name or domain to search, or an explicit ASN/IP/CIDR target for --routeviews.',
required=True,
)
parser.add_argument(
'-l',
'--limit',
@@ -229,6 +240,16 @@ async def start(
default=False,
action='store_true',
)
parser.add_argument(
'--routeviews',
help=(
'Enrich discovered IPs with sourced ASN attribution, or an explicitly targeted ASN, IP, or prefix, through '
'RouteViews. Returned routing relationships do not establish ownership or target scope. Uses authenticated '
'access when a RouteViews API key is configured.'
),
default=False,
action='store_true',
)
parser.add_argument(
'--screenshot',
help='Save screenshots of reachable discovered hosts to DIR. This sends direct browser requests.',
@@ -506,11 +527,19 @@ async def start(
ips: list = []
host_ip: list = []
limit: int = args.limit
routeviews_enabled = args.routeviews
shodan = args.shodan
start: int = args.start
all_urls: list = []
vhost_observations: list[VirtualHostObservation] = []
word: str = args.domain.rstrip('\n')
explicit_asn_target: str | None = None
if word.strip()[:2].casefold() == 'as':
try:
explicit_asn_target = normalize_asn(word)
word = explicit_asn_target
except ValueError:
pass
takeover_status = args.take_over
use_proxy = args.proxies
linkedin_people_list_tracker: list = []
@@ -534,6 +563,10 @@ async def start(
source_executions: list[SourceExecution] = []
observations: set[ResultObservation] = set()
action_executions: list[ActionExecution] = []
network_prefixes: set[str] = set()
network_observations: list[NetworkObservation] = []
asn_attributions: list[AsnAttributionObservation] = []
displayed_asn_attributions: set[AsnAttributionObservation] = set()
dns_resolution_duration_ms = 0.0
dns_resolution_ips: set[str] = set()
dns_resolution_completed_count = 0
@@ -548,6 +581,21 @@ async def start(
def confirmed_virtual_hostnames() -> list[str]:
return sorted({observation.hostname for observation in vhost_observations})
def display_new_asn_attributions() -> None:
pending = sorted(set(asn_attributions) - displayed_asn_attributions, key=AsnAttributionObservation.sort_key)
if not pending:
return
print_section(
f'\n[*] ASN organization attributions found: {len(pending)}',
(
f'{item.asn} | {item.organization_label} | {item.producer_kind}:{item.producer} | '
f'{item.subject_kind}:{item.subject_value}'
for item in pending
),
'------------------------------------',
)
displayed_asn_attributions.update(pending)
def finish_completed_result(
*,
extra_hostnames: Iterable[str] = (),
@@ -570,6 +618,7 @@ async def start(
'language': map(str, all_languages),
'linkedin-person': map(str, linkedin_people_list_tracker),
'person': (json.dumps(person, ensure_ascii=False, separators=(',', ':'), sort_keys=True) for person in all_people),
'prefix': network_prefixes,
'server': map(str, all_servers),
'twitter-person': map(str, twitter_people_list_tracker),
'url': map(str, all_urls),
@@ -595,6 +644,8 @@ async def start(
source_executions=source_executions,
observations=observations,
active_evidence=ActiveEvidence(tuple(action_executions)),
network_observations=network_observations,
asn_attributions=asn_attributions,
virtual_hosts=vhost_observations,
)
except (ValueError, TypeError) as error:
@@ -794,6 +845,14 @@ async def start(
fasns = await search_engine.get_asns()
total_asns.extend(fasns)
record_source_observations(source, 'asn', fasns)
get_asn_attributions = getattr(search_engine, 'get_asn_attributions', None)
if get_asn_attributions is not None:
asn_attributions.extend(
attribution
for attribution in await get_asn_attributions()
if ResultObservation(source, 'asn', attribution.asn) in source_observations
and ResultObservation(source, attribution.subject_kind, attribution.subject_value) in source_observations
)
if ResultRoute.BREACHES in routes:
breach_names = await search_engine.get_breach_names()
@@ -888,9 +947,30 @@ async def start(
stor_lst = []
if args.source is not None:
engines = Core.expand_source_selection(args.source)
if explicit_asn_target is not None and (
not routeviews_enabled
or engines
or shodan
or dnslookup
or dnsbrute[0]
or dnsresolve != ''
or recursive_limits is not None
or takeover_status
or args.screenshot
or args.api_scan
or vhost_enabled
):
raise ValueError('ASN target requires --routeviews without discovery sources or other actions')
if routeviews_enabled and not engines and explicit_asn_target is None:
try:
ip_network(word, strict=False)
except ValueError as error:
raise ValueError('RouteViews hostname target requires a discovery source') from error
activities = {get_source_spec(engine).activity for engine in engines if engine in SOURCE_SPECS}
if shodan:
activities.add(ActivityClass.PASSIVE)
if routeviews_enabled:
activities.add(ActivityClass.PASSIVE)
if dnslookup or dnsbrute[0] or dnsresolve != '' or recursive_limits is not None:
activities.add(ActivityClass.DNS)
if takeover_status or args.screenshot or args.api_scan or vhost_enabled:
@@ -1968,6 +2048,7 @@ async def start(
and rest_args.dns_brute is False
and not dnslookup
and not return_completed_result
and not routeviews_enabled
and not vhost_enabled
):
# Indicates user is using REST api but not wanting output to be saved to a file
@@ -2005,6 +2086,7 @@ async def start(
if len(total_asns) > 0:
print_section(f'\n[*] ASNS found: {len(total_asns)}', total_asns, '--------------------')
total_asns = sorted_unique(total_asns)
display_new_asn_attributions()
if len(twitter_people_list_tracker) == 0 and 'twitter' in engines:
output_logger.info('\n[*] No Twitter users found.\n\n')
@@ -2174,6 +2256,68 @@ async def start(
await persist_result(finish_completed_result())
return resolved_pair
if routeviews_enabled:
routeviews_started = time.perf_counter()
routeviews_asns = {explicit_asn_target} if explicit_asn_target is not None else set()
routeviews_network_seeds = {
attribution.subject_value
for attribution in asn_attributions
if attribution.producer_kind == 'source' and attribution.subject_kind == 'ip'
}
try:
routeviews_network_seeds.add(
str(ip_network(word.strip(), strict=False)) if '/' in word else str(ip_address(word.strip()))
)
except ValueError:
pass
def record_routeviews_result(result: RouteViewsResult) -> None:
network_prefixes.update(result.prefixes)
total_asns.extend(result.origin_asns)
network_observations.extend(result.observations)
action_executions.append(
ActionExecution.finish(
action='routeviews',
status=result.status,
duration_ms=(time.perf_counter() - routeviews_started) * 1000,
groups={'asn': result.origin_asns, 'prefix': result.prefixes},
error_type=result.error_type,
stop_reason=result.stop_reason,
)
)
try:
routeviews_result = await enrich_routeviews(
routeviews_asns,
routeviews_network_seeds,
api_key=Core.routeviews_key(),
)
except RouteViewsCancelled as error:
record_routeviews_result(error.result)
cancelled_result = finish_completed_result()
try:
if completed_result_checkpoint is not None and cancelled_result is not None:
await completed_result_checkpoint(cancelled_result)
except (asyncio.CancelledError, Exception) as checkpoint_error:
output_logger.info(f'[!] RouteViews cancellation checkpoint failed: {checkpoint_error}')
finally:
await persist_result(cancelled_result)
raise
record_routeviews_result(routeviews_result)
if routeviews_result.prefixes:
print_section(
f'\n[*] RouteViews prefixes found: {len(routeviews_result.prefixes)}',
routeviews_result.prefixes,
'--------------------',
)
output_logger.info(
'[*] RouteViews: '
f'prefixes={len(routeviews_result.prefixes)}; origins={len(routeviews_result.origin_asns)}; '
f'requests={routeviews_result.request_count}; errors={routeviews_result.error_count}; '
f'status={routeviews_result.status}; stop={routeviews_result.stop_reason or "complete"}'
)
await checkpoint_completed_result()
# TakeOver Checking
if takeover_status:
takeover_started = time.perf_counter()
@@ -2630,6 +2774,8 @@ async def start(
if shodan is True:
shodan_started = time.perf_counter()
shodan_error_types: set[str] = set()
shodan_asns: set[str] = set()
shodan_ips: set[str] = set()
output_logger.info('[*] Searching Shodan. ')
try:
for ip_index, ip in enumerate(host_ip):
@@ -2637,6 +2783,13 @@ async def start(
output_logger.info('\tSearching for ' + ip)
shodan_search = shodansearch.SearchShodan()
shodandict = await shodan_search.search_ip(ip)
get_asn_attributions = getattr(shodan_search, 'get_asn_attributions', None)
if get_asn_attributions is not None:
collected_attributions = await get_asn_attributions()
asn_attributions.extend(collected_attributions)
shodan_asns.update(attribution.asn for attribution in collected_attributions)
shodan_ips.update(attribution.subject_value for attribution in collected_attributions)
total_asns.extend(attribution.asn for attribution in collected_attributions)
if shodan_search.error_type:
shodan_error_types.add(shodan_search.error_type)
@@ -2672,7 +2825,7 @@ async def start(
action='shodan',
status='partial' if shodan_evidence else 'failed',
duration_ms=(time.perf_counter() - shodan_started) * 1000,
groups={'shodan': shodan_evidence},
groups={'shodan': shodan_evidence, 'asn': shodan_asns, 'ip': shodan_ips},
error_type='CancelledError',
stop_reason='cancelled',
)
@@ -2692,11 +2845,12 @@ async def start(
action='shodan',
status=shodan_status,
duration_ms=(time.perf_counter() - shodan_started) * 1000,
groups={'shodan': shodan_evidence},
groups={'shodan': shodan_evidence, 'asn': shodan_asns, 'ip': shodan_ips},
error_type=next(iter(sorted(shodan_error_types)), None),
stop_reason=shodan_stop_reason,
)
)
display_new_asn_attributions()
await checkpoint_action_result(extra_hostnames=dnsrev)
else:
pass
@@ -2947,6 +3101,9 @@ async def start(
if len(total_asns) > 0:
json_dict['asns'] = total_asns
if network_prefixes:
json_dict['prefixes'] = sorted(network_prefixes)
if len(twitter_people_list_tracker) > 0:
json_dict['twitter_people'] = twitter_people_list_tracker
+3
View File
@@ -80,6 +80,9 @@ apikeys:
rocketreach:
key:
routeviews:
key:
securityscorecard:
key:
+56 -4
View File
@@ -1,10 +1,13 @@
import logging
from datetime import UTC, datetime
from ipaddress import ip_address
from urllib.parse import urlparse
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.asn_attribution import AsnAttributionObservation, SubjectKind
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.result_values import normalize_asn
logger = logging.getLogger(__name__)
@@ -12,12 +15,20 @@ logger = logging.getLogger(__name__)
class SearchOnyphe:
"""Collect ONYPHE results and retain physical/logical IP attribution.
ONYPHE documents the root ``asn``/``organization`` pair as physical
hosting data and ``geolocus.asn``/``geolocus.organization`` as logical
WHOIS data. Both stay separate and are linked to the record's primary IP.
"""
def __init__(self, word) -> None:
self.word = word
self.response = ''
self.totalhosts: set = set()
self.totalips: set = set()
self.asns: set = set()
self.asn_attributions: set[AsnAttributionObservation] = set()
self.key = Core.onyphe_key()
if self.key is None:
raise MissingKey('onyphe')
@@ -93,18 +104,22 @@ class SearchOnyphe:
malformed = True
continue
primary_ip = None
alternative_ips = result.get('alternativeip', [])
if not isinstance(alternative_ips, list):
malformed = True
alternative_ips = []
ip_candidates = [result['ip']] if 'ip' in result else []
ip_candidates.extend(alternative_ips)
for candidate in ip_candidates:
for index, candidate in enumerate(ip_candidates):
if not isinstance(candidate, str):
malformed = True
continue
try:
self.totalips.add(str(ip_address(candidate.strip())))
normalized_ip = str(ip_address(candidate.strip()))
self.totalips.add(normalized_ip)
if index == 0 and 'ip' in result:
primary_ip = normalized_ip
except ValueError:
malformed = True
@@ -141,10 +156,12 @@ class SearchOnyphe:
geolocus = result.get('geolocus')
geolocus_asn = None
geolocus_organization = None
if geolocus is not None:
if isinstance(geolocus, dict) and isinstance(geolocus.get('domain', []), list):
host_candidates.extend(geolocus.get('domain', []))
geolocus_asn = geolocus.get('asn')
geolocus_organization = geolocus.get('organization')
else:
malformed = True
@@ -155,13 +172,45 @@ class SearchOnyphe:
if hostname := normalize_scoped_hostname(candidate, self.word):
self.totalhosts.add(hostname)
for asn in (result.get('asn'), geolocus_asn):
attribution_pairs = (
(result.get('asn'), result.get('organization')),
(geolocus_asn, geolocus_organization),
)
for asn, organization in attribution_pairs:
if asn is None:
continue
if isinstance(asn, (str, int)):
self.asns.add(asn)
try:
normalized_asn = normalize_asn(asn)
except ValueError:
malformed = True
continue
self.asns.add(normalized_asn)
else:
malformed = True
continue
if organization is None:
continue
if not isinstance(organization, str) or not organization.strip():
malformed = True
continue
collected_at = datetime.now(UTC)
subjects: list[tuple[SubjectKind, str]] = [('ip', primary_ip)] if primary_ip is not None else []
for subject_kind, subject_value in subjects:
try:
self.asn_attributions.add(
AsnAttributionObservation(
'source',
'onyphe',
normalized_asn,
organization,
subject_kind,
subject_value,
collected_at,
)
)
except ValueError:
malformed = True
if malformed:
self.execution_status = 'partial' if self.totalhosts or self.totalips or self.asns else 'failed'
@@ -173,6 +222,9 @@ class SearchOnyphe:
async def get_asns(self) -> set:
return self.asns
async def get_asn_attributions(self) -> set[AsnAttributionObservation]:
return self.asn_attributions
async def get_hostnames(self) -> set:
return self.totalhosts
+31
View File
@@ -1,15 +1,25 @@
import logging
from collections import OrderedDict
from datetime import UTC, datetime
from ipaddress import ip_address
from shodan import Shodan, exception
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.asn_attribution import AsnAttributionObservation
from theHarvester.lib.core import Core
logger = logging.getLogger(__name__)
class SearchShodan:
"""Collect Shodan host data and retain ``asn`` + ``org`` attribution.
Shodan documents ``org`` and ``isp`` as separate Host API fields. The
organization attribution uses ``org`` and stays linked to the queried IP;
``isp`` remains ordinary Shodan output and is not treated as equivalent.
"""
def __init__(self) -> None:
self.key = Core.shodan_key()
if self.key is None:
@@ -18,6 +28,7 @@ class SearchShodan:
self.hostdatarow: list = []
self.tracker: OrderedDict = OrderedDict()
self.error_type: str | None = None
self.asn_attributions: set[AsnAttributionObservation] = set()
async def search_ip(self, ip) -> OrderedDict:
self.error_type = None
@@ -108,6 +119,23 @@ class SearchShodan:
'technologies': technologies,
'title': title.strip(),
}
organization_label = results.get('org')
if asn.strip() and isinstance(organization_label, str) and organization_label.strip():
try:
subject_ip = str(ip_address(str(ip).strip()))
self.asn_attributions.add(
AsnAttributionObservation(
'action',
'shodan',
asn,
organization_label,
'ip',
subject_ip,
datetime.now(UTC),
)
)
except ValueError:
logger.info('Shodan returned invalid ASN organization attribution')
return self.tracker
except exception.APIError as error:
@@ -122,3 +150,6 @@ class SearchShodan:
self.tracker[ip] = 'Shodan request failed'
return self.tracker
async def get_asn_attributions(self) -> set[AsnAttributionObservation]:
return self.asn_attributions
+45 -7
View File
@@ -1,9 +1,12 @@
import logging
from datetime import UTC, datetime
from ipaddress import ip_address
from urllib.parse import urlsplit
from theHarvester.lib.asn_attribution import AsnAttributionObservation, SubjectKind
from theHarvester.lib.core import AsyncFetcher, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.result_values import normalize_asn
logger = logging.getLogger(__name__)
@@ -18,6 +21,7 @@ class SearchUrlscan:
self.totalips: set = set()
self.urls: set = set()
self.totalasns: set = set()
self.asn_attributions: set[AsnAttributionObservation] = set()
self.proxy = False
self.execution_status: str | None = None
self.stop_reason: str | None = None
@@ -46,7 +50,7 @@ class SearchUrlscan:
values.append(serialized)
return ','.join(values)
def _parse_results(self, results: list) -> bool:
def _parse_results(self, results: list, collected_at: datetime) -> bool:
malformed = False
for result in results:
if not isinstance(result, dict):
@@ -59,12 +63,14 @@ class SearchUrlscan:
malformed = True
continue
scoped = False
scoped_hosts: set[str] = set()
if 'domain' in page:
domain = page['domain']
if not isinstance(domain, str) or not domain:
malformed = True
elif normalized_domain := normalize_scoped_hostname(domain, self.word):
self.totalhosts.add(normalized_domain)
scoped_hosts.add(normalized_domain)
scoped = True
if 'url' in page:
value = page['url']
@@ -83,30 +89,59 @@ class SearchUrlscan:
scoped = True
if not scoped:
continue
scoped_ips: set[str] = set()
if 'ip' in page:
address = page['ip']
if not isinstance(address, str):
malformed = True
else:
try:
self.totalips.add(str(ip_address(address.strip())))
normalized_ip = str(ip_address(address.strip()))
self.totalips.add(normalized_ip)
scoped_ips.add(normalized_ip)
except ValueError:
malformed = True
normalized_asn = None
if 'asn' in page:
asn = page['asn']
if not isinstance(asn, str):
malformed = True
else:
normalized_asn = asn.strip().upper()
asn_number = normalized_asn.removeprefix('AS')
if normalized_asn.startswith('AS') and asn_number.isdigit() and int(asn_number) > 0:
self.totalasns.add(f'AS{int(asn_number)}')
try:
normalized_asn = normalize_asn(asn)
except ValueError:
malformed = True
else:
self.totalasns.add(normalized_asn)
organization = page.get('asnname')
if organization is not None and not isinstance(organization, str):
malformed = True
organization = None
if normalized_asn is not None and isinstance(organization, str) and organization.strip():
subjects: list[tuple[SubjectKind, str]] = [
*(('hostname', hostname) for hostname in scoped_hosts),
*(('ip', address) for address in scoped_ips),
]
for subject_kind, subject_value in subjects:
try:
self.asn_attributions.add(
AsnAttributionObservation(
'source',
'urlscan',
normalized_asn,
organization,
subject_kind,
subject_value,
collected_at,
)
)
except ValueError:
malformed = True
return malformed
async def do_search(self) -> None:
url = 'https://urlscan.io/api/v1/search/'
collected_at = datetime.now(UTC)
cursor = None
seen_cursors: set[str] = set()
malformed = False
@@ -153,7 +188,7 @@ class SearchUrlscan:
self.stop_reason = None if self._has_results() else 'no-results'
return
malformed = self._parse_results(results) or malformed
malformed = self._parse_results(results, collected_at) or malformed
next_cursor = self._cursor(results[-1])
if next_cursor is None:
self._stop('failed', 'invalid-cursor')
@@ -179,6 +214,9 @@ class SearchUrlscan:
async def get_asns(self) -> set:
return self.totalasns
async def get_asn_attributions(self) -> set[AsnAttributionObservation]:
return self.asn_attributions
async def process(self, proxy: bool = False) -> None:
self.proxy = proxy
await self.do_search()
+20
View File
@@ -7,6 +7,7 @@ from uuid import UUID
from fastapi import HTTPException, status
from theHarvester.lib.asn_attribution import asn_attribution_details, parse_asn_attribution_details
from theHarvester.lib.completed_result import parse_result_jsonl, parse_virtual_host_details, virtual_host_details
from theHarvester.lib.evidence_types import EVIDENCE_STATUSES
from theHarvester.lib.network_evidence import (
@@ -137,6 +138,25 @@ def validate_evidence(evidence: dict[str, Any]) -> dict[str, Any]:
result['sources'] = sorted(set(sources))
result['actions'] = sorted(set(actions))
continue
if result.get('type') == 'asn' and 'observations' in result:
allowed_keys = {'type', 'value', 'sources', 'actions', 'observations'}
if set(result) - allowed_keys:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail='ASN attribution contains unsupported fields',
)
result_value = result.get('value')
if not isinstance(result_value, str):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail='ASN attribution must identify a canonical ASN',
)
try:
attributions = parse_asn_attribution_details(result_value, result.get('observations'))
except ValueError as error:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(error)) from error
result['observations'] = asn_attribution_details(attributions)
continue
if 'observations' not in result:
continue
if result.get('type') != 'hostname':
+53 -6
View File
@@ -14,6 +14,7 @@ from theHarvester.lib.enumeration import (
)
from theHarvester.lib.evidence_types import EvidenceStatus # noqa: TC001 - Pydantic resolves this annotation at runtime
from theHarvester.lib.resolver_selection import DEFAULT_DNS_RESOLVERS, normalize_resolver_addresses
from theHarvester.lib.result_values import normalize_asn
from theHarvester.lib.source_catalog import SOURCE_SPECS, ActivityClass, selected_action_names
from theHarvester.lib.virtual_host import (
DEFAULT_VHOST_CONCURRENCY,
@@ -32,7 +33,10 @@ def utc_now() -> str:
def _normalize_target(value: str) -> str:
target = value.strip().rstrip('.').lower()
target = value.strip().rstrip('.')
if target[:2].casefold() == 'as' and target[2:].isascii() and target[2:].isdecimal():
return normalize_asn(target)
target = target.lower()
if not target or len(target) > 253 or any(character in target for character in '/?#@'):
raise ValueError('Target must be a hostname or IP address')
try:
@@ -58,7 +62,9 @@ def _normalize_target(value: str) -> str:
class RunRequest(BaseModel):
model_config = ConfigDict(extra='forbid')
target: str = Field(description='Authorized domain name or IP address to enumerate.')
target: str = Field(
description='Authorized domain name or IP address, or an explicit AS-prefixed network pivot for RouteViews.'
)
sources: list[str] = Field(
max_length=len(SOURCE_SPECS),
description=(
@@ -117,6 +123,13 @@ class RunRequest(BaseModel):
allow_inf_nan=False,
description='Maximum wall-clock seconds spent in recursive DNS discovery.',
)
routeviews: bool = Field(
default=False,
description=(
'Enrich discovered IPs with sourced ASN attribution, or an explicitly targeted ASN or IP address, through '
'RouteViews. Returned routing relationships do not establish ownership, authorization, or target scope.'
),
)
shodan: bool = Field(default=False, description='Enrich discovered hosts with configured Shodan access.')
screenshot: bool = Field(
default=False,
@@ -251,8 +264,17 @@ class RunRequest(BaseModel):
@model_validator(mode='after')
def validate_selected_work(self) -> RunRequest:
if not self.sources and not selected_action_names(self.model_dump()):
action_names = selected_action_names(self.model_dump())
if not self.sources and not action_names:
raise ValueError('Select at least one discovery source or action')
if self.target.startswith('AS') and self.target[2:].isdecimal():
if self.sources or action_names != ('routeviews',):
raise ValueError('ASN target requires RouteViews as the only selected work')
elif self.routeviews and not self.sources:
try:
ipaddress.ip_address(self.target)
except ValueError as error:
raise ValueError('RouteViews hostname target requires a discovery source') from error
if self.dns_recursive_depth > 0 and len(self.dns_resolvers) != 3:
raise ValueError('Recursive DNS requires exactly three distinct resolver IPs')
return self
@@ -347,6 +369,24 @@ class RpkiValidationObservationResponse(BaseModel):
collected_at: str
class AsnAttributionSubjectResponse(BaseModel):
model_config = ConfigDict(extra='forbid')
type: Literal['hostname', 'ip']
value: str
class AsnAttributionObservationResponse(BaseModel):
model_config = ConfigDict(extra='forbid')
type: Literal['organization-attribution']
producer_kind: Literal['source', 'action']
producer: str
organization_label: str
subject: AsnAttributionSubjectResponse
collected_at: str
class NormalizedResult(BaseModel):
model_config = ConfigDict(extra='forbid')
@@ -361,6 +401,7 @@ class NormalizedResult(BaseModel):
| PrefixOriginObservationResponse
| BgpRouteObservationResponse
| RpkiValidationObservationResponse
| AsnAttributionObservationResponse
]
| None
) = Field(default=None, min_length=1)
@@ -379,9 +420,15 @@ class NormalizedResult(BaseModel):
elif self.scope is not None:
raise ValueError('Only prefix results have relationship scope')
elif self.observations is not None:
if self.type != 'hostname':
raise ValueError('Structured observations belong to hostname or prefix results')
parse_virtual_host_details(self.value, [details.model_dump() for details in self.observations])
details = [details.model_dump() for details in self.observations]
if self.type == 'asn':
from theHarvester.lib.asn_attribution import parse_asn_attribution_details
parse_asn_attribution_details(self.value, details)
elif self.type == 'hostname':
parse_virtual_host_details(self.value, details)
else:
raise ValueError('Structured observations belong to ASN, hostname, or prefix results')
return self
+1 -1
View File
@@ -38,7 +38,7 @@ def normalized_results(evidence: dict[str, Any] | None) -> list[dict[str, Any]]:
'sources': sorted({str(source) for source in item.get('sources', [])}),
'actions': sorted({str(action) for action in item.get('actions', [])}),
}
if item.get('type') in {'hostname', 'prefix'} and item.get('observations'):
if item.get('type') in {'asn', 'hostname', 'prefix'} and item.get('observations'):
result['observations'] = [
dict(observation) for observation in item.get('observations', []) if isinstance(observation, dict)
]
+5
View File
@@ -9,6 +9,7 @@ from uuid import UUID, uuid4
from fastapi import HTTPException, status
from theHarvester.lib.active_evidence import ActionExecution, ActiveEvidence, ArtifactReference
from theHarvester.lib.asn_attribution import AsnAttributionObservation, parse_asn_attribution_details
from theHarvester.lib.completed_result import (
CompletedResult,
ResultObservation,
@@ -53,6 +54,7 @@ def _completed_result(
action_groups: dict[str, dict[ResultKind, set[str]]] = defaultdict(lambda: defaultdict(set))
virtual_hosts: list[VirtualHostObservation] = []
network_observations: list[NetworkObservation] = []
asn_attributions: list[AsnAttributionObservation] = []
for item in results:
kind = cast('ResultKind', str(item['type']))
value = str(item['value'])
@@ -61,6 +63,8 @@ def _completed_result(
virtual_hosts.extend(parse_virtual_host_details(value, item.get('observations')))
elif kind == 'prefix' and item.get('observations'):
network_observations.extend(parse_network_observation_details(value, item.get('observations')))
elif kind == 'asn' and item.get('observations'):
asn_attributions.extend(parse_asn_attribution_details(value, item.get('observations')))
for source in set(item.get('sources', [])):
source_name = str(source)
source_origins.add(ResultObservation(source_name, kind, value))
@@ -147,6 +151,7 @@ def _completed_result(
active_evidence=active_evidence,
virtual_hosts=virtual_hosts,
network_observations=network_observations,
asn_attributions=asn_attributions,
evidence_status=(
cast('EvidenceStatus', str(evidence['status']))
if evidence.get('status') is not None and not execution_status_is_authoritative
+1
View File
@@ -316,6 +316,7 @@ async def _child_execute(run_id: str, database: Path) -> None:
limit=request['limit'],
proxies=request.get('proxies', False),
quiet=True,
routeviews=request.get('routeviews', False),
screenshot=str(screenshot_dir) if request.get('screenshot') else '',
shodan=request.get('shodan', False),
source=','.join(request['sources']),
+68 -3
View File
@@ -4,11 +4,11 @@
const $ = selector => document.querySelector(selector);
const $$ = selector => [...document.querySelectorAll(selector)];
const ROUTE_ORDER = [
'hostname', 'ip', 'asn', 'email', 'url', 'person', 'person-link', 'takeover', 'shodan',
'hostname', 'ip', 'prefix', 'asn', 'email', 'url', 'person', 'person-link', 'takeover', 'shodan',
'scope-extension', 'external-relationship', 'other'
];
const ROUTE_LABELS = {
hostname: 'Hostnames', ip: 'IP addresses', asn: 'ASNs', email: 'Emails', url: 'URLs',
hostname: 'Hostnames', ip: 'IP addresses', prefix: 'Network prefixes', asn: 'ASNs', email: 'Emails', url: 'URLs',
person: 'People', 'person-link': 'People links', takeover: 'Takeover evidence', shodan: 'Shodan evidence',
'scope-extension': 'Scope extensions', 'external-relationship': 'External relationships', other: 'Other'
};
@@ -298,6 +298,7 @@
['Recursive DNS depth', request.dns_recursive_depth ?? 'Not recorded'],
['Recursive DNS query budget', request.dns_recursive_query_limit ?? 'Not recorded'],
['Recursive DNS runtime', request.dns_recursive_runtime_seconds ? `${request.dns_recursive_runtime_seconds} seconds` : 'Not recorded'],
['RouteViews enrichment', request.routeviews ? 'Selected' : 'Off'],
['Screenshots', request.screenshot ? 'Selected' : 'Off'],
['Takeover transport', request.takeover ? (request.proxies ? 'Configured proxy' : 'Direct') : 'Off'],
['API endpoint interaction', request.api_scan ? 'Selected' : 'Off'],
@@ -418,6 +419,56 @@
return text.includes(query);
}
function asnAttributionsFormatter(cell) {
const observations = Array.isArray(cell.getValue()) ? cell.getValue() : [];
if (!observations.length) return 'No organization attribution';
return `<div class="vhost-observations">${observations.map(observation => {
const subject = observation.subject || {};
const producer = `${observation.producer_kind || 'producer'}:${observation.producer || 'unknown'}`;
const text = `${observation.organization_label || 'Unknown organization'} · ${producer} · ${subject.type || 'subject'}:${subject.value || 'unknown'}`;
return `<span title="${escapeHtml(text)}">${escapeHtml(text)}</span>`;
}).join('')}</div>`;
}
function asnAttributionsFilter(headerValue, rowValue) {
const query = String(headerValue || '').trim().toLowerCase().replaceAll('-', ' ');
const text = (Array.isArray(rowValue) ? rowValue : []).flatMap(observation => [
observation.organization_label, observation.producer_kind, observation.producer,
observation.subject?.type, observation.subject?.value
]).join(' ').toLowerCase().replaceAll('-', ' ');
return text.includes(query);
}
function networkObservationsFormatter(cell) {
const observations = Array.isArray(cell.getValue()) ? cell.getValue() : [];
if (!observations.length) return 'No routing evidence';
const validations = new Map(observations
.filter(observation => observation.type === 'rpki-validation')
.map(observation => [observation.origin_asn, observation.state]));
const origins = [...new Set(observations.map(observation => observation.origin_asn).filter(Boolean))].sort();
const originSummary = origins.map(origin => {
const state = validations.get(origin);
return `${origin} · RPKI ${state ? state.replaceAll('-', ' ') : 'not recorded'}`;
}).join(' · ');
const routes = observations.filter(observation => observation.type === 'bgp-route');
const routeDetails = routes.length ? `<details><summary>${routes.length.toLocaleString()} BGP route observation${routes.length === 1 ? '' : 's'}</summary>${routes.map(route => {
const peer = `${route.peer_asn || 'unknown peer'}${route.peer_address ? ` (${route.peer_address})` : ''}`;
const path = route.as_path ? ` · path ${route.as_path}` : '';
const communities = route.communities ? ` · communities ${route.communities}` : '';
return `<span>${escapeHtml(`${route.collector || 'unknown collector'} · peer ${peer}${path}${communities}`)}</span>`;
}).join('')}</details>` : '';
return `<div class="vhost-observations"><span>${escapeHtml(originSummary || 'Origin not recorded')}</span>${routeDetails}</div>`;
}
function networkObservationsFilter(headerValue, rowValue) {
const query = String(headerValue || '').trim().toLowerCase().replaceAll('-', ' ');
const text = (Array.isArray(rowValue) ? rowValue : []).flatMap(observation => [
observation.type, observation.origin_asn, observation.state, observation.collector,
observation.peer_asn, observation.peer_address, observation.as_path, observation.communities
]).join(' ').toLowerCase().replaceAll('-', ' ');
return text.includes(query);
}
function provenanceFormatter(cell) {
const values = Array.isArray(cell.getValue()) ? cell.getValue() : [];
return escapeHtml(values.join(', ') || '-');
@@ -438,6 +489,19 @@
{title: 'Produced by', field: 'actions', formatter: provenanceFormatter, minWidth: 130, responsive: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter},
);
}
if (state.route === 'asn' && rows.some(row => Array.isArray(row.observations) && row.observations.length)) {
columns.push(
{title: 'Organization attributions', field: 'observations', formatter: asnAttributionsFormatter, minWidth: 420, widthGrow: 4, variableHeight: true, headerFilter: 'input', headerFilterFunc: asnAttributionsFilter, headerFilterPlaceholder: 'Filter organization evidence'},
{title: 'Sources', field: 'sources', formatter: provenanceFormatter, minWidth: 130, responsive: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter},
{title: 'Produced by', field: 'actions', formatter: provenanceFormatter, minWidth: 130, responsive: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter},
);
}
if (state.route === 'prefix' && rows.some(row => Array.isArray(row.observations) && row.observations.length)) {
columns.push(
{title: 'Routing evidence', field: 'observations', formatter: networkObservationsFormatter, minWidth: 420, widthGrow: 4, variableHeight: true, headerFilter: 'input', headerFilterFunc: networkObservationsFilter, headerFilterPlaceholder: 'Filter routing evidence'},
{title: 'Produced by', field: 'actions', formatter: provenanceFormatter, minWidth: 130, responsive: 2, headerFilter: 'input', headerFilterFunc: columnTextFilter},
);
}
if (state.route === 'hostname') {
columns.push({
title: 'Actions', field: 'value', formatter: resultActionFormatter, headerSort: false,
@@ -817,7 +881,8 @@
dns_recursive_depth: Number(form.get('dns_recursive_depth')),
dns_recursive_query_limit: Number(form.get('dns_recursive_query_limit')),
dns_recursive_runtime_seconds: Number(form.get('dns_recursive_runtime_seconds')),
dns_brute: form.has('dns_brute'), shodan: form.has('shodan'), screenshot: form.has('screenshot'),
dns_brute: form.has('dns_brute'), shodan: form.has('shodan'), routeviews: form.has('routeviews'),
screenshot: form.has('screenshot'),
takeover: form.has('takeover'), api_scan: form.has('api_scan'),
api_scan_paths: form.has('api_scan')
? String(form.get('api_scan_paths')).split(/\r?\n/).map(value => value.trim()).filter(Boolean)
@@ -259,6 +259,7 @@
<legend><span class="heading-with-help">Additional activity <button class="help-tip" type="button" aria-label="Explain additional activity" data-tooltip="These options expand authorization beyond selected sources. P1 performs DNS interaction; P2 can contact the target directly.">?</button></span></legend>
<div class="action-grid">
<label class="action-choice p0"><input type="checkbox" name="shodan"><span><strong>P0 · Shodan enrichment</strong><small>Query Shodan for discovered IP evidence.</small></span></label>
<label class="action-choice p0"><input type="checkbox" name="routeviews"><span><strong>P0 · RouteViews enrichment</strong><small>Enrich source-attributed IPs with external routing evidence.</small></span></label>
<label class="action-choice p1"><input type="checkbox" name="dns_lookup"><span><strong>P1 · DNS lookup</strong><small>Reverse-query every discovered IPv4 address and each address in its /24.</small></span></label>
<label class="action-choice p1"><input type="checkbox" name="dns_resolve"><span><strong>P1 · DNS resolution</strong><small>Resolve with the configured resolver addresses.</small></span></label>
<label class="action-choice p1"><input type="checkbox" name="dns_brute"><span><strong>P1 · DNS brute force</strong><small>Query candidate labels against DNS.</small></span></label>
+165
View File
@@ -0,0 +1,165 @@
from __future__ import annotations
import unicodedata
from dataclasses import dataclass
from datetime import UTC, datetime
from ipaddress import ip_address
from typing import Literal, cast
from theHarvester.lib.evidence_types import format_utc
from theHarvester.lib.result_values import normalize_asn
from theHarvester.lib.virtual_host import normalize_virtual_host_hostname
ProducerKind = Literal['source', 'action']
SubjectKind = Literal['hostname', 'ip']
MAX_ORGANIZATION_LABEL_LENGTH = 255
def _normalize_name(value: str, label: str, *, max_length: int = 255) -> str:
if not isinstance(value, str) or not (normalized := value.strip()):
raise ValueError(f'{label} must not be empty')
try:
normalized.encode('utf-8')
except UnicodeEncodeError as error:
raise ValueError(f'{label} is invalid') from error
if len(normalized) > max_length or any(unicodedata.category(character) in {'Cc', 'Cf'} for character in normalized):
raise ValueError(f'{label} is invalid')
return normalized
def _normalize_subject(kind: SubjectKind, value: str) -> str:
if kind == 'hostname':
return normalize_virtual_host_hostname(value)
if kind == 'ip':
if not isinstance(value, str) or '%' in value:
raise ValueError('ASN attribution IP subject must be a canonical IP address')
try:
return str(ip_address(value.strip()))
except ValueError as error:
raise ValueError('ASN attribution IP subject must be a canonical IP address') from error
raise ValueError('ASN attribution subject type must be hostname or ip')
def _normalize_time(value: datetime) -> datetime:
if not isinstance(value, datetime) or value.tzinfo is None or value.utcoffset() is None:
raise ValueError('ASN attribution collected_at must be timezone-aware')
return value.astimezone(UTC)
@dataclass(frozen=True, slots=True)
class AsnAttributionObservation:
producer_kind: ProducerKind
producer: str
asn: str
organization_label: str
subject_kind: SubjectKind
subject_value: str
collected_at: datetime
def __post_init__(self) -> None:
if self.producer_kind not in {'source', 'action'}:
raise ValueError('ASN attribution producer kind must be source or action')
object.__setattr__(self, 'producer', _normalize_name(self.producer, 'ASN attribution producer'))
object.__setattr__(self, 'asn', normalize_asn(self.asn))
object.__setattr__(
self,
'organization_label',
_normalize_name(
self.organization_label,
'ASN organization label',
max_length=MAX_ORGANIZATION_LABEL_LENGTH,
),
)
object.__setattr__(self, 'subject_value', _normalize_subject(self.subject_kind, self.subject_value))
object.__setattr__(self, 'collected_at', _normalize_time(self.collected_at))
def sort_key(self) -> tuple[object, ...]:
return (
self.asn,
self.producer_kind,
self.producer,
self.organization_label,
self.subject_kind,
self.subject_value,
self.collected_at,
)
def detail(self) -> dict[str, object]:
return {
'type': 'organization-attribution',
'producer_kind': self.producer_kind,
'producer': self.producer,
'organization_label': self.organization_label,
'subject': {'type': self.subject_kind, 'value': self.subject_value},
'collected_at': format_utc(self.collected_at),
}
def canonical_asn_attributions(
observations: tuple[AsnAttributionObservation, ...] | list[AsnAttributionObservation],
) -> tuple[AsnAttributionObservation, ...]:
return tuple(sorted(set(observations), key=AsnAttributionObservation.sort_key))
def asn_attribution_details(observations: tuple[AsnAttributionObservation, ...]) -> list[dict[str, object]]:
return [observation.detail() for observation in canonical_asn_attributions(observations)]
def parse_asn_attribution_details(asn: str, details: object) -> tuple[AsnAttributionObservation, ...]:
if not isinstance(details, list) or not details:
raise ValueError('ASN attribution details must be a non-empty array')
normalized_asn = normalize_asn(asn)
observations: list[AsnAttributionObservation] = []
for detail in details:
if not isinstance(detail, dict) or set(detail) != {
'type',
'producer_kind',
'producer',
'organization_label',
'subject',
'collected_at',
}:
raise ValueError('ASN attribution details must contain canonical observation objects')
subject = detail.get('subject')
if (
detail.get('type') != 'organization-attribution'
or not isinstance(subject, dict)
or set(subject)
!= {
'type',
'value',
}
):
raise ValueError('ASN attribution details must contain canonical observation objects')
collected_at = detail.get('collected_at')
producer_kind = detail.get('producer_kind')
producer = detail.get('producer')
organization_label = detail.get('organization_label')
subject_kind = subject.get('type')
subject_value = subject.get('value')
if (
producer_kind not in {'source', 'action'}
or not isinstance(producer, str)
or not isinstance(organization_label, str)
or subject_kind not in {'hostname', 'ip'}
or not isinstance(subject_value, str)
or not isinstance(collected_at, str)
):
raise ValueError('ASN attribution details must contain canonical observation values')
try:
parsed_time = datetime.fromisoformat(collected_at)
except ValueError as error:
raise ValueError('ASN attribution collected_at must be a canonical UTC timestamp') from error
observation = AsnAttributionObservation(
cast('ProducerKind', producer_kind),
producer,
normalized_asn,
organization_label,
cast('SubjectKind', subject_kind),
subject_value,
parsed_time,
)
if detail != observation.detail():
raise ValueError('ASN attribution details must use canonical structured evidence')
observations.append(observation)
return canonical_asn_attributions(observations)
+47 -1
View File
@@ -7,6 +7,12 @@ from typing import Self
from uuid import UUID, uuid4
from theHarvester.lib.active_evidence import ActiveEvidence
from theHarvester.lib.asn_attribution import (
AsnAttributionObservation,
asn_attribution_details,
canonical_asn_attributions,
parse_asn_attribution_details,
)
from theHarvester.lib.evidence_types import (
EVIDENCE_STATUSES,
EXECUTION_STATUSES,
@@ -80,7 +86,7 @@ def parse_result_jsonl(payload: bytes | str) -> tuple[dict[str, object], list[di
actions = record.get('actions', [])
result_kind = record.get('type')
allowed_keys = {'type', 'value', 'sources', 'actions'}
if result_kind in {'hostname', 'prefix'} and 'observations' in record:
if result_kind in {'asn', 'hostname', 'prefix'} and 'observations' in record:
allowed_keys.add('observations')
if result_kind == 'prefix':
allowed_keys.add('scope')
@@ -120,6 +126,12 @@ def parse_result_jsonl(payload: bytes | str) -> tuple[dict[str, object], list[di
except ValueError as error:
raise ValueError(f'JSONL prefix has invalid network observations: {error}') from error
record['observations'] = network_observation_details(network_observations)
elif result_kind == 'asn' and 'observations' in record:
try:
asn_attributions = parse_asn_attribution_details(record['value'], record.get('observations'))
except ValueError as error:
raise ValueError(f'JSONL ASN has invalid organization attributions: {error}') from error
record['observations'] = asn_attribution_details(asn_attributions)
return summary, findings
@@ -202,6 +214,7 @@ class CompletedResult:
active_evidence: ActiveEvidence = field(default_factory=ActiveEvidence)
virtual_hosts: tuple[VirtualHostObservation, ...] = ()
network_observations: tuple[NetworkObservation, ...] = ()
asn_attributions: tuple[AsnAttributionObservation, ...] = ()
evidence_status: EvidenceStatus | None = None
def __post_init__(self) -> None:
@@ -308,6 +321,32 @@ class CompletedResult:
not in origin_observations
):
raise ValueError('BGP route and RPKI observations require matching observed-origin evidence')
sorted_asn_attributions = canonical_asn_attributions(list(self.asn_attributions))
if self.asn_attributions != sorted_asn_attributions:
raise ValueError('ASN attributions must be deduplicated and sorted')
source_results = {(observation.source, observation.kind, observation.value) for observation in self.observations}
action_results = {
(action, observation.kind, observation.value) for action, observation in self.active_evidence.observations
}
for attribution in self.asn_attributions:
if attribution.collected_at < self.started_at or attribution.collected_at > self.completed_at:
raise ValueError('ASN attribution collection time must fall within the completed run')
if ('asn', attribution.asn) not in result_set or (
attribution.subject_kind,
attribution.subject_value,
) not in result_set:
raise ValueError('ASN attribution must reference completed ASN and subject results')
producer_result_set = source_results if attribution.producer_kind == 'source' else action_results
if (
attribution.producer,
'asn',
attribution.asn,
) not in producer_result_set or (
attribution.producer,
attribution.subject_kind,
attribution.subject_value,
) not in producer_result_set:
raise ValueError('ASN attribution must reference matching producer provenance')
if self.evidence_status is not None and self.evidence_status not in EVIDENCE_STATUSES:
raise ValueError('evidence status must be complete, partial, or failed')
@@ -325,6 +364,7 @@ class CompletedResult:
active_evidence: ActiveEvidence | None = None,
virtual_hosts: Iterable[VirtualHostObservation] = (),
network_observations: Iterable[NetworkObservation] = (),
asn_attributions: Iterable[AsnAttributionObservation] = (),
evidence_status: EvidenceStatus | None = None,
) -> Self:
completed_active_evidence = active_evidence if active_evidence is not None else ActiveEvidence()
@@ -353,6 +393,7 @@ class CompletedResult:
active_evidence=completed_active_evidence,
virtual_hosts=completed_virtual_hosts,
network_observations=canonical_network_observations(network_observations),
asn_attributions=canonical_asn_attributions(list(asn_attributions)),
evidence_status=evidence_status,
)
@@ -415,6 +456,9 @@ class CompletedResult:
network_by_prefix: dict[str, list[NetworkObservation]] = {}
for observation in self.network_observations:
network_by_prefix.setdefault(observation.prefix, []).append(observation)
attribution_by_asn: dict[str, list[AsnAttributionObservation]] = {}
for attribution in self.asn_attributions:
attribution_by_asn.setdefault(attribution.asn, []).append(attribution)
records: list[dict[str, object]] = []
for kind, value in self.results:
record: dict[str, object] = {
@@ -430,5 +474,7 @@ class CompletedResult:
record['observations'] = virtual_host_details(virtual_hosts)
elif kind == 'prefix' and (network_observations := network_by_prefix.get(value)):
record['observations'] = network_observation_details(tuple(network_observations))
elif kind == 'asn' and (asn_attributions := attribution_by_asn.get(value)):
record['observations'] = asn_attribution_details(tuple(asn_attributions))
records.append(record)
return records
+6
View File
@@ -181,6 +181,7 @@ class Core:
'pentestTools': ('key',),
'projectDiscovery': ('key',),
'rocketreach': ('key',),
'routeviews': ('key',),
'securityscorecard': ('key',),
'securityTrails': ('key',),
'sherlockeye': ('key',),
@@ -331,6 +332,11 @@ class Core:
def rocketreach_key() -> str:
return Core._api_key_value('rocketreach')
@staticmethod
def routeviews_key() -> str | None:
value = Core.api_keys().get('routeviews', {}).get('key')
return value.strip() if isinstance(value, str) and value.strip() else None
@staticmethod
def securityscorecard_key() -> str:
return Core._api_key_value('securityscorecard')
+87
View File
@@ -39,6 +39,12 @@ from theHarvester.lib.active_evidence import (
ActiveEvidence,
ArtifactReference,
)
from theHarvester.lib.asn_attribution import (
AsnAttributionObservation,
ProducerKind,
SubjectKind,
canonical_asn_attributions,
)
from theHarvester.lib.completed_result import (
CompletedResult,
ExecutionStatus,
@@ -182,6 +188,37 @@ class _ResultOriginRow(_Base):
execution_position: Mapped[int] = mapped_column(primary_key=True)
class _AsnAttributionRow(_Base):
"""One sourced organization label connecting an ASN to a hostname or IP result."""
__tablename__ = 'asn_attributions'
__table_args__ = (
ForeignKeyConstraint(
('run_id', 'asn_result_position'),
('results.run_id', 'results.position'),
ondelete='CASCADE',
),
ForeignKeyConstraint(
('run_id', 'subject_result_position'),
('results.run_id', 'results.position'),
ondelete='CASCADE',
),
ForeignKeyConstraint(
('run_id', 'execution_position'),
('executions.run_id', 'executions.position'),
ondelete='CASCADE',
),
)
run_id: Mapped[str] = mapped_column(Text, primary_key=True)
position: Mapped[int] = mapped_column(primary_key=True)
asn_result_position: Mapped[int]
subject_result_position: Mapped[int]
execution_position: Mapped[int]
organization_label: Mapped[str] = mapped_column(Text)
collected_at: Mapped[str] = mapped_column(Text)
class _ArtifactRow(_Base):
"""Metadata for a file created by an action and attached to one result."""
@@ -806,6 +843,18 @@ class ResultStore:
)
for position, (action, artifact) in enumerate(result.active_evidence.artifacts)
)
session.add_all(
_AsnAttributionRow(
run_id=run_id,
position=position,
asn_result_position=result_positions[('asn', attribution.asn)],
subject_result_position=result_positions[(attribution.subject_kind, attribution.subject_value)],
execution_position=execution_positions[(attribution.producer_kind, attribution.producer)],
organization_label=attribution.organization_label,
collected_at=attribution.collected_at.isoformat(),
)
for position, attribution in enumerate(result.asn_attributions)
)
await session.commit()
except IntegrityError as error:
duplicate_codes = {
@@ -835,6 +884,13 @@ class ResultStore:
select(_ArtifactRow).where(_ArtifactRow.run_id == str(run_id)).order_by(_ArtifactRow.position)
)
).all()
attribution_rows = (
await session.scalars(
select(_AsnAttributionRow)
.where(_AsnAttributionRow.run_id == str(run_id))
.order_by(_AsnAttributionRow.position)
)
).all()
results_by_position = {row.position: row for row in rows}
executions_by_position = {row.position: row for row in execution_rows}
vhost_execution_positions = {
@@ -933,6 +989,36 @@ class ResultStore:
stop_reason=execution_row.stop_reason,
)
)
asn_attributions: list[AsnAttributionObservation] = []
for attribution in attribution_rows:
asn_result = results_by_position.get(attribution.asn_result_position)
subject_result = results_by_position.get(attribution.subject_result_position)
attribution_execution = executions_by_position.get(attribution.execution_position)
if asn_result is None or subject_result is None or attribution_execution is None:
raise ResultStoreError('Persisted ASN attribution references missing evidence')
if (
asn_result.kind != 'asn'
or attribution_execution.producer_kind not in {'source', 'action'}
or subject_result.kind not in {'hostname', 'ip'}
):
raise ResultStoreError('Persisted ASN attribution is invalid')
try:
asn_attributions.append(
AsnAttributionObservation(
cast('ProducerKind', attribution_execution.producer_kind),
attribution_execution.name,
asn_result.value,
attribution.organization_label,
cast('SubjectKind', subject_result.kind),
subject_result.value,
datetime.datetime.fromisoformat(attribution.collected_at),
)
)
except ValueError as error:
raise ResultStoreError('Persisted ASN attribution is invalid') from error
canonical_attributions = canonical_asn_attributions(asn_attributions)
if tuple(asn_attributions) != canonical_attributions:
raise ResultStoreError('Persisted ASN attribution is invalid')
return CompletedResult(
run_id=UUID(parent.run_id),
target=parent.target,
@@ -955,6 +1041,7 @@ class ResultStore:
active_evidence=ActiveEvidence(executions=tuple(action_executions)),
virtual_hosts=tuple(sorted(set(virtual_hosts), key=VirtualHostObservation.sort_key)),
network_observations=tuple(sorted(set(network_observations), key=network_observation_sort_key)),
asn_attributions=canonical_attributions,
evidence_status=cast('EvidenceStatus', parent.evidence_status) if parent.evidence_status is not None else None,
)
+2
View File
@@ -26,6 +26,7 @@ class EnumerationOptions:
limit: int = DEFAULT_RESULT_LIMIT
start: int = DEFAULT_RESULT_START
proxies: bool = False
routeviews: bool = False
shodan: bool = False
screenshot: str = ''
dns_server: str | None = None
@@ -60,6 +61,7 @@ class EnumerationOptions:
limit=getattr(value, 'limit', DEFAULT_RESULT_LIMIT),
start=getattr(value, 'start', DEFAULT_RESULT_START),
proxies=getattr(value, 'proxies', False),
routeviews=getattr(value, 'routeviews', False),
shodan=getattr(value, 'shodan', False),
screenshot=getattr(value, 'screenshot', ''),
dns_server=getattr(value, 'dns_server', None),
+42 -20
View File
@@ -197,9 +197,32 @@ def _encoded_detail_size(observation: NetworkObservation) -> int:
def _rpki_identity(observation: RpkiValidationObservation) -> tuple[object, ...]:
return observation.action, observation.prefix, observation.origin_asn, observation.state
def _rpki_conflict_identity(observation: RpkiValidationObservation) -> tuple[object, ...]:
return observation.action, observation.prefix, observation.origin_asn, observation.observed_at
def _network_observation_identity(observation: NetworkObservation) -> tuple[object, ...]:
if isinstance(observation, PrefixOriginObservation):
return type(observation), observation.action, observation.prefix, observation.origin_asn
if isinstance(observation, BgpRouteObservation):
return (
type(observation),
observation.action,
observation.prefix,
observation.origin_asn,
observation.collector,
observation.peer_asn,
observation.peer_address,
observation.as_path,
observation.communities,
observation.observed_at,
)
return type(observation), *_rpki_identity(observation)
class NetworkEvidenceAccumulator:
"""Incrementally deduplicate network evidence within its persisted envelope."""
@@ -216,7 +239,7 @@ class NetworkEvidenceAccumulator:
self._max_observations_per_prefix = max_observations_per_prefix
self._max_details_bytes = max_details_bytes
self._observations: set[NetworkObservation] = set()
self._origin_identities: set[tuple[str, str, str]] = set()
self._observation_identities: set[tuple[object, ...]] = set()
self._rpki_states: dict[tuple[object, ...], RpkiState] = {}
self._prefix_counts: Counter[str] = Counter()
self._prefix_bytes: dict[str, int] = {}
@@ -224,16 +247,15 @@ class NetworkEvidenceAccumulator:
def add(self, observation: NetworkObservation) -> bool:
if not isinstance(observation, (PrefixOriginObservation, BgpRouteObservation, RpkiValidationObservation)):
raise TypeError('network evidence contains an unsupported observation')
if isinstance(observation, PrefixOriginObservation):
origin_identity = observation.action, observation.prefix, str(observation.origin_asn)
if origin_identity in self._origin_identities:
return False
elif observation in self._observations:
return False
if isinstance(observation, RpkiValidationObservation):
identity = _rpki_identity(observation)
identity = _rpki_conflict_identity(observation)
if (existing := self._rpki_states.get(identity)) is not None and existing != observation.state:
raise ValueError('network evidence contains conflicting RPKI states')
observation_identity = _network_observation_identity(observation)
if observation_identity in self._observation_identities:
if isinstance(observation, RpkiValidationObservation):
self._rpki_states[identity] = observation.state
return False
count = self._prefix_counts[observation.prefix] + 1
if count > self._max_observations_per_prefix:
@@ -243,11 +265,10 @@ class NetworkEvidenceAccumulator:
raise NetworkEvidenceLimitError('network evidence details exceed the serialized size limit')
self._observations.add(observation)
self._observation_identities.add(observation_identity)
self._prefix_counts[observation.prefix] = count
self._prefix_bytes[observation.prefix] = encoded_size
if isinstance(observation, PrefixOriginObservation):
self._origin_identities.add(origin_identity)
elif isinstance(observation, RpkiValidationObservation):
if isinstance(observation, RpkiValidationObservation):
self._rpki_states[identity] = observation.state
return True
@@ -268,18 +289,19 @@ def network_observation_sort_key(observation: NetworkObservation) -> tuple[objec
def canonical_network_observations(observations: Iterable[NetworkObservation]) -> tuple[NetworkObservation, ...]:
canonical = tuple(sorted(set(observations), key=network_observation_sort_key))
deduplicated: dict[tuple[object, ...], NetworkObservation] = {}
rpki_states: dict[tuple[object, ...], RpkiState] = {}
for observation in sorted(set(observations), key=network_observation_sort_key):
if isinstance(observation, RpkiValidationObservation):
identity = _rpki_conflict_identity(observation)
if (existing := rpki_states.get(identity)) is not None and existing != observation.state:
raise ValueError('network evidence contains conflicting RPKI states')
rpki_states[identity] = observation.state
deduplicated.setdefault(_network_observation_identity(observation), observation)
canonical = tuple(sorted(deduplicated.values(), key=network_observation_sort_key))
counts = Counter(observation.prefix for observation in canonical)
if any(count > MAX_NETWORK_OBSERVATIONS_PER_PREFIX for count in counts.values()):
raise ValueError('network evidence contains too many observations for one prefix')
rpki_states: dict[tuple[object, ...], RpkiState] = {}
for observation in canonical:
if not isinstance(observation, RpkiValidationObservation):
continue
identity = _rpki_identity(observation)
if (existing := rpki_states.get(identity)) is not None and existing != observation.state:
raise ValueError('network evidence contains conflicting RPKI states')
rpki_states[identity] = observation.state
return canonical
+492
View File
@@ -0,0 +1,492 @@
from __future__ import annotations
import asyncio
import json
import math
import time
from dataclasses import dataclass
from datetime import UTC, datetime
from ipaddress import ip_address, ip_network
from typing import TYPE_CHECKING, Any, cast
from urllib.parse import quote
from theHarvester.lib.core import AsyncFetcher, FetcherResponse, ResponseStreamError
from theHarvester.lib.network_evidence import (
MAX_NETWORK_DETAILS_BYTES,
MAX_NETWORK_OBSERVATIONS_PER_PREFIX,
BgpRouteObservation,
NetworkEvidenceAccumulator,
NetworkEvidenceLimitError,
PrefixOriginObservation,
RpkiValidationObservation,
)
from theHarvester.lib.result_values import normalize_asn, normalize_prefix
if TYPE_CHECKING:
from collections.abc import Iterable
from theHarvester.lib.evidence_types import ExecutionStatus
from theHarvester.lib.network_evidence import NetworkObservation, RpkiState
ROUTEVIEWS_BASE = 'https://api.routeviews.org'
ROUTEVIEWS_GUEST_BASE = f'{ROUTEVIEWS_BASE}/guest'
MAX_ROUTEVIEWS_REQUESTS = 300
MAX_ROUTEVIEWS_RUNTIME_SECONDS = 300.0
MAX_ROUTEVIEWS_INPUT_ITEMS = 1_000
MAX_ROUTEVIEWS_RUN_ITEMS = 100_000
MAX_ROUTEVIEWS_RUN_JSON_BYTES = 32 * 1024 * 1024
ROUTEVIEWS_GUEST_INTERVAL_SECONDS = 1.0
ROUTEVIEWS_AUTHENTICATED_INTERVAL_SECONDS = 0.1
ROUTEVIEWS_REQUEST_TIMEOUT_SECONDS = 30
async def _fetch_json(url: str, **kwargs: Any) -> FetcherResponse:
return await AsyncFetcher.fetch_json(url, **kwargs)
async def _sleep(seconds: float) -> None:
await asyncio.sleep(seconds)
def _monotonic() -> float:
return time.monotonic()
def _now() -> datetime:
return datetime.now(UTC)
@dataclass(frozen=True, slots=True)
class RouteViewsResult:
prefixes: tuple[str, ...]
origin_asns: tuple[str, ...]
observations: tuple[NetworkObservation, ...]
request_count: int
error_count: int
status: ExecutionStatus
error_type: str | None = None
stop_reason: str | None = None
class RouteViewsCancelled(asyncio.CancelledError):
def __init__(self, result: RouteViewsResult) -> None:
self.result = result
super().__init__('RouteViews enrichment cancelled')
class _RouteViewsStopError(Exception):
def __init__(self, error_type: str, stop_reason: str, *, terminal: bool = True) -> None:
self.error_type = error_type
self.stop_reason = stop_reason
self.terminal = terminal
super().__init__(stop_reason)
def _canonical_asns(values: Iterable[str | int]) -> tuple[tuple[str, ...], bool]:
normalized: set[str] = set()
if isinstance(values, (str, bytes)):
return (), False
truncated = False
for index, value in enumerate(values):
if index >= MAX_ROUTEVIEWS_INPUT_ITEMS:
truncated = True
break
if isinstance(value, str) and len(value) > 32:
continue
try:
normalized.add(normalize_asn(value))
except (TypeError, ValueError):
continue
return tuple(sorted(normalized, key=lambda value: int(value[2:]))), truncated
def _canonical_network_seeds(values: Iterable[str]) -> tuple[tuple[str, ...], bool]:
normalized: set[str] = set()
if isinstance(values, (str, bytes)):
return (), False
truncated = False
for index, value in enumerate(values):
if index >= MAX_ROUTEVIEWS_INPUT_ITEMS:
truncated = True
break
if not isinstance(value, str) or len(value) > 128 or '%' in value:
continue
try:
normalized.add(normalize_prefix(value) if '/' in value else str(ip_address(value.strip())))
except ValueError:
continue
def sort_key(value: str) -> tuple[int, int, int]:
network = ip_network(value, strict=False) if '/' in value else ip_network(value)
return network.version, int(network.network_address), network.prefixlen
return tuple(sorted(normalized, key=sort_key)), truncated
def _provider_time(value: object) -> datetime:
if not isinstance(value, str):
raise ValueError('RouteViews timestamp must be a string')
try:
parsed = datetime.fromisoformat(value)
if parsed.tzinfo is None or parsed.utcoffset() is None:
raise ValueError('RouteViews timestamp must be timezone-aware')
return parsed.astimezone(UTC)
except (OverflowError, ValueError) as error:
raise ValueError('RouteViews timestamp is invalid') from error
def _rpki_state(value: object) -> RpkiState:
if value == 'notfound':
return 'not-found'
if value in {'valid', 'invalid', 'not-found'}:
return cast('RpkiState', value)
raise ValueError('RouteViews RPKI state is invalid')
class _RouteViewsRuntime:
def __init__(self, api_key: str | None = None) -> None:
self.base_url = ROUTEVIEWS_BASE if api_key else ROUTEVIEWS_GUEST_BASE
self.headers = {'Api-Key': api_key} if api_key else None
self.request_interval = ROUTEVIEWS_AUTHENTICATED_INTERVAL_SECONDS if api_key else ROUTEVIEWS_GUEST_INTERVAL_SECONDS
self.started_at = _monotonic()
self.last_request_at: float | None = None
self.request_count = 0
self.error_count = 0
self.reported_error_type: str | None = None
self.reported_stop_reason: str | None = None
self.run_item_count = 0
self.run_json_bytes = 0
self.prefixes: set[str] = set()
self.origin_asns: set[str] = set()
self.evidence = NetworkEvidenceAccumulator(
max_observations_per_prefix=MAX_NETWORK_OBSERVATIONS_PER_PREFIX,
max_details_bytes=MAX_NETWORK_DETAILS_BYTES,
)
def _record_error(self, error_type: str, stop_reason: str, *, override: bool = False) -> None:
self.error_count += 1
if self.reported_error_type is None or override:
self.reported_error_type = error_type
self.reported_stop_reason = stop_reason
def _charge_items(self, count: int) -> None:
self.run_item_count += count
if self.run_item_count > MAX_ROUTEVIEWS_RUN_ITEMS:
raise _RouteViewsStopError('RouteViewsLimitError', 'result-limit')
def _accept_observation(self, observation: NetworkObservation) -> bool:
try:
return self.evidence.add(observation)
except NetworkEvidenceLimitError as error:
raise _RouteViewsStopError('RouteViewsLimitError', 'result-limit') from error
def _add_origin(self, prefix: object, origin_asn: object, collected_at: datetime) -> tuple[str, str]:
if not isinstance(prefix, str) or isinstance(origin_asn, bool) or not isinstance(origin_asn, (str, int)):
raise ValueError('RouteViews origin must contain a prefix and ASN')
normalized_prefix = normalize_prefix(prefix)
normalized_asn = normalize_asn(origin_asn)
key = normalized_prefix, normalized_asn
self._accept_observation(
PrefixOriginObservation(
action='routeviews',
prefix=normalized_prefix,
origin_asn=normalized_asn,
collected_at=collected_at,
)
)
self.prefixes.add(normalized_prefix)
self.origin_asns.add(normalized_asn)
return key
def _add_route(
self,
*,
prefix: object,
origin_asn: object,
peer: object,
collected_at: datetime,
) -> None:
if not isinstance(peer, dict):
raise ValueError('RouteViews reporting peer must be an object')
collector = peer.get('collector')
peer_asn = peer.get('peer_asn')
peer_address = peer.get('peer_addr')
as_path = peer.get('as_path')
communities = peer.get('communities')
if (
not isinstance(collector, str)
or isinstance(peer_asn, bool)
or not isinstance(peer_asn, (str, int))
or not isinstance(peer_address, str)
or not isinstance(as_path, str)
or not isinstance(communities, str)
):
raise ValueError('RouteViews reporting peer has invalid fields')
normalized_prefix, normalized_asn = self._add_origin(prefix, origin_asn, collected_at)
route = BgpRouteObservation(
action='routeviews',
prefix=normalized_prefix,
origin_asn=normalized_asn,
collector=collector,
peer_asn=peer_asn,
peer_address=peer_address,
as_path=as_path,
communities=communities,
observed_at=_provider_time(peer.get('timestamp')),
collected_at=collected_at,
)
self._accept_observation(route)
def _add_rpki(
self,
*,
prefix: object,
origin_asn: object,
state: object,
observed_at: datetime,
collected_at: datetime,
) -> None:
normalized_prefix, normalized_asn = self._add_origin(prefix, origin_asn, collected_at)
observation = RpkiValidationObservation(
action='routeviews',
prefix=normalized_prefix,
origin_asn=normalized_asn,
state=_rpki_state(state),
observed_at=observed_at,
collected_at=collected_at,
)
self._accept_observation(observation)
async def _request(self, url: str, *, params: object = '') -> FetcherResponse:
if self.request_count >= MAX_ROUTEVIEWS_REQUESTS:
raise _RouteViewsStopError('RouteViewsLimitError', 'request-limit')
elapsed = _monotonic() - self.started_at
if elapsed >= MAX_ROUTEVIEWS_RUNTIME_SECONDS:
raise _RouteViewsStopError('RouteViewsLimitError', 'runtime-limit')
if self.last_request_at is not None:
delay = self.request_interval - (_monotonic() - self.last_request_at)
if delay > 0:
if elapsed + delay >= MAX_ROUTEVIEWS_RUNTIME_SECONDS:
raise _RouteViewsStopError('RouteViewsLimitError', 'runtime-limit')
await _sleep(delay)
remaining = MAX_ROUTEVIEWS_RUNTIME_SECONDS - (_monotonic() - self.started_at)
if remaining <= 0:
raise _RouteViewsStopError('RouteViewsLimitError', 'runtime-limit')
self.request_count += 1
self.last_request_at = _monotonic()
request_kwargs: dict[str, Any] = {
'params': params,
'request_timeout': min(ROUTEVIEWS_REQUEST_TIMEOUT_SECONDS, max(1, math.ceil(remaining))),
}
if self.headers is not None:
request_kwargs['headers'] = self.headers
try:
async with asyncio.timeout(remaining):
response = await _fetch_json(url, **request_kwargs)
except ResponseStreamError as error:
raise _RouteViewsStopError(
type(error).__name__,
error.reason,
terminal=error.reason == 'response-limit',
) from error
except TimeoutError as error:
raise _RouteViewsStopError('RouteViewsLimitError', 'runtime-limit') from error
if response.status == 429:
raise _RouteViewsStopError('HTTPStatusError', 'http-429')
if not 200 <= response.status < 300:
raise _RouteViewsStopError(
'HTTPStatusError',
f'http-{response.status}',
terminal=response.status in {401, 403},
)
if response.body is None:
raise _RouteViewsStopError('ValueError', 'invalid-response', terminal=False)
try:
encoded_size = len(
json.dumps(response.body, ensure_ascii=False, separators=(',', ':'), allow_nan=False).encode('utf-8')
)
except (RecursionError, TypeError, ValueError, UnicodeEncodeError) as error:
raise _RouteViewsStopError('ValueError', 'invalid-response', terminal=False) from error
self.run_json_bytes += encoded_size
if self.run_json_bytes > MAX_ROUTEVIEWS_RUN_JSON_BYTES:
raise _RouteViewsStopError('RouteViewsLimitError', 'result-limit')
return response
def _require_items(self, value: object, label: str) -> list[object]:
if not isinstance(value, list):
raise ValueError(f'RouteViews {label} must be an array')
self._charge_items(len(value))
return value
def _parse_asn(self, body: object, origin_asn: str, collected_at: datetime) -> None:
for prefix in self._require_items(body, 'ASN response'):
try:
self._add_origin(prefix, origin_asn, collected_at)
except ValueError:
self._record_error('ValueError', 'invalid-response')
def _parse_rpki(self, body: object, origin_asn: str, collected_at: datetime) -> None:
if not isinstance(body, dict):
raise ValueError('RouteViews RPKI response must be an object')
record = body.get(origin_asn[2:])
if record is None:
return
if not isinstance(record, dict):
raise ValueError('RouteViews RPKI ASN record must be an object or null')
observed_at = _provider_time(record.get('timestamp'))
for item in self._require_items(record.get('prefix'), 'RPKI prefix list'):
try:
if not isinstance(item, dict) or len(item) != 1:
raise ValueError('RouteViews RPKI prefix entry must contain one prefix and state')
prefix, state = next(iter(item.items()))
self._add_rpki(
prefix=prefix,
origin_asn=origin_asn,
state=state,
observed_at=observed_at,
collected_at=collected_at,
)
except ValueError:
self._record_error('ValueError', 'invalid-response')
def _parse_prefix(self, body: object, seed: str, collected_at: datetime) -> None:
requested_network = ip_network(seed, strict=False) if '/' in seed else None
requested_address = ip_address(seed) if requested_network is None else None
matching_groups: list[dict[object, object]] = []
longest_prefix_length = -1
for group in self._require_items(body, 'prefix response'):
try:
if not isinstance(group, dict):
raise ValueError('RouteViews prefix group must be an object')
prefix = group.get('prefix')
if not isinstance(prefix, str):
raise ValueError('RouteViews prefix group must contain a prefix')
returned_network = ip_network(normalize_prefix(prefix))
if (requested_network is not None and returned_network != requested_network) or (
requested_address is not None and requested_address not in returned_network
):
raise ValueError('RouteViews prefix response is unrelated to its seed')
if requested_address is not None:
if returned_network.prefixlen > longest_prefix_length:
matching_groups.clear()
longest_prefix_length = returned_network.prefixlen
elif returned_network.prefixlen < longest_prefix_length:
continue
matching_groups.append(group)
except ValueError:
self._record_error('ValueError', 'invalid-response')
for group in matching_groups:
prefix = group['prefix']
try:
origin_asn = group.get('origin_asn')
self._add_origin(prefix, origin_asn, collected_at)
except ValueError:
self._record_error('ValueError', 'invalid-response')
continue
try:
self._add_rpki(
prefix=prefix,
origin_asn=origin_asn,
state=group.get('rpki_state'),
observed_at=collected_at,
collected_at=collected_at,
)
except ValueError:
self._record_error('ValueError', 'invalid-response')
try:
peers = self._require_items(group.get('reporting_peers'), 'reporting peers')
except ValueError:
self._record_error('ValueError', 'invalid-response')
continue
for peer in peers:
try:
self._add_route(prefix=prefix, origin_asn=origin_asn, peer=peer, collected_at=collected_at)
except ValueError:
self._record_error('ValueError', 'invalid-response')
def _result(self) -> RouteViewsResult:
observations = self.evidence.observations()
if self.error_count == 0:
status: ExecutionStatus = 'completed'
stop_reason = None if self.prefixes else 'no-results'
elif self.prefixes:
status = 'partial'
stop_reason = self.reported_stop_reason
elif self.reported_stop_reason == 'http-429':
status = 'rate-limited'
stop_reason = self.reported_stop_reason
else:
status = 'failed'
stop_reason = self.reported_stop_reason
return RouteViewsResult(
prefixes=tuple(sorted(self.prefixes)),
origin_asns=tuple(sorted(self.origin_asns, key=lambda value: int(value[2:]))),
observations=observations,
request_count=self.request_count,
error_count=self.error_count,
status=status,
error_type=self.reported_error_type,
stop_reason=stop_reason,
)
async def run(self, asns: Iterable[str | int], network_seeds: Iterable[str]) -> RouteViewsResult:
canonical_asns, asns_truncated = _canonical_asns(asns)
canonical_seeds, seeds_truncated = _canonical_network_seeds(network_seeds)
if asns_truncated or seeds_truncated:
self._record_error('RouteViewsLimitError', 'input-limit')
if not canonical_asns and not canonical_seeds:
if self.error_count:
return self._result()
return RouteViewsResult((), (), (), 0, 0, 'skipped', stop_reason='no-input')
async def collect(url: str, parser: Any, *, params: object = '') -> None:
try:
response = await self._request(url, params=params)
parser(response.body, _now())
except _RouteViewsStopError as error:
self._record_error(error.error_type, error.stop_reason, override=error.terminal)
if error.terminal:
raise
except (TypeError, ValueError):
self._record_error('ValueError', 'invalid-response')
try:
for origin_asn in canonical_asns:
await collect(
f'{self.base_url}/asn/{origin_asn[2:]}',
lambda body, collected_at, asn=origin_asn: self._parse_asn(body, asn, collected_at),
)
await collect(
f'{self.base_url}/rpki',
lambda body, collected_at, asn=origin_asn: self._parse_rpki(body, asn, collected_at),
params={'asn': origin_asn[2:]},
)
for seed in canonical_seeds:
query_seed = seed if '/' in seed else str(ip_network(seed, strict=False))
await collect(
f'{self.base_url}/prefix/{quote(query_seed, safe="")}',
lambda body, collected_at, requested_seed=seed: self._parse_prefix(body, requested_seed, collected_at),
params={'strict-match': 'yes'} if '/' in seed else '',
)
except _RouteViewsStopError:
return self._result()
except asyncio.CancelledError:
self._record_error('CancelledError', 'cancelled', override=True)
raise RouteViewsCancelled(self._result()) from None
return self._result()
async def enrich_routeviews(
asns: Iterable[str | int],
network_seeds: Iterable[str],
*,
api_key: str | None = None,
) -> RouteViewsResult:
"""Collect bounded routing evidence for caller-approved network pivots.
Domain runs automatically pass harvested IPs that have sourced IP-to-ASN
attribution. Bare ASN findings are not expanded into complete prefix
inventories; that requires an explicit ASN target.
"""
return await _RouteViewsRuntime(api_key).run(asns, network_seeds)
+1
View File
@@ -15,6 +15,7 @@ ACTION_ACTIVITIES: Final = {
'dns-lookup': ActivityClass.DNS,
'dns-recursive': ActivityClass.DNS,
'dns-resolve': ActivityClass.DNS,
'routeviews': ActivityClass.PASSIVE,
'shodan': ActivityClass.PASSIVE,
'api-scan': ActivityClass.DIRECT,
'screenshot': ActivityClass.DIRECT,