diff --git a/AGENTS.md b/AGENTS.md index c4f7e3bb..e972aca3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,3 +28,10 @@ Read [CONTEXT.md](CONTEXT.md) when changing discovery terminology, evidence clas - Typing: `uv run mypy theHarvester` Run focused checks first and expand according to risk. Report any skipped check and its reason. + +### Test budget + +- During implementation, run the narrowest test that covers the changed behavior. Do not rerun the full suite after every small edit. +- Run the full non-browser suite once at the publication head. Dependent stack layers do not need to repeat it unless they change Python behavior. +- Run the HarvestView browser suite once at the final UI head or rely on its GitHub workflow. Static UI edits should use focused UI tests and a JavaScript syntax check first. +- Before retrying a long-running test, confirm the previous process exited. Poll the existing command or stop only its exact owned process instead of starting an overlapping run. diff --git a/CHANGELOG.md b/CHANGELOG.md index b90ca292..96434d70 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Added bounded, keyless subdomain discovery through Arquivo.pt's public CDX API with offline response contracts. - Added transactional SQLite storage and loading for completed full-pipeline runs without changing legacy result rows. - Added deterministic JSONL report companions finalized after selected one-shot actions complete. +- Added a unified model and SQLite schema for active-action provenance and screenshot artifact metadata. +- Added authenticated bulk import for completed runs from validated theHarvester SQLite databases. +- Added bounded custom endpoint-path input for REST API scans without exposing server-side file paths. +- Recorded DNS resolution, recursive DNS, DNS brute force, and PTR lookup outcomes through the unified action model. +- Recorded takeover, Shodan, and API endpoint scan outcomes through the unified action model. - Added normalized BuiltWith framework, language, server, CMS, and analytics findings to JSONL and completed-result SQLite output. - Added DNSDB passive DNS discovery with API key configuration, shared transport handling, result parsing, and offline tests ([9b41b78e](https://github.com/laramies/theHarvester/commit/9b41b78e), [aba9fec6](https://github.com/laramies/theHarvester/commit/aba9fec6)). - Added `--verbose` diagnostic logging while keeping normal operator output available at the default log level ([8a7b8b71](https://github.com/laramies/theHarvester/commit/8a7b8b71)). @@ -22,6 +27,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Added root contributor and security policies, structured issue forms, repository agent guidance, discovery terminology, and an operator-focused documentation wiki ([d090a29a](https://github.com/laramies/theHarvester/commit/d090a29a), [7c491ef5](https://github.com/laramies/theHarvester/commit/7c491ef5), [8b9d420b](https://github.com/laramies/theHarvester/commit/8b9d420b)). ### Changed +- Standardized SQLite, JSONL, API, and HarvestView result names on `hostname` and `ip` without a presentation alias. +- Standardized URL-producing adapters, JSON, JSONL, SQLite, and API evidence on one `url` result kind while preserving producer provenance. - Fixed proxied POST requests so they retain the request method, body, and query parameters. - Migrated Pentest-Tools discovery to its API v2 Bearer-authenticated scan, status, and output endpoints. - Included HIBP verified-domain in `all` and matching capability selectors like every other P0 source, with REST operator authentication applied after source expansion when its provider key is configured. @@ -35,20 +42,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Replaced deprecated hostname resolution with `getaddrinfo`-based handling ([6a847435](https://github.com/laramies/theHarvester/commit/6a847435)). - Reworked routine CI to use read-only permissions, non-mutating Ruff checks, offline tests, and explicit opt-in live provider checks ([72e5820f](https://github.com/laramies/theHarvester/commit/72e5820f)). - Grouped GitHub Actions, Python, and Docker Dependabot updates with a seven-day cooldown, and added a seven-day `uv` dependency freshness window ([7a947b66](https://github.com/laramies/theHarvester/commit/7a947b66), [52a79cdb](https://github.com/laramies/theHarvester/commit/52a79cdb)). -- Updated runtime dependencies: `aiohttp` to `3.14.1`, `beautifulsoup4` to `4.15.0`, `certifi` to `2026.6.17`, `fastapi` to `0.138.1`, `slowapi` to `0.1.10`, `ujson` to `5.13.0`, and `uvicorn` to `0.49.0`. +- Updated runtime dependencies: `aiohttp` to `3.14.1`, `beautifulsoup4` to `4.15.0`, `certifi` to `2026.6.17`, `fastapi` to `0.138.1`, `ujson` to `5.13.0`, and `uvicorn` to `0.49.0`. - Updated development dependencies: `pytest` to `9.1.1`, `ruff` to `0.15.20`, and `ty` to `0.0.54`. - Updated CI and container maintenance pins, including `actions/checkout`, `astral-sh/setup-uv`, `astral-sh/ruff-action`, `github/codeql-action`, StepSecurity Harden-Runner, Docker actions, and the Python base image. - Expanded offline regression coverage for discovery providers, configuration contracts, logging, output, documentation, workflow policy, and scope boundaries. ### Removed +- Removed the obsolete bundled IP-range and resolver snapshots. +- Removed the REST API's built-in SlowAPI request limiter and its launcher option without adding a replacement. - Removed Bitbucket domain discovery because its current REST APIs require workspace, repository, or user scope that the domain-only CLI contract cannot provide. - Removed the nonfunctional ThreatCrowd source because its service hostnames terminate at deleted AWS load balancers and return NXDOMAIN; OTX remains available through its separate adapter. ### Fixed +- Kept API endpoint scan URLs canonical instead of prefixing targets onto already complete URLs. - Made DeHashed pagination honor the CLI limit, retain only normalized email and IP evidence, and discard raw breach rows; aligned LeakIX with its authenticated subdomain endpoint and documented rate-limit retry. - Added offline contracts for explicitly selected DNS and direct sources, retained normalized Pentest-Tools host and IP results, and hardened Shodan InternetDB, SubdomainFinder C99, and Windvane evidence boundaries. - Retained relevant GitLab project, profile, and website URLs in consolidated JSONL and SQLite results while excluding unrelated user URLs. -- Removed BuiltWith's duplicate interesting-URL getter by allowing the shared collector to use either established getter spelling. +- Standardized BuiltWith and every other URL-producing adapter on `get_urls()`. - Made no-filename REST `/query` executions reach completed-result construction and SQLite persistence without changing the legacy response fields. - Made Chaos reject empty credentials, report HTTP and malformed-response failures, and preserve supported subdomain response shapes. - Made Fofa reject incomplete credentials, report HTTP and malformed-response failures, normalize scoped hosts, and discard invalid IP values. diff --git a/CONTEXT.md b/CONTEXT.md index 96586490..5d8f3d9e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -1,6 +1,6 @@ # theHarvester discovery context -This glossary is the source of truth for discussing subdomain discovery across code, issues, pull requests, output, and documentation. It separates current addressability from historical, indirect, or unresolved evidence. +This glossary is the source of truth for discussing subdomain discovery and HarvestView lifecycle behavior across code, issues, pull requests, output, and documentation. It separates current addressability from historical, indirect, or unresolved evidence. The definitions state intended semantics; they do not imply that every discovery adapter already produces every evidence class. Update this glossary when a change alters a term's meaning or boundary. @@ -50,13 +50,53 @@ _Avoid_: Result, duplicate, hit A deduplicated operator-facing entity backed by one or more discovery observations and their retained provenance. _Avoid_: Raw finding, source result +**Hostname result**: +One normalized DNS-name merged result. It can be the authorized target itself or a subordinate name and does not by itself imply current DNS addressability. +_Avoid_: Subdomain result, live host, resolved host + +**IP result**: +One canonical IPv4 or IPv6 address merged result. +_Avoid_: IP-address result, resolved host + +**URL result**: +One normalized URL merged result. Source and action origins identify how it was found; provider-specific URL categories are not separate result kinds. +_Avoid_: Interesting URL, LinkedIn link, API endpoint result + **DNS validation observation**: One resolver vantage's time-bound DNS evidence about one in-scope candidate. It supports classifying the candidate as currently addressable or wildcard-indistinguishable without replacing its discovery observations. _Avoid_: DNS result, resolved host, validation status **Enumeration run**: One finite execution of theHarvester against an explicit target and selected options, identified independently from every other execution. -_Avoid_: Scan, monitoring cycle, session +_Avoid_: Scan, monitoring cycle, session, job + +**Action-only run**: +An enumeration run with no discovery sources that performs an explicitly selected DNS or direct action against an explicitly authorized target. It creates its own run record and never mutates the evidence of a parent run. +_Avoid_: Result action, parent-run update, inline scan + +**Run record**: +The durable operator-facing record that begins when an enumeration is submitted or evidence is imported and retains lifecycle, authorization, and available evidence under one stable identifier. +_Avoid_: Task, worker job, scan record + +**HarvestView**: +The browser-based analysis workspace for creating and inspecting run records, normalized evidence, source outcomes, and managed artifacts from theHarvester. +_Avoid_: Internal workflow names, operator app, console, dashboard + +**Imported run**: +A run record created from an existing theHarvester result file. Import records evidence but never executes discovery or contacts a target. +_Avoid_: Uploaded scan, replayed run + +**Lifecycle status**: +The durable state of a run record: queued, running, cancelling, cancelled, completed, or failed. It describes control flow, not evidence quality. +_Avoid_: Run result, provider status + +**Terminal evidence status**: +The completeness classification reported by a finished enumeration result: complete, partial, or failed. It does not describe queue or cancellation state. +_Avoid_: Lifecycle status, completion state + +**Cancellation request**: +The operator's durable request that the run worker prevent queued work from starting or ask the running child process to stop. A request is not itself proof that execution has ended. +_Avoid_: Cancelled run, process killed **Source execution**: One attempt to run one canonical discovery source within an enumeration run, with an explicit completion status and summary counts. diff --git a/README.md b/README.md index b2faac60..de352077 100644 --- a/README.md +++ b/README.md @@ -79,11 +79,11 @@ uv run theHarvester -h Options such as DNS brute force (`-c`), bounded recursive DNS (`--dns-recursive-depth`), reverse DNS lookup (`-n`), takeover checks (`-t`), API endpoint scanning (`-a`), DNS resolution (`-r`), and screenshots (`--screenshot`) generate additional network activity. Use them only within an explicitly authorized scope. -Recursive DNS requires exactly three distinct resolver IPs through `--dns-resolve`. It advances only names with two-vantage address consensus that are distinguishable from closest-encloser wildcard controls. Depth, DNS record query, and runtime limits are configurable through the three `--dns-recursive-*` options; the default query ceiling is 3,000 record queries across resolver vantages, and three consecutive zero-yield batches also stop recursion. PTR names for current addresses are retained as secondary evidence, but they do not establish current addressability or become recursion seeds. REST `/query` exposes the same options and requires the configured operator API key when recursion is enabled. +Recursive DNS requires exactly three distinct resolver IPs through `--dns-resolvers` or the compatible `--dns-resolve` value. It advances only names with two-vantage address consensus that are distinguishable from closest-encloser wildcard controls. Depth, DNS record query, and runtime limits are configurable through the three `--dns-recursive-*` options; the default query ceiling is 3,000 record queries across resolver vantages, and three consecutive zero-yield batches also stop recursion. PTR names for current addresses are retained as secondary evidence, but they do not establish current addressability or become recursion seeds. `POST /api/v1/runs` exposes the same controls. Screenshot capture also requires a Playwright-compatible browser; see the installation guide for setup. -## Browser interface and REST API +## REST API `restfulHarvest` starts a FastAPI service on `127.0.0.1:5000` by default: @@ -95,33 +95,27 @@ Open [http://127.0.0.1:5000/docs](http://127.0.0.1:5000/docs) for interactive Sw | Route | Purpose | | --- | --- | -| `GET /sources` | List registered discovery sources. | -| `GET /query` | Return consolidated discovery results, including emails and breach names, as JSON. | -| `GET /dnsbrute` | Run DNS brute force for a domain. | -| `POST /additional/breaches` | Return Have I Been Pwned breach data. | -| `POST /additional/leaks` | Return Leak-Lookup data. | -| `POST /additional/security-score` | Return SecurityScorecard data. | -| `POST /additional/tech-stack` | Return BuiltWith technology data. | -| `POST /additional/all` | Run all additional API lookups. | +| `GET /api/v1/sources` | List registered discovery sources and capabilities. | +| `POST /api/v1/runs` | Submit a finite enumeration run. | +| `GET /api/v1/runs` | List durable run records. | +| `GET /api/v1/runs/{run_id}` | Retrieve lifecycle state, normalized results, and source outcomes. | +| `POST /api/v1/runs/{run_id}/cancel` | Cancel queued or running work. | +| `POST /api/v1/runs/import` | Import JSONL evidence without executing discovery. | +| `POST /api/v1/runs/import-database` | Import completed runs from a theHarvester SQLite database. | +| `GET /api/v1/runs/{run_id}/export` | Export normalized evidence as JSONL. | -The service rate limit defaults to five requests per minute and can be changed with `--rate-limit`. The `/additional/*` routes require `THEHARVESTER_API_KEY` on the server and the same value in the `X-API-Key` request header. +Every `/api/v1/*` route requires `THEHARVESTER_API_KEY` in the `X-API-Key` header. Provider credentials stay in server-side configuration and cannot be supplied in a request. Keep the service bound to localhost. If you require remote access, add network access controls and TLS. -The core `/query`, `/sources`, and `/dnsbrute` routes do not normally require authentication. When a `/query` selection includes `dehashed`, `hibpverified`, or `leaklookup` and that source's provider key is configured, the request requires `THEHARVESTER_API_KEY` in the `X-API-Key` header because these sources can access breach-account data. Keep the service bound to localhost. If you require remote access, add authentication, access controls, and TLS. - -Docker Compose publishes port `5000` on every host interface unless you narrow the port mapping: - -```bash -docker compose up --build -``` +When `--proxies` and `--take-over` are combined, supported discovery and takeover requests use the configured proxies. ## Discovery sources -The table shows which result types each source can add to consolidated CLI results. Legacy JSON and XML keep their existing schemas; breach names are retained in JSONL and SQLite. Some adapters parse fields that the reports do not store. +The table shows which result types each source can add to consolidated CLI results. XML keeps its existing schema. Legacy JSON now consolidates `interesting_urls`, `linkedin_links`, and `trello_urls` into one `urls` field. Breach names are retained in JSONL and SQLite. Some adapters parse fields that the reports do not store. -The report groups findings by result type. It does not record which source found each item. Empty optional fields may be omitted. +JSON and XML group findings by result type without source attribution. JSONL and SQLite retain source attribution when the collection adapter provides it. Empty optional fields may be omitted. BuiltWith's normalized frameworks, languages, servers, CMS products, and analytics products are retained in JSONL and completed-result SQLite rows. -A checkmark means the source can add that result type. The **Separate output** column lists REST endpoints and optional actions that return other data. +A checkmark means the source can add that result type. The **Additional action output** column lists optional actions that return other data. Read the **API key** column as follows: @@ -132,13 +126,13 @@ Read the **API key** column as follows:
View the source and result matrix -| Source | Subdomains | Emails | IPs | ASNs | URLs / links | People | Breaches | Separate REST/action output (not consolidated report) | API key | +| Source | Subdomains | Emails | IPs | ASNs | URLs | People | Breaches | Additional action output (not consolidated report) | API key | | --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | --- | :---: | | `arquivo` | ✓ | No | No | No | No | No | No | No | No | | `baidu` | ✓ | ✓ | No | No | No | No | No | No | No | | `bevigil` | ✓ | No | No | No | ✓ | No | No | No | ✓ | | `bufferoverun` | ✓ | No | ✓ | No | No | No | No | No | ✓ | -| `builtwith` | ✓ | No | No | No | ✓ | No | No | `POST /additional/tech-stack` response | ✓ | +| `builtwith` | ✓ | No | No | No | ✓ | No | No | No | ✓ | | `brave` | ✓ | ✓ | No | No | No | No | No | No | ✓ | | `censys` | ✓ | ✓ | No | No | No | No | No | No | ✓ | | `certspotter` | ✓ | No | No | No | No | No | No | No | No | @@ -156,14 +150,14 @@ Read the **API key** column as follows: | `github-code` | ✓ | ✓ | No | No | No | No | No | No | ✓ | | `gitlab` | ✓ | ✓ | No | No | ✓ | No | No | No | No | | `hackertarget` | ✓ | No | No | No | No | No | No | No | Optional | -| `haveibeenpwned` | No | No | No | No | No | No | ✓ | `POST /additional/breaches` response | No | +| `haveibeenpwned` | No | No | No | No | No | No | ✓ | No | No | | `hibpverified` | No | ✓ | No | No | No | No | ✓ | No | ✓ | | `hudsonrock` | ✓ | ✓ | ✓ | No | No | No | No | No | No | | `hunter` | ✓ | ✓ | No | No | No | No | No | No | ✓ | | `hunterhow` | ✓ | No | No | No | No | No | No | No | ✓ | | `intelx` | ✓ | ✓ | No | No | ✓ | No | No | No | ✓ | | `leakix` | ✓ | No | No | No | No | No | No | No | ✓ | -| `leaklookup` | No | ✓ | No | No | No | No | ✓ | `POST /additional/leaks` response | ✓ | +| `leaklookup` | No | ✓ | No | No | No | No | ✓ | No | ✓ | | `mojeek` | ✓ | ✓ | No | No | No | No | No | No | Optional | | `netlas` | ✓ | No | No | No | No | No | No | No | ✓ | | `onyphe` | ✓ | No | ✓ | ✓ | No | No | No | No | ✓ | @@ -173,7 +167,7 @@ Read the **API key** column as follows: | `rapiddns` | ✓ | No | ✓ | No | No | No | No | No | No | | `robtex` | ✓ | No | ✓ | No | No | No | No | No | No | | `rocketreach` | No | ✓ | No | No | ✓ | No | No | No | ✓ | -| `securityscorecard` | ✓ | No | ✓ | No | No | No | No | `POST /additional/security-score` response | ✓ | +| `securityscorecard` | ✓ | No | ✓ | No | No | No | No | No | ✓ | | `securityTrails` | ✓ | No | ✓ | No | No | No | No | No | ✓ | | `sherlockeye` | ✓ | ✓ | ✓ | No | No | No | No | No | ✓ | | `shodan` | ✓ | No | No | No | No | No | No | `-s` / `--shodan` host-enrichment output | ✓ | @@ -195,9 +189,9 @@ Read the **API key** column as follows: Provider pricing is intentionally omitted because plans and quotas change frequently. See [Configuration and API Keys](docs/wiki/Configuration-and-API-Keys.md) and each provider's current documentation. -`haveibeenpwned` remains the keyless public breach catalogue. `hibpverified` is a separate authenticated source for HIBP's `breachedDomain` endpoint. It participates in `all` and matching capability selectors just like every other P0 source, and skips normally when its provider key is absent. REST selections that include it require the operator `X-API-Key` when the provider key is configured and return normalized emails plus stable breach names. A live run requires a user-owned paid HIBP API key and a user-owned domain verified in that account; routine tests use offline responses. +`haveibeenpwned` remains the keyless public breach catalogue. `hibpverified` is a separate authenticated source for HIBP's `breachedDomain` endpoint. It participates in `all` and matching capability selectors just like every other P0 source, and skips normally when its provider key is absent. API run requests can select it through the shared source contract and return normalized emails plus stable breach names. A live run requires a user-owned paid HIBP API key and a user-owned domain verified in that account; routine tests use offline responses. -The runtime registry also reports the legacy identifiers `linkedin`, `linkedin_links`, `netcraft`, `omnisint`, `sublist3r`, and `zoomeyeapi`. These identifiers have no active CLI handlers. The table does not present them as usable sources. +The runtime registry also reports the legacy identifiers `linkedin`, `netcraft`, `omnisint`, `sublist3r`, and `zoomeyeapi`. These identifiers have no active CLI handlers. The table does not present them as usable sources. ## Configuration @@ -214,14 +208,14 @@ Never commit populated configuration files, API keys, account details, or provid - `-f NAME` writes `NAME.json`, `NAME.xml`, and `NAME.jsonl`. - Screenshots are written to the directory passed to `--screenshot`. - Host, email, IP, and related scan records are stored in `~/.local/share/theHarvester/stash.sqlite`. -- Full-pipeline runs are also stored transactionally by run UUID with their completed, deduplicated findings. Early REST returns and DNS-brute utility requests are not recorded as completed runs. -- REST queries return JSON. +- Full CLI pipeline runs are also stored transactionally by run UUID with their completed, deduplicated findings. +- API executions use the same SQLite database as CLI results. Durable lifecycle rows stay separate from terminal evidence, while typed results and source or action origins remain queryable. JSONL handles individual run interchange, and the API can import completed runs from another theHarvester SQLite database. Treat collected OSINT as potentially sensitive. Keep report files, screenshots, and the local database out of source control and share them only within the authorized engagement. ### Report formats -The JSON report is a single object that preserves the legacy automation contract. Host entries remain plain hostnames or `hostname:address[,address...]` values when DNS resolution is enabled. DNS resolution and DNS brute force retain candidates only when A, AAAA, or CNAME evidence is available; CNAME-only candidates remain plain hostnames in existing CLI, REST, JSON, and XML output. +The JSON report is a single object. Host entries remain plain hostnames or `hostname:address[,address...]` values when DNS resolution is enabled. DNS resolution and DNS brute force retain candidates only when A, AAAA, or CNAME evidence is available; CNAME-only candidates remain plain hostnames in existing CLI, REST, JSON, and XML output. `Checker.check()` and `DnsForce.run()` retain their existing `(resolved, hosts, addresses)` return shape. Normalized A, AAAA, and CNAME values are available through each object's `records` mapping. @@ -231,7 +225,7 @@ The JSON report is a single object that preserves the legacy automation contract | `hosts` | Always | Discovered hosts; an empty array when none are found. | | `shodan` | Always | Shodan enrichment rows; an empty array when Shodan is not used. | | `ips`, `emails`, `vhosts`, `asns` | When non-empty | Network and contact findings. | -| `interesting_urls`, `trello_urls`, `linkedin_links` | When non-empty | Discovered links and URLs. | +| `urls` | When non-empty | Discovered URLs from every URL-producing source or action. | | `people`, `twitter_people`, `linkedin_people` | When non-empty | People and profile findings. | | `takeover_results` | When non-empty | Optional takeover-check results. | @@ -240,11 +234,11 @@ The XML report contains the command, emails, hosts, and virtual hosts. Use JSON The JSONL report is finalized after the selected one-shot actions finish. The first line identifies the run with its UUID, target, UTC timestamps, and result counts. Each later line is one sorted, deduplicated finding. When you concatenate report files, treat each summary line as the start of a new run. ```jsonl -{"completed_at":"2026-08-07T12:01:00Z","counts":{"hostname":1},"result_count":1,"run_id":"123e4567-e89b-12d3-a456-426614174000","started_at":"2026-08-07T12:00:00Z","target":"example.com","type":"summary"} +{"action_executions":[],"artifacts":[],"completed_at":"2026-08-07T12:01:00Z","counts":{"hostname":1},"evidence_status":"complete","result_count":1,"run_id":"123e4567-e89b-12d3-a456-426614174000","source_executions":[],"started_at":"2026-08-07T12:00:00Z","target":"example.com","type":"summary"} {"sources":[],"type":"hostname","value":"api.example.com"} ``` -JSONL is easy to stream for simple findings, but it is not uniformly self-describing. Finding lines inherit their run ID and target from the preceding summary. Structured result types, including recursive DNS records plus `person`, `infostealer`, `shodan`, and `takeover`, store a JSON object inside the string `value`. Parse those values a second time with `fromjson`. JSONL does not include source execution records. Finding records include source attribution when it is available. +JSONL is easy to stream one record at a time. The summary preserves the evidence status, source and action outcomes, and screenshot artifact metadata. Finding lines carry `sources` and, when applicable, `actions`; they inherit their run ID and target from the preceding summary. Hostnames, IP addresses, and URLs use the same `hostname`, `ip`, and `url` result kinds in JSONL, SQLite, the API, and HarvestView. Provenance identifies which source or action produced each finding. Structured result types, including recursive DNS records plus `person`, `infostealer`, `shodan`, and `takeover`, store a JSON object inside the string `value`. Parse those values a second time with `fromjson`. Parse recursive DNS findings as JSON objects: diff --git a/docs/adr/0003-run-worker-lifecycle.md b/docs/adr/0003-run-worker-lifecycle.md new file mode 100644 index 00000000..5998d1e0 --- /dev/null +++ b/docs/adr/0003-run-worker-lifecycle.md @@ -0,0 +1,19 @@ +# Keep run records separate with one isolated worker + +Status: proposed + +## Decision + +The HTTP application owns a durable run record separate from theHarvester's optional terminal `RunResult` evidence. A submission receives its stable ID while queued. One local worker claims one queued run at a time and executes the finite theHarvester core in an isolated child process. + +Lifecycle transitions are `queued -> running -> completed|failed`, `queued -> cancelled`, and `running -> cancelling -> cancelled`. A fixed whole-run deadline applies to the child. Running cancellation first requests cooperative termination, waits a short grace period, and then forces termination if needed. Queued cancellation is an atomic transition that prevents the worker claim. + +Evidence already persisted remains attached after failure or cancellation. Terminal evidence status (`complete`, `partial`, or `failed`) is reported independently from orchestration lifecycle status. On service restart, queued runs may resume; orphaned running or cancelling records become failed because their process ownership cannot be proven. + +## Why + +The existing core is a finite one-shot enumerator and its result object is created only when execution begins. Reusing it as queue state would conflate operator intent, process ownership, cancellation acknowledgement, and evidence quality. A single worker matches the local single-operator product, avoids concurrent output and credential contention, and can be widened later only if measured demand justifies it. + +## Consequences + +Run records need one small SQLite table and a lifecycle API. Child-process boundaries make deadline and forced cancellation reliable across blocking provider code. Work is serialized by design. Imported evidence enters as an already completed run record and never enters the queue. diff --git a/docs/wiki/Configuration-and-API-Keys.md b/docs/wiki/Configuration-and-API-Keys.md index 3e45d314..4558e38a 100644 --- a/docs/wiki/Configuration-and-API-Keys.md +++ b/docs/wiki/Configuration-and-API-Keys.md @@ -42,7 +42,7 @@ The [README source matrix](https://github.com/laramies/theHarvester/blob/dev/REA Provider pricing, quotas, and terms change frequently. Check the provider's current documentation for these details. -`hibpverified` queries [HIBP's authenticated verified-domain endpoint](https://haveibeenpwned.com/API/v3#BreachedDomain) only when explicitly named, either alone or in a combination such as `breaches,hibpverified`. Capability selectors and `all` exclude it. Live use requires a user-owned paid HIBP API key and a user-owned domain verified in that account. REST queries selecting it also require the operator `X-API-Key`; the keyless `haveibeenpwned` source continues to query only the public breach catalogue. +`hibpverified` queries [HIBP's authenticated verified-domain endpoint](https://haveibeenpwned.com/API/v3#BreachedDomain). It is selected by its name, the `breaches` capability, and `all`. Without a configured HIBP API key it is skipped like other unavailable keyed sources. Live use requires a user-owned paid HIBP API key and a user-owned domain verified in that account. The keyless `haveibeenpwned` source continues to query only the public breach catalogue. ## Proxies @@ -63,13 +63,13 @@ uv run theHarvester -d example.com -b crtsh -p A proxy does not make an assessment anonymous and does not change the authorization boundary. -## REST API protection +## API protection -The `/additional/*` routes require a server-side key: +Every `/api/v1/*` route requires a server-side key: ```bash export THEHARVESTER_API_KEY='replace-with-a-long-random-value' uv run restfulHarvest ``` -Clients send the same value in the `X-API-Key` header. This key protects only `/additional/*`; the core query routes remain unauthenticated. +API clients send the same value in the `X-API-Key` header. Provider credentials remain in `api-keys.yaml` and cannot be supplied through an API request. diff --git a/docs/wiki/How-to-add-a-new-module.md b/docs/wiki/How-to-add-a-new-module.md index 3f70b465..b555ec4e 100644 --- a/docs/wiki/How-to-add-a-new-module.md +++ b/docs/wiki/How-to-add-a-new-module.md @@ -21,7 +21,7 @@ An adapter normally provides: - an initializer for the target and local result sets; - an asynchronous `process()` method; -- only the getters it actually supports, such as `get_hostnames()`, `get_emails()`, `get_ips()`, `get_asns()`, `get_interesting_urls()`, or `get_results()`. +- only the getters it actually supports, such as `get_hostnames()`, `get_emails()`, `get_ips()`, `get_asns()`, `get_urls()`, or `get_results()`. Do not return fields the provider did not supply. Normalize and deduplicate before returning results. diff --git a/docs/wiki/Operator-Workflows.md b/docs/wiki/Operator-Workflows.md index c2d69954..c74b5cbc 100644 --- a/docs/wiki/Operator-Workflows.md +++ b/docs/wiki/Operator-Workflows.md @@ -25,14 +25,14 @@ AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -r ``` -To control the resolvers used: +To control the resolvers used, create a resolver file with one IP address per line and pass its path: ```bash AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh -r resolvers.txt ``` -Resolver files contain one IP address per line. DNS requests disclose candidate names to the selected resolver. +DNS requests disclose candidate names to each selected resolver. ## Shodan enrichment diff --git a/docs/wiki/Quick-Start.md b/docs/wiki/Quick-Start.md index 9344a8a0..880d7cee 100644 --- a/docs/wiki/Quick-Start.md +++ b/docs/wiki/Quick-Start.md @@ -35,7 +35,7 @@ AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -r ``` -Pass a resolver IP, comma-separated resolver IPs, or a file containing one resolver IP per line: +Pass a resolver IP, comma-separated resolver IPs, or a resolver file you create with one IP per line: ```bash AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' diff --git a/docs/wiki/Responsible-Use-and-Scope.md b/docs/wiki/Responsible-Use-and-Scope.md index c361b296..741ee155 100644 --- a/docs/wiki/Responsible-Use-and-Scope.md +++ b/docs/wiki/Responsible-Use-and-Scope.md @@ -22,6 +22,8 @@ The following options require additional care: | `--screenshot DIR` | Opens discovered web services in a browser. | | `-a`, `--api-scan` | Requests common API paths from the target. | +Use `--dns-resolvers IPS_OR_FILE` to select resolver addresses for DNS brute force, reverse lookup, or recursive DNS without also enabling hostname resolution. The compatible `--dns-resolve` value still selects resolvers and enables hostname resolution. + Use an owned or explicitly authorized domain for active examples. Do not substitute universities, public companies, bounty targets, or reserved example domains for recurring active scans. ## Protect collected data @@ -33,8 +35,8 @@ Results may contain private infrastructure, employee addresses, account identifi - Redact credentials, private target data, account information, and unnecessary response content before filing an issue. - Never publish raw provider responses merely to demonstrate a parsing or availability problem. -## Service exposure +## API exposure -The `restfulHarvest` core query routes do not require authentication. `THEHARVESTER_API_KEY` protects the optional `/additional/*` routes only. It does not protect `/query`, `/sources`, or `/dnsbrute`. +Every `/api/v1/*` route requires `THEHARVESTER_API_KEY`. Provider credentials remain server-side. -Keep the service on localhost. If you require remote access, add authentication, network controls, and TLS. +Keep the service on localhost. If you require remote access, add network controls and TLS in front of the existing API authentication. diff --git a/docs/wiki/Rest-API.md b/docs/wiki/Rest-API.md index 175eb61f..3b3163a1 100644 --- a/docs/wiki/Rest-API.md +++ b/docs/wiki/Rest-API.md @@ -1,25 +1,17 @@ # REST API -`restfulHarvest` runs a FastAPI service for local automation and interactive Swagger/ReDoc documentation. +`restfulHarvest` serves one versioned API for local automation. ## Start the service +Set a long random API key before startup: + ```bash +export THEHARVESTER_API_KEY='replace-with-a-long-random-value' uv run restfulHarvest ``` -Defaults: - -- host: `127.0.0.1` -- port: `5000` -- log level: `info` -- rate limit: `5/minute` per client address - -Use `uv run restfulHarvest -h` for current launcher options. For example: - -```bash -uv run restfulHarvest --rate-limit 10/minute -``` +The service binds to `127.0.0.1:5000` by default. Use `uv run restfulHarvest -h` for launcher options. Open: @@ -28,104 +20,132 @@ Open: Treat the runtime OpenAPI document as the exact request and response reference. -## Core routes +## Routes | Route | Purpose | | --- | --- | -| `GET /sources` | List current discovery sources. | -| `GET /query` | Run selected discovery sources and return consolidated JSON. | -| `GET /dnsbrute` | Run active DNS brute force for an authorized domain. | -| `GET /runs` | List recent completed enumeration runs. | -| `GET /runs/{run_id}` | Retrieve one completed run and its normalized evidence. | +| `GET /api/v1/sources` | List discovery sources, capabilities, activity classes, and credential names. | +| `POST /api/v1/runs` | Submit one finite enumeration run. | +| `GET /api/v1/runs` | List run records with `limit` and `offset` pagination. | +| `GET /api/v1/runs/{run_id}` | Retrieve lifecycle state, options, results, source outcomes, and artifacts. | +| `POST /api/v1/runs/{run_id}/cancel` | Cancel queued work or request cancellation of running work. | +| `POST /api/v1/runs/import` | Import a JSONL result file without executing discovery. | +| `POST /api/v1/runs/import-database` | Import completed runs from a theHarvester SQLite database. | +| `GET /api/v1/runs/{run_id}/export` | Export normalized results as JSONL. | +| `GET /api/v1/runs/{run_id}/screenshots/{name}` | Retrieve one managed screenshot. | -List sources: +There are no provider-specific routes. Sources such as `builtwith`, `haveibeenpwned`, `hibpverified`, `leaklookup`, and `securityscorecard` use the same run request as every other source. + +The versioned API is asynchronous by design. A successful submission returns a durable run record instead of waiting for every provider and action to finish. + +## Authentication + +Every `/api/v1/*` route requires the configured key in `X-API-Key`: ```bash -curl -s http://127.0.0.1:5000/sources | jq -r '.sources[]' -``` - -Run a passive query: - -```bash -curl -sG http://127.0.0.1:5000/query \ - --data-urlencode 'domain=example.com' \ - --data-urlencode 'source=crtsh' \ - --data-urlencode 'source=certspotter' \ - | jq -``` - -The `source` parameter also accepts the same capability selectors as the CLI: -`subdomains`, `emails`, `ips`, `asns`, `urls`, `people`, and `breaches`. -Repeat `source` to combine capabilities with explicit source names. Selection is -a union and does not filter fields returned by a selected source. - -```bash -curl -sG http://127.0.0.1:5000/query \ - --data-urlencode 'domain=example.com' \ - --data-urlencode 'source=emails' \ - --data-urlencode 'source=certspotter' \ - | jq -``` - -A completed `/query` also retains its normalized terminal record in the local -SQLite database. No JSON, XML, or JSONL report file is written unless `filename` -is supplied. - -HIBP verified-domain participates in `all`, `emails`, and `breaches` selections. -When its provider key is configured, any selection that includes it also requires -the operator API key: - -```bash -curl -sG http://127.0.0.1:5000/query \ - -H "X-API-Key: $THEHARVESTER_API_KEY" \ - --data-urlencode "domain=$VERIFIED_DOMAIN" \ - --data-urlencode 'source=hibpverified' \ - | jq '{emails, breaches}' -``` - -Completed-run routes require the operator API key because retained evidence can -contain sensitive results: - -```bash -curl -s http://127.0.0.1:5000/runs \ +curl -s http://127.0.0.1:5000/api/v1/sources \ -H "X-API-Key: $THEHARVESTER_API_KEY" \ | jq ``` -## Additional API routes +Provider credentials remain in theHarvester's server-side configuration. Requests cannot supply provider API keys. -The following `POST /additional/*` routes provide optional breach, leak, security-score, and technology-stack lookups: +## Submit and inspect a run -- `/additional/breaches` -- `/additional/leaks` -- `/additional/security-score` -- `/additional/tech-stack` -- `/additional/all` - -Set a server key before startup: +Source names and capability selectors share the `sources` array. Multiple capabilities select the union of matching sources and do not filter fields returned by those sources. ```bash -export THEHARVESTER_API_KEY='replace-with-a-long-random-value' -uv run restfulHarvest +run_id="$(curl -s http://127.0.0.1:5000/api/v1/runs \ + -X POST \ + -H "X-API-Key: $THEHARVESTER_API_KEY" \ + -H 'Content-Type: application/json' \ + -d '{ + "target": "example.com", + "sources": ["emails", "crtsh"], + "limit": 500, + "deadline_seconds": 1800 + }' \ + | jq -r '.run_id')" + +curl -s "http://127.0.0.1:5000/api/v1/runs/$run_id" \ + -H "X-API-Key: $THEHARVESTER_API_KEY" \ + | jq '{status, evidence_status, results, source_executions, action_executions, artifacts}' ``` -Send that value in `X-API-Key`: +Run submission is asynchronous. Lifecycle status is `queued`, `running`, `cancelling`, `cancelled`, `completed`, or `failed`. Terminal evidence status is reported separately as `complete`, `partial`, or `failed` when evidence exists. + +P1 DNS and P2 direct options are fields on the same run request. The OpenAPI schema shows their current defaults, limits, and descriptions. The server uses the operator-selected target and does not impose a public-only egress policy. + +### Run an action against one result + +Screenshots and DNS brute force can run directly against an authorized hostname without repeating discovery. Submit an empty `sources` array and select one action: ```bash -curl -s http://127.0.0.1:5000/additional/tech-stack \ +curl -s http://127.0.0.1:5000/api/v1/runs \ + -X POST \ + -H "X-API-Key: $THEHARVESTER_API_KEY" \ + -H 'Content-Type: application/json' \ + -d '{ + "target": "subdomain.example.com", + "sources": [], + "screenshot": true + }' \ + | jq +``` + +For DNS brute force, set `dns_brute` to `true`. You may also provide `dns_resolvers` as one or more distinct IPv4 or IPv6 addresses. Recursive DNS is the only action that requires exactly three resolver addresses. + +The action catalog and run request use the same names. For example, set `takeover` to `true` for takeover checks. API endpoint scans can use the bundled paths or an explicit bounded list: + +```json +{ + "target": "api.example.com", + "sources": [], + "api_scan": true, + "api_scan_paths": ["/api/v2", "/health"] +} +``` + +Every custom API scan entry must be a URL path beginning with `/`. The API does not accept a server-side file path. + +## Import and export + +Import records existing evidence and never contacts the target. For one run, send the same JSONL written by `theHarvester -f NAME`: + +```bash +curl -s "http://127.0.0.1:5000/api/v1/runs/import?filename=report.jsonl" \ -X POST \ -H "X-API-Key: $THEHARVESTER_API_KEY" \ - -H 'Content-Type: application/json' \ - -d '{"domain":"example.com"}' \ + -H 'Content-Type: application/x-ndjson' \ + --data-binary @report.jsonl \ | jq ``` -These routes may also require provider credentials in the request body or local configuration. Consult `/docs` for the current schema. +JSONL is a terminal report, so an import is recorded as completed. The summary retains evidence status, source and action outcomes, and screenshot artifact metadata. Each finding's `sources` and `actions` arrays rebuild result attribution and must name an execution in the summary. Result kinds such as `hostname`, `ip`, and `url` use the same names in JSONL, SQLite, and the API. + +To load every completed run from another theHarvester database: + +```bash +curl -s "http://127.0.0.1:5000/api/v1/runs/import-database?filename=stash.sqlite" \ + -X POST \ + -H "X-API-Key: $THEHARVESTER_API_KEY" \ + -H 'Content-Type: application/vnd.sqlite3' \ + --data-binary @stash.sqlite \ + | jq +``` + +The server checks the SQLite header, integrity, schema, and each completed run before copying it. Original run IDs are preserved. Exact duplicates are skipped, while a reused ID with different evidence is rejected. Close the source process or checkpoint its WAL before uploading the database. Screenshot metadata is imported, but screenshot files must be copied separately. The default upload ceiling is 1 GiB and can be changed with `THEHARVESTER_MAX_DATABASE_IMPORT_BYTES`. + +Export one normalized result set in the same streamable format: + +```bash +curl -s "http://127.0.0.1:5000/api/v1/runs/$run_id/export" \ + -H "X-API-Key: $THEHARVESTER_API_KEY" \ + -o results.jsonl +``` + +The first line is the `summary` record, including evidence status, source and action outcomes, and artifacts. Each remaining line is one normalized finding with `type`, `value`, `sources`, and optional `actions`. This keeps the file easy to stream with `jq -c` and makes API exports importable again without a format conversion. Lifecycle details and the submitted request remain available from `GET /api/v1/runs/{run_id}`. ## Security boundary -`THEHARVESTER_API_KEY` protects `/additional/*`, `/runs*`, and `/query` selections that include a configured `hibpverified` source. Other `/query` requests, `/sources`, and `/dnsbrute` remain unauthenticated. - -Keep the default localhost binding. If you require remote access, add authentication, network allowlists, TLS, request logging, and an appropriate rate limit. - -The supplied Docker Compose configuration binds host port `5000` on every interface unless you narrow the mapping. +Keep the default localhost binding. If remote access is required, add TLS, network access controls, request logging, and an appropriate rate limit. The supplied Docker Compose configuration publishes only to `127.0.0.1` by default. diff --git a/docs/wiki/Results-and-Local-Data.md b/docs/wiki/Results-and-Local-Data.md index c994b0dc..256b4bb8 100644 --- a/docs/wiki/Results-and-Local-Data.md +++ b/docs/wiki/Results-and-Local-Data.md @@ -1,6 +1,6 @@ # Results and local data -theHarvester can print findings, write reports, retain selected records in SQLite, save screenshots, and return REST JSON. These outputs have different schemas and sensitivity. +theHarvester can print findings, write reports, retain selected records in SQLite, save screenshots, and expose durable run records through the API. These outputs have different schemas and sensitivity. ## Terminal output @@ -34,17 +34,31 @@ Host, email, IP, and related records are stored at: The database persists across runs. Account for it in engagement cleanup and retention procedures. -Completed CLI and REST `/query` executions also store one normalized terminal -record keyed by run UUID. REST keeps its existing response shape and does not -write report files unless a filename is requested. +Completed CLI executions store one normalized terminal record keyed by run UUID. API executions use the same database by default and may override its path with `THEHARVESTER_RUN_DB`. Lifecycle rows keep queue, cancellation, and worker state separate from terminal evidence. Imported JSONL is stored without executing discovery, and source attribution is rebuilt from each finding's `sources` array. A SQLite import copies every completed run after validating the database and keeps the original run IDs. + +The normalized persistence model can represent active-action provenance and artifact metadata through five core tables: + +- `runs`: one finite enumeration run; +- `executions`: each passive source or active action represented by the model; +- `results`: deduplicated hostnames, IPs, emails, URLs, and structured outputs; +- `result_origins`: which execution produced each result; and +- `artifacts`: files such as screenshots, linked to their creating action and subject result. + +Current runtime collection populates passive source executions plus DNS, takeover, Shodan, and API endpoint scan executions and origins. Screenshot actions attach file metadata to their captured hostname or URL without creating fake screenshot findings. + +Every discovered URL is stored as the `url` result kind. Its source or action origins identify whether it came from BuiltWith, GitLab, RocketReach, API scanning, or another producer; provider-specific URL kinds are not stored. + +Hostname and IP evidence use the `hostname` and `ip` result kinds in SQLite, JSONL, the API, and HarvestView. A hostname may be the authorized target itself or a subordinate name, so the result kind does not claim that every value is a subdomain. + +Two operational tables support the API without changing those five evidence concepts: `run_records` stores queue and lifecycle state, and `run_worker_leases` prevents two local workers from claiming the same queue. Older runless rows remain in `legacy_observations`. SQLite upgrades supported schemas automatically during normal initialization. ## Screenshots `--screenshot DIR` writes browser captures to the selected directory. Screenshots may contain authentication pages, internal names, or other sensitive visual data even when no credentials were used. -## REST JSON +## API results -The REST `/query` response returns arrays for ASNs, interesting URLs, Twitter/LinkedIn data, Trello URLs, IPs, emails, and hosts. The corresponding normalized terminal record is retained in SQLite. Treat runtime `/docs`, `/redoc`, and OpenAPI as the exact request/response reference. +`GET /api/v1/runs/{run_id}` returns lifecycle state plus a normalized `results` array. Each result has `type`, `value`, `sources`, and `actions`. Run-level source and action outcomes remain available in `source_executions` and `action_executions`, while file metadata is returned through `artifacts`. JSONL imports or exports one run, and SQLite import loads completed runs in bulk. Treat runtime `/docs`, `/redoc`, and OpenAPI as the exact request and response reference. ## Handling and sharing diff --git a/docs/wiki/Troubleshooting.md b/docs/wiki/Troubleshooting.md index 7e3e967b..c90975d8 100644 --- a/docs/wiki/Troubleshooting.md +++ b/docs/wiki/Troubleshooting.md @@ -53,7 +53,7 @@ Do not post credentials, private targets, account details, or raw provider respo ## DNS resolution -`-r` accepts no value, a resolver IP, comma-separated resolver IPs, or a file with one IP per line: +`-r` accepts no value, a resolver IP, comma-separated resolver IPs, or a resolver file you create with one IP per line: ```bash AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' @@ -82,10 +82,10 @@ uv run restfulHarvest --log-level debug Then open [http://127.0.0.1:5000/docs](http://127.0.0.1:5000/docs). -- `401` on `/additional/*`: the `X-API-Key` header is absent or does not match. -- `503` on `/additional/*`: `THEHARVESTER_API_KEY` was not configured before startup. -- `429`: the client exceeded the configured API rate limit. -- Core routes are intentionally not protected by that key; do not expose the service directly. +- `401` on `/api/v1/*`: the `X-API-Key` header does not match. +- `503` on `/api/v1/*`: `THEHARVESTER_API_KEY` was not configured before startup. +- `429`: a reverse proxy or remote provider applied its own rate limit. `restfulHarvest` has no built-in request limiter. +- `503` when creating a run: the execution worker is disabled or unavailable. ## Docker @@ -94,7 +94,7 @@ docker compose ps docker compose logs theharvester.svc.local ``` -The container runs the REST API on container port `80`, published as host port `5000` by the supplied Compose file. +The container runs the REST API on container port `8000`, published as host port `5000` by the supplied Compose file. ## File an actionable issue diff --git a/pyproject.toml b/pyproject.toml index 3b5eb9ed..04dcf99e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -38,7 +38,6 @@ dependencies = [ "httpx==0.28.1", "retrying==1.4.2", "shodan==1.31.0", - "slowapi==0.1.10", "sqlalchemy==2.0.51", "ujson==5.13.0", "uvicorn==0.49.0", diff --git a/tests/discovery/test_api_endpoints.py b/tests/discovery/test_api_endpoints.py index 7608478a..4915ff37 100644 --- a/tests/discovery/test_api_endpoints.py +++ b/tests/discovery/test_api_endpoints.py @@ -4,6 +4,7 @@ import aiohttp import pytest from theHarvester.discovery import api_endpoints +from theHarvester.lib.core import FetcherResponse class FakeResponse: @@ -62,6 +63,7 @@ async def test_api_endpoint_scan_uses_only_observational_http_methods(monkeypatc methods = [] monkeypatch.setattr(search, '_load_wordlist', lambda: []) + async def detect_schema(): return 'https' @@ -77,6 +79,107 @@ async def test_api_endpoint_scan_uses_only_observational_http_methods(monkeypatc assert methods == ['GET', 'HEAD', 'OPTIONS'] +@pytest.mark.asyncio +async def test_api_endpoint_scan_uses_only_the_configured_wordlist(monkeypatch, tmp_path) -> None: + wordlist = tmp_path / 'operator-paths.txt' + wordlist.write_text('/health\n', encoding='utf-8') + search = api_endpoints.SearchApiEndpoints('example.com', wordlist=str(wordlist), exact_paths=True) + detected_paths: list[str] = [] + requested_urls: list[str] = [] + + async def detect_schema(path: str = '') -> str: + detected_paths.append(path) + return 'https' + + async def fetch(url: str, *_args, **_kwargs): + requested_urls.append(url) + return '' + + monkeypatch.setattr(search, '_detect_schema', detect_schema) + monkeypatch.setattr(api_endpoints.AsyncFetcher, 'fetch', fetch) + + await search.do_search() + + assert detected_paths == ['/health'] + assert requested_urls == ['https://example.com/health'] * 3 + + +@pytest.mark.asyncio +async def test_schema_detection_can_probe_an_exact_listed_path() -> None: + search = api_endpoints.SearchApiEndpoints('example.com', exact_paths=True) + session = FakeSession() + search._session = session + + assert await search._detect_schema('/health') == 'https' + assert session.requests[0][0] == 'https://example.com/health' + + +@pytest.mark.asyncio +async def test_api_endpoint_scan_exposes_shared_fetcher_metadata(monkeypatch: pytest.MonkeyPatch) -> None: + search = api_endpoints.SearchApiEndpoints('example.com') + search.common_api_paths = ['/api'] + monkeypatch.setattr(search, '_load_wordlist', lambda: []) + + async def detect_schema() -> str: + return 'https' + + async def fetch(*_args, **kwargs): + assert kwargs['include_metadata'] is True + return FetcherResponse( + body='{"status":"ok"}', + status=200, + headers={'content-type': 'application/json'}, + ) + + monkeypatch.setattr(search, '_detect_schema', detect_schema) + monkeypatch.setattr(api_endpoints.AsyncFetcher, 'fetch', fetch) + + await search.do_search() + + result = search.get_found_endpoints()['https://example.com/api'] + assert result.status_code == 200 + assert result.method == 'GET' + assert result.content_type == 'application/json' + assert result.content_length == len('{"status":"ok"}') + assert result.content_preview == '{"status":"ok"}' + + +@pytest.mark.asyncio +async def test_api_endpoint_scan_counts_suppressed_request_failures(monkeypatch: pytest.MonkeyPatch) -> None: + search = api_endpoints.SearchApiEndpoints('example.com') + search.common_api_paths = ['/api'] + monkeypatch.setattr(search, '_load_wordlist', lambda: []) + + async def detect_schema() -> str: + return 'https' + + async def fail_request(*_args, **kwargs): + assert kwargs['include_metadata'] is True + return None + + monkeypatch.setattr(search, '_detect_schema', detect_schema) + monkeypatch.setattr(api_endpoints.AsyncFetcher, 'fetch', fail_request) + + await search.do_search() + + assert search.request_error_count == 3 + assert search.request_error_types == {'TransportError'} + assert search.get_found_endpoints() == {} + + +@pytest.mark.asyncio +async def test_api_endpoint_scan_reports_suppressed_top_level_failure(monkeypatch: pytest.MonkeyPatch) -> None: + search = api_endpoints.SearchApiEndpoints('example.com') + + async def fail_schema_detection() -> str: + raise RuntimeError('scan setup failed') + + monkeypatch.setattr(search, '_detect_schema', fail_schema_detection) + + assert await search.do_search() is None + assert search.scan_error_type == 'RuntimeError' + + @pytest.mark.asyncio async def test_api_endpoint_scan_allows_an_operator_selected_private_target(monkeypatch) -> None: search = api_endpoints.SearchApiEndpoints('100.64.0.1') diff --git a/tests/discovery/test_builtwith.py b/tests/discovery/test_builtwith.py index 2f1aef5f..5c1f11eb 100644 --- a/tests/discovery/test_builtwith.py +++ b/tests/discovery/test_builtwith.py @@ -13,12 +13,12 @@ if 'aiohttp_socks' not in sys.modules: def from_url(*_args, **_kwargs): return None - setattr(aiohttp_socks_stub, 'ProxyConnector', _ProxyConnector) + aiohttp_socks_stub.ProxyConnector = _ProxyConnector # type: ignore[attr-defined] sys.modules['aiohttp_socks'] = aiohttp_socks_stub +from theHarvester import __main__ as theharvester_main from theHarvester.discovery import builtwith from theHarvester.discovery.constants import MissingKey -from theHarvester import __main__ as theharvester_main from theHarvester.lib.completed_result import CompletedResult @@ -87,7 +87,7 @@ async def test_process_accepts_text_json_content_type(monkeypatch) -> None: await search.process() assert await search.get_hostnames() == {'sub.example.com'} - assert await search.get_interesting_urls() == {'https://example.com/login'} + assert await search.get_urls() == {'https://example.com/login'} assert await search.get_frameworks() == {'Django'} assert await search.get_languages() == {'Python'} assert await search.get_servers() == {'nginx'} @@ -157,7 +157,7 @@ async def test_normalized_builtwith_results_reach_completed_jsonl( async def get_hostnames(self) -> set[str]: return set() - async def get_interesting_urls(self) -> set[str]: + async def get_urls(self) -> set[str]: return {'https://example.com/login'} async def get_frameworks(self) -> set[str]: @@ -188,12 +188,12 @@ async def test_normalized_builtwith_results_reach_completed_jsonl( ('analytics', 'Google Analytics'), ('cms', 'WordPress'), ('framework', 'Django'), - ('interesting-url', 'https://example.com/login'), ('language', 'Python'), ('server', 'nginx'), + ('url', 'https://example.com/login'), ) records = [json.loads(line) for line in report.with_suffix('.jsonl').read_text().splitlines()] - assert {'type': 'interesting-url', 'value': 'https://example.com/login', 'sources': ['builtwith']} in records + assert {'type': 'url', 'value': 'https://example.com/login', 'sources': ['builtwith']} in records assert {'type': 'framework', 'value': 'Django', 'sources': ['builtwith']} in records assert {'type': 'language', 'value': 'Python', 'sources': ['builtwith']} in records assert {'type': 'server', 'value': 'nginx', 'sources': ['builtwith']} in records diff --git a/tests/discovery/test_haveibeenpwned.py b/tests/discovery/test_haveibeenpwned.py index e07eb5a1..c78c7af8 100644 --- a/tests/discovery/test_haveibeenpwned.py +++ b/tests/discovery/test_haveibeenpwned.py @@ -8,7 +8,6 @@ import pytest from theHarvester.discovery import haveibeenpwned from theHarvester import __main__ as theharvester_main -from theHarvester.lib.api import additional_endpoints from theHarvester.lib.completed_result import CompletedResult from theHarvester.lib.core import FetcherResponse @@ -113,21 +112,6 @@ async def test_public_breach_catalog_attributes_http_failures( assert 'HaveIBeenPwned request failed with HTTP 429' in caplog.text -@pytest.mark.asyncio -async def test_breach_rest_handler_does_not_initialize_unrelated_providers(monkeypatch: pytest.MonkeyPatch) -> None: - async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]: - return [FetcherResponse(body=[{'Domain': 'example.com'}], status=200, headers={})] - - monkeypatch.setattr(haveibeenpwned.AsyncFetcher, 'fetch_all', fake_fetch_all) - - result = await additional_endpoints.get_breaches( - additional_endpoints.DomainRequest(domain='example.com'), - _api_key='local-api-key', - ) - - assert result == {'status': 'success', 'data': [{'Domain': 'example.com'}]} - - @pytest.mark.asyncio async def test_public_breach_names_reach_completed_result_and_jsonl( monkeypatch: pytest.MonkeyPatch, diff --git a/tests/discovery/test_intelxsearch.py b/tests/discovery/test_intelxsearch.py index cfed0de8..902620fe 100644 --- a/tests/discovery/test_intelxsearch.py +++ b/tests/discovery/test_intelxsearch.py @@ -59,6 +59,8 @@ async def test_process_exposes_flat_normalized_in_scope_results(monkeypatch: pyt {'selectorvalue': 'https://portal.example.com/path'}, {'selectorvalue': 'api.example.com.'}, {'selectorvalue': 'foo.example.com.evil'}, + {'selectorvalue': 'https://foo.example.com.evil/path'}, + {'selectorvalue': 'ftp://api.example.com/archive'}, {'selectorvalue': 'http://['}, {'selectorvalue': None}, None, @@ -78,11 +80,7 @@ async def test_process_exposes_flat_normalized_in_scope_results(monkeypatch: pyt assert await search.get_emails() == ['admin@example.com'] assert await search.get_hostnames() == ['api.example.com', 'portal.example.com'] - assert await search.get_interestingurls() == [ - 'api.example.com.', - 'foo.example.com.evil', - 'https://portal.example.com/path', - ] + assert await search.get_urls() == ['https://portal.example.com/path'] @pytest.mark.asyncio @@ -112,7 +110,7 @@ async def test_process_treats_denied_and_malformed_responses_as_empty( assert await search.get_emails() == [] assert await search.get_hostnames() == [] - assert await search.get_interestingurls() == [] + assert await search.get_urls() == [] @pytest.mark.asyncio @@ -139,7 +137,7 @@ async def test_orchestrator_stores_intelx_subdomains_without_dns(monkeypatch: py async def get_emails(self) -> list[str]: return [] - async def get_interestingurls(self) -> list[str]: + async def get_urls(self) -> list[str]: return [] class _UnexpectedChecker: diff --git a/tests/discovery/test_leaklookup.py b/tests/discovery/test_leaklookup.py index f314c7a4..ca352265 100644 --- a/tests/discovery/test_leaklookup.py +++ b/tests/discovery/test_leaklookup.py @@ -6,10 +6,9 @@ from typing import Any import pytest -from theHarvester.discovery import additional_apis, leaklookup +from theHarvester.discovery import leaklookup from theHarvester.discovery.constants import MissingKey from theHarvester.lib.completed_result import CompletedResult -from theHarvester.lib.api import additional_endpoints from theHarvester.lib.core import FetcherResponse from theHarvester import __main__ as theharvester_main @@ -150,77 +149,6 @@ async def test_leaklookup_empty_success_returns_no_results(monkeypatch) -> None: assert await search.get_breach_names() == set() -@pytest.mark.asyncio -async def test_additional_leaks_endpoint_uses_only_leaklookup(monkeypatch) -> None: - class FakeSearchLeakLookup: - def __init__(self, domain: str) -> None: - assert domain == 'example.test' - - async def process(self) -> None: - return None - - async def get_leaks(self) -> list[dict[str, str]]: - return [{'breach': 'Example Breach', 'email': 'alice@example.test'}] - - class UnexpectedAdditionalAPIs: - def __init__(self, *_args: object, **_kwargs: object) -> None: - raise AssertionError('unrelated providers must not be initialized') - - monkeypatch.setattr(additional_endpoints, 'SearchLeakLookup', FakeSearchLeakLookup, raising=False) - monkeypatch.setattr(additional_endpoints, 'AdditionalAPIs', UnexpectedAdditionalAPIs) - - result = await additional_endpoints.get_leaks( - additional_endpoints.DomainRequest(domain='example.test'), - _api_key='local-api-key', - ) - - assert result == { - 'status': 'success', - 'data': [{'breach': 'Example Breach', 'email': 'alice@example.test'}], - } - - -@pytest.mark.asyncio -async def test_missing_leaklookup_key_does_not_break_security_score_endpoint(monkeypatch) -> None: - class PassiveProvider: - def __init__(self, _domain: str) -> None: - return None - - class FakeSecurityScorecard(PassiveProvider): - score = 95 - grades = {'network': 'A'} - issues: list[object] = [] - recommendations: list[object] = [] - hosts: set[str] = set() - - async def process(self, _proxy: bool) -> None: - return None - - class UnexpectedLeakLookup: - def __init__(self, _domain: str) -> None: - raise AssertionError('Leak-Lookup must be constructed only when requested') - - monkeypatch.setattr(additional_apis, 'SearchHaveIBeenPwned', PassiveProvider) - monkeypatch.setattr(additional_apis, 'SearchBuiltWith', PassiveProvider) - monkeypatch.setattr(additional_apis, 'SearchSecurityScorecard', FakeSecurityScorecard) - monkeypatch.setattr(additional_apis, 'SearchLeakLookup', UnexpectedLeakLookup) - - result = await additional_endpoints.get_security_score( - additional_endpoints.DomainRequest(domain='example.test'), - _api_key='local-api-key', - ) - - assert result == { - 'status': 'success', - 'data': { - 'score': 95, - 'grades': {'network': 'A'}, - 'issues': [], - 'recommendations': [], - }, - } - - @pytest.mark.asyncio async def test_leaklookup_emails_and_breaches_reach_completed_result_and_jsonl(monkeypatch, tmp_path: Path) -> None: completed_results: list[CompletedResult] = [] diff --git a/tests/discovery/test_rapiddns.py b/tests/discovery/test_rapiddns.py index 104914f5..36615fed 100644 --- a/tests/discovery/test_rapiddns.py +++ b/tests/discovery/test_rapiddns.py @@ -154,15 +154,18 @@ async def test_rapiddns_evidence_reaches_existing_outputs( def __init__(self, *, word: str, wordlist: str) -> None: assert word == 'example.com' assert wordlist.endswith('api_endpoints.txt') + self.scan_error_type = None + self.request_error_count = 0 + self.request_error_types: set[str] = set() async def do_search(self) -> None: return None def get_found_endpoints(self) -> dict[str, object]: - return {'/health': object()} + return {'https://example.com/health': object()} def get_interesting_endpoints(self) -> dict[str, object]: - return {'/health': object()} + return {'https://example.com/health': object()} def get_auth_required(self) -> dict[str, object]: return {} @@ -198,10 +201,14 @@ async def test_rapiddns_evidence_reaches_existing_outputs( iprange: str, callback: Any, nameservers: list[str] | None = None, + error_types: set[str] | None = None, ) -> None: assert iprange in {'192.0.2.0/24', '198.51.100.0/24', '2001:d00::/24'} assert nameservers is None callback('reverse.example.com') + if iprange == '198.51.100.0/24': + assert error_types is not None + error_types.add('TimeoutError') report = tmp_path / 'rapiddns-report' monkeypatch.setattr(rapiddns.AsyncFetcher, 'fetch_all', fake_fetch_all) @@ -232,12 +239,15 @@ async def test_rapiddns_evidence_reaches_existing_outputs( await theharvester_main.start(completed_result_checkpoint=capture_checkpoint) assert exit_info.value.code == 0 - assert stored.count(('api-endpoint', ('/health',), 'api_scan')) == 1 + assert stored == [] report_json = json.loads(report.with_suffix('.json').read_text()) assert report_json['hosts'] == ['alias.example.com', 'api.example.com', 'broken.example.com'] assert report_json['ips'] == ['192.0.2.1', '198.51.100.9', '2001:db8::1'] + assert report_json['urls'] == ['https://example.com/health'] assert 'interesting_urls' not in report_json + assert 'linkedin_links' not in report_json + assert 'trello_urls' not in report_json jsonl_records = [json.loads(line) for line in report.with_suffix('.jsonl').read_text().splitlines()] assert jsonl_records[0]['type'] == 'summary' @@ -246,24 +256,43 @@ async def test_rapiddns_evidence_reaches_existing_outputs( assert [str(result.run_id) for result in completed_results] == [jsonl_records[0]['run_id']] assert checkpoints assert {result.run_id for result in checkpoints} == {completed_results[0].run_id} - assert {'type': 'interesting-url', 'value': 'https://example.com/health', 'sources': []} in jsonl_records - assert {'type': 'url', 'value': 'https://example.com/health', 'sources': []} in jsonl_records - assert {'type': 'hostname', 'value': 'reverse.example.com', 'sources': []} in jsonl_records - assert {'type': 'ip-address', 'value': '198.51.100.9', 'sources': ['securityscorecard']} in jsonl_records - assert {'type': 'ip-address', 'value': '2001:db8::1', 'sources': ['securityscorecard']} in jsonl_records + assert {'type': 'url', 'value': 'https://example.com/health', 'sources': [], 'actions': ['api-scan']} in jsonl_records + assert { + 'type': 'hostname', + 'value': 'reverse.example.com', + 'sources': [], + 'actions': ['dns-lookup'], + } in jsonl_records + assert {'type': 'ip', 'value': '198.51.100.9', 'sources': ['securityscorecard']} in jsonl_records + assert {'type': 'ip', 'value': '2001:db8::1', 'sources': ['securityscorecard']} in jsonl_records assert not any(record.get('value') == 'not-an-ip' for record in jsonl_records) assert {(observation.source, observation.kind, observation.value) for observation in completed_results[0].observations} >= { ('rapiddns', 'hostname', 'api.example.com'), - ('rapiddns', 'ip-address', '192.0.2.1'), - ('securityscorecard', 'ip-address', '198.51.100.9'), - ('securityscorecard', 'ip-address', '2001:db8::1'), + ('rapiddns', 'ip', '192.0.2.1'), + ('securityscorecard', 'ip', '198.51.100.9'), + ('securityscorecard', 'ip', '2001:db8::1'), } securityscorecard_execution = next( execution for execution in completed_results[0].source_executions if execution.source == 'securityscorecard' ) assert securityscorecard_execution.status == 'completed' assert securityscorecard_execution.result_count == 2 - + reverse_execution = next( + execution for execution in completed_results[0].active_evidence.executions if execution.action == 'dns-lookup' + ) + assert reverse_execution.status == 'partial' + assert reverse_execution.error_type == 'TimeoutError' + assert reverse_execution.stop_reason == 'query-errors' + assert {(observation.kind, observation.value) for observation in reverse_execution.observations} == { + ('hostname', 'reverse.example.com') + } + api_execution = next( + execution for execution in completed_results[0].active_evidence.executions if execution.action == 'api-scan' + ) + assert api_execution.status == 'completed' + assert {(observation.kind, observation.value) for observation in api_execution.observations} == { + ('url', 'https://example.com/health') + } xml_hosts = { (element.findtext('hostname') or (element.text or '').strip(), element.findtext('ip')) for element in ElementTree.parse(report.with_suffix('.xml')).getroot().findall('host') @@ -325,13 +354,13 @@ async def test_rapiddns_evidence_reaches_existing_outputs( assert len(completed_results) == 2 assert FakeSecurityScorecard.created == 1 assert completed_results[1].target == 'example.com' - assert {'192.0.2.1', '198.51.100.2'} <= {value for kind, value in completed_results[1].results if kind == 'ip-address'} + assert {'192.0.2.1', '198.51.100.2'} <= {value for kind, value in completed_results[1].results if kind == 'ip'} assert ('email', 'user@example.com') in completed_results[1].results assert {(observation.source, observation.kind, observation.value) for observation in completed_results[1].observations} >= { ('dehashed', 'email', 'user@example.com'), - ('dehashed', 'ip-address', '198.51.100.2'), + ('dehashed', 'ip', '198.51.100.2'), ('rapiddns', 'hostname', 'api.example.com'), - ('rapiddns', 'ip-address', '192.0.2.1'), + ('rapiddns', 'ip', '192.0.2.1'), } monkeypatch.setattr(sys, 'argv', ['theHarvester', '-d', 'example.com', '-b', 'rapiddns']) diff --git a/tests/discovery/test_rocketreach.py b/tests/discovery/test_rocketreach.py index 94f25987..e2d54202 100644 --- a/tests/discovery/test_rocketreach.py +++ b/tests/discovery/test_rocketreach.py @@ -11,7 +11,7 @@ if 'aiohttp_socks' not in sys.modules: def from_url(*_args, **_kwargs): return None - setattr(aiohttp_socks_stub, 'ProxyConnector', _ProxyConnector) + aiohttp_socks_stub.ProxyConnector = _ProxyConnector # type: ignore[attr-defined] sys.modules['aiohttp_socks'] = aiohttp_socks_stub from theHarvester.discovery import rocketreach @@ -84,7 +84,7 @@ async def test_do_search_uses_people_data_endpoint_and_start_pagination(monkeypa assert first_data == {'query': {'current_employer_domain': ['example.com']}, 'start': 0, 'page_size': 100} assert second_data == {'query': {'current_employer_domain': ['example.com']}, 'start': 100, 'page_size': 50} - links = await search.get_links() + links = await search.get_urls() emails = await search.get_emails() assert len(links) == 150 assert len(emails) == 150 diff --git a/tests/discovery/test_shodan_engine.py b/tests/discovery/test_shodan_engine.py index 9a86d1c4..6e67cdf7 100644 --- a/tests/discovery/test_shodan_engine.py +++ b/tests/discovery/test_shodan_engine.py @@ -11,6 +11,25 @@ class TestShodanEngine: async def test_shodan_provider_failure_returns_attributed_empty_evidence(self, monkeypatch, caplog): from theHarvester.discovery import shodansearch + class FailingShodan: + def host(self, _ip): + raise shodansearch.exception.APIError('No information available for that IP.') + + monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') + monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: FailingShodan()) + caplog.set_level(logging.INFO, logger=shodansearch.__name__) + + search = shodansearch.SearchShodan() + result = await search.search_ip('203.0.113.1') + + assert result == OrderedDict({'203.0.113.1': 'Not in Shodan'}) + assert search.error_type is None + assert '203.0.113.1: Not in Shodan' in caplog.text + + @pytest.mark.asyncio + async def test_shodan_api_failure_exposes_only_its_error_type(self, monkeypatch, caplog): + from theHarvester.discovery import shodansearch + class FailingShodan: def host(self, _ip): raise shodansearch.exception.APIError('provider-secret-payload') @@ -19,12 +38,30 @@ class TestShodanEngine: monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: FailingShodan()) caplog.set_level(logging.INFO, logger=shodansearch.__name__) - result = await shodansearch.SearchShodan().search_ip('203.0.113.1') + search = shodansearch.SearchShodan() + result = await search.search_ip('203.0.113.1') - assert result == OrderedDict({'203.0.113.1': 'Not in Shodan'}) - assert '203.0.113.1: Not in Shodan' in caplog.text + assert result == OrderedDict({'203.0.113.1': 'Shodan request failed'}) + assert search.error_type == 'APIError' assert 'provider-secret-payload' not in caplog.text + @pytest.mark.asyncio + async def test_shodan_unexpected_failure_exposes_only_its_error_type(self, monkeypatch): + from theHarvester.discovery import shodansearch + + class FailingShodan: + def host(self, _ip): + raise RuntimeError('provider-secret-payload') + + monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') + monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: FailingShodan()) + + search = shodansearch.SearchShodan() + result = await search.search_ip('203.0.113.1') + + assert result == OrderedDict({'203.0.113.1': 'Shodan request failed'}) + assert search.error_type == 'RuntimeError' + @pytest.mark.asyncio async def test_shodan_engine_processes_without_work_item_error_and_yields_hostnames(self, monkeypatch, capsys): # Import inside the test so monkeypatching affects the already-imported module namespace. diff --git a/tests/discovery/test_takeover.py b/tests/discovery/test_takeover.py new file mode 100644 index 00000000..9771571e --- /dev/null +++ b/tests/discovery/test_takeover.py @@ -0,0 +1,37 @@ +import pytest + +from theHarvester.discovery import takeover +from theHarvester.lib.core import FetcherResponse + + +@pytest.mark.asyncio +async def test_takeover_distinguishes_transport_failure_from_successful_empty_body( + monkeypatch: pytest.MonkeyPatch, +) -> None: + search = takeover.TakeOver(['api.example.com', 'timeout.example.com']) + monkeypatch.setattr(search, 'fingerprints', {'No such app': 'Heroku'}) + + async def fake_fetch_all(urls, **kwargs): + assert kwargs['include_metadata'] is True + assert set(urls) == { + 'https://api.example.com', + 'http://api.example.com', + 'https://timeout.example.com', + 'http://timeout.example.com', + } + return [ + ('https://api.example.com', FetcherResponse(body='No such app', status=200, headers={})), + ('http://api.example.com', FetcherResponse(body='', status=204, headers={})), + ('https://timeout.example.com', None), + ('http://timeout.example.com', FetcherResponse(body='not vulnerable', status=200, headers={})), + ] + + monkeypatch.setattr(takeover.AsyncFetcher, 'fetch_all', fake_fetch_all) + + assert await search.process() is None + + assert search.request_count == 4 + assert search.request_error_count == 1 + assert search.request_error_types == {'TransportError'} + assert search.scan_error_type is None + assert await search.get_takeover_results() == {'https://api.example.com': [{'No such app': 'Heroku'}]} diff --git a/tests/discovery/test_zoomeyesearch.py b/tests/discovery/test_zoomeyesearch.py new file mode 100644 index 00000000..224f597e --- /dev/null +++ b/tests/discovery/test_zoomeyesearch.py @@ -0,0 +1,23 @@ +import pytest + +from theHarvester.discovery import zoomeyesearch + + +@pytest.mark.asyncio +async def test_banner_urls_are_absolute_http_and_scoped(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(zoomeyesearch.Core, 'zoomeye_key', staticmethod(lambda: 'test-key')) + search = zoomeyesearch.SearchZoomEye('example.com', 1) + banner = '\n'.join( + f'"{value}"' + for value in ( + 'https://api.example.com/v1', + '//example.com/path', + '/assets/example.com/config.js', + 'https://evil.test/?target=example.com', + 'ftp://api.example.com/archive', + ) + ) + + _hostnames, _emails, _ips, _asns, urls = await search.parse_matches([{'service': {'banner': banner}}]) + + assert urls == {'https://api.example.com/v1'} diff --git a/tests/lib/test_active_evidence.py b/tests/lib/test_active_evidence.py new file mode 100644 index 00000000..f8b4d14f --- /dev/null +++ b/tests/lib/test_active_evidence.py @@ -0,0 +1,110 @@ +from datetime import UTC, datetime + +import pytest + +from theHarvester.lib.active_evidence import ActionExecution, ActionObservation, ActiveEvidence, ArtifactReference + + +def screenshot_artifact() -> ArtifactReference: + return ArtifactReference( + kind='screenshot', + subject_kind='hostname', + subject_value='api.example.com', + path='screenshots/api.example.com.png', + media_type='image/png', + size_bytes=3, + sha256='0' * 64, + created_at=datetime(2026, 8, 9, 12, 0, tzinfo=UTC), + ) + + +def test_active_evidence_owns_action_results_and_artifacts() -> None: + evidence = ActiveEvidence( + executions=( + ActionExecution.finish( + action='dns-resolve', + status='completed', + duration_ms=12.5, + groups={'ip': ['192.0.2.10', '192.0.2.10']}, + ), + ActionExecution.finish( + action='screenshot', + status='completed', + duration_ms=4.0, + groups={}, + artifacts=(screenshot_artifact(),), + ), + ActionExecution.finish(action='takeover', status='completed', duration_ms=2.0, groups={}), + ) + ) + + assert evidence.executions[0].result_count == 1 + assert evidence.executions[0].observations == (ActionObservation('ip', '192.0.2.10'),) + assert evidence.executions[1].result_count == 0 + assert evidence.executions[1].artifacts == (screenshot_artifact(),) + + +def test_active_evidence_rejects_duplicate_actions() -> None: + with pytest.raises(ValueError, match='action executions must be unique'): + ActiveEvidence( + executions=( + ActionExecution('dns-resolve', 'completed', 1.0), + ActionExecution('dns-resolve', 'failed', 2.0, error_type='RuntimeError'), + ) + ) + + +def test_action_execution_rejects_noncanonical_observations_and_artifacts() -> None: + observation = ActionObservation('hostname', 'api.example.com') + artifact = screenshot_artifact() + + with pytest.raises(ValueError, match='action observations must be deduplicated and sorted'): + ActionExecution('dns-brute', 'completed', 1.0, observations=(observation, observation)) + with pytest.raises(ValueError, match='artifacts must be deduplicated and sorted'): + ActionExecution('screenshot', 'completed', 1.0, artifacts=(artifact, artifact)) + + +def test_active_evidence_rejects_screenshot_results() -> None: + with pytest.raises(ValueError, match='screenshots must be stored as artifacts'): + ActionExecution.finish( + action='screenshot', + status='completed', + duration_ms=1.0, + groups={'screenshot': ['https://api.example.com']}, + ) + with pytest.raises(ValueError, match='known non-empty result'): + ArtifactReference( + kind='screenshot', + subject_kind='screenshot', + subject_value='https://api.example.com', + path='screenshots/api.example.com.png', + media_type='image/png', + size_bytes=3, + sha256='0' * 64, + created_at=datetime(2026, 8, 9, 12, 0, tzinfo=UTC), + ) + + +def test_artifact_reference_rejects_invalid_metadata() -> None: + with pytest.raises(ValueError, match='sha256'): + ArtifactReference( + kind='screenshot', + subject_kind='hostname', + subject_value='api.example.com', + path='screenshots/api.example.com.png', + media_type='image/png', + size_bytes=3, + sha256='not-a-hash', + created_at=datetime(2026, 8, 9, 12, 0, tzinfo=UTC), + ) + with pytest.raises(ValueError, match='timezone-aware'): + ArtifactReference( + kind='screenshot', + subject_kind='hostname', + subject_value='api.example.com', + path='screenshots/api.example.com.png', + media_type='image/png', + size_bytes=3, + sha256='0' * 64, + created_at=datetime(2026, 8, 9, 12, 0), + ) diff --git a/tests/lib/test_api_v1.py b/tests/lib/test_api_v1.py new file mode 100644 index 00000000..55d9fcf3 --- /dev/null +++ b/tests/lib/test_api_v1.py @@ -0,0 +1,871 @@ +from __future__ import annotations + +import asyncio +import json + +import pytest +from fastapi.testclient import TestClient + + +def _jsonl_result( + *, + target: str = 'example.test', + finding_type: str = 'email', + value: str = 'a@example.test', + finding_fields: dict[str, object] | None = None, + summary_fields: dict[str, object] | None = None, +) -> str: + summary = { + 'type': 'summary', + 'run_id': '9f9b4383-6cc4-4f3f-80a4-c8d21930dc2d', + 'target': target, + 'started_at': '2026-08-08T01:00:00Z', + 'completed_at': '2026-08-08T01:01:00Z', + 'evidence_status': 'complete', + 'result_count': 1, + 'counts': {finding_type: 1}, + } + summary.update(summary_fields or {}) + return '\n'.join( + ( + json.dumps(summary), + json.dumps({'type': finding_type, 'value': value, 'sources': [], **(finding_fields or {})}), + '', + ) + ) + + +@pytest.mark.parametrize( + ('finding_type', 'finding_fields'), + [ + ('made-up-kind', {}), + ('api-endpoint', {}), + ('interesting-url', {}), + ('ip-address', {}), + ('linkedin-link', {}), + ('subdomain', {}), + ('hostname', {'dns_status': 'made-up-status'}), + ], +) +def test_api_rejects_jsonl_findings_outside_the_contract( + tmp_path, + monkeypatch, + finding_type: str, + finding_fields: dict[str, object], +) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + + with TestClient(api.app) as client: + response = client.post( + '/api/v1/runs/import', + params={'filename': 'invalid.jsonl'}, + headers={'X-API-Key': 'test-key'}, + content=_jsonl_result(finding_type=finding_type, finding_fields=finding_fields), + ) + + assert response.status_code == 400 + + +def test_api_exposes_one_fresh_run_contract(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + + with TestClient(api.app, base_url='http://127.0.0.1', client=('127.0.0.1', 50000)) as client: + schema = client.get('/openapi.json').json() + paths = set(schema['paths']) + old_responses = [ + client.get('/'), + client.get('/query?domain=example.test&source=crtsh'), + client.get('/sources'), + client.get('/dnsbrute?domain=example.test'), + client.get('/runs'), + client.post('/additional/all', json={'domain': 'example.test'}), + ] + + assert paths == { + '/api/v1/sources', + '/api/v1/runs', + '/api/v1/runs/import', + '/api/v1/runs/import-database', + '/api/v1/runs/{run_id}', + '/api/v1/runs/{run_id}/cancel', + '/api/v1/runs/{run_id}/export', + '/api/v1/runs/{run_id}/screenshots/{name}', + } + assert all(response.status_code == 404 for response in old_responses) + + +def test_screenshot_route_serves_only_a_run_owned_png(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + headers = {'X-API-Key': 'test-key'} + + with TestClient(api.app, client=('127.0.0.2', 50000)) as client: + imported = client.post( + '/api/v1/runs/import', + params={'filename': 'smoke.jsonl'}, + headers=headers, + content=_jsonl_result( + finding_type='hostname', + value='owned.example.test', + summary_fields={ + 'action_executions': [ + { + 'action': 'screenshot', + 'status': 'completed', + 'duration_ms': 1, + 'result_count': 0, + 'error_type': None, + 'stop_reason': None, + } + ], + 'artifacts': [ + { + 'action': 'screenshot', + 'kind': 'screenshot', + 'subject': {'kind': 'hostname', 'value': 'owned.example.test'}, + 'file': { + 'path': 'screenshots/owned.example.test.png', + 'media_type': 'image/png', + 'size_bytes': 16, + 'sha256': '0' * 64, + }, + 'created_at': '2026-08-08T01:01:00Z', + } + ], + }, + ), + ) + assert imported.status_code == 201 + run_id = imported.json()['run_id'] + screenshot_dir = tmp_path / 'artifacts' / run_id / 'screenshots' + screenshot_dir.mkdir(parents=True) + (screenshot_dir / 'owned.example.test.png').write_bytes(b'owned screenshot') + (screenshot_dir / 'unrecorded.png').write_bytes(b'unrecorded screenshot') + outside = tmp_path / 'outside.png' + outside.write_bytes(b'outside screenshot') + (screenshot_dir / 'linked.png').symlink_to(outside) + + owned = client.get(f'/api/v1/runs/{run_id}/screenshots/owned.example.test.png', headers=headers) + unrecorded = client.get(f'/api/v1/runs/{run_id}/screenshots/unrecorded.png', headers=headers) + linked = client.get(f'/api/v1/runs/{run_id}/screenshots/linked.png', headers=headers) + traversal = client.get(f'/api/v1/runs/{run_id}/screenshots/%2E%2E%2Foutside.png', headers=headers) + + assert owned.status_code == 200 + assert owned.content == b'owned screenshot' + assert unrecorded.status_code == 404 + assert linked.status_code == 404 + assert traversal.status_code == 404 + + +def test_openapi_names_the_public_response_shapes(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + + with TestClient(api.app) as client: + schema = client.get('/openapi.json').json() + + paths = schema['paths'] + assert paths['/api/v1/sources']['get']['responses']['200']['content']['application/json']['schema'] == { + '$ref': '#/components/schemas/SourceCatalogResponse' + } + assert paths['/api/v1/runs']['get']['responses']['200']['content']['application/json']['schema']['items'] == { + '$ref': '#/components/schemas/RunSummary' + } + for path, method in ( + ('/api/v1/runs', 'post'), + ('/api/v1/runs/import', 'post'), + ('/api/v1/runs/{run_id}', 'get'), + ('/api/v1/runs/{run_id}/cancel', 'post'), + ): + assert paths[path][method]['responses']['201' if path in {'/api/v1/runs', '/api/v1/runs/import'} else '200']['content'][ + 'application/json' + ]['schema'] == {'$ref': '#/components/schemas/RunDetail'} + + +def test_source_catalog_exposes_shared_action_activities(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + + with TestClient(api.app) as client: + response = client.get('/api/v1/sources', headers={'X-API-Key': 'test-key'}) + + assert response.status_code == 200 + catalog = response.json() + assert catalog['sources'] + assert catalog['actions'] == [ + {'name': 'api-scan', 'activity': 'P2'}, + {'name': 'dns-brute', 'activity': 'P1'}, + {'name': 'dns-lookup', 'activity': 'P1'}, + {'name': 'dns-recursive', 'activity': 'P1'}, + {'name': 'dns-resolve', 'activity': 'P1'}, + {'name': 'screenshot', 'activity': 'P2'}, + {'name': 'shodan', 'activity': 'P0'}, + {'name': 'takeover', 'activity': 'P2'}, + ] + + +def test_openapi_explains_scope_and_execution_controls(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + + with TestClient(api.app) as client: + schema = client.get('/openapi.json').json() + + request_body = schema['paths']['/api/v1/runs']['post']['requestBody'] + properties = request_body['content']['application/json']['schema']['properties'] + + assert request_body['required'] is True + assert 'union' in properties['sources']['description'] + assert 'do not filter' in properties['sources']['description'] + assert '/24' in properties['dns_lookup']['description'] + assert 'whole run' in properties['deadline_seconds']['description'] + assert 'three resolver' in properties['dns_recursive_query_limit']['description'] + assert 'discovery sources' in properties['proxies']['description'] + assert 'configured proxies' in properties['takeover']['description'] + assert 'take_over' not in properties + assert 'endpoint paths' in properties['api_scan_paths']['description'] + import_content = schema['paths']['/api/v1/runs/import']['post']['requestBody']['content'] + assert set(import_content) == {'application/x-ndjson'} + export_content = schema['paths']['/api/v1/runs/{run_id}/export']['get']['responses']['200']['content'] + assert set(export_content) == {'application/x-ndjson'} + assert set(schema['components']['schemas']['NormalizedResult']['properties']) == { + 'type', + 'value', + 'sources', + 'actions', + } + assert export_content['application/x-ndjson']['schema']['description'] == ( + 'UTF-8 JSONL with one summary followed by normalized findings.' + ) + + def references(value): + if isinstance(value, dict): + if '$ref' in value: + yield value['$ref'] + for child in value.values(): + yield from references(child) + elif isinstance(value, list): + for child in value: + yield from references(child) + + components = schema['components']['schemas'] + for reference in references(schema): + assert reference.startswith('#/components/schemas/') + assert reference.removeprefix('#/components/schemas/') in components + + +def test_run_detail_exposes_one_normalized_evidence_surface(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + + with TestClient(api.app) as client: + imported = client.post( + '/api/v1/runs/import', + params={'filename': 'result.jsonl'}, + headers={'X-API-Key': 'test-key'}, + content=_jsonl_result(), + ) + + assert imported.status_code == 201 + assert 'evidence' not in imported.json() + + +def test_api_scan_can_run_without_discovery_sources(tmp_path, monkeypatch) -> None: + from pydantic import ValidationError + + from theHarvester.lib.api.run_models import RunRequest + + request = RunRequest(target='example.test', sources=[], api_scan=True, api_scan_paths=['/api/v2', '/health']) + + assert request.api_scan is True + assert request.api_scan_paths == ['/api/v2', '/health'] + with pytest.raises(ValidationError): + RunRequest(target='example.test', sources=[], api_scan=True, api_scan_paths=['https://other.example/api']) + + +def test_fresh_api_uses_catalog_takeover_name_and_rejects_unknown_fields() -> None: + from pydantic import ValidationError + + from theHarvester.lib.api.run_models import RunRequest + + request = RunRequest(target='example.test', sources=[], takeover=True) + + assert request.takeover is True + with pytest.raises(ValidationError): + RunRequest(target='example.test', sources=[], take_over=True) + + +@pytest.mark.parametrize( + ('evidence_status', 'execution_status'), + [('complete', 'failed'), ('partial', 'completed')], +) +def test_api_rejects_evidence_status_that_disagrees_with_executions( + tmp_path, + monkeypatch, + evidence_status, + execution_status, +) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + payload = _jsonl_result( + summary_fields={ + 'evidence_status': evidence_status, + 'source_executions': [ + { + 'source': 'crtsh', + 'status': execution_status, + 'duration_ms': 1, + 'result_count': 0, + 'error_type': 'RuntimeError', + 'stop_reason': 'provider-error', + } + ], + }, + finding_fields={'sources': []}, + ) + + with TestClient(api.app) as client: + response = client.post( + '/api/v1/runs/import', + params={'filename': 'inconsistent.jsonl'}, + headers={'X-API-Key': 'test-key'}, + content=payload, + ) + + assert response.status_code == 400 + assert response.json()['detail'] == 'Evidence status does not match its execution outcomes' + + +def test_api_preserves_sparse_failed_status_without_executions(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + database = tmp_path / 'runs.sqlite' + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(database)) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + payload = _jsonl_result(summary_fields={'evidence_status': 'failed'}) + + with TestClient(api.app) as client: + imported = client.post( + '/api/v1/runs/import', + params={'filename': 'failed.jsonl'}, + headers={'X-API-Key': 'test-key'}, + content=payload, + ) + exported = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/export', headers={'X-API-Key': 'test-key'}) + + assert imported.status_code == 201 + assert imported.json()['evidence_status'] == 'failed' + assert json.loads(exported.text.splitlines()[0])['evidence_status'] == 'failed' + + +def test_api_import_and_export_accept_only_jsonl(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + headers = {'X-API-Key': 'test-key'} + + with TestClient(api.app, client=('127.0.0.3', 50000)) as client: + rejected = client.post( + '/api/v1/runs/import', + params={'filename': 'legacy.json'}, + headers=headers, + content='{"target":"example.test"}', + ) + imported = client.post( + '/api/v1/runs/import', + params={'filename': 'result.jsonl'}, + headers={**headers, 'Content-Type': 'application/x-ndjson'}, + content=_jsonl_result(), + ) + exported = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/export', headers=headers) + reimported = client.post( + '/api/v1/runs/import', + params={'filename': 'round-trip.jsonl'}, + headers={**headers, 'Content-Type': 'application/x-ndjson'}, + content=exported.content, + ) + old_json = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/exports/json', headers=headers) + old_csv = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/exports/csv', headers=headers) + + assert rejected.status_code == 400 + assert rejected.json()['detail'] == 'Choose a .jsonl result file' + assert imported.status_code == 201 + assert exported.status_code == 200 + assert exported.headers['content-type'] == 'application/x-ndjson' + assert exported.headers['content-disposition'].endswith('.jsonl"') + records = [json.loads(line) for line in exported.text.splitlines()] + assert 'schema' not in records[0] + assert 'schema_version' not in records[0] + assert records[0]['type'] == 'summary' + assert records[0]['target'] == 'example.test' + assert records[1] == {'sources': [], 'type': 'email', 'value': 'a@example.test'} + assert reimported.status_code == 201 + assert reimported.json()['results'] == [{'type': 'email', 'value': 'a@example.test', 'sources': [], 'actions': []}] + assert old_json.status_code == 404 + assert old_csv.status_code == 404 + + +def test_api_jsonl_round_trip_preserves_canonical_url_sources(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + headers = {'X-API-Key': 'test-key', 'Content-Type': 'application/x-ndjson'} + sources = ['builtwith', 'gitlab', 'rocketreach'] + executions = [ + { + 'source': source, + 'status': 'completed', + 'duration_ms': 1, + 'result_count': 1, + 'error_type': None, + 'stop_reason': None, + } + for source in sources + ] + payload = _jsonl_result( + finding_type='url', + value='https://example.test/profile', + finding_fields={'sources': sources}, + summary_fields={'source_executions': executions}, + ) + + with TestClient(api.app) as client: + imported = client.post( + '/api/v1/runs/import', + params={'filename': 'urls.jsonl'}, + headers=headers, + content=payload, + ) + run_id = imported.json()['run_id'] + detail = client.get(f'/api/v1/runs/{run_id}', headers={'X-API-Key': 'test-key'}) + exported = client.get(f'/api/v1/runs/{run_id}/export', headers={'X-API-Key': 'test-key'}) + + assert imported.status_code == 201 + assert detail.json()['results'] == [ + { + 'type': 'url', + 'value': 'https://example.test/profile', + 'sources': sources, + 'actions': [], + } + ] + records = [json.loads(line) for line in exported.text.splitlines()] + assert records[1] == {'sources': sources, 'type': 'url', 'value': 'https://example.test/profile'} + + +def test_api_database_import_rejects_non_sqlite_content(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + + with TestClient(api.app) as client: + response = client.post( + '/api/v1/runs/import-database', + params={'filename': 'results.sqlite'}, + headers={'X-API-Key': 'test-key', 'Content-Type': 'application/vnd.sqlite3'}, + content=b'not a sqlite database', + ) + + assert response.status_code == 400 + assert response.json()['detail'] == 'Uploaded file is not a SQLite database' + + +def test_api_database_import_exposes_completed_cli_runs(tmp_path, monkeypatch) -> None: + from datetime import UTC, datetime + + from theHarvester.lib.api import api + from theHarvester.lib.completed_result import CompletedResult + from theHarvester.lib.database import ResultStore, dispose_sqlite_databases + + source_database = tmp_path / 'source.sqlite' + destination_database = tmp_path / 'destination.sqlite' + now = datetime.now(UTC) + completed = CompletedResult.finish( + target='imported.example.test', + started_at=now, + completed_at=now, + groups={'hostname': ['api.imported.example.test']}, + ) + + async def seed() -> None: + store = ResultStore(source_database) + await store.initialize() + await store.save_run(completed) + await dispose_sqlite_databases() + + asyncio.run(seed()) + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(destination_database)) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + + with TestClient(api.app) as client: + imported = client.post( + '/api/v1/runs/import-database', + params={'filename': 'source.sqlite'}, + headers={'X-API-Key': 'test-key', 'Content-Type': 'application/vnd.sqlite3'}, + content=source_database.read_bytes(), + ) + detail = client.get( + f'/api/v1/runs/{completed.run_id}', + headers={'X-API-Key': 'test-key'}, + ) + + assert imported.status_code == 201 + assert imported.json()['imported_run_ids'] == [str(completed.run_id)] + assert detail.status_code == 200 + assert detail.json()['results'] == [{'type': 'hostname', 'value': 'api.imported.example.test', 'sources': [], 'actions': []}] + + +def test_api_jsonl_round_trip_preserves_source_attribution(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + source_execution = { + 'source': 'crtsh', + 'status': 'completed', + 'duration_ms': 0, + 'result_count': 1, + 'error_type': None, + 'stop_reason': None, + } + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + headers = {'X-API-Key': 'test-key'} + + with TestClient(api.app, client=('127.0.0.4', 50000)) as client: + imported = client.post( + '/api/v1/runs/import', + params={'filename': 'complete.jsonl'}, + headers=headers, + content=_jsonl_result( + finding_fields={'sources': ['crtsh']}, + summary_fields={'source_executions': [source_execution]}, + ), + ) + exported = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/export', headers=headers) + reimported = client.post( + '/api/v1/runs/import', + params={'filename': 'complete-round-trip.jsonl'}, + headers=headers, + content=exported.content, + ) + + summary = json.loads(exported.text.splitlines()[0]) + assert imported.json()['evidence_status'] == 'complete' + assert imported.json()['source_executions'] == [source_execution] + assert imported.json()['results'] == [{'type': 'email', 'value': 'a@example.test', 'sources': ['crtsh'], 'actions': []}] + assert summary['run_id'] == imported.json()['run_id'] + assert summary['evidence_status'] == 'complete' + assert summary['source_executions'] == [source_execution] + assert json.loads(exported.text.splitlines()[1])['sources'] == ['crtsh'] + assert reimported.json()['evidence_status'] == 'complete' + assert reimported.json()['request']['source_run_id'] == imported.json()['run_id'] + assert reimported.json()['source_executions'] == [source_execution] + + +def test_api_jsonl_export_uses_evidence_timestamps_not_lifecycle_timestamps(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + store = RunStore() + queued = asyncio.run(store.create(RunRequest(target='example.test', sources=['crtsh']))) + asyncio.run(store.claim_next()) + asyncio.run( + store.finish( + queued['run_id'], + { + 'run_id': '3e7cf0c1-214b-4429-80ba-058b2cb68b06', + 'target': 'example.test', + 'status': 'complete', + 'started_at': '2026-08-07T01:00:00Z', + 'completed_at': '2026-08-07T01:01:00Z', + 'results': [], + 'source_executions': [], + }, + '', + ) + ) + + with TestClient(api.app, client=('127.0.0.16', 50000)) as client: + response = client.get( + f'/api/v1/runs/{queued["run_id"]}/export', + headers={'X-API-Key': 'test-key'}, + ) + + summary = json.loads(response.text.splitlines()[0]) + completed = asyncio.run(store.load_completed_result(queued['run_id'])) + assert completed is not None + assert response.text == completed.jsonl() + assert summary['started_at'] == '2026-08-07T01:00:00Z' + assert summary['completed_at'] == '2026-08-07T01:01:00Z' + + +def test_api_jsonl_export_uses_lifecycle_timestamps_for_sparse_partial_evidence(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + store = RunStore() + queued = asyncio.run(store.create(RunRequest(target='example.test', sources=['crtsh']))) + asyncio.run(store.claim_next()) + asyncio.run( + store.fail( + queued['run_id'], + 'Provider process exited.', + '', + evidence={ + 'run_id': 'eb470313-d813-4d81-bd75-c1221a8bc00e', + 'target': 'example.test', + 'status': 'partial', + 'results': [], + 'source_executions': [], + }, + ) + ) + + with TestClient(api.app, client=('127.0.0.17', 50000)) as client: + exported = client.get( + f'/api/v1/runs/{queued["run_id"]}/export', + headers={'X-API-Key': 'test-key'}, + ) + reimported = client.post( + '/api/v1/runs/import', + params={'filename': 'partial.jsonl'}, + headers={'X-API-Key': 'test-key'}, + content=exported.content, + ) + + summary = json.loads(exported.text.splitlines()[0]) + assert exported.status_code == 200 + assert isinstance(summary['started_at'], str) + assert isinstance(summary['completed_at'], str) + assert reimported.status_code == 201 + assert summary['evidence_status'] == 'partial' + assert reimported.json()['evidence_status'] == 'partial' + + +def test_api_jsonl_round_trip_uses_canonical_hostname_and_ip_kinds(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + headers = {'X-API-Key': 'test-key'} + + for client_ip, finding_type, value, run_id in ( + ('127.0.0.5', 'hostname', 'www.example.test', '0f17b751-dd31-46da-968f-31580e233b72'), + ('127.0.0.6', 'ip', '192.0.2.1', 'f7419165-d78c-4aef-9023-e9686f864ff0'), + ): + with TestClient(api.app, client=(client_ip, 50000)) as client: + imported = client.post( + '/api/v1/runs/import', + params={'filename': f'{finding_type}.jsonl'}, + headers=headers, + content=_jsonl_result(finding_type=finding_type, value=value, summary_fields={'run_id': run_id}), + ) + exported = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/export', headers=headers) + reimported = client.post( + '/api/v1/runs/import', + params={'filename': f'{finding_type}-round-trip.jsonl'}, + headers=headers, + content=exported.content, + ) + + assert imported.json()['results'] == [{'type': finding_type, 'value': value, 'sources': [], 'actions': []}] + assert json.loads(exported.text.splitlines()[1]) == {'sources': [], 'type': finding_type, 'value': value} + assert reimported.json()['results'] == [{'type': finding_type, 'value': value, 'sources': [], 'actions': []}] + + +def test_api_jsonl_round_trip_preserves_execution_outcomes_and_action_origins(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + headers = {'X-API-Key': 'test-key'} + payload = _jsonl_result( + finding_type='hostname', + value='api.example.test', + finding_fields={'sources': ['crtsh', 'crtsh'], 'actions': ['dns-brute', 'dns-brute']}, + summary_fields={ + 'evidence_status': 'partial', + 'source_executions': [ + { + 'source': 'crtsh', + 'status': 'completed', + 'duration_ms': 1, + 'result_count': 1, + 'error_type': None, + 'stop_reason': None, + }, + { + 'source': 'certspotter', + 'status': 'rate-limited', + 'duration_ms': 2, + 'result_count': 0, + 'error_type': None, + 'stop_reason': 'http-429', + }, + ], + 'action_executions': [ + { + 'action': 'dns-brute', + 'status': 'partial', + 'duration_ms': 3, + 'result_count': 1, + 'error_type': 'TimeoutError', + 'stop_reason': 'query-errors', + } + ], + }, + ) + + with TestClient(api.app, client=('127.0.0.18', 50000)) as client: + imported = client.post( + '/api/v1/runs/import', + params={'filename': 'attributed.jsonl'}, + headers=headers, + content=payload, + ) + exported = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/export', headers=headers) + reimported = client.post( + '/api/v1/runs/import', + params={'filename': 'attributed-round-trip.jsonl'}, + headers=headers, + content=exported.content, + ) + + assert imported.status_code == 201 + assert exported.status_code == 200 + assert reimported.status_code == 201 + assert reimported.json()['evidence_status'] == 'partial' + assert reimported.json()['source_executions'] == imported.json()['source_executions'] + assert reimported.json()['action_executions'] == imported.json()['action_executions'] + assert reimported.json()['results'] == [ + { + 'type': 'hostname', + 'value': 'api.example.test', + 'sources': ['crtsh'], + 'actions': ['dns-brute'], + } + ] + + +def test_api_rejects_non_string_jsonl_timestamps(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + + with TestClient(api.app, client=('127.0.0.7', 50000)) as client: + response = client.post( + '/api/v1/runs/import', + params={'filename': 'invalid.jsonl'}, + headers={'X-API-Key': 'test-key'}, + content=_jsonl_result(summary_fields={'completed_at': {'not': 'a timestamp'}}), + ) + + assert response.status_code == 400 + assert response.json()['detail'] == 'JSONL summary must contain an ISO-8601 UTC completed_at' + + +def test_api_rejects_invalid_jsonl_summary_identity_and_timestamps(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + cases = ( + ({'run_id': None}, None, 'JSONL summary must contain a UUID run_id'), + ({}, 'run_id', 'JSONL summary must contain a UUID run_id'), + ({'run_id': 'not-a-uuid'}, None, 'JSONL summary must contain a UUID run_id'), + ({}, 'started_at', 'JSONL summary must contain an ISO-8601 UTC started_at'), + ({}, 'completed_at', 'JSONL summary must contain an ISO-8601 UTC completed_at'), + ({'started_at': 'not-a-time'}, None, 'JSONL summary must contain an ISO-8601 UTC started_at'), + ( + {'completed_at': '2026-08-08T02:01:00+01:00'}, + None, + 'JSONL summary must contain an ISO-8601 UTC completed_at', + ), + ( + {'started_at': '2026-08-08T03:00:00Z', 'completed_at': '2026-08-08T02:00:00Z'}, + None, + 'JSONL summary completed_at must not be earlier than started_at', + ), + ) + + for index, (updates, removed_field, detail) in enumerate(cases, start=8): + records = [json.loads(line) for line in _jsonl_result().splitlines()] + records[0].update(updates) + if removed_field: + records[0].pop(removed_field) + content = ''.join(json.dumps(record) + '\n' for record in records) + with TestClient(api.app, client=(f'127.0.0.{index}', 50000)) as client: + response = client.post( + '/api/v1/runs/import', + params={'filename': 'invalid.jsonl'}, + headers={'X-API-Key': 'test-key'}, + content=content, + ) + + assert response.status_code == 400 + assert response.json()['detail'] == detail + + +def test_api_refuses_to_export_before_evidence_exists(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + run = asyncio.run(RunStore().create(RunRequest(target='example.test', sources=['crtsh']))) + + with TestClient(api.app) as client: + response = client.get(f'/api/v1/runs/{run["run_id"]}/export', headers={'X-API-Key': 'test-key'}) + + assert response.status_code == 409 + assert response.json()['detail'] == 'No run evidence is available to export' diff --git a/tests/lib/test_completed_persistence.py b/tests/lib/test_completed_persistence.py index c11cb8bd..a2e2910a 100644 --- a/tests/lib/test_completed_persistence.py +++ b/tests/lib/test_completed_persistence.py @@ -8,6 +8,7 @@ from uuid import UUID import pytest from theHarvester.lib import database as database_module +from theHarvester.lib.active_evidence import ActionExecution, ActiveEvidence, ArtifactReference from theHarvester.lib.completed_result import CompletedResult, ResultObservation, SourceExecution from theHarvester.lib.database import ( DuplicateRunError, @@ -56,6 +57,78 @@ CREATE TABLE discovery_observations ( PRAGMA user_version = 1; """ +SCHEMA_V2_RUN_PROVENANCE = """ +CREATE TABLE runs ( + run_id TEXT PRIMARY KEY, + target TEXT NOT NULL, + started_at TEXT NOT NULL, + completed_at TEXT NOT NULL +); +CREATE TABLE results ( + run_id TEXT NOT NULL REFERENCES runs(run_id) ON DELETE CASCADE, + position INTEGER NOT NULL, + kind TEXT NOT NULL, + value TEXT NOT NULL, + PRIMARY KEY (run_id, position), + UNIQUE (run_id, kind, value) +); +CREATE TABLE executions ( + run_id TEXT NOT NULL REFERENCES runs(run_id) ON DELETE CASCADE, + position INTEGER NOT NULL, + producer_kind TEXT NOT NULL, + name TEXT NOT NULL, + status TEXT NOT NULL, + duration_ms REAL NOT NULL, + result_count INTEGER NOT NULL, + error_type TEXT, + stop_reason TEXT, + PRIMARY KEY (run_id, position), + UNIQUE (run_id, producer_kind, name) +); +CREATE TABLE result_origins ( + run_id TEXT NOT NULL, + result_position INTEGER NOT NULL, + execution_position INTEGER NOT NULL, + PRIMARY KEY (run_id, result_position, execution_position), + FOREIGN KEY (run_id, result_position) REFERENCES results(run_id, position) ON DELETE CASCADE, + FOREIGN KEY (run_id, execution_position) REFERENCES executions(run_id, position) ON DELETE CASCADE +); +CREATE TABLE legacy_observations ( + id INTEGER PRIMARY KEY, + domain TEXT, + resource TEXT, + kind TEXT, + discovered_on DATE, + source TEXT +); +PRAGMA user_version = 2; +""" + +SCHEMA_V4_URL_KINDS = ( + SCHEMA_V2_RUN_PROVENANCE + + """ +CREATE TABLE artifacts ( + run_id TEXT NOT NULL, + position INTEGER NOT NULL, + result_position INTEGER NOT NULL, + execution_position INTEGER NOT NULL, + kind TEXT NOT NULL, + path TEXT NOT NULL, + media_type TEXT NOT NULL, + size_bytes INTEGER NOT NULL, + sha256 TEXT NOT NULL, + created_at TEXT NOT NULL, + PRIMARY KEY (run_id, position), + FOREIGN KEY (run_id, result_position) REFERENCES results(run_id, position) ON DELETE CASCADE, + FOREIGN KEY (run_id, execution_position) REFERENCES executions(run_id, position) ON DELETE CASCADE +); +PRAGMA user_version = 4; +""" +) + +SCHEMA_V5_RESULT_KINDS = SCHEMA_V4_URL_KINDS + 'PRAGMA user_version = 5;' +SCHEMA_V6_RESULT_KINDS = SCHEMA_V5_RESULT_KINDS + 'PRAGMA user_version = 6;' + def completed_result(run_id: str = 'f047261c-0afb-4e18-89d5-28a7d977f51f') -> CompletedResult: return CompletedResult.finish( @@ -71,12 +144,33 @@ def completed_result(run_id: str = 'f047261c-0afb-4e18-89d5-28a7d977f51f') -> Co ], 'dns-recursive-summary': ['{"depth_reached":1,"query_count":24,"stop_reason":"depth-limit","zero_yield_batches":0}'], 'hostname': ['api.example.com'], - 'ip-address': ['192.0.2.1'], + 'ip': ['192.0.2.1'], 'person': ['{"firstname":"Ada","lastname":"Lovelace"}'], }, ) +def screenshot_execution(completed_at: datetime) -> ActionExecution: + return ActionExecution.finish( + action='screenshot', + status='completed', + duration_ms=4.0, + groups={}, + artifacts=( + ArtifactReference( + kind='screenshot', + subject_kind='hostname', + subject_value='api.example.com', + path='screenshots/api.example.com.png', + media_type='image/png', + size_bytes=3, + sha256='0' * 64, + created_at=completed_at, + ), + ), + ) + + @pytest.mark.asyncio async def test_initialization_enables_wal_when_sqlite_contains_the_reset_fix(tmp_path) -> None: database = tmp_path / 'stash.sqlite' @@ -124,10 +218,10 @@ async def test_newer_schema_is_rejected_without_changing_journal_mode(tmp_path) database = tmp_path / 'stash.sqlite' store = ResultStore(database) with sqlite3.connect(database) as db: - db.execute('PRAGMA user_version = 3') + db.execute('PRAGMA user_version = 8') original_journal_mode = db.execute('PRAGMA journal_mode').fetchone()[0] - with pytest.raises(RuntimeError, match='schema version 3 is newer than supported version 2'): + with pytest.raises(RuntimeError, match='schema version 8 is newer than supported version 7'): await store.initialize() with sqlite3.connect(database) as db: @@ -197,7 +291,13 @@ async def test_completed_result_round_trip_preserves_legacy_observations(tmp_pat jsonl_items = {(record['type'], record['value']) for line in result.jsonl().splitlines()[1:] if (record := json.loads(line))} assert stored == [('example.com', 'legacy.example.com', 'hostname', 'legacy-source')] assert stored_items == jsonl_items - assert run_types == {'run_id': 'TEXT', 'target': 'TEXT', 'started_at': 'TEXT', 'completed_at': 'TEXT'} + assert run_types == { + 'run_id': 'TEXT', + 'target': 'TEXT', + 'started_at': 'TEXT', + 'completed_at': 'TEXT', + 'evidence_status': 'TEXT', + } assert result_types == { 'run_id': 'TEXT', 'position': 'INTEGER', @@ -206,6 +306,26 @@ async def test_completed_result_round_trip_preserves_legacy_observations(tmp_pat } +@pytest.mark.asyncio +@pytest.mark.parametrize('evidence_status', ['partial', 'failed']) +async def test_sparse_evidence_status_survives_result_store_round_trip(tmp_path, evidence_status: str) -> None: + store = ResultStore(tmp_path / 'stash.sqlite') + await store.initialize() + result = CompletedResult.finish( + target='example.com', + started_at=datetime(2026, 8, 5, 12, 0, tzinfo=UTC), + completed_at=datetime(2026, 8, 5, 12, 1, tzinfo=UTC), + groups={}, + evidence_status=evidence_status, + ) + + await store.save_run(result) + + loaded = await store.load_run(result.run_id) + assert loaded == result + assert loaded.status == evidence_status + + @pytest.mark.asyncio async def test_completed_result_round_trip_preserves_source_provenance(tmp_path) -> None: database = tmp_path / 'stash.sqlite' @@ -253,6 +373,124 @@ async def test_completed_result_round_trip_preserves_source_provenance(tmp_path) ] +@pytest.mark.asyncio +async def test_mixed_source_action_artifact_round_trip_uses_unified_tables(tmp_path) -> None: + database = tmp_path / 'stash.sqlite' + store = ResultStore(database) + await store.initialize() + completed_at = datetime(2026, 8, 9, 12, 1, tzinfo=UTC) + result = CompletedResult.finish( + run_id=UUID('d721f4c5-1c76-4e7a-904a-23c5d6755834'), + target='example.com', + started_at=datetime(2026, 8, 9, 12, 0, tzinfo=UTC), + completed_at=completed_at, + groups={'hostname': ['api.example.com']}, + source_executions=(SourceExecution('shared-name', 'completed', 2.0, 1),), + observations=(ResultObservation('shared-name', 'hostname', 'api.example.com'),), + active_evidence=ActiveEvidence( + executions=( + ActionExecution.finish( + action='shared-name', + status='completed', + duration_ms=3.0, + groups={'ip': ['192.0.2.10']}, + ), + screenshot_execution(completed_at), + ActionExecution.finish( + action='takeover', + status='completed', + duration_ms=1.0, + groups={}, + ), + ) + ), + ) + + await store.save_run(result) + + assert await store.load_run(result.run_id) == result + assert [item.to_dict() for item in await store.action_yields(result.run_id)] == [ + { + 'action': 'screenshot', + 'observed_result_count': 0, + 'unique_result_count': 0, + 'shared_result_count': 0, + }, + { + 'action': 'shared-name', + 'observed_result_count': 1, + 'unique_result_count': 1, + 'shared_result_count': 0, + }, + { + 'action': 'takeover', + 'observed_result_count': 0, + 'unique_result_count': 0, + 'shared_result_count': 0, + }, + ] + with sqlite3.connect(database) as db: + tables = { + row[0] for row in db.execute("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'") + } + executions = db.execute('SELECT position, producer_kind, name, result_count FROM executions ORDER BY position').fetchall() + origins = db.execute( + 'SELECT e.producer_kind, e.name, r.kind, r.value ' + 'FROM result_origins AS o ' + 'JOIN executions AS e ON e.run_id = o.run_id AND e.position = o.execution_position ' + 'JOIN results AS r ON r.run_id = o.run_id AND r.position = o.result_position ' + 'ORDER BY e.producer_kind, e.name' + ).fetchall() + artifacts = db.execute( + 'SELECT e.name, r.kind, r.value, a.kind, a.path, a.media_type, a.size_bytes, a.sha256, a.created_at ' + 'FROM artifacts AS a ' + 'JOIN executions AS e ON e.run_id = a.run_id AND e.position = a.execution_position ' + 'JOIN results AS r ON r.run_id = a.run_id AND r.position = a.result_position' + ).fetchall() + assert tables == { + 'runs', + 'executions', + 'results', + 'result_origins', + 'artifacts', + 'legacy_observations', + 'run_records', + 'run_worker_leases', + } + assert executions == [ + (0, 'source', 'shared-name', 1), + (1, 'action', 'shared-name', 1), + (2, 'action', 'screenshot', 0), + (3, 'action', 'takeover', 0), + ] + assert origins == [ + ('action', 'shared-name', 'ip', '192.0.2.10'), + ('source', 'shared-name', 'hostname', 'api.example.com'), + ] + assert artifacts == [ + ( + 'screenshot', + 'hostname', + 'api.example.com', + 'screenshot', + 'screenshots/api.example.com.png', + 'image/png', + 3, + '0' * 64, + completed_at.isoformat(), + ) + ] + + with sqlite3.connect(database) as db: + db.execute('PRAGMA foreign_keys = ON') + db.execute('DELETE FROM runs WHERE run_id = ?', (str(result.run_id),)) + db.commit() + assert db.execute('SELECT COUNT(*) FROM executions').fetchone()[0] == 0 + assert db.execute('SELECT COUNT(*) FROM results').fetchone()[0] == 0 + assert db.execute('SELECT COUNT(*) FROM result_origins').fetchone()[0] == 0 + assert db.execute('SELECT COUNT(*) FROM artifacts').fetchone()[0] == 0 + + @pytest.mark.asyncio async def test_source_yields_distinguish_unique_and_shared_results(tmp_path) -> None: store = ResultStore(tmp_path / 'stash.sqlite') @@ -346,6 +584,265 @@ async def test_current_schema_reopens_without_running_legacy_migration(tmp_path) assert await reopened.load_run(existing.run_id) == existing +@pytest.mark.asyncio +async def test_schema_v2_upgrades_to_v7_without_rewriting_existing_rows(tmp_path) -> None: + database = tmp_path / 'stash.sqlite' + run_id = UUID('251d4047-190b-4a4d-9c4e-9eed3f23c8c7') + with sqlite3.connect(database) as db: + db.executescript(SCHEMA_V2_RUN_PROVENANCE) + db.execute( + 'INSERT INTO runs (run_id, target, started_at, completed_at) VALUES (?, ?, ?, ?)', + ( + str(run_id), + 'example.com', + '2026-08-09T12:00:00+00:00', + '2026-08-09T12:01:00+00:00', + ), + ) + db.execute( + 'INSERT INTO results (run_id, position, kind, value) VALUES (?, ?, ?, ?)', + (str(run_id), 0, 'hostname', 'api.example.com'), + ) + db.execute( + 'INSERT INTO executions ' + '(run_id, position, producer_kind, name, status, duration_ms, result_count) ' + 'VALUES (?, ?, ?, ?, ?, ?, ?)', + (str(run_id), 0, 'source', 'crtsh', 'completed', 12.5, 1), + ) + db.execute( + 'INSERT INTO result_origins (run_id, result_position, execution_position) VALUES (?, ?, ?)', + (str(run_id), 0, 0), + ) + + store = ResultStore(database) + await store.initialize() + await store.initialize() + + loaded = await store.load_run(run_id) + assert loaded == CompletedResult.finish( + run_id=run_id, + target='example.com', + started_at=datetime(2026, 8, 9, 12, 0, tzinfo=UTC), + completed_at=datetime(2026, 8, 9, 12, 1, tzinfo=UTC), + groups={'hostname': ['api.example.com']}, + source_executions=(SourceExecution('crtsh', 'completed', 12.5, 1),), + observations=(ResultObservation('crtsh', 'hostname', 'api.example.com'),), + ) + with sqlite3.connect(database) as db: + assert db.execute('PRAGMA user_version').fetchone()[0] == 7 + assert db.execute('SELECT COUNT(*) FROM artifacts').fetchone()[0] == 0 + + +@pytest.mark.asyncio +async def test_schema_v6_adds_nullable_evidence_status_without_rewriting_execution_status(tmp_path) -> None: + database = tmp_path / 'stash.sqlite' + run_id = UUID('cb34987b-9dd5-44f5-a58a-7ca7d34b0743') + with sqlite3.connect(database) as db: + db.executescript(SCHEMA_V6_RESULT_KINDS) + db.execute( + 'INSERT INTO runs (run_id, target, started_at, completed_at) VALUES (?, ?, ?, ?)', + ( + str(run_id), + 'example.com', + '2026-08-09T12:00:00+00:00', + '2026-08-09T12:01:00+00:00', + ), + ) + db.execute( + 'INSERT INTO executions ' + '(run_id, position, producer_kind, name, status, duration_ms, result_count) ' + 'VALUES (?, ?, ?, ?, ?, ?, ?)', + (str(run_id), 0, 'source', 'crtsh', 'partial', 12.5, 0), + ) + + store = ResultStore(database) + await store.initialize() + + loaded = await store.load_run(run_id) + with sqlite3.connect(database) as db: + run_columns = {row[1] for row in db.execute('PRAGMA table_info(runs)')} + stored_status = db.execute('SELECT evidence_status FROM runs WHERE run_id = ?', (str(run_id),)).fetchone()[0] + schema_version = db.execute('PRAGMA user_version').fetchone()[0] + assert loaded.status == 'partial' + assert stored_status is None + assert 'evidence_status' in run_columns + assert schema_version == 7 + + +@pytest.mark.asyncio +async def test_schema_v4_merges_deprecated_url_kinds_without_losing_origins(tmp_path) -> None: + database = tmp_path / 'stash.sqlite' + run_id = UUID('d299651b-21c1-4511-8cac-63ba70f926f4') + target_url = 'https://portal.example.com/login' + with sqlite3.connect(database) as db: + db.executescript(SCHEMA_V4_URL_KINDS) + db.execute( + 'INSERT INTO runs (run_id, target, started_at, completed_at) VALUES (?, ?, ?, ?)', + (str(run_id), 'example.com', '2026-08-09T12:00:00+00:00', '2026-08-09T12:01:00+00:00'), + ) + db.executemany( + 'INSERT INTO results (run_id, position, kind, value) VALUES (?, ?, ?, ?)', + [ + (str(run_id), 0, 'api-endpoint', target_url), + (str(run_id), 1, 'hostname', 'portal.example.com'), + (str(run_id), 2, 'interesting-url', target_url), + (str(run_id), 3, 'linkedin-link', target_url), + (str(run_id), 4, 'url', target_url), + ], + ) + db.executemany( + 'INSERT INTO executions ' + '(run_id, position, producer_kind, name, status, duration_ms, result_count, error_type, stop_reason) ' + 'VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)', + [ + (str(run_id), 0, 'source', 'builtwith', 'completed', 1.0, 1, None, None), + (str(run_id), 1, 'source', 'rocketreach', 'completed', 1.0, 1, None, None), + (str(run_id), 2, 'source', 'gitlab', 'completed', 1.0, 1, None, None), + (str(run_id), 3, 'action', 'api-scan', 'completed', 1.0, 3, None, None), + (str(run_id), 4, 'action', 'screenshot', 'completed', 1.0, 0, None, None), + ], + ) + db.executemany( + 'INSERT INTO result_origins (run_id, result_position, execution_position) VALUES (?, ?, ?)', + [ + (str(run_id), 2, 0), + (str(run_id), 3, 1), + (str(run_id), 4, 2), + (str(run_id), 0, 3), + (str(run_id), 2, 3), + (str(run_id), 4, 3), + ], + ) + db.execute( + 'INSERT INTO artifacts ' + '(run_id, position, result_position, execution_position, kind, path, media_type, size_bytes, sha256, created_at) ' + 'VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', + ( + str(run_id), + 0, + 2, + 4, + 'screenshot', + 'screenshots/portal.png', + 'image/png', + 3, + '0' * 64, + '2026-08-09T12:01:00+00:00', + ), + ) + db.executemany( + 'INSERT INTO legacy_observations (domain, resource, kind, discovered_on, source) VALUES (?, ?, ?, ?, ?)', + [ + ('example.com', target_url, 'interesting-url', '2026-08-09', 'builtwith'), + ('example.com', target_url, 'linkedinlinks', '2026-08-09', 'rocketreach'), + ], + ) + + store = ResultStore(database) + await store.initialize() + loaded = await store.load_run(run_id) + + assert loaded.results == (('hostname', 'portal.example.com'), ('url', target_url)) + assert {(item.source, item.kind, item.value) for item in loaded.observations} == { + ('builtwith', 'url', target_url), + ('gitlab', 'url', target_url), + ('rocketreach', 'url', target_url), + } + api_scan = next(item for item in loaded.active_evidence.executions if item.action == 'api-scan') + assert api_scan.result_count == 1 + assert {(item.kind, item.value) for item in api_scan.observations} == {('url', target_url)} + screenshot = next(item for item in loaded.active_evidence.executions if item.action == 'screenshot') + assert screenshot.artifacts[0].subject_kind == 'url' + assert screenshot.artifacts[0].subject_value == target_url + with sqlite3.connect(database) as db: + assert db.execute('PRAGMA user_version').fetchone()[0] == 7 + assert db.execute('SELECT DISTINCT kind FROM legacy_observations').fetchall() == [('url',)] + assert db.execute('SELECT result_count FROM executions WHERE name = ?', ('api-scan',)).fetchone()[0] == 1 + + +@pytest.mark.asyncio +async def test_schema_v5_merges_ip_address_into_ip_without_losing_provenance_or_artifacts(tmp_path) -> None: + database = tmp_path / 'stash.sqlite' + run_id = UUID('5b240ef2-e714-45de-b38f-174f20447f8b') + address = '192.0.2.1' + with sqlite3.connect(database) as db: + db.executescript(SCHEMA_V5_RESULT_KINDS) + db.execute( + 'INSERT INTO runs (run_id, target, started_at, completed_at) VALUES (?, ?, ?, ?)', + (str(run_id), 'example.com', '2026-08-09T12:00:00+00:00', '2026-08-09T12:01:00+00:00'), + ) + db.executemany( + 'INSERT INTO results (run_id, position, kind, value) VALUES (?, ?, ?, ?)', + [ + (str(run_id), 0, 'ip-address', address), + (str(run_id), 1, 'ip', address), + ], + ) + db.executemany( + 'INSERT INTO executions ' + '(run_id, position, producer_kind, name, status, duration_ms, result_count, error_type, stop_reason) ' + 'VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)', + [ + (str(run_id), 0, 'source', 'dns', 'completed', 1.0, 2, None, None), + (str(run_id), 1, 'action', 'screenshot', 'completed', 2.0, 2, None, None), + ], + ) + db.executemany( + 'INSERT INTO result_origins (run_id, result_position, execution_position) VALUES (?, ?, ?)', + [ + (str(run_id), 0, 0), + (str(run_id), 1, 0), + (str(run_id), 0, 1), + (str(run_id), 1, 1), + ], + ) + db.execute( + 'INSERT INTO artifacts ' + '(run_id, position, result_position, execution_position, kind, path, media_type, size_bytes, sha256, created_at) ' + 'VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', + ( + str(run_id), + 0, + 0, + 1, + 'screenshot', + 'screenshots/192.0.2.1.png', + 'image/png', + 3, + '0' * 64, + '2026-08-09T12:01:00+00:00', + ), + ) + db.executemany( + 'INSERT INTO legacy_observations (domain, resource, kind, discovered_on, source) VALUES (?, ?, ?, ?, ?)', + [ + ('example.com', address, 'ip-address', '2026-08-09', 'dns'), + ('example.com', address, 'ip', '2026-08-09', 'dns'), + ], + ) + + store = ResultStore(database) + await store.initialize() + loaded = await store.load_run(run_id) + + assert loaded.results == (('ip', address),) + assert [(item.source, item.kind, item.value) for item in loaded.observations] == [('dns', 'ip', address)] + assert loaded.source_executions[0].result_count == 1 + screenshot = loaded.active_evidence.executions[0] + assert [(item.kind, item.value) for item in screenshot.observations] == [('ip', address)] + assert screenshot.artifacts[0].subject_kind == 'ip' + assert screenshot.artifacts[0].subject_value == address + with sqlite3.connect(database) as db: + assert db.execute('PRAGMA user_version').fetchone()[0] == 7 + assert db.execute('SELECT kind, value FROM results').fetchall() == [('ip', address)] + assert db.execute('SELECT execution_position FROM result_origins ORDER BY execution_position').fetchall() == [ + (0,), + (1,), + ] + assert db.execute('SELECT result_count FROM executions ORDER BY position').fetchall() == [(1,), (1,)] + assert db.execute('SELECT DISTINCT kind FROM legacy_observations').fetchall() == [('ip',)] + + @pytest.mark.asyncio async def test_released_results_migrate_to_legacy_observations(tmp_path) -> None: database = tmp_path / 'stash.sqlite' @@ -376,15 +873,15 @@ async def test_released_results_migrate_to_legacy_observations(tmp_path) -> None assert result_columns == ['run_id', 'position', 'kind', 'value'] assert observations == [ ('api.example.com', 'hostname'), - ('192.0.2.1', 'ip-address'), + ('192.0.2.1', 'ip'), ('Ada Lovelace', 'person'), - ('https://linkedin.test/ada', 'linkedin-link'), - ('https://admin.example.com', 'interesting-url'), + ('https://linkedin.test/ada', 'url'), + ('https://admin.example.com', 'url'), ('AS64496', 'asn'), - ('/api/v1', 'api-endpoint'), + ('/api/v1', 'url'), ('admin@example.com', 'email'), ] - assert schema_version == 2 + assert schema_version == 7 @pytest.mark.asyncio @@ -456,6 +953,35 @@ async def test_completed_result_write_is_atomic_and_rejects_duplicate_run_id(tmp result_count = db.execute('SELECT COUNT(*) FROM results').fetchone()[0] assert (run_count, result_count) == (1, 7) + artifact_run_id = UUID('7ff120b6-4aec-4d27-b2db-d3ac9fd87340') + completed_at = datetime(2026, 8, 9, 12, 1, tzinfo=UTC) + failing_artifact = CompletedResult.finish( + run_id=artifact_run_id, + target='example.com', + started_at=datetime(2026, 8, 9, 12, 0, tzinfo=UTC), + completed_at=completed_at, + groups={'hostname': ['api.example.com']}, + active_evidence=ActiveEvidence(executions=(screenshot_execution(completed_at),)), + ) + with sqlite3.connect(database) as db: + db.execute( + f""" + CREATE TRIGGER fail_artifact + BEFORE INSERT ON artifacts + WHEN NEW.run_id = '{artifact_run_id}' + BEGIN + SELECT RAISE(ABORT, 'forced artifact failure'); + END + """ + ) + + with pytest.raises(ResultStoreError, match='Could not save enumeration run'): + await store.save_run(failing_artifact) + + with sqlite3.connect(database) as db: + assert db.execute('SELECT COUNT(*) FROM runs').fetchone()[0] == 1 + assert db.execute('SELECT COUNT(*) FROM artifacts').fetchone()[0] == 0 + @pytest.mark.asyncio async def test_legacy_observations_keep_the_released_normalized_schema(tmp_path) -> None: @@ -464,7 +990,7 @@ async def test_legacy_observations_keep_the_released_normalized_schema(tmp_path) await store.initialize() await store.record_observations('example.com', ['api.example.com', 'www.example.com'], 'hostname', 'crtsh') await store.record_observations('example.com', ['admin@example.com'], 'email', 'hunter') - await store.record_observations('example.com', ['192.0.2.1'], 'ip-address', 'dns') + await store.record_observations('example.com', ['192.0.2.1'], 'ip', 'dns') await store.record_observations('example.com', ['{"firstname":"Ada","lastname":"Lovelace"}'], 'person', 'hunter') await store.record_observations('example.com', ['vhost.example.com'], 'vhost', 'virtual-host') await store.record_observations('example.com', ['443'], 'shodan', 'shodan') @@ -478,7 +1004,7 @@ async def test_legacy_observations_keep_the_released_normalized_schema(tmp_path) ('example.com', 'api.example.com', 'hostname', 'crtsh'), ('example.com', 'www.example.com', 'hostname', 'crtsh'), ('example.com', 'admin@example.com', 'email', 'hunter'), - ('example.com', '192.0.2.1', 'ip-address', 'dns'), + ('example.com', '192.0.2.1', 'ip', 'dns'), ('example.com', '{"firstname":"Ada","lastname":"Lovelace"}', 'person', 'hunter'), ('example.com', 'vhost.example.com', 'vhost', 'virtual-host'), ('example.com', '443', 'shodan', 'shodan'), @@ -504,7 +1030,7 @@ async def test_schema_v1_observations_upgrade_without_losing_rows(tmp_path) -> N schema_version = db.execute('PRAGMA user_version').fetchone()[0] assert 'discovery_observations' not in tables assert rows == [('example.com', 'api.example.com', 'hostname', 'crtsh')] - assert schema_version == 2 + assert schema_version == 7 @pytest.mark.asyncio diff --git a/tests/lib/test_completed_result.py b/tests/lib/test_completed_result.py index f1481d22..3488bd6a 100644 --- a/tests/lib/test_completed_result.py +++ b/tests/lib/test_completed_result.py @@ -5,9 +5,24 @@ from uuid import UUID import pytest +from theHarvester.lib.active_evidence import ActionExecution, ActionObservation, ActiveEvidence, ArtifactReference from theHarvester.lib.completed_result import CompletedResult, ResultObservation, SourceExecution +@pytest.mark.parametrize('evidence_status', ['partial', 'failed']) +def test_sparse_completed_result_retains_explicit_status(evidence_status: str) -> None: + result = CompletedResult.finish( + target='example.com', + started_at=datetime(2026, 8, 5, 12, 0, tzinfo=UTC), + completed_at=datetime(2026, 8, 5, 12, 1, tzinfo=UTC), + groups={}, + evidence_status=evidence_status, + ) + + assert result.status == evidence_status + assert json.loads(result.jsonl().splitlines()[0])['evidence_status'] == evidence_status + + def test_completed_result_is_deterministic_and_deduplicated() -> None: result = CompletedResult.finish( run_id=UUID('f047261c-0afb-4e18-89d5-28a7d977f51f'), @@ -26,8 +41,12 @@ def test_completed_result_is_deterministic_and_deduplicated() -> None: { 'completed_at': '2026-08-05T12:01:00Z', 'counts': {'email': 1, 'hostname': 2}, + 'evidence_status': 'complete', 'result_count': 3, 'run_id': 'f047261c-0afb-4e18-89d5-28a7d977f51f', + 'source_executions': [], + 'action_executions': [], + 'artifacts': [], 'started_at': '2026-08-05T12:00:00Z', 'target': 'example.com', 'type': 'summary', @@ -167,29 +186,111 @@ def test_completed_result_rejects_source_count_without_matching_origins() -> Non ) -def test_completed_result_keeps_terminal_action_evidence_in_jsonl() -> None: +def test_completed_result_merges_active_results_and_keeps_screenshot_as_an_artifact() -> None: + completed_at = datetime(2026, 8, 5, 12, 1, tzinfo=UTC) + artifact = ArtifactReference( + kind='screenshot', + subject_kind='hostname', + subject_value='api.example.com', + path='screenshots/api.example.com.png', + media_type='image/png', + size_bytes=3, + sha256='0' * 64, + created_at=completed_at, + ) + result = CompletedResult.finish( + target='example.com', + started_at=completed_at, + completed_at=completed_at, + groups={'hostname': ['api.example.com']}, + active_evidence=ActiveEvidence( + executions=( + ActionExecution.finish( + action='dns-resolve', + status='completed', + duration_ms=12.5, + groups={'ip': ['192.0.2.10']}, + ), + ActionExecution.finish( + action='screenshot', + status='completed', + duration_ms=4.0, + groups={}, + artifacts=(artifact,), + ), + ) + ), + ) + + assert result.results == (('hostname', 'api.example.com'), ('ip', '192.0.2.10')) + assert result.active_evidence.executions[0].observations == (ActionObservation('ip', '192.0.2.10'),) + assert result.active_evidence.executions[1].artifacts == (artifact,) + assert not any(kind == 'screenshot' for kind, _value in result.results) + assert result.evidence_dict()['results'] == [ + {'type': 'hostname', 'value': 'api.example.com', 'sources': []}, + {'type': 'ip', 'value': '192.0.2.10', 'sources': [], 'actions': ['dns-resolve']}, + ] + assert [json.loads(line) for line in result.jsonl().splitlines()][1:] == [ + {'type': 'hostname', 'value': 'api.example.com', 'sources': []}, + {'type': 'ip', 'value': '192.0.2.10', 'sources': [], 'actions': ['dns-resolve']}, + ] + + +def test_completed_result_rejects_artifact_without_a_real_subject_result() -> None: + completed_at = datetime(2026, 8, 5, 12, 1, tzinfo=UTC) + artifact = ArtifactReference( + kind='screenshot', + subject_kind='hostname', + subject_value='missing.example.com', + path='screenshots/missing.example.com.png', + media_type='image/png', + size_bytes=3, + sha256='0' * 64, + created_at=completed_at, + ) + + with pytest.raises(ValueError, match='artifact must reference a completed result'): + CompletedResult.finish( + target='example.com', + started_at=completed_at, + completed_at=completed_at, + groups={}, + active_evidence=ActiveEvidence( + executions=( + ActionExecution.finish( + action='screenshot', + status='completed', + duration_ms=4.0, + groups={}, + artifacts=(artifact,), + ), + ) + ), + ) + + +def test_action_status_contributes_to_completed_result_status() -> None: completed_at = datetime(2026, 8, 5, 12, 1, tzinfo=UTC) result = CompletedResult.finish( target='example.com', started_at=completed_at, completed_at=completed_at, - groups={ - 'api-endpoint': ['/api/v1'], - 'screenshot': ['https://api.example.com'], - 'shodan': ['{"ip":"192.0.2.10","ports":[443]}'], - 'takeover': ['{"matches":[{"No such app":"Heroku"}],"url":"https://old.example.com"}'], - }, + groups={}, + source_executions=(SourceExecution('crtsh', 'completed', 1.0, 0),), + active_evidence=ActiveEvidence( + executions=( + ActionExecution.finish( + action='takeover', + status='failed', + duration_ms=2.0, + groups={}, + error_type='RuntimeError', + ), + ) + ), ) - records = [json.loads(line) for line in result.jsonl().splitlines()] - - assert records[0]['counts'] == {'api-endpoint': 1, 'screenshot': 1, 'shodan': 1, 'takeover': 1} - assert {(record['type'], record['value']) for record in records[1:]} == { - ('api-endpoint', '/api/v1'), - ('screenshot', 'https://api.example.com'), - ('shodan', '{"ip":"192.0.2.10","ports":[443]}'), - ('takeover', '{"matches":[{"No such app":"Heroku"}],"url":"https://old.example.com"}'), - } + assert result.evidence_dict()['status'] == 'partial' @pytest.mark.parametrize('value', ['', ' ', 7]) diff --git a/tests/lib/test_core.py b/tests/lib/test_core.py index 0b44920e..76c7e6d8 100644 --- a/tests/lib/test_core.py +++ b/tests/lib/test_core.py @@ -441,6 +441,7 @@ async def test_takeover_fetch_uses_the_shared_transport( 'url': url, 'proxy': proxy, 'request_timeout': 15, + 'include_metadata': False, } ] @@ -462,7 +463,25 @@ async def test_takeover_fetch_all_falls_back_to_direct_when_proxy_pool_is_empty( assert result == [('http://example.com', 'direct response')] assert len(calls) == 1 - assert calls[0][1] == {'proxy': None} + assert calls[0][1] == {'proxy': None, 'include_metadata': False} + + +@pytest.mark.asyncio +async def test_takeover_fetch_all_propagates_metadata_opt_in(monkeypatch) -> None: + reset_dummy_sessions() + seen: list[bool] = [] + monkeypatch.setattr(core_module.aiohttp, 'ClientSession', DummySession) + + async def fake_takeover_fetch(*_args, include_metadata: bool = False, **_kwargs): + seen.append(include_metadata) + return 'https://example.com', FetcherResponse(body='', status=204, headers={}) + + monkeypatch.setattr(AsyncFetcher, 'takeover_fetch', fake_takeover_fetch) + + result = await AsyncFetcher.fetch_all(['https://example.com'], takeover=True, include_metadata=True) + + assert seen == [True] + assert result[0][1].status == 204 @pytest.mark.asyncio diff --git a/tests/lib/test_hostchecker.py b/tests/lib/test_hostchecker.py index dcf087fb..03d7d188 100644 --- a/tests/lib/test_hostchecker.py +++ b/tests/lib/test_hostchecker.py @@ -75,6 +75,8 @@ async def test_dns_force_preserves_legacy_result_and_typed_records(monkeypatch: def __init__(self, _hosts: list[str], nameservers: list[str]) -> None: assert nameservers == ['192.0.2.53'] self.records = records + self.query_error_count = 2 + self.query_error_types = {'TimeoutError'} async def check(self) -> tuple[list[str], list[str], list[str]]: return ['www.example.com:192.0.2.10'], ['www.example.com'], ['192.0.2.10'] @@ -87,6 +89,15 @@ async def test_dns_force_preserves_legacy_result_and_typed_records(monkeypatch: assert result == (['www.example.com:192.0.2.10'], ['www.example.com'], ['192.0.2.10']) assert dns_force.records is records + assert dns_force.query_error_count == 2 + assert dns_force.query_error_types == {'TimeoutError'} + + +def test_dns_force_preserves_selected_www_target() -> None: + dns_force = dnssearch.DnsForce('www.example.com', ['192.0.2.53']) + + assert dns_force.domain == 'www.example.com' + assert all(candidate.endswith('.www.example.com') for candidate in dns_force.list) @pytest.mark.asyncio @@ -145,6 +156,87 @@ async def test_check_excludes_candidate_without_usable_evidence( assert checker.records == {} +@pytest.mark.asyncio +async def test_check_distinguishes_expected_absence_from_query_failures(monkeypatch: pytest.MonkeyPatch) -> None: + not_found = hostchecker.aiodns.error.DNSError(hostchecker.aiodns.error.ARES_ENOTFOUND, 'not found') + no_data = hostchecker.aiodns.error.DNSError(hostchecker.aiodns.error.ARES_ENODATA, 'no data') + + class FakeResolver: + async def query_dns(self, host: str, _record_type: str): + if host == 'missing.example.com': + raise not_found + if host == 'empty.example.com': + raise no_data + raise TimeoutError('resolver timed out') + + monkeypatch.setattr(hostchecker.aiodns, 'DNSResolver', lambda **_kwargs: FakeResolver()) + checker = hostchecker.Checker( + ['missing.example.com', 'empty.example.com', 'timeout.example.com'], + nameservers=[], + ) + + assert await checker.check() == ([], [], []) + assert checker.query_error_count == 3 + assert checker.query_error_types == {'TimeoutError'} + + +@pytest.mark.asyncio +async def test_dns_force_defaults_diagnostics_for_existing_checker_contract(monkeypatch: pytest.MonkeyPatch) -> None: + class ExistingChecker: + records: dict[str, hostchecker.HostDnsRecords] = {} + + def __init__(self, _hosts: list[str], nameservers: list[str]) -> None: + assert nameservers == [] + + async def check(self) -> tuple[list[str], list[str], list[str]]: + return [], [], [] + + monkeypatch.setattr(dnssearch.hostchecker, 'Checker', ExistingChecker) + dns_force = dnssearch.DnsForce('example.com', []) + dns_force.list = [] + + assert await dns_force.run() == ([], [], []) + assert dns_force.query_error_count == 0 + assert dns_force.query_error_types == set() + + +@pytest.mark.asyncio +async def test_reverse_single_ip_keeps_transport_failures_as_empty_results() -> None: + class FakeResolver: + async def gethostbyaddr(self, _ip: str): + raise TimeoutError('resolver timed out') + + assert await dnssearch.reverse_single_ip('192.0.2.10', FakeResolver()) == '' + + +@pytest.mark.asyncio +async def test_reverse_range_reports_only_unexpected_ptr_errors(monkeypatch: pytest.MonkeyPatch) -> None: + not_found = hostchecker.aiodns.error.DNSError(hostchecker.aiodns.error.ARES_ENOTFOUND, 'not found') + + class FakeResolver: + async def gethostbyaddr(self, ip: str): + if ip == '192.0.2.1': + return SimpleNamespace(name='api.example.com') + if ip == '192.0.2.2': + raise not_found + raise TimeoutError('resolver timed out') + + monkeypatch.setattr(dnssearch, 'list_ips_in_network_range', lambda _range: ['192.0.2.1', '192.0.2.2', '192.0.2.3']) + monkeypatch.setattr(dnssearch, 'DNSResolver', lambda **_kwargs: FakeResolver()) + monkeypatch.setattr(dnssearch, 'log_query', lambda _ip: None) + results: list[str] = [] + error_types: set[str] = set() + + await dnssearch.reverse_all_ips_in_range( + '192.0.2.0/24', + results.append, + error_types=error_types, + ) + + assert results == ['api.example.com', '', ''] + assert error_types == {'TimeoutError'} + + @pytest.mark.asyncio async def test_check_propagates_cancellation(monkeypatch: pytest.MonkeyPatch) -> None: class FakeResolver: diff --git a/tests/lib/test_output.py b/tests/lib/test_output.py index e94dd4ad..9122eaaa 100644 --- a/tests/lib/test_output.py +++ b/tests/lib/test_output.py @@ -1,39 +1,25 @@ from __future__ import annotations -from theHarvester.lib.output import configure_logging, print_linkedin_sections, sorted_unique +from theHarvester.lib.output import configure_logging, print_linkedin_people, sorted_unique def test_sorted_unique_sorts_and_deduplicates() -> None: - assert sorted_unique(["b", "a", "b"]) == ["a", "b"] + assert sorted_unique(['b', 'a', 'b']) == ['a', 'b'] -def test_print_linkedin_sections_prints_links_when_present(capsys) -> None: - # Regression coverage: the CLI previously never printed LinkedIn links when the list was non-empty. +def test_print_linkedin_people_reports_no_users(capsys) -> None: configure_logging(verbose=False) - print_linkedin_sections( - engines=["linkedin"], - people=[], - links=["https://b.example", "https://a.example", "https://a.example"], - ) + print_linkedin_people(engines=['linkedin'], people=[]) out = capsys.readouterr().out - assert "No LinkedIn users found" in out - assert "LinkedIn Links found: 3" in out - assert "https://a.example" in out - assert "https://b.example" in out + assert 'No LinkedIn users found' in out -def test_print_linkedin_sections_prints_people_and_links(capsys) -> None: +def test_print_linkedin_people_prints_people(capsys) -> None: configure_logging(verbose=False) - print_linkedin_sections( - engines=["rocketreach"], - people=["bob", "alice", "bob"], - links=["https://z.example", "https://z.example"], - ) + print_linkedin_people(engines=['rocketreach'], people=['bob', 'alice', 'bob']) out = capsys.readouterr().out - assert "LinkedIn Users found: 3" in out - assert "alice" in out - assert "bob" in out - assert "LinkedIn Links found: 2" in out - assert "https://z.example" in out + assert 'LinkedIn Users found: 3' in out + assert 'alice' in out + assert 'bob' in out diff --git a/tests/lib/test_run_backend.py b/tests/lib/test_run_backend.py new file mode 100644 index 00000000..0d9ac818 --- /dev/null +++ b/tests/lib/test_run_backend.py @@ -0,0 +1,984 @@ +from __future__ import annotations + +import asyncio +import inspect +import json +import os +import sqlite3 +import sys +import time +from datetime import UTC, datetime +from pathlib import Path +from uuid import UUID + +import pytest +from fastapi.testclient import TestClient + + +def test_run_paths_use_one_expanded_database_and_artifact_root(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api.run_store import RunStore + + monkeypatch.setenv('HOME', str(tmp_path)) + monkeypatch.setenv('THEHARVESTER_RUN_DB', '~/state/runs.sqlite') + monkeypatch.delenv('THEHARVESTER_RUN_ARTIFACTS', raising=False) + + store = RunStore() + + assert store.database == tmp_path / 'state' / 'runs.sqlite' + assert store.artifact_directory('run-id') == tmp_path / 'state' / 'run-artifacts' / 'run-id' + + +def test_run_store_does_not_change_caller_owned_directory_permissions(tmp_path) -> None: + from theHarvester.lib.api.run_store import RunStore + + tmp_path.chmod(0o755) + asyncio.run(RunStore(tmp_path / 'runs.sqlite').initialize()) + + assert os.stat(tmp_path).st_mode & 0o777 == 0o755 + + +def test_run_history_is_bounded_without_loading_completed_evidence(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + + async def fail_load(*_args, **_kwargs): + raise AssertionError('run summaries must not hydrate terminal evidence') + + async def scenario(): + store = RunStore(tmp_path / 'runs.sqlite') + for index in range(4): + await store.create(RunRequest(target=f'{index}.example.test', sources=['crtsh'])) + monkeypatch.setattr(store.results, 'load_run', fail_load) + return await store.list_runs(limit=2, offset=1) + + history = asyncio.run(scenario()) + + assert len(history) == 2 + assert [run['target'] for run in history] == ['2.example.test', '1.example.test'] + assert all(run['result_count'] == 0 for run in history) + + +def test_run_history_breaks_timestamp_ties_by_run_id(tmp_path) -> None: + from theHarvester.lib.database import RunLifecycleStore + + async def scenario() -> list[str]: + store = RunLifecycleStore(tmp_path / 'runs.sqlite') + await store.initialize() + for run_id in ('run-a', 'run-c', 'run-b'): + await store.create( + run_id=run_id, + target='example.test', + status='completed', + origin='imported', + created_at='2026-08-09T12:00:00+00:00', + request_json='{}', + ) + first = await store.list_records(limit=2, offset=0) + second = await store.list_records(limit=2, offset=2) + return [str(run['run_id']) for run in first + second] + + assert asyncio.run(scenario()) == ['run-c', 'run-b', 'run-a'] + + +def test_api_lifespan_disposes_shared_sqlite_engines(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + disposed = False + + async def no_op() -> None: + return None + + async def dispose() -> None: + nonlocal disposed + disposed = True + + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + monkeypatch.setattr(api, 'start_worker', no_op) + monkeypatch.setattr(api, 'stop_worker', no_op) + monkeypatch.setattr(api, 'dispose_sqlite_databases', dispose) + + with TestClient(api.app): + pass + + assert disposed is True + + +def test_static_assets_are_resolved_from_the_installed_module(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + monkeypatch.chdir(tmp_path) + + assert api.STATIC_DIRECTORY == Path(api.__file__).resolve().parent / 'static' + + +def test_explicit_run_database_keeps_screenshot_artifacts_attached(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api.run_store import RunStore + + monkeypatch.delenv('THEHARVESTER_RUN_DB', raising=False) + monkeypatch.delenv('THEHARVESTER_RUN_ARTIFACTS', raising=False) + + async def scenario(): + store = RunStore(tmp_path / 'state' / 'runs.sqlite') + imported = await store.import_evidence( + { + 'run_id': '4a6e5a15-fae5-462c-a34b-122ced6bb86d', + 'target': 'example.test', + 'status': 'complete', + 'started_at': '2026-08-09T12:00:00+00:00', + 'completed_at': '2026-08-09T12:01:00+00:00', + 'results': [{'type': 'hostname', 'value': 'owned.example.test', 'actions': []}], + 'source_executions': [], + 'action_executions': [ + { + 'action': 'screenshot', + 'status': 'completed', + 'duration_ms': 1, + 'result_count': 0, + 'error_type': None, + 'stop_reason': None, + } + ], + 'artifacts': [ + { + 'action': 'screenshot', + 'kind': 'screenshot', + 'subject': {'kind': 'hostname', 'value': 'owned.example.test'}, + 'file': { + 'path': 'screenshots/owned.example.test.png', + 'media_type': 'image/png', + 'size_bytes': 3, + 'sha256': '0' * 64, + }, + 'created_at': '2026-08-09T12:01:00+00:00', + } + ], + }, + 'evidence.jsonl', + ) + screenshot_dir = store.artifact_directory(imported['run_id']) / 'screenshots' + screenshot_dir.mkdir(parents=True) + (screenshot_dir / 'owned.example.test.png').write_bytes(b'png') + return await store.get(imported['run_id']) + + run = asyncio.run(scenario()) + + assert run is not None + assert run['evidence']['run_id'] == run['run_id'] + assert run['request']['source_run_id'] == '4a6e5a15-fae5-462c-a34b-122ced6bb86d' + assert [screenshot['name'] for screenshot in run['screenshots']] == ['owned.example.test.png'] + + +def test_api_lifecycle_and_terminal_evidence_share_the_sqlalchemy_database(tmp_path) -> None: + from theHarvester.lib.api import run_store as run_store_module + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + + database = tmp_path / 'stash.sqlite' + + async def scenario() -> str: + store = RunStore(database) + queued = await store.create(RunRequest(target='example.test', sources=['crtsh'])) + await store.claim_next() + await store.finish( + queued['run_id'], + { + 'run_id': '4a6e5a15-fae5-462c-a34b-122ced6bb86d', + 'target': 'example.test', + 'status': 'complete', + 'started_at': '2026-08-09T12:00:00+00:00', + 'completed_at': '2026-08-09T12:01:00+00:00', + 'results': [{'type': 'hostname', 'value': 'api.example.test', 'sources': ['crtsh']}], + 'source_executions': [ + { + 'source': 'crtsh', + 'status': 'completed', + 'duration_ms': 1, + 'result_count': 1, + 'error_type': None, + 'stop_reason': None, + } + ], + }, + '', + ) + return queued['run_id'] + + lifecycle_run_id = asyncio.run(scenario()) + + with sqlite3.connect(database) as db: + evidence_run_id = db.execute( + 'SELECT evidence_run_id FROM run_records WHERE run_id = ?', + (lifecycle_run_id,), + ).fetchone()[0] + stored_target = db.execute('SELECT target FROM runs WHERE run_id = ?', (evidence_run_id,)).fetchone()[0] + schema_version = db.execute('PRAGMA user_version').fetchone()[0] + assert evidence_run_id == lifecycle_run_id + assert stored_target == 'example.test' + assert schema_version == 7 + assert 'import aiosqlite' not in inspect.getsource(run_store_module) + + +def test_run_store_distinguishes_no_evidence_from_a_broken_evidence_link(tmp_path) -> None: + from uuid import uuid4 + + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + from theHarvester.lib.database import ResultStoreError + + database = tmp_path / 'runs.sqlite' + store = RunStore(database) + queued = asyncio.run(store.create(RunRequest(target='example.test', sources=['crtsh']))) + assert asyncio.run(store.load_completed_result(queued['run_id'])) is None + + with sqlite3.connect(database) as db: + db.execute( + 'UPDATE run_records SET evidence_run_id = ? WHERE run_id = ?', + (str(uuid4()), queued['run_id']), + ) + + with pytest.raises(ResultStoreError, match='Attached run evidence does not exist'): + asyncio.run(store.load_completed_result(queued['run_id'])) + + +def test_child_execution_passes_the_configured_database_to_core(tmp_path, monkeypatch) -> None: + from theHarvester import __main__ as main_module + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + from theHarvester.lib.api.run_worker import _child_execute + from theHarvester.lib.completed_result import CompletedResult + + database = tmp_path / 'state' / 'runs.sqlite' + seen_database = None + seen_run_id = None + + async def fake_start(_args, **kwargs): + nonlocal seen_database, seen_run_id + seen_database = kwargs.get('result_database') + seen_run_id = kwargs.get('completed_run_id') + now = datetime.now(UTC) + result = CompletedResult.finish( + run_id=seen_run_id, + target='example.test', + started_at=now, + completed_at=now, + groups={}, + ) + return (result,) + + async def scenario() -> None: + store = RunStore(database) + queued = await store.create(RunRequest(target='example.test', sources=['crtsh'])) + await store.claim_next() + monkeypatch.setattr(main_module, 'start', fake_start) + await _child_execute(queued['run_id'], database) + + asyncio.run(scenario()) + + assert seen_database == database + with sqlite3.connect(database) as db: + lifecycle_run_id = db.execute('SELECT run_id FROM run_records').fetchone()[0] + assert str(seen_run_id) == lifecycle_run_id + + +def test_child_screenshot_run_persists_downloadable_artifact_metadata(tmp_path, monkeypatch) -> None: + from theHarvester import __main__ as main_module + from theHarvester.lib.api.run_artifacts import read_child_evidence + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + from theHarvester.lib.api.run_worker import _child_execute + + class FakeScreenShotter: + slash = '/' + + def __init__(self, output: str) -> None: + self.output = output + + def verify_path(self) -> bool: + return True + + async def verify_installation(self) -> None: + return None + + async def visit(self, host: str) -> tuple[str, str]: + return f'https://{host}', 'reachable' + + @staticmethod + def chunk_list(values: list[str], _size: int) -> list[list[str]]: + return [values] + + def screenshot_path(self, url: str) -> Path: + return Path(self.output) / f'{url.removeprefix("https://")}.png' + + async def take_screenshot(self, url: str, *, output_path: Path | None = None) -> str: + captured_url = url if url.startswith('https://') else f'https://{url}' + (output_path or self.screenshot_path(captured_url)).write_bytes(b'png') + return captured_url + + class FakePool: + def __init__(self, _workers: int) -> None: + pass + + async def __aenter__(self): + return self + + async def __aexit__(self, *_args) -> None: + return None + + async def map(self, function, values): + return [await function(value) for value in values] + + database = tmp_path / 'state' / 'runs.sqlite' + monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts')) + monkeypatch.setattr(main_module, 'ScreenShotter', FakeScreenShotter) + monkeypatch.setattr(main_module, 'Pool', FakePool) + + async def scenario(): + store = RunStore(database) + queued = await store.create(RunRequest(target='api.example.test', sources=[], screenshot=True)) + await store.claim_next() + await _child_execute(queued['run_id'], database) + evidence, error = read_child_evidence(store.artifact_directory(queued['run_id'])) + assert error is None + assert evidence is not None + await store.finish(queued['run_id'], evidence, '') + return await store.get(queued['run_id']), store.artifact_directory(queued['run_id']) + + run, artifact_dir = asyncio.run(scenario()) + + assert run is not None + assert run['action_executions'][0]['action'] == 'screenshot' + assert run['action_executions'][0]['status'] == 'completed' + assert run['results'] == [{'type': 'hostname', 'value': 'api.example.test', 'sources': [], 'actions': []}] + assert [screenshot['name'] for screenshot in run['screenshots']] == ['api.example.test.png'] + assert (artifact_dir / 'screenshots' / 'api.example.test.png').read_bytes() == b'png' + + +def test_child_screenshot_cancellation_reuses_the_checkpointed_evidence(tmp_path, monkeypatch) -> None: + from theHarvester import __main__ as main_module + from theHarvester.lib.api.run_artifacts import read_child_evidence + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + from theHarvester.lib.api.run_worker import _child_execute + + first_captured = asyncio.Event() + + class FakeScreenShotter: + slash = '/' + + def __init__(self, output: str) -> None: + self.output = output + + def verify_path(self) -> bool: + return True + + async def verify_installation(self) -> None: + return None + + async def visit(self, host: str) -> tuple[str, str]: + return f'https://{host}', 'reachable' + + async def take_screenshot(self, url: str, *, output_path: Path | None = None) -> str: + assert output_path is not None + if 'first.' in url: + output_path.write_bytes(b'png') # noqa: ASYNC240 - tiny in-memory screenshot fixture + first_captured.set() + return url + await first_captured.wait() + raise asyncio.CancelledError + + def screenshot_path(self, url: str) -> Path: + return Path(self.output) / f'{url.removeprefix("https://")}.png' + + class TwoHostSource: + def __init__(self, _word: str) -> None: + pass + + async def process(self, _proxy: bool) -> None: + return None + + async def get_hostnames(self) -> set[str]: + return {'first.example.test', 'second.example.test'} + + class FakePool: + def __init__(self, _workers: int) -> None: + pass + + async def __aenter__(self): + return self + + async def __aexit__(self, *_args) -> None: + return None + + async def map(self, function, values): + return [await function(value) for value in values] + + database = tmp_path / 'runs.sqlite' + monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts')) + monkeypatch.setattr(main_module.crtsh, 'SearchCrtsh', TwoHostSource) + monkeypatch.setattr(main_module, 'ScreenShotter', FakeScreenShotter) + monkeypatch.setattr(main_module, 'Pool', FakePool) + + async def scenario(): + store = RunStore(database) + queued = await store.create(RunRequest(target='example.test', sources=['crtsh'], screenshot=True)) + await store.claim_next() + await _child_execute(queued['run_id'], database) + evidence, error = read_child_evidence(store.artifact_directory(queued['run_id'])) + assert error is None + assert evidence is not None + await store.fail(queued['run_id'], 'cancelled', '', cancelled=True, evidence=evidence) + return await store.get(queued['run_id']) + + run = asyncio.run(scenario()) + + assert run is not None + assert run['status'] == 'cancelled' + assert run['action_executions'][0]['status'] == 'partial' + assert [screenshot['target'] for screenshot in run['screenshots']] == ['first.example.test'] + + +def test_sqlite_import_preserves_run_ids_and_is_idempotent(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import run_store + from theHarvester.lib.api.run_store import RunStore + from theHarvester.lib.completed_result import CompletedResult + from theHarvester.lib.database import ResultStore, dispose_sqlite_databases + + source_database = tmp_path / 'source.sqlite' + destination_database = tmp_path / 'destination.sqlite' + now = datetime.now(UTC) + first = CompletedResult.finish( + target='first.example.test', + started_at=now, + completed_at=now, + groups={'hostname': ['api.first.example.test']}, + ) + second = CompletedResult.finish( + target='second.example.test', + started_at=now, + completed_at=now, + groups={'email': ['security@second.example.test']}, + ) + list_calls: list[tuple[int | None, int]] = [] + original_list_runs = ResultStore.list_runs + + async def track_source_batches(self, *, limit=50, offset=0): + if Path(self.database) == source_database.resolve(): + list_calls.append((limit, offset)) + return await original_list_runs(self, limit=limit, offset=offset) + + monkeypatch.setattr(run_store, 'DATABASE_IMPORT_BATCH_SIZE', 1) + monkeypatch.setattr(ResultStore, 'list_runs', track_source_batches) + + async def scenario(): + source = ResultStore(source_database) + await source.initialize() + await source.save_run(first) + await source.save_run(second) + await dispose_sqlite_databases() + destination = RunStore(destination_database) + imported = await destination.import_database(source_database, 'source.sqlite') + first_import_calls = list(list_calls) + list_calls.clear() + repeated = await destination.import_database(source_database, 'source.sqlite') + history = await destination.list_runs() + return imported, repeated, history, first_import_calls + + imported, repeated, history, first_import_calls = asyncio.run(scenario()) + + expected_ids = sorted((str(first.run_id), str(second.run_id))) + assert imported == {'filename': 'source.sqlite', 'imported_run_ids': expected_ids, 'skipped_run_ids': []} + assert repeated == {'filename': 'source.sqlite', 'imported_run_ids': [], 'skipped_run_ids': expected_ids} + assert sorted(run['run_id'] for run in history) == expected_ids + assert first_import_calls == [(1, 0), (1, 1), (1, 2), (1, 0), (1, 1), (1, 2)] + + +def test_orphan_recovery_reattaches_partial_checkpoint_and_leaves_queued_work(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api.run_artifacts import ensure_private_directory, write_child_evidence + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + from theHarvester.lib.completed_result import CompletedResult + + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts')) + + async def scenario(): + store = RunStore() + for target in ('first.example', 'second.example', 'third.example'): + await store.create(RunRequest(target=target, sources=['crtsh'])) + cancelling = await store.claim_next() + assert cancelling is not None + await store.cancel(cancelling['run_id']) + artifact_dir = store.artifact_directory(cancelling['run_id']) + ensure_private_directory(artifact_dir) + now = datetime.now(UTC) + checkpoint = CompletedResult.finish( + target='first.example', + started_at=now, + completed_at=now, + groups={'email': ['saved@first.example']}, + ) + write_child_evidence(artifact_dir, checkpoint, partial=True) + running = await store.claim_next() + assert running is not None + persisted = CompletedResult.finish( + run_id=UUID(running['run_id']), + target=str(running['target']), + started_at=now, + completed_at=now, + groups={'hostname': [f'api.{running["target"]}']}, + ) + await store.results.save_run(persisted) + await store.recover_orphans() + return await store.get(cancelling['run_id']), await store.get(running['run_id']), await store.list_runs() + + cancelling, running, history = asyncio.run(scenario()) + + assert cancelling is not None + assert cancelling['status'] == 'failed' + assert cancelling['evidence_status'] == 'partial' + assert cancelling['results'] == [{'type': 'email', 'value': 'saved@first.example', 'sources': [], 'actions': []}] + assert running is not None + assert running['status'] == 'failed' + assert running['results'] == [{'type': 'hostname', 'value': f'api.{running["target"]}', 'sources': [], 'actions': []}] + assert {run['target']: run['status'] for run in history}['third.example'] == 'queued' + + +def test_orphan_recovery_does_not_attach_same_id_evidence_for_another_target(tmp_path) -> None: + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + from theHarvester.lib.completed_result import CompletedResult + + async def scenario(): + store = RunStore(tmp_path / 'runs.sqlite') + queued = await store.create(RunRequest(target='expected.example', sources=['crtsh'])) + await store.claim_next() + now = datetime.now(UTC) + await store.results.save_run( + CompletedResult.finish( + run_id=UUID(queued['run_id']), + target='different.example', + started_at=now, + completed_at=now, + groups={'hostname': ['api.different.example']}, + ) + ) + await store.recover_orphans() + return await store.get(queued['run_id']) + + run = asyncio.run(scenario()) + + assert run is not None + assert run['status'] == 'failed' + assert run['results'] == [] + + +def test_orphan_recovery_rejects_checkpoint_for_another_target(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api.run_artifacts import ensure_private_directory, write_child_evidence + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + from theHarvester.lib.completed_result import CompletedResult + + monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts')) + + async def scenario(): + store = RunStore(tmp_path / 'runs.sqlite') + queued = await store.create(RunRequest(target='expected.example', sources=['crtsh'])) + await store.claim_next() + artifact_dir = store.artifact_directory(queued['run_id']) + ensure_private_directory(artifact_dir) + now = datetime.now(UTC) + write_child_evidence( + artifact_dir, + CompletedResult.finish( + target='different.example', + started_at=now, + completed_at=now, + groups={'hostname': ['api.different.example']}, + ), + partial=True, + ) + await store.recover_orphans() + return await store.get(queued['run_id']) + + run = asyncio.run(scenario()) + + assert run is not None + assert run['status'] == 'failed' + assert run['results'] == [] + assert 'does not match run target' in run['error'] + + +@pytest.mark.parametrize('operation', ['finish', 'fail']) +def test_terminal_run_rejects_evidence_for_another_target(tmp_path, operation) -> None: + from fastapi import HTTPException + + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + from theHarvester.lib.completed_result import CompletedResult + + async def scenario(): + store = RunStore(tmp_path / 'runs.sqlite') + queued = await store.create(RunRequest(target='expected.example', sources=['crtsh'])) + await store.claim_next() + now = datetime.now(UTC) + evidence = CompletedResult.finish( + target='different.example', + started_at=now, + completed_at=now, + groups={'hostname': ['api.different.example']}, + ).evidence_dict() + terminal = ( + store.finish(queued['run_id'], evidence, '') + if operation == 'finish' + else store.fail(queued['run_id'], 'failed', '', evidence=evidence) + ) + with pytest.raises(HTTPException, match='Evidence target does not match run target'): + await terminal + with pytest.raises(LookupError): + await store.results.load_run(UUID(queued['run_id'])) + + asyncio.run(scenario()) + + +def test_worker_lease_serializes_execution_owners(tmp_path) -> None: + from theHarvester.lib.api.run_store import RunStore + + async def scenario() -> tuple[bool, bool, bool]: + store = RunStore(tmp_path / 'runs.sqlite') + first = await store.acquire_worker_lease('worker-a') + second = await store.acquire_worker_lease('worker-b') + await store.release_worker_lease('worker-a') + replacement = await store.acquire_worker_lease('worker-b') + return first, second, replacement + + assert asyncio.run(scenario()) == (True, False, True) + + +def test_submission_fails_closed_when_worker_supervisor_stops(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api, run_worker + + async def no_op() -> None: + return None + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setattr(api, 'start_worker', no_op) + monkeypatch.setattr(api, 'stop_worker', no_op) + monkeypatch.setattr(run_worker, 'worker_enabled', lambda: True) + monkeypatch.setattr(run_worker, '_worker_task', type('StoppedTask', (), {'done': lambda self: True})()) + + with TestClient(api.app) as client: + response = client.post( + '/api/v1/runs', + headers={'X-API-Key': 'test-key'}, + json={'target': 'example.com', 'sources': ['crtsh']}, + ) + history = client.get('/api/v1/runs', headers={'X-API-Key': 'test-key'}) + + assert response.status_code == 503 + assert response.json()['detail'] == 'theHarvester execution worker is unavailable' + assert history.json() == [] + + +def test_authenticated_operator_can_queue_direct_activity_for_selected_target(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api, run_worker + + async def no_op() -> None: + return None + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setattr(api, 'start_worker', no_op) + monkeypatch.setattr(api, 'stop_worker', no_op) + monkeypatch.setattr(run_worker, 'worker_enabled', lambda: True) + monkeypatch.setattr(run_worker, '_worker_task', type('RunningTask', (), {'done': lambda self: False})()) + + with TestClient(api.app) as client: + response = client.post( + '/api/v1/runs', + headers={'X-API-Key': 'test-key'}, + json={'target': '192.0.2.8', 'sources': ['criminalip']}, + ) + + assert response.status_code == 201 + assert response.json()['target'] == '192.0.2.8' + assert response.json()['activities'] == ['P2'] + + +def test_authenticated_operator_can_queue_screenshot_only_run(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api, run_worker + + async def no_op() -> None: + return None + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setattr(api, 'start_worker', no_op) + monkeypatch.setattr(api, 'stop_worker', no_op) + monkeypatch.setattr(run_worker, 'worker_enabled', lambda: True) + monkeypatch.setattr(run_worker, '_worker_task', type('RunningTask', (), {'done': lambda self: False})()) + + with TestClient(api.app) as client: + response = client.post( + '/api/v1/runs', + headers={'X-API-Key': 'test-key'}, + json={'target': 'api.example.com', 'sources': [], 'screenshot': True}, + ) + + assert response.status_code == 201 + assert response.json()['target'] == 'api.example.com' + assert response.json()['sources'] == [] + assert response.json()['activities'] == ['P2'] + + +def test_dns_brute_run_accepts_operator_resolver_list(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api, run_worker + + async def no_op() -> None: + return None + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setattr(api, 'start_worker', no_op) + monkeypatch.setattr(api, 'stop_worker', no_op) + monkeypatch.setattr(run_worker, 'worker_enabled', lambda: True) + monkeypatch.setattr(run_worker, '_worker_task', type('RunningTask', (), {'done': lambda self: False})()) + + with TestClient(api.app) as client: + response = client.post( + '/api/v1/runs', + headers={'X-API-Key': 'test-key'}, + json={ + 'target': 'dev.api.example.com', + 'sources': [], + 'dns_brute': True, + 'dns_resolvers': ['192.0.2.53'], + }, + ) + + assert response.status_code == 201 + assert response.json()['activities'] == ['P1'] + assert response.json()['request']['dns_resolvers'] == ['192.0.2.53'] + + +def test_dns_brute_child_uses_operator_resolver_list(tmp_path, monkeypatch) -> None: + from theHarvester import __main__ as main_module + from theHarvester.lib.api import run_worker + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + from theHarvester.lib.completed_result import CompletedResult + + received_options = [] + + async def fake_start(options, **_kwargs): + received_options.append(options) + now = datetime.now(UTC) + return (CompletedResult.finish(target=options.domain, started_at=now, completed_at=now, groups={}),) + + monkeypatch.setattr(main_module, 'start', fake_start) + + async def scenario() -> None: + store = RunStore(tmp_path / 'runs.sqlite') + created = await store.create( + RunRequest( + target='dev.api.example.com', + sources=[], + dns_brute=True, + dns_resolvers=['192.0.2.53'], + ) + ) + assert await store.claim_next() is not None + await run_worker._child_execute(created['run_id'], store.database) + + asyncio.run(scenario()) + + assert received_options[0].source == '' + assert received_options[0].dns_brute is True + assert received_options[0].dns_resolve == '' + assert received_options[0].dns_resolvers == ('192.0.2.53',) + + +def test_api_scan_child_uses_operator_endpoint_paths(tmp_path, monkeypatch) -> None: + from theHarvester import __main__ as main_module + from theHarvester.lib.api import run_worker + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + from theHarvester.lib.completed_result import CompletedResult + + received_wordlists: list[Path] = [] + + async def fake_start(options, **_kwargs): + received_wordlists.append(Path(options.wordlist)) + now = datetime.now(UTC) + return (CompletedResult.finish(target=options.domain, started_at=now, completed_at=now, groups={}),) + + monkeypatch.setattr(main_module, 'start', fake_start) + + async def scenario() -> None: + store = RunStore(tmp_path / 'runs.sqlite') + created = await store.create( + RunRequest( + target='api.example.test', + sources=[], + api_scan=True, + api_scan_paths=['/api/v2', '/health'], + ) + ) + assert await store.claim_next() is not None + await run_worker._child_execute(created['run_id'], store.database) + + asyncio.run(scenario()) + + assert received_wordlists[0].read_text(encoding='utf-8') == '/api/v2\n/health\n' + + +def test_running_cancellation_terminates_child_and_retains_partial_evidence(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api, run_worker + + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'enabled') + + async def slow_process(_run_id, _database, artifact_dir): + artifact_dir.mkdir(parents=True, exist_ok=True) + (artifact_dir / 'evidence.json').write_text( + json.dumps( + { + 'target': 'example.test', + 'status': 'partial', + 'results': [{'type': 'email', 'value': 'saved@example.test'}], + } + ), + encoding='utf-8', + ) + return await asyncio.create_subprocess_exec( + sys.executable, + '-c', + 'import time; time.sleep(60)', + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + + monkeypatch.setattr(run_worker, '_process_factory', slow_process) + headers = {'X-API-Key': 'test-key'} + with TestClient(api.app) as client: + submitted = client.post( + '/api/v1/runs', + headers=headers, + json={'target': 'example.test', 'sources': ['crtsh'], 'deadline_seconds': 60}, + ).json() + deadline = time.monotonic() + 3 + while time.monotonic() < deadline: + detail = client.get(f'/api/v1/runs/{submitted["run_id"]}', headers=headers).json() + if detail['status'] == 'running': + break + time.sleep(0.02) + requested = client.post(f'/api/v1/runs/{submitted["run_id"]}/cancel', headers=headers) + while time.monotonic() < deadline: + detail = client.get(f'/api/v1/runs/{submitted["run_id"]}', headers=headers).json() + if detail['status'] == 'cancelled': + break + time.sleep(0.02) + + assert requested.json()['status'] == 'cancelling' + assert detail['status'] == 'cancelled' + assert detail['completed_at'] is not None + assert detail['evidence_status'] == 'partial' + assert detail['results'] == [{'type': 'email', 'value': 'saved@example.test', 'sources': [], 'actions': []}] + + +def test_whole_run_deadline_terminates_child_and_retains_partial_evidence(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import run_worker + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts')) + monkeypatch.setattr(run_worker, '_worker_stop', None) + + async def slow_process(_run_id, _database, artifact_dir): + artifact_dir.mkdir(parents=True, exist_ok=True) + (artifact_dir / 'evidence.json').write_text( + json.dumps( + { + 'target': 'example.test', + 'status': 'partial', + 'results': [{'type': 'email', 'value': 'saved@example.test'}], + } + ), + encoding='utf-8', + ) + return await asyncio.create_subprocess_exec( + sys.executable, + '-c', + 'import time; time.sleep(60)', + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + + monkeypatch.setattr(run_worker, '_process_factory', slow_process) + + async def scenario(): + store = RunStore() + await store.create(RunRequest(target='example.test', sources=['crtsh'])) + run = await store.claim_next() + assert run is not None + run['request']['deadline_seconds'] = 0 + await run_worker._execute_claimed(store, run) + return await store.get(run['run_id']) + + detail = asyncio.run(scenario()) + + assert detail is not None + assert detail['status'] == 'failed' + assert 'deadline' in detail['error'] + assert detail['evidence_status'] == 'partial' + assert detail['results'] == [{'type': 'email', 'value': 'saved@example.test', 'sources': [], 'actions': []}] + + +def test_worker_fails_run_without_attaching_child_evidence_for_another_target(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import run_worker + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + + monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts')) + monkeypatch.setattr(run_worker, '_worker_stop', None) + + async def mismatched_process(_run_id, _database, artifact_dir): + artifact_dir.mkdir(parents=True, exist_ok=True) + (artifact_dir / 'evidence.json').write_text( + json.dumps( + { + 'target': 'different.example', + 'status': 'complete', + 'results': [{'type': 'email', 'value': 'saved@different.example'}], + } + ), + encoding='utf-8', + ) + return await asyncio.create_subprocess_exec( + sys.executable, + '-c', + 'pass', + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + + monkeypatch.setattr(run_worker, '_process_factory', mismatched_process) + + async def scenario(): + store = RunStore(tmp_path / 'runs.sqlite') + await store.create(RunRequest(target='expected.example', sources=['crtsh'])) + run = await store.claim_next() + assert run is not None + await run_worker._execute_claimed(store, run) + return await store.get(run['run_id']) + + detail = asyncio.run(scenario()) + + assert detail is not None + assert detail['status'] == 'failed' + assert detail['results'] == [] + assert 'does not match run target' in detail['error'] diff --git a/tests/lib/test_source_catalog.py b/tests/lib/test_source_catalog.py index a913f1e9..124a0e41 100644 --- a/tests/lib/test_source_catalog.py +++ b/tests/lib/test_source_catalog.py @@ -1,6 +1,7 @@ import ast from pathlib import Path +from theHarvester.discovery import bevigil, builtwith, gitlabsearch, intelxsearch, rocketreach, urlscan, zoomeyesearch from theHarvester.lib.core import Core from theHarvester.lib.source_catalog import SOURCE_SPECS, ResultRoute, SourceSpec, get_source_spec @@ -42,13 +43,7 @@ def test_invalid_bitbucket_domain_source_is_not_selectable() -> None: def test_subdomain_route_drives_subdomain_capability() -> None: spec = SourceSpec( name='example', - routes=frozenset( - { - ResultRoute.SUBDOMAINS, - ResultRoute.LINKS, - ResultRoute.INTERESTING_URLS, - } - ), + routes=frozenset({ResultRoute.SUBDOMAINS, ResultRoute.URLS}), ) assert spec.capabilities == frozenset({'subdomains', 'urls'}) @@ -64,11 +59,43 @@ def test_source_specs_describe_consolidated_routes_not_getter_presence() -> None ResultRoute.SUBDOMAINS, ResultRoute.IPS, ResultRoute.ASNS, - ResultRoute.INTERESTING_URLS, + ResultRoute.URLS, } ) +def test_every_url_source_uses_one_route() -> None: + url_sources = {'bevigil', 'builtwith', 'gitlab', 'intelx', 'rocketreach', 'urlscan', 'zoomeye'} + + assert {spec.name for spec in SOURCE_SPECS.values() if ResultRoute.URLS in spec.routes} == url_sources + assert {route.name for route in ResultRoute} == { + 'SUBDOMAINS', + 'EMAILS', + 'IPS', + 'ASNS', + 'PEOPLE', + 'URLS', + 'BREACHES', + } + + +def test_every_url_adapter_uses_one_getter() -> None: + adapters = ( + bevigil.SearchBeVigil, + builtwith.SearchBuiltWith, + gitlabsearch.SearchGitlab, + intelxsearch.SearchIntelx, + rocketreach.SearchRocketReach, + urlscan.SearchUrlscan, + zoomeyesearch.SearchZoomEye, + ) + + assert all(hasattr(adapter, 'get_urls') for adapter in adapters) + assert not any(hasattr(adapter, 'get_links') for adapter in adapters) + assert not any(hasattr(adapter, 'get_interestingurls') for adapter in adapters) + assert not any(hasattr(adapter, 'get_interesting_urls') for adapter in adapters) + + def test_rapiddns_declares_separate_subdomain_and_ip_routes() -> None: assert SOURCE_SPECS['rapiddns'].routes == frozenset({ResultRoute.SUBDOMAINS, ResultRoute.IPS}) diff --git a/tests/test_all_source_orchestration.py b/tests/test_all_source_orchestration.py index d6978079..15738916 100644 --- a/tests/test_all_source_orchestration.py +++ b/tests/test_all_source_orchestration.py @@ -13,7 +13,6 @@ import pytest from theHarvester import __main__ as theharvester_main from theHarvester.lib.source_catalog import SOURCE_SPECS, ActivityClass - NON_PASSIVE_SOURCES = ( 'criminalip', 'pentesttools', @@ -234,18 +233,9 @@ async def test_all_schedules_each_passive_catalog_source_once_and_reports_result async def get_people(self) -> list[dict[str, str]]: return [{'name': 'Example Person'}] - async def get_links(self) -> set[str]: - return {'https://sub.example.test/profile'} - async def get_urls(self) -> set[str]: - return {'https://gitlab.com/example/project'} - - async def get_interestingurls(self) -> set[str]: return {'https://sub.example.test/evidence'} - async def get_interesting_urls(self) -> set[str]: - return await self.get_interestingurls() - async def get_host_ip_pairs(self) -> set[tuple[str, str]]: return set() @@ -296,18 +286,28 @@ async def test_all_schedules_each_passive_catalog_source_once_and_reports_result json_report = json.loads(report.with_suffix('.json').read_text()) assert 'sub.example.test' in json_report['hosts'] assert 'user@example.test' in json_report['emails'] + assert json_report['urls'] == ['https://sub.example.test/evidence'] + assert not {'interesting_urls', 'linkedin_links', 'trello_urls'} & json_report.keys() jsonl_records = [json.loads(line) for line in report.with_suffix('.jsonl').read_text().splitlines()] findings = {(record['type'], record['value']): record for record in jsonl_records[1:]} assert findings[('hostname', 'sub.example.test')]['sources'] assert findings[('email', 'user@example.test')]['sources'] - assert findings[('url', 'https://gitlab.com/example/project')]['sources'] + assert findings[('url', 'https://sub.example.test/evidence')]['sources'] == [ + 'bevigil', + 'builtwith', + 'gitlab', + 'intelx', + 'rocketreach', + 'urlscan', + 'zoomeye', + ] completed = await TestResultStore().load_run(UUID(jsonl_records[0]['run_id'])) assert completed.target == 'example.test' assert ('hostname', 'sub.example.test') in completed.results assert ('email', 'user@example.test') in completed.results - assert ('ip-address', '192.0.2.1') in completed.results + assert ('ip', '192.0.2.1') in completed.results xml_hosts = { (element.findtext('hostname') or (element.text or '').strip()) diff --git a/tests/test_logging.py b/tests/test_logging.py index 0574a9f2..27613040 100644 --- a/tests/test_logging.py +++ b/tests/test_logging.py @@ -54,26 +54,6 @@ def test_operator_output_uses_stdout_without_verbose_logging() -> None: assert result.stderr == '' -def test_api_example_entry_point_configures_output_and_diagnostics() -> None: - result = run_python( - """ - import logging - from theHarvester.lib.api import api_example - from theHarvester.lib.output import output_logger - - async def fake_main(): - output_logger.info('example result') - logging.getLogger(api_example.__name__).info('example diagnostic') - - api_example.main = fake_main - api_example.entry_point() - """ - ) - - assert result.stdout == 'example result\n' - assert 'INFO theHarvester.lib.api.api_example: example diagnostic' in result.stderr - - def test_diagnostics_use_stderr_only_when_verbose() -> None: result = run_python( """ @@ -132,41 +112,6 @@ def test_verbose_logging_does_not_overwrite_a_later_host_level() -> None: assert result.stdout == f'{logging.ERROR}\n' -def test_rest_errors_are_visible_with_uvicorn_logging() -> None: - result = run_python( - """ - import logging.config - import sys - from unittest.mock import AsyncMock, patch - - from fastapi.testclient import TestClient - from uvicorn.config import LOGGING_CONFIG - - logging.config.dictConfig(LOGGING_CONFIG) - - from theHarvester.lib.api import api - - client = TestClient(api.app) - statuses = [] - with patch.object(api.__main__.Core, 'get_supportedengines', side_effect=RuntimeError('sources failure')): - statuses.append(client.get('/sources').status_code) - with patch.object(api.__main__, 'start', AsyncMock(side_effect=RuntimeError('dnsbrute failure'))): - statuses.append(client.get('/dnsbrute?domain=example.com').status_code) - with ( - patch.object(api.__main__.Core, 'get_supportedengines', return_value=['baidu']), - patch.object(api.__main__, 'start', AsyncMock(side_effect=RuntimeError('query failure'))), - ): - statuses.append(client.get('/query?domain=example.com&source=baidu').status_code) - sys.stdout.write(repr(statuses) + '\\n') - """ - ) - - assert result.stdout == '[500, 500, 500]\n' - assert 'Error in getsources endpoint' in result.stderr - assert 'Error in dnsbrute endpoint' in result.stderr - assert 'Error in query endpoint' in result.stderr - - def test_verbose_enables_info_diagnostics(tmp_path: Path) -> None: script = """import asyncio import logging diff --git a/tests/test_main.py b/tests/test_main.py index 01c214b3..e429d8b5 100644 --- a/tests/test_main.py +++ b/tests/test_main.py @@ -1,5 +1,6 @@ import asyncio import json +import logging import sys import xml.etree.ElementTree as ElementTree from pathlib import Path @@ -7,6 +8,7 @@ from pathlib import Path import pytest from theHarvester import __main__ as theharvester_main +from theHarvester.discovery.constants import MissingKey from theHarvester.lib.completed_result import CompletedResult, ResultObservation from theHarvester.lib.dns_consensus import Addressability from theHarvester.lib.enumeration import EnumerationOptions @@ -26,7 +28,9 @@ async def test_cli_help_explains_proxy_and_direct_action_scope( help_text = ' '.join(capsys.readouterr().out.split()) assert exit_info.value.code == 0 assert 'Use proxies.yaml for supported discovery-source and takeover requests.' in help_text - assert 'Accepted for compatibility but currently unused; use --dns-resolve to select resolvers.' in help_text + assert 'Accepted for compatibility but currently unused; use --dns-resolvers to select resolvers.' in help_text + assert 'Select resolver IPs for DNS actions without enabling hostname resolution.' in help_text + assert 'text file with one IP per line' in help_text assert 'Perform PTR lookups across the /24 network containing each discovered IPv4 address.' in help_text assert 'Multiple capabilities select the union of matching sources; they do not filter returned fields.' in help_text assert 'Check common API paths with GET, HEAD, and OPTIONS.' in help_text @@ -96,6 +100,8 @@ async def test_rapiddns_hostnames_honor_explicit_dns_resolution(monkeypatch: pyt def __init__(self, hosts: list[str], nameservers: list[str]) -> None: assert nameservers == ['192.0.2.53'] self.hosts = hosts + self.query_error_count = 1 + self.query_error_types = {'TimeoutError'} async def check(self) -> tuple[list[str], list[str], list[str]]: if self.hosts == ['crt.example.com']: @@ -135,20 +141,31 @@ async def test_rapiddns_hostnames_honor_explicit_dns_resolution(monkeypatch: pyt assert ('hostname', 'api.example.com') in completed[0].results assert ('hostname', 'crt.example.com') in completed[0].results assert ('hostname', 'reported.example.com') in completed[0].results - assert ('ip-address', '192.0.2.10') in completed[0].results - assert ('ip-address', '192.0.2.20') in completed[0].results - assert ('ip-address', '192.0.2.21') in completed[0].results - assert ('ip-address', '192.0.2.30') in completed[0].results + assert ('ip', '192.0.2.10') in completed[0].results + assert ('ip', '192.0.2.20') in completed[0].results + assert ('ip', '192.0.2.21') in completed[0].results + assert ('ip', '192.0.2.30') in completed[0].results assert {execution.source for execution in completed[0].source_executions} == {'crtsh', 'rapiddns'} crtsh_execution = next(execution for execution in completed[0].source_executions if execution.source == 'crtsh') assert crtsh_execution.status == 'partial' assert crtsh_execution.stop_reason == 'invalid-response' + dns_execution = completed[0].active_evidence.executions[0] + assert dns_execution.action == 'dns-resolve' + assert dns_execution.status == 'partial' + assert dns_execution.error_type == 'TimeoutError' + assert dns_execution.stop_reason == 'query-errors' + assert {(observation.kind, observation.value) for observation in dns_execution.observations} == { + ('ip', '192.0.2.10'), + ('ip', '192.0.2.21'), + ('ip', '192.0.2.30'), + } + assert ('ip', '192.0.2.20') not in {(observation.kind, observation.value) for observation in dns_execution.observations} assert completed[0].evidence_dict()['status'] == 'partial' assert {(observation.source, observation.kind, observation.value) for observation in completed[0].observations} >= { ('crtsh', 'hostname', 'crt.example.com'), ('rapiddns', 'hostname', 'api.example.com'), ('rapiddns', 'hostname', 'reported.example.com'), - ('rapiddns', 'ip-address', '192.0.2.20'), + ('rapiddns', 'ip', '192.0.2.20'), } assert 'reported.example.com:192.0.2.21' in json.loads(output_path.with_suffix('.json').read_text())['hosts'] assert output_path.with_suffix('.jsonl').is_file() @@ -160,6 +177,518 @@ async def test_rapiddns_hostnames_honor_explicit_dns_resolution(monkeypatch: pyt assert xml_pairs.count(('reported.example.com', '192.0.2.21')) == 1 +@pytest.mark.asyncio +async def test_dns_brute_utility_persists_action_evidence_before_return(monkeypatch: pytest.MonkeyPatch) -> None: + completed: list[CompletedResult] = [] + legacy_writes: list[tuple[object, ...]] = [] + resolved = ['api.example.com:192.0.2.10'] + + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def record_observations(self, *args: object) -> None: + legacy_writes.append(args) + + async def save_run(self, result: CompletedResult) -> None: + completed.append(result) + + class FakeDnsForce: + query_error_count = 1 + query_error_types = {'TimeoutError'} + + def __init__(self, domain: str, nameservers: list[str], verbose: bool) -> None: + assert domain == 'example.com' + assert nameservers == [] + assert verbose is True + + async def run(self) -> tuple[list[str], list[str], list[str]]: + return resolved, ['API.Example.COM.'], ['192.0.2.10', 'not-an-ip'] + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + monkeypatch.setattr(theharvester_main.dnssearch, 'DnsForce', FakeDnsForce) + + response = await theharvester_main.start( + EnumerationOptions(domain='example.com', source='', dns_brute=True, quiet=True), + return_dns_brute_result=True, + ) + + assert response == resolved + assert len(completed) == 1 + execution = completed[0].active_evidence.executions[0] + assert execution.action == 'dns-brute' + assert execution.status == 'partial' + assert execution.error_type == 'TimeoutError' + assert execution.stop_reason == 'query-errors' + assert {(observation.kind, observation.value) for observation in execution.observations} == { + ('hostname', 'api.example.com'), + ('ip', '192.0.2.10'), + } + assert legacy_writes == [] + + +@pytest.mark.asyncio +async def test_dns_brute_query_errors_are_partial_even_without_findings(monkeypatch: pytest.MonkeyPatch) -> None: + completed: list[CompletedResult] = [] + + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def save_run(self, result: CompletedResult) -> None: + completed.append(result) + + class EmptyDnsForce: + query_error_count = 1 + query_error_types = {'TimeoutError'} + + def __init__(self, *_args, **_kwargs) -> None: + pass + + async def run(self) -> tuple[list[str], list[str], list[str]]: + return [], [], [] + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + monkeypatch.setattr(theharvester_main.dnssearch, 'DnsForce', EmptyDnsForce) + + assert ( + await theharvester_main.start( + EnumerationOptions(domain='example.com', source='', dns_brute=True, quiet=True), + return_dns_brute_result=True, + ) + == [] + ) + + execution = completed[0].active_evidence.executions[0] + assert execution.status == 'partial' + assert execution.result_count == 0 + assert execution.error_type == 'TimeoutError' + assert execution.stop_reason == 'query-errors' + + +@pytest.mark.asyncio +async def test_dns_brute_keeps_legacy_json_and_xml_while_persisting_canonical_evidence( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + stored: list[CompletedResult] = [] + output_path = tmp_path / 'dns-brute' + + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def save_run(self, result: CompletedResult) -> None: + stored.append(result) + + class FakeDnsForce: + query_error_count = 0 + query_error_types: set[str] = set() + + def __init__(self, *_args, **_kwargs) -> None: + pass + + async def run(self) -> tuple[list[str], list[str], list[str]]: + return ['api.example.com:192.0.2.10'], ['api.example.com'], ['192.0.2.10'] + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + monkeypatch.setattr(theharvester_main.dnssearch, 'DnsForce', FakeDnsForce) + + response = await theharvester_main.start( + EnumerationOptions( + domain='example.com', + source='', + dns_brute=True, + filename=str(output_path), + quiet=True, + ), + return_completed_result=True, + ) + + legacy_json = json.loads(output_path.with_suffix('.json').read_text()) + assert legacy_json['hosts'] == ['api.example.com:192.0.2.10'] + assert 'ips' not in legacy_json + assert response[6] == [] + xml_pairs = [ + (element.findtext('hostname'), element.findtext('ip')) + for element in ElementTree.parse(output_path.with_suffix('.xml')).getroot().findall('host') + ] + assert xml_pairs == [('api.example.com', '192.0.2.10')] + completed = response[-1] + assert isinstance(completed, CompletedResult) + assert ('hostname', 'api.example.com') in completed.results + assert ('ip', '192.0.2.10') in completed.results + assert stored == [completed] + + +@pytest.mark.parametrize('error_type', [RuntimeError, asyncio.CancelledError]) +@pytest.mark.asyncio +async def test_dns_brute_failure_persists_before_propagation( + monkeypatch: pytest.MonkeyPatch, + error_type: type[BaseException], +) -> None: + completed: list[CompletedResult] = [] + + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def save_run(self, result: CompletedResult) -> None: + completed.append(result) + + class FailingDnsForce: + def __init__(self, *_args, **_kwargs) -> None: + pass + + async def run(self): + raise error_type() + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + monkeypatch.setattr(theharvester_main.dnssearch, 'DnsForce', FailingDnsForce) + + with pytest.raises(error_type): + await theharvester_main.start( + EnumerationOptions(domain='example.com', source='', dns_brute=True, quiet=True), + return_dns_brute_result=True, + ) + + assert len(completed) == 1 + execution = completed[0].active_evidence.executions[0] + assert execution.action == 'dns-brute' + assert execution.status == 'failed' + assert execution.error_type == error_type.__name__ + assert execution.stop_reason == ('cancelled' if issubclass(error_type, asyncio.CancelledError) else None) + + +@pytest.mark.asyncio +async def test_dns_resolve_cancellation_closes_workers_and_persists_before_propagation( + monkeypatch: pytest.MonkeyPatch, +) -> None: + completed: list[CompletedResult] = [] + + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def save_run(self, result: CompletedResult) -> None: + completed.append(result) + + class FakeSource: + def __init__(self, _word: str) -> None: + pass + + async def process(self, _proxy: bool) -> None: + return None + + async def get_hostnames(self) -> set[str]: + return {'api.example.com'} + + class CancelledChecker: + def __init__(self, _hosts: list[str], _nameservers: list[str]) -> None: + pass + + async def check(self): + raise asyncio.CancelledError + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + monkeypatch.setattr(theharvester_main.certspottersearch, 'SearchCertspoter', FakeSource) + monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', FakeSource) + monkeypatch.setattr(theharvester_main.shodanct, 'SearchShodanCt', FakeSource) + monkeypatch.setattr(theharvester_main.subdomaincenter, 'SubdomainCenter', FakeSource) + monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', CancelledChecker) + + with pytest.raises(asyncio.CancelledError): + await asyncio.wait_for( + theharvester_main.start( + EnumerationOptions( + domain='example.com', + source='certspotter,crtsh,shodanct,subdomaincenter', + dns_resolve='192.0.2.53', + quiet=True, + ) + ), + timeout=1, + ) + + assert len(completed) == 1 + execution = completed[0].active_evidence.executions[0] + assert execution.action == 'dns-resolve' + assert execution.status == 'failed' + assert execution.error_type == 'CancelledError' + assert execution.stop_reason == 'cancelled' + + +@pytest.mark.asyncio +async def test_source_cancellation_before_dns_resolution_does_not_claim_dns_failure( + monkeypatch: pytest.MonkeyPatch, +) -> None: + completed: list[CompletedResult] = [] + + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def save_run(self, result: CompletedResult) -> None: + completed.append(result) + + class CancelledSource: + def __init__(self, _word: str) -> None: + pass + + async def process(self, _proxy: bool) -> None: + raise asyncio.CancelledError + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + monkeypatch.setattr(theharvester_main.certspottersearch, 'SearchCertspoter', CancelledSource) + + with pytest.raises(asyncio.CancelledError): + await theharvester_main.start( + EnumerationOptions( + domain='example.com', + source='certspotter', + dns_resolve='192.0.2.53', + quiet=True, + ) + ) + + assert len(completed) == 1 + assert completed[0].active_evidence.executions == () + assert len(completed[0].source_executions) == 1 + source_execution = completed[0].source_executions[0] + assert source_execution.source == 'certspotter' + assert source_execution.status == 'failed' + assert source_execution.error_type == 'CancelledError' + assert source_execution.stop_reason == 'cancelled' + assert completed[0].evidence_dict()['status'] == 'failed' + + +@pytest.mark.asyncio +async def test_dns_resolve_query_errors_are_partial_even_without_findings(monkeypatch: pytest.MonkeyPatch) -> None: + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def save_run(self, _result: CompletedResult) -> None: + return None + + class FakeSource: + def __init__(self, _word: str) -> None: + pass + + async def process(self, _proxy: bool) -> None: + return None + + async def get_hostnames(self) -> set[str]: + return {'api.example.com'} + + class EmptyChecker: + query_error_count = 1 + query_error_types = {'TimeoutError'} + + def __init__(self, hosts: list[str], _nameservers: list[str]) -> None: + assert hosts == ['api.example.com'] + + async def check(self) -> tuple[list[str], list[str], list[str]]: + return [], [], [] + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + monkeypatch.setattr(theharvester_main.certspottersearch, 'SearchCertspoter', FakeSource) + monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', EmptyChecker) + + response = await theharvester_main.start( + EnumerationOptions( + domain='example.com', + source='certspotter', + dns_resolve='192.0.2.53', + quiet=True, + ), + return_completed_result=True, + ) + + completed = response[-1] + assert isinstance(completed, CompletedResult) + execution = completed.active_evidence.executions[0] + assert execution.action == 'dns-resolve' + assert execution.status == 'partial' + assert execution.result_count == 0 + assert execution.error_type == 'TimeoutError' + assert execution.stop_reason == 'query-errors' + + +@pytest.mark.asyncio +async def test_requested_dns_resolve_without_inputs_is_skipped(monkeypatch: pytest.MonkeyPatch) -> None: + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def save_run(self, _result: CompletedResult) -> None: + return None + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + + response = await theharvester_main.start( + EnumerationOptions(domain='example.com', source='', dns_resolve='192.0.2.53', quiet=True), + return_completed_result=True, + ) + + completed = response[-1] + assert isinstance(completed, CompletedResult) + assert len(completed.active_evidence.executions) == 1 + execution = completed.active_evidence.executions[0] + assert execution.action == 'dns-resolve' + assert execution.status == 'skipped' + assert execution.stop_reason == 'no-input' + + +@pytest.mark.asyncio +async def test_rest_dns_lookup_runs_before_return_and_retains_action_evidence(monkeypatch: pytest.MonkeyPatch) -> None: + completed: list[CompletedResult] = [] + + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def save_run(self, result: CompletedResult) -> None: + completed.append(result) + + class FakeSecurityScorecard: + def __init__(self, domain: str) -> None: + assert domain == 'example.com' + + async def process(self, _proxy: bool) -> None: + return None + + async def get_hostnames(self) -> set[str]: + return set() + + async def get_ips(self) -> set[str]: + return {'192.0.2.10'} + + async def fake_reverse( + iprange: str, + callback, + nameservers: list[str] | None = None, + error_types: set[str] | None = None, + ) -> None: + assert iprange == '192.0.2.0/24' + assert nameservers is None + callback('PTR.example.com.') + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + monkeypatch.setattr(theharvester_main.securityscorecard, 'SearchSecurityScorecard', FakeSecurityScorecard) + monkeypatch.setattr(theharvester_main.dnssearch, 'reverse_all_ips_in_range', fake_reverse) + + response = await theharvester_main.start( + EnumerationOptions(domain='example.com', source='securityscorecard', dns_lookup=True, quiet=True), + persist_completed_result=True, + ) + + assert len(response) == 9 + assert response[8] == ['ptr.example.com'] + assert len(completed) == 1 + execution = completed[0].active_evidence.executions[0] + assert execution.action == 'dns-lookup' + assert execution.status == 'completed' + assert execution.error_type is None + assert execution.stop_reason is None + assert execution.observations[0].kind == 'hostname' + assert execution.observations[0].value == 'ptr.example.com' + + +@pytest.mark.asyncio +async def test_requested_dns_lookup_without_ip_ranges_is_skipped(monkeypatch: pytest.MonkeyPatch) -> None: + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def save_run(self, _result: CompletedResult) -> None: + return None + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + + response = await theharvester_main.start( + EnumerationOptions(domain='example.com', source='', dns_lookup=True, quiet=True), + return_completed_result=True, + ) + + completed = response[-1] + assert isinstance(completed, CompletedResult) + assert len(completed.active_evidence.executions) == 1 + execution = completed.active_evidence.executions[0] + assert execution.action == 'dns-lookup' + assert execution.status == 'skipped' + assert execution.result_count == 0 + assert execution.stop_reason == 'no-input' + + +@pytest.mark.asyncio +async def test_dns_lookup_cancels_sibling_ranges_and_persists_partial_evidence( + monkeypatch: pytest.MonkeyPatch, +) -> None: + completed: list[CompletedResult] = [] + sibling_started = asyncio.Event() + sibling_cancelled = asyncio.Event() + + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def save_run(self, result: CompletedResult) -> None: + completed.append(result) + + class FakeSecurityScorecard: + def __init__(self, _domain: str) -> None: + pass + + async def process(self, _proxy: bool) -> None: + return None + + async def get_hostnames(self) -> set[str]: + return set() + + async def get_ips(self) -> set[str]: + return {'192.0.2.10', '198.51.100.10'} + + async def fake_reverse( + iprange: str, + callback, + nameservers: list[str] | None = None, + error_types: set[str] | None = None, + ) -> None: + assert nameservers is None + if iprange == '192.0.2.0/24': + callback('partial.example.com') + await sibling_started.wait() + raise asyncio.CancelledError + sibling_started.set() + try: + await asyncio.Event().wait() + except asyncio.CancelledError: + sibling_cancelled.set() + raise + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + monkeypatch.setattr(theharvester_main.securityscorecard, 'SearchSecurityScorecard', FakeSecurityScorecard) + monkeypatch.setattr(theharvester_main.dnssearch, 'reverse_all_ips_in_range', fake_reverse) + + with pytest.raises(asyncio.CancelledError): + await theharvester_main.start( + EnumerationOptions(domain='example.com', source='securityscorecard', dns_lookup=True, quiet=True), + persist_completed_result=True, + ) + + assert sibling_cancelled.is_set() + assert len(completed) == 1 + execution = completed[0].active_evidence.executions[0] + assert execution.action == 'dns-lookup' + assert execution.status == 'partial' + assert execution.error_type == 'CancelledError' + assert execution.stop_reason == 'cancelled' + assert [(observation.kind, observation.value) for observation in execution.observations] == [ + ('hostname', 'partial.example.com') + ] + + @pytest.mark.asyncio async def test_source_failure_retains_normalized_partial_results(monkeypatch: pytest.MonkeyPatch) -> None: completed: list[CompletedResult] = [] @@ -181,7 +710,7 @@ async def test_source_failure_retains_normalized_partial_results(monkeypatch: py async def get_hostnames(self) -> list[str]: return ['API.Example.COM.', 'api.example.com', 'outside.test'] - async def get_interesting_urls(self) -> set[str]: + async def get_urls(self) -> set[str]: raise RuntimeError('provider page failed') monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) @@ -225,7 +754,7 @@ async def test_source_checkpoint_excludes_other_source_work_in_progress(monkeypa builtwith_collected.set() return {'early.example.com'} - async def get_interesting_urls(self) -> set[str]: + async def get_urls(self) -> set[str]: await release_builtwith.wait() return set() @@ -337,10 +866,81 @@ async def test_recursive_dns_requires_canonically_distinct_resolvers(monkeypatch ], ) - with pytest.raises(ValueError, match='exactly three resolver vantages'): + with pytest.raises(ValueError, match='exactly three resolver addresses'): await theharvester_main.start() +@pytest.mark.asyncio +async def test_cli_rejects_resolver_file_with_non_ip_value(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + class FakeResultStore: + async def initialize(self) -> None: + return None + + resolvers = tmp_path / 'resolvers.txt' + resolvers.write_text('192.0.2.53\nnot-an-ip\n', encoding='utf-8') + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + + with pytest.raises(ValueError, match='Invalid DNS resolver address: not-an-ip'): + await theharvester_main.start(EnumerationOptions(domain='example.com', dns_resolve=str(resolvers), quiet=True)) + + +@pytest.mark.asyncio +async def test_dns_brute_resolver_configuration_does_not_enable_dns_resolution( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def save_run(self, _result: CompletedResult) -> None: + return None + + class FakeDnsForce: + def __init__(self, domain: str, nameservers: list[str], verbose: bool) -> None: + assert domain == 'www.example.com' + assert nameservers == ['192.0.2.53'] + assert verbose is True + + async def run(self) -> tuple[list[str], list[str], list[str]]: + return ( + ['dev.www.example.com:192.0.2.10'], + ['dev.www.example.com'], + ['192.0.2.10'], + ) + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + monkeypatch.setattr(theharvester_main.dnssearch, 'DnsForce', FakeDnsForce) + resolvers = tmp_path / 'resolvers.txt' + resolvers.write_text('192.0.2.53\n', encoding='utf-8') + monkeypatch.setattr( + sys, + 'argv', + [ + 'theHarvester', + '-d', + 'www.example.com', + '-c', + '--dns-resolvers', + str(resolvers), + '--quiet', + ], + ) + checkpoints: list[CompletedResult] = [] + + async def checkpoint(result: CompletedResult) -> None: + checkpoints.append(result) + + with pytest.raises(SystemExit) as exit_info: + await theharvester_main.start(completed_result_checkpoint=checkpoint) + + assert exit_info.value.code == 0 + completed = checkpoints[-1] + actions = {execution.action for execution in completed.active_evidence.executions} + assert 'dns-brute' in actions + assert 'dns-resolve' not in actions + + @pytest.mark.asyncio async def test_dns_proven_cname_hosts_reach_screenshot_filter( monkeypatch: pytest.MonkeyPatch, @@ -399,8 +999,13 @@ async def test_dns_proven_cname_hosts_reach_screenshot_filter( def chunk_list(values: list[str], _size: int) -> list[list[str]]: return [values] - async def take_screenshot(self, host: str) -> tuple[str, str]: - return host, f'{host}.png' + async def take_screenshot(self, host: str, *, output_path: Path | None = None) -> str: + path = output_path or self.screenshot_path(host) + path.write_bytes(b'png') + return f'https://{host}' + + def screenshot_path(self, host: str) -> Path: + return Path(self.output) / f'{host.removeprefix("https://")}.png' class FakePool: def __init__(self, _workers: int) -> None: @@ -443,6 +1048,528 @@ async def test_dns_proven_cname_hosts_reach_screenshot_filter( assert visited == {'address.example.com', 'alias.example.com'} +class _NoopResultStore: + async def initialize(self) -> None: + return None + + async def record_observations(self, *_args: object) -> None: + return None + + async def save_run(self, _result: CompletedResult) -> None: + return None + + +class _ApiHostSource: + def __init__(self, _word: str) -> None: + pass + + async def process(self, _proxy: bool) -> None: + return None + + async def get_hostnames(self) -> set[str]: + return {'api.example.com'} + + +class _ApiHostChecker: + def __init__(self, _hosts: list[str], _nameservers: list[str]) -> None: + pass + + async def check(self) -> tuple[list[str], list[str], list[str]]: + return ['api.example.com:192.0.2.10'], ['api.example.com'], ['192.0.2.10'] + + +def _recording_result_store(saved: list[CompletedResult]) -> type[_NoopResultStore]: + class RecordingResultStore(_NoopResultStore): + async def save_run(self, result: CompletedResult) -> None: + saved.append(result) + + return RecordingResultStore + + +@pytest.mark.asyncio +async def test_cli_can_capture_an_explicit_target_without_discovery_sources( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + captured: list[str] = [] + saved: list[CompletedResult] = [] + + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def save_run(self, result: CompletedResult) -> None: + saved.append(result) + + class FakeScreenShotter: + slash = '/' + + def __init__(self, output: str) -> None: + self.output = output + + def verify_path(self) -> bool: + return True + + async def verify_installation(self) -> None: + return None + + async def visit(self, host: str) -> tuple[str, str]: + return host, 'https' + + @staticmethod + def chunk_list(values: list[str], _size: int) -> list[list[str]]: + return [values] + + async def take_screenshot(self, host: str, *, output_path: Path | None = None) -> str: + captured.append(host) + (output_path or self.screenshot_path(host)).write_bytes(b'png') + return f'https://{host}' + + def screenshot_path(self, host: str) -> Path: + return Path(self.output) / f'{host.removeprefix("https://")}.png' + + class FakePool: + def __init__(self, _workers: int) -> None: + pass + + async def __aenter__(self) -> 'FakePool': + return self + + async def __aexit__(self, *_args) -> None: + return None + + async def map(self, function, values): + return [await function(value) for value in values] + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + monkeypatch.setattr(theharvester_main, 'ScreenShotter', FakeScreenShotter) + monkeypatch.setattr(theharvester_main, 'Pool', FakePool) + monkeypatch.setattr( + sys, + 'argv', + ['theHarvester', '-d', 'api.example.com', '--screenshot', str(tmp_path), '--quiet'], + ) + + with pytest.raises(SystemExit) as exit_info: + await theharvester_main.start() + + assert exit_info.value.code == 0 + assert captured == ['api.example.com'] + completed = saved[-1] + execution = next(item for item in completed.active_evidence.executions if item.action == 'screenshot') + assert execution.status == 'completed' + assert execution.result_count == 0 + assert ('screenshot', 'https://api.example.com') not in completed.results + assert ('hostname', 'api.example.com') in completed.results + assert len(execution.artifacts) == 1 + artifact = execution.artifacts[0] + assert artifact.subject_value == 'api.example.com' + assert artifact.path == f'{tmp_path.name}/api.example.com.png' + assert artifact.media_type == 'image/png' + assert artifact.size_bytes == 3 + + +@pytest.mark.asyncio +async def test_screenshot_reports_no_reachable_target_as_failed( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + class FakeScreenShotter: + slash = '/' + + def __init__(self, output: str) -> None: + self.output = output + + def verify_path(self) -> bool: + return True + + async def verify_installation(self) -> None: + return None + + async def visit(self, _host: str) -> tuple[str, str]: + return '', '' + + class FakePool: + def __init__(self, _workers: int) -> None: + pass + + async def __aenter__(self): + return self + + async def __aexit__(self, *_args) -> None: + return None + + async def map(self, function, values): + return [await function(value) for value in values] + + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + monkeypatch.setattr(theharvester_main, 'ScreenShotter', FakeScreenShotter) + monkeypatch.setattr(theharvester_main, 'Pool', FakePool) + + result = await theharvester_main.start( + EnumerationOptions(domain='api.example.test', screenshot=str(tmp_path), quiet=True), + return_completed_result=True, + ) + + execution = next(item for item in result[-1].active_evidence.executions if item.action == 'screenshot') + assert execution.status == 'failed' + assert execution.stop_reason == 'no-reachable-targets' + + +@pytest.mark.asyncio +async def test_screenshot_redirect_stays_attached_to_the_authorized_host( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + class FakeScreenShotter: + slash = '/' + + def __init__(self, output: str) -> None: + self.output = output + + def verify_path(self) -> bool: + return True + + async def verify_installation(self) -> None: + return None + + async def visit(self, host: str) -> tuple[str, str]: + assert host == 'api.example.test' + return 'https://outside.example/path', 'reachable' + + async def take_screenshot(self, url: str, *, output_path: Path | None = None) -> str: + (output_path or self.screenshot_path(url)).write_bytes(b'png') + return url + + def screenshot_path(self, _url: str) -> Path: + return Path(self.output) / 'outside.example.png' + + class FakePool: + def __init__(self, _workers: int) -> None: + pass + + async def __aenter__(self): + return self + + async def __aexit__(self, *_args) -> None: + return None + + async def map(self, function, values): + return [await function(value) for value in values] + + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + monkeypatch.setattr(theharvester_main, 'ScreenShotter', FakeScreenShotter) + monkeypatch.setattr(theharvester_main, 'Pool', FakePool) + + result = await theharvester_main.start( + EnumerationOptions(domain='api.example.test', screenshot=str(tmp_path), quiet=True), + return_completed_result=True, + ) + + completed = result[-1] + execution = next(item for item in completed.active_evidence.executions if item.action == 'screenshot') + assert execution.artifacts[0].subject_value == 'api.example.test' + assert ('hostname', 'outside.example') not in completed.results + + +@pytest.mark.asyncio +async def test_target_only_ip_screenshot_keeps_ip_result_and_artifact_subject( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + class FakeScreenShotter: + slash = '/' + + def __init__(self, output: str) -> None: + self.output = output + + def verify_path(self) -> bool: + return True + + async def verify_installation(self) -> None: + return None + + async def visit(self, host: str) -> tuple[str, str]: + return f'https://{host}', 'reachable' + + async def take_screenshot(self, url: str, *, output_path: Path | None = None) -> str: + assert output_path is not None + output_path.write_bytes(b'png') # noqa: ASYNC240 - tiny in-memory screenshot fixture + return url + + def screenshot_path(self, url: str) -> Path: + return Path(self.output) / f'{url.removeprefix("https://")}.png' + + class FakePool: + def __init__(self, _workers: int) -> None: + pass + + async def __aenter__(self): + return self + + async def __aexit__(self, *_args) -> None: + return None + + async def map(self, function, values): + return [await function(value) for value in values] + + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + monkeypatch.setattr(theharvester_main, 'ScreenShotter', FakeScreenShotter) + monkeypatch.setattr(theharvester_main, 'Pool', FakePool) + + result = await theharvester_main.start( + EnumerationOptions(domain='192.0.2.1', screenshot=str(tmp_path), quiet=True), + return_completed_result=True, + ) + + completed = result[-1] + execution = next(item for item in completed.active_evidence.executions if item.action == 'screenshot') + assert ('ip', '192.0.2.1') in completed.results + assert ('hostname', '192.0.2.1') not in completed.results + assert execution.artifacts[0].subject_kind == 'ip' + assert execution.artifacts[0].subject_value == '192.0.2.1' + + +@pytest.mark.asyncio +async def test_screenshot_redirects_to_one_login_keep_distinct_subject_artifacts( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + class FakeScreenShotter: + slash = '/' + + def __init__(self, output: str) -> None: + self.output = output + + def verify_path(self) -> bool: + return True + + async def verify_installation(self) -> None: + return None + + async def visit(self, _host: str) -> tuple[str, str]: + return 'https://login.example.net/session', 'reachable' + + async def take_screenshot(self, url: str, *, output_path: Path | None = None) -> str: + assert url == 'https://login.example.net/session' + assert output_path is not None + output_path.write_bytes(output_path.name.encode()) # noqa: ASYNC240 - tiny in-memory screenshot fixture + return url + + def screenshot_path(self, url: str) -> Path: + hostname = url.removeprefix('https://').split('/', maxsplit=1)[0] + return Path(self.output) / f'{hostname}.png' + + class TwoHostSource: + def __init__(self, _word: str) -> None: + pass + + async def process(self, _proxy: bool) -> None: + return None + + async def get_hostnames(self) -> set[str]: + return {'first.example.test', 'second.example.test'} + + class FakePool: + def __init__(self, _workers: int) -> None: + pass + + async def __aenter__(self): + return self + + async def __aexit__(self, *_args) -> None: + return None + + async def map(self, function, values): + return [await function(value) for value in values] + + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', TwoHostSource) + monkeypatch.setattr(theharvester_main, 'ScreenShotter', FakeScreenShotter) + monkeypatch.setattr(theharvester_main, 'Pool', FakePool) + + result = await theharvester_main.start( + EnumerationOptions( + domain='example.test', + screenshot=str(tmp_path), + quiet=True, + source='crtsh', + ), + return_completed_result=True, + ) + + execution = next(item for item in result[-1].active_evidence.executions if item.action == 'screenshot') + assert [(artifact.subject_value, Path(artifact.path).name) for artifact in execution.artifacts] == [ + ('first.example.test', 'first.example.test.png'), + ('second.example.test', 'second.example.test.png'), + ] + + +@pytest.mark.asyncio +async def test_screenshot_cancellation_persists_failed_execution_and_propagates( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + saved: list[CompletedResult] = [] + captured = asyncio.Event() + + class RecordingResultStore(_NoopResultStore): + async def save_run(self, result: CompletedResult) -> None: + saved.append(result) + + class FakeScreenShotter: + slash = '/' + + def __init__(self, output: str) -> None: + self.output = output + + def verify_path(self) -> bool: + return True + + async def verify_installation(self) -> None: + return None + + async def visit(self, host: str) -> tuple[str, str]: + return f'https://{host}', 'reachable' + + async def take_screenshot(self, host: str, *, output_path: Path | None = None) -> str: + if 'first.' in host: + (output_path or self.screenshot_path(host)).write_bytes(b'png') + captured.set() + return host + await asyncio.Event().wait() + return '' + + def screenshot_path(self, host: str) -> Path: + return Path(self.output) / f'{host.removeprefix("https://")}.png' + + class TwoHostSource: + def __init__(self, _word: str) -> None: + pass + + async def process(self, _proxy: bool) -> None: + return None + + async def get_hostnames(self) -> set[str]: + return {'first.example.test', 'second.example.test'} + + class FakePool: + def __init__(self, _workers: int) -> None: + pass + + async def __aenter__(self): + return self + + async def __aexit__(self, *_args) -> None: + return None + + async def map(self, function, values): + return [await function(value) for value in values] + + monkeypatch.setattr(theharvester_main, 'ResultStore', RecordingResultStore) + monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', TwoHostSource) + monkeypatch.setattr(theharvester_main, 'ScreenShotter', FakeScreenShotter) + monkeypatch.setattr(theharvester_main, 'Pool', FakePool) + + task = asyncio.create_task( + theharvester_main.start( + EnumerationOptions( + domain='example.test', + screenshot=str(tmp_path), + quiet=True, + source='crtsh', + ), + return_completed_result=True, + ) + ) + await captured.wait() + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + execution = next(item for item in saved[-1].active_evidence.executions if item.action == 'screenshot') + assert execution.status == 'partial' + assert execution.stop_reason == 'cancelled' + assert [artifact.subject_value for artifact in execution.artifacts] == ['first.example.test'] + + +@pytest.mark.asyncio +async def test_screenshot_capture_failure_cancels_sibling_tasks( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + sibling_cancelled = asyncio.Event() + + class FakeScreenShotter: + slash = '/' + + def __init__(self, output: str) -> None: + self.output = output + + def verify_path(self) -> bool: + return True + + async def verify_installation(self) -> None: + return None + + async def visit(self, host: str) -> tuple[str, str]: + return f'https://{host}', 'reachable' + + async def take_screenshot(self, url: str, *, output_path: Path | None = None) -> str: + if 'first.' in url: + raise RuntimeError('capture failed') + try: + await asyncio.Event().wait() + finally: + sibling_cancelled.set() + return '' + + def screenshot_path(self, url: str) -> Path: + return Path(self.output) / f'{url.removeprefix("https://")}.png' + + class TwoHostSource: + def __init__(self, _word: str) -> None: + pass + + async def process(self, _proxy: bool) -> None: + return None + + async def get_hostnames(self) -> set[str]: + return {'first.example.test', 'second.example.test'} + + class FakePool: + def __init__(self, _workers: int) -> None: + pass + + async def __aenter__(self): + return self + + async def __aexit__(self, *_args) -> None: + return None + + async def map(self, function, values): + return [await function(value) for value in values] + + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', TwoHostSource) + monkeypatch.setattr(theharvester_main, 'ScreenShotter', FakeScreenShotter) + monkeypatch.setattr(theharvester_main, 'Pool', FakePool) + + result = await theharvester_main.start( + EnumerationOptions( + domain='example.test', + screenshot=str(tmp_path), + quiet=True, + source='crtsh', + ), + return_completed_result=True, + ) + + execution = next(item for item in result[-1].active_evidence.executions if item.action == 'screenshot') + assert execution.status == 'failed' + assert sibling_cancelled.is_set() + + @pytest.mark.asyncio async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: class FakeResultStore: @@ -476,6 +1603,10 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte class FakeTakeOver: def __init__(self, hosts: list[str]) -> None: assert hosts == ['api.example.com'] + self.request_count = 2 + self.request_error_count = 0 + self.request_error_types: set[str] = set() + self.scan_error_type = None async def populate_fingerprints(self) -> None: return None @@ -505,26 +1636,35 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte def chunk_list(values: list[str], _size: int) -> list[list[str]]: return [values] - async def take_screenshot(self, host: str) -> str: - return host + async def take_screenshot(self, host: str, *, output_path: Path | None = None) -> str: + (output_path or self.screenshot_path(host)).write_bytes(b'png') + return f'https://{host}' + + def screenshot_path(self, host: str) -> Path: + return Path(self.output) / f'{host.removeprefix("https://")}.png' class FakeShodan: + error_type = None + async def search_ip(self, ip: str) -> dict[str, dict[str, list[int]]]: return {ip: {'ports': [443]}} class FakeApiScanner: - def __init__(self, word: str, wordlist: str) -> None: + def __init__(self, word: str, wordlist: str, exact_paths: bool = False) -> None: assert word == 'example.com' assert wordlist == str(tmp_path / 'api.txt') + self.scan_error_type = None + self.request_error_count = 0 + self.request_error_types: set[str] = set() async def do_search(self) -> None: return None def get_found_endpoints(self) -> set[str]: - return {'/api/v1'} + return {'https://example.com/api/v1'} def get_interesting_endpoints(self) -> set[str]: - return {'/api/v1'} + return {'https://example.com/api/v1'} def get_auth_required(self) -> set[str]: return set() @@ -588,13 +1728,569 @@ async def test_direct_action_evidence_reaches_completed_result(monkeypatch: pyte completed = result[-1] assert isinstance(completed, CompletedResult) - assert ('api-endpoint', '/api/v1') in completed.results - assert ('screenshot', 'api.example.com') in completed.results + assert ('url', 'https://example.com/api/v1') in completed.results + assert ('screenshot', 'api.example.com') not in completed.results assert ('shodan', '{"ip":"192.0.2.10","result":{"ports":[443]}}') in completed.results - assert ( + takeover_result = ( 'takeover', '{"matches":[{"No such app":"Heroku"}],"url":"https://api.example.com"}', - ) in completed.results + ) + assert takeover_result in completed.results + takeover_execution = next(execution for execution in completed.active_evidence.executions if execution.action == 'takeover') + assert takeover_execution.status == 'completed' + assert takeover_execution.result_count == 1 + assert takeover_execution.error_type is None + assert takeover_execution.stop_reason is None + screenshot_execution = next( + execution for execution in completed.active_evidence.executions if execution.action == 'screenshot' + ) + assert screenshot_execution.status == 'completed' + assert screenshot_execution.result_count == 0 + assert screenshot_execution.artifacts[0].subject_value == 'api.example.com' + shodan_execution = next(execution for execution in completed.active_evidence.executions if execution.action == 'shodan') + assert shodan_execution.status == 'completed' + assert shodan_execution.result_count == 1 + assert shodan_execution.error_type is None + assert shodan_execution.stop_reason is None + api_executions = [execution for execution in completed.active_evidence.executions if execution.action == 'api-scan'] + assert len(api_executions) == 1 + api_execution = api_executions[0] + assert api_execution.status == 'completed' + assert api_execution.result_count == 1 + assert api_execution.error_type is None + assert api_execution.stop_reason is None + assert {(observation.kind, observation.value) for observation in api_execution.observations} == { + ('url', 'https://example.com/api/v1') + } + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ('request_count', 'request_errors', 'scan_error', 'expected_status', 'expected_error', 'expected_reason'), + [ + (2, 1, None, 'partial', 'TransportError', 'request-errors'), + (2, 2, None, 'failed', 'TransportError', 'request-errors'), + (0, 0, 'RuntimeError', 'failed', 'RuntimeError', 'scan-error'), + ], +) +async def test_takeover_action_records_suppressed_outcome( + monkeypatch: pytest.MonkeyPatch, + request_count: int, + request_errors: int, + scan_error: str | None, + expected_status: str, + expected_error: str, + expected_reason: str, +) -> None: + class FakeTakeOver: + def __init__(self, _hosts: list[str]) -> None: + self.request_count = request_count + self.request_error_count = request_errors + self.request_error_types = {'TransportError'} if request_errors else set() + self.scan_error_type = scan_error + + async def populate_fingerprints(self) -> None: + return None + + async def process(self, proxy: bool = False) -> None: + assert proxy is False + return None + + async def get_takeover_results(self) -> dict: + return {} + + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', _ApiHostSource) + monkeypatch.setattr(theharvester_main.takeover, 'TakeOver', FakeTakeOver) + + result = await theharvester_main.start( + EnumerationOptions(domain='example.com', quiet=True, source='crtsh', take_over=True), + return_completed_result=True, + ) + + execution = next(item for item in result[-1].active_evidence.executions if item.action == 'takeover') + assert execution.status == expected_status + assert execution.result_count == 0 + assert execution.error_type == expected_error + assert execution.stop_reason == expected_reason + + +@pytest.mark.asyncio +async def test_shodan_action_records_all_target_errors_as_failed( + monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture +) -> None: + caplog.set_level(logging.INFO) + + class FailedShodan: + error_type = None + + async def search_ip(self, ip: str) -> dict[str, str]: + raise RuntimeError(f'provider-secret-payload for {ip}') + + async def no_sleep(_seconds: float) -> None: + return None + + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', _ApiHostSource) + monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', _ApiHostChecker) + monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', FailedShodan) + monkeypatch.setattr(theharvester_main.asyncio, 'sleep', no_sleep) + + result = await theharvester_main.start( + EnumerationOptions( + dns_resolve='192.0.2.53', + domain='example.com', + quiet=True, + shodan=True, + source='crtsh', + ), + return_completed_result=True, + ) + + completed = result[-1] + shodan_execution = next(execution for execution in completed.active_evidence.executions if execution.action == 'shodan') + assert shodan_execution.status == 'failed' + assert shodan_execution.result_count == 0 + assert shodan_execution.error_type == 'RuntimeError' + assert shodan_execution.stop_reason == 'target-errors' + assert 'provider-secret-payload' not in caplog.text + + +@pytest.mark.asyncio +async def test_shodan_no_data_is_a_completed_zero_yield_action(monkeypatch: pytest.MonkeyPatch) -> None: + class EmptyShodan: + error_type = None + + async def search_ip(self, _ip: str) -> dict: + return {} + + async def no_sleep(_seconds: float) -> None: + return None + + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', _ApiHostSource) + monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', _ApiHostChecker) + monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', EmptyShodan) + monkeypatch.setattr(theharvester_main.asyncio, 'sleep', no_sleep) + + result = await theharvester_main.start( + EnumerationOptions(dns_resolve='192.0.2.53', domain='example.com', quiet=True, shodan=True, source='crtsh'), + return_completed_result=True, + ) + + completed = result[-1] + execution = next(item for item in completed.active_evidence.executions if item.action == 'shodan') + assert execution.status == 'completed' + assert execution.result_count == 0 + assert execution.error_type is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ('scan_error', 'request_errors', 'rate_limited', 'expected_status', 'expected_error', 'expected_reason'), + [ + ('RuntimeError', 0, False, 'failed', 'RuntimeError', 'scan-error'), + (None, 3, False, 'partial', 'TransportError', 'request-errors'), + (None, 0, True, 'rate-limited', None, 'rate-limited'), + ], +) +async def test_api_scan_records_suppressed_scan_failure( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + scan_error: str | None, + request_errors: int, + rate_limited: bool, + expected_status: str, + expected_error: str | None, + expected_reason: str, +) -> None: + class FailedApiScanner: + scan_error_type = scan_error + request_error_count = request_errors + request_error_types = {'TransportError'} if request_errors else set() + + def __init__(self, word: str, wordlist: str, exact_paths: bool = False) -> None: + assert word == 'example.com' + assert wordlist == str(tmp_path / 'api.txt') + assert exact_paths is True + + async def do_search(self) -> None: + return None + + def get_found_endpoints(self) -> dict: + return {} + + def get_interesting_endpoints(self) -> dict: + return {} + + def get_auth_required(self) -> dict: + return {} + + def get_api_versions(self) -> set[str]: + return set() + + def get_rate_limits(self) -> dict: + if rate_limited: + return {'/api': type('RateLimitInfo', (), {'method': 'GET'})()} + return {} + + def get_methods(self) -> set[str]: + return set() + + def get_status_codes(self) -> set[int]: + return set() + + wordlist = tmp_path / 'api.txt' + wordlist.write_text('/api\n', encoding='utf-8') + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + monkeypatch.setattr(theharvester_main.api_endpoints, 'SearchApiEndpoints', FailedApiScanner) + + result = await theharvester_main.start( + EnumerationOptions( + api_scan=True, + domain='example.com', + quiet=True, + wordlist=str(wordlist), + ), + return_completed_result=True, + ) + + completed = result[-1] + api_execution = next(execution for execution in completed.active_evidence.executions if execution.action == 'api-scan') + assert api_execution.status == expected_status + assert api_execution.result_count == 0 + assert api_execution.error_type == expected_error + assert api_execution.stop_reason == expected_reason + + +@pytest.mark.asyncio +async def test_takeover_without_hosts_is_skipped_without_starting(monkeypatch: pytest.MonkeyPatch) -> None: + class UnexpectedTakeOver: + def __init__(self, _hosts: list[str]) -> None: + raise AssertionError('takeover should not start without hosts') + + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + monkeypatch.setattr(theharvester_main.takeover, 'TakeOver', UnexpectedTakeOver) + + result = await theharvester_main.start( + EnumerationOptions(domain='example.com', quiet=True, take_over=True), + return_completed_result=True, + ) + + completed = result[-1] + takeover_execution = next(execution for execution in completed.active_evidence.executions if execution.action == 'takeover') + assert takeover_execution.status == 'skipped' + assert takeover_execution.result_count == 0 + assert takeover_execution.stop_reason == 'no-input' + + +@pytest.mark.asyncio +async def test_shodan_without_ips_is_skipped_without_starting(monkeypatch: pytest.MonkeyPatch) -> None: + class UnexpectedShodan: + def __init__(self) -> None: + raise AssertionError('Shodan should not start without IP addresses') + + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', UnexpectedShodan) + + result = await theharvester_main.start( + EnumerationOptions(domain='example.com', quiet=True, shodan=True), + return_completed_result=True, + ) + + completed = result[-1] + shodan_execution = next(execution for execution in completed.active_evidence.executions if execution.action == 'shodan') + assert shodan_execution.status == 'skipped' + assert shodan_execution.result_count == 0 + assert shodan_execution.stop_reason == 'no-input' + + +@pytest.mark.asyncio +async def test_api_scan_cancellation_persists_failure_and_propagates(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + saved: list[CompletedResult] = [] + + class CancelledApiScanner: + def __init__(self, word: str, wordlist: str, exact_paths: bool = False) -> None: + assert word == 'example.com' + assert wordlist == str(tmp_path / 'api.txt') + assert exact_paths is True + + async def do_search(self) -> None: + raise asyncio.CancelledError + + def get_found_endpoints(self) -> set[str]: + return {'https://example.com/api/v1'} + + def get_interesting_endpoints(self) -> set[str]: + return {'https://example.com/api/v1'} + + wordlist = tmp_path / 'api.txt' + wordlist.write_text('/api\n', encoding='utf-8') + monkeypatch.setattr(theharvester_main, 'ResultStore', _recording_result_store(saved)) + monkeypatch.setattr(theharvester_main.api_endpoints, 'SearchApiEndpoints', CancelledApiScanner) + + with pytest.raises(asyncio.CancelledError): + await theharvester_main.start( + EnumerationOptions( + api_scan=True, + domain='example.com', + quiet=True, + wordlist=str(wordlist), + ), + return_completed_result=True, + ) + + execution = next(item for item in saved[-1].active_evidence.executions if item.action == 'api-scan') + assert execution.status == 'partial' + assert execution.result_count == 1 + assert execution.error_type == 'CancelledError' + assert execution.stop_reason == 'cancelled' + assert {(observation.kind, observation.value) for observation in execution.observations} == { + ('url', 'https://example.com/api/v1') + } + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ('raised_error', 'failure_stage', 'expected_status', 'expected_count'), + [ + (RuntimeError('scan failed'), 'search', 'partial', 1), + (MissingKey('API endpoints'), 'init', 'failed', 0), + (RuntimeError('getter failed'), 'getter', 'partial', 1), + ], +) +async def test_api_scan_raised_failure_is_persisted( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + raised_error: Exception, + failure_stage: str, + expected_status: str, + expected_count: int, +) -> None: + class FailedApiScanner: + def __init__(self, word: str, wordlist: str, exact_paths: bool = False) -> None: + assert exact_paths is True + assert word == 'example.com' + assert wordlist == str(tmp_path / 'api.txt') + self.scan_error_type = None + self.request_error_count = 0 + self.request_error_types: set[str] = set() + if failure_stage == 'init': + raise raised_error + + async def do_search(self) -> None: + if failure_stage == 'search': + raise raised_error + + def get_found_endpoints(self) -> set[str]: + if failure_stage == 'getter': + raise raised_error + return {'https://example.com/api/v1'} + + def get_interesting_endpoints(self) -> set[str]: + return {'https://example.com/api/v1'} + + def get_auth_required(self) -> dict: + return {} + + def get_api_versions(self) -> set[str]: + return set() + + def get_rate_limits(self) -> dict: + return {} + + def get_methods(self) -> set[str]: + return set() + + def get_status_codes(self) -> set[int]: + return set() + + wordlist = tmp_path / 'api.txt' + wordlist.write_text('/api\n', encoding='utf-8') + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + monkeypatch.setattr(theharvester_main.api_endpoints, 'SearchApiEndpoints', FailedApiScanner) + + result = await theharvester_main.start( + EnumerationOptions(api_scan=True, domain='example.com', quiet=True, wordlist=str(wordlist)), + return_completed_result=True, + ) + + executions = [item for item in result[-1].active_evidence.executions if item.action == 'api-scan'] + assert len(executions) == 1 + execution = executions[0] + assert execution.status == expected_status + assert execution.result_count == expected_count + assert execution.error_type == type(raised_error).__name__ + assert execution.stop_reason == 'scan-error' + if failure_stage == 'getter': + assert {(observation.kind, observation.value) for observation in execution.observations} == { + ('url', 'https://example.com/api/v1') + } + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ('raised_error', 'failure_stage', 'expected_reason'), + [ + (asyncio.CancelledError(), 'process', 'cancelled'), + (RuntimeError('scan failed'), 'process', 'scan-error'), + (RuntimeError('constructor failed'), 'init', 'scan-error'), + (RuntimeError('getter failed'), 'getter', 'scan-error'), + ], +) +async def test_takeover_failure_persists_and_propagates( + monkeypatch: pytest.MonkeyPatch, + raised_error: BaseException, + failure_stage: str, + expected_reason: str, +) -> None: + saved: list[CompletedResult] = [] + + class CancelledTakeOver: + def __init__(self, _hosts: list[str]) -> None: + self.request_error_count = 0 + self.request_error_types: set[str] = set() + self.scan_error_type = None + if failure_stage == 'init': + raise raised_error + + async def populate_fingerprints(self) -> None: + return None + + async def process(self, _proxy: bool = False, **_kwargs) -> None: + if failure_stage == 'process': + raise raised_error + + async def get_takeover_results(self) -> dict: + if failure_stage == 'getter': + raise raised_error + return {} + + monkeypatch.setattr(theharvester_main, 'ResultStore', _recording_result_store(saved)) + monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', _ApiHostSource) + monkeypatch.setattr(theharvester_main.takeover, 'TakeOver', CancelledTakeOver) + + with pytest.raises(type(raised_error)): + await theharvester_main.start( + EnumerationOptions(domain='example.com', quiet=True, source='crtsh', take_over=True), + return_completed_result=True, + ) + + execution = next(item for item in saved[-1].active_evidence.executions if item.action == 'takeover') + assert execution.status == 'failed' + assert execution.result_count == 0 + assert execution.error_type == type(raised_error).__name__ + assert execution.stop_reason == expected_reason + + +@pytest.mark.asyncio +async def test_shodan_cancellation_persists_failure_and_propagates(monkeypatch: pytest.MonkeyPatch) -> None: + saved: list[CompletedResult] = [] + + class FakeChecker: + def __init__(self, _hosts: list[str], _nameservers: list[str]) -> None: + pass + + async def check(self) -> tuple[list[str], list[str], list[str]]: + return ( + ['api.example.com:192.0.2.10', 'www.example.com:192.0.2.11'], + ['api.example.com', 'www.example.com'], + ['192.0.2.10', '192.0.2.11'], + ) + + class CancelledShodan: + error_type = None + + async def search_ip(self, ip: str) -> dict: + return {ip: {'ports': [443]}} + + async def cancel_during_throttle(_seconds: float) -> None: + raise asyncio.CancelledError + + monkeypatch.setattr(theharvester_main, 'ResultStore', _recording_result_store(saved)) + monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', _ApiHostSource) + monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', FakeChecker) + monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', CancelledShodan) + monkeypatch.setattr(theharvester_main.asyncio, 'sleep', cancel_during_throttle) + + with pytest.raises(asyncio.CancelledError): + await theharvester_main.start( + EnumerationOptions( + dns_resolve='192.0.2.53', + domain='example.com', + quiet=True, + shodan=True, + source='crtsh', + ), + return_completed_result=True, + ) + + execution = next(item for item in saved[-1].active_evidence.executions if item.action == 'shodan') + assert execution.status == 'partial' + assert execution.result_count == 1 + assert execution.error_type == 'CancelledError' + assert execution.stop_reason == 'cancelled' + + +@pytest.mark.asyncio +@pytest.mark.parametrize('action', ['takeover', 'shodan']) +async def test_direct_action_checkpoint_cancellation_persists_and_propagates( + monkeypatch: pytest.MonkeyPatch, action: str +) -> None: + checkpoints: list[CompletedResult] = [] + saved: list[CompletedResult] = [] + + class FakeTakeOver: + def __init__(self, _hosts: list[str]) -> None: + self.request_count = 1 + self.request_error_count = 0 + self.request_error_types: set[str] = set() + self.scan_error_type = None + + async def populate_fingerprints(self) -> None: + return None + + async def process(self, proxy: bool = False) -> None: + assert proxy is False + + async def get_takeover_results(self) -> dict: + return {} + + class FakeShodan: + error_type = None + + async def search_ip(self, ip: str) -> dict: + return {ip: {'ports': [443]}} + + async def cancel_after_action(result: CompletedResult) -> None: + if any(execution.action == action for execution in result.active_evidence.executions): + checkpoints.append(result) + raise asyncio.CancelledError + + monkeypatch.setattr(theharvester_main, 'ResultStore', _recording_result_store(saved)) + monkeypatch.setattr(theharvester_main.crtsh, 'SearchCrtsh', _ApiHostSource) + monkeypatch.setattr(theharvester_main.takeover, 'TakeOver', FakeTakeOver) + monkeypatch.setattr(theharvester_main.hostchecker, 'Checker', _ApiHostChecker) + monkeypatch.setattr(theharvester_main.shodansearch, 'SearchShodan', FakeShodan) + + options = EnumerationOptions( + dns_resolve='192.0.2.53' if action == 'shodan' else '', + domain='example.com', + quiet=True, + shodan=action == 'shodan', + source='crtsh', + take_over=action == 'takeover', + ) + with pytest.raises(asyncio.CancelledError): + await theharvester_main.start( + options, + completed_result_checkpoint=cancel_after_action, + return_completed_result=True, + ) + + execution = next(item for item in saved[-1].active_evidence.executions if item.action == action) + assert execution.status == 'completed' + assert saved[-1] == checkpoints[-1] @pytest.mark.asyncio @@ -604,6 +2300,7 @@ async def test_recursive_dns_results_reach_completed_output_without_changing_leg ) -> None: completed: list[CompletedResult] = [] captured: list[tuple[str, tuple[str, ...], int, int]] = [] + closed: list[str] = [] output_path = tmp_path / 'recursive-dns' class FakeResultStore: @@ -639,7 +2336,7 @@ async def test_recursive_dns_results_reach_completed_output_without_changing_leg assert target == 'example.com' async def close(self) -> None: - return None + closed.append(self.name) async def fake_recursive(target, seeds, _labels, _resolvers, limits): captured.append((target, tuple(seeds), limits.depth, limits.query_limit)) @@ -695,10 +2392,11 @@ async def test_recursive_dns_results_reach_completed_output_without_changing_leg assert exit_info.value.code == 0 assert captured == [('example.com', ('api.example.com',), 1, 3_000)] + assert sorted(closed) == ['192.0.2.53', '192.0.2.54', '192.0.2.55'] assert completed assert ('hostname', 'dev.api.example.com') in completed[0].results - assert ('ip-address', '192.0.2.2') in completed[0].results - assert ('ip-address', '2001:db8::2') in completed[0].results + assert ('ip', '192.0.2.2') in completed[0].results + assert ('ip', '2001:db8::2') in completed[0].results assert ( 'dns-recursive-finding', json.dumps( @@ -746,6 +2444,53 @@ async def test_recursive_dns_results_reach_completed_output_without_changing_leg sort_keys=True, ), ) in completed[0].results + recursive_execution = next( + execution for execution in completed[0].active_evidence.executions if execution.action == 'dns-recursive' + ) + assert recursive_execution.status == 'completed' + assert recursive_execution.stop_reason == 'depth-limit' + assert recursive_execution.result_count == 6 + assert {(observation.kind, observation.value) for observation in recursive_execution.observations} >= { + ('hostname', 'dev.api.example.com'), + ('ip', '192.0.2.2'), + ('ip', '2001:db8::2'), + } + + +@pytest.mark.asyncio +async def test_requested_recursive_dns_without_seed_hosts_is_skipped(monkeypatch: pytest.MonkeyPatch) -> None: + class FakeResultStore: + async def initialize(self) -> None: + return None + + async def save_run(self, _result: CompletedResult) -> None: + return None + + async def unexpected_recursive(*_args, **_kwargs): + raise AssertionError('recursive discovery must not start without seed hostnames') + + monkeypatch.setattr(theharvester_main, 'ResultStore', FakeResultStore) + monkeypatch.setattr(theharvester_main, 'discover_recursive_dns', unexpected_recursive) + + response = await theharvester_main.start( + EnumerationOptions( + domain='example.com', + source='', + dns_resolve='192.0.2.53,192.0.2.54,192.0.2.55', + dns_recursive_depth=1, + quiet=True, + ), + return_completed_result=True, + ) + + completed = response[-1] + assert isinstance(completed, CompletedResult) + recursive_execution = next( + execution for execution in completed.active_evidence.executions if execution.action == 'dns-recursive' + ) + assert recursive_execution.status == 'skipped' + assert recursive_execution.result_count == 0 + assert recursive_execution.stop_reason == 'no-input' @pytest.mark.parametrize('error_type', [RuntimeError, asyncio.CancelledError]) @@ -754,6 +2499,7 @@ async def test_recursive_dns_closes_resolvers_on_failure_and_preserves_cancellat monkeypatch: pytest.MonkeyPatch, error_type: type[BaseException] ) -> None: closed: list[str] = [] + completed: list[CompletedResult] = [] class FakeResultStore: async def initialize(self) -> None: @@ -762,8 +2508,8 @@ async def test_recursive_dns_closes_resolvers_on_failure_and_preserves_cancellat async def record_observations(self, *_args) -> None: return None - async def save_run(self, _result: CompletedResult) -> None: - return None + async def save_run(self, result: CompletedResult) -> None: + completed.append(result) class FakeCrtsh: def __init__(self, _word: str) -> None: @@ -822,3 +2568,10 @@ async def test_recursive_dns_closes_resolvers_on_failure_and_preserves_cancellat assert exit_info.value.code == 0 assert sorted(closed) == ['192.0.2.53', '192.0.2.54', '192.0.2.55'] + assert len(completed) == 1 + recursive_execution = next( + execution for execution in completed[0].active_evidence.executions if execution.action == 'dns-recursive' + ) + assert recursive_execution.status == 'failed' + assert recursive_execution.error_type == error_type.__name__ + assert recursive_execution.stop_reason == ('cancelled' if issubclass(error_type, asyncio.CancelledError) else None) diff --git a/tests/test_readme.py b/tests/test_readme.py index 633fa20f..9c791f5d 100644 --- a/tests/test_readme.py +++ b/tests/test_readme.py @@ -7,15 +7,13 @@ import yaml from theHarvester.lib.source_catalog import SOURCE_SPECS, ResultRoute -RESULT_COLUMNS = ('Subdomains', 'Emails', 'IPs', 'ASNs', 'URLs / links', 'People', 'Breaches') +RESULT_COLUMNS = ('Subdomains', 'Emails', 'IPs', 'ASNs', 'URLs', 'People', 'Breaches') ROUTE_COLUMNS = { ResultRoute.SUBDOMAINS: 'Subdomains', ResultRoute.EMAILS: 'Emails', ResultRoute.IPS: 'IPs', ResultRoute.ASNS: 'ASNs', - ResultRoute.LINKS: 'URLs / links', - ResultRoute.URLS: 'URLs / links', - ResultRoute.INTERESTING_URLS: 'URLs / links', + ResultRoute.URLS: 'URLs', ResultRoute.PEOPLE: 'People', ResultRoute.BREACHES: 'Breaches', } @@ -80,7 +78,7 @@ def test_readme_matches_declared_source_contracts() -> None: documented = _documented_source_contracts(readme) declared = _declared_source_contracts() - assert '| Source | Subdomains | Emails | IPs | ASNs | URLs / links | People | Breaches |' in readme + assert '| Source | Subdomains | Emails | IPs | ASNs | URLs | People | Breaches |' in readme assert len(declared) == 56 assert len(documented) == 56 assert documented == declared @@ -133,6 +131,6 @@ def test_readme_explains_jsonl_record_and_structured_value_parsing() -> None: assert '{"sources":[],"type":"hostname","value":"api.example.com"}' in readme assert 'select(.type == "dns-recursive-finding") | .value | fromjson' in readme - assert 'JSONL is easy to stream for simple findings, but it is not uniformly self-describing.' in readme + assert 'JSONL is easy to stream one record at a time.' in readme assert '`person`, `infostealer`, `shodan`, and `takeover`' in readme - assert 'JSONL does not include source execution records. Finding records include source attribution' in readme + assert 'The summary preserves the evidence status, source and action outcomes' in readme diff --git a/tests/test_rest_api.py b/tests/test_rest_api.py deleted file mode 100644 index 0b8111b2..00000000 --- a/tests/test_rest_api.py +++ /dev/null @@ -1,358 +0,0 @@ -from argparse import Namespace - -import pytest -from fastapi.testclient import TestClient - -from theHarvester.lib.api import api -from theHarvester.lib.core import Core - - -@pytest.fixture(autouse=True) -def reset_rate_limiter() -> None: - api.limiter.reset() - yield - api.limiter.reset() - - -def test_query_expands_source_capability(monkeypatch) -> None: - captured: list[tuple[Namespace, bool]] = [] - - async def fake_start( - args: Namespace, - *, - persist_completed_result: bool = False, - include_breaches: bool = False, - ): - captured.append((args, persist_completed_result)) - assert include_breaches is True - return ([], [], [], [], [], [], [], [], [], []) - - monkeypatch.setattr(api.__main__, 'start', fake_start) - - response = TestClient(api.app).get('/query?domain=example.test&source=subdomains') - - assert response.status_code == 200 - assert captured[0][0].source == ','.join(Core.expand_source_selection('subdomains')) - assert captured[0][1] is True - - -def test_query_allows_api_scan_of_operator_selected_private_target(monkeypatch) -> None: - captured: list[Namespace] = [] - - async def fake_start( - args: Namespace, - *, - persist_completed_result: bool = False, - include_breaches: bool = False, - ): - captured.append(args) - return ([], [], [], [], [], [], [], [], [], []) - - monkeypatch.setattr(api.__main__, 'start', fake_start) - - response = TestClient(api.app).get('/query?domain=192.0.2.8&source=certspotter&api_scan=true') - - assert response.status_code == 200 - assert captured[0].domain == '192.0.2.8' - assert captured[0].api_scan is True - - -def test_query_forwards_bounded_recursive_dns_options(monkeypatch) -> None: - captured: list[Namespace] = [] - - async def fake_start( - args: Namespace, - *, - persist_completed_result: bool = False, - include_breaches: bool = False, - ): - captured.append(args) - return ([], [], [], [], [], [], [], [], [], []) - - monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret') - monkeypatch.setattr(api.__main__, 'start', fake_start) - - response = TestClient(api.app).get( - '/query?domain=example.test&source=certspotter&dns_recursive_depth=2' - '&dns_recursive_query_limit=321&dns_recursive_runtime_seconds=4.5' - '&dns_resolve=192.0.2.53,192.0.2.54,192.0.2.55', - headers={'X-API-Key': 'operator-secret'}, - ) - - assert response.status_code == 200 - assert captured[0].dns_recursive_depth == 2 - assert captured[0].dns_recursive_query_limit == 321 - assert captured[0].dns_recursive_runtime_seconds == 4.5 - - -def test_query_requires_operator_key_for_recursive_dns(monkeypatch) -> None: - async def unexpected_start(*_args, **_kwargs): - raise AssertionError('enumeration must not start') - - monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False) - monkeypatch.setattr(api.__main__, 'start', unexpected_start) - - response = TestClient(api.app).get( - '/query?domain=example.test&source=certspotter&dns_recursive_depth=1&dns_resolve=192.0.2.53,192.0.2.54,192.0.2.55' - ) - - assert response.status_code == 503 - assert response.json()['detail'] == 'THEHARVESTER_API_KEY is not configured' - - -def test_query_documents_safe_recursive_dns_query_default() -> None: - query_operation = api.app.openapi()['paths']['/query']['get'] - query_limit = next( - parameter for parameter in query_operation['parameters'] if parameter['name'] == 'dns_recursive_query_limit' - ) - - assert query_limit['schema']['default'] == 3_000 - - -def test_query_documents_proxy_and_direct_action_scope() -> None: - parameters = {parameter['name']: parameter for parameter in api.app.openapi()['paths']['/query']['get']['parameters']} - - assert parameters['proxies']['description'] == ( - 'Use proxies.yaml for supported discovery-source and takeover requests.' - ) - assert 'using configured proxies when enabled' in parameters['take_over']['description'] - assert parameters['api_scan']['description'] == ( - 'Check common API paths with GET, HEAD, and OPTIONS. Requests follow redirects.' - ) - assert parameters['dns_server']['description'] == ( - 'Accepted for compatibility but currently unused; use dns_resolve to select resolvers.' - ) - assert parameters['dns_lookup']['description'] == ( - 'Perform PTR lookups across the /24 network containing each discovered IPv4 address. ' - 'This sends active DNS queries.' - ) - assert parameters['source']['description'] == ( - 'Source names or source capabilities to query. Multiple capabilities select the union of matching sources; ' - 'they do not filter returned fields.' - ) - assert parameters['filename']['description'] == ( - 'Write uniquely prefixed server-side XML, JSON, and JSONL files using NAME as the filename suffix.' - ) - - -@pytest.mark.parametrize('runtime_seconds', ['nan', 'inf']) -def test_query_rejects_non_finite_recursive_dns_runtime(monkeypatch, runtime_seconds: str) -> None: - async def unexpected_start(*_args, **_kwargs): - raise AssertionError('enumeration must not start') - - monkeypatch.setattr(api.__main__, 'start', unexpected_start) - - response = TestClient(api.app).get( - f'/query?domain=example.test&source=certspotter&dns_recursive_runtime_seconds={runtime_seconds}' - ) - - assert response.status_code == 422 - - -def test_query_rejects_recursive_dns_without_three_distinct_resolvers(monkeypatch) -> None: - async def unexpected_start(*_args, **_kwargs): - raise AssertionError('enumeration must not start') - - monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret') - monkeypatch.setattr(api.__main__, 'start', unexpected_start) - - response = TestClient(api.app).get( - '/query?domain=example.test&source=certspotter&dns_recursive_depth=1&dns_resolve=192.0.2.53,192.0.2.54', - headers={'X-API-Key': 'operator-secret'}, - ) - - assert response.status_code == 400 - assert response.json()['detail'] == 'recursive DNS requires exactly three distinct resolver IPs' - - -def test_query_unions_capabilities_and_explicit_sources(monkeypatch) -> None: - captured: list[tuple[Namespace, bool]] = [] - - async def fake_start( - args: Namespace, - *, - persist_completed_result: bool = False, - include_breaches: bool = False, - ): - captured.append((args, persist_completed_result)) - assert include_breaches is True - return ([], [], [], [], [], [], [], [], [], []) - - monkeypatch.setattr(api.__main__, 'start', fake_start) - - response = TestClient(api.app).get('/query?domain=example.test&source=emails&source=certspotter') - - assert response.status_code == 200 - expected_sources = Core.expand_source_selection('emails,certspotter') - assert captured[0][0].source == ','.join(expected_sources) - assert captured[0][1] is True - - -def test_query_rejects_unknown_source_or_capability(monkeypatch) -> None: - async def unexpected_start(_args: Namespace): - raise AssertionError('enumeration must not start') - - monkeypatch.setattr(api.__main__, 'start', unexpected_start) - - response = TestClient(api.app).get('/query?domain=example.test&source=unknown') - - assert response.status_code == 400 - assert response.json()['detail'].startswith("Source 'unknown' is not supported") - - -@pytest.mark.parametrize('source', ['hibpverified', 'breaches', 'emails', 'all']) -def test_query_requires_operator_key_when_selection_includes_verified_hibp(monkeypatch, source: str) -> None: - async def unexpected_start( - _args: Namespace, - *, - persist_completed_result: bool = False, - include_breaches: bool = False, - ): - raise AssertionError('enumeration must not start') - - monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False) - monkeypatch.setattr(api.__main__.Core, 'hibpverified_key', lambda: 'provider-secret') - monkeypatch.setattr(api.__main__, 'start', unexpected_start) - - response = TestClient(api.app).get(f'/query?domain=example.test&source={source}') - - assert response.status_code == 503 - assert response.json()['detail'] == 'THEHARVESTER_API_KEY is not configured' - - -def test_authenticated_query_returns_verified_hibp_emails_and_breaches(monkeypatch) -> None: - captured: list[Namespace] = [] - - async def fake_start( - args: Namespace, - *, - persist_completed_result: bool = False, - include_breaches: bool = False, - ): - captured.append(args) - assert persist_completed_result is True - assert include_breaches is True - return ([], [], [], [], [], [], [], ['alice@example.test'], [], ['ExampleBreach']) - - monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret') - monkeypatch.setattr(api.__main__.Core, 'hibpverified_key', lambda: 'provider-secret') - monkeypatch.setattr(api.__main__, 'start', fake_start) - - response = TestClient(api.app).get( - '/query?domain=example.test&source=hibpverified', - headers={'X-API-Key': 'operator-secret'}, - ) - - assert response.status_code == 200 - assert captured[0].source == 'hibpverified' - assert response.json()['emails'] == ['alice@example.test'] - assert response.json()['breaches'] == ['ExampleBreach'] - - -def test_authenticated_query_includes_verified_hibp_from_capability_selection(monkeypatch) -> None: - captured: list[Namespace] = [] - - async def fake_start( - args: Namespace, - *, - persist_completed_result: bool = False, - include_breaches: bool = False, - ): - captured.append(args) - assert include_breaches is True - return ([], [], [], [], [], [], [], [], [], []) - - monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret') - monkeypatch.setattr(api.__main__.Core, 'hibpverified_key', lambda: 'provider-secret') - monkeypatch.setattr(api.__main__, 'start', fake_start) - - response = TestClient(api.app).get( - '/query?domain=example.test&source=breaches', - headers={'X-API-Key': 'operator-secret'}, - ) - - assert response.status_code == 200 - assert captured[0].source == 'haveibeenpwned,hibpverified,leaklookup' - - -def test_query_requires_operator_auth_for_configured_leaklookup(monkeypatch) -> None: - async def unexpected_start(*_args, **_kwargs): - raise AssertionError('collection must not start without operator authentication') - - monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret') - monkeypatch.setattr(api.__main__.Core, 'leaklookup_key', lambda: 'provider-secret') - monkeypatch.setattr(api.__main__, 'start', unexpected_start) - - response = TestClient(api.app).get('/query?domain=example.test&source=leaklookup') - - assert response.status_code == 401 - - -@pytest.mark.parametrize('source', ['dehashed', 'emails']) -def test_query_requires_operator_auth_for_configured_dehashed(monkeypatch, source) -> None: - async def unexpected_start(*_args, **_kwargs): - raise AssertionError('collection must not start without operator authentication') - - monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret') - monkeypatch.setattr(api.__main__.Core, 'dehashed_key', lambda: 'provider-secret') - monkeypatch.setattr(api.__main__, 'start', unexpected_start) - - response = TestClient(api.app).get(f'/query?domain=example.test&source={source}') - - assert response.status_code == 401 - - -@pytest.mark.parametrize('dehashed_key', [None, '', ' ']) -def test_query_skips_operator_auth_when_dehashed_key_is_blank(monkeypatch, dehashed_key) -> None: - captured: list[Namespace] = [] - - async def fake_start( - args: Namespace, - *, - persist_completed_result: bool = False, - include_breaches: bool = False, - ): - captured.append(args) - return ([], [], [], [], [], [], [], [], [], []) - - monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False) - monkeypatch.setattr(api.__main__.Core, 'dehashed_key', lambda: dehashed_key) - monkeypatch.setattr(api.__main__, 'start', fake_start) - - response = TestClient(api.app).get('/query?domain=example.test&source=dehashed') - - assert response.status_code == 200 - assert captured[0].source == 'dehashed' - - -@pytest.mark.parametrize('leaklookup_key', [None, '', ' ']) -def test_query_skips_operator_auth_when_credentialed_provider_keys_are_blank(monkeypatch, leaklookup_key) -> None: - captured: list[Namespace] = [] - - async def fake_start( - args: Namespace, - *, - persist_completed_result: bool = False, - include_breaches: bool = False, - ): - captured.append(args) - return ([], [], [], [], [], [], [], [], [], []) - - monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False) - monkeypatch.setattr(api.__main__.Core, 'hibpverified_key', lambda: None) - monkeypatch.setattr(api.__main__.Core, 'leaklookup_key', lambda: leaklookup_key) - monkeypatch.setattr(api.__main__, 'start', fake_start) - - response = TestClient(api.app).get('/query?domain=example.test&source=breaches') - - assert response.status_code == 200 - assert captured[0].source == 'haveibeenpwned,hibpverified,leaklookup' - - -def test_sources_advertises_authenticated_verified_hibp(monkeypatch) -> None: - monkeypatch.setattr(api.__main__.Core, 'get_supportedengines', lambda: ['crtsh', 'hibpverified']) - - response = TestClient(api.app).get('/sources') - - assert response.status_code == 200 - assert response.json() == {'sources': ['crtsh', 'hibpverified']} diff --git a/tests/test_rest_completed_runs.py b/tests/test_rest_completed_runs.py deleted file mode 100644 index 628f113a..00000000 --- a/tests/test_rest_completed_runs.py +++ /dev/null @@ -1,94 +0,0 @@ -from datetime import UTC, datetime, timedelta -from uuid import UUID - -import pytest -from fastapi.testclient import TestClient - -from theHarvester.lib.api import api -from theHarvester.lib.completed_result import CompletedResult -from theHarvester.lib.database import ResultStore - - -def completed_result(run_id: str, completed_at: datetime) -> CompletedResult: - return CompletedResult.finish( - run_id=UUID(run_id), - target='example.com', - started_at=completed_at - timedelta(minutes=1), - completed_at=completed_at, - groups={'hostname': ['www.example.com']}, - ) - - -@pytest.mark.asyncio -async def test_authenticated_operator_lists_recent_completed_runs(tmp_path, monkeypatch) -> None: - monkeypatch.setattr('theHarvester.lib.database._DEFAULT_DATABASE', tmp_path / 'stash.sqlite') - monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-secret') - store = ResultStore() - await store.initialize() - older = completed_result('11111111-1111-4111-8111-111111111111', datetime(2026, 8, 6, 12, 0, tzinfo=UTC)) - newer = completed_result('22222222-2222-4222-8222-222222222222', datetime(2026, 8, 6, 13, 0, tzinfo=UTC)) - offset_older = completed_result( - '55555555-5555-4555-8555-555555555555', datetime.fromisoformat('2026-08-06T14:30:00+05:00') - ) - await store.save_run(older) - await store.save_run(newer) - await store.save_run(offset_older) - - response = TestClient(api.app).get('/runs?limit=1', headers={'X-API-Key': 'test-secret'}) - - assert response.status_code == 200 - assert response.json() == [ - { - 'run_id': str(newer.run_id), - 'target': 'example.com', - 'started_at': '2026-08-06T12:59:00Z', - 'completed_at': '2026-08-06T13:00:00Z', - 'result_count': 1, - } - ] - - -@pytest.mark.asyncio -async def test_authenticated_operator_gets_one_completed_run(tmp_path, monkeypatch) -> None: - monkeypatch.setattr('theHarvester.lib.database._DEFAULT_DATABASE', tmp_path / 'stash.sqlite') - monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-secret') - store = ResultStore() - await store.initialize() - result = CompletedResult.finish( - run_id=UUID('33333333-3333-4333-8333-333333333333'), - target='example.com', - started_at=datetime(2026, 8, 6, 14, 0, tzinfo=UTC), - completed_at=datetime(2026, 8, 6, 14, 1, tzinfo=UTC), - groups={'email': ['security@example.com'], 'hostname': ['www.example.com']}, - ) - await store.save_run(result) - client = TestClient(api.app) - - response = client.get(f'/runs/{result.run_id}', headers={'X-API-Key': 'test-secret'}) - missing = client.get('/runs/44444444-4444-4444-8444-444444444444', headers={'X-API-Key': 'test-secret'}) - - assert response.status_code == 200 - assert response.json() == { - 'run_id': str(result.run_id), - 'target': 'example.com', - 'started_at': '2026-08-06T14:00:00Z', - 'completed_at': '2026-08-06T14:01:00Z', - 'result_count': 2, - 'results': [ - {'type': 'email', 'value': 'security@example.com'}, - {'type': 'hostname', 'value': 'www.example.com'}, - ], - } - assert missing.status_code == 404 - assert missing.json() == {'detail': 'Completed run not found'} - - -def test_completed_run_routes_fail_closed_without_operator_key(monkeypatch) -> None: - monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False) - client = TestClient(api.app) - - list_response = client.get('/runs') - detail_response = client.get('/runs/33333333-3333-4333-8333-333333333333') - - assert list_response.status_code == 503 - assert detail_response.status_code == 503 diff --git a/tests/test_rest_terminal_evidence.py b/tests/test_rest_terminal_evidence.py deleted file mode 100644 index e97ec352..00000000 --- a/tests/test_rest_terminal_evidence.py +++ /dev/null @@ -1,26 +0,0 @@ -from argparse import Namespace - -from fastapi.testclient import TestClient - -from theHarvester.lib.api import api - - -def test_query_requests_completed_result_persistence(monkeypatch) -> None: - persistence_flags: list[bool] = [] - - async def fake_start( - args: Namespace, - *, - persist_completed_result: bool = False, - include_breaches: bool = False, - ): - persistence_flags.append(persist_completed_result) - assert include_breaches is True - return ([], [], [], [], [], [], [], [], [], []) - - monkeypatch.setattr(api.__main__, 'start', fake_start) - - response = TestClient(api.app).get('/query?domain=example.test&source=crtsh') - - assert response.status_code == 200 - assert persistence_flags == [True] diff --git a/tests/test_restful_harvest.py b/tests/test_restful_harvest.py new file mode 100644 index 00000000..58b3d7f6 --- /dev/null +++ b/tests/test_restful_harvest.py @@ -0,0 +1,15 @@ +import sys + +import pytest + +from theHarvester import restfulHarvest + + +def test_help_does_not_offer_rate_limit_configuration(monkeypatch, capsys): + monkeypatch.setattr(sys, 'argv', ['restfulHarvest', '--help']) + + with pytest.raises(SystemExit) as exit_info: + restfulHarvest.main() + + assert exit_info.value.code == 0 + assert '--rate-limit' not in capsys.readouterr().out diff --git a/tests/test_screenshot.py b/tests/test_screenshot.py index a496e250..5013ccd8 100644 --- a/tests/test_screenshot.py +++ b/tests/test_screenshot.py @@ -24,10 +24,35 @@ def test_screenshot_output_separator_matches_platform( assert ScreenShotter('screenshots').slash == expected_separator +@pytest.mark.parametrize( + ('target', 'filename'), + [ + ('https://www.example.com:8443/login?next=/admin', 'www.example.com_8443.png'), + ('2001:db8::1', '2001_db8__1.png'), + ], +) +def test_screenshot_path_uses_the_target_host_and_port(tmp_path: Path, target: str, filename: str) -> None: + screenshotter = ScreenShotter(str(tmp_path)) + + assert screenshotter.screenshot_path(target) == tmp_path / filename + + +@pytest.mark.parametrize( + ('target', 'response_url', 'request_url'), + [ + ('www.example.com', 'https://www.example.com/landing', 'https://www.example.com'), + ('2001:db8::1', 'https://[2001:db8::1]/', 'https://[2001:db8::1]'), + ], +) @pytest.mark.asyncio -async def test_visit_prefers_https_and_returns_final_www_url(monkeypatch: pytest.MonkeyPatch) -> None: +async def test_visit_prefers_https_and_normalizes_the_target( + monkeypatch: pytest.MonkeyPatch, + target: str, + response_url: str, + request_url: str, +) -> None: response = MagicMock() - response.url = 'https://www.example.com/landing' + response.url = response_url response.__aenter__ = AsyncMock(return_value=response) response.__aexit__ = AsyncMock(return_value=False) response.text = AsyncMock(return_value='reachable') @@ -38,10 +63,10 @@ async def test_visit_prefers_https_and_returns_final_www_url(monkeypatch: pytest monkeypatch.setattr(screenshot_module.aiohttp, 'ClientSession', MagicMock(return_value=session)) monkeypatch.setattr(screenshot_module.aiohttp, 'TCPConnector', MagicMock()) monkeypatch.setattr(screenshot_module.ssl, 'create_default_context', MagicMock()) - result = await ScreenShotter.visit('www.example.com') + result = await ScreenShotter.visit(target) - assert result == ('https://www.example.com/landing', 'reachable') - assert session.get.call_args.args[0] == 'https://www.example.com' + assert result == (response_url, 'reachable') + assert session.get.call_args.args[0] == request_url @pytest.mark.asyncio @@ -67,8 +92,44 @@ async def test_visit_falls_back_to_http_when_https_is_unreachable(monkeypatch: p ] +@pytest.mark.parametrize( + ('target', 'normalized_url', 'filename'), + [ + ('www.example.com', 'https://www.example.com', 'www.example.com.png'), + ('2001:db8::1', 'https://[2001:db8::1]', '2001_db8__1.png'), + ], +) @pytest.mark.asyncio -async def test_take_screenshot_preserves_www_hostname( +async def test_take_screenshot_normalizes_the_target( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + target: str, + normalized_url: str, + filename: str, +) -> None: + page = AsyncMock() + context = AsyncMock() + context.new_page.return_value = page + browser = AsyncMock() + browser.new_context.return_value = context + playwright = MagicMock() + playwright.chromium.launch = AsyncMock(return_value=browser) + manager = MagicMock() + manager.__aenter__ = AsyncMock(return_value=playwright) + manager.__aexit__ = AsyncMock(return_value=False) + monkeypatch.setattr(screenshot_module, 'async_playwright', MagicMock(return_value=manager)) + monkeypatch.setattr(screenshot_module.os, 'chmod', MagicMock()) + + captured_url = await ScreenShotter(str(tmp_path)).take_screenshot(target) + + assert captured_url == normalized_url + page.goto.assert_awaited_once_with(normalized_url, timeout=35000) + screenshot_path = page.screenshot.await_args.kwargs['path'] + assert Path(screenshot_path) == tmp_path / filename + + +@pytest.mark.asyncio +async def test_take_screenshot_can_name_the_artifact_for_the_authorized_subject( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, ) -> None: @@ -84,13 +145,15 @@ async def test_take_screenshot_preserves_www_hostname( manager.__aexit__ = AsyncMock(return_value=False) monkeypatch.setattr(screenshot_module, 'async_playwright', MagicMock(return_value=manager)) monkeypatch.setattr(screenshot_module.os, 'chmod', MagicMock()) + output_path = tmp_path / 'authorized.example.com.png' - captured_url = await ScreenShotter(str(tmp_path)).take_screenshot('www.example.com') + captured_url = await ScreenShotter(str(tmp_path)).take_screenshot( + 'https://login.example.net/session', + output_path=output_path, + ) - assert captured_url == 'https://www.example.com' - page.goto.assert_awaited_once_with('https://www.example.com', timeout=35000) - screenshot_path = page.screenshot.await_args.kwargs['path'] - assert screenshot_path.endswith('www.example.com.png') + assert captured_url == 'https://login.example.net/session' + assert page.screenshot.await_args.kwargs['path'] == output_path @pytest.mark.asyncio diff --git a/tests/test_security.py b/tests/test_security.py index 4904d57c..0f214e21 100644 --- a/tests/test_security.py +++ b/tests/test_security.py @@ -1,7 +1,6 @@ import os import re import tempfile -from pathlib import Path from unittest.mock import AsyncMock import pytest @@ -13,64 +12,10 @@ from theHarvester.__main__ import sanitize_filename, sanitize_for_xml class TestCORSConfiguration: """Test CORS security configuration.""" - def test_cors_does_not_allow_credentials_with_wildcard_origins(self): - """ - Security Test: CORS should not allow credentials with wildcard origins. - - This prevents credential theft attacks where any origin can make - authenticated requests to the API. - """ + def test_api_does_not_enable_cross_origin_requests(self): from theHarvester.lib.api.api import app - # Find CORS middleware in the app - cors_middleware = None - for middleware in app.user_middleware: - if 'CORSMiddleware' in str(middleware.cls): - cors_middleware = middleware - break - - assert cors_middleware is not None, 'CORS middleware should be configured' - - # Check that if allow_origins contains '*', allow_credentials must be False - # Access kwargs from the middleware - options = cors_middleware.kwargs - allow_origins = options.get('allow_origins', []) - allow_credentials = options.get('allow_credentials', False) - - if isinstance(allow_origins, (list, tuple, set)) and '*' in allow_origins: - assert ( - allow_credentials is False - ), 'CRITICAL: CORS must not allow credentials with wildcard origins (CVE risk)' - - def test_cors_restricts_http_methods(self): - """ - Security Test: CORS should restrict HTTP methods to only what's needed. - - Reduces attack surface by limiting available methods. - """ - from theHarvester.lib.api.api import app - - cors_middleware = None - for middleware in app.user_middleware: - if 'CORSMiddleware' in str(middleware.cls): - cors_middleware = middleware - break - - assert cors_middleware is not None - - options = cors_middleware.kwargs - allow_methods = options.get('allow_methods', []) - - # Should not allow all methods - assert allow_methods != ['*'], 'CORS should restrict HTTP methods, not allow all (*)' - - # Should only allow necessary methods (GET, POST for this API) - if isinstance(allow_methods, list): - dangerous_methods = {'DELETE', 'PUT', 'PATCH', 'TRACE', 'CONNECT'} - allowed_set = {m.upper() for m in allow_methods} - assert not ( - allowed_set & dangerous_methods - ), f'Unnecessary HTTP methods detected: {allowed_set & dangerous_methods}' + assert all('CORSMiddleware' not in str(middleware.cls) for middleware in app.user_middleware) class TestXMLInjectionPrevention: @@ -153,8 +98,7 @@ class TestInformationDisclosure: Stack traces can reveal sensitive information about the system. """ - # Test the /sources endpoint with a simulated error condition - response = client.get('/sources') + response = client.get('/api/v1/sources') # Even if there's an error, traceback should not be in response if response.status_code >= 400: @@ -163,20 +107,19 @@ class TestInformationDisclosure: assert 'Traceback' not in str(response_data), 'Traceback text found in response' assert 'File "' not in str(response_data), 'File paths exposed in response' - def test_error_responses_do_not_leak_internal_paths(self, client, monkeypatch): + def test_error_responses_do_not_leak_internal_paths(self, client, tmp_path, monkeypatch): """ Security Test: Error messages should not reveal internal file paths. """ - start = AsyncMock(return_value=([], [], [], [], [], [], [], [], [])) fetch_all = AsyncMock(side_effect=AssertionError('API security test attempted a provider request')) - monkeypatch.setattr('theHarvester.lib.api.api.__main__.start', start) + monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) monkeypatch.setattr('theHarvester.lib.core.AsyncFetcher.fetch_all', fetch_all) - # Try various endpoints - endpoints = ['/sources', '/dnsbrute?domain=test', '/query?domain=test&source=baidu'] + endpoints = ['/api/v1/sources', '/api/v1/runs/not-found'] for endpoint in endpoints: - response = client.get(endpoint) + response = client.get(endpoint, headers={'X-API-Key': 'operator-secret'}) response_text = str(response.json() if response.status_code != 200 else {}) # Check for common path leakage patterns @@ -193,7 +136,6 @@ class TestInformationDisclosure: matches = re.findall(pattern, response_text) assert not matches, f'Internal path leaked in {endpoint}: {matches}' - assert start.await_count == 2 fetch_all.assert_not_awaited() def test_debug_mode_does_not_expose_sensitive_info(self, client, monkeypatch): @@ -204,7 +146,7 @@ class TestInformationDisclosure: monkeypatch.setenv('DEBUG', '1') # Make request that might trigger an error - response = client.get('/dnsbrute?domain=') # Invalid request + response = client.get('/api/v1/runs/not-found') if response.status_code >= 400: response_data = response.json() @@ -212,8 +154,8 @@ class TestInformationDisclosure: assert 'traceback' not in response_data, 'DEBUG mode exposes tracebacks to clients' -class TestAdditionalAPIAuthentication: - """Test authentication and error handling for protected additional API routes.""" +class TestAPIAuthentication: + """Test authentication and error handling for the versioned API.""" @pytest.fixture def client(self): @@ -222,40 +164,48 @@ class TestAdditionalAPIAuthentication: return TestClient(app) - def test_additional_endpoints_fail_closed_without_configured_api_key(self, client, monkeypatch): + def test_api_fails_closed_without_configured_api_key(self, client, monkeypatch): monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False) + monkeypatch.delenv('THEHARVESTER_API_KEY_FILE', raising=False) - response = client.post('/additional/all', json={'domain': 'example.com'}) + response = client.get('/api/v1/sources') assert response.status_code == 503 - def test_additional_endpoints_reject_missing_or_invalid_api_key(self, client, monkeypatch): + def test_api_key_can_be_read_from_a_docker_secret_file(self, client, tmp_path, monkeypatch): + secret = tmp_path / 'operator-api-key' + secret.write_text('test-secret\n', encoding='utf-8') + monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False) + monkeypatch.setenv('THEHARVESTER_API_KEY_FILE', str(secret)) + + response = client.get('/api/v1/sources', headers={'X-API-Key': 'test-secret'}) + + assert response.status_code == 200 + + def test_api_rejects_missing_or_invalid_api_key(self, client, monkeypatch): monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-secret') - missing_response = client.post('/additional/all', json={'domain': 'example.com'}) - invalid_response = client.post('/additional/all', headers={'X-API-Key': 'wrong'}, json={'domain': 'example.com'}) + missing_response = client.get('/api/v1/sources') + invalid_response = client.get('/api/v1/sources', headers={'X-API-Key': 'wrong'}) assert missing_response.status_code == 401 assert invalid_response.status_code == 401 - def test_additional_endpoints_do_not_expose_internal_errors(self, client, monkeypatch): - from theHarvester.lib.api import additional_endpoints + def test_api_does_not_expose_internal_errors(self, monkeypatch): + from theHarvester.lib.api import api + from theHarvester.lib.api.run_store import RunStore - class FailingAdditionalAPIs: - def __init__(self, domain, api_keys): - self.domain = domain - self.api_keys = api_keys - - async def process(self): - raise RuntimeError('/home/user/project/secret.py:123 internal failure') + async def fail(_self): + raise RuntimeError('/home/user/project/secret.py:123 internal failure') monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-secret') - monkeypatch.setattr(additional_endpoints, 'AdditionalAPIs', FailingAdditionalAPIs) + monkeypatch.setattr(RunStore, 'list_runs', fail) + client = TestClient(api.app, raise_server_exceptions=False) - response = client.post('/additional/all', headers={'X-API-Key': 'test-secret'}, json={'domain': 'example.com'}) + response = client.get('/api/v1/runs', headers={'X-API-Key': 'test-secret'}) assert response.status_code == 500 - response_text = str(response.json()) + response_text = response.text assert 'internal failure' not in response_text assert '/home/user/project' not in response_text @@ -403,24 +353,11 @@ class TestSecurityBestPractices: real_matches = [ m for m in matches - if 'example' not in m.lower() - and 'your_' not in m.lower() - and '""' not in m - and "''" not in m + if 'example' not in m.lower() and 'your_' not in m.lower() and '""' not in m and "''" not in m ] assert not real_matches, f'Potential hardcoded secret in {file_path}: {real_matches}' - def test_api_has_rate_limiting(self): - """ - Security Test: Verify API endpoints have rate limiting enabled. - """ - from theHarvester.lib.api.api import app - - # Check that rate limiting is configured - assert hasattr(app.state, 'limiter'), 'Rate limiter not configured' - assert app.state.limiter is not None, 'Rate limiter is None' - - def test_sensitive_endpoints_require_validation(self): + def test_sensitive_endpoints_require_validation(self, monkeypatch): """ Security Test: Ensure sensitive endpoints validate input. """ @@ -428,24 +365,15 @@ class TestSecurityBestPractices: from theHarvester.lib.api.api import app + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-secret') client = TestClient(app) + headers = {'X-API-Key': 'test-secret'} - # Test that endpoints reject invalid input - # Note: The /query endpoint requires 'source' as a list parameter - test_cases = [ - ('/dnsbrute?domain=', 400), # Empty domain should be rejected - ('/dnsbrute?domain=a', 422), # Too short domain should be rejected by FastAPI validation - ] + missing_target = client.post('/api/v1/runs', headers=headers, json={'sources': ['crtsh']}) + empty_sources = client.post('/api/v1/runs', headers=headers, json={'target': 'example.test', 'sources': []}) - for endpoint, expected_status in test_cases: - response = client.get(endpoint) - assert ( - response.status_code >= 400 - ), f'Endpoint {endpoint} should reject invalid input (got {response.status_code})' - - # Test query endpoint with proper parameter format but invalid domain - response = client.get('/query?domain=a&source=baidu') # Too short domain - assert response.status_code == 422 + assert missing_target.status_code == 422 + assert empty_sources.status_code == 422 if __name__ == '__main__': diff --git a/theHarvester/__main__.py b/theHarvester/__main__.py index 5cee3309..f70cbc7d 100644 --- a/theHarvester/__main__.py +++ b/theHarvester/__main__.py @@ -1,5 +1,7 @@ import argparse import asyncio +import hashlib +import inspect import json import logging import os @@ -8,13 +10,14 @@ import secrets import string import sys import time -import traceback -from collections.abc import Awaitable, Callable, Iterable +from collections.abc import Awaitable, Callable, Iterable, Mapping from contextlib import AsyncExitStack from datetime import UTC, datetime from ipaddress import ip_address +from pathlib import Path from typing import Any, cast -from uuid import uuid4 +from urllib.parse import urlsplit +from uuid import UUID, uuid4 import anyio import netaddr @@ -84,6 +87,7 @@ from theHarvester.discovery import ( ) from theHarvester.discovery.constants import MissingKey from theHarvester.lib import hostchecker +from theHarvester.lib.active_evidence import ActionExecution, ActiveEvidence, ArtifactReference from theHarvester.lib.completed_result import ( EXECUTION_STATUSES, CompletedResult, @@ -102,14 +106,20 @@ from theHarvester.lib.enumeration import ( EnumerationOptions, ) from theHarvester.lib.hostnames import normalize_scoped_hostname -from theHarvester.lib.output import configure_logging, output_logger, print_linkedin_sections, print_section, sorted_unique +from theHarvester.lib.output import configure_logging, output_logger, print_linkedin_people, print_section, sorted_unique from theHarvester.lib.recursive_dns import ( DEFAULT_RECURSIVE_DNS_QUERY_LIMIT, RecursiveDNSLimits, - RecursiveDNSResult, discover_recursive_dns, ) -from theHarvester.lib.source_catalog import SOURCE_SPECS, ActivityClass, ResultRoute, SourceSpec, get_source_spec +from theHarvester.lib.resolver_selection import DEFAULT_DNS_RESOLVERS, normalize_resolver_addresses +from theHarvester.lib.source_catalog import ( + SOURCE_SPECS, + ActivityClass, + ResultRoute, + SourceSpec, + get_source_spec, +) from theHarvester.screenshot.screenshot import ScreenShotter logger = logging.getLogger(__name__) @@ -168,6 +178,8 @@ async def start( include_breaches: bool = False, return_completed_result: bool = False, return_dns_brute_result: bool = False, + result_database: str | Path | None = None, + completed_run_id: UUID | None = None, ): """Main program function""" parser = argparse.ArgumentParser( @@ -213,7 +225,7 @@ async def start( parser.add_argument( '-e', '--dns-server', - help='Accepted for compatibility but currently unused; use --dns-resolve to select resolvers.', + help='Accepted for compatibility but currently unused; use --dns-resolvers to select resolvers.', ) parser.add_argument( '-t', @@ -225,11 +237,24 @@ async def start( parser.add_argument( '-r', '--dns-resolve', - help='Resolve discovered hostnames. Pass resolver IPs or a resolver file; omit the value to use defaults.', + help=( + 'Resolve discovered hostnames. Pass comma-separated resolver IPs or a text file with one IP per line; ' + 'omit the value to use defaults.' + ), default='', type=str, nargs='?', ) + parser.add_argument( + '--dns-resolvers', + dest='dns_resolver_input', + help=( + 'Select resolver IPs for DNS actions without enabling hostname resolution. ' + 'Pass comma-separated IPs or a text file with one IP per line.' + ), + default='', + metavar='IPS_OR_FILE', + ) parser.add_argument( '-n', '--dns-lookup', @@ -306,12 +331,12 @@ async def start( if rest_args.source and rest_args.source == 'getsources': return list(sorted(Core.get_supportedengines())) args = EnumerationOptions.from_namespace(rest_args) + filename = args.filename if args.dns_brute: dnsbrute = (args.dns_brute, return_dns_brute_result) else: dnsbrute = (args.dns_brute, False) # We need to make sure the filename is random as to not overwrite other files - filename: str = args.filename alphabet = string.ascii_letters + string.digits rest_filename += f'{"".join(secrets.choice(alphabet) for _ in range(32))}_{filename}' if len(filename) != 0 else '' else: @@ -323,7 +348,7 @@ async def start( logger.info('Verbose logging enabled') Core.quiet = getattr(args, 'quiet', False) try: - db = ResultStore() + db = ResultStore() if result_database is None else ResultStore(result_database) await db.initialize() except (AttributeError, OSError, RuntimeError, ValueError) as init_error: if not args.quiet: @@ -345,7 +370,7 @@ async def start( # For relative paths, sanitize the entire filename filename = sanitize_filename(filename) run_started_at = datetime.now(UTC) - run_id = uuid4() + run_id = completed_run_id or uuid4() all_emails: list = [] all_hosts: list = [] @@ -355,39 +380,21 @@ async def start( dnslookup = args.dns_lookup dnsserver = args.dns_server # TODO arg is not used anywhere replace with resolvers wordlist arg dnsresolve dnsresolve: str | None = args.dns_resolve - final_dns_resolver_list = [] - if dnsresolve is not None and len(dnsresolve) > 0: - # Three scenarios: - # 8.8.8.8 - # 1.1.1.1,8.8.8.8 or 1.1.1.1, 8.8.8.8 - # resolvers.txt - if await anyio.Path(dnsresolve).exists(): - async with await anyio.open_file(dnsresolve, encoding='UTF-8') as fp: + final_dns_resolver_list = normalize_resolver_addresses(args.dns_resolvers) if args.dns_resolvers else [] + if args.dns_resolver_input and dnsresolve not in {'', None}: + raise ValueError('Pass resolver values through either --dns-resolvers or --dns-resolve, not both') + resolver_input = args.dns_resolver_input or (dnsresolve if dnsresolve is not None else '') + if resolver_input: + resolver_candidates: list[str] = [] + if await anyio.Path(resolver_input).exists(): + async with await anyio.open_file(resolver_input, encoding='UTF-8') as fp: async for line in fp: - line = line.strip() - if len(line) == 0: - continue - try: - final_dns_resolver_list.append(str(netaddr.IPAddress(line))) - except (netaddr.core.AddrFormatError, ValueError, TypeError) as e: - output_logger.info(f'An exception has occurred while reading from: {dnsresolve}, {e}') - output_logger.info(f'Current line: {line}') + resolver_candidates.append(line) else: - cleaned = dnsresolve.replace(' ', '') - resolver_candidates = cleaned.split(',') if ',' in cleaned else [cleaned] - for item in resolver_candidates: - if len(item) == 0: - continue - try: - # Verify user passed in an IP; this does not validate resolver behavior - final_dns_resolver_list.append(str(netaddr.IPAddress(item))) - except (netaddr.core.AddrFormatError, ValueError, TypeError) as e: - output_logger.info(f'Passed DNS resolver is invalid, skipping: {item} ({e})') - - # if for some reason, there are duplicates - final_dns_resolver_list = sorted(set(final_dns_resolver_list)) - if len(final_dns_resolver_list) == 0: - output_logger.info('No valid DNS resolvers were parsed from --dns-resolve; continuing without custom resolvers.') + resolver_candidates.extend(resolver_input.split(',')) + final_dns_resolver_list = normalize_resolver_addresses(resolver_candidates) + elif dnsresolve is None and not final_dns_resolver_list: + final_dns_resolver_list = list(DEFAULT_DNS_RESOLVERS) recursive_depth = getattr(args, 'dns_recursive_depth', 0) recursive_limits = None @@ -395,7 +402,7 @@ async def start( raise ValueError('--dns-recursive-depth cannot be negative') if recursive_depth > 0: if len(final_dns_resolver_list) != 3: - raise ValueError('--dns-recursive-depth requires --dns-resolve with exactly three resolver vantages') + raise ValueError('--dns-recursive-depth requires exactly three resolver addresses') recursive_limits = RecursiveDNSLimits( depth=recursive_depth, query_limit=getattr(args, 'dns_recursive_query_limit', DEFAULT_RECURSIVE_DNS_QUERY_LIMIT), @@ -418,9 +425,7 @@ async def start( takeover_status = args.take_over use_proxy = args.proxies linkedin_people_list_tracker: list = [] - linkedin_links_tracker: list = [] twitter_people_list_tracker: list = [] - interesting_urls: list = [] total_asns: list = [] all_breaches: list[str] = [] all_frameworks: list[str] = [] @@ -429,19 +434,24 @@ async def start( all_cms: list[str] = [] all_analytics: list[str] = [] endpoints_found: set[str] = set() - screenshot_results: list[str] = [] + screenshot_artifacts: list[ArtifactReference] = [] + screenshot_hostnames: set[str] = set() + screenshot_ip_addresses: set[str] = set() shodan_evidence: list[str] = [] takeover_results: dict[str, list[dict[str, str]]] = {} - recursive_result: RecursiveDNSResult | None = None - linkedin_people_list_tracker = [] - linkedin_links_tracker = [] twitter_people_list_tracker = [] - - interesting_urls = [] total_asns = [] source_executions: list[SourceExecution] = [] observations: set[ResultObservation] = set() + action_executions: list[ActionExecution] = [] + dns_resolution_duration_ms = 0.0 + dns_resolution_ips: set[str] = set() + dns_resolution_completed_count = 0 + dns_resolution_query_error_count = 0 + dns_resolution_error_types: set[str] = set() + dns_resolution_failure_types: set[str] = set() + dns_resolution_cancelled = False def finish_completed_result( *, @@ -451,81 +461,20 @@ async def start( ) -> CompletedResult | None: groups: dict[ResultKind, Iterable[str]] = { 'analytics': map(str, all_analytics), - 'api-endpoint': map(str, endpoints_found), 'asn': map(str, total_asns), 'breach': map(str, all_breaches), 'cms': map(str, all_cms), - 'dns-recursive-finding': ( - ( - json.dumps( - { - 'addresses': list(finding.records.addresses), - 'hostname': finding.hostname, - 'parent': finding.parent, - 'ptrs': list(finding.ptrs), - }, - separators=(',', ':'), - sort_keys=True, - ) - for finding in recursive_result.findings - ) - if recursive_result is not None - else () - ), - 'dns-recursive-classification': ( - ( - json.dumps( - { - 'addressability': classification.addressability.value, - 'addresses': list(classification.records.addresses), - 'cnames': list(classification.records.cnames), - 'hostname': classification.hostname, - 'parent': classification.parent, - 'ptrs': list(classification.ptrs), - }, - separators=(',', ':'), - sort_keys=True, - ) - for classification in recursive_result.classifications - ) - if recursive_result is not None - else () - ), - 'dns-recursive-summary': ( - ( - json.dumps( - { - 'depth_reached': recursive_result.depth_reached, - 'query_count': recursive_result.query_count, - 'stop_reason': recursive_result.stop_reason, - 'zero_yield_batches': recursive_result.zero_yield_batches, - }, - separators=(',', ':'), - sort_keys=True, - ), - ) - if recursive_result is not None - else () - ), 'email': map(str, all_emails), 'framework': map(str, all_frameworks), - 'hostname': _normalize_hosts_for_storage(all_hosts, word), + 'hostname': _normalize_hosts_for_storage(all_hosts, word) | screenshot_hostnames, 'infostealer': ( json.dumps(stealer, ensure_ascii=False, separators=(',', ':'), sort_keys=True) for stealer in all_infostealers ), - 'interesting-url': map(str, interesting_urls), - 'ip-address': _normalize_ip_addresses(all_ip), + 'ip': _normalize_ip_addresses(all_ip) | screenshot_ip_addresses, 'language': map(str, all_languages), - 'linkedin-link': map(str, linkedin_links_tracker), 'linkedin-person': map(str, linkedin_people_list_tracker), 'person': (json.dumps(person, ensure_ascii=False, separators=(',', ':'), sort_keys=True) for person in all_people), 'server': map(str, all_servers), - 'screenshot': map(str, screenshot_results), - 'shodan': shodan_evidence, - 'takeover': ( - json.dumps({'matches': matches, 'url': url}, separators=(',', ':'), sort_keys=True) - for url, matches in takeover_results.items() - ), 'twitter-person': map(str, twitter_people_list_tracker), 'url': map(str, all_urls), 'vhost': map(str, virtual_hosts), @@ -536,7 +485,7 @@ async def start( committed_groups.setdefault(observation.kind, []).append(observation.value) groups = {kind: iter(values) for kind, values in committed_groups.items()} elif extra_hostnames: - groups['hostname'] = _normalize_hosts_for_storage((*all_hosts, *extra_hostnames), word) + groups['hostname'] = _normalize_hosts_for_storage((*all_hosts, *extra_hostnames), word) | screenshot_hostnames try: return CompletedResult.finish( run_id=run_id, @@ -546,6 +495,7 @@ async def start( groups=groups, source_executions=source_executions, observations=observations, + active_evidence=ActiveEvidence(tuple(action_executions)), ) except (ValueError, TypeError) as error: output_logger.info(f'[!] An error occurred while completing the result: {error}') @@ -570,6 +520,81 @@ async def start( ): await completed_result_checkpoint(result) + async def persist_result(completed_result: CompletedResult | None) -> None: + if completed_result is None: + return + try: + await db.save_run(completed_result) + except Exception as error: + output_logger.info(f'[!] An error occurred while storing the completed result: {error}') + + async def checkpoint_action_result( + *, + extra_hostnames: Iterable[str] = (), + virtual_hosts: Iterable[str] = (), + ) -> None: + result = finish_completed_result(extra_hostnames=extra_hostnames, virtual_hosts=virtual_hosts) + if result is None: + return + try: + if completed_result_checkpoint is not None: + await completed_result_checkpoint(result) + except asyncio.CancelledError: + await persist_result(result) + raise + + def record_dns_resolution_execution(*, handler_cancelled: bool = False) -> None: + if dnsresolve == '': + return + if dnsresolve is not None and not final_dns_resolver_list: + action_executions.append( + ActionExecution.finish( + action='dns-resolve', + status='skipped', + duration_ms=0, + groups={}, + stop_reason='no-valid-resolvers', + ) + ) + return + if handler_cancelled and not ( + dns_resolution_cancelled + or dns_resolution_completed_count + or dns_resolution_failure_types + or dns_resolution_query_error_count + ): + return + + status: ExecutionStatus + error_type: str | None = None + stop_reason: str | None = None + if dns_resolution_cancelled: + status = 'partial' if dns_resolution_completed_count else 'failed' + error_type = 'CancelledError' + stop_reason = 'cancelled' + elif dns_resolution_failure_types: + status = 'partial' if dns_resolution_completed_count else 'failed' + error_type = next(iter(sorted(dns_resolution_failure_types))) + elif dns_resolution_completed_count: + status = 'partial' if dns_resolution_query_error_count else 'completed' + error_type = next(iter(sorted(dns_resolution_error_types)), None) + stop_reason = 'query-errors' if dns_resolution_query_error_count else None + elif not handler_cancelled: + status = 'skipped' + stop_reason = 'no-input' + else: + return + action_executions.append( + ActionExecution.finish( + action='dns-resolve', + status=status, + duration_ms=dns_resolution_duration_ms, + groups={'ip': dns_resolution_ips}, + error_type=error_type, + stop_reason=stop_reason, + ) + ) + async def collect_and_store( search_engine: Any, source_spec: SourceSpec, @@ -580,6 +605,9 @@ async def start( :param search_engine: search engine to fetch details from :param source_spec: canonical source identity and declared result routes """ + nonlocal dns_resolution_cancelled, dns_resolution_completed_count + nonlocal dns_resolution_duration_ms, dns_resolution_query_error_count + await search_engine.process(use_proxy) source = source_spec.name routes = source_spec.routes @@ -606,17 +634,31 @@ async def start( if source != 'hackertarget' and source != 'pentesttools': # If a source is inside this conditional, it means the hosts returned must be resolved to obtain ip # This should only be checked if --dns-resolve has a wordlist - if dnsresolve is None or len(final_dns_resolver_list) > 0: + hosts_to_resolve = [host for host in host_names if host not in paired_hosts] + if dnsresolve != '' and hosts_to_resolve: # indicates that -r was passed in if dnsresolve is None - full_hosts_checker = hostchecker.Checker( - [host for host in host_names if host not in paired_hosts], final_dns_resolver_list - ) - # If full, this is only getting resolved hosts - ( - resolved_pair, - resolved_hosts, - temp_ips, - ) = await full_hosts_checker.check() + dns_resolution_started = time.perf_counter() + try: + full_hosts_checker = hostchecker.Checker(hosts_to_resolve, final_dns_resolver_list) + # If full, this is only getting resolved hosts + ( + resolved_pair, + resolved_hosts, + temp_ips, + ) = await full_hosts_checker.check() + except asyncio.CancelledError: + dns_resolution_duration_ms += (time.perf_counter() - dns_resolution_started) * 1000 + dns_resolution_cancelled = True + raise + except Exception as error: + dns_resolution_duration_ms += (time.perf_counter() - dns_resolution_started) * 1000 + dns_resolution_failure_types.add(type(error).__name__) + raise + dns_resolution_duration_ms += (time.perf_counter() - dns_resolution_started) * 1000 + dns_resolution_completed_count += 1 + dns_resolution_query_error_count += getattr(full_hosts_checker, 'query_error_count', 0) + dns_resolution_error_types.update(getattr(full_hosts_checker, 'query_error_types', set())) + dns_resolution_ips.update(_normalize_ip_addresses(temp_ips)) all_ip.extend(temp_ips) full.extend(resolved_pair) if source == 'rapiddns': @@ -637,7 +679,7 @@ async def start( if ResultRoute.IPS in routes: ips_list = await search_engine.get_ips() all_ip.extend(ips_list) - record_source_observations(source, 'ip-address', _normalize_ip_addresses(ips_list)) + record_source_observations(source, 'ip', _normalize_ip_addresses(ips_list)) if ResultRoute.PEOPLE in routes: people_list = await search_engine.get_people() @@ -647,22 +689,11 @@ async def start( ) record_source_observations(source, 'person', people_evidence) - if ResultRoute.LINKS in routes: - links = await search_engine.get_links() - linkedin_links_tracker.extend(links) - record_source_observations(source, 'linkedin-link', links) - if ResultRoute.URLS in routes: urls = await search_engine.get_urls() all_urls.extend(urls) record_source_observations(source, 'url', urls) - if ResultRoute.INTERESTING_URLS in routes: - get_interesting_urls = getattr(search_engine, 'get_interesting_urls', None) - iurls = await get_interesting_urls() if get_interesting_urls else await search_engine.get_interestingurls() - interesting_urls.extend(iurls) - record_source_observations(source, 'interesting-url', iurls) - if ResultRoute.ASNS in routes: fasns = await search_engine.get_asns() total_asns.extend(fasns) @@ -708,6 +739,20 @@ async def start( execution_status = cast('ExecutionStatus', reported_status) else: raise ValueError(f'Source {source_name} reported invalid execution status: {reported_status!r}') + except asyncio.CancelledError: + result_count = len(source_observations) + source_executions.append( + SourceExecution( + source_name, + 'partial' if result_count else 'failed', + (time.perf_counter() - started) * 1000, + result_count, + 'CancelledError', + 'cancelled', + ) + ) + observations.update(source_observations) + raise except Exception as error: logger.exception(f'Source {source_name} failed') result_count = len(source_observations) @@ -1586,12 +1631,12 @@ async def start( stor = await queue.get() try: await stor - queue.task_done() - # Notify the queue that the "work item" has been processed. except Exception as work_item_error: output_logger.info( f'\n An error occurred while processing a "work item": {type(work_item_error).__name__}: {work_item_error}\n' ) + finally: + # Notify the queue that the "work item" has been processed. queue.task_done() async def handler(lst): @@ -1605,16 +1650,36 @@ async def start( task = asyncio.create_task(worker(queue)) tasks.append(task) - # Wait until the queue is fully processed. - await queue.join() + join_task = asyncio.create_task(queue.join()) + try: + done, _pending = await asyncio.wait((join_task, *tasks), return_when=asyncio.FIRST_COMPLETED) + finished_workers = [task for task in tasks if task in done] + if any(task.cancelled() for task in finished_workers): + raise asyncio.CancelledError + for task in finished_workers: + if error := task.exception(): + raise error + if finished_workers: + raise RuntimeError('A source worker stopped before the queue was drained') + await join_task + finally: + join_task.cancel() + for task in tasks: + task.cancel() + await asyncio.gather(join_task, *tasks, return_exceptions=True) + while not queue.empty(): + pending_work = queue.get_nowait() + if inspect.iscoroutine(pending_work): + pending_work.close() + queue.task_done() - # Cancel our worker tasks. - for task in tasks: - task.cancel() - # Wait until all worker tasks are cancelled. - await asyncio.gather(*tasks, return_exceptions=True) - - await handler(lst=stor_lst) + try: + await handler(lst=stor_lst) + except asyncio.CancelledError: + record_dns_resolution_execution(handler_cancelled=True) + await checkpoint_completed_result(committed_sources_only=True) + await persist_result(finish_completed_result(committed_sources_only=True)) + raise recorded_sources = {result.source.casefold() for result in source_executions} source_executions.extend( @@ -1622,9 +1687,23 @@ async def start( for engine in engines if engine.casefold() not in recorded_sources ) + record_dns_resolution_execution() await checkpoint_completed_result() - if recursive_limits is not None: + recursive_seeds = sorted(_normalize_hosts_for_storage(all_hosts, word)) if recursive_limits is not None else [] + if recursive_limits is not None and not recursive_seeds: + action_executions.append( + ActionExecution.finish( + action='dns-recursive', + status='skipped', + duration_ms=0, + groups={}, + stop_reason='no-input', + ) + ) + await checkpoint_completed_result() + elif recursive_limits is not None: + recursive_started = time.perf_counter() try: async with AsyncExitStack() as resolver_stack: resolvers = [] @@ -1634,11 +1713,51 @@ async def start( resolver_stack.push_async_callback(resolver.close) recursive_result = await discover_recursive_dns( word, - all_hosts, + recursive_seeds, dnssearch.DNS_NAMES.read_text(encoding='utf-8').splitlines(), resolvers, recursive_limits, ) + recursive_finding_evidence = tuple( + json.dumps( + { + 'addresses': list(finding.records.addresses), + 'hostname': finding.hostname, + 'parent': finding.parent, + 'ptrs': list(finding.ptrs), + }, + separators=(',', ':'), + sort_keys=True, + ) + for finding in recursive_result.findings + ) + recursive_classification_evidence = tuple( + json.dumps( + { + 'addressability': classification.addressability.value, + 'addresses': list(classification.records.addresses), + 'cnames': list(classification.records.cnames), + 'hostname': classification.hostname, + 'parent': classification.parent, + 'ptrs': list(classification.ptrs), + }, + separators=(',', ':'), + sort_keys=True, + ) + for classification in recursive_result.classifications + ) + recursive_summary_evidence = ( + json.dumps( + { + 'depth_reached': recursive_result.depth_reached, + 'query_count': recursive_result.query_count, + 'stop_reason': recursive_result.stop_reason, + 'zero_yield_batches': recursive_result.zero_yield_batches, + }, + separators=(',', ':'), + sort_keys=True, + ), + ) recursive_hosts = [finding.hostname for finding in recursive_result.findings] recursive_ips = [address for finding in recursive_result.findings for address in finding.records.addresses] all_hosts.extend(recursive_hosts) @@ -1650,27 +1769,63 @@ async def start( reported_host_ip_pairs.update((finding.hostname, address) for address in finding.records.addresses) else: full.append(finding.hostname) - await db.record_observations(word, recursive_hosts, 'hostname', 'dns_recursive') - await db.record_observations(word, recursive_ips, 'ip-address', 'dns_recursive') + except asyncio.CancelledError: + action_executions.append( + ActionExecution.finish( + action='dns-recursive', + status='failed', + duration_ms=(time.perf_counter() - recursive_started) * 1000, + groups={}, + error_type='CancelledError', + stop_reason='cancelled', + ) + ) + await checkpoint_completed_result() + await persist_result(finish_completed_result()) + raise + except Exception as error: + action_executions.append( + ActionExecution.finish( + action='dns-recursive', + status='failed', + duration_ms=(time.perf_counter() - recursive_started) * 1000, + groups={}, + error_type=type(error).__name__, + ) + ) + await checkpoint_completed_result() + output_logger.info(f'[!] Recursive DNS discovery failed: {type(error).__name__}') + else: + action_executions.append( + ActionExecution.finish( + action='dns-recursive', + status='partial' if recursive_result.stop_reason in {'query-limit', 'runtime-limit'} else 'completed', + duration_ms=(time.perf_counter() - recursive_started) * 1000, + groups={ + 'hostname': recursive_hosts, + 'ip': recursive_ips, + 'dns-recursive-finding': recursive_finding_evidence, + 'dns-recursive-classification': recursive_classification_evidence, + 'dns-recursive-summary': recursive_summary_evidence, + }, + stop_reason=recursive_result.stop_reason, + ) + ) output_logger.info( '[*] Recursive DNS: ' f'hosts={len(recursive_hosts)}; queries={recursive_result.query_count}; ' f'depth={recursive_result.depth_reached}; stop={recursive_result.stop_reason}' ) await checkpoint_completed_result() - except Exception as error: - output_logger.info(f'[!] Recursive DNS discovery failed: {type(error).__name__}') - - async def persist_result(completed_result: CompletedResult | None) -> None: - if completed_result is None: - return - try: - await db.save_run(completed_result) - except Exception as error: - output_logger.info(f'[!] An error occurred while storing the completed result: {error}') return_ips: list = [] - if rest_args is not None and len(rest_filename) == 0 and rest_args.dns_brute is False and not return_completed_result: + if ( + rest_args is not None + and len(rest_filename) == 0 + and rest_args.dns_brute is False + and not dnslookup + and not return_completed_result + ): # Indicates user is using REST api but not wanting output to be saved to a file # cast to string so Rest API can understand the type return_ips.extend([str(ip) for ip in sorted([netaddr.IPAddress(ip.strip()) for ip in set(all_ip)])]) @@ -1680,10 +1835,10 @@ async def start( await persist_result(finish_completed_result()) result = ( total_asns, - interesting_urls, + list[str](), twitter_people_list_tracker, linkedin_people_list_tracker, - linkedin_links_tracker, + list[str](), all_urls, all_ip, all_emails, @@ -1707,10 +1862,6 @@ async def start( print_section(f'\n[*] ASNS found: {len(total_asns)}', total_asns, '--------------------') total_asns = sorted_unique(total_asns) - if len(interesting_urls) > 0: - print_section(f'\n[*] Interesting Urls found: {len(interesting_urls)}', interesting_urls, '--------------------') - interesting_urls = sorted_unique(interesting_urls) - if len(twitter_people_list_tracker) == 0 and 'twitter' in engines: output_logger.info('\n[*] No Twitter users found.\n\n') elif len(twitter_people_list_tracker) >= 1: @@ -1721,9 +1872,8 @@ async def start( ) twitter_people_list_tracker = sorted_unique(twitter_people_list_tracker) - print_linkedin_sections(engines, linkedin_people_list_tracker, linkedin_links_tracker) + print_linkedin_people(engines, linkedin_people_list_tracker) linkedin_people_list_tracker = sorted_unique(linkedin_people_list_tracker) - linkedin_links_tracker = sorted_unique(linkedin_links_tracker) length_urls = len(all_urls) if length_urls == 0: @@ -1776,7 +1926,7 @@ async def start( if len(all_hosts) == 0: output_logger.info('\n[*] No hosts found.\n\n') else: - if dnsresolve is None or len(final_dns_resolver_list) > 0: + if dnsresolve != '': temp = set() for host in full: if ':' in host: @@ -1793,13 +1943,6 @@ async def start( output_logger.info('---------------------') for host in full: output_logger.info(host) - try: - if ':' in host: - _, addr = host.split(':', 1) - await db.record_observations(word, [addr], 'ip-address', 'DNS-resolver') - except (OSError, RuntimeError, ValueError, TypeError) as e: - output_logger.info(f'An exception has occurred while attempting to insert: {host} IP into DB: {e}') - continue else: all_hosts = sorted_unique(all_hosts) output_logger.info('\n[*] Hosts found: ' + str(len(all_hosts))) @@ -1809,13 +1952,41 @@ async def start( # DNS brute force if dnsbrute and dnsbrute[0] is True: + dns_brute_started = time.perf_counter() output_logger.info('\n[*] Starting DNS brute force.') - dns_force = dnssearch.DnsForce(word, final_dns_resolver_list, verbose=True) - resolved_pair, hosts, ips = await dns_force.run() + try: + dns_force = dnssearch.DnsForce(word, final_dns_resolver_list, verbose=True) + resolved_pair, hosts, ips = await dns_force.run() + except asyncio.CancelledError: + action_executions.append( + ActionExecution.finish( + action='dns-brute', + status='failed', + duration_ms=(time.perf_counter() - dns_brute_started) * 1000, + groups={}, + error_type='CancelledError', + stop_reason='cancelled', + ) + ) + await checkpoint_completed_result() + await persist_result(finish_completed_result()) + raise + except Exception as error: + action_executions.append( + ActionExecution.finish( + action='dns-brute', + status='failed', + duration_ms=(time.perf_counter() - dns_brute_started) * 1000, + groups={}, + error_type=type(error).__name__, + ) + ) + await checkpoint_completed_result() + await persist_result(finish_completed_result()) + raise resolved_screenshot_hosts.update(hosts) - # Check if Rest API is being used if so return found hosts - if dnsbrute[1]: - return resolved_pair + normalized_brute_hosts = _normalize_hosts_for_storage(hosts, word) + normalized_brute_ips = _normalize_ip_addresses(ips) temp = set() for host in resolved_pair: if ':' in host: @@ -1838,25 +2009,111 @@ async def start( output_logger.info('\n[*] Hosts found after DNS brute force:') for sub in temp: output_logger.info(sub) - await db.record_observations(word, list(sorted(temp)), 'hostname', 'dns_bruteforce') + dns_brute_error_count = getattr(dns_force, 'query_error_count', 0) + dns_brute_error_types: set[str] = set(getattr(dns_force, 'query_error_types', set())) + dns_brute_status: ExecutionStatus = 'completed' + if dns_brute_error_count: + dns_brute_status = 'partial' + action_executions.append( + ActionExecution.finish( + action='dns-brute', + status=dns_brute_status, + duration_ms=(time.perf_counter() - dns_brute_started) * 1000, + groups={'hostname': normalized_brute_hosts, 'ip': normalized_brute_ips}, + error_type=next(iter(sorted(dns_brute_error_types)), None), + stop_reason='query-errors' if dns_brute_error_count else None, + ) + ) await checkpoint_completed_result() + # Preserve the dedicated utility response after retaining its completed evidence. + if dnsbrute[1]: + await persist_result(finish_completed_result()) + return resolved_pair # TakeOver Checking if takeover_status: + takeover_started = time.perf_counter() output_logger.info('\n[*] Performing subdomain takeover check') output_logger.info('\n[*] Subdomain Takeover checking IS ACTIVE RECON') - search_take = takeover.TakeOver(all_hosts) - await search_take.populate_fingerprints() - await search_take.process(proxy=use_proxy) - takeover_results = await search_take.get_takeover_results() - await checkpoint_completed_result() + if not all_hosts: + action_executions.append( + ActionExecution.finish( + action='takeover', + status='skipped', + duration_ms=(time.perf_counter() - takeover_started) * 1000, + groups={}, + stop_reason='no-input', + ) + ) + else: + search_take: takeover.TakeOver | None = None + + def normalize_takeover_evidence(results: Mapping[str, object]) -> set[str]: + return { + json.dumps({'matches': matches, 'url': url}, separators=(',', ':'), sort_keys=True) + for url, matches in results.items() + } + + async def collect_takeover_evidence(*, best_effort: bool = False) -> tuple[dict[str, list[dict[str, str]]], set[str]]: + if search_take is None: + return {}, set() + try: + results = await search_take.get_takeover_results() + except (asyncio.CancelledError, Exception): + if not best_effort: + raise + return {}, set() + return results, normalize_takeover_evidence(results) + + try: + search_take = takeover.TakeOver(all_hosts) + await search_take.populate_fingerprints() + await search_take.process(proxy=use_proxy) + takeover_results, takeover_evidence = await collect_takeover_evidence() + except (asyncio.CancelledError, Exception) as error: + takeover_results, takeover_evidence = await collect_takeover_evidence(best_effort=True) + action_executions.append( + ActionExecution.finish( + action='takeover', + status='partial' if takeover_evidence else 'failed', + duration_ms=(time.perf_counter() - takeover_started) * 1000, + groups={'takeover': takeover_evidence}, + error_type=type(error).__name__, + stop_reason='cancelled' if isinstance(error, asyncio.CancelledError) else 'scan-error', + ) + ) + await persist_result(finish_completed_result()) + raise + assert search_take is not None + takeover_request_errors = search_take.request_error_count + takeover_scan_error = search_take.scan_error_type + takeover_status_value: ExecutionStatus = 'completed' + if takeover_scan_error: + takeover_status_value = 'partial' if takeover_evidence else 'failed' + elif takeover_request_errors: + takeover_status_value = ( + 'partial' if takeover_evidence or takeover_request_errors < search_take.request_count else 'failed' + ) + action_executions.append( + ActionExecution.finish( + action='takeover', + status=takeover_status_value, + duration_ms=(time.perf_counter() - takeover_started) * 1000, + groups={'takeover': takeover_evidence}, + error_type=takeover_scan_error or next(iter(sorted(search_take.request_error_types)), None), + stop_reason=('scan-error' if takeover_scan_error else 'request-errors' if takeover_request_errors else None), + ) + ) + await checkpoint_action_result() # DNS reverse lookup dnsrev: list = [] if dnslookup is True: + dns_lookup_started = time.perf_counter() + dns_lookup_error_types: set[str] = set() output_logger.info('\n[*] Starting active queries for DNSLookup.') # reverse each iprange in a separate task - __reverse_dns_tasks: dict = {} + __reverse_dns_tasks: dict[str, asyncio.Task[None]] = {} for entry in host_ip: __ip_range = dnssearch.serialize_ip_range(ip=entry, netmask='24') if __ip_range and __ip_range not in set(__reverse_dns_tasks.keys()): @@ -1868,57 +2125,231 @@ async def start( target=word, local_results=dnsrev, overall_results=full ), nameservers=(final_dns_resolver_list if len(final_dns_resolver_list) > 0 else None), + error_types=dns_lookup_error_types, ) ) # nameservers=list(map(str, dnsserver.split(','))) if dnsserver else None)) # run all the reversing tasks concurrently - await asyncio.gather(*__reverse_dns_tasks.values()) + try: + await asyncio.gather(*__reverse_dns_tasks.values()) + except (asyncio.CancelledError, Exception) as error: + for task in __reverse_dns_tasks.values(): + if not task.done(): + task.cancel() + await asyncio.gather(*__reverse_dns_tasks.values(), return_exceptions=True) + normalized_reverse_hosts = _normalize_hosts_for_storage(dnsrev, word) + action_executions.append( + ActionExecution.finish( + action='dns-lookup', + status='partial' if normalized_reverse_hosts else 'failed', + duration_ms=(time.perf_counter() - dns_lookup_started) * 1000, + groups={'hostname': normalized_reverse_hosts}, + error_type=type(error).__name__, + stop_reason='cancelled' if isinstance(error, asyncio.CancelledError) else None, + ) + ) + await checkpoint_completed_result(extra_hostnames=dnsrev) + await persist_result(finish_completed_result(extra_hostnames=dnsrev)) + raise output_logger.info('\n[*] Hosts found after reverse lookup (in target domain):') output_logger.info('--------------------------------------------------------') for xh in dnsrev: output_logger.info(xh) + normalized_reverse_hosts = _normalize_hosts_for_storage(dnsrev, word) + dns_lookup_status: ExecutionStatus = 'completed' + dns_lookup_stop_reason = None + if not __reverse_dns_tasks: + dns_lookup_status = 'skipped' + dns_lookup_stop_reason = 'no-input' + elif dns_lookup_error_types: + dns_lookup_status = 'partial' + dns_lookup_stop_reason = 'query-errors' + action_executions.append( + ActionExecution.finish( + action='dns-lookup', + status=dns_lookup_status, + duration_ms=(time.perf_counter() - dns_lookup_started) * 1000, + groups={'hostname': normalized_reverse_hosts}, + error_type=next(iter(sorted(dns_lookup_error_types)), None), + stop_reason=dns_lookup_stop_reason, + ) + ) await checkpoint_completed_result(extra_hostnames=dnsrev) # Screenshots if len(args.screenshot) > 0: + screenshot_started = time.perf_counter() screen_shotter = ScreenShotter(args.screenshot) + + async def persist_screenshot_cancellation() -> None: + action_executions.append( + ActionExecution.finish( + action='screenshot', + status='partial' if screenshot_artifacts else 'failed', + duration_ms=(time.perf_counter() - screenshot_started) * 1000, + groups={}, + artifacts=screenshot_artifacts, + error_type='CancelledError', + stop_reason='cancelled', + ) + ) + completed = finish_completed_result(extra_hostnames=dnsrev) + if completed is None: + return + try: + if completed_result_checkpoint is not None: + await completed_result_checkpoint(completed) + finally: + await persist_result(completed) + path_exists = screen_shotter.verify_path() # Verify the path exists, if not create it or if user does not create it skips screenshot - if path_exists: - await screen_shotter.verify_installation() + if not path_exists: + action_executions.append( + ActionExecution.finish( + action='screenshot', + status='skipped', + duration_ms=(time.perf_counter() - screenshot_started) * 1000, + groups={}, + stop_reason='path-unavailable', + ) + ) + else: + try: + await screen_shotter.verify_installation() + except asyncio.CancelledError: + await persist_screenshot_cancellation() + raise output_logger.info(f'\nScreenshots can be found in: {screen_shotter.output}{screen_shotter.slash}') - start_time = time.perf_counter() output_logger.info('Filtering domains for ones we can reach') - if dnsresolve is None or len(final_dns_resolver_list) > 0: + if not engines: + unique_resolved_domains = resolved_screenshot_hosts | {word} + elif dnsresolve != '': unique_resolved_domains = resolved_screenshot_hosts else: # Technically not resolved in this case, which is not ideal # You should always use dns resolve when doing screenshotting output_logger.info('NOTE for future use cases you should only use screenshotting in tandem with DNS resolving') unique_resolved_domains = set(all_hosts) + reachable_targets: list[tuple[str, str]] = [] + capture_error_types: set[str] = set() if len(unique_resolved_domains) > 0: # First filter out ones that didn't resolve output_logger.info('Attempting to visit unique resolved domains, this is ACTIVE RECON') + + async def visit_screenshot_target(host: str) -> tuple[str, str]: + final_url, body = await screen_shotter.visit(host) + return host, final_url if body else '' + async with Pool(10) as pool: - results = await pool.map(screen_shotter.visit, list(unique_resolved_domains)) - # Filter out domains that we couldn't connect to - unique_resolved_domains_list = list(sorted({tup[0] for tup in results if len(tup[1]) > 0})) - async with Pool(3) as pool: - output_logger.info(f'Length of unique resolved domains: {len(unique_resolved_domains_list)} chunking now!\n') - # If you have the resources, you could make the function faster by increasing the chunk number - chunk_number = 14 - for chunk in screen_shotter.chunk_list(unique_resolved_domains_list, chunk_number): + try: + results = await pool.map(visit_screenshot_target, list(unique_resolved_domains)) + except asyncio.CancelledError: + await persist_screenshot_cancellation() + raise + reachable_targets = sorted((host, final_url) for host, final_url in results if final_url) + + semaphore = asyncio.Semaphore(3) + + async def capture_screenshot_target(target: tuple[str, str]) -> tuple[str, str, Path]: + subject, final_url = target + output_path = screen_shotter.screenshot_path(subject) + async with semaphore: + return subject, await screen_shotter.take_screenshot(final_url, output_path=output_path), output_path + + async def record_screenshot_artifact(subject: str, captured_url: str, screenshot_path: Path) -> None: + if not captured_url: + capture_error_types.add('CaptureError') + return + if not await anyio.Path(screenshot_path).is_file(): + capture_error_types.add('ArtifactMissing') + return + raw_subject = subject.strip() + try: + subject_value = str(ip_address(raw_subject)) + subject_kind: ResultKind = 'ip' + except ValueError: + parsed_subject = urlsplit( + raw_subject if raw_subject.startswith(('http://', 'https://')) else f'https://{raw_subject}' + ) + if not parsed_subject.hostname: + capture_error_types.add('InvalidScreenshotURL') + return + subject_value = parsed_subject.hostname.lower() try: - screenshot_results.extend( - result for result in await pool.map(screen_shotter.take_screenshot, chunk) if result - ) - await checkpoint_completed_result(extra_hostnames=dnsrev) - except Exception as ee: - output_logger.info(f'An exception has occurred while mapping: {ee}') + subject_value = str(ip_address(subject_value)) + subject_kind = 'ip' + except ValueError: + subject_kind = 'hostname' + recorded_subjects = screenshot_ip_addresses if subject_kind == 'ip' else screenshot_hostnames + if subject_value in recorded_subjects: + return + recorded_subjects.add(subject_value) + screenshot_bytes = await anyio.Path(screenshot_path).read_bytes() + screenshot_artifacts.append( + ArtifactReference( + kind='screenshot', + subject_kind=subject_kind, + subject_value=subject_value, + path=str(Path(Path(screen_shotter.output).name) / screenshot_path.name), + media_type='image/png', + size_bytes=len(screenshot_bytes), + sha256=hashlib.sha256(screenshot_bytes).hexdigest(), + created_at=datetime.now(UTC), + ) + ) + + capture_tasks = [asyncio.create_task(capture_screenshot_target(target)) for target in reachable_targets] + try: + for capture_task in asyncio.as_completed(capture_tasks): + subject, captured_url, screenshot_path = await capture_task + await record_screenshot_artifact(subject, captured_url, screenshot_path) + except asyncio.CancelledError: + for capture_task in capture_tasks: + capture_task.cancel() + outcomes = await asyncio.gather(*capture_tasks, return_exceptions=True) + for outcome in outcomes: + if isinstance(outcome, tuple): + await record_screenshot_artifact(*outcome) + await persist_screenshot_cancellation() + raise + except Exception as ee: + for capture_task in capture_tasks: + capture_task.cancel() + await asyncio.gather(*capture_tasks, return_exceptions=True) + capture_error_types.add(type(ee).__name__) + output_logger.info(f'An exception has occurred while mapping: {ee}') + if not unique_resolved_domains: + screenshot_status: ExecutionStatus = 'skipped' + screenshot_stop_reason = 'no-input' + elif not reachable_targets: + screenshot_status = 'failed' + screenshot_stop_reason = 'no-reachable-targets' + elif capture_error_types and screenshot_artifacts: + screenshot_status = 'partial' + screenshot_stop_reason = 'capture-errors' + elif capture_error_types or (reachable_targets and not screenshot_artifacts): + screenshot_status = 'failed' + screenshot_stop_reason = 'capture-errors' + else: + screenshot_status = 'completed' + screenshot_stop_reason = None + action_executions.append( + ActionExecution.finish( + action='screenshot', + status=screenshot_status, + duration_ms=(time.perf_counter() - screenshot_started) * 1000, + groups={}, + artifacts=screenshot_artifacts, + error_type=next(iter(sorted(capture_error_types)), None), + stop_reason=screenshot_stop_reason, + ) + ) + await checkpoint_action_result(extra_hostnames=dnsrev) end = time.perf_counter() # There is probably an easier way to do this - total = int(end - start_time) + total = int(end - screenshot_started) mon, sec = divmod(total, 60) hr, mon = divmod(mon, 60) total_time = f'{mon:02d}:{sec:02d}' @@ -1928,24 +2359,27 @@ async def start( # Shodan shodanres = [] if shodan is True: + shodan_started = time.perf_counter() + shodan_error_types: set[str] = set() output_logger.info('[*] Searching Shodan. ') try: - for ip in host_ip: + for ip_index, ip in enumerate(host_ip): try: output_logger.info('\tSearching for ' + ip) shodan_search = shodansearch.SearchShodan() shodandict = await shodan_search.search_ip(ip) - await asyncio.sleep(5) + if shodan_search.error_type: + shodan_error_types.add(shodan_search.error_type) + shodan_result = shodandict.get(ip) # Check if the result is a string (error message) - if isinstance(shodandict[ip], str): - output_logger.info(f'{ip}: {shodandict[ip]}') - continue + if isinstance(shodan_result, str): + output_logger.info(f'{ip}: {shodan_result}') # Process the results if it's a dictionary - if isinstance(shodandict[ip], dict): + if isinstance(shodan_result, dict): rowdata = [] - for _key, value in shodandict[ip].items(): + for _key, value in shodan_result.items(): if isinstance(value, int): value = str(value) if isinstance(value, list): @@ -1953,16 +2387,48 @@ async def start( rowdata.append(value) shodanres.append(rowdata) shodan_evidence.append( - json.dumps({'ip': ip, 'result': shodandict[ip]}, separators=(',', ':'), sort_keys=True) + json.dumps({'ip': ip, 'result': shodan_result}, separators=(',', ':'), sort_keys=True) ) - await checkpoint_completed_result(extra_hostnames=dnsrev) - output_logger.info(ujson.dumps(shodandict[ip], indent=4, sort_keys=True)) + output_logger.info(ujson.dumps(shodan_result, indent=4, sort_keys=True)) output_logger.info('\n') + if ip_index + 1 < len(host_ip): + await asyncio.sleep(5) except Exception as ip_error: - output_logger.info(f'[SHODAN-error] Error searching {ip}: {ip_error}') + shodan_error_types.add(type(ip_error).__name__) + output_logger.info(f'[SHODAN-error] Error searching {ip}: {type(ip_error).__name__}') continue - except Exception as e: - output_logger.info(f'[!] An error occurred with Shodan: {e} ') + except asyncio.CancelledError: + action_executions.append( + ActionExecution.finish( + action='shodan', + status='partial' if shodan_evidence else 'failed', + duration_ms=(time.perf_counter() - shodan_started) * 1000, + groups={'shodan': shodan_evidence}, + error_type='CancelledError', + stop_reason='cancelled', + ) + ) + await persist_result(finish_completed_result(extra_hostnames=dnsrev)) + raise + shodan_status: ExecutionStatus = 'completed' + shodan_stop_reason = None + if not host_ip: + shodan_status = 'skipped' + shodan_stop_reason = 'no-input' + elif shodan_error_types: + shodan_status = 'partial' if shodan_evidence else 'failed' + shodan_stop_reason = 'target-errors' + action_executions.append( + ActionExecution.finish( + action='shodan', + status=shodan_status, + duration_ms=(time.perf_counter() - shodan_started) * 1000, + groups={'shodan': shodan_evidence}, + error_type=next(iter(sorted(shodan_error_types)), None), + stop_reason=shodan_stop_reason, + ) + ) + await checkpoint_action_result(extra_hostnames=dnsrev) else: pass @@ -2007,67 +2473,35 @@ async def start( except (OSError, ValueError, TypeError, UnicodeEncodeError) as error: output_logger.info(f'[!] An error occurred while saving the XML file: {error}') - try: - # JSON REPORT SECTION - filename = os.path.splitext(filename)[0] + '.json' - # create dict with values for JSON output - json_dict: dict = dict() - # start by adding the command line arguments - json_dict['cmd'] = ' '.join([f'"{arg}"' if ' ' in arg else arg for arg in sys.argv[1:]]) - # to determine if a variable exists - # it should but just a validation check - if 'ip_list' in locals(): - if all_ip and len(all_ip) >= 1 and ip_list and len(ip_list) > 0: - json_dict['ips'] = ip_list - - if len(all_emails) > 0: - json_dict['emails'] = all_emails - - if dnsresolve is None or (len(final_dns_resolver_list) > 0 and len(full) > 0): - json_dict['hosts'] = full - elif len(all_hosts) > 0: - json_dict['hosts'] = all_hosts - else: - json_dict['hosts'] = [] - - if vhost and len(vhost) > 0: - json_dict['vhosts'] = vhost - - if len(interesting_urls) > 0: - json_dict['interesting_urls'] = interesting_urls - - if len(all_urls) > 0: - json_dict['trello_urls'] = all_urls - - if len(total_asns) > 0: - json_dict['asns'] = total_asns - - if len(twitter_people_list_tracker) > 0: - json_dict['twitter_people'] = twitter_people_list_tracker - - if len(linkedin_people_list_tracker) > 0: - json_dict['linkedin_people'] = linkedin_people_list_tracker - - if len(linkedin_links_tracker) > 0: - json_dict['linkedin_links'] = linkedin_links_tracker - - if len(all_people) > 0: - json_dict['people'] = all_people - - if takeover_status and len(takeover_results) > 0: - json_dict['takeover_results'] = takeover_results - - json_dict['shodan'] = shodanres - async with await anyio.open_file(filename, 'w+') as fp: - dumped_json = ujson.dumps(json_dict, sort_keys=True) - await fp.write(dumped_json) - output_logger.info('[*] JSON File saved.') - except (OSError, ValueError, TypeError, UnicodeEncodeError) as er: - output_logger.info(f'[!] An error occurred while saving the JSON file: {er} ') - output_logger.info('\n\n') - # Enhanced code block for API Endpoint scanning feature if args.api_scan or 'api_endpoints' in engines: + api_scan_started = time.perf_counter() + api_scanner = None + + def collect_api_action_groups( + scanner: 'api_endpoints.SearchApiEndpoints | None', + *, + best_effort: bool = False, + ) -> tuple[set[str], set[str], dict[ResultKind, Iterable[str]]]: + endpoints: set[str] = set() + interesting: set[str] = set() + + def collect(getter: Callable[[], Iterable[str]]) -> set[str]: + if not best_effort: + return set(getter()) + try: + return set(getter()) + except Exception: + return set() + + if scanner is not None: + endpoints = collect(scanner.get_found_endpoints) + interesting = collect(scanner.get_interesting_endpoints) + if best_effort: + endpoints.update(interesting) + groups: dict[ResultKind, Iterable[str]] = {'url': endpoints | interesting} + return endpoints, interesting, groups + try: # Define a default wordlist if none is specified wordlist = args.wordlist or str(DATA_DIR / 'wordlists' / 'api_endpoints.txt') @@ -2104,16 +2538,22 @@ async def start( output_logger.info(f'Basic API wordlist created with {len(basic_endpoints)} endpoints.') output_logger.info(f'\n[*] Starting API endpoint scanning with wordlist: {wordlist}') - api_scanner = api_endpoints.SearchApiEndpoints(word=args.domain, wordlist=wordlist) + if args.wordlist: + api_scanner = api_endpoints.SearchApiEndpoints( + word=args.domain, + wordlist=wordlist, + exact_paths=True, + ) + else: + api_scanner = api_endpoints.SearchApiEndpoints(word=args.domain, wordlist=wordlist) await api_scanner.do_search() # Print results - endpoints_found = set(api_scanner.get_found_endpoints()) + endpoints_found, interesting_endpoints, api_action_groups = collect_api_action_groups(api_scanner) output_logger.info(f'\n[*] API Endpoints found: {len(endpoints_found)}') for endpoint in endpoints_found: output_logger.info(f' - {endpoint}') - interesting_endpoints = api_scanner.get_interesting_endpoints() output_logger.info(f'\n[*] Interesting endpoints (200, 201, 202): {len(interesting_endpoints)}') for endpoint in interesting_endpoints: output_logger.info(f' - {endpoint}') @@ -2139,28 +2579,131 @@ async def start( status_codes = api_scanner.get_status_codes() output_logger.info(f'\n[*] HTTP status codes encountered: {", ".join(map(str, status_codes))}') - # Add results to storage - await db.record_observations(word, endpoints_found, 'api-endpoint', 'api_scan') + if endpoints_found or interesting_endpoints: + all_urls.extend(sorted(endpoints_found | interesting_endpoints)) - # Add to interesting URLs if any endpoints were found - if interesting_endpoints: - new_urls = [f'https://{args.domain}{endpoint}' for endpoint in interesting_endpoints] - interesting_urls.extend(new_urls) - - # Also add complete domain paths to the interesting_urls list - all_urls.extend(new_urls) + api_scan_error = api_scanner.scan_error_type + api_request_errors = api_scanner.request_error_count + api_scan_status: ExecutionStatus = 'completed' + api_error_type = None + api_stop_reason = None + if api_scan_error: + api_scan_status = 'partial' if any(api_action_groups.values()) else 'failed' + api_error_type = api_scan_error + api_stop_reason = 'scan-error' + elif rate_limits: + api_scan_status = 'rate-limited' + api_stop_reason = 'rate-limited' + elif api_request_errors: + api_scan_status = 'partial' + api_error_type = next(iter(sorted(api_scanner.request_error_types)), None) + api_stop_reason = 'request-errors' + action_executions.append( + ActionExecution.finish( + action='api-scan', + status=api_scan_status, + duration_ms=(time.perf_counter() - api_scan_started) * 1000, + groups=api_action_groups, + error_type=api_error_type, + stop_reason=api_stop_reason, + ) + ) output_logger.info('\n[+] API scanning completed successfully.') - await checkpoint_completed_result(extra_hostnames=dnsrev, virtual_hosts=vhost) - except MissingKey: - output_logger.info('\n[!] API endpoint scanning requires a wordlist. Use -w to specify a wordlist file.') - output_logger.info(' Creating a basic wordlist and trying again...') - # The wordlist creation code above could be used here - except Exception as e: - output_logger.info(f'\n[!] An exception has occurred in API Endpoints scanning: {e}') - output_logger.info(' Continuing with the rest of the scan...') - traceback.print_exc() # More detailed error information for developers + except asyncio.CancelledError: + if not any(execution.action == 'api-scan' for execution in action_executions): + _endpoints, _interesting, api_action_groups = collect_api_action_groups(api_scanner, best_effort=True) + action_executions.append( + ActionExecution.finish( + action='api-scan', + status='partial' if any(api_action_groups.values()) else 'failed', + duration_ms=(time.perf_counter() - api_scan_started) * 1000, + groups=api_action_groups, + error_type='CancelledError', + stop_reason='cancelled', + ) + ) + await persist_result(finish_completed_result(extra_hostnames=dnsrev, virtual_hosts=vhost)) + raise + except Exception as error: + endpoints_found, _interesting, api_action_groups = collect_api_action_groups(api_scanner, best_effort=True) + if endpoints_found: + all_urls.extend(sorted(endpoints_found)) + if not any(execution.action == 'api-scan' for execution in action_executions): + action_executions.append( + ActionExecution.finish( + action='api-scan', + status='partial' if any(api_action_groups.values()) else 'failed', + duration_ms=(time.perf_counter() - api_scan_started) * 1000, + groups=api_action_groups, + error_type=type(error).__name__, + stop_reason='scan-error', + ) + ) + if isinstance(error, MissingKey): + output_logger.info('\n[!] API endpoint scanning could not start because a required key is missing.') + else: + output_logger.info(f'\n[!] API endpoint scanning failed with {type(error).__name__}.') + output_logger.info(' Continuing with the rest of the scan...') + + await checkpoint_action_result(extra_hostnames=dnsrev, virtual_hosts=vhost) + + all_urls = sorted_unique(all_urls) + + if filename != '': + try: + # JSON REPORT SECTION + filename = os.path.splitext(filename)[0] + '.json' + # create dict with values for JSON output + json_dict: dict = dict() + # start by adding the command line arguments + json_dict['cmd'] = ' '.join([f'"{arg}"' if ' ' in arg else arg for arg in sys.argv[1:]]) + # to determine if a variable exists + # it should but just a validation check + if 'ip_list' in locals(): + if all_ip and len(all_ip) >= 1 and ip_list and len(ip_list) > 0: + json_dict['ips'] = ip_list + + if len(all_emails) > 0: + json_dict['emails'] = all_emails + + if dnsresolve != '' and len(full) > 0: + json_dict['hosts'] = full + elif len(all_hosts) > 0: + json_dict['hosts'] = all_hosts + else: + json_dict['hosts'] = [] + + if vhost and len(vhost) > 0: + json_dict['vhosts'] = vhost + + if len(all_urls) > 0: + json_dict['urls'] = all_urls + + if len(total_asns) > 0: + json_dict['asns'] = total_asns + + if len(twitter_people_list_tracker) > 0: + json_dict['twitter_people'] = twitter_people_list_tracker + + if len(linkedin_people_list_tracker) > 0: + json_dict['linkedin_people'] = linkedin_people_list_tracker + + if len(all_people) > 0: + json_dict['people'] = all_people + + if takeover_status and len(takeover_results) > 0: + json_dict['takeover_results'] = takeover_results + + json_dict['shodan'] = shodanres + async with await anyio.open_file(filename, 'w+') as fp: + dumped_json = ujson.dumps(json_dict, sort_keys=True) + await fp.write(dumped_json) + output_logger.info('[*] JSON File saved.') + except (OSError, ValueError, TypeError, UnicodeEncodeError) as er: + output_logger.info(f'[!] An error occurred while saving the JSON file: {er} ') + output_logger.info('\n\n') completed_result = finish_completed_result(extra_hostnames=dnsrev, virtual_hosts=vhost) @@ -2176,13 +2719,13 @@ async def start( await persist_result(completed_result) if rest_args is not None: - all_hosts = sorted_unique(all_hosts) + all_hosts = sorted_unique((*all_hosts, *_normalize_hosts_for_storage(dnsrev, word))) result = ( total_asns, - interesting_urls, + list[str](), twitter_people_list_tracker, linkedin_people_list_tracker, - linkedin_links_tracker, + list[str](), all_urls, all_ip, all_emails, diff --git a/theHarvester/discovery/additional_apis.py b/theHarvester/discovery/additional_apis.py deleted file mode 100644 index 6620565b..00000000 --- a/theHarvester/discovery/additional_apis.py +++ /dev/null @@ -1,183 +0,0 @@ -import asyncio -import logging -from typing import Any - -from theHarvester.discovery.builtwith import SearchBuiltWith -from theHarvester.discovery.haveibeenpwned import SearchHaveIBeenPwned -from theHarvester.discovery.leaklookup import SearchLeakLookup -from theHarvester.discovery.securityscorecard import SearchSecurityScorecard -from theHarvester.discovery.shodansearch import SearchShodan -from theHarvester.lib.output import output_logger - -logger = logging.getLogger(__name__) - - -class AdditionalAPIs: - """Wrapper class for additional API services.""" - - def __init__(self, domain: str, api_keys: dict[str, str] | None = None): - self.domain = domain - self.api_keys = api_keys or {} - - # Initialize API services - self.haveibeenpwned = SearchHaveIBeenPwned(domain) - self.leaklookup: SearchLeakLookup | None = None - self.securityscorecard = SearchSecurityScorecard(domain) - self.builtwith = SearchBuiltWith(domain) - self.shodan: SearchShodan | None = None # Will be initialized when needed - - # Aggregated sets for results - self.hosts: set[str] = set() - self.emails: set[str] = set() - - # Results storage - self.results: dict[str, Any] = { - 'breaches': [], - 'leaks': [], - 'security_score': {}, - 'tech_stack': {}, - 'shodan_data': {}, - 'hosts': [], - 'emails': [], - } - self.shodan_data: dict[str, Any] = {} - - async def process(self, proxy: bool = False) -> dict[str, Any]: - """Process all additional API services and return combined results.""" - tasks = [ - self._process_haveibeenpwned(proxy), - self._process_leaklookup(proxy), - self._process_securityscorecard(proxy), - self._process_builtwith(proxy), - self._process_shodan(proxy), - ] - - await asyncio.gather(*tasks, return_exceptions=True) - - # Convert aggregated sets to lists for JSON serialization - self.results['hosts'] = list(self.hosts) - self.results['emails'] = list(self.emails) - self.results['shodan_data'] = self.shodan_data - - return self.results - - async def _process_haveibeenpwned(self, proxy: bool = False): - """Process HaveIBeenPwned API.""" - try: - await self.haveibeenpwned.process(proxy) - self.results['breaches'] = self.haveibeenpwned.breaches - self.hosts.update(self.haveibeenpwned.hosts) - self.emails.update(self.haveibeenpwned.emails) - except Exception as e: - logger.info(f'Error processing HaveIBeenPwned: {e}') - - async def _process_leaklookup(self, proxy: bool = False): - """Process Leak-Lookup API.""" - try: - if self.leaklookup is None: - self.leaklookup = SearchLeakLookup(self.domain) - await self.leaklookup.process(proxy) - self.results['leaks'] = self.leaklookup.leaks - self.hosts.update(self.leaklookup.hosts) - self.emails.update(self.leaklookup.emails) - except Exception as e: - logger.info(f'Error processing Leak-Lookup: {e}') - - async def _process_securityscorecard(self, proxy: bool = False): - """Process SecurityScorecard API.""" - try: - await self.securityscorecard.process(proxy) - self.results['security_score'] = { - 'score': self.securityscorecard.score, - 'grades': self.securityscorecard.grades, - 'issues': self.securityscorecard.issues, - 'recommendations': self.securityscorecard.recommendations, - } - self.hosts.update(self.securityscorecard.hosts) - except Exception as e: - logger.info(f'Error processing SecurityScorecard: {e}') - - async def _process_builtwith(self, proxy: bool = False): - """Process BuiltWith API.""" - try: - await self.builtwith.process(proxy) - self.results['tech_stack'] = { - 'frameworks': list(self.builtwith.frameworks), - 'languages': list(self.builtwith.languages), - 'servers': list(self.builtwith.servers), - 'cms': list(self.builtwith.cms), - 'analytics': list(self.builtwith.analytics), - 'interesting_urls': list(self.builtwith.interesting_urls), - } - self.hosts.update(self.builtwith.hosts) - except Exception as e: - logger.info(f'Error processing BuiltWith: {e}') - - async def _process_shodan(self, proxy: bool = False): - """Process Shodan API for IP information.""" - try: - # Initialize Shodan only when needed - if self.shodan is None: - self.shodan = SearchShodan() - - # Get IPs from hosts for Shodan lookup - import socket - - ips_to_search: set[str] = set() - - # Try to resolve domain to IP - try: - ip = socket.gethostbyname(self.domain) - ips_to_search.add(ip) - except socket.gaierror as e: - logger.info(f"Failed to resolve domain '{self.domain}': {e}") - except Exception as e: - logger.info(f"Unexpected error while resolving domain '{self.domain}': {e}") - - # Add any IPs from other results - for host in self.hosts: - if ':' in host: - # Extract IP from host:ip format - parts = host.split(':') - if len(parts) == 2 and self._is_valid_ip(parts[1]): - ips_to_search.add(parts[1]) - elif self._is_valid_ip(host): - ips_to_search.add(host) - - # Search each IP in Shodan - for ip in ips_to_search: - try: - output_logger.info(f'\tSearching Shodan for {ip}') - shodan_result = await self.shodan.search_ip(ip) - - if ip in shodan_result and isinstance(shodan_result[ip], dict): - self.shodan_data[ip] = shodan_result[ip] - elif ip in shodan_result and isinstance(shodan_result[ip], str): - output_logger.info(f'{ip}: {shodan_result[ip]}') - - await asyncio.sleep(2) # Rate limiting - except Exception as ip_error: - logger.info(f'Error searching Shodan for {ip}: {ip_error}') - continue - - except Exception as e: - logger.info(f'Error processing Shodan: {e}') - - @staticmethod - def _is_valid_ip(ip_str: str) -> bool: - """Check if a string is a valid IP address.""" - import ipaddress - - try: - ipaddress.ip_address(ip_str) - return True - except ValueError: - return False - - async def get_hosts(self) -> set[str]: - """Get all discovered hosts.""" - return self.hosts - - async def get_emails(self) -> set[str]: - """Get all discovered emails.""" - return self.emails diff --git a/theHarvester/discovery/api_endpoints.py b/theHarvester/discovery/api_endpoints.py index 858fd441..6a50fd85 100644 --- a/theHarvester/discovery/api_endpoints.py +++ b/theHarvester/discovery/api_endpoints.py @@ -55,6 +55,7 @@ class SearchApiEndpoints: follow_redirects: bool = True, verify_ssl: bool = True, additional_headers: dict[str, str] | None = None, + exact_paths: bool = False, ) -> None: """Configure an API path scan. @@ -68,6 +69,7 @@ class SearchApiEndpoints: follow_redirects: Whether requests follow redirects. verify_ssl: Whether to verify TLS certificates. additional_headers: Extra HTTP headers to send. + exact_paths: Check only paths listed in the configured wordlist. """ self.word = word @@ -92,12 +94,16 @@ class SearchApiEndpoints: self.user_agent = user_agent or Core.get_user_agent() self.additional_headers = additional_headers or {} self._session: aiohttp.ClientSession | None = None + self.scan_error_type: str | None = None + self.request_error_count = 0 + self.request_error_types: set[str] = set() # Set default wordlist path default_wordlist = os.path.join( os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'wordlists', 'api_endpoints.txt' ) self.wordlist = wordlist or default_wordlist + self.exact_paths = exact_paths # Add comprehensive API paths categorized by functionality self.common_api_paths = [ @@ -384,6 +390,9 @@ class SearchApiEndpoints: async def do_search(self) -> None: """Check common paths with GET, HEAD, and OPTIONS.""" + self.scan_error_type = None + self.request_error_count = 0 + self.request_error_types.clear() session: aiohttp.ClientSession | None = None try: session = aiohttp.ClientSession( @@ -399,12 +408,14 @@ class SearchApiEndpoints: self.logger.warning(f'No endpoints found in wordlist: {self.wordlist}') endpoints = [] - # Add common API paths that might not be in the wordlist - endpoints.extend(self.common_api_paths) - endpoints = list(set(endpoints)) # Remove duplicates + if not self.exact_paths: + endpoints.extend(self.common_api_paths) + endpoints = list(dict.fromkeys(endpoints)) + if not endpoints: + return # Detect base URL schema (http or https) - schema = await self._detect_schema() + schema = await self._detect_schema(endpoints[0]) if self.exact_paths else await self._detect_schema() self.logger.info(f'Detected schema for {self.word}: {schema}') # Generate batches of tasks to control concurrency @@ -426,15 +437,16 @@ class SearchApiEndpoints: await self._post_scan_analysis() except Exception as e: + self.scan_error_type = type(e).__name__ self.logger.error(f'Error in API endpoint scan: {e!s}', exc_info=True) finally: self._session = None if session is not None: await session.close() - async def _detect_schema(self) -> str: + async def _detect_schema(self, path: str = '') -> str: """Detect if the domain supports HTTPS or fall back to HTTP.""" - https_url = f'https://{self.word}' + https_url = f'https://{self.word}{path}' if self._session is None: raise RuntimeError('API endpoint session is not initialized') try: @@ -463,6 +475,9 @@ class SearchApiEndpoints: # Ensure all paths start with / endpoints = [line if line.startswith('/') else f'/{line}' for line in lines] + if self.exact_paths: + return list(dict.fromkeys(endpoints)) + # Add some path variations (with and without trailing slash) variations = [] for endpoint in endpoints: @@ -507,21 +522,28 @@ class SearchApiEndpoints: verify=self.verify_ssl, follow_redirects=self.follow_redirects, request_timeout=self.timeout, + include_metadata=True, ) # Calculate response time response_time = asyncio.get_event_loop().time() - start_time - # If we get a response, process it - if response: - result = self._process_response(url, method, response, response_time) - if result: - return result + if response is None: + self.request_error_count += 1 + self.request_error_types.add('TransportError') + continue + result = self._process_response(url, method, response, response_time) + if result: + return result except TimeoutError: + self.request_error_count += 1 + self.request_error_types.add('TimeoutError') self.logger.debug(f'Timeout for {method} {url}') continue except (aiohttp.ClientError, OSError, TypeError, ValueError, AttributeError) as e: + self.request_error_count += 1 + self.request_error_types.add(type(e).__name__) self.logger.debug(f'Error checking {method} {url}: {e!s}') continue @@ -567,7 +589,10 @@ class SearchApiEndpoints: headers = {} try: - content = getattr(response, 'content', b'') + content_value = getattr(response, 'body', getattr(response, 'content', b'')) + content = content_value.encode() if isinstance(content_value, str) else content_value + if not isinstance(content, bytes): + content = b'' except (TypeError, AttributeError) as e: self.logger.error(f'Failed to get content from response for URL {url}: {e}') content = b'' @@ -576,7 +601,7 @@ class SearchApiEndpoints: self.response_sizes[url] = content_length # Try to get content type from headers - content_type = headers.get('Content-Type', '') + content_type = next((value for name, value in headers.items() if name.casefold() == 'content-type'), '') # Try to create a preview of the response content (up to 200 characters) content_preview = '' diff --git a/theHarvester/discovery/bevigil.py b/theHarvester/discovery/bevigil.py index 09f5ed25..af52b72b 100644 --- a/theHarvester/discovery/bevigil.py +++ b/theHarvester/discovery/bevigil.py @@ -6,7 +6,7 @@ class SearchBeVigil: def __init__(self, word) -> None: self.word = word self.totalhosts: set = set() - self.interestingurls: set = set() + self.urls: set = set() self.key = Core.bevigil_key() if self.key is None: self.key = '' @@ -26,13 +26,13 @@ class SearchBeVigil: responses = await AsyncFetcher.fetch_all([url_endpoint], json=True, proxy=self.proxy, headers=headers) response = responses[0] for url in response['urls']: - self.interestingurls.add(url) + self.urls.add(url) async def get_hostnames(self) -> set: return self.totalhosts - async def get_interestingurls(self) -> set: - return self.interestingurls + async def get_urls(self) -> set: + return self.urls async def process(self, proxy: bool = False) -> None: self.proxy = proxy diff --git a/theHarvester/discovery/builtwith.py b/theHarvester/discovery/builtwith.py index 9c6d1239..d2710ff0 100644 --- a/theHarvester/discovery/builtwith.py +++ b/theHarvester/discovery/builtwith.py @@ -19,7 +19,7 @@ class SearchBuiltWith: self.headers = {'Authorization': f'Bearer {self.api_key}', 'Content-Type': 'application/json'} self.hosts: set[str] = set() self.tech_stack: dict[str, Any] = {} - self.interesting_urls: set[str] = set() + self.urls: set[str] = set() self.frameworks: set[str] = set() self.languages: set[str] = set() self.servers: set[str] = set() @@ -51,7 +51,7 @@ class SearchBuiltWith: if 'domains' in self.tech_stack: self.hosts.update(self.tech_stack['domains']) if 'paths' in self.tech_stack: - self.interesting_urls.update(self.tech_stack['paths']) + self.urls.update(self.tech_stack['paths']) if 'technologies' in self.tech_stack: for tech in self.tech_stack['technologies']: if not isinstance(tech, dict): @@ -80,8 +80,8 @@ class SearchBuiltWith: async def get_tech_stack(self) -> dict: return self.tech_stack - async def get_interesting_urls(self) -> set[str]: - return self.interesting_urls + async def get_urls(self) -> set[str]: + return self.urls async def get_frameworks(self) -> set[str]: return self.frameworks diff --git a/theHarvester/discovery/criminalip.py b/theHarvester/discovery/criminalip.py index 3ecebb9a..1002bc30 100644 --- a/theHarvester/discovery/criminalip.py +++ b/theHarvester/discovery/criminalip.py @@ -181,7 +181,7 @@ class SearchCriminalIP: async def parser(self, jlines): # TODO when new scope field is added to parse lines for potential new scope! # TODO map as_name to asn for asn data - # TODO determine if worth storing interesting urls + # TODO determine if returned URLs are useful if not isinstance(jlines, dict) or 'data' not in jlines.keys() or not isinstance(jlines['data'], dict): logger.info('CriminalIP report has an unexpected structure') return diff --git a/theHarvester/discovery/dnssearch.py b/theHarvester/discovery/dnssearch.py index 1ef64fad..ca283f12 100644 --- a/theHarvester/discovery/dnssearch.py +++ b/theHarvester/discovery/dnssearch.py @@ -32,12 +32,13 @@ class DnsForce: self.subdo = False self.verbose = verbose self.records: dict[str, hostchecker.HostDnsRecords] = {} + self.query_error_count = 0 + self.query_error_types: set[str] = set() # self.dnsserver = [dnsserver] if isinstance(dnsserver, str) else dnsserver # self.dnsserver = list(map(str, dnsserver.split(','))) if isinstance(dnsserver, str) else dnsserver self.dnsserver = dnsserver with DNS_NAMES.open('r') as file: self.list = file.readlines() - self.domain = domain.replace('www.', '') self.list = [f'{word.strip()}.{self.domain}' for word in self.list] async def run(self): @@ -45,6 +46,8 @@ class DnsForce: checker = hostchecker.Checker(self.list, nameservers=self.dnsserver) resolved_pair, hosts, ips = await checker.check() self.records = checker.records + self.query_error_count = getattr(checker, 'query_error_count', 0) + self.query_error_types = set(getattr(checker, 'query_error_types', set())) return resolved_pair, hosts, ips @@ -112,7 +115,7 @@ def list_ips_in_network_range(iprange: str) -> list[str]: return [] -async def reverse_single_ip(ip: str, resolver: DNSResolver) -> str: +async def reverse_single_ip(ip: str, resolver: DNSResolver, error_types: set[str] | None = None) -> str: """Reverse a single IP and output the linked CNAME, if it exists. Parameters @@ -126,13 +129,20 @@ async def reverse_single_ip(ip: str, resolver: DNSResolver) -> str: """ try: - __host = await resolver.gethostbyaddr(ip) - return __host.name if __host else '' - except Exception: + host = await resolver.gethostbyaddr(ip) + return host.name if host else '' + except Exception as error: + if error_types is not None and not hostchecker.is_expected_dns_absence(error): + error_types.add(type(error).__name__) return '' -async def reverse_all_ips_in_range(iprange: str, callback: Callable, nameservers: list[str] | None = None) -> None: +async def reverse_all_ips_in_range( + iprange: str, + callback: Callable, + nameservers: list[str] | None = None, + error_types: set[str] | None = None, +) -> None: """Reverse all the IPs stored in a network range. All the queries are made concurrently. @@ -146,7 +156,8 @@ async def reverse_all_ips_in_range(iprange: str, callback: Callable, nameservers Arbitrary postprocessing function. nameservers: List[str]. Optional list of DNS servers. - + error_types: set[str]. + Optional sink for unexpected resolver or transport error names. Returns ------- out: None. @@ -156,7 +167,7 @@ async def reverse_all_ips_in_range(iprange: str, callback: Callable, nameservers __resolver = DNSResolver(loop=loop, timeout=8, nameservers=nameservers) for __ip in list_ips_in_network_range(iprange): log_query(__ip) - __host = await reverse_single_ip(ip=__ip, resolver=__resolver) + __host = await reverse_single_ip(ip=__ip, resolver=__resolver, error_types=error_types) callback(__host) log_result(__host) diff --git a/theHarvester/discovery/intelxsearch.py b/theHarvester/discovery/intelxsearch.py index 178a4496..03fd5341 100644 --- a/theHarvester/discovery/intelxsearch.py +++ b/theHarvester/discovery/intelxsearch.py @@ -30,7 +30,7 @@ class SearchIntelx: self.results: dict[str, Any] = {} self.emails: list[str] = [] self.hostnames: list[str] = [] - self.interesting_urls: list[str] = [] + self.urls: list[str] = [] self.limit: int = 10000 self.proxy = False self.offset = 0 @@ -80,7 +80,7 @@ class SearchIntelx: intelx_parser = intelxparser.Parser() raw_emails, raw_selectors = await intelx_parser.parse_dictionaries(self.results) emails: set[str] = set() - interesting_urls: set[str] = set() + urls: set[str] = set() hostnames: set[str] = set() for email in raw_emails: @@ -94,16 +94,19 @@ class SearchIntelx: emails.add(f'{address.username}@{normalized_domain}') for selector in raw_selectors: + selector = selector.strip() try: parsed = urlparse(selector if '://' in selector else f'//{selector}') except ValueError: continue - interesting_urls.add(selector) - if normalized_hostname := normalize_scoped_hostname(parsed.hostname, self.word): + normalized_hostname = normalize_scoped_hostname(parsed.hostname, self.word) + if normalized_hostname: hostnames.add(normalized_hostname) + if parsed.scheme in {'http', 'https'} and parsed.netloc: + urls.add(selector) self.emails = sorted(emails) - self.interesting_urls = sorted(interesting_urls) + self.urls = sorted(urls) self.hostnames = sorted(hostnames) async def get_emails(self) -> list[str]: @@ -112,5 +115,5 @@ class SearchIntelx: async def get_hostnames(self) -> list[str]: return self.hostnames - async def get_interestingurls(self) -> list[str]: - return self.interesting_urls + async def get_urls(self) -> list[str]: + return self.urls diff --git a/theHarvester/discovery/rocketreach.py b/theHarvester/discovery/rocketreach.py index b2db7005..a57b160f 100644 --- a/theHarvester/discovery/rocketreach.py +++ b/theHarvester/discovery/rocketreach.py @@ -17,7 +17,7 @@ class SearchRocketReach: self.hosts: set = set() self.proxy = False self.baseurl = 'https://api.rocketreach.co/api/v2/person/search' - self.links: set = set() + self.urls: set = set() self.emails: set = set() self.limit = limit @@ -64,7 +64,7 @@ class SearchRocketReach: for profile in profiles: if 'linkedin_url' in profile: - self.links.add(profile['linkedin_url']) + self.urls.add(profile['linkedin_url']) if profile.get('emails'): for email in profile['emails']: if email.get('email'): @@ -86,8 +86,8 @@ class SearchRocketReach: except Exception as e: logger.info(f'An exception has occurred rocketreach: {e}') - async def get_links(self): - return self.links + async def get_urls(self): + return self.urls async def get_emails(self): return self.emails diff --git a/theHarvester/discovery/shodansearch.py b/theHarvester/discovery/shodansearch.py index 4b504634..3d976f47 100644 --- a/theHarvester/discovery/shodansearch.py +++ b/theHarvester/discovery/shodansearch.py @@ -17,8 +17,10 @@ class SearchShodan: self.api = Shodan(self.key) self.hostdatarow: list = [] self.tracker: OrderedDict = OrderedDict() + self.error_type: str | None = None async def search_ip(self, ip) -> OrderedDict: + self.error_type = None try: ipaddress = ip results = self.api.host(ipaddress) @@ -108,10 +110,15 @@ class SearchShodan: } return self.tracker - except exception.APIError: - logger.info(f'{ip}: Not in Shodan') - self.tracker[ip] = 'Not in Shodan' + except exception.APIError as error: + if str(error).strip().rstrip('.').casefold() == 'no information available for that ip': + logger.info(f'{ip}: Not in Shodan') + self.tracker[ip] = 'Not in Shodan' + else: + self.error_type = type(error).__name__ + self.tracker[ip] = 'Shodan request failed' except Exception as e: - self.tracker[ip] = f'Error occurred in the Shodan IP search module: {e}' + self.error_type = type(e).__name__ + self.tracker[ip] = 'Shodan request failed' return self.tracker diff --git a/theHarvester/discovery/takeover.py b/theHarvester/discovery/takeover.py index 5a60d3f6..47ee3953 100644 --- a/theHarvester/discovery/takeover.py +++ b/theHarvester/discovery/takeover.py @@ -5,7 +5,7 @@ from random import shuffle import ujson -from theHarvester.lib.core import AsyncFetcher, Core +from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse from theHarvester.lib.output import output_logger logger = logging.getLogger(__name__) @@ -19,6 +19,10 @@ class TakeOver: self.fingerprints: dict[str, str] = dict() # https://stackoverflow.com/questions/33080869/python-how-to-create-a-dict-of-dict-of-list-with-defaultdict self.results: defaultdict[str, list] = defaultdict(list) + self.request_count = 0 + self.request_error_count = 0 + self.request_error_types: set[str] = set() + self.scan_error_type: str | None = None async def populate_fingerprints(self): # Thank you to https://github.com/EdOverflow/can-i-take-over-xyz for these fingerprints @@ -78,6 +82,10 @@ class TakeOver: self.results[url].append({match: service}) async def do_take(self) -> None: + self.request_count = 0 + self.request_error_count = 0 + self.request_error_types.clear() + self.scan_error_type = None try: if len(self.hosts) > 0: # Returns a list of tuples in this format: (url, response) @@ -85,21 +93,37 @@ class TakeOver: https_hosts = [f'https://{host}' for host in self.hosts] http_hosts = [f'http://{host}' for host in self.hosts] all_hosts = https_hosts + http_hosts + self.request_count = len(all_hosts) shuffle(all_hosts) - resps: list = await AsyncFetcher.fetch_all(all_hosts, takeover=True, proxy=self.proxy) - for url, resp in tuple(resp for resp in resps if len(resp[1]) >= 1): - await self.check(url, resp) + responses: list[tuple[str, FetcherResponse | None]] = await AsyncFetcher.fetch_all( + all_hosts, + takeover=True, + proxy=self.proxy, + include_metadata=True, + ) + for url, response in responses: + if response is None: + self.request_error_count += 1 + self.request_error_types.add('TransportError') + continue + if response.body: + await self.check(url, response.body) else: return except IndexError: + self.scan_error_type = 'IndexError' logger.info('Response was empty: possible network error or invalid URL.') except ujson.JSONDecodeError: + self.scan_error_type = 'JSONDecodeError' logger.info('Failed to parse JSON: cert fingerprints might be unavailable.') except KeyError as ke: + self.scan_error_type = 'KeyError' logger.info(f'Missing expected field in fingerprint: {ke}') except TypeError as te: + self.scan_error_type = 'TypeError' logger.info(f'Invalid response structure: {te}') except Exception as e: + self.scan_error_type = type(e).__name__ logger.info(f'Unexpected error: {e}') async def process(self, proxy: bool = False) -> None: diff --git a/theHarvester/discovery/urlscan.py b/theHarvester/discovery/urlscan.py index f4cd70aa..1ed8751c 100644 --- a/theHarvester/discovery/urlscan.py +++ b/theHarvester/discovery/urlscan.py @@ -6,7 +6,7 @@ class SearchUrlscan: self.word = word self.totalhosts: set = set() self.totalips: set = set() - self.interestingurls: set = set() + self.urls: set = set() self.totalasns: set = set() self.proxy = False @@ -16,7 +16,7 @@ class SearchUrlscan: resp = response[0] self.totalhosts = {f'{page["page"]["domain"]}' for page in resp['results']} self.totalips = {f'{page["page"]["ip"]}' for page in resp['results'] if 'ip' in page['page']} - self.interestingurls = { + self.urls = { f'{page["page"]["url"]}' for page in resp['results'] if self.word in page['page']['url'] and 'url' in page['page'] } self.totalasns = {f'{page["page"]["asn"]}' for page in resp['results'] if 'asn' in page['page']} @@ -27,8 +27,8 @@ class SearchUrlscan: async def get_ips(self) -> set: return self.totalips - async def get_interestingurls(self) -> set: - return self.interestingurls + async def get_urls(self) -> set: + return self.urls async def get_asns(self) -> set: return self.totalasns diff --git a/theHarvester/discovery/zoomeyesearch.py b/theHarvester/discovery/zoomeyesearch.py index 9fb0a870..fc07afa2 100644 --- a/theHarvester/discovery/zoomeyesearch.py +++ b/theHarvester/discovery/zoomeyesearch.py @@ -4,9 +4,11 @@ import math import re from collections.abc import Iterable from typing import Any +from urllib.parse import urlparse from theHarvester.discovery.constants import MissingKey, get_delay from theHarvester.lib.core import AsyncFetcher, Core +from theHarvester.lib.hostnames import normalize_scoped_hostname from theHarvester.parsers import myparser logger = logging.getLogger(__name__) @@ -29,7 +31,7 @@ class SearchZoomEye: self.proxy = False self.totalasns: list = list() self.totalhosts: list = list() - self.interestingurls: list = list() + self.urls: list = list() self.totalips: list = list() self.totalemails: list = list() # Regex used is directly from: https://github.com/GerbenJavado/LinkFinder/blob/master/linkfinder.py#L29 @@ -62,7 +64,7 @@ class SearchZoomEye: ) (?:"|') # End newline delimiter """ - self.iurl_regex = re.compile(regex_str, re.VERBOSE) + self.url_regex = re.compile(regex_str, re.VERBOSE) def _build_headers(self) -> dict[str, str]: # API v2 uses API-KEY header @@ -205,24 +207,24 @@ class SearchZoomEye: if isinstance(payload, dict): matches = extract_matches(payload) if matches: - hostnames, emails, ips, asns, iurls = await self.parse_matches(matches) + hostnames, emails, ips, asns, urls = await self.parse_matches(matches) self.totalhosts.extend(hostnames) self.totalemails.extend(emails) self.totalips.extend(ips) self.totalasns.extend(asns) - self.interestingurls.extend(iurls) + self.urls.extend(urls) return # Parse first page then loop if isinstance(payload, dict): matches = extract_matches(payload) if matches: - hostnames, emails, ips, asns, iurls = await self.parse_matches(matches) + hostnames, emails, ips, asns, urls = await self.parse_matches(matches) self.totalhosts.extend(hostnames) self.totalemails.extend(emails) self.totalips.extend(ips) self.totalasns.extend(asns) - self.interestingurls.extend(iurls) + self.urls.extend(urls) for num in range(2, self.limit + 1): params = (('query', f'site:{self.word}'), ('page', str(num)), ('size', str(size))) @@ -244,9 +246,9 @@ class SearchZoomEye: break continue - hostnames, emails, ips, asns, iurls = await self.parse_matches(matches) + hostnames, emails, ips, asns, urls = await self.parse_matches(matches) - if len(hostnames) == 0 and len(emails) == 0 and len(ips) == 0 and len(asns) == 0 and len(iurls) == 0: + if len(hostnames) == 0 and len(emails) == 0 and len(ips) == 0 and len(asns) == 0 and len(urls) == 0: nomatches_counter += 1 if nomatches_counter >= 5: break @@ -255,7 +257,7 @@ class SearchZoomEye: self.totalemails.extend(emails) self.totalips.extend(ips) self.totalasns.extend(asns) - self.interestingurls.extend(iurls) + self.urls.extend(urls) if num % 10 == 0: await asyncio.sleep(get_delay() + 1) @@ -263,7 +265,7 @@ class SearchZoomEye: async def parse_matches(self, matches): # Helper function to parse items from match json ips: set[str] = set() - iurls: set[str] = set() + urls: set[str] = set() hostnames: set[str] = set() asns: set[str] = set() emails: set[str] = set() @@ -317,7 +319,7 @@ class SearchZoomEye: if isinstance(v, str): self._safe_add_hostname(hostnames, v) - # Banner/content extraction for emails, hostnames, iurls + # Banner/content extraction for emails, hostnames, and URLs banners = [] portinfo = match.get('portinfo') @@ -344,18 +346,21 @@ class SearchZoomEye: temp_emails = set(await self.parse_emails(content_blob)) emails.update(temp_emails) hostnames.update(set(await self.parse_hostnames(content_blob))) - found_urls = { - str(iurl.group(1)).replace('"', '') - for iurl in re.finditer(self.iurl_regex, content_blob) - if self.word in str(iurl.group(1)) - } - iurls.update(found_urls) + for url_match in re.finditer(self.url_regex, content_blob): + candidate = str(url_match.group(1)).replace('"', '') + try: + parsed = urlparse(candidate) + hostname = normalize_scoped_hostname(parsed.hostname, self.word) + except ValueError: + continue + if parsed.scheme in {'http', 'https'} and parsed.netloc and hostname: + urls.add(candidate) except Exception as e: # Continue processing other matches instead of failing completely logger.info(f'ZoomEye parsing error: {e}') - return hostnames, emails, ips, asns, iurls + return hostnames, emails, ips, asns, urls async def process(self, proxy: bool = False) -> None: self.proxy = proxy @@ -381,5 +386,5 @@ class SearchZoomEye: async def get_asns(self): return set(self.totalasns) - async def get_interestingurls(self): - return set(self.interestingurls) + async def get_urls(self): + return set(self.urls) diff --git a/theHarvester/lib/active_evidence.py b/theHarvester/lib/active_evidence.py new file mode 100644 index 00000000..8154e019 --- /dev/null +++ b/theHarvester/lib/active_evidence.py @@ -0,0 +1,169 @@ +from collections.abc import Iterable, Mapping +from dataclasses import dataclass +from datetime import datetime +from typing import Self + +from theHarvester.lib.evidence_types import EXECUTION_STATUSES, RESULT_KINDS, ExecutionStatus, ResultKind, format_utc + + +@dataclass(frozen=True, order=True, slots=True) +class ActionObservation: + kind: ResultKind + value: str + + def __post_init__(self) -> None: + if self.kind not in RESULT_KINDS: + raise ValueError(f'unknown action observation kind: {self.kind}') + if self.kind == 'screenshot': + raise ValueError('screenshots must be stored as artifacts, not results') + if not isinstance(self.value, str) or not self.value.strip(): + raise ValueError('action observation value must be a non-empty string') + + +@dataclass(frozen=True, order=True, slots=True) +class ArtifactReference: + kind: str + subject_kind: ResultKind + subject_value: str + path: str + media_type: str + size_bytes: int + sha256: str + created_at: datetime + + def __post_init__(self) -> None: + if any(not isinstance(value, str) or not value.strip() for value in (self.kind, self.path, self.media_type)): + raise ValueError('artifact kind, path, and media type must not be empty') + if ( + self.subject_kind not in RESULT_KINDS + or self.subject_kind == 'screenshot' + or not isinstance(self.subject_value, str) + or not self.subject_value.strip() + ): + raise ValueError('artifact must reference a known non-empty result') + if self.size_bytes < 0: + raise ValueError('artifact size must not be negative') + if ( + not isinstance(self.sha256, str) + or len(self.sha256) != 64 + or any(character not in '0123456789abcdef' for character in self.sha256) + ): + raise ValueError('artifact sha256 must be 64 lowercase hexadecimal characters') + if not isinstance(self.created_at, datetime) or self.created_at.tzinfo is None or self.created_at.utcoffset() is None: + raise ValueError('artifact created_at must be timezone-aware') + + def to_dict(self) -> dict[str, object]: + return { + 'kind': self.kind, + 'subject': {'kind': self.subject_kind, 'value': self.subject_value}, + 'file': { + 'path': self.path, + 'media_type': self.media_type, + 'size_bytes': self.size_bytes, + 'sha256': self.sha256, + }, + 'created_at': format_utc(self.created_at), + } + + +@dataclass(frozen=True, slots=True) +class ActionExecution: + action: str + status: ExecutionStatus + duration_ms: float + observations: tuple[ActionObservation, ...] = () + artifacts: tuple[ArtifactReference, ...] = () + error_type: str | None = None + stop_reason: str | None = None + + def __post_init__(self) -> None: + if not isinstance(self.action, str) or not self.action.strip(): + raise ValueError('action must not be empty') + if self.status not in EXECUTION_STATUSES: + raise ValueError(f'unknown execution status: {self.status}') + if self.duration_ms < 0: + raise ValueError('execution duration must not be negative') + if self.observations != tuple(sorted(set(self.observations))): + raise ValueError('action observations must be deduplicated and sorted') + if self.artifacts != tuple(sorted(set(self.artifacts))): + raise ValueError('artifacts must be deduplicated and sorted') + + @classmethod + def finish( + cls, + *, + action: str, + status: ExecutionStatus, + duration_ms: float, + groups: Mapping[ResultKind, Iterable[str]], + artifacts: Iterable[ArtifactReference] = (), + error_type: str | None = None, + stop_reason: str | None = None, + ) -> Self: + observations: set[ActionObservation] = set() + for kind, values in groups.items(): + if kind not in RESULT_KINDS: + raise ValueError(f'unknown action observation kind: {kind}') + if kind == 'screenshot': + raise ValueError('screenshots must be stored as artifacts, not results') + for value in values: + if not isinstance(value, str) or not value.strip(): + raise ValueError('action observation value must be a non-empty string') + observations.add(ActionObservation(kind, value.strip())) + return cls( + action=action.strip(), + status=status, + duration_ms=duration_ms, + observations=tuple(sorted(observations)), + artifacts=tuple(sorted(set(artifacts))), + error_type=error_type, + stop_reason=stop_reason, + ) + + @property + def result_count(self) -> int: + return len(self.observations) + + def to_dict(self) -> dict[str, str | float | int | None]: + return { + 'action': self.action, + 'status': self.status, + 'duration_ms': self.duration_ms, + 'result_count': self.result_count, + 'error_type': self.error_type, + 'stop_reason': self.stop_reason, + } + + +@dataclass(frozen=True, slots=True) +class ActiveEvidence: + executions: tuple[ActionExecution, ...] = () + + def __post_init__(self) -> None: + actions = [execution.action for execution in self.executions] + if len(actions) != len(set(actions)): + raise ValueError('action executions must be unique') + + @property + def observations(self) -> tuple[tuple[str, ActionObservation], ...]: + return tuple((execution.action, observation) for execution in self.executions for observation in execution.observations) + + @property + def artifacts(self) -> tuple[tuple[str, ArtifactReference], ...]: + return tuple((execution.action, artifact) for execution in self.executions for artifact in execution.artifacts) + + +@dataclass(frozen=True, slots=True) +class ActionYield: + action: str + observed_result_count: int + unique_result_count: int + shared_result_count: int + + def to_dict(self) -> dict[str, str | int]: + return { + 'action': self.action, + 'observed_result_count': self.observed_result_count, + 'unique_result_count': self.unique_result_count, + 'shared_result_count': self.shared_result_count, + } diff --git a/theHarvester/lib/api/additional_endpoints.py b/theHarvester/lib/api/additional_endpoints.py deleted file mode 100644 index 8070ff0b..00000000 --- a/theHarvester/lib/api/additional_endpoints.py +++ /dev/null @@ -1,80 +0,0 @@ -import logging -from typing import Annotated, NoReturn - -from fastapi import APIRouter, Depends, HTTPException -from pydantic import BaseModel, Field - -from theHarvester.discovery.additional_apis import AdditionalAPIs -from theHarvester.discovery.haveibeenpwned import SearchHaveIBeenPwned -from theHarvester.discovery.leaklookup import SearchLeakLookup -from theHarvester.lib.api.auth import get_api_key - -router = APIRouter() -logger = logging.getLogger(__name__) - - -class DomainRequest(BaseModel): - domain: str = Field(..., min_length=3) - api_keys: dict[str, str] | None = None - - -def _raise_processing_error(endpoint: str, exc: Exception) -> NoReturn: - logger.exception(f'Error processing additional API endpoint {endpoint}') - raise HTTPException(status_code=500, detail='An error occurred while processing your request') from exc - - -@router.post('/breaches') -async def get_breaches(request: DomainRequest, _api_key: Annotated[str, Depends(get_api_key)]): - """Get breach information for a domain using HaveIBeenPwned.""" - try: - search = SearchHaveIBeenPwned(request.domain) - await search.process() - return {'status': 'success', 'data': await search.get_breaches()} - except Exception as e: - _raise_processing_error('breaches', e) - - -@router.post('/leaks') -async def get_leaks(request: DomainRequest, _api_key: Annotated[str, Depends(get_api_key)]): - """Get leaked credentials for a domain using Leak-Lookup.""" - try: - search = SearchLeakLookup(request.domain) - await search.process() - return {'status': 'success', 'data': await search.get_leaks()} - except Exception as e: - _raise_processing_error('leaks', e) - - -@router.post('/security-score') -async def get_security_score(request: DomainRequest, _api_key: Annotated[str, Depends(get_api_key)]): - """Get security scorecard for a domain.""" - try: - apis = AdditionalAPIs(request.domain, request.api_keys or {}) - await apis._process_securityscorecard() - results = apis.results['security_score'] - return {'status': 'success', 'data': results} - except Exception as e: - _raise_processing_error('security-score', e) - - -@router.post('/tech-stack') -async def get_tech_stack(request: DomainRequest, _api_key: Annotated[str, Depends(get_api_key)]): - """Get technology stack information for a domain using BuiltWith.""" - try: - apis = AdditionalAPIs(request.domain, request.api_keys or {}) - await apis._process_builtwith() - results = apis.results['tech_stack'] - return {'status': 'success', 'data': results} - except Exception as e: - _raise_processing_error('tech-stack', e) - - -@router.post('/all') -async def get_all_info(request: DomainRequest, _api_key: Annotated[str, Depends(get_api_key)]): - """Get all additional information for a domain.""" - try: - apis = AdditionalAPIs(request.domain, request.api_keys or {}) - results = await apis.process() - return {'status': 'success', 'data': results} - except Exception as e: - _raise_processing_error('all', e) diff --git a/theHarvester/lib/api/api.py b/theHarvester/lib/api/api.py index 81b222f1..e53b26d7 100644 --- a/theHarvester/lib/api/api.py +++ b/theHarvester/lib/api/api.py @@ -1,528 +1,41 @@ -import argparse -import ipaddress -import logging -import os -from collections.abc import AsyncIterator -from contextlib import asynccontextmanager -from datetime import datetime -from typing import Annotated, Any, cast -from uuid import UUID +from __future__ import annotations -from fastapi import Depends, FastAPI, Header, HTTPException, Query, Request, status -from fastapi.middleware.cors import CORSMiddleware -from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse, Response -from pydantic import BaseModel, Field -from slowapi import Limiter, _rate_limit_exceeded_handler -from slowapi.errors import RateLimitExceeded -from slowapi.util import get_remote_address +from contextlib import asynccontextmanager +from pathlib import Path +from typing import TYPE_CHECKING + +from fastapi import FastAPI from starlette.staticfiles import StaticFiles -from theHarvester import __main__ -from theHarvester.lib.api.additional_endpoints import router as additional_router -from theHarvester.lib.api.auth import get_api_key -from theHarvester.lib.completed_result import ResultKind -from theHarvester.lib.database import ResultStore, dispose_sqlite_databases -from theHarvester.lib.recursive_dns import DEFAULT_RECURSIVE_DNS_QUERY_LIMIT +from theHarvester import __version__ +from theHarvester.lib.api.run_worker import start_worker, stop_worker +from theHarvester.lib.api.runs import router as api_router +from theHarvester.lib.database import dispose_sqlite_databases -logger = logging.getLogger(__name__) - -API_RATE_LIMIT = os.getenv('API_RATE_LIMIT', '5/minute') - - -# Define Pydantic models for request and response validation -class QueryResponse(BaseModel): - asns: list[str] = Field(default_factory=list, description='List of ASNs') - interesting_urls: list[str] = Field(default_factory=list, description='List of interesting URLs') - twitter_people: list[str] = Field(default_factory=list, description='List of Twitter people') - linkedin_people: list[dict] = Field(default_factory=list, description='List of LinkedIn people') - linkedin_links: list[str] = Field(default_factory=list, description='List of LinkedIn links') - trello_urls: list[str] = Field(default_factory=list, description='List of discovered URLs (legacy field name)') - ips: list[str] = Field(default_factory=list, description='List of IPs') - emails: list[str] = Field(default_factory=list, description='List of emails') - hosts: list[str] = Field(default_factory=list, description='List of hosts') - breaches: list[str] = Field(default_factory=list, description='List of breach names') - - -class ErrorResponse(BaseModel): - detail: str = Field(..., description='Error message') - error_type: str | None = Field(None, description='Type of error') - traceback: str | None = Field(None, description='Error traceback') - - -limiter = Limiter(key_func=get_remote_address) +if TYPE_CHECKING: + from collections.abc import AsyncIterator @asynccontextmanager async def lifespan(_app: FastAPI) -> AsyncIterator[None]: - manager = ResultStore() - await manager.initialize() + await start_worker() try: yield finally: - await dispose_sqlite_databases() + try: + await stop_worker() + finally: + await dispose_sqlite_databases() app = FastAPI( - title='Restful Harvest', - description='Rest API for theHarvester powered by FastAPI', - version='0.0.4', + title='theHarvester API', + description='Local API for finite theHarvester enumeration runs', + version=__version__, docs_url='/docs', redoc_url='/redoc', lifespan=lifespan, ) -app.state.limiter = limiter -app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) # type: ignore - -# Add CORS middleware -app.add_middleware( - cast('Any', CORSMiddleware), - allow_origins=['*'], - allow_credentials=False, - allow_methods=['GET', 'POST'], - allow_headers=['*'], -) - -# Include additional endpoints -app.include_router(additional_router, prefix='/additional', tags=['Additional APIs']) - -# This is where we will host files that arise if the user specifies a filename -try: - app.mount('/static', StaticFiles(directory='theHarvester/lib/api/static/'), name='static') -except RuntimeError: - static_path = os.path.expanduser('~/.local/share/theHarvester/static/') - if not os.path.isdir(static_path): - os.makedirs(static_path) - app.mount( - '/static', - StaticFiles(directory=static_path), - name='static', - ) - - -@app.get('/', response_class=HTMLResponse) -async def root(*, user_agent: Annotated[str | None, Header()] = None) -> Response: - """Root endpoint that displays the theHarvester logo and links to the GitHub repository. - - Also performs basic user agent filtering to redirect suspicious bots. - """ - # Very basic user agent filtering - if user_agent and ('gobuster' in user_agent or 'sqlmap' in user_agent or 'rustbuster' in user_agent): - response = RedirectResponse(app.url_path_for('bot')) - return response - - return HTMLResponse( - """ - - - - theHarvester API - - - -
- - - theHarvester logo - - - - - - """ - ) - - -# Define Pydantic model for bot response -class BotResponse(BaseModel): - bot: str = Field(..., description='Bot message') - - -@app.get('/nicebot', response_model=BotResponse) -async def bot() -> Response: - """Easter egg endpoint for bots. - - Returns a Star Wars reference when accessed. - """ - return JSONResponse({'bot': 'These are not the droids you are looking for'}) - - -# Define Pydantic model for sources response -class SourcesResponse(BaseModel): - sources: list[str] = Field(..., description='List of supported data sources') - - -class CompletedRunSummary(BaseModel): - run_id: UUID - target: str - started_at: datetime - completed_at: datetime - result_count: int - - -class CompletedResultItem(BaseModel): - type: ResultKind - value: str - - -class CompletedRunDetail(CompletedRunSummary): - results: list[CompletedResultItem] - - -@app.get( - '/sources', - response_model=SourcesResponse, - responses={ - status.HTTP_500_INTERNAL_SERVER_ERROR: {'model': ErrorResponse}, - status.HTTP_429_TOO_MANY_REQUESTS: {'model': ErrorResponse}, - }, -) -@limiter.limit(API_RATE_LIMIT) -async def getsources(request: Request) -> Response: - """Endpoint to query for available sources theHarvester supports. - - Returns a list of all supported data sources that can be used with the query endpoint. - Rate limit is configurable via CLI argument (default: 5 requests per minute). - """ - try: - sources = __main__.Core.get_supportedengines() - return JSONResponse({'sources': sources}) - except Exception as e: - logger.exception('Error in getsources endpoint') - - return JSONResponse( - { - 'detail': 'An error occurred while retrieving sources', - 'error_type': type(e).__name__, - }, - status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, - ) - - -@app.get( - '/runs', - response_model=list[CompletedRunSummary], - responses={status.HTTP_429_TOO_MANY_REQUESTS: {'model': ErrorResponse}}, -) -@limiter.limit(API_RATE_LIMIT) -async def list_runs( - request: Request, - _api_key: Annotated[str, Depends(get_api_key)], - limit: Annotated[int, Query(ge=1, le=500)] = 50, -) -> list[dict[str, object]]: - """List recently completed enumeration runs.""" - manager = ResultStore() - await manager.initialize() - return await manager.list_runs(limit=limit) - - -@app.get( - '/runs/{run_id}', - response_model=CompletedRunDetail, - responses={ - status.HTTP_404_NOT_FOUND: {'model': ErrorResponse}, - status.HTTP_429_TOO_MANY_REQUESTS: {'model': ErrorResponse}, - }, -) -@limiter.limit(API_RATE_LIMIT) -async def get_run( - request: Request, - run_id: UUID, - _api_key: Annotated[str, Depends(get_api_key)], -) -> dict[str, object]: - """Retrieve one completed enumeration run with its normalized evidence.""" - manager = ResultStore() - await manager.initialize() - try: - result = await manager.load_run(run_id) - except LookupError as error: - raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='Completed run not found') from error - return { - 'run_id': str(result.run_id), - 'target': result.target, - 'started_at': result.started_at.isoformat(), - 'completed_at': result.completed_at.isoformat(), - 'result_count': len(result.results), - 'results': [{'type': kind, 'value': value} for kind, value in result.results], - } - - -# Define Pydantic model for DNS brute force response -class DnsBruteResponse(BaseModel): - dns_bruteforce: list[str] = Field(default_factory=list, description='List of DNS brute force results') - - -@app.get( - '/dnsbrute', - response_model=DnsBruteResponse, - responses={ - status.HTTP_500_INTERNAL_SERVER_ERROR: {'model': ErrorResponse}, - status.HTTP_400_BAD_REQUEST: {'model': ErrorResponse}, - status.HTTP_429_TOO_MANY_REQUESTS: {'model': ErrorResponse}, - }, -) -@limiter.limit(API_RATE_LIMIT) -async def dnsbrute( - request: Request, - domain: Annotated[str, Query(min_length=3, description='Domain to be brute forced')], - user_agent: Annotated[str | None, Header()] = None, - dns_resolve: Annotated[ - str, Query(description='Perform DNS resolution on subdomains with a resolver list or passed in resolvers') - ] = '', -) -> Response: - """Endpoint for DNS brute forcing. - - This endpoint performs DNS brute force on the specified domain and returns the results. - Rate limit is configurable via CLI argument (default: 5 requests per minute). - """ - # Basic user agent filtering - if user_agent and ('gobuster' in user_agent or 'sqlmap' in user_agent or 'rustbuster' in user_agent): - response = RedirectResponse(app.url_path_for('bot')) - return response - - try: - # Validate domain - if not domain or len(domain) < 3: - raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail='Domain must be at least 3 characters long') - - # Call the main function with the provided parameters - dns_bruteforce = await __main__.start( - argparse.Namespace( - dns_brute=True, - dns_lookup=False, - dns_server=False, - dns_tld=False, - domain=domain, - filename='', - google_dork=False, - limit=500, - proxies=False, - shodan=False, - source=','.join([]), - start=0, - take_over=False, - wordlist='', - api_scan=False, - dns_resolve=dns_resolve, - ) - ) - - return JSONResponse({'dns_bruteforce': dns_bruteforce}) - - except HTTPException as e: - # Re-raise HTTP exceptions - raise e - except Exception as e: - logger.exception('Error in dnsbrute endpoint') - - return JSONResponse( - { - 'detail': 'An error occurred while processing your request', - 'error_type': type(e).__name__, - }, - status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, - ) - - -@app.get( - '/query', - response_model=QueryResponse, - responses={ - status.HTTP_500_INTERNAL_SERVER_ERROR: {'model': ErrorResponse}, - status.HTTP_400_BAD_REQUEST: {'model': ErrorResponse}, - status.HTTP_429_TOO_MANY_REQUESTS: {'model': ErrorResponse}, - }, -) -@limiter.limit(API_RATE_LIMIT) -async def query( - request: Request, - source: Annotated[ - list[str], - Query( - description=( - 'Source names or source capabilities to query. Multiple capabilities select the union of matching ' - 'sources; they do not filter returned fields.' - ) - ), - ], - domain: Annotated[str, Query(min_length=3, description='Domain to be harvested')], - dns_server: Annotated[ - str, - Query(description='Accepted for compatibility but currently unused; use dns_resolve to select resolvers.'), - ] = '', - user_agent: Annotated[str | None, Header()] = None, - x_api_key: Annotated[str | None, Header(alias='X-API-Key')] = None, - dns_brute: Annotated[bool, Query(description='Perform a DNS brute force on the domain')] = False, - dns_lookup: Annotated[ - bool, - Query( - description=( - 'Perform PTR lookups across the /24 network containing each discovered IPv4 address. ' - 'This sends active DNS queries.' - ) - ), - ] = False, - dns_resolve: Annotated[str, Query(description='Resolve discovered hostnames using resolver IPs or a resolver file')] = '', - dns_recursive_depth: Annotated[int, Query(ge=0, description='Maximum recursive DNS discovery depth. Zero disables it.')] = 0, - dns_recursive_query_limit: Annotated[ - int, Query(gt=0, description='Hard cap on recursive DNS record queries across all resolver vantages') - ] = DEFAULT_RECURSIVE_DNS_QUERY_LIMIT, - dns_recursive_runtime_seconds: Annotated[ - float, Query(gt=0, allow_inf_nan=False, description='Hard runtime cap in seconds for recursive DNS discovery') - ] = 60.0, - filename: Annotated[ - str, - Query(description=('Write uniquely prefixed server-side XML, JSON, and JSONL files using NAME as the filename suffix.')), - ] = '', - proxies: Annotated[ - bool, - Query(description='Use proxies.yaml for supported discovery-source and takeover requests.'), - ] = False, - shodan: Annotated[bool, Query(description='Use Shodan to query discovered hosts')] = False, - take_over: Annotated[ - bool, - Query(description='Check discovered hosts for known takeover indicators, using configured proxies when enabled.'), - ] = False, - wordlist: Annotated[str, Query(description='Path to the endpoint wordlist used by api_scan')] = '', - api_scan: Annotated[ - bool, - Query(description='Check common API paths with GET, HEAD, and OPTIONS. Requests follow redirects.'), - ] = False, - limit: Annotated[int, Query(description='Maximum results requested from each source that supports result limits')] = 500, - start: Annotated[int, Query(description='Result offset for sources that support pagination')] = 0, -) -> Response: - """Query function that allows user to query theHarvester rest API. - - This endpoint performs searches using the specified data sources and returns the results. - Rate limit is configurable via CLI argument (default: 5 requests per minute). - """ - # Basic user agent filtering - if user_agent and ('gobuster' in user_agent or 'sqlmap' in user_agent or 'rustbuster' in user_agent): - response = RedirectResponse(app.url_path_for('bot')) - return response - - try: - # Validate sources - selected_sources = __main__.Core.expand_source_selection(','.join(source)) - credentialed_source = any( - source_name in selected_sources and bool((key_getter() or '').strip()) - for source_name, key_getter in ( - ('dehashed', __main__.Core.dehashed_key), - ('hibpverified', __main__.Core.hibpverified_key), - ('leaklookup', __main__.Core.leaklookup_key), - ) - ) - if credentialed_source: - get_api_key(x_api_key) - supported_engines = __main__.Core.get_supportedengines() - for s in selected_sources: - if s not in supported_engines: - raise HTTPException( - status_code=status.HTTP_400_BAD_REQUEST, - detail=f"Source '{s}' is not supported. Supported sources: {', '.join(supported_engines)}", - ) - - if dns_recursive_depth > 0: - get_api_key(x_api_key) - try: - recursive_resolvers = { - str(ipaddress.ip_address(value.strip())) for value in dns_resolve.split(',') if value.strip() - } - except ValueError as error: - raise HTTPException( - status_code=status.HTTP_400_BAD_REQUEST, - detail='recursive DNS requires exactly three distinct resolver IPs', - ) from error - if len(recursive_resolvers) != 3: - raise HTTPException( - status_code=status.HTTP_400_BAD_REQUEST, - detail='recursive DNS requires exactly three distinct resolver IPs', - ) - - # Call the main function with the provided parameters - ( - asns, - iurls, - twitter_people_list, - linkedin_people_list, - linkedin_links, - aurls, - aips, - aemails, - ahosts, - abreaches, - ) = await __main__.start( - argparse.Namespace( - dns_brute=dns_brute, - dns_lookup=dns_lookup, - dns_server=dns_server, - domain=domain, - filename=filename, - limit=limit, - proxies=proxies, - shodan=shodan, - source=','.join(selected_sources), - start=start, - take_over=take_over, - wordlist=wordlist, - api_scan=api_scan, - dns_resolve=dns_resolve, - dns_recursive_depth=dns_recursive_depth, - dns_recursive_query_limit=dns_recursive_query_limit, - dns_recursive_runtime_seconds=dns_recursive_runtime_seconds, - quiet=False, - screenshot='', - ), - persist_completed_result=True, - include_breaches=True, - ) - - # Return the results using the Pydantic model - return JSONResponse( - { - 'asns': asns, - 'interesting_urls': iurls, - 'twitter_people': twitter_people_list, - 'linkedin_people': linkedin_people_list, - 'linkedin_links': linkedin_links, - 'trello_urls': aurls, - 'ips': aips, - 'emails': aemails, - 'hosts': ahosts, - 'breaches': abreaches, - } - ) - except HTTPException as e: - # Re-raise HTTP exceptions - raise e - except Exception as e: - logger.exception('Error in query endpoint') - - return JSONResponse( - { - 'detail': 'An error occurred while processing your request', - 'error_type': type(e).__name__, - }, - status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, - ) +app.include_router(api_router) +STATIC_DIRECTORY = Path(__file__).resolve().parent / 'static' +app.mount('/static', StaticFiles(directory=STATIC_DIRECTORY), name='static') diff --git a/theHarvester/lib/api/api_example.py b/theHarvester/lib/api/api_example.py deleted file mode 100644 index 04d616bb..00000000 --- a/theHarvester/lib/api/api_example.py +++ /dev/null @@ -1,127 +0,0 @@ -"""Example script to query theHarvester rest API, obtain results, and write out to stdout as well as an html""" - -import asyncio -import logging - -import aiohttp -import netaddr - -from theHarvester.lib.output import configure_logging, output_logger, print_section, sorted_unique - -logger = logging.getLogger(__name__) - - -async def fetch_json(session, url): - try: - async with session.get(url) as response: - response.raise_for_status() # Raise an exception for 4XX/5XX responses - return await response.json() - except Exception as e: - logger.info(f'Error fetching data from {url}: {e}') - return {} - - -async def fetch(session, url): - try: - async with session.get(url) as response: - response.raise_for_status() # Raise an exception for 4XX/5XX responses - return await response.text() - except Exception as e: - logger.info(f'Error fetching data from {url}: {e}') - return '' - - -async def main() -> None: - """Just a simple example of how to interact with the rest api - you can use httpx instead of aiohttp or whatever you best see fit - """ - url = 'http://127.0.0.1:5000' - domain = 'netflix.com' - query_url = f'{url}/query?dns_brute=false&dns_lookup=false&dns_tld=false&proxies=false&shodan=false&take_over=false&virtual_host=false&api_scan=false&source=otx&source=subdomaincenter&limit=500&start=0&domain={domain}' - - async with aiohttp.ClientSession() as session: - fetched_json = await fetch_json(session, query_url) - total_asns = fetched_json.get('asns', []) - interesting_urls = fetched_json.get('interesting_urls', []) - twitter_people_list_tracker = fetched_json.get('twitter_people', []) - linkedin_people_list_tracker = fetched_json.get('linkedin_people', []) - linkedin_links_tracker = fetched_json.get('linkedin_links', []) - trello_urls = fetched_json.get('trello_urls', []) - ips = fetched_json.get('ips', []) - emails = fetched_json.get('emails', []) - hosts = fetched_json.get('hosts', []) - - if len(total_asns) > 0: - print_section(f'\n[*] ASNS found: {len(total_asns)}', total_asns, '--------------------') - total_asns = sorted_unique(total_asns) - - if len(interesting_urls) > 0: - print_section(f'\n[*] Interesting Urls found: {len(interesting_urls)}', interesting_urls, '--------------------') - interesting_urls = sorted_unique(interesting_urls) - - if len(twitter_people_list_tracker) == 0: - output_logger.info('\n[*] No Twitter users found.') - elif len(twitter_people_list_tracker) >= 1: - print_section( - '\n[*] Twitter Users found: ' + str(len(twitter_people_list_tracker)), - twitter_people_list_tracker, - '---------------------', - ) - twitter_people_list_tracker = sorted_unique(twitter_people_list_tracker) - - if len(linkedin_people_list_tracker) == 0: - output_logger.info('\n[*] No LinkedIn users found.') - elif len(linkedin_people_list_tracker) >= 1: - print_section( - '\n[*] LinkedIn Users found: ' + str(len(linkedin_people_list_tracker)), - linkedin_people_list_tracker, - '---------------------', - ) - linkedin_people_list_tracker = sorted_unique(linkedin_people_list_tracker) - - if len(linkedin_links_tracker) == 0: - output_logger.info('\n[*] No LinkedIn links found.') - else: - print_section( - f'\n[*] LinkedIn Links found: {len(linkedin_links_tracker)}', linkedin_links_tracker, '---------------------' - ) - linkedin_links_tracker = sorted_unique(linkedin_links_tracker) - - length_urls = len(trello_urls) - if length_urls == 0: - output_logger.info('\n[*] No Trello URLs found.') - else: - print_section('\n[*] Trello URLs found: ' + str(length_urls), trello_urls, '--------------------') - - if len(ips) == 0: - output_logger.info('\n[*] No IPs found.') - else: - output_logger.info('\n[*] IPs found: ' + str(len(ips))) - output_logger.info('-------------------') - # use netaddr as the list may contain ipv4 and ipv6 addresses - ip_list = sorted([netaddr.IPAddress(ip.strip()) for ip in set(ips)]) - output_logger.info('\n'.join(map(str, ip_list))) - - if len(emails) == 0: - output_logger.info('\n[*] No emails found.') - else: - output_logger.info('\n[*] Emails found: ' + str(len(emails))) - output_logger.info('----------------------') - all_emails = sorted_unique(emails) - output_logger.info('\n'.join(all_emails)) - - if len(hosts) == 0: - output_logger.info('\n[*] No hosts found.\n\n') - else: - output_logger.info('\n[*] Hosts found: ' + str(len(hosts))) - output_logger.info('---------------------') - output_logger.info('\n'.join(hosts)) - - -def entry_point() -> None: - configure_logging(verbose=True) - asyncio.run(main()) - - -if __name__ == '__main__': - entry_point() diff --git a/theHarvester/lib/api/auth.py b/theHarvester/lib/api/auth.py index a3f69bde..95579b47 100644 --- a/theHarvester/lib/api/auth.py +++ b/theHarvester/lib/api/auth.py @@ -1,15 +1,32 @@ import os import secrets +from pathlib import Path from typing import Annotated from fastapi import Header, HTTPException, status API_KEY_ENV_VAR = 'THEHARVESTER_API_KEY' +API_KEY_FILE_ENV_VAR = f'{API_KEY_ENV_VAR}_FILE' -def get_api_key(x_api_key: Annotated[str | None, Header(alias='X-API-Key')] = None) -> str: - """Validate the API key used by protected API routes.""" +def _configured_api_key() -> str | None: configured_api_key = os.getenv(API_KEY_ENV_VAR) + if configured_api_key: + return configured_api_key + configured_api_key_file = os.getenv(API_KEY_FILE_ENV_VAR) + if not configured_api_key_file: + return None + try: + return Path(configured_api_key_file).read_text(encoding='utf-8').strip() or None + except OSError: + return None + + +def get_api_key( + x_api_key: Annotated[str | None, Header(alias='X-API-Key')] = None, +) -> str: + """Validate the API key used by protected API routes.""" + configured_api_key = _configured_api_key() if not configured_api_key: raise HTTPException( status_code=status.HTTP_503_SERVICE_UNAVAILABLE, @@ -22,4 +39,4 @@ def get_api_key(x_api_key: Annotated[str | None, Header(alias='X-API-Key')] = No detail='Invalid API key', ) - return x_api_key + return configured_api_key diff --git a/theHarvester/lib/api/run_artifacts.py b/theHarvester/lib/api/run_artifacts.py new file mode 100644 index 00000000..3fd5c449 --- /dev/null +++ b/theHarvester/lib/api/run_artifacts.py @@ -0,0 +1,67 @@ +from __future__ import annotations + +import json +import os +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from fastapi import HTTPException + +from theHarvester.lib.database import ResultStore + +from .run_evidence import validate_evidence + + +@dataclass(frozen=True, slots=True) +class RunPaths: + database: Path + artifacts: Path + + @classmethod + def configured(cls, database: str | Path | None = None) -> RunPaths: + database_path = Path(database or os.getenv('THEHARVESTER_RUN_DB') or ResultStore().database) + database_path = database_path.expanduser() + configured_artifacts = os.getenv('THEHARVESTER_RUN_ARTIFACTS') + artifact_root = ( + Path(configured_artifacts).expanduser() if configured_artifacts else database_path.parent / 'run-artifacts' + ) + return cls(database=database_path, artifacts=artifact_root) + + def artifact_directory(self, run_id: str) -> Path: + return self.artifacts / run_id + + +def ensure_private_directory(path: Path) -> None: + path.mkdir(parents=True, exist_ok=True, mode=0o700) + if path.is_symlink(): + raise OSError(f'Refusing symlinked theHarvester directory: {path}') + path.chmod(0o700) + + +def read_child_evidence( + artifact_dir: Path, + expected_target: str | None = None, +) -> tuple[dict[str, Any] | None, str | None]: + evidence_path = artifact_dir / 'evidence.json' + if not evidence_path.is_file(): + return None, None + try: + evidence = validate_evidence(json.loads(evidence_path.read_text(encoding='utf-8'))) + if expected_target is not None and evidence.get('target') != expected_target: + return None, 'Child evidence target does not match run target' + return evidence, None + except (OSError, json.JSONDecodeError, HTTPException) as error: + return None, f'Child evidence is invalid: {error}' + + +def write_child_evidence(artifact_dir: Path, evidence: Any, *, partial: bool) -> None: + payload = evidence.evidence_dict() + if partial: + payload['status'] = 'partial' + temporary = artifact_dir / 'evidence.json.tmp' + temporary.write_text(json.dumps(payload), encoding='utf-8') + temporary.chmod(0o600) + evidence_path = artifact_dir / 'evidence.json' + temporary.replace(evidence_path) + evidence_path.chmod(0o600) diff --git a/theHarvester/lib/api/run_evidence.py b/theHarvester/lib/api/run_evidence.py new file mode 100644 index 00000000..7c484614 --- /dev/null +++ b/theHarvester/lib/api/run_evidence.py @@ -0,0 +1,98 @@ +from __future__ import annotations + +from datetime import datetime, timedelta +from typing import Any +from uuid import UUID + +from fastapi import HTTPException, status + +from theHarvester.lib.completed_result import parse_result_jsonl +from theHarvester.lib.evidence_types import EVIDENCE_STATUSES + +from .run_models import _normalize_target + + +def parse_jsonl_import(body: bytes) -> dict[str, Any]: + try: + summary, findings = parse_result_jsonl(body) + except ValueError as error: + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(error)) from error + try: + raw_run_id = summary.get('run_id') + if not isinstance(raw_run_id, str): + raise ValueError + run_id = str(UUID(raw_run_id)) + except ValueError as error: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail='JSONL summary must contain a UUID run_id', + ) from error + timestamps: dict[str, datetime] = {} + for field in ('started_at', 'completed_at'): + value = summary.get(field) + try: + timestamp = datetime.fromisoformat(value) if isinstance(value, str) else None + except ValueError: + timestamp = None + if timestamp is None or timestamp.tzinfo is None or timestamp.utcoffset() != timedelta(0): + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f'JSONL summary must contain an ISO-8601 UTC {field}', + ) + timestamps[field] = timestamp + if timestamps['completed_at'] < timestamps['started_at']: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail='JSONL summary completed_at must not be earlier than started_at', + ) + evidence = { + 'run_id': run_id, + 'target': summary.get('target'), + 'status': summary.get('evidence_status'), + 'started_at': summary.get('started_at'), + 'completed_at': summary.get('completed_at'), + 'results': [ + { + 'type': record['type'], + 'value': record['value'], + 'sources': record['sources'], + 'actions': record['actions'], + } + for record in findings + ], + 'source_executions': summary.get('source_executions', []), + 'action_executions': summary.get('action_executions', []), + 'artifacts': summary.get('artifacts', []), + } + return validate_evidence(evidence) + + +def validate_evidence(evidence: dict[str, Any]) -> dict[str, Any]: + if not evidence.get('target'): + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail='Result file does not identify a target') + try: + evidence['target'] = _normalize_target(str(evidence['target'])) + except ValueError as error: + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(error)) from error + if evidence.get('status') not in EVIDENCE_STATUSES: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail='Evidence status must be complete, partial, or failed', + ) + for field in ('started_at', 'completed_at'): + value = evidence.get(field) + if value is not None and not isinstance(value, str): + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f'Evidence field {field} must be a string', + ) + for field in ('results', 'source_executions', 'action_executions', 'artifacts'): + value = evidence.get(field) + if value is None: + evidence[field] = [] + elif not isinstance(value, list): + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f'Evidence field {field} must be an array', + ) + return evidence diff --git a/theHarvester/lib/api/run_models.py b/theHarvester/lib/api/run_models.py new file mode 100644 index 00000000..07c30824 --- /dev/null +++ b/theHarvester/lib/api/run_models.py @@ -0,0 +1,274 @@ +from __future__ import annotations + +import ipaddress +from datetime import UTC, datetime +from typing import Any, Literal + +from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator + +from theHarvester.lib.enumeration import ( + DEFAULT_DNS_RECURSIVE_QUERY_LIMIT, + DEFAULT_DNS_RECURSIVE_RUNTIME_SECONDS, + DEFAULT_RESULT_START, +) +from theHarvester.lib.evidence_types import EvidenceStatus # noqa: TC001 - Pydantic resolves this annotation at runtime +from theHarvester.lib.resolver_selection import DEFAULT_DNS_RESOLVERS, normalize_resolver_addresses +from theHarvester.lib.source_catalog import SOURCE_SPECS, ActivityClass, selected_action_names + + +def utc_now() -> str: + return datetime.now(UTC).isoformat() + + +def _normalize_target(value: str) -> str: + target = value.strip().rstrip('.').lower() + if not target or len(target) > 253 or any(character in target for character in '/?#@'): + raise ValueError('Target must be a hostname or IP address') + try: + return str(ipaddress.ip_address(target)) + except ValueError: + try: + target = target.encode('idna').decode('ascii') + except UnicodeError as error: + raise ValueError('Target must be a valid hostname') from error + labels = target.split('.') + if any( + not label + or len(label) > 63 + or label.startswith('-') + or label.endswith('-') + or not all(character.isalnum() or character == '-' for character in label) + for label in labels + ): + raise ValueError('Target must be a valid hostname') + return target + + +class RunRequest(BaseModel): + model_config = ConfigDict(extra='forbid') + + target: str = Field(description='Authorized domain name or IP address to enumerate.') + sources: list[str] = Field( + max_length=len(SOURCE_SPECS), + description=( + 'Discovery source names or source capabilities. Multiple capabilities select the union of matching ' + 'sources and do not filter result fields. May be empty when a target-only action is selected.' + ), + ) + limit: int = Field( + default=500, + ge=1, + le=10_000, + description='Maximum results requested from each source when that provider supports a limit.', + ) + start: int = Field( + default=DEFAULT_RESULT_START, + ge=0, + description='Starting result offset for providers that support pagination.', + ) + deadline_seconds: int = Field( + default=1800, + ge=30, + le=86_400, + description='Hard deadline in seconds for the whole run, including every selected source and action.', + ) + proxies: bool = Field( + default=False, + description='Use configured proxies for supported discovery sources and takeover requests.', + ) + dns_brute: bool = Field(default=False, description='Try wordlist candidates below the authorized target through DNS.') + dns_lookup: bool = Field( + default=False, + description="Perform reverse DNS lookup across each discovered IPv4 address's /24 network.", + ) + dns_resolve: bool = Field( + default=False, + description='Validate discovered hostnames through the configured resolver addresses.', + ) + dns_resolvers: list[str] = Field( + default_factory=lambda: list(DEFAULT_DNS_RESOLVERS), + min_length=1, + description=('Distinct resolver IPv4 or IPv6 addresses used by DNS actions. Recursive DNS requires exactly three.'), + ) + dns_recursive_depth: int = Field( + default=0, + ge=0, + description='Maximum recursive label depth. Zero disables recursive DNS discovery.', + ) + dns_recursive_query_limit: int = Field( + default=DEFAULT_DNS_RECURSIVE_QUERY_LIMIT, + gt=0, + description='Maximum DNS record queries shared across all three resolver vantages.', + ) + dns_recursive_runtime_seconds: float = Field( + default=DEFAULT_DNS_RECURSIVE_RUNTIME_SECONDS, + gt=0, + allow_inf_nan=False, + description='Maximum wall-clock seconds spent in recursive DNS discovery.', + ) + shodan: bool = Field(default=False, description='Enrich discovered hosts with configured Shodan access.') + screenshot: bool = Field( + default=False, + description='Capture discovered web services, or the authorized target when no discovery sources are selected.', + ) + takeover: bool = Field( + default=False, + description='Check discovered hosts for takeover indicators, using configured proxies when enabled.', + ) + api_scan: bool = Field( + default=False, + description='Request common API paths directly from the authorized target.', + ) + api_scan_paths: list[str] = Field( + default_factory=list, + max_length=500, + description='Optional endpoint paths used by API scan instead of its bundled wordlist.', + ) + + @field_validator('target') + @classmethod + def normalize_target(cls, value: str) -> str: + return _normalize_target(value) + + @field_validator('sources') + @classmethod + def validate_sources(cls, values: list[str]) -> list[str]: + if len(values) != len(set(values)): + raise ValueError('Sources must not contain duplicates') + return values + + @field_validator('dns_resolvers') + @classmethod + def validate_dns_resolvers(cls, values: list[str]) -> list[str]: + return normalize_resolver_addresses(values) + + @field_validator('api_scan_paths') + @classmethod + def validate_api_scan_paths(cls, values: list[str]) -> list[str]: + paths = [value.strip() for value in values] + if any( + not path + or not path.startswith('/') + or len(path) > 2048 + or '://' in path + or any(character in path for character in '\r\n') + for path in paths + ): + raise ValueError('API scan paths must be non-empty URL paths beginning with /') + if len(paths) != len(set(paths)): + raise ValueError('API scan paths must not contain duplicates') + return paths + + @model_validator(mode='after') + def validate_selected_work(self) -> RunRequest: + if not self.sources and not selected_action_names(self.model_dump()): + raise ValueError('Select at least one discovery source or action') + if self.dns_recursive_depth > 0 and len(self.dns_resolvers) != 3: + raise ValueError('Recursive DNS requires exactly three distinct resolver IPs') + return self + + +class SourceResponse(BaseModel): + name: str + activity: ActivityClass + credentials: list[str] + capabilities: list[str] + + +class ActionResponse(BaseModel): + name: str + activity: ActivityClass + + +class SourceCatalogResponse(BaseModel): + sources: list[SourceResponse] + actions: list[ActionResponse] + + +class DatabaseImportResponse(BaseModel): + filename: str + imported_run_ids: list[str] + skipped_run_ids: list[str] + + +class NormalizedResult(BaseModel): + type: str + value: str + sources: list[str] = Field(default_factory=list) + actions: list[str] = Field(default_factory=list) + + +class ScreenshotRecord(BaseModel): + name: str + target: str + url: str + + +RunStatus = Literal['queued', 'running', 'cancelling', 'cancelled', 'completed', 'failed'] +Activity = Literal['P0', 'P1', 'P2'] + + +class ImportedRunRequest(BaseModel): + filename: str + source_run_id: str + sources: list[str] + activities: list[Activity] + + +class RunSummary(BaseModel): + run_id: str + target: str + status: RunStatus + origin: Literal['local', 'imported'] + created_at: str + started_at: str | None + completed_at: str | None + cancellation_requested_at: str | None + error: str | None + sources: list[str] + activities: list[Activity] + evidence_status: EvidenceStatus | None + result_count: int + + +class RunDetail(RunSummary): + request: RunRequest | ImportedRunRequest + results: list[NormalizedResult] + source_executions: list[dict[str, Any]] + action_executions: list[dict[str, Any]] + artifacts: list[dict[str, Any]] + screenshots: list[ScreenshotRecord] + log: str + + +RUN_REQUEST_OPENAPI = { + 'requestBody': { + 'required': True, + 'content': {'application/json': {'schema': RunRequest.model_json_schema()}}, + } +} +IMPORT_REQUEST_OPENAPI = { + 'requestBody': { + 'required': True, + 'content': {'application/x-ndjson': {'schema': {'type': 'string', 'format': 'binary'}}}, + } +} +DATABASE_IMPORT_REQUEST_OPENAPI = { + 'requestBody': { + 'required': True, + 'content': {'application/vnd.sqlite3': {'schema': {'type': 'string', 'format': 'binary'}}}, + } +} +EXPORT_RESPONSES: dict[int | str, dict[str, Any]] = { + 200: { + 'description': 'Normalized run results as JSONL.', + 'content': { + 'application/x-ndjson': { + 'schema': { + 'type': 'string', + 'description': 'UTF-8 JSONL with one summary followed by normalized findings.', + } + }, + }, + } +} diff --git a/theHarvester/lib/api/run_projection.py b/theHarvester/lib/api/run_projection.py new file mode 100644 index 00000000..c09bb859 --- /dev/null +++ b/theHarvester/lib/api/run_projection.py @@ -0,0 +1,97 @@ +from __future__ import annotations + +from pathlib import Path +from typing import Any + +from theHarvester.lib.source_catalog import ( + ActivityClass, + SourceSpec, + activity_classes_for_selection, + get_source_spec, + selected_action_names, +) + + +def activities_for_request(request: dict[str, Any]) -> list[str]: + if request.get('activities'): + return list(request['activities']) + actions = selected_action_names(request) + return [activity.value for activity in activity_classes_for_selection(request.get('sources', []), actions)] + + +def source_spec(name: str) -> SourceSpec | None: + try: + return get_source_spec(name) + except KeyError: + return None + + +def normalized_results(evidence: dict[str, Any] | None) -> list[dict[str, Any]]: + if not evidence: + return [] + results: list[dict[str, Any]] = [] + for item in evidence.get('results') or []: + if isinstance(item, dict) and item.get('type') != 'screenshot': + results.append( + { + 'type': str(item.get('type', 'other')), + 'value': str(item.get('value', '')), + 'sources': sorted({str(source) for source in item.get('sources', [])}), + 'actions': sorted({str(action) for action in item.get('actions', [])}), + } + ) + return results + + +def source_executions(evidence: dict[str, Any] | None) -> list[dict[str, Any]]: + if not evidence: + return [] + executions = evidence.get('source_executions') or [] + return [dict(execution) for execution in executions if isinstance(execution, dict)] + + +def screenshots(evidence: dict[str, Any] | None, run_id: str, artifact_dir: Path) -> list[dict[str, Any]]: + screenshot_dir = artifact_dir / 'screenshots' + if not screenshot_dir.is_dir(): + return [] + allowed_names: set[str] = set() + targets: dict[str, str] = {} + for artifact in (evidence or {}).get('artifacts') or []: + if not isinstance(artifact, dict) or artifact.get('kind') != 'screenshot': + continue + file = artifact.get('file') + subject = artifact.get('subject') + if not isinstance(file, dict) or not isinstance(subject, dict): + continue + name = Path(str(file.get('path', ''))).name + if name and name.endswith('.png'): + allowed_names.add(name) + targets[name] = str(subject.get('value') or Path(name).stem) + return [ + { + 'name': path.name, + 'target': targets.get(path.name, path.stem), + 'url': f'/api/v1/runs/{run_id}/screenshots/{path.name}', + } + for path in sorted(screenshot_dir.glob('*.png')) + if path.name in allowed_names and path.is_file() + ] + + +def activities_for_evidence( + source_executions: list[dict[str, Any]], + action_executions: list[dict[str, Any]], +) -> list[str]: + sources = [str(execution.get('source', '')) for execution in source_executions] + actions = [str(execution.get('action', '')) for execution in action_executions] + explicit: set[ActivityClass] = set() + for execution in (*source_executions, *action_executions): + activity = str(execution.get('activity') or '') + try: + explicit.add(ActivityClass(activity)) + except ValueError: + pass + activities = set(activity_classes_for_selection(sources, actions)) | explicit + if not activities: + activities.add(ActivityClass.PASSIVE) + return [activity.value for activity in ActivityClass if activity in activities] diff --git a/theHarvester/lib/api/run_store.py b/theHarvester/lib/api/run_store.py new file mode 100644 index 00000000..a552921f --- /dev/null +++ b/theHarvester/lib/api/run_store.py @@ -0,0 +1,539 @@ +from __future__ import annotations + +import json +from collections import Counter, defaultdict +from datetime import datetime +from typing import TYPE_CHECKING, Any, cast +from uuid import UUID, uuid4 + +from fastapi import HTTPException, status + +from theHarvester.lib.active_evidence import ActionExecution, ActiveEvidence, ArtifactReference +from theHarvester.lib.completed_result import CompletedResult, ResultObservation, SourceExecution +from theHarvester.lib.database import DuplicateRunError, ResultStore, ResultStoreError, RunLifecycleStore +from theHarvester.lib.evidence_types import EXECUTION_STATUSES, EvidenceStatus, ExecutionStatus, ResultKind + +from .run_artifacts import RunPaths, read_child_evidence +from .run_models import RunRequest, _normalize_target, utc_now +from .run_projection import activities_for_evidence, activities_for_request, normalized_results, screenshots, source_executions + +if TYPE_CHECKING: + from pathlib import Path + +WORKER_LEASE_TIMEOUT_SECONDS = 30 +DATABASE_IMPORT_BATCH_SIZE = 100 + + +def _execution_status(value: object) -> ExecutionStatus: + normalized = str(value) + if normalized not in EXECUTION_STATUSES: + raise ValueError(f'unknown execution status: {normalized}') + return cast('ExecutionStatus', normalized) + + +def _completed_result( + evidence: dict[str, Any], + *, + run_id: UUID, + fallback_started_at: str, + fallback_completed_at: str, +) -> CompletedResult: + results = [item for item in evidence.get('results', []) if isinstance(item, dict)] + groups: dict[ResultKind, set[str]] = defaultdict(set) + source_origins: set[ResultObservation] = set() + source_counts: Counter[str] = Counter() + action_groups: dict[str, dict[ResultKind, set[str]]] = defaultdict(lambda: defaultdict(set)) + for item in results: + kind = cast('ResultKind', str(item['type'])) + value = str(item['value']) + groups[kind].add(value) + for source in set(item.get('sources', [])): + source_name = str(source) + source_origins.add(ResultObservation(source_name, kind, value)) + source_counts[source_name] += 1 + for action in set(item.get('actions', [])): + action_groups[str(action)][kind].add(value) + + source_details = { + str(item['source']): item + for item in evidence.get('source_executions', []) + if isinstance(item, dict) and item.get('source') + } + if len(source_details) != len(evidence.get('source_executions', [])): + raise ValueError('source executions must have unique non-empty names') + if missing_sources := set(source_counts) - set(source_details): + raise ValueError(f'missing source execution: {sorted(missing_sources)[0]}') + source_names = sorted(source_details) + completed_sources = tuple( + SourceExecution( + source=name, + status=_execution_status(source_details.get(name, {}).get('status', 'completed')), + duration_ms=float(source_details.get(name, {}).get('duration_ms', 0)), + result_count=source_counts[name], + error_type=source_details.get(name, {}).get('error_type'), + stop_reason=source_details[name].get('stop_reason'), + ) + for name in source_names + ) + + artifacts_by_action: dict[str, list[ArtifactReference]] = defaultdict(list) + for item in evidence.get('artifacts', []): + if not isinstance(item, dict) or not item.get('action'): + continue + subject = item.get('subject') + file = item.get('file') + if not isinstance(subject, dict) or not isinstance(file, dict): + continue + artifacts_by_action[str(item['action'])].append( + ArtifactReference( + kind=str(item['kind']), + subject_kind=cast('ResultKind', str(subject['kind'])), + subject_value=str(subject['value']), + path=str(file['path']), + media_type=str(file['media_type']), + size_bytes=int(file['size_bytes']), + sha256=str(file['sha256']), + created_at=datetime.fromisoformat(str(item.get('created_at') or fallback_completed_at)), + ) + ) + action_details = { + str(item['action']): item + for item in evidence.get('action_executions', []) + if isinstance(item, dict) and item.get('action') + } + if len(action_details) != len(evidence.get('action_executions', [])): + raise ValueError('action executions must have unique non-empty names') + missing_actions = (set(action_groups) | set(artifacts_by_action)) - set(action_details) + if missing_actions: + raise ValueError(f'missing action execution: {sorted(missing_actions)[0]}') + action_names = sorted(action_details) + active_evidence = ActiveEvidence( + executions=tuple( + ActionExecution.finish( + action=name, + status=_execution_status(action_details.get(name, {}).get('status', 'completed')), + duration_ms=float(action_details.get(name, {}).get('duration_ms', 0)), + groups=action_groups[name], + artifacts=artifacts_by_action[name], + error_type=action_details.get(name, {}).get('error_type'), + stop_reason=action_details.get(name, {}).get('stop_reason'), + ) + for name in action_names + ) + ) + execution_status_is_authoritative = bool(completed_sources or action_names) + return CompletedResult.finish( + run_id=run_id, + target=str(evidence['target']), + started_at=datetime.fromisoformat(str(evidence.get('started_at') or fallback_started_at)), + completed_at=datetime.fromisoformat(str(evidence.get('completed_at') or fallback_completed_at)), + groups=groups, + source_executions=completed_sources, + observations=sorted(source_origins), + active_evidence=active_evidence, + evidence_status=( + cast('EvidenceStatus', str(evidence['status'])) + if evidence.get('status') is not None and not execution_status_is_authoritative + else None + ), + ) + + +def _imported_request(completed: CompletedResult, filename: str, source_run_id: str) -> dict[str, object]: + evidence = completed.evidence_dict() + executions = source_executions(evidence) + raw_action_executions = evidence.get('action_executions', []) + action_executions = ( + [dict(execution) for execution in raw_action_executions if isinstance(execution, dict)] + if isinstance(raw_action_executions, list) + else [] + ) + return { + 'filename': filename, + 'source_run_id': source_run_id, + 'sources': sorted( + { + str(execution.get('source') or execution.get('name')) + for execution in executions + if execution.get('source') or execution.get('name') + } + ), + 'activities': activities_for_evidence(executions, action_executions), + } + + +class RunStore: + """Join API lifecycle state with the canonical SQLAlchemy result store.""" + + def __init__(self, database: str | Path | None = None) -> None: + self.paths = RunPaths.configured(database) + self.database = self.paths.database + self.lifecycle = RunLifecycleStore(self.database) + self.results = ResultStore(self.database) + + def artifact_directory(self, run_id: str) -> Path: + return self.paths.artifact_directory(run_id) + + async def initialize(self) -> None: + self.database.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + await self.lifecycle.initialize() + self.database.chmod(0o600) + + async def _row(self, record: dict[str, object], *, detail: bool = False) -> dict[str, Any]: + request = json.loads(str(record['request_json'])) + evidence = None + if detail and record['evidence_run_id'] is not None: + evidence = (await self.results.load_run(UUID(str(record['evidence_run_id'])))).evidence_dict() + summary_result_count = record.get('result_count') + result = { + 'run_id': record['run_id'], + 'target': record['target'], + 'status': record['status'], + 'origin': record['origin'], + 'created_at': record['created_at'], + 'started_at': record['started_at'], + 'completed_at': record['completed_at'], + 'cancellation_requested_at': record['cancellation_requested_at'], + 'error': record['error'], + 'sources': request.get('sources', []), + 'activities': activities_for_request(request), + 'evidence_status': record['evidence_status'] or (evidence.get('status') if evidence else None), + 'result_count': ( + len(normalized_results(evidence)) + if evidence + else summary_result_count + if isinstance(summary_result_count, int) + else 0 + ), + } + if detail: + result.update( + request=request, + evidence=evidence, + results=normalized_results(evidence), + source_executions=source_executions(evidence), + action_executions=evidence.get('action_executions', []) if evidence else [], + artifacts=evidence.get('artifacts', []) if evidence else [], + screenshots=screenshots(evidence, str(record['run_id']), self.artifact_directory(str(record['run_id']))), + log=record['log'], + ) + return result + + async def create(self, request: RunRequest) -> dict[str, Any]: + await self.initialize() + run_id = str(uuid4()) + await self.lifecycle.create( + run_id=run_id, + target=request.target, + status='queued', + origin='local', + created_at=utc_now(), + request_json=request.model_dump_json(), + ) + run = await self.get(run_id) + assert run is not None + return run + + async def import_evidence(self, evidence: dict[str, Any], filename: str) -> dict[str, Any]: + await self.initialize() + created_at = utc_now() + target = _normalize_target(str(evidence['target'])) + source_run_id = str(evidence['run_id']) + run_id = str(uuid4()) + try: + completed = _completed_result( + evidence, + run_id=UUID(run_id), + fallback_started_at=created_at, + fallback_completed_at=created_at, + ) + except (KeyError, TypeError, ValueError) as error: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f'Invalid run evidence: {error}', + ) from error + if evidence['status'] != completed.status: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail='Evidence status does not match its execution outcomes', + ) + await self._persist_completed(completed) + request = _imported_request(completed, filename, source_run_id) + await self.lifecycle.create( + run_id=run_id, + target=target, + status='completed', + origin='imported', + created_at=created_at, + started_at=completed.started_at.isoformat(), + completed_at=completed.completed_at.isoformat(), + request_json=json.dumps(request), + evidence_run_id=run_id, + evidence_status=completed.status, + ) + run = await self.get(run_id) + assert run is not None + return run + + async def import_database(self, source_path: Path, filename: str) -> dict[str, object]: + await self.initialize() + source = ResultStore(source_path) + imported_run_ids: list[str] = [] + skipped_run_ids: set[str] = set() + reuse_evidence_ids: set[str] = set() + + async def source_summaries(): + offset = 0 + while batch := await source.list_runs(limit=DATABASE_IMPORT_BATCH_SIZE, offset=offset): + for summary in batch: + yield summary + offset += len(batch) + + try: + await source.validate_import_database() + await source.initialize() + async for summary in source_summaries(): + completed = await source.load_run(UUID(str(summary['run_id']))) + run_id = str(completed.run_id) + record = await self.lifecycle.get(run_id) + existing = None + try: + existing = await self.results.load_run(completed.run_id) + except (LookupError, ResultStoreError): + pass + if record is not None or existing is not None: + if record is not None and existing == completed: + skipped_run_ids.add(run_id) + continue + if existing != completed: + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=f'Run ID conflicts with different evidence: {run_id}', + ) + reuse_evidence_ids.add(run_id) + + async for summary in source_summaries(): + completed = await source.load_run(UUID(str(summary['run_id']))) + run_id = str(completed.run_id) + if run_id in skipped_run_ids: + continue + if run_id not in reuse_evidence_ids: + await self.results.save_run(completed) + await self.lifecycle.create( + run_id=run_id, + target=completed.target, + status='completed', + origin='imported', + created_at=completed.completed_at.isoformat(), + started_at=completed.started_at.isoformat(), + completed_at=completed.completed_at.isoformat(), + request_json=json.dumps(_imported_request(completed, filename, run_id)), + evidence_run_id=run_id, + evidence_status=completed.status, + ) + imported_run_ids.append(run_id) + except (ResultStoreError, RuntimeError, ValueError) as error: + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(error)) from error + finally: + await source.dispose() + return { + 'filename': filename, + 'imported_run_ids': sorted(imported_run_ids), + 'skipped_run_ids': sorted(skipped_run_ids), + } + + async def list_runs(self, *, limit: int = 100, offset: int = 0) -> list[dict[str, Any]]: + await self.initialize() + return [await self._row(record) for record in await self.lifecycle.list_records(limit=limit, offset=offset)] + + async def get(self, run_id: str) -> dict[str, Any] | None: + await self.initialize() + record = await self.lifecycle.get(run_id) + return await self._row(record, detail=True) if record else None + + async def load_completed_result(self, run_id: str) -> CompletedResult | None: + """Load the canonical evidence attached to an API run.""" + await self.initialize() + record = await self.lifecycle.get(run_id) + if record is None: + raise LookupError(run_id) + if record['evidence_run_id'] is None: + return None + try: + completed = await self.results.load_run(UUID(str(record['evidence_run_id']))) + except (LookupError, ValueError) as error: + raise ResultStoreError('Attached run evidence does not exist') from error + if completed.target != str(record['target']): + raise ResultStoreError('Attached run evidence target does not match its lifecycle run') + return completed + + async def cancel(self, run_id: str) -> dict[str, Any] | None: + await self.initialize() + try: + record = await self.lifecycle.cancel(run_id, utc_now()) + except ValueError as error: + raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail=f'Run is already {error.args[0]}') from error + return await self._row(record, detail=True) if record else None + + async def recover_orphans(self) -> None: + await self.initialize() + recovered_at = utc_now() + for record in await self.lifecycle.running(): + run_id = str(record['run_id']) + target = str(record['target']) + evidence, evidence_error = read_child_evidence(self.artifact_directory(run_id), target) + error = 'theHarvester restarted before child completion' + if evidence_error: + error += f'; {evidence_error}' + evidence_run_id = None + completed = None + if evidence: + completed = await self._save_evidence( + evidence, + str(record['started_at'] or record['created_at']), + recovered_at, + run_id=UUID(run_id), + expected_target=target, + ) + evidence_run_id = str(completed.run_id) + else: + completed = await self._existing_evidence(run_id, target) + if completed is not None: + evidence_run_id = str(completed.run_id) + evidence_status = completed.status if completed is not None else None + if evidence is not None and completed is not None: + evidence_status = str(evidence.get('status', completed.status)) + await self.lifecycle.fail( + run_id, + status='failed', + completed_at=recovered_at, + error=error, + log=str(record['log']), + evidence_run_id=evidence_run_id, + evidence_status=evidence_status, + ) + + async def acquire_worker_lease(self, owner_id: str) -> bool: + await self.initialize() + return await self.lifecycle.acquire_lease(owner_id, utc_now(), WORKER_LEASE_TIMEOUT_SECONDS) + + async def heartbeat_worker_lease(self, owner_id: str) -> bool: + return await self.lifecycle.heartbeat_lease(owner_id, utc_now()) + + async def release_worker_lease(self, owner_id: str) -> None: + return await self.lifecycle.release_lease(owner_id) + + async def claim_next(self) -> dict[str, Any] | None: + await self.initialize() + record = await self.lifecycle.claim_next(utc_now()) + return await self._row(record, detail=True) if record else None + + async def finish(self, run_id: str, evidence: dict[str, Any] | None, log: str) -> None: + record = await self.lifecycle.get(run_id) + if record is None: + return + evidence_run_id = None + if evidence: + completed = await self._save_evidence( + evidence, + str(record['started_at'] or record['created_at']), + utc_now(), + run_id=UUID(run_id), + expected_target=str(record['target']), + ) + evidence_run_id = str(completed.run_id) + await self.lifecycle.finish( + run_id, + completed_at=utc_now(), + evidence_run_id=evidence_run_id, + evidence_status=str(evidence.get('status', completed.status)) if evidence else None, + log=log[-200_000:], + ) + + async def fail( + self, + run_id: str, + error: str, + log: str, + *, + cancelled: bool = False, + evidence: dict[str, Any] | None = None, + ) -> None: + record = await self.lifecycle.get(run_id) + if record is None: + return + completed_at = utc_now() + evidence_run_id = None + completed = None + if evidence: + completed = await self._save_evidence( + evidence, + str(record['started_at'] or record['created_at']), + completed_at, + run_id=UUID(run_id), + expected_target=str(record['target']), + ) + evidence_run_id = str(completed.run_id) + else: + completed = await self._existing_evidence(run_id, str(record['target'])) + if completed is not None: + evidence_run_id = str(completed.run_id) + evidence_status = completed.status if completed is not None else None + if evidence is not None and completed is not None: + evidence_status = str(evidence.get('status', completed.status)) + await self.lifecycle.fail( + run_id, + status='cancelled' if cancelled else 'failed', + completed_at=completed_at, + error=error, + log=log[-200_000:], + evidence_run_id=evidence_run_id, + evidence_status=evidence_status, + ) + + async def _existing_evidence(self, run_id: str, target: str) -> CompletedResult | None: + try: + completed = await self.results.load_run(UUID(run_id)) + except (LookupError, ResultStoreError, ValueError): + return None + return completed if completed.target == target else None + + async def _save_evidence( + self, + evidence: dict[str, Any], + fallback_started_at: str, + fallback_completed_at: str, + *, + run_id: UUID, + expected_target: str, + ) -> CompletedResult: + try: + completed = _completed_result( + evidence, + run_id=run_id, + fallback_started_at=fallback_started_at, + fallback_completed_at=fallback_completed_at, + ) + except (KeyError, TypeError, ValueError) as error: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f'Invalid run evidence: {error}', + ) from error + if completed.target != expected_target: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail='Evidence target does not match run target', + ) + await self._persist_completed(completed) + return completed + + async def _persist_completed(self, completed: CompletedResult) -> None: + try: + await self.results.save_run(completed) + except DuplicateRunError: + existing = await self.results.load_run(completed.run_id) + if existing != completed: + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=f'Run evidence already exists with different contents: {completed.run_id}', + ) from None diff --git a/theHarvester/lib/api/run_worker.py b/theHarvester/lib/api/run_worker.py new file mode 100644 index 00000000..86809705 --- /dev/null +++ b/theHarvester/lib/api/run_worker.py @@ -0,0 +1,353 @@ +from __future__ import annotations + +import asyncio +import os +import signal +import subprocess +import sys +from argparse import ArgumentParser +from pathlib import Path +from typing import TYPE_CHECKING, Any +from uuid import UUID, uuid4 +from weakref import WeakKeyDictionary + +import anyio + +from theHarvester.lib.enumeration import ( + DEFAULT_DNS_RECURSIVE_QUERY_LIMIT, + DEFAULT_DNS_RECURSIVE_RUNTIME_SECONDS, + DEFAULT_RESULT_START, + EnumerationOptions, +) +from theHarvester.lib.resolver_selection import DEFAULT_DNS_RESOLVERS + +from .run_artifacts import ensure_private_directory, read_child_evidence, write_child_evidence +from .run_store import RunStore + +if TYPE_CHECKING: + from collections.abc import Awaitable, Callable + +_worker_task: asyncio.Task[None] | None = None +_worker_stop: asyncio.Event | None = None +_worker_wakeup: asyncio.Event | None = None +_worker_owner: str | None = None +_process_groups: WeakKeyDictionary[asyncio.subprocess.Process, int] = WeakKeyDictionary() + + +def worker_enabled() -> bool: + return os.getenv('THEHARVESTER_RUN_WORKER', 'enabled').casefold() != 'disabled' + + +def worker_available() -> bool: + return _worker_task is not None and not _worker_task.done() + + +async def _default_process_factory(run_id: str, database: Path, _artifact_dir_path: Path) -> asyncio.subprocess.Process: + process_options = ( + {'creationflags': getattr(subprocess, 'CREATE_NEW_PROCESS_GROUP', 0)} if os.name == 'nt' else {'start_new_session': True} + ) + process = await asyncio.create_subprocess_exec( + sys.executable, + '-m', + 'theHarvester.lib.api.run_worker', + '--execute', + run_id, + '--database', + str(database), + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + **process_options, + ) + if process.pid is not None: + _process_groups[process] = process.pid + return process + + +_process_factory: Callable[[str, Path, Path], Awaitable[asyncio.subprocess.Process]] = _default_process_factory + + +async def _process_output(process: asyncio.subprocess.Process) -> str: + async def read(stream: asyncio.StreamReader | None) -> bytes: + return await stream.read() if stream is not None else b'' + + stdout, stderr = await asyncio.gather(read(process.stdout), read(process.stderr)) + return '\n'.join(part.decode('utf-8', errors='replace').strip() for part in (stdout, stderr) if part).strip() + + +async def _signal_process_tree(process: asyncio.subprocess.Process, *, force: bool) -> None: + process_group = _process_groups.get(process) + if process_group is not None and os.name != 'nt': + try: + os.killpg(process_group, signal.SIGKILL if force else signal.SIGTERM) + except ProcessLookupError: + pass + return + if process_group is not None and os.name == 'nt': + if force: + killer = await asyncio.create_subprocess_exec( + 'taskkill', + '/PID', + str(process.pid), + '/T', + '/F', + stdout=asyncio.subprocess.DEVNULL, + stderr=asyncio.subprocess.DEVNULL, + ) + await killer.wait() + else: + try: + process.send_signal(getattr(signal, 'CTRL_BREAK_EVENT', 1)) + except ProcessLookupError: + pass + return + try: + process.kill() if force else process.terminate() + except ProcessLookupError: + pass + + +async def _stop_process(process: asyncio.subprocess.Process, wait_task: asyncio.Task[int]) -> None: + if process.returncode is None: + await _signal_process_tree(process, force=False) + try: + await asyncio.wait_for(asyncio.shield(wait_task), timeout=2) + except TimeoutError: + if process.returncode is None: + await _signal_process_tree(process, force=True) + await wait_task + + +async def _execute_claimed(store: RunStore, run: dict[str, Any], owner_id: str | None = None) -> None: + run_id = run['run_id'] + artifact_dir = store.artifact_directory(run_id) + ensure_private_directory(artifact_dir) + try: + process = await _process_factory(run_id, store.database, artifact_dir) + except (OSError, RuntimeError) as error: + await store.fail(run_id, f'Could not start child process: {error}', '') + return + wait_task = asyncio.create_task(process.wait()) + output_task = asyncio.create_task(_process_output(process)) + deadline = asyncio.get_running_loop().time() + int(run['request']['deadline_seconds']) + next_heartbeat = 0.0 + while not wait_task.done(): + await asyncio.sleep(0.05) + if owner_id is not None and asyncio.get_running_loop().time() >= next_heartbeat: + if not await store.heartbeat_worker_lease(owner_id): + await _stop_process(process, wait_task) + await store.fail(run_id, 'Worker lost its execution lease', await output_task) + return + next_heartbeat = asyncio.get_running_loop().time() + 5 + current = await store.get(run_id) + stopping = _worker_stop is not None and _worker_stop.is_set() + if current is not None and current['status'] == 'cancelling': + await _stop_process(process, wait_task) + evidence, evidence_error = read_child_evidence(artifact_dir, str(run['target'])) + failure_message = 'Cancelled by operator' + (f'; {evidence_error}' if evidence_error else '') + await store.fail(run_id, failure_message, await output_task, cancelled=True, evidence=evidence) + return + if stopping: + await _stop_process(process, wait_task) + evidence, evidence_error = read_child_evidence(artifact_dir, str(run['target'])) + failure_message = 'theHarvester stopped before child completion' + (f'; {evidence_error}' if evidence_error else '') + await store.fail(run_id, failure_message, await output_task, evidence=evidence) + return + if asyncio.get_running_loop().time() >= deadline: + await _stop_process(process, wait_task) + evidence, evidence_error = read_child_evidence(artifact_dir, str(run['target'])) + failure_message = f'Run exceeded its {run["request"]["deadline_seconds"]} second deadline' + if evidence_error: + failure_message += f'; {evidence_error}' + await store.fail( + run_id, + failure_message, + await output_task, + evidence=evidence, + ) + return + log = await output_task + current = await store.get(run_id) + evidence, evidence_error = read_child_evidence(artifact_dir, str(run['target'])) + if evidence_error: + await store.fail( + run_id, + evidence_error, + log, + cancelled=current is not None and current['status'] == 'cancelling', + ) + return + if current is not None and current['status'] == 'cancelling': + await store.finish(run_id, evidence, log) + elif process.returncode == 0 and evidence is not None: + await store.finish(run_id, evidence, log) + else: + await store.fail( + run_id, f'Child process exited with status {process.returncode} without terminal completion', log, evidence=evidence + ) + + +async def _worker_loop(store: RunStore, owner_id: str) -> None: + assert _worker_stop is not None + assert _worker_wakeup is not None + while not _worker_stop.is_set(): + if not await store.heartbeat_worker_lease(owner_id): + return + run = await store.claim_next() + if run is not None: + await _execute_claimed(store, run, owner_id) + continue + _worker_wakeup.clear() + try: + await asyncio.wait_for(_worker_wakeup.wait(), timeout=0.5) + except TimeoutError: + continue + + +async def _supervise_worker(store: RunStore, owner_id: str) -> None: + assert _worker_stop is not None + assert _worker_wakeup is not None + while not _worker_stop.is_set(): + if await store.acquire_worker_lease(owner_id): + try: + await store.recover_orphans() + except BaseException: + await store.release_worker_lease(owner_id) + raise + await _worker_loop(store, owner_id) + return + _worker_wakeup.clear() + try: + await asyncio.wait_for(_worker_wakeup.wait(), timeout=0.5) + except TimeoutError: + continue + + +async def start_worker() -> None: + global _worker_owner, _worker_stop, _worker_task, _worker_wakeup + if not worker_enabled(): + return + if _worker_task is not None and not _worker_task.done(): + return + store = RunStore() + owner_id = str(uuid4()) + _worker_owner = owner_id + _worker_stop = asyncio.Event() + _worker_wakeup = asyncio.Event() + _worker_task = asyncio.create_task(_supervise_worker(store, owner_id)) + + +async def stop_worker() -> None: + global _worker_owner, _worker_stop, _worker_task, _worker_wakeup + task = _worker_task + owner_id = _worker_owner + try: + if task is not None: + if _worker_stop is not None: + _worker_stop.set() + if _worker_wakeup is not None: + _worker_wakeup.set() + await task + finally: + try: + if owner_id is not None: + await RunStore().release_worker_lease(owner_id) + finally: + _worker_task = None + _worker_owner = None + _worker_stop = None + _worker_wakeup = None + + +def wake_worker() -> None: + if _worker_wakeup is not None: + _worker_wakeup.set() + + +async def _child_execute(run_id: str, database: Path) -> None: + from theHarvester import __main__ as main_module + from theHarvester.lib.completed_result import CompletedResult + + store = RunStore(database) + run = await store.get(run_id) + if run is None or run['status'] not in {'running', 'cancelling'}: + raise RuntimeError('theHarvester run is not executable') + request = run['request'] + artifact_dir = store.artifact_directory(run_id) + ensure_private_directory(artifact_dir) + screenshot_dir = artifact_dir / 'screenshots' + if request.get('screenshot'): + ensure_private_directory(screenshot_dir) + recursive_depth = request.get('dns_recursive_depth', 0) + resolver_list = request.get('dns_resolvers', list(DEFAULT_DNS_RESOLVERS)) + api_scan_wordlist = '' + if request.get('api_scan_paths'): + api_scan_wordlist_path = artifact_dir / 'api-scan-paths.txt' + await anyio.Path(api_scan_wordlist_path).write_text( + ''.join(f'{path}\n' for path in request['api_scan_paths']), + encoding='utf-8', + ) + api_scan_wordlist = str(api_scan_wordlist_path) + args = EnumerationOptions( + api_scan=request.get('api_scan', False), + dns_brute=request.get('dns_brute', False), + dns_lookup=request.get('dns_lookup', False), + dns_recursive_depth=recursive_depth, + dns_recursive_query_limit=request.get('dns_recursive_query_limit', DEFAULT_DNS_RECURSIVE_QUERY_LIMIT), + dns_recursive_runtime_seconds=request.get('dns_recursive_runtime_seconds', DEFAULT_DNS_RECURSIVE_RUNTIME_SECONDS), + dns_resolve=','.join(resolver_list) if request.get('dns_resolve') else '', + dns_resolvers=tuple(resolver_list), + dns_server=None, + domain=run['target'], + filename='', + limit=request['limit'], + proxies=request.get('proxies', False), + quiet=True, + screenshot=str(screenshot_dir) if request.get('screenshot') else '', + shodan=request.get('shodan', False), + source=','.join(request['sources']), + start=request.get('start', DEFAULT_RESULT_START), + take_over=request.get('takeover', False), + wordlist=api_scan_wordlist, + ) + checkpoint_lock = asyncio.Lock() + + async def checkpoint(evidence: CompletedResult) -> None: + async with checkpoint_lock: + write_child_evidence(artifact_dir, evidence, partial=True) + + task = asyncio.create_task( + main_module.start( + args, + completed_result_checkpoint=checkpoint, + return_completed_result=True, + result_database=database, + completed_run_id=UUID(run_id), + ) + ) + loop = asyncio.get_running_loop() + signal_handler_installed = False + if os.name != 'nt': + try: + loop.add_signal_handler(signal.SIGTERM, task.cancel) + signal_handler_installed = True + except (NotImplementedError, RuntimeError): + pass + try: + response = await task + except asyncio.CancelledError: + return + finally: + if signal_handler_installed: + loop.remove_signal_handler(signal.SIGTERM) + evidence = response[-1] + if not isinstance(evidence, CompletedResult): + raise RuntimeError('theHarvester did not return terminal evidence') + write_child_evidence(artifact_dir, evidence, partial=False) + + +if __name__ == '__main__': + parser = ArgumentParser() + parser.add_argument('--execute', required=True) + parser.add_argument('--database', required=True, type=Path) + child_args = parser.parse_args() + asyncio.run(_child_execute(child_args.execute, child_args.database)) diff --git a/theHarvester/lib/api/runs.py b/theHarvester/lib/api/runs.py new file mode 100644 index 00000000..46139e67 --- /dev/null +++ b/theHarvester/lib/api/runs.py @@ -0,0 +1,275 @@ +from __future__ import annotations + +import os +import tempfile +from pathlib import Path +from typing import Annotated + +import anyio +from fastapi import APIRouter, Depends, HTTPException, Query, Request, Response, status +from fastapi.responses import FileResponse +from pydantic import ValidationError + +from theHarvester.lib.api.auth import get_api_key +from theHarvester.lib.source_catalog import ACTION_ACTIVITIES, SOURCE_SPECS, SourceSpec, get_source_spec, resolve_sources + +from . import run_worker +from .run_evidence import parse_jsonl_import +from .run_models import ( + DATABASE_IMPORT_REQUEST_OPENAPI, + EXPORT_RESPONSES, + IMPORT_REQUEST_OPENAPI, + RUN_REQUEST_OPENAPI, + ActionResponse, + DatabaseImportResponse, + RunDetail, + RunRequest, + RunSummary, + SourceCatalogResponse, + SourceResponse, +) +from .run_projection import source_spec +from .run_store import RunStore + +router = APIRouter(prefix='/api/v1', tags=['Runs']) +MAX_IMPORT_BYTES = 10 * 1024 * 1024 +MAX_RUN_REQUEST_BYTES = 64 * 1024 +DEFAULT_MAX_DATABASE_IMPORT_BYTES = 1024 * 1024 * 1024 + + +async def _read_limited_body(request: Request, limit: int, detail: str) -> bytes: + content_length = request.headers.get('content-length') + if content_length and content_length.isdigit() and int(content_length) > limit: + raise HTTPException(status_code=status.HTTP_413_CONTENT_TOO_LARGE, detail=detail) + body = bytearray() + async for chunk in request.stream(): + if len(body) + len(chunk) > limit: + raise HTTPException(status_code=status.HTTP_413_CONTENT_TOO_LARGE, detail=detail) + body.extend(chunk) + return bytes(body) + + +async def _stream_limited_body(request: Request, path: Path, limit: int, detail: str) -> None: + content_length = request.headers.get('content-length') + if content_length and content_length.isdigit() and int(content_length) > limit: + raise HTTPException(status_code=status.HTTP_413_CONTENT_TOO_LARGE, detail=detail) + size = 0 + async with await anyio.open_file(path, 'wb') as file: + async for chunk in request.stream(): + size += len(chunk) + if size > limit: + raise HTTPException(status_code=status.HTTP_413_CONTENT_TOO_LARGE, detail=detail) + await file.write(chunk) + + +@router.get('/runs') +async def list_runs( + _api_key: Annotated[str, Depends(get_api_key)], + limit: Annotated[int, Query(ge=1, le=500, description='Maximum run summaries to return.')] = 100, + offset: Annotated[int, Query(ge=0, description='Number of newer run summaries to skip.')] = 0, +) -> list[RunSummary]: + return [RunSummary.model_validate(run) for run in await RunStore().list_runs(limit=limit, offset=offset)] + + +@router.get('/sources') +async def list_sources(_api_key: Annotated[str, Depends(get_api_key)]) -> SourceCatalogResponse: + from theHarvester.lib.core import Core + + provider_aliases = {'chaos': 'projectDiscovery', 'github-code': 'github', 'pentesttools': 'pentestTools'} + api_key_fields = Core.api_key_fields() + provider_names = {provider.casefold(): provider for provider in api_key_fields} + + def credentials(source: SourceSpec) -> list[str]: + provider = provider_aliases.get(source.name, provider_names.get(source.name.casefold())) + if provider is None: + return [] + return [f'api-{field}' for field in api_key_fields.get(provider, ())] + + return SourceCatalogResponse( + sources=[ + SourceResponse( + name=source.name, + activity=source.activity, + credentials=credentials(source), + capabilities=sorted(source.capabilities), + ) + for source in sorted(SOURCE_SPECS.values(), key=lambda item: item.name) + ], + actions=[ActionResponse(name=name, activity=activity) for name, activity in sorted(ACTION_ACTIVITIES.items())], + ) + + +@router.post( + '/runs', + status_code=status.HTTP_201_CREATED, + response_model_exclude_unset=True, + openapi_extra=RUN_REQUEST_OPENAPI, +) +async def create_run( + request: Request, + _api_key: Annotated[str, Depends(get_api_key)], +) -> RunDetail: + body = await _read_limited_body(request, MAX_RUN_REQUEST_BYTES, 'Run request exceeds the 64 KiB limit') + try: + run_request = RunRequest.model_validate_json(body) + except ValidationError as error: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_CONTENT, + detail=error.errors(include_url=False, include_context=False), + ) from error + selected_sources = resolve_sources(run_request.sources) + unsupported_sources = [source for source in selected_sources if source_spec(source) is None] + if unsupported_sources: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_CONTENT, + detail=f'Unsupported sources: {", ".join(sorted(unsupported_sources))}', + ) + run_request.sources = [get_source_spec(source).name for source in selected_sources] + if not run_worker.worker_enabled(): + raise HTTPException( + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, + detail='theHarvester execution worker is disabled', + ) + if not run_worker.worker_available(): + raise HTTPException( + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, + detail='theHarvester execution worker is unavailable', + ) + run = await RunStore().create(run_request) + run_worker.wake_worker() + return RunDetail.model_validate(run) + + +@router.post( + '/runs/import', + status_code=status.HTTP_201_CREATED, + response_model_exclude_unset=True, + openapi_extra=IMPORT_REQUEST_OPENAPI, +) +async def import_run( + request: Request, + _api_key: Annotated[str, Depends(get_api_key)], + filename: Annotated[ + str, + Query( + min_length=1, + max_length=255, + description='Original .jsonl file name.', + ), + ], +) -> RunDetail: + safe_filename = Path(filename).name + if Path(safe_filename).suffix.casefold() != '.jsonl': + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail='Choose a .jsonl result file') + body = await _read_limited_body(request, MAX_IMPORT_BYTES, 'Result file exceeds the 10 MiB limit') + evidence = parse_jsonl_import(body) + return RunDetail.model_validate(await RunStore().import_evidence(evidence, safe_filename)) + + +@router.post( + '/runs/import-database', + status_code=status.HTTP_201_CREATED, + openapi_extra=DATABASE_IMPORT_REQUEST_OPENAPI, +) +async def import_database( + request: Request, + _api_key: Annotated[str, Depends(get_api_key)], + filename: Annotated[ + str, + Query( + min_length=1, + max_length=255, + description='Original .sqlite, .sqlite3, or .db file name.', + ), + ], +) -> DatabaseImportResponse: + safe_filename = Path(filename).name + if Path(safe_filename).suffix.casefold() not in {'.sqlite', '.sqlite3', '.db'}: + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail='Choose a SQLite database file') + try: + maximum_size = int(os.getenv('THEHARVESTER_MAX_DATABASE_IMPORT_BYTES', DEFAULT_MAX_DATABASE_IMPORT_BYTES)) + except ValueError as error: + raise HTTPException( + status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, + detail='THEHARVESTER_MAX_DATABASE_IMPORT_BYTES must be an integer', + ) from error + descriptor, temporary_name = tempfile.mkstemp(prefix='theharvester-import-', suffix='.sqlite') + os.close(descriptor) + temporary_path = Path(temporary_name) + await anyio.Path(temporary_path).chmod(0o600) + try: + await _stream_limited_body( + request, + temporary_path, + maximum_size, + 'SQLite database exceeds the configured import limit', + ) + async with await anyio.open_file(temporary_path, 'rb') as file: + header = await file.read(16) + if header != b'SQLite format 3\x00': + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail='Uploaded file is not a SQLite database') + return DatabaseImportResponse.model_validate(await RunStore().import_database(temporary_path, safe_filename)) + finally: + await anyio.Path(temporary_path).unlink(missing_ok=True) + + +@router.get('/runs/{run_id}', response_model_exclude_unset=True) +async def get_run(run_id: str, _api_key: Annotated[str, Depends(get_api_key)]) -> RunDetail: + run = await RunStore().get(run_id) + if run is None: + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='theHarvester run not found') + return RunDetail.model_validate(run) + + +@router.post('/runs/{run_id}/cancel', response_model_exclude_unset=True) +async def cancel_run( + run_id: str, + _api_key: Annotated[str, Depends(get_api_key)], +) -> RunDetail: + run = await RunStore().cancel(run_id) + if run is None: + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='theHarvester run not found') + return RunDetail.model_validate(run) + + +@router.get('/runs/{run_id}/export', response_class=Response, responses=EXPORT_RESPONSES) +async def export_run( + run_id: str, + _api_key: Annotated[str, Depends(get_api_key)], +) -> Response: + store = RunStore() + try: + completed = await store.load_completed_result(run_id) + except LookupError: + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='theHarvester run not found') + if completed is None: + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail='No run evidence is available to export', + ) + return Response( + completed.jsonl(), + media_type='application/x-ndjson', + headers={'Content-Disposition': f'attachment; filename="{completed.target}-{run_id}.jsonl"'}, + ) + + +@router.get('/runs/{run_id}/screenshots/{name}') +async def get_screenshot( + run_id: str, + name: str, + _api_key: Annotated[str, Depends(get_api_key)], +) -> FileResponse: + store = RunStore() + run = await store.get(run_id) + if run is None: + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='Screenshot not found') + screenshot = next((item for item in run['screenshots'] if item['name'] == name), None) + if screenshot is None: + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='Screenshot not found') + artifact_dir = store.artifact_directory(str(run['run_id'])) + screenshot_dir = artifact_dir / 'screenshots' + path = screenshot_dir / screenshot['name'] + if artifact_dir.is_symlink() or screenshot_dir.is_symlink() or path.is_symlink() or not path.is_file(): + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='Screenshot not found') + return FileResponse(path, media_type='image/png', filename=screenshot['name']) diff --git a/theHarvester/lib/completed_result.py b/theHarvester/lib/completed_result.py index 9badaf95..ca57a7f4 100644 --- a/theHarvester/lib/completed_result.py +++ b/theHarvester/lib/completed_result.py @@ -1,46 +1,62 @@ import json from collections import Counter from collections.abc import Iterable, Mapping -from dataclasses import dataclass -from datetime import UTC, datetime -from typing import Literal, Self, get_args +from dataclasses import dataclass, field +from datetime import datetime +from typing import Self from uuid import UUID, uuid4 -ResultKind = Literal[ - 'analytics', - 'api-endpoint', - 'asn', - 'breach', - 'cms', - 'dns-recursive-classification', - 'dns-recursive-finding', - 'dns-recursive-summary', - 'email', - 'framework', - 'hostname', - 'infostealer', - 'interesting-url', - 'ip-address', - 'language', - 'linkedin-link', - 'linkedin-person', - 'person', - 'server', - 'screenshot', - 'shodan', - 'takeover', - 'twitter-person', - 'url', - 'vhost', -] -ExecutionStatus = Literal['completed', 'partial', 'failed', 'rate-limited', 'skipped'] - -RESULT_KINDS: frozenset[str] = frozenset(get_args(ResultKind)) -EXECUTION_STATUSES: frozenset[str] = frozenset(get_args(ExecutionStatus)) +from theHarvester.lib.active_evidence import ActiveEvidence +from theHarvester.lib.evidence_types import ( + EVIDENCE_STATUSES, + EXECUTION_STATUSES, + RESULT_KINDS, + EvidenceStatus, + ExecutionStatus, + ResultKind, + format_utc, +) -def _isoformat_utc(value: datetime) -> str: - return value.astimezone(UTC).isoformat().replace('+00:00', 'Z') +def encode_result_jsonl( + summary: Mapping[str, object], + findings: Iterable[Mapping[str, object]], +) -> str: + records = [{**summary, 'type': 'summary'}, *findings] + return ''.join(json.dumps(record, ensure_ascii=False, separators=(',', ':'), sort_keys=True) + '\n' for record in records) + + +def parse_result_jsonl(payload: bytes | str) -> tuple[dict[str, object], list[dict[str, object]]]: + try: + text = payload.decode('utf-8') if isinstance(payload, bytes) else payload + records = [json.loads(line) for line in text.splitlines() if line.strip()] + except (UnicodeDecodeError, json.JSONDecodeError) as error: + raise ValueError('result file is not valid JSONL') from error + if any(not isinstance(record, dict) for record in records): + raise ValueError('JSONL records must be objects') + summary = records[0] if records else None + if not summary or summary.get('type') != 'summary': + raise ValueError('JSONL must start with a summary record') + if 'schema' in summary or 'schema_version' in summary: + raise ValueError('JSONL must not contain a schema version') + findings = records[1:] + for record in findings: + sources = record.get('sources', []) + actions = record.get('actions', []) + if ( + set(record) - {'type', 'value', 'sources', 'actions'} + or record.get('type') not in RESULT_KINDS + or not isinstance(record.get('value'), str) + or not record['value'].strip() + or not isinstance(sources, list) + or any(not isinstance(source, str) or not source.strip() for source in sources) + or not isinstance(actions, list) + or any(not isinstance(action, str) or not action.strip() for action in actions) + ): + raise ValueError('JSONL findings must contain a known type, non-empty value, and producer names') + record['sources'] = sorted(set(sources)) + record['actions'] = sorted(set(actions)) + return summary, findings @dataclass(frozen=True, order=True, slots=True) @@ -118,6 +134,8 @@ class CompletedResult: results: tuple[tuple[ResultKind, str], ...] source_executions: tuple[SourceExecution, ...] = () observations: tuple[ResultObservation, ...] = () + active_evidence: ActiveEvidence = field(default_factory=ActiveEvidence) + evidence_status: EvidenceStatus | None = None def __post_init__(self) -> None: if not self.target.strip(): @@ -148,6 +166,17 @@ class CompletedResult: observation_counts = Counter(observation.source for observation in self.observations) if any(execution.result_count != observation_counts[execution.source] for execution in self.source_executions): raise ValueError('source execution result count must match its attributed observations') + if any( + (observation.kind, observation.value) not in result_set for _action, observation in self.active_evidence.observations + ): + raise ValueError('every action observation must reference a completed result') + if any( + (artifact.subject_kind, artifact.subject_value) not in result_set + for _action, artifact in self.active_evidence.artifacts + ): + raise ValueError('every artifact must reference a completed result') + if self.evidence_status is not None and self.evidence_status not in EVIDENCE_STATUSES: + raise ValueError('evidence status must be complete, partial, or failed') @classmethod def finish( @@ -160,7 +189,10 @@ class CompletedResult: groups: Mapping[ResultKind, Iterable[str]], source_executions: Iterable[SourceExecution] = (), observations: Iterable[ResultObservation] = (), + active_evidence: ActiveEvidence | None = None, + evidence_status: EvidenceStatus | None = None, ) -> Self: + completed_active_evidence = active_evidence if active_evidence is not None else ActiveEvidence() results: set[tuple[ResultKind, str]] = set() for kind, values in groups.items(): if kind not in RESULT_KINDS: @@ -169,6 +201,8 @@ class CompletedResult: if not isinstance(value, str) or not value.strip(): raise ValueError('results must contain non-empty string values') results.add((kind, value.strip())) + for _action, observation in completed_active_evidence.observations: + results.add((observation.kind, observation.value)) return cls( run_id=run_id or uuid4(), target=target.strip(), @@ -177,45 +211,69 @@ class CompletedResult: results=tuple(sorted(results)), source_executions=tuple(source_executions), observations=tuple(sorted(set(observations))), + active_evidence=completed_active_evidence, + evidence_status=evidence_status, ) def evidence_dict(self) -> dict[str, object]: - incomplete = {'partial', 'failed', 'rate-limited', 'skipped'} - status = 'complete' - if self.source_executions and all(execution.status == 'failed' for execution in self.source_executions): - status = 'failed' - elif any(execution.status in incomplete for execution in self.source_executions): - status = 'partial' return { 'run_id': str(self.run_id), 'target': self.target, 'started_at': self.started_at.isoformat(), 'completed_at': self.completed_at.isoformat(), - 'status': status, - 'results': self._result_records(), + 'status': self.status, + 'results': self._result_records(include_actions=True), 'source_executions': [execution.to_dict() for execution in self.source_executions], + 'action_executions': [execution.to_dict() for execution in self.active_evidence.executions], + 'artifacts': [{'action': action, **artifact.to_dict()} for action, artifact in self.active_evidence.artifacts], } def jsonl(self) -> str: counts = Counter(kind for kind, _value in self.results) - records = [ + return encode_result_jsonl( { - 'completed_at': _isoformat_utc(self.completed_at), + 'completed_at': format_utc(self.completed_at), 'counts': dict(sorted(counts.items())), + 'evidence_status': self.status, 'result_count': len(self.results), 'run_id': str(self.run_id), - 'started_at': _isoformat_utc(self.started_at), + 'source_executions': [execution.to_dict() for execution in self.source_executions], + 'action_executions': [execution.to_dict() for execution in self.active_evidence.executions], + 'artifacts': [{'action': action, **artifact.to_dict()} for action, artifact in self.active_evidence.artifacts], + 'started_at': format_utc(self.started_at), 'target': self.target, - 'type': 'summary', }, - *self._result_records(), - ] - return ''.join(json.dumps(record, ensure_ascii=False, separators=(',', ':'), sort_keys=True) + '\n' for record in records) + self._result_records(include_actions=True), + ) - def _result_records(self) -> list[dict[str, object]]: + @property + def status(self) -> str: + incomplete = {'partial', 'failed', 'rate-limited', 'skipped'} + execution_statuses = [execution.status for execution in self.source_executions] + execution_statuses.extend(execution.status for execution in self.active_evidence.executions) + if execution_statuses and all(status == 'failed' for status in execution_statuses): + return 'failed' + if any(execution_status in incomplete for execution_status in execution_statuses): + return 'partial' + if execution_statuses: + return 'complete' + return self.evidence_status or 'complete' + + def _result_records(self, *, include_actions: bool) -> list[dict[str, object]]: sources_by_result: dict[tuple[ResultKind, str], list[str]] = {} for observation in self.observations: sources_by_result.setdefault((observation.kind, observation.value), []).append(observation.source) - return [ - {'type': kind, 'value': value, 'sources': sources_by_result.get((kind, value), [])} for kind, value in self.results - ] + actions_by_result: dict[tuple[ResultKind, str], list[str]] = {} + for action, action_observation in self.active_evidence.observations: + actions_by_result.setdefault((action_observation.kind, action_observation.value), []).append(action) + records: list[dict[str, object]] = [] + for kind, value in self.results: + record: dict[str, object] = { + 'type': kind, + 'value': value, + 'sources': sources_by_result.get((kind, value), []), + } + if include_actions and (actions := actions_by_result.get((kind, value))): + record['actions'] = actions + records.append(record) + return records diff --git a/theHarvester/lib/core.py b/theHarvester/lib/core.py index a0c25d44..d65a6e16 100644 --- a/theHarvester/lib/core.py +++ b/theHarvester/lib/core.py @@ -105,6 +105,10 @@ class Core: keys = yaml.safe_load(Core._read_config('api-keys.yaml')) return keys['apikeys'] + @staticmethod + def api_key_fields() -> dict[str, tuple[str, ...]]: + return dict(Core._API_KEY_FIELDS) + @staticmethod def _api_key_value(provider: str) -> Any: provider_keys = Core.api_keys()[provider] @@ -316,7 +320,6 @@ class Core: 'leakix', 'leaklookup', 'linkedin', - 'linkedin_links', 'mojeek', 'netcraft', 'netlas', @@ -710,6 +713,7 @@ class AsyncFetcher: session, url: str, proxy: str | None = None, + include_metadata: bool = False, ) -> tuple[Any, Any] | str: _, proxy_type = AsyncFetcher._resolve_proxy(proxy) response = await AsyncFetcher.fetch( @@ -717,6 +721,7 @@ class AsyncFetcher: url=url, proxy=proxy, request_timeout=15, + include_metadata=include_metadata, ) return url, response @@ -745,13 +750,22 @@ class AsyncFetcher: return list( await asyncio.gather( *[ - AsyncFetcher.takeover_fetch(session, url, proxy=proxy_url) + AsyncFetcher.takeover_fetch( + session, + url, + proxy=proxy_url, + include_metadata=include_metadata, + ) for url, proxy_url in zip(urls, proxy_urls, strict=False) ] ) ) else: - return list(await asyncio.gather(*[AsyncFetcher.takeover_fetch(session, url) for url in urls])) + return list( + await asyncio.gather( + *[AsyncFetcher.takeover_fetch(session, url, include_metadata=include_metadata) for url in urls] + ) + ) if len(params) == 0: async with aiohttp.ClientSession(headers=headers, timeout=timeout) as session: diff --git a/theHarvester/lib/database.py b/theHarvester/lib/database.py index fcb23725..2f208957 100644 --- a/theHarvester/lib/database.py +++ b/theHarvester/lib/database.py @@ -5,17 +5,39 @@ import sqlite3 from collections import Counter from collections.abc import AsyncIterator, Iterable from contextlib import asynccontextmanager -from datetime import date +from datetime import date, timedelta from pathlib import Path -from typing import Any, cast +from typing import TYPE_CHECKING, Any, cast from uuid import UUID -from sqlalchemy import Date, Float, ForeignKey, ForeignKeyConstraint, Text, UniqueConstraint, event, func, select +from sqlalchemy import ( + CheckConstraint, + Date, + Float, + ForeignKey, + ForeignKeyConstraint, + Text, + UniqueConstraint, + delete, + event, + func, + or_, + select, + update, +) +from sqlalchemy.dialects.sqlite import insert as sqlite_insert from sqlalchemy.engine import URL from sqlalchemy.exc import IntegrityError, SQLAlchemyError -from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker, create_async_engine +from sqlalchemy.ext.asyncio import AsyncConnection, AsyncEngine, AsyncSession, async_sessionmaker, create_async_engine from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column +from theHarvester.lib.active_evidence import ( + ActionExecution, + ActionObservation, + ActionYield, + ActiveEvidence, + ArtifactReference, +) from theHarvester.lib.completed_result import ( CompletedResult, ExecutionStatus, @@ -25,11 +47,24 @@ from theHarvester.lib.completed_result import ( SourceYield, ) +if TYPE_CHECKING: + from theHarvester.lib.evidence_types import EvidenceStatus + logger = logging.getLogger(__name__) -SCHEMA_VERSION = 2 +SCHEMA_VERSION = 7 _DEFAULT_DATABASE = Path('~/.local/share/theHarvester/stash.sqlite').expanduser() +_LEGACY_RESULT_KIND_RENAMES = { + 'api-endpoint': 'url', + 'api_endpoint': 'url', + 'interesting-url': 'url', + 'interestingurls': 'url', + 'ip-address': 'ip', + 'linkedin-link': 'url', + 'linkedinlinks': 'url', +} + class ResultStoreError(RuntimeError): """The result store could not complete an operation.""" @@ -73,6 +108,7 @@ class _RunRow(_Base): target: Mapped[str] = mapped_column(Text) started_at: Mapped[str] = mapped_column(Text) completed_at: Mapped[str] = mapped_column(Text) + evidence_status: Mapped[str | None] = mapped_column(Text) class _ResultRow(_Base): @@ -134,6 +170,67 @@ class _ResultOriginRow(_Base): execution_position: Mapped[int] = mapped_column(primary_key=True) +class _ArtifactRow(_Base): + """Metadata for a file created by an action and attached to one result.""" + + __tablename__ = 'artifacts' + __table_args__ = ( + ForeignKeyConstraint( + ('run_id', 'result_position'), + ('results.run_id', 'results.position'), + ondelete='CASCADE', + ), + ForeignKeyConstraint( + ('run_id', 'execution_position'), + ('executions.run_id', 'executions.position'), + ondelete='CASCADE', + ), + CheckConstraint('size_bytes >= 0'), + CheckConstraint("length(sha256) = 64 AND sha256 NOT GLOB '*[^0-9a-f]*'"), + ) + + run_id: Mapped[str] = mapped_column(Text, primary_key=True) + position: Mapped[int] = mapped_column(primary_key=True) + result_position: Mapped[int] + execution_position: Mapped[int] + kind: Mapped[str] = mapped_column(Text) + path: Mapped[str] = mapped_column(Text) + media_type: Mapped[str] = mapped_column(Text) + size_bytes: Mapped[int] + sha256: Mapped[str] = mapped_column(Text) + created_at: Mapped[str] = mapped_column(Text) + + +class _RunRecordRow(_Base): + """Lifecycle state for one API-submitted or imported run.""" + + __tablename__ = 'run_records' + + run_id: Mapped[str] = mapped_column(Text, primary_key=True) + target: Mapped[str] = mapped_column(Text) + status: Mapped[str] = mapped_column(Text) + origin: Mapped[str] = mapped_column(Text) + created_at: Mapped[str] = mapped_column(Text) + started_at: Mapped[str | None] = mapped_column(Text) + completed_at: Mapped[str | None] = mapped_column(Text) + request_json: Mapped[str] = mapped_column(Text) + evidence_run_id: Mapped[str | None] = mapped_column(Text, ForeignKey('runs.run_id', ondelete='SET NULL')) + evidence_status: Mapped[str | None] = mapped_column(Text) + cancellation_requested_at: Mapped[str | None] = mapped_column(Text) + error: Mapped[str | None] = mapped_column(Text) + log: Mapped[str] = mapped_column(Text, default='') + + +class _WorkerLeaseRow(_Base): + """The current owner of the single local API execution worker.""" + + __tablename__ = 'run_worker_leases' + + name: Mapped[str] = mapped_column(Text, primary_key=True) + owner_id: Mapped[str] = mapped_column(Text) + heartbeat_at: Mapped[str] = mapped_column(Text) + + def _configure_sqlite_connection(dbapi_connection: Any, _connection_record: Any) -> None: dbapi_connection.isolation_level = None cursor = dbapi_connection.cursor() @@ -155,6 +252,102 @@ def _sqlite_engine(database: str | Path) -> AsyncEngine: return engine +async def _canonicalize_result_kinds(connection: AsyncConnection) -> None: + """Merge result-kind aliases without losing provenance or artifact references.""" + aliases = ', '.join(f"'{kind}'" for kind in sorted(_LEGACY_RESULT_KIND_RENAMES)) + run_rows = await connection.exec_driver_sql(f'SELECT DISTINCT run_id FROM results WHERE kind IN ({aliases})') + for (run_id,) in run_rows: + result_rows = list( + await connection.exec_driver_sql( + 'SELECT position, kind, value FROM results WHERE run_id = ? ORDER BY position', + (run_id,), + ) + ) + origin_rows = list( + await connection.exec_driver_sql( + 'SELECT result_position, execution_position FROM result_origins WHERE run_id = ?', + (run_id,), + ) + ) + artifact_rows = list( + await connection.exec_driver_sql( + 'SELECT position, result_position, execution_position, kind, path, media_type, ' + 'size_bytes, sha256, created_at FROM artifacts WHERE run_id = ? ORDER BY position', + (run_id,), + ) + ) + + canonical_results = sorted( + {(_LEGACY_RESULT_KIND_RENAMES.get(kind, kind), value) for _position, kind, value in result_rows} + ) + new_positions = {result: position for position, result in enumerate(canonical_results)} + old_positions = { + position: new_positions[(_LEGACY_RESULT_KIND_RENAMES.get(kind, kind), value)] for position, kind, value in result_rows + } + canonical_origins = sorted( + {(old_positions[result_position], execution_position) for result_position, execution_position in origin_rows} + ) + + await connection.exec_driver_sql('DELETE FROM artifacts WHERE run_id = ?', (run_id,)) + await connection.exec_driver_sql('DELETE FROM result_origins WHERE run_id = ?', (run_id,)) + await connection.exec_driver_sql('DELETE FROM results WHERE run_id = ?', (run_id,)) + if canonical_results: + await connection.exec_driver_sql( + 'INSERT INTO results (run_id, position, kind, value) VALUES (?, ?, ?, ?)', + [(run_id, position, kind, value) for position, (kind, value) in enumerate(canonical_results)], + ) + if canonical_origins: + await connection.exec_driver_sql( + 'INSERT INTO result_origins (run_id, result_position, execution_position) VALUES (?, ?, ?)', + [(run_id, result_position, execution_position) for result_position, execution_position in canonical_origins], + ) + if artifact_rows: + await connection.exec_driver_sql( + 'INSERT INTO artifacts ' + '(run_id, position, result_position, execution_position, kind, path, media_type, size_bytes, sha256, created_at) ' + 'VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', + [ + ( + run_id, + position, + old_positions[result_position], + execution_position, + kind, + path, + media_type, + size_bytes, + sha256, + created_at, + ) + for ( + position, + result_position, + execution_position, + kind, + path, + media_type, + size_bytes, + sha256, + created_at, + ) in artifact_rows + ], + ) + await connection.exec_driver_sql( + 'UPDATE executions SET result_count = (' + 'SELECT COUNT(*) FROM result_origins ' + 'WHERE result_origins.run_id = executions.run_id ' + 'AND result_origins.execution_position = executions.position' + ') WHERE run_id = ?', + (run_id,), + ) + + for alias, canonical in _LEGACY_RESULT_KIND_RENAMES.items(): + await connection.exec_driver_sql( + 'UPDATE legacy_observations SET kind = ? WHERE kind = ?', + (canonical, alias), + ) + + class _SQLiteDatabase: def __init__(self, database: str | Path) -> None: self.database = str(Path(database).expanduser().resolve()) @@ -205,20 +398,21 @@ class _SQLiteDatabase: if 'discovery_observations' in tables and 'legacy_observations' not in tables: await connection.exec_driver_sql('ALTER TABLE discovery_observations RENAME TO legacy_observations') await connection.run_sync(_Base.metadata.create_all) + run_column_rows = await connection.exec_driver_sql('PRAGMA table_info(runs)') + if 'evidence_status' not in {row[1] for row in run_column_rows}: + await connection.exec_driver_sql('ALTER TABLE runs ADD COLUMN evidence_status TEXT') if has_legacy_results: await connection.exec_driver_sql( 'INSERT INTO legacy_observations (domain, resource, kind, discovered_on, source) ' 'SELECT domain, resource, CASE type ' "WHEN 'host' THEN 'hostname' " - "WHEN 'ip' THEN 'ip-address' " "WHEN 'people' THEN 'person' " - "WHEN 'linkedinlinks' THEN 'linkedin-link' " - "WHEN 'interestingurls' THEN 'interesting-url' " "WHEN 'asns' THEN 'asn' " - "WHEN 'api_endpoint' THEN 'api-endpoint' " 'ELSE type END, find_date, source FROM legacy_results' ) await connection.exec_driver_sql('DROP TABLE legacy_results') + if schema_version < SCHEMA_VERSION: + await _canonicalize_result_kinds(connection) await connection.exec_driver_sql(f'PRAGMA user_version = {SCHEMA_VERSION}') await connection.commit() except BaseException: @@ -251,6 +445,240 @@ def _database_for(database: str | Path) -> _SQLiteDatabase: return _databases[path] +def _row_count(result: Any) -> int: + return int(result.rowcount) + + +class RunLifecycleStore: + """Persist API run state in the same SQLite database as terminal evidence.""" + + def __init__(self, database: str | Path | None = None) -> None: + self.database = str(Path(database or _DEFAULT_DATABASE).expanduser().resolve()) + + async def initialize(self) -> None: + Path(self.database).parent.mkdir(parents=True, exist_ok=True) + await _database_for(self.database).initialize() + + async def create( + self, + *, + run_id: str, + target: str, + status: str, + origin: str, + created_at: str, + request_json: str, + started_at: str | None = None, + completed_at: str | None = None, + evidence_run_id: str | None = None, + evidence_status: str | None = None, + ) -> None: + async with self._session() as session: + session.add( + _RunRecordRow( + run_id=run_id, + target=target, + status=status, + origin=origin, + created_at=created_at, + started_at=started_at, + completed_at=completed_at, + request_json=request_json, + evidence_run_id=evidence_run_id, + evidence_status=evidence_status, + cancellation_requested_at=None, + error=None, + log='', + ) + ) + await session.commit() + + async def list_records(self, *, limit: int = 100, offset: int = 0) -> list[dict[str, object]]: + async with self._session() as session: + result_count = ( + select(func.count(_ResultRow.position)) + .where(_ResultRow.run_id == _RunRecordRow.evidence_run_id) + .correlate(_RunRecordRow) + .scalar_subquery() + ) + rows = ( + await session.execute( + select(_RunRecordRow, result_count.label('result_count')) + .order_by(_RunRecordRow.created_at.desc(), _RunRecordRow.run_id.desc()) + .limit(limit) + .offset(offset) + ) + ).all() + return [self._record(row, result_count=count) for row, count in rows] + + async def get(self, run_id: str) -> dict[str, object] | None: + async with self._session() as session: + row = await session.get(_RunRecordRow, run_id) + return self._record(row) if row is not None else None + + async def cancel(self, run_id: str, requested_at: str) -> dict[str, object] | None: + async with self._session() as session: + queued = await session.execute( + update(_RunRecordRow) + .where(_RunRecordRow.run_id == run_id, _RunRecordRow.status == 'queued') + .values(status='cancelled', cancellation_requested_at=requested_at, completed_at=requested_at) + ) + running = None + if _row_count(queued) != 1: + running = await session.execute( + update(_RunRecordRow) + .where(_RunRecordRow.run_id == run_id, _RunRecordRow.status == 'running') + .values(status='cancelling', cancellation_requested_at=requested_at) + ) + await session.commit() + row = await self.get(run_id) + if row is None: + return None + if ( + _row_count(queued) == 1 + or (running is not None and _row_count(running) == 1) + or row['status'] in {'cancelling', 'cancelled'} + ): + return row + raise ValueError(row['status']) + + async def claim_next(self, started_at: str) -> dict[str, object] | None: + async with self._session() as session: + candidate = ( + select(_RunRecordRow.run_id) + .where(_RunRecordRow.status == 'queued') + .order_by(_RunRecordRow.created_at) + .limit(1) + .scalar_subquery() + ) + result = await session.execute( + update(_RunRecordRow) + .where(_RunRecordRow.run_id == candidate, _RunRecordRow.status == 'queued') + .values(status='running', started_at=started_at) + .returning(_RunRecordRow.run_id) + ) + run_id = result.scalar_one_or_none() + if run_id is None: + await session.rollback() + return None + await session.commit() + return await self.get(run_id) + + async def finish( + self, + run_id: str, + *, + completed_at: str, + evidence_run_id: str | None, + evidence_status: str | None, + log: str, + ) -> None: + async with self._session() as session: + await session.execute( + update(_RunRecordRow) + .where(_RunRecordRow.run_id == run_id, _RunRecordRow.status.in_({'running', 'cancelling'})) + .values( + status=func.iif(_RunRecordRow.status == 'cancelling', 'cancelled', 'completed'), + completed_at=completed_at, + evidence_run_id=func.coalesce(evidence_run_id, _RunRecordRow.evidence_run_id), + evidence_status=func.coalesce(evidence_status, _RunRecordRow.evidence_status), + log=log, + ) + ) + await session.commit() + + async def fail( + self, + run_id: str, + *, + status: str, + completed_at: str, + error: str, + log: str, + evidence_run_id: str | None, + evidence_status: str | None, + ) -> None: + async with self._session() as session: + await session.execute( + update(_RunRecordRow) + .where(_RunRecordRow.run_id == run_id) + .values( + status=status, + completed_at=completed_at, + error=error, + log=log, + evidence_run_id=func.coalesce(evidence_run_id, _RunRecordRow.evidence_run_id), + evidence_status=func.coalesce(evidence_status, _RunRecordRow.evidence_status), + ) + ) + await session.commit() + + async def running(self) -> list[dict[str, object]]: + async with self._session() as session: + rows = (await session.scalars(select(_RunRecordRow).where(_RunRecordRow.status.in_({'running', 'cancelling'})))).all() + return [self._record(row) for row in rows] + + async def acquire_lease(self, owner_id: str, now: str, timeout_seconds: int) -> bool: + async with self._session() as session: + stale_before = (datetime.datetime.fromisoformat(now) - timedelta(seconds=timeout_seconds)).isoformat() + statement = sqlite_insert(_WorkerLeaseRow).values(name='executor', owner_id=owner_id, heartbeat_at=now) + statement = statement.on_conflict_do_update( + index_elements=[_WorkerLeaseRow.name], + set_={'owner_id': owner_id, 'heartbeat_at': now}, + where=or_(_WorkerLeaseRow.owner_id == owner_id, _WorkerLeaseRow.heartbeat_at < stale_before), + ) + result = await session.execute(statement) + await session.commit() + return _row_count(result) == 1 + + async def heartbeat_lease(self, owner_id: str, now: str) -> bool: + async with self._session() as session: + result = await session.execute( + update(_WorkerLeaseRow) + .where(_WorkerLeaseRow.name == 'executor', _WorkerLeaseRow.owner_id == owner_id) + .values(heartbeat_at=now) + ) + await session.commit() + return _row_count(result) == 1 + + async def release_lease(self, owner_id: str) -> None: + async with self._session() as session: + await session.execute( + delete(_WorkerLeaseRow).where( + _WorkerLeaseRow.name == 'executor', + _WorkerLeaseRow.owner_id == owner_id, + ) + ) + await session.commit() + + @asynccontextmanager + async def _session(self) -> AsyncIterator[AsyncSession]: + await self.initialize() + async with _database_for(self.database).session() as session: + yield session + + @staticmethod + def _record(row: _RunRecordRow, *, result_count: int | None = None) -> dict[str, object]: + record: dict[str, object] = { + 'run_id': row.run_id, + 'target': row.target, + 'status': row.status, + 'origin': row.origin, + 'created_at': row.created_at, + 'started_at': row.started_at, + 'completed_at': row.completed_at, + 'request_json': row.request_json, + 'evidence_run_id': row.evidence_run_id, + 'evidence_status': row.evidence_status, + 'cancellation_requested_at': row.cancellation_requested_at, + 'error': row.error, + 'log': row.log, + } + if result_count is not None: + record['result_count'] = int(result_count) + return record + + class ResultStore: """Persist enumeration results without exposing SQLAlchemy to callers.""" @@ -274,6 +702,7 @@ class ResultStore: target=result.target, started_at=result.started_at.isoformat(), completed_at=result.completed_at.isoformat(), + evidence_status=result.evidence_status, ) ) await session.flush() @@ -281,30 +710,58 @@ class ResultStore: _ResultRow(run_id=run_id, position=position, kind=kind, value=value) for position, (kind, value) in enumerate(result.results) ) + producers: list[tuple[str, str, SourceExecution | ActionExecution]] = [ + ('source', execution.source, execution) for execution in result.source_executions + ] + producers.extend(('action', execution.action, execution) for execution in result.active_evidence.executions) session.add_all( _ExecutionRow( run_id=run_id, position=position, - producer_kind='source', - name=execution.source, + producer_kind=producer_kind, + name=name, status=execution.status, duration_ms=execution.duration_ms, result_count=execution.result_count, error_type=execution.error_type, stop_reason=execution.stop_reason, ) - for position, execution in enumerate(result.source_executions) + for position, (producer_kind, name, execution) in enumerate(producers) ) await session.flush() result_positions = {item: position for position, item in enumerate(result.results)} - execution_positions = {execution.source: position for position, execution in enumerate(result.source_executions)} + execution_positions = { + (producer_kind, name): position for position, (producer_kind, name, _execution) in enumerate(producers) + } + origins: list[tuple[str, str, ResultKind, str]] = [ + ('source', observation.source, observation.kind, observation.value) for observation in result.observations + ] + origins.extend( + ('action', action, observation.kind, observation.value) + for action, observation in result.active_evidence.observations + ) session.add_all( _ResultOriginRow( run_id=run_id, - result_position=result_positions[(observation.kind, observation.value)], - execution_position=execution_positions[observation.source], + result_position=result_positions[(kind, value)], + execution_position=execution_positions[(producer_kind, name)], ) - for observation in result.observations + for producer_kind, name, kind, value in origins + ) + session.add_all( + _ArtifactRow( + run_id=run_id, + position=position, + result_position=result_positions[(artifact.subject_kind, artifact.subject_value)], + execution_position=execution_positions[('action', action)], + kind=artifact.kind, + path=artifact.path, + media_type=artifact.media_type, + size_bytes=artifact.size_bytes, + sha256=artifact.sha256, + created_at=artifact.created_at.isoformat(), + ) + for position, (action, artifact) in enumerate(result.active_evidence.artifacts) ) await session.commit() except IntegrityError as error: @@ -326,14 +783,76 @@ class ResultStore: ).all() execution_rows = ( await session.scalars( - select(_ExecutionRow) - .where(_ExecutionRow.run_id == str(run_id), _ExecutionRow.producer_kind == 'source') - .order_by(_ExecutionRow.position) + select(_ExecutionRow).where(_ExecutionRow.run_id == str(run_id)).order_by(_ExecutionRow.position) ) ).all() origin_rows = (await session.scalars(select(_ResultOriginRow).where(_ResultOriginRow.run_id == str(run_id)))).all() + artifact_rows = ( + await session.scalars( + select(_ArtifactRow).where(_ArtifactRow.run_id == str(run_id)).order_by(_ArtifactRow.position) + ) + ).all() results_by_position = {row.position: row for row in rows} executions_by_position = {row.position: row for row in execution_rows} + unknown_producer_kinds = {row.producer_kind for row in execution_rows} - {'source', 'action'} + if unknown_producer_kinds: + raise ResultStoreError(f'Unknown persisted producer kind: {sorted(unknown_producer_kinds)[0]}') + observations_by_execution: dict[int, list[ActionObservation]] = {} + source_observations: list[ResultObservation] = [] + for origin in origin_rows: + execution = executions_by_position[origin.execution_position] + stored_result = results_by_position[origin.result_position] + if execution.producer_kind == 'source': + source_observations.append( + ResultObservation( + source=execution.name, + kind=cast('ResultKind', stored_result.kind), + value=stored_result.value, + ) + ) + else: + observations_by_execution.setdefault(execution.position, []).append( + ActionObservation( + kind=cast('ResultKind', stored_result.kind), + value=stored_result.value, + ) + ) + artifacts_by_execution: dict[int, list[ArtifactReference]] = {} + for artifact in artifact_rows: + execution = executions_by_position[artifact.execution_position] + if execution.producer_kind != 'action': + raise ResultStoreError('Persisted artifact must reference an action execution') + subject = results_by_position[artifact.result_position] + artifacts_by_execution.setdefault(execution.position, []).append( + ArtifactReference( + kind=artifact.kind, + subject_kind=cast('ResultKind', subject.kind), + subject_value=subject.value, + path=artifact.path, + media_type=artifact.media_type, + size_bytes=artifact.size_bytes, + sha256=artifact.sha256, + created_at=datetime.datetime.fromisoformat(artifact.created_at), + ) + ) + action_executions: list[ActionExecution] = [] + for row in execution_rows: + if row.producer_kind != 'action': + continue + action_observations = tuple(sorted(observations_by_execution.get(row.position, []))) + if row.result_count != len(action_observations): + raise ResultStoreError(f'Persisted action result count does not match origins: {row.name}') + action_executions.append( + ActionExecution( + action=row.name, + status=cast('ExecutionStatus', row.status), + duration_ms=row.duration_ms, + observations=action_observations, + artifacts=tuple(sorted(artifacts_by_execution.get(row.position, []))), + error_type=row.error_type, + stop_reason=row.stop_reason, + ) + ) return CompletedResult( run_id=UUID(parent.run_id), target=parent.target, @@ -350,30 +869,25 @@ class ResultStore: stop_reason=row.stop_reason, ) for row in execution_rows + if row.producer_kind == 'source' ), - observations=tuple( - sorted( - ResultObservation( - source=executions_by_position[row.execution_position].name, - kind=cast('ResultKind', results_by_position[row.result_position].kind), - value=results_by_position[row.result_position].value, - ) - for row in origin_rows - ) - ), + observations=tuple(sorted(source_observations)), + active_evidence=ActiveEvidence(executions=tuple(action_executions)), + evidence_status=cast('EvidenceStatus', parent.evidence_status) if parent.evidence_status is not None else None, ) - async def list_runs(self, *, limit: int = 50) -> list[dict[str, object]]: + async def list_runs(self, *, limit: int | None = 50, offset: int = 0) -> list[dict[str, object]]: async with self._session() as session: - rows = ( - await session.execute( - select(_RunRow, func.count(_ResultRow.position)) - .outerjoin(_ResultRow, _ResultRow.run_id == _RunRow.run_id) - .group_by(_RunRow.run_id) - .order_by(func.julianday(_RunRow.completed_at).desc(), _RunRow.run_id.desc()) - .limit(limit) - ) - ).all() + statement = ( + select(_RunRow, func.count(_ResultRow.position)) + .outerjoin(_ResultRow, _ResultRow.run_id == _RunRow.run_id) + .group_by(_RunRow.run_id) + .order_by(func.julianday(_RunRow.completed_at).desc(), _RunRow.run_id.desc()) + ) + if limit is not None: + statement = statement.limit(limit) + statement = statement.offset(offset) + rows = (await session.execute(statement)).all() return [ { 'run_id': run.run_id, @@ -385,6 +899,32 @@ class ResultStore: for run, result_count in rows ] + async def validate_import_database(self) -> None: + engine = create_async_engine(URL.create('sqlite+aiosqlite', database=self.database)) + try: + async with engine.connect() as connection: + quick_check = await connection.exec_driver_sql('PRAGMA quick_check') + if quick_check.scalar_one() != 'ok': + raise ResultStoreError('SQLite integrity check failed') + version = (await connection.exec_driver_sql('PRAGMA user_version')).scalar_one() + if version > SCHEMA_VERSION: + raise ResultStoreError(f'Database schema version {version} is newer than supported version {SCHEMA_VERSION}') + table_rows = await connection.exec_driver_sql("SELECT name FROM sqlite_master WHERE type = 'table'") + tables = {str(row[0]) for row in table_rows} + current_schema = {'runs', 'results'}.issubset(tables) + released_schema = {'completed_results', 'completed_result_items'}.issubset(tables) + if not current_schema and not released_schema: + raise ResultStoreError('SQLite database does not contain theHarvester completed runs') + except SQLAlchemyError as error: + raise ResultStoreError('Could not validate SQLite database') from error + finally: + await engine.dispose() + + async def dispose(self) -> None: + database = _databases.pop(self.database, None) + if database is not None: + await database.dispose() + async def record_observations( self, target: str, @@ -414,40 +954,59 @@ class ResultStore: A result is unique when one source reported it and shared when more than one source reported it. Sources that ran without results still appear with zero counts. """ + yields = await self._producer_yields(run_id, 'source') + return [ + SourceYield( + source=name, + observed_result_count=observed, + unique_result_count=unique, + shared_result_count=shared, + ) + for name, observed, unique, shared in yields + ] + + async def action_yields(self, run_id: UUID) -> list[ActionYield]: + yields = await self._producer_yields(run_id, 'action') + return [ + ActionYield( + action=name, + observed_result_count=observed, + unique_result_count=unique, + shared_result_count=shared, + ) + for name, observed, unique, shared in yields + ] + + async def _producer_yields(self, run_id: UUID, producer_kind: str) -> list[tuple[str, int, int, int]]: async with self._session() as session: execution_rows = ( await session.scalars( select(_ExecutionRow).where( _ExecutionRow.run_id == str(run_id), - _ExecutionRow.producer_kind == 'source', + _ExecutionRow.producer_kind == producer_kind, ) ) ).all() result_rows = (await session.scalars(select(_ResultRow).where(_ResultRow.run_id == str(run_id)))).all() origin_rows = (await session.scalars(select(_ResultOriginRow).where(_ResultOriginRow.run_id == str(run_id)))).all() - source_by_position = {row.position: row.name for row in execution_rows} + producer_by_position = {row.position: row.name for row in execution_rows} result_by_position = {row.position: (row.kind, row.value) for row in result_rows} - sources_by_result: dict[tuple[str, str], set[str]] = {} + producers_by_result: dict[tuple[str, str], set[str]] = {} for origin in origin_rows: - source = source_by_position.get(origin.execution_position) + producer = producer_by_position.get(origin.execution_position) result = result_by_position.get(origin.result_position) - if source is not None and result is not None: - sources_by_result.setdefault(result, set()).add(source) + if producer is not None and result is not None: + producers_by_result.setdefault(result, set()).add(producer) observed_counts: Counter[str] = Counter() unique_counts: Counter[str] = Counter() shared_counts: Counter[str] = Counter() - for sources in sources_by_result.values(): - for source in sources: - observed_counts[source] += 1 - (unique_counts if len(sources) == 1 else shared_counts)[source] += 1 + for producers in producers_by_result.values(): + for producer in producers: + observed_counts[producer] += 1 + (unique_counts if len(producers) == 1 else shared_counts)[producer] += 1 return [ - SourceYield( - source=source, - observed_result_count=observed_counts[source], - unique_result_count=unique_counts[source], - shared_result_count=shared_counts[source], - ) - for source in sorted(source_by_position.values()) + (name, observed_counts[name], unique_counts[name], shared_counts[name]) + for name in sorted(producer_by_position.values()) ] @asynccontextmanager diff --git a/theHarvester/lib/enumeration.py b/theHarvester/lib/enumeration.py index c6d5aca5..6cb6d697 100644 --- a/theHarvester/lib/enumeration.py +++ b/theHarvester/lib/enumeration.py @@ -25,6 +25,8 @@ class EnumerationOptions: dns_server: str | None = None take_over: bool = False dns_resolve: str | None = '' + dns_resolvers: tuple[str, ...] = () + dns_resolver_input: str = '' dns_lookup: bool = False dns_brute: bool = False dns_recursive_depth: int = 0 @@ -49,6 +51,8 @@ class EnumerationOptions: dns_server=getattr(value, 'dns_server', None), take_over=getattr(value, 'take_over', False), dns_resolve=getattr(value, 'dns_resolve', ''), + dns_resolvers=tuple(getattr(value, 'dns_resolvers', ())), + dns_resolver_input=getattr(value, 'dns_resolver_input', ''), dns_lookup=getattr(value, 'dns_lookup', False), dns_brute=getattr(value, 'dns_brute', False), dns_recursive_depth=getattr(value, 'dns_recursive_depth', 0), diff --git a/theHarvester/lib/evidence_types.py b/theHarvester/lib/evidence_types.py new file mode 100644 index 00000000..b2685d5a --- /dev/null +++ b/theHarvester/lib/evidence_types.py @@ -0,0 +1,37 @@ +from datetime import UTC, datetime +from typing import Literal, get_args + +ResultKind = Literal[ + 'analytics', + 'asn', + 'breach', + 'cms', + 'dns-recursive-classification', + 'dns-recursive-finding', + 'dns-recursive-summary', + 'email', + 'framework', + 'hostname', + 'infostealer', + 'ip', + 'language', + 'linkedin-person', + 'person', + 'server', + 'screenshot', + 'shodan', + 'takeover', + 'twitter-person', + 'url', + 'vhost', +] +ExecutionStatus = Literal['completed', 'partial', 'failed', 'rate-limited', 'skipped'] +EvidenceStatus = Literal['complete', 'partial', 'failed'] + +RESULT_KINDS: frozenset[str] = frozenset(get_args(ResultKind)) +EXECUTION_STATUSES: frozenset[str] = frozenset(get_args(ExecutionStatus)) +EVIDENCE_STATUSES: frozenset[str] = frozenset(get_args(EvidenceStatus)) + + +def format_utc(value: datetime) -> str: + return value.astimezone(UTC).isoformat().replace('+00:00', 'Z') diff --git a/theHarvester/lib/hostchecker.py b/theHarvester/lib/hostchecker.py index 72b5f49c..d2d7a681 100644 --- a/theHarvester/lib/hostchecker.py +++ b/theHarvester/lib/hostchecker.py @@ -28,6 +28,14 @@ class HostDnsRecords: return self.ipv4 + self.ipv6 +def is_expected_dns_absence(error: BaseException) -> bool: + return ( + isinstance(error, aiodns.error.DNSError) + and bool(error.args) + and error.args[0] in {aiodns.error.ARES_ENODATA, aiodns.error.ARES_ENOTFOUND} + ) + + class Checker: """Resolve hosts while preserving the legacy ``check()`` return tuple. @@ -41,6 +49,8 @@ class Checker: self.addresses: set[str] = set() self.records: dict[str, HostDnsRecords] = {} self.nameservers: list[str] = nameservers + self.query_error_count = 0 + self.query_error_types: set[str] = set() # @staticmethod # async def query(host, resolver) -> Tuple[str, Any]: @@ -54,8 +64,7 @@ class Checker: # except Exception: # return f"{host}", tuple() - @staticmethod - async def resolve_host(host: str, resolver: aiodns.DNSResolver) -> tuple[str, HostDnsRecords] | None: + async def resolve_host(self, host: str, resolver: aiodns.DNSResolver) -> tuple[str, HostDnsRecords] | None: record_types = ('A', 'AAAA', 'CNAME') results = await asyncio.gather( *(resolver.query_dns(host, record_type) for record_type in record_types), @@ -65,6 +74,9 @@ class Checker: for record_type, result in zip(record_types, results, strict=True): if isinstance(result, BaseException): if isinstance(result, Exception): + if not is_expected_dns_absence(result): + self.query_error_count += 1 + self.query_error_types.add(type(result).__name__) continue raise result for record in result.answer: diff --git a/theHarvester/lib/ip-ranges.json b/theHarvester/lib/ip-ranges.json deleted file mode 100644 index 678c8e6b..00000000 --- a/theHarvester/lib/ip-ranges.json +++ /dev/null @@ -1,8978 +0,0 @@ -{ - "syncToken": "1546032855", - "createDate": "2018-12-28-21-34-15", - "prefixes": [ - { - "ip_prefix": "18.208.0.0/13", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.245.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.194.0.0/15", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.155.0.0/16", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.196.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.22.0/24", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.255.112/28", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "13.210.0.0/15", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.17.0/24", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.154.0/23", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.212.0/22", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.240/28", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.241.0.0/16", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "184.169.128.0/17", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "216.182.224.0/21", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.74.0.0/16", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.168.0.0/16", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.54.0/23", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.224.0/21", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.64.0/22", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.238.0.0/16", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "216.182.232.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.72.0/22", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.184.0/22", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "172.96.98.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.125.0.0/16", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "13.248.24.0/22", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ip_prefix": "54.193.0.0/16", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.104.0/22", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.249.0/24", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.64.0/22", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.5.0/24", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.193.128/26", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.250.0.0/16", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "107.20.0.0/14", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.8.0/22", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.224.0/20", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.224.0/20", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.156.0/24", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "54.180.0.0/15", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.30.0.0/15", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.8.0/24", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.249.64/28", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.92.0.0/17", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.154.0.0/16", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "67.202.0.0/18", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "103.246.148.0/23", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.20.17/32", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.0.0/20", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.246.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.112/28", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.39.0/24", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.150.0/24", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.60.0/23", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.198.32/28", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.232.0.0/16", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.249.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "207.171.160.0/20", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.48.0/22", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.116.0/22", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.215.200/31", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.248.99.0/24", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.37.223/32", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.192/28", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.20.0/24", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.0.0/20", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.80.0/20", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "184.73.0.0/16", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "46.137.0.0/17", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.249.16/28", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.208.64/26", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "3.80.0.0/12", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.40.0.0/14", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.170.0/23", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.124.128.0/17", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "35.181.0.0/16", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.138.252/32", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.80.0.0/13", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.214.0.0/16", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.254.0.0/16", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.40.0/24", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.254.0/24", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "176.32.64.0/19", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "3.224.0.0/12", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.216.0/21", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.192.192/26", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.196.192/26", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.221.0.0/16", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.202.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.255.0.0/16", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.253.0.0/16", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.192.0/20", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.82.187.0/24", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.139.253/32", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.249.112/28", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.230.0.0/16", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "13.208.0.0/16", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.96.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.156.0.0/14", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.231.224.0/21", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.236.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.249.0/24", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.244.0.0/16", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.174.0/24", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.12.12/32", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.255.128/28", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.208.0.0/13", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.208/28", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "103.246.150.0/23", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.228.0.0/15", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.96/28", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.196.0.0/14", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.32.0.0/14", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.252.0/24", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.192.0/22", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.222.36.0/22", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.18.0.0/15", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.56.0/22", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.21.14/32", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.19.0/24", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.52.0/23", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "175.41.192.0/18", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.228.0/22", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.160/28", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.151.0.0/17", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.54.0.0/15", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.142.0/23", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.241.0/24", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.231.232.0/21", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.128.0/18", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.209.192/26", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "99.80.0.0/15", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.172.0/22", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.65.0.0/16", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.19.236/32", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.200.0/24", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.188.0/22", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.194.0/26", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.150.0.0/16", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.200.0.0/16", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.206.0.0/16", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "13.248.128.0/17", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.82.128.0/19", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.255.96/28", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.231.128.0/19", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.226.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.106.253/32", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.149.0/24", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.218.128.0/17", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "76.223.0.0/17", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "99.84.0.0/16", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "18.144.0.0/15", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.90.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.138.253/32", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.157.0/24", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.208.192/26", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.10.0.0/15", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.230.0/23", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "100.24.0.0/13", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.74.0.0/15", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "3.104.0.0/14", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.80.0.0/16", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.216.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.232.0/21", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.231.244.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "175.41.128.0/18", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.32.0/20", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.76.0/22", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.216.0.0/15", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.32/28", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.34.57/32", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.13.0/24", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.78.0.0/16", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.231.253.0/24", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.160.0/24", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "204.246.160.0/22", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.248.97.0/24", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "162.213.232.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.200.0.0/15", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.1.16/28", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "185.143.16.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.244.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.248.0.0/20", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.112.35/32", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.29.0/26", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "35.160.0.0/13", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.48.0.0/14", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.198.80/28", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.231.0.0/17", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.192.0/26", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "204.236.128.0/18", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.20.16/32", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.216.0/31", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.1.0/28", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.48.0.0/15", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.64.0.0/17", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.239.0/24", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.210.0/23", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "35.155.0.0/16", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.210.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.2.0/23", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.34.56/32", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.198.16/28", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.225.128/26", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.5.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.199.0.0/16", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.199.0/24", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.198.0.0/16", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.69.0/24", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.120.0/22", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "13.248.98.0/24", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.20.0.0/14", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.208/28", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.20.0/22", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.24.0/21", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.161.0/24", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "46.137.192.0/19", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.200.0.0/13", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.96.0/20", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.222.32.0/22", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.232.0/22", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.76.0.0/17", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.48.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.216.6/31", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.220.0/22", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.211.196/31", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.72.0/22", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.153.128.0/17", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.222.58.0/28", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "122.248.192.0/18", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.207.0.0/16", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "35.154.0.0/16", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.82.0.0/17", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.249.32/28", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.160/28", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.227.0/24", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.23.0/24", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.48.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.232.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.224.64/26", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.170.0.0/15", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.171.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.4.0/24", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.72.0/22", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.222.48.0/22", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.228.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "176.32.120.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.210.192/26", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.211.200/31", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.56.0/22", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.160.0.0/13", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "157.175.0.0/16", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "176.34.32.0/19", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.108.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.236.0.0/15", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.249.80/28", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.198.0/24", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.192.0/19", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "46.51.192.0/20", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.174.0/24", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.106.252/32", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.96.0/20", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.192.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.231.248.0/22", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "178.236.0.0/20", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "35.176.0.0/15", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.112.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.34.0/24", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.247.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "35.153.0.0/16", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.61.0.0/16", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.79.0.0/16", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.107.252/32", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.16.0/24", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.195.0/26", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.58.0.0/15", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.218.0.0/17", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.62.0.0/15", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.0.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.19.237/32", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.44.0/22", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.192.0/19", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.162.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.144/28", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "46.51.216.0/21", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.28.0.0/16", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.166.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.176.0/22", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.57.0.0/16", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.124.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.211.192/31", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.70.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.0/28", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.212.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.216.10/31", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.99.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.29.0.0/16", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.15.0/24", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.35.0/24", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.62.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.144.0/24", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.194.64/26", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.209.0/26", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.211.198/31", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "184.72.0.0/18", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.246.0/24", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.26.0/23", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.247.0.0/16", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.248.0.0/15", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "27.0.0.0/22", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.180.0/22", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.1.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.198.144/28", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.208.0/21", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.227.0/24", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.68.0.0/14", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.93.0.0/16", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "70.132.0.0/18", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.54.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.3.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.225.0/24", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.182.0.0/16", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.152.0.0/16", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.32.0.0/15", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "13.112.0.0/14", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.68.0.0/15", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.67.0.0/16", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.173.0/24", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.194.0.0/15", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.64.0/20", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.197.0/24", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.249.128/28", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.193.64/26", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.184.0.0/13", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.16.0/20", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.163.0/24", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.92.128.0/17", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.0/28", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.0.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.253.0/24", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.120.0/21", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "13.53.0.0/16", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.48.0/20", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "184.72.128.0/17", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.248.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.240.0/21", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.80/28", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.216.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.128.0/20", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.166.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.58.0.0/15", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.51.29/32", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.194.0.0/15", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.244.0/22", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "35.156.0.0/14", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.18.178/32", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.209.64/26", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "23.20.0.0/14", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.168.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.151.0/24", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.80/28", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.16.0/20", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.64.0/20", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.225.0/24", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ip_prefix": "172.96.97.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.229.0.0/16", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.68.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.192.0/20", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.219.0.0/16", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.204.0.0/14", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "35.178.0.0/15", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.9.0/24", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.204.0/23", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.88.0.0/14", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "75.2.0.0/17", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.12.0.0/15", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.249.0/28", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.220.0.0/15", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.231.252.0/24", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "13.35.0.0/16", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "34.240.0.0/13", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.14.19/32", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.16/28", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.249.96/28", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.216.8/31", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.200.0/24", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.253.0.0/16", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.240.0/22", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.28.0/23", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.128/28", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.100.0/23", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "204.246.172.0/23", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.64/28", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.72.0.0/15", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.192/28", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.11.0/24", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.196.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "204.246.164.0/22", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "54.223.0.0/16", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.48/28", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.24.0/22", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.196.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "79.125.0.0/17", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.88.0.0/15", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.0.0/20", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.248.0/21", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.32/28", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.40.0/22", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.220.0.0/16", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "100.20.0.0/14", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.24.0/23", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "3.8.0.0/14", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "18.246.0.0/16", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.139.252/32", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.198.0/28", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.204.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.163.0/24", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "216.182.236.0/23", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "34.208.0.0/12", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.15.0.0/16", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.17.16/32", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.209.128/26", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.30.0/23", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.96.0/22", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.145.0/24", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.86.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.44.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.76.128.0/17", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.40.0/21", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.32.0/21", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.95.0.0/16", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.212.0.0/15", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "18.232.0.0/14", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.224/28", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.48.0/22", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.47.0.0/16", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.16.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.136.0/23", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.255.64/28", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.225.64/26", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "204.246.168.0/22", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.62.0/23", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.175.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.208.0/21", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.51.28/32", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.12.0/24", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "63.32.0.0/14", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.83.0.0/16", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.14.18/32", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.6.0/24", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.197.192/26", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "150.222.2.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.79.0.0/16", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.251.0.0/16", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.52.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.153.0.0/16", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.202.0.0/15", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.1.48/28", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ip_prefix": "176.32.104.0/21", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.196.0.0/15", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.76.0.0/15", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.80.0/20", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.198.112/28", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.197.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "71.152.0.0/17", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "216.137.32.0/19", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.252.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.255.16/28", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "13.232.0.0/14", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.243.0/24", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.80.0/20", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.174.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "50.16.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.249.0/24", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.52.0.0/15", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.197.128/26", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.233.64.0/18", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "35.168.0.0/13", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.64.128.0/17", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.80.0/22", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.198.48/28", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.228.0/24", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.222.128.0/17", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "96.127.0.0/17", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.252.0/22", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.148.0.0/15", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "35.182.0.0/15", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "3.112.0.0/14", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.244.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.148.0/23", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "3.208.0.0/12", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.88.0/22", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "185.48.120.0/22", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.192.64/26", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.192.0/22", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "18.220.0.0/14", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.36.0.0/14", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.112.0/22", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.94.0.0/16", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.191.0.0/16", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.210.0/26", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.169.0/24", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.222.0.0/19", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.112/28", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.8.0/21", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.204.0/22", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "99.86.0.0/16", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "207.171.176.0/20", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.164.0/23", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.208.128/26", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.202.0.0/15", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.208.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.240.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.210.64/26", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "34.248.0.0/13", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.237.0/24", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.107.253/32", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "50.18.0.0/16", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.14.0.0/16", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.0.0/18", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.88.0/22", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.17.17/32", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.124.0.0/16", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.84.0.0/15", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.144/28", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.192.0.0/15", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.255.32/28", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "160.1.0.0/16", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.236.0.0/14", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.220.0/22", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.32.0/22", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.100.0/22", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.172.0/23", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "174.129.0.0/16", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.209.0.0/16", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.60.0.0/16", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.78.0.0/16", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "72.44.32.0/19", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.236.0/22", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "34.224.0.0/12", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.75.0.0/16", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.215.194/31", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.164.0/24", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.68.0/22", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.0.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.240.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.230.0.0/15", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.4.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.198.96/28", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.194.128/26", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.210.128/26", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.211.202/31", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.112.0/21", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.224.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.32.0/21", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "15.164.0.0/15", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.96.0/19", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.128.0/19", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.128/28", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "176.34.128.0/17", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.240.0/24", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.16.0/22", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "75.101.128.0/17", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.164.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.178.0.0/16", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.168.0/24", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "108.128.0.0/13", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.61.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.56.0.0/16", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.184.0.0/15", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "72.21.192.0/19", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.63.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.252.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.215.198/31", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.222.57.0/24", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.83.128.0/17", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "18.216.0.0/14", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "34.192.0.0/12", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.37.222/32", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.64.0/22", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.231.160.0/19", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.18.179/32", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.112.34/32", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.196.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.215.0.0/16", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "177.71.128.0/17", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.175.0.0/16", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.216.0/22", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.76.0/22", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.208.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.228.0.0/16", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "64.252.64.0/18", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.52.0/22", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.60.0/22", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.215.192/31", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.219.68.0/22", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.229.0.0/16", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.14.0/24", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.198.64/28", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.216.0/21", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.138.0.0/15", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.255.144/28", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "204.246.174.0/23", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "3.120.0.0/14", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.9.0.0/16", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.216.4/31", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.48/28", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.242.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "177.72.240.0/21", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "216.182.238.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "35.180.0.0/16", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.76.0/22", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.36.0/22", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "18.228.0.0/16", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.16.0.0/15", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.28.0/24", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.146.0/23", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.242.0/24", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.52.0.0/16", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "46.137.128.0/18", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.2.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.176/28", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.16.0/21", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.234.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.188.0.0/16", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "46.51.128.0/18", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "64.252.128.0/18", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.152.0/22", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.167.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.254.0/24", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.254.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.153.0.0/17", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.24.0.0/14", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.170.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.56.0/22", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.160.0/20", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.222.0.0/17", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.82.192.0/18", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.12.13/32", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.96.0/24", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.226.0/24", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.248.224/28", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.255.48/28", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.218.0.0/16", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "3.124.0.0/14", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.82.176.0/22", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.194.192/26", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.183.0.0/16", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.255.0/28", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.176.0.0/15", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.246.0.0/16", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.108.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.193.0/26", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "143.204.0.0/16", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "18.231.0.0/16", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.252.0.0/16", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "46.137.224.0/19", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.248.0/22", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.156.0/22", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.199.0/24", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.128.0/21", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.206.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.252.0/23", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.176.0/21", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.144.0.0/14", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.169.0.0/16", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.66.0.0/16", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.2.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "103.4.8.0/21", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "176.32.96.0/21", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "184.72.64.0/18", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.244.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.208.0/23", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.112.0/20", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.179.0.0/16", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.138.0/24", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.224.0/22", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.110.0/24", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "46.51.224.0/19", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.111.0/24", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.179.0.0/16", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.203.0/24", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.233.0.0/18", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.172.0/24", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.184.0/22", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.211.194/31", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.104.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "204.246.176.0/20", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.8.0.0/16", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.247.0/24", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.66.0.0/16", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.64/28", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.176.0/21", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "204.236.192.0/18", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.64.0.0/15", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "103.8.172.0/22", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "176.34.0.0/19", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.248.96.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.158.0/23", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.192.128/26", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.216.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.144.0/21", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.169.0/24", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.198.128/28", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.248.0/24", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.176/28", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.92.0/22", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.236.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.98.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.82.188.0/22", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.231.240.0/22", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "176.32.125.0/25", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "13.249.0.0/16", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "13.248.28.0/22", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.56.0/21", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.165.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "3.0.0.0/15", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "43.250.193.0/24", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.77.0.0/16", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.21.15/32", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.205.0/24", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.1.64/28", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.224.0.0/14", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.56.0.0/16", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.212.0/22", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.245.0.0/16", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "43.250.192.0/24", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.113.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "176.32.112.0/21", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.10.0/24", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "99.82.170.0/24", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.7.0/24", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.60.0/24", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.248.16.0/21", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.84.0/22", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.240.128.0/18", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "150.222.12.0/24", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "205.251.250.0/23", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.211.128/26", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.251.0/24", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.4.0.0/14", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.80.0/21", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ip_prefix": "52.46.184.0/22", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.67.0.0/16", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.116.0/22", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ip_prefix": "18.201.0.0/16", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.119.214.0/23", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.215.202/31", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.151.128.0/17", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.81.0.0/16", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.222.128.0/17", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "13.250.0.0/15", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.166.0/23", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.216.2/31", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "3.16.0.0/14", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ip_prefix": "18.130.0.0/16", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.72.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.82.180.0/22", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.182.0.0/16", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.168.0/24", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.224.128/26", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.192.0.0/16", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.16/28", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.0.96/28", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "18.136.0.0/16", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "50.112.0.0/16", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ip_prefix": "52.93.97.0/24", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.215.196/31", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ip_prefix": "87.238.80.0/21", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.255.80/28", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.92.252.0/22", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.95.250.0/24", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.144.211.0/26", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "50.19.0.0/16", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "99.79.0.0/16", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.57.0.0/16", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.126.0.0/15", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.239.4.0/22", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.172.0.0/15", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "176.34.64.0/18", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ip_prefix": "52.94.206.0/23", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.231.192.0/20", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.233.128.0/17", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ip_prefix": "203.83.220.0/22", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "54.245.168.0/26", - "region": "us-west-2", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "54.243.31.192/26", - "region": "us-east-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "177.71.207.128/26", - "region": "sa-east-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "54.255.254.192/26", - "region": "ap-southeast-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "54.244.52.192/26", - "region": "us-west-2", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "176.34.159.192/26", - "region": "eu-west-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "54.251.31.128/26", - "region": "ap-southeast-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "54.183.255.128/26", - "region": "us-west-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "54.241.32.64/26", - "region": "us-west-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "54.252.254.192/26", - "region": "ap-southeast-2", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "107.23.255.0/26", - "region": "us-east-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "54.248.220.0/26", - "region": "ap-northeast-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "54.228.16.0/26", - "region": "eu-west-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "54.250.253.192/26", - "region": "ap-northeast-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "54.232.40.64/26", - "region": "sa-east-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "54.252.79.128/26", - "region": "ap-southeast-2", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ip_prefix": "52.95.154.0/23", - "region": "eu-west-3", - "service": "S3" - }, - { - "ip_prefix": "52.219.64.0/22", - "region": "ap-south-1", - "service": "S3" - }, - { - "ip_prefix": "52.92.72.0/22", - "region": "sa-east-1", - "service": "S3" - }, - { - "ip_prefix": "52.92.64.0/22", - "region": "sa-east-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.156.0/24", - "region": "eu-west-3", - "service": "S3" - }, - { - "ip_prefix": "52.92.39.0/24", - "region": "sa-east-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.150.0/24", - "region": "eu-west-2", - "service": "S3" - }, - { - "ip_prefix": "52.219.60.0/23", - "region": "ap-northeast-2", - "service": "S3" - }, - { - "ip_prefix": "52.92.48.0/22", - "region": "us-west-1", - "service": "S3" - }, - { - "ip_prefix": "52.92.0.0/20", - "region": "ap-northeast-2", - "service": "S3" - }, - { - "ip_prefix": "52.95.170.0/23", - "region": "eu-north-1", - "service": "S3" - }, - { - "ip_prefix": "54.231.224.0/21", - "region": "ap-northeast-1", - "service": "S3" - }, - { - "ip_prefix": "52.92.56.0/22", - "region": "ap-southeast-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.142.0/23", - "region": "us-gov-west-1", - "service": "S3" - }, - { - "ip_prefix": "54.231.232.0/21", - "region": "us-west-1", - "service": "S3" - }, - { - "ip_prefix": "54.231.128.0/19", - "region": "eu-west-1", - "service": "S3" - }, - { - "ip_prefix": "52.218.128.0/17", - "region": "us-west-2", - "service": "S3" - }, - { - "ip_prefix": "52.95.157.0/24", - "region": "ap-northeast-3", - "service": "S3" - }, - { - "ip_prefix": "52.219.76.0/22", - "region": "ap-southeast-1", - "service": "S3" - }, - { - "ip_prefix": "54.231.253.0/24", - "region": "sa-east-1", - "service": "S3" - }, - { - "ip_prefix": "54.231.0.0/17", - "region": "us-east-1", - "service": "S3" - }, - { - "ip_prefix": "52.219.20.0/22", - "region": "us-west-1", - "service": "S3" - }, - { - "ip_prefix": "52.219.24.0/21", - "region": "us-west-1", - "service": "S3" - }, - { - "ip_prefix": "52.219.96.0/20", - "region": "us-east-2", - "service": "S3" - }, - { - "ip_prefix": "52.219.72.0/22", - "region": "eu-central-1", - "service": "S3" - }, - { - "ip_prefix": "54.222.48.0/22", - "region": "cn-north-1", - "service": "S3" - }, - { - "ip_prefix": "52.219.56.0/22", - "region": "ap-northeast-2", - "service": "S3" - }, - { - "ip_prefix": "52.95.174.0/24", - "region": "me-south-1", - "service": "S3" - }, - { - "ip_prefix": "54.231.248.0/22", - "region": "ap-southeast-2", - "service": "S3" - }, - { - "ip_prefix": "52.218.0.0/17", - "region": "eu-west-1", - "service": "S3" - }, - { - "ip_prefix": "52.219.44.0/22", - "region": "eu-central-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.144.0/24", - "region": "us-gov-west-1", - "service": "S3" - }, - { - "ip_prefix": "52.92.16.0/20", - "region": "us-east-1", - "service": "S3" - }, - { - "ip_prefix": "54.231.252.0/24", - "region": "ap-southeast-2", - "service": "S3" - }, - { - "ip_prefix": "52.219.0.0/20", - "region": "ap-northeast-1", - "service": "S3" - }, - { - "ip_prefix": "52.219.40.0/22", - "region": "ap-southeast-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.163.0/24", - "region": "sa-east-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.145.0/24", - "region": "ca-central-1", - "service": "S3" - }, - { - "ip_prefix": "52.92.40.0/21", - "region": "eu-west-1", - "service": "S3" - }, - { - "ip_prefix": "52.219.32.0/21", - "region": "ap-southeast-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.136.0/23", - "region": "sa-east-1", - "service": "S3" - }, - { - "ip_prefix": "52.219.62.0/23", - "region": "ap-south-1", - "service": "S3" - }, - { - "ip_prefix": "52.219.80.0/20", - "region": "us-east-2", - "service": "S3" - }, - { - "ip_prefix": "52.92.80.0/22", - "region": "ap-northeast-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.148.0/23", - "region": "eu-west-2", - "service": "S3" - }, - { - "ip_prefix": "52.92.88.0/22", - "region": "eu-west-2", - "service": "S3" - }, - { - "ip_prefix": "52.95.169.0/24", - "region": "eu-north-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.164.0/23", - "region": "sa-east-1", - "service": "S3" - }, - { - "ip_prefix": "52.92.32.0/22", - "region": "us-west-2", - "service": "S3" - }, - { - "ip_prefix": "52.95.172.0/23", - "region": "me-south-1", - "service": "S3" - }, - { - "ip_prefix": "52.92.68.0/22", - "region": "eu-central-1", - "service": "S3" - }, - { - "ip_prefix": "52.219.112.0/21", - "region": "us-west-1", - "service": "S3" - }, - { - "ip_prefix": "52.219.16.0/22", - "region": "ap-northeast-1", - "service": "S3" - }, - { - "ip_prefix": "54.231.160.0/19", - "region": "us-west-2", - "service": "S3" - }, - { - "ip_prefix": "52.92.76.0/22", - "region": "us-east-2", - "service": "S3" - }, - { - "ip_prefix": "52.92.52.0/22", - "region": "ap-southeast-2", - "service": "S3" - }, - { - "ip_prefix": "52.92.60.0/22", - "region": "ap-northeast-1", - "service": "S3" - }, - { - "ip_prefix": "52.219.68.0/22", - "region": "ap-northeast-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.146.0/23", - "region": "ca-central-1", - "service": "S3" - }, - { - "ip_prefix": "52.92.248.0/22", - "region": "ap-south-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.128.0/21", - "region": "ap-southeast-2", - "service": "S3" - }, - { - "ip_prefix": "52.95.138.0/24", - "region": "sa-east-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.158.0/23", - "region": "ap-northeast-3", - "service": "S3" - }, - { - "ip_prefix": "52.216.0.0/15", - "region": "us-east-1", - "service": "S3" - }, - { - "ip_prefix": "52.82.188.0/22", - "region": "cn-northwest-1", - "service": "S3" - }, - { - "ip_prefix": "54.231.240.0/22", - "region": "ap-southeast-1", - "service": "S3" - }, - { - "ip_prefix": "52.92.84.0/22", - "region": "ca-central-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.166.0/23", - "region": "us-gov-east-1", - "service": "S3" - }, - { - "ip_prefix": "52.95.168.0/24", - "region": "us-gov-east-1", - "service": "S3" - }, - { - "ip_prefix": "52.92.252.0/22", - "region": "us-gov-west-1", - "service": "S3" - }, - { - "ip_prefix": "54.231.192.0/20", - "region": "eu-central-1", - "service": "S3" - }, - { - "ip_prefix": "18.208.0.0/13", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.245.0/24", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.194.0.0/15", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.155.0.0/16", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.196.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.255.112/28", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "13.210.0.0/15", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "54.241.0.0/16", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "184.169.128.0/17", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "216.182.224.0/21", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.74.0.0/16", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.168.0.0/16", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.238.0.0/16", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "216.182.232.0/22", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "13.125.0.0/16", - "region": "ap-northeast-2", - "service": "EC2" - }, - { - "ip_prefix": "54.193.0.0/16", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.250.0.0/16", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "107.20.0.0/14", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.180.0.0/15", - "region": "ap-northeast-2", - "service": "EC2" - }, - { - "ip_prefix": "52.30.0.0/15", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.249.64/28", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "54.92.0.0/17", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.154.0.0/16", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "67.202.0.0/18", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.112/28", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "54.232.0.0/16", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.116.0/22", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.192/28", - "region": "eu-west-2", - "service": "EC2" - }, - { - "ip_prefix": "184.73.0.0/16", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "46.137.0.0/17", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.249.16/28", - "region": "cn-northwest-1", - "service": "EC2" - }, - { - "ip_prefix": "3.80.0.0/12", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.40.0.0/14", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "35.181.0.0/16", - "region": "eu-west-3", - "service": "EC2" - }, - { - "ip_prefix": "54.80.0.0/13", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.214.0.0/16", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "54.254.0.0/16", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.254.0/24", - "region": "eu-west-3", - "service": "EC2" - }, - { - "ip_prefix": "176.32.64.0/19", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "3.224.0.0/12", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.221.0.0/16", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.255.0.0/16", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "18.253.0.0/16", - "region": "us-gov-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.249.112/28", - "region": "us-gov-east-1", - "service": "EC2" - }, - { - "ip_prefix": "13.208.0.0/16", - "region": "ap-northeast-3", - "service": "EC2" - }, - { - "ip_prefix": "54.156.0.0/14", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.236.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.249.0/24", - "region": "ap-south-1", - "service": "EC2" - }, - { - "ip_prefix": "54.244.0.0/16", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.95.255.128/28", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "52.208.0.0/13", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "13.228.0.0/15", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.96/28", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.196.0.0/14", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.32.0.0/14", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.95.252.0/24", - "region": "ap-northeast-2", - "service": "EC2" - }, - { - "ip_prefix": "54.222.36.0/22", - "region": "cn-north-1", - "service": "EC2" - }, - { - "ip_prefix": "52.18.0.0/15", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "175.41.192.0/18", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.160/28", - "region": "us-east-2", - "service": "EC2" - }, - { - "ip_prefix": "54.151.0.0/17", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "13.54.0.0/15", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "52.95.241.0/24", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "99.80.0.0/15", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.65.0.0/16", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "54.150.0.0/16", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "18.200.0.0/16", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.206.0.0/16", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "52.95.255.96/28", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.226.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "18.144.0.0/15", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.90.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.10.0.0/15", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "100.24.0.0/13", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.74.0.0/15", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "3.104.0.0/14", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "52.80.0.0/16", - "region": "cn-north-1", - "service": "EC2" - }, - { - "ip_prefix": "175.41.128.0/18", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.216.0.0/15", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.78.0.0/16", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "162.213.232.0/24", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.200.0.0/15", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "35.160.0.0/13", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.48.0.0/14", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "204.236.128.0/18", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "13.48.0.0/15", - "region": "eu-north-1", - "service": "EC2" - }, - { - "ip_prefix": "52.64.0.0/17", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "52.95.239.0/24", - "region": "eu-west-2", - "service": "EC2" - }, - { - "ip_prefix": "35.155.0.0/16", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "54.210.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.199.0.0/16", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.198.0.0/16", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.20.0.0/14", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.208/28", - "region": "ca-central-1", - "service": "EC2" - }, - { - "ip_prefix": "46.137.192.0/19", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.200.0.0/13", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.222.32.0/22", - "region": "cn-north-1", - "service": "EC2" - }, - { - "ip_prefix": "52.76.0.0/17", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.153.128.0/17", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "122.248.192.0/18", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.207.0.0/16", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "35.154.0.0/16", - "region": "ap-south-1", - "service": "EC2" - }, - { - "ip_prefix": "52.82.0.0/17", - "region": "cn-northwest-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.249.32/28", - "region": "eu-west-3", - "service": "EC2" - }, - { - "ip_prefix": "54.170.0.0/15", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.160.0.0/13", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "157.175.0.0/16", - "region": "me-south-1", - "service": "EC2" - }, - { - "ip_prefix": "176.34.32.0/19", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "18.236.0.0/15", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.94.249.80/28", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "46.51.192.0/20", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "35.176.0.0/15", - "region": "eu-west-2", - "service": "EC2" - }, - { - "ip_prefix": "35.153.0.0/16", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.61.0.0/16", - "region": "us-gov-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.79.0.0/16", - "region": "ap-northeast-2", - "service": "EC2" - }, - { - "ip_prefix": "52.58.0.0/15", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "52.62.0.0/15", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "46.51.216.0/21", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.28.0.0/16", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "52.57.0.0/16", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "52.70.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.0/28", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.29.0.0/16", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "184.72.0.0/18", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.246.0/24", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.247.0.0/16", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.248.0.0/15", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.46.180.0/22", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.95.227.0/24", - "region": "eu-north-1", - "service": "EC2" - }, - { - "ip_prefix": "54.68.0.0/14", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "54.93.0.0/16", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "52.54.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "18.182.0.0/16", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.152.0.0/16", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "13.112.0.0/14", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.68.0.0/15", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.67.0.0/16", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "18.194.0.0/15", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.249.128/28", - "region": "eu-north-1", - "service": "EC2" - }, - { - "ip_prefix": "54.184.0.0/13", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "54.92.128.0/17", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.0.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.253.0/24", - "region": "eu-west-2", - "service": "EC2" - }, - { - "ip_prefix": "13.53.0.0/16", - "region": "eu-north-1", - "service": "EC2" - }, - { - "ip_prefix": "184.72.128.0/17", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "13.58.0.0/15", - "region": "us-east-2", - "service": "EC2" - }, - { - "ip_prefix": "54.194.0.0/15", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "35.156.0.0/14", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "23.20.0.0/14", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.80/28", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.225.0/24", - "region": "ap-northeast-3", - "service": "EC2" - }, - { - "ip_prefix": "18.229.0.0/16", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.219.0.0/16", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "18.204.0.0/14", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "35.178.0.0/15", - "region": "eu-west-2", - "service": "EC2" - }, - { - "ip_prefix": "54.88.0.0/14", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.12.0.0/15", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.94.249.0/28", - "region": "cn-north-1", - "service": "EC2" - }, - { - "ip_prefix": "52.220.0.0/15", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "34.240.0.0/13", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.16/28", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.249.96/28", - "region": "ap-northeast-3", - "service": "EC2" - }, - { - "ip_prefix": "54.253.0.0/16", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.128/28", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.64/28", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "54.72.0.0/15", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.223.0.0/16", - "region": "cn-north-1", - "service": "EC2" - }, - { - "ip_prefix": "79.125.0.0/17", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.88.0.0/15", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.32/28", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.220.0.0/16", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "100.20.0.0/14", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "3.8.0.0/14", - "region": "eu-west-2", - "service": "EC2" - }, - { - "ip_prefix": "18.246.0.0/16", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "54.204.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "216.182.236.0/23", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "34.208.0.0/12", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.15.0.0/16", - "region": "us-east-2", - "service": "EC2" - }, - { - "ip_prefix": "52.86.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.44.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.76.128.0/17", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.95.0.0/16", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.212.0.0/15", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "18.232.0.0/14", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.47.0.0/16", - "region": "eu-west-3", - "service": "EC2" - }, - { - "ip_prefix": "52.95.255.64/28", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "63.32.0.0/14", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.83.0.0/16", - "region": "cn-northwest-1", - "service": "EC2" - }, - { - "ip_prefix": "54.79.0.0/16", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "54.251.0.0/16", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "18.153.0.0/16", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "18.202.0.0/15", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "18.196.0.0/15", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "54.76.0.0/15", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.255.16/28", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "13.232.0.0/14", - "region": "ap-south-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.243.0/24", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.174.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "50.16.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.52.0.0/15", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.233.64.0/18", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "35.168.0.0/13", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.64.128.0/17", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "52.95.228.0/24", - "region": "me-south-1", - "service": "EC2" - }, - { - "ip_prefix": "54.222.128.0/17", - "region": "cn-north-1", - "service": "EC2" - }, - { - "ip_prefix": "96.127.0.0/17", - "region": "us-gov-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.148.0.0/15", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "35.182.0.0/15", - "region": "ca-central-1", - "service": "EC2" - }, - { - "ip_prefix": "3.112.0.0/14", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.244.0/24", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "3.208.0.0/12", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "185.48.120.0/22", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "18.220.0.0/14", - "region": "us-east-2", - "service": "EC2" - }, - { - "ip_prefix": "52.36.0.0/14", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "54.94.0.0/16", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "18.191.0.0/16", - "region": "us-east-2", - "service": "EC2" - }, - { - "ip_prefix": "54.202.0.0/15", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "34.248.0.0/13", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "50.18.0.0/16", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.14.0.0/16", - "region": "us-east-2", - "service": "EC2" - }, - { - "ip_prefix": "13.124.0.0/16", - "region": "ap-northeast-2", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.144/28", - "region": "ap-south-1", - "service": "EC2" - }, - { - "ip_prefix": "52.192.0.0/15", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.255.32/28", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "160.1.0.0/16", - "region": "us-gov-west-1", - "service": "EC2" - }, - { - "ip_prefix": "13.236.0.0/14", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "174.129.0.0/16", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "13.209.0.0/16", - "region": "ap-northeast-2", - "service": "EC2" - }, - { - "ip_prefix": "52.60.0.0/16", - "region": "ca-central-1", - "service": "EC2" - }, - { - "ip_prefix": "52.78.0.0/16", - "region": "ap-northeast-2", - "service": "EC2" - }, - { - "ip_prefix": "72.44.32.0/19", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "34.224.0.0/12", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.75.0.0/16", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "13.230.0.0/15", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.224.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "15.164.0.0/15", - "region": "ap-northeast-2", - "service": "EC2" - }, - { - "ip_prefix": "176.34.128.0/17", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.240.0/24", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "75.101.128.0/17", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.178.0.0/16", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "108.128.0.0/13", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "13.56.0.0/16", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "18.184.0.0/15", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "18.216.0.0/14", - "region": "us-east-2", - "service": "EC2" - }, - { - "ip_prefix": "34.192.0.0/12", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.215.0.0/16", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "177.71.128.0/17", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "18.175.0.0/16", - "region": "eu-west-2", - "service": "EC2" - }, - { - "ip_prefix": "54.208.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.228.0.0/16", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.229.0.0/16", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "18.138.0.0/15", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.255.144/28", - "region": "cn-north-1", - "service": "EC2" - }, - { - "ip_prefix": "3.120.0.0/14", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "52.9.0.0/16", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.48/28", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.242.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "216.182.238.0/23", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "35.180.0.0/16", - "region": "eu-west-3", - "service": "EC2" - }, - { - "ip_prefix": "18.228.0.0/16", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.16.0.0/15", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.242.0/24", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "13.52.0.0/16", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "46.137.128.0/18", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.176/28", - "region": "ap-northeast-2", - "service": "EC2" - }, - { - "ip_prefix": "54.234.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "18.188.0.0/16", - "region": "us-east-2", - "service": "EC2" - }, - { - "ip_prefix": "46.51.128.0/18", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.153.0.0/17", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.24.0.0/14", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.222.0.0/17", - "region": "us-gov-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.94.248.224/28", - "region": "us-gov-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.255.48/28", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.218.0.0/16", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "3.124.0.0/14", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "52.82.176.0/22", - "region": "cn-northwest-1", - "service": "EC2" - }, - { - "ip_prefix": "54.183.0.0/16", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.255.0/28", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.176.0.0/15", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.246.0.0/16", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "18.231.0.0/16", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.252.0.0/16", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "46.137.224.0/19", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.144.0.0/14", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.169.0.0/16", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.66.0.0/16", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ip_prefix": "52.2.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "103.4.8.0/21", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "184.72.64.0/18", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "18.179.0.0/16", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "46.51.224.0/19", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.179.0.0/16", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "54.233.0.0/18", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.8.0.0/16", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.247.0/24", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.66.0.0/16", - "region": "ap-south-1", - "service": "EC2" - }, - { - "ip_prefix": "204.236.192.0/18", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "54.64.0.0/15", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "176.34.0.0/19", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.248.0/24", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "3.0.0.0/15", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.77.0.0/16", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.119.205.0/24", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "18.224.0.0/14", - "region": "us-east-2", - "service": "EC2" - }, - { - "ip_prefix": "52.56.0.0/16", - "region": "eu-west-2", - "service": "EC2" - }, - { - "ip_prefix": "54.245.0.0/16", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.95.251.0/24", - "region": "us-east-2", - "service": "EC2" - }, - { - "ip_prefix": "52.4.0.0/14", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.46.184.0/22", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ip_prefix": "52.67.0.0/16", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "18.201.0.0/16", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.151.128.0/17", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "52.81.0.0/16", - "region": "cn-north-1", - "service": "EC2" - }, - { - "ip_prefix": "13.250.0.0/15", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "3.16.0.0/14", - "region": "us-east-2", - "service": "EC2" - }, - { - "ip_prefix": "18.130.0.0/16", - "region": "eu-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.72.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.82.180.0/22", - "region": "cn-northwest-1", - "service": "EC2" - }, - { - "ip_prefix": "18.136.0.0/16", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ip_prefix": "50.112.0.0/16", - "region": "us-west-2", - "service": "EC2" - }, - { - "ip_prefix": "52.95.255.80/28", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "52.95.250.0/24", - "region": "ca-central-1", - "service": "EC2" - }, - { - "ip_prefix": "50.19.0.0/16", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "99.79.0.0/16", - "region": "ca-central-1", - "service": "EC2" - }, - { - "ip_prefix": "13.57.0.0/16", - "region": "us-west-1", - "service": "EC2" - }, - { - "ip_prefix": "13.126.0.0/15", - "region": "ap-south-1", - "service": "EC2" - }, - { - "ip_prefix": "54.172.0.0/15", - "region": "us-east-1", - "service": "EC2" - }, - { - "ip_prefix": "176.34.64.0/18", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ip_prefix": "54.233.128.0/17", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ip_prefix": "205.251.192.0/21", - "region": "GLOBAL", - "service": "ROUTE53" - }, - { - "ip_prefix": "52.95.110.0/24", - "region": "GLOBAL", - "service": "ROUTE53" - }, - { - "ip_prefix": "13.124.199.0/24", - "region": "ap-northeast-2", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "34.226.14.0/24", - "region": "us-east-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.124.128.0/17", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "54.230.0.0/16", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "54.239.128.0/18", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.82.128.0/19", - "region": "cn-northwest-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "99.84.0.0/16", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.15.127.128/26", - "region": "us-east-2", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "35.158.136.0/24", - "region": "eu-central-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.57.254.0/24", - "region": "eu-central-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "18.216.170.128/25", - "region": "us-east-2", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "13.54.63.128/26", - "region": "ap-southeast-2", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "13.59.250.0/26", - "region": "us-east-2", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "13.210.67.128/26", - "region": "ap-southeast-2", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "35.167.191.128/26", - "region": "us-west-2", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.47.139.0/24", - "region": "eu-west-3", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.199.127.192/26", - "region": "ap-northeast-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.212.248.0/26", - "region": "eu-west-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "205.251.192.0/19", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.66.194.128/26", - "region": "ap-south-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "54.239.192.0/19", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "70.132.0.0/18", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "13.32.0.0/15", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "13.113.203.0/24", - "region": "ap-northeast-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "34.195.252.0/24", - "region": "us-east-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "35.162.63.192/26", - "region": "us-west-2", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "34.223.12.224/27", - "region": "us-west-2", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "13.35.0.0/16", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "204.246.172.0/23", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "204.246.164.0/22", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.56.127.0/25", - "region": "eu-west-2", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "204.246.168.0/22", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "13.228.69.0/24", - "region": "ap-southeast-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "34.216.51.0/25", - "region": "us-west-2", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "71.152.0.0/17", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "216.137.32.0/19", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "205.251.249.0/24", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "99.86.0.0/16", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.46.0.0/18", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.84.0.0/15", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "54.233.255.128/26", - "region": "sa-east-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "64.252.64.0/18", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.52.191.128/26", - "region": "us-west-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "204.246.174.0/23", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "64.252.128.0/18", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "205.251.254.0/24", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "143.204.0.0/16", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "205.251.252.0/23", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.78.247.128/26", - "region": "ap-northeast-2", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "204.246.176.0/20", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.220.191.0/26", - "region": "ap-southeast-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "13.249.0.0/16", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "54.240.128.0/18", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "205.251.250.0/23", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.222.128.0/17", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "54.182.0.0/16", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "54.192.0.0/16", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "34.232.163.208/29", - "region": "us-east-1", - "service": "CLOUDFRONT" - }, - { - "ip_prefix": "52.47.73.72/29", - "region": "eu-west-3", - "service": "CODEBUILD" - }, - { - "ip_prefix": "13.55.255.216/29", - "region": "ap-southeast-2", - "service": "CODEBUILD" - }, - { - "ip_prefix": "52.15.247.208/29", - "region": "us-east-2", - "service": "CODEBUILD" - }, - { - "ip_prefix": "13.112.191.184/29", - "region": "ap-northeast-1", - "service": "CODEBUILD" - }, - { - "ip_prefix": "34.250.63.248/29", - "region": "eu-west-1", - "service": "CODEBUILD" - }, - { - "ip_prefix": "52.221.221.128/29", - "region": "ap-southeast-1", - "service": "CODEBUILD" - }, - { - "ip_prefix": "13.127.70.136/29", - "region": "ap-south-1", - "service": "CODEBUILD" - }, - { - "ip_prefix": "52.82.1.0/29", - "region": "cn-northwest-1", - "service": "CODEBUILD" - }, - { - "ip_prefix": "177.71.207.16/29", - "region": "sa-east-1", - "service": "CODEBUILD" - }, - { - "ip_prefix": "13.124.145.16/29", - "region": "ap-northeast-2", - "service": "CODEBUILD" - }, - { - "ip_prefix": "35.157.127.248/29", - "region": "eu-central-1", - "service": "CODEBUILD" - }, - { - "ip_prefix": "35.182.14.48/29", - "region": "ca-central-1", - "service": "CODEBUILD" - }, - { - "ip_prefix": "35.176.92.32/29", - "region": "eu-west-2", - "service": "CODEBUILD" - }, - { - "ip_prefix": "52.43.76.88/29", - "region": "us-west-2", - "service": "CODEBUILD" - }, - { - "ip_prefix": "18.231.194.8/29", - "region": "sa-east-1", - "service": "CODEBUILD" - }, - { - "ip_prefix": "52.80.198.136/29", - "region": "cn-north-1", - "service": "CODEBUILD" - }, - { - "ip_prefix": "13.56.32.200/29", - "region": "us-west-1", - "service": "CODEBUILD" - }, - { - "ip_prefix": "34.228.4.208/28", - "region": "us-east-1", - "service": "CODEBUILD" - }, - { - "ip_prefix": "13.248.99.0/24", - "region": "us-west-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.174.0/24", - "region": "ca-central-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "13.248.128.0/17", - "region": "GLOBAL", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "76.223.0.0/17", - "region": "GLOBAL", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.160.0/24", - "region": "ap-south-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "13.248.97.0/24", - "region": "eu-central-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "13.248.98.0/24", - "region": "ap-northeast-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.161.0/24", - "region": "eu-west-3", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.171.0/24", - "region": "us-east-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.162.0/24", - "region": "eu-west-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.173.0/24", - "region": "ap-southeast-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.163.0/24", - "region": "eu-central-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.166.0/24", - "region": "us-east-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "75.2.0.0/17", - "region": "GLOBAL", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.175.0/24", - "region": "us-east-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.164.0/24", - "region": "sa-east-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.168.0/24", - "region": "ap-northeast-2", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.83.128.0/17", - "region": "GLOBAL", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.167.0/24", - "region": "us-east-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.156.0/22", - "region": "GLOBAL", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.172.0/24", - "region": "us-west-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "13.248.96.0/24", - "region": "eu-west-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.169.0/24", - "region": "eu-west-2", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.165.0/24", - "region": "us-east-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "99.82.170.0/24", - "region": "ap-northeast-1", - "service": "GLOBALACCELERATOR" - }, - { - "ip_prefix": "13.251.113.64/26", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.251.116.0/23", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ip_prefix": "13.210.2.192/26", - "region": "ap-southeast-2", - "service": "AMAZON_CONNECT" - }, - { - "ip_prefix": "13.236.8.0/25", - "region": "ap-southeast-2", - "service": "AMAZON_CONNECT" - }, - { - "ip_prefix": "18.182.96.64/26", - "region": "ap-northeast-1", - "service": "AMAZON_CONNECT" - }, - { - "ip_prefix": "18.184.2.128/25", - "region": "eu-central-1", - "service": "AMAZON_CONNECT" - }, - { - "ip_prefix": "18.233.213.128/25", - "region": "us-east-1", - "service": "AMAZON_CONNECT" - }, - { - "ip_prefix": "18.236.61.0/25", - "region": "us-west-2", - "service": "AMAZON_CONNECT" - }, - { - "ip_prefix": "35.158.127.64/26", - "region": "eu-central-1", - "service": "AMAZON_CONNECT" - }, - { - "ip_prefix": "52.55.191.224/27", - "region": "us-east-1", - "service": "AMAZON_CONNECT" - }, - { - "ip_prefix": "54.190.198.32/28", - "region": "us-west-2", - "service": "AMAZON_CONNECT" - }, - { - "ip_prefix": "13.250.186.128/27", - "region": "ap-southeast-1", - "service": "CLOUD9" - }, - { - "ip_prefix": "13.250.186.160/27", - "region": "ap-southeast-1", - "service": "CLOUD9" - }, - { - "ip_prefix": "18.188.9.0/27", - "region": "us-east-2", - "service": "CLOUD9" - }, - { - "ip_prefix": "18.188.9.32/27", - "region": "us-east-2", - "service": "CLOUD9" - }, - { - "ip_prefix": "34.217.141.224/27", - "region": "us-west-2", - "service": "CLOUD9" - }, - { - "ip_prefix": "34.218.119.32/27", - "region": "us-west-2", - "service": "CLOUD9" - }, - { - "ip_prefix": "34.245.205.0/27", - "region": "eu-west-1", - "service": "CLOUD9" - }, - { - "ip_prefix": "34.245.205.64/27", - "region": "eu-west-1", - "service": "CLOUD9" - }, - { - "ip_prefix": "35.172.155.192/27", - "region": "us-east-1", - "service": "CLOUD9" - }, - { - "ip_prefix": "35.172.155.96/27", - "region": "us-east-1", - "service": "CLOUD9" - } - ], - "ipv6_prefixes": [ - { - "ipv6_prefix": "2a05:d07c:2000::/40", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:a300::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d000:8000::/40", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafe:2000::/40", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:4860::/47", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daff:8000::/40", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:da1a::/36", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daf8:c000::/40", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daf9:6000::/40", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d050:2000::/40", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daa0:8000::/40", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d000:4000::/40", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffc:8000::/40", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2620:107:300f::/64", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07c:8000::/40", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:da18::/36", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafa:2000::/40", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2400:6500:ff00::/64", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a01:578:0:7000::/56", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daf9:8000::/40", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80ff:4000::/40", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2400:6500:0:7500::/56", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:48b0::/47", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07e:e000::/40", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:4840::/47", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d050:e000::/40", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:a600::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07c:e000::/40", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:a500::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2400:6500:0:7100::/56", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d050:c000::/40", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafa:6000::/40", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07f:c000::/40", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafa:a000::/40", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f14::/35", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2620:107:4000:7000::/56", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d079:c000::/40", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2400:6500:100:7200::/56", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafa:4000::/40", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daf9:a000::/40", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffe:6000::/40", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f00:e000::/40", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffa:2000::/40", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffa:6000::/40", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daf9:4000::/40", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff9:4000::/40", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffc:e000::/40", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d050:8000::/40", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafe:8000::/40", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:4820::/47", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07e:8000::/40", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafc:4000::/40", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d000:c000::/40", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07f:e000::/40", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:eee::/48", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daa0:6000::/40", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffc:1000::/40", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2620:107:4000:5::/64", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff8:1000::/40", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2400:6500:0:7200::/56", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff9:c000::/40", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffe:e000::/40", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:aa00::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fff:1000::/40", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:4870::/47", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d079:e000::/40", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2620:108:d000::/44", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fff:4000::/40", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daff:4000::/40", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07f:4000::/40", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fa0:4000::/40", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f00:1000::/40", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d000:e000::/40", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d050:4000::/40", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fa0:8000::/40", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffe:8000::/40", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:da16::/36", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daa0:c000::/40", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2620:107:4000:7200::/56", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80fe:4000::/40", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07e:2000::/40", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffa:c000::/40", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07f:2000::/40", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d016::/36", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafa:c000::/40", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:da00:8000::/40", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f00:c000::/40", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2620:107:4000:7100::/56", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafe:4000::/40", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:4810::/47", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f12::/36", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffc:4000::/40", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:4830::/47", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fa0:e000::/40", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffc:2000::/40", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d079:8000::/40", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d050:6000::/40", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:ab00::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daf8:8000::/40", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffe:1000::/40", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff8:4000::/40", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07f:6000::/40", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80a0:8000::/40", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fff:e000::/40", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daf9:c000::/40", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d078:2000::/40", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafc:a000::/40", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafe:6000::/40", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daff:c000::/40", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:da00:2000::/40", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff8:5000::/36", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:af00::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fff:8000::/40", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:a100::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f1c::/36", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:4000::/36", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d012::/36", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f00:6000::/40", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:ac00::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80fc:4000::/40", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f11::/36", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80f9:8000::/40", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffe:2000::/40", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffe:5000::/40", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07c:4000::/40", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daff:2000::/40", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:8014::/36", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80a0:4000::/40", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffa:4000::/40", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d078:8000::/40", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d079:4000::/40", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80f8:4000::/40", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:3000::/36", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:f000::/36", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2804:800:0:7000::/56", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f00:8000::/40", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:4850::/47", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:48a0::/47", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:fff::/48", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80fa:8000::/40", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff9:e000::/40", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80f9:4000::/40", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f1e::/36", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2620:108:7000::/44", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:a400::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d079:2000::/40", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffa:8000::/40", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80fc:8000::/40", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:48d0::/47", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:a800::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d01e::/36", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fa0:6000::/40", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fff:c000::/40", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fff:2000::/40", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07a:c000::/40", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fa0:c000::/40", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2400:6700:ff00::/64", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2403:b300:ff00::/64", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daa0:2000::/40", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f16::/36", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d078:c000::/40", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07a:2000::/40", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07a:6000::/40", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daff:a000::/40", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f00:4000::/40", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fa0:2000::/40", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d078:e000::/40", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff9:1000::/40", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daf8:2000::/40", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafc:2000::/40", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff8:6000::/40", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffe:4000::/40", - "region": "us-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafc:6000::/40", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daf8:a000::/40", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f00:2000::/40", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:2000::/36", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffa:1000::/40", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2400:6500:0:7400::/56", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:1000::/36", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07f:8000::/40", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:8018::/36", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffa:5000::/40", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a01:578:13::/64", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:da00:a000::/40", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fa0:1000::/40", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffc:c000::/40", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:4880::/47", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2620:107:4007::/64", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07c:6000::/40", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff9:2000::/40", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:a900::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07a:4000::/40", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2620:107:4000:7400::/56", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a01:578:0:7100::/56", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fa0:5000::/40", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:8000:4000::/40", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07e:6000::/40", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daf8:6000::/40", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2400:6500:0:7000::/56", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafa:8000::/40", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff9:5000::/40", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff8:8000::/40", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2400:6500:100:7100::/56", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff8:2000::/40", - "region": "us-gov-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f15::/32", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:4800::/47", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d078:4000::/40", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07a:8000::/40", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07e:c000::/40", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:da00:c000::/40", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:da14::/36", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff9:8000::/40", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d078:6000::/40", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d000:2000::/40", - "region": "eu-west-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffe:c000::/40", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07c:c000::/40", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafc:c000::/40", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:ae00::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07e:4000::/40", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff8:c000::/40", - "region": "us-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:ddd::/48", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:da1c::/36", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:da00:6000::/40", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daa0:4000::/40", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daf9:2000::/40", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2804:800:ff00::/64", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d079:6000::/40", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:da00:4000::/40", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffa:e000::/40", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffc:6000::/40", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:8000:8000::/40", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:ad00::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d018::/36", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80ff:8000::/40", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:48c0::/47", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d01c::/36", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafc:8000::/40", - "region": "ap-southeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff9:6000::/40", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d000:6000::/40", - "region": "eu-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80f8:8000::/40", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:a200::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2620:107:4000:7800::/56", - "region": "ca-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a01:578:0:7200::/56", - "region": "eu-west-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d014::/36", - "region": "eu-central-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daf8:4000::/40", - "region": "ap-northeast-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafe:c000::/40", - "region": "ap-southeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f18::/33", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ff8:e000::/40", - "region": "sa-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80fe:8000::/40", - "region": "cn-north-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2620:107:4000:7700::/56", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daa0:a000::/40", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:dafe:a000::/40", - "region": "ap-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fff:5000::/40", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:da12::/36", - "region": "ap-northeast-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f01:4890::/47", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:daff:6000::/40", - "region": "ap-northeast-3", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1f00:5000::/40", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2406:da00:ff00::/64", - "region": "us-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:5300::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:9000:a700::/40", - "region": "GLOBAL", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a05:d07a:e000::/40", - "region": "me-south-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1ffc:5000::/40", - "region": "us-gov-east-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2600:1fff:6000::/40", - "region": "us-east-2", - "service": "AMAZON" - }, - { - "ipv6_prefix": "240f:80fa:4000::/40", - "region": "cn-northwest-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2a01:578:3::/64", - "region": "eu-west-1", - "service": "AMAZON" - }, - { - "ipv6_prefix": "2804:800:ff00::b147:cf80/122", - "region": "sa-east-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2400:6700:ff00::36f8:dc00/122", - "region": "ap-northeast-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2403:b300:ff00::36fc:4f80/122", - "region": "ap-southeast-2", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2406:da00:ff00::36f3:1fc0/122", - "region": "us-east-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2406:da14:fff:f800::/53", - "region": "ap-northeast-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2400:6500:ff00::36ff:fec0/122", - "region": "ap-southeast-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2a01:578:3::36e4:1000/122", - "region": "eu-west-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2600:1f14:fff:f800::/53", - "region": "us-west-2", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2600:1f14:7ff:f800::/53", - "region": "us-west-2", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2406:da1c:7ff:f800::/53", - "region": "ap-southeast-2", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2600:1f18:7fff:f800::/53", - "region": "us-east-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2600:1f18:3fff:f800::/53", - "region": "us-east-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2804:800:ff00::36e8:2840/122", - "region": "sa-east-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2620:107:300f::36f1:2040/122", - "region": "us-west-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2620:108:700f::36f4:34c0/122", - "region": "us-west-2", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2620:107:300f::36b7:ff80/122", - "region": "us-west-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2403:b300:ff00::36fc:fec0/122", - "region": "ap-southeast-2", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2a05:d018:7ff:f800::/53", - "region": "eu-west-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2406:da00:ff00::6b17:ff00/122", - "region": "us-east-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2400:6700:ff00::36fa:fdc0/122", - "region": "ap-northeast-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2406:da18:fff:f800::/53", - "region": "ap-southeast-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2620:108:700f::36f5:a800/122", - "region": "us-west-2", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2600:1f1c:7ff:f800::/53", - "region": "us-west-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2406:da1c:fff:f800::/53", - "region": "ap-southeast-2", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2600:1f1e:fff:f800::/53", - "region": "sa-east-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2600:1f1c:fff:f800::/53", - "region": "us-west-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2406:da14:7ff:f800::/53", - "region": "ap-northeast-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2a05:d018:fff:f800::/53", - "region": "eu-west-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2400:6500:ff00::36fb:1f80/122", - "region": "ap-southeast-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2406:da18:7ff:f800::/53", - "region": "ap-southeast-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2600:1f1e:7ff:f800::/53", - "region": "sa-east-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2a01:578:3::b022:9fc0/122", - "region": "eu-west-1", - "service": "ROUTE53_HEALTHCHECKS" - }, - { - "ipv6_prefix": "2406:daf8:c000::/40", - "region": "ap-southeast-2", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daf9:6000::/40", - "region": "ap-northeast-3", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d050:2000::/40", - "region": "eu-west-3", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daa0:8000::/40", - "region": "ap-southeast-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:dafa:2000::/40", - "region": "ap-northeast-2", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daf9:8000::/40", - "region": "ap-southeast-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d050:e000::/40", - "region": "me-south-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d050:c000::/40", - "region": "eu-west-2", - "service": "S3" - }, - { - "ipv6_prefix": "2406:dafa:6000::/40", - "region": "ap-northeast-3", - "service": "S3" - }, - { - "ipv6_prefix": "2406:dafa:a000::/40", - "region": "ap-south-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d079:c000::/40", - "region": "eu-west-2", - "service": "S3" - }, - { - "ipv6_prefix": "2406:dafa:4000::/40", - "region": "ap-northeast-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daf9:a000::/40", - "region": "ap-south-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ffa:2000::/40", - "region": "us-gov-west-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ffa:6000::/40", - "region": "us-east-2", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daf9:4000::/40", - "region": "ap-northeast-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff9:4000::/40", - "region": "us-west-2", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d050:8000::/40", - "region": "eu-west-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daa0:6000::/40", - "region": "ap-northeast-3", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff8:1000::/40", - "region": "ca-central-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff9:c000::/40", - "region": "us-west-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d079:e000::/40", - "region": "me-south-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1fa0:4000::/40", - "region": "us-west-2", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d050:4000::/40", - "region": "eu-central-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1fa0:8000::/40", - "region": "us-east-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daa0:c000::/40", - "region": "ap-southeast-2", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ffa:c000::/40", - "region": "us-west-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:dafa:c000::/40", - "region": "ap-southeast-2", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1fa0:e000::/40", - "region": "sa-east-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d079:8000::/40", - "region": "eu-west-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daf8:8000::/40", - "region": "ap-southeast-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff8:4000::/40", - "region": "us-west-2", - "service": "S3" - }, - { - "ipv6_prefix": "240f:80a0:8000::/40", - "region": "cn-north-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daf9:c000::/40", - "region": "ap-southeast-2", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d078:2000::/40", - "region": "eu-west-3", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff8:5000::/36", - "region": "us-gov-east-1", - "service": "S3" - }, - { - "ipv6_prefix": "240f:80f9:8000::/40", - "region": "cn-north-1", - "service": "S3" - }, - { - "ipv6_prefix": "240f:80a0:4000::/40", - "region": "cn-northwest-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ffa:4000::/40", - "region": "us-west-2", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d078:8000::/40", - "region": "eu-west-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d079:4000::/40", - "region": "eu-central-1", - "service": "S3" - }, - { - "ipv6_prefix": "240f:80f8:4000::/40", - "region": "cn-northwest-1", - "service": "S3" - }, - { - "ipv6_prefix": "240f:80fa:8000::/40", - "region": "cn-north-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff9:e000::/40", - "region": "sa-east-1", - "service": "S3" - }, - { - "ipv6_prefix": "240f:80f9:4000::/40", - "region": "cn-northwest-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d079:2000::/40", - "region": "eu-west-3", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ffa:8000::/40", - "region": "us-east-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1fa0:6000::/40", - "region": "us-east-2", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d07a:c000::/40", - "region": "eu-west-2", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1fa0:c000::/40", - "region": "us-west-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daa0:2000::/40", - "region": "ap-northeast-2", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d078:c000::/40", - "region": "eu-west-2", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d07a:2000::/40", - "region": "eu-west-3", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d07a:6000::/40", - "region": "eu-north-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1fa0:2000::/40", - "region": "us-gov-west-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d078:e000::/40", - "region": "me-south-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff9:1000::/40", - "region": "ca-central-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daf8:2000::/40", - "region": "ap-northeast-2", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff8:6000::/40", - "region": "us-east-2", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daf8:a000::/40", - "region": "ap-south-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ffa:1000::/40", - "region": "ca-central-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ffa:5000::/40", - "region": "us-gov-east-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1fa0:1000::/40", - "region": "ca-central-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff9:2000::/40", - "region": "us-gov-west-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d07a:4000::/40", - "region": "eu-central-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1fa0:5000::/40", - "region": "us-gov-east-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daf8:6000::/40", - "region": "ap-northeast-3", - "service": "S3" - }, - { - "ipv6_prefix": "2406:dafa:8000::/40", - "region": "ap-southeast-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff9:5000::/40", - "region": "us-gov-east-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff8:8000::/40", - "region": "us-east-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff8:2000::/40", - "region": "us-gov-west-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d078:4000::/40", - "region": "eu-central-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d07a:8000::/40", - "region": "eu-west-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff9:8000::/40", - "region": "us-east-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d078:6000::/40", - "region": "eu-north-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff8:c000::/40", - "region": "us-west-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daa0:4000::/40", - "region": "ap-northeast-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daf9:2000::/40", - "region": "ap-northeast-2", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d079:6000::/40", - "region": "eu-north-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ffa:e000::/40", - "region": "sa-east-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff9:6000::/40", - "region": "us-east-2", - "service": "S3" - }, - { - "ipv6_prefix": "240f:80f8:8000::/40", - "region": "cn-north-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daf8:4000::/40", - "region": "ap-northeast-1", - "service": "S3" - }, - { - "ipv6_prefix": "2600:1ff8:e000::/40", - "region": "sa-east-1", - "service": "S3" - }, - { - "ipv6_prefix": "2406:daa0:a000::/40", - "region": "ap-south-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d07a:e000::/40", - "region": "me-south-1", - "service": "S3" - }, - { - "ipv6_prefix": "240f:80fa:4000::/40", - "region": "cn-northwest-1", - "service": "S3" - }, - { - "ipv6_prefix": "2a05:d000:8000::/40", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:daff:8000::/40", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:da1a::/36", - "region": "ap-south-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d000:4000::/40", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2620:107:300f::/64", - "region": "us-west-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:da18::/36", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2400:6500:ff00::/64", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ipv6_prefix": "240f:80ff:4000::/40", - "region": "cn-northwest-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d07f:c000::/40", - "region": "eu-west-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f14::/35", - "region": "us-west-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f00:e000::/40", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d000:c000::/40", - "region": "eu-west-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d07f:e000::/40", - "region": "me-south-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1fff:1000::/40", - "region": "ca-central-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1fff:4000::/40", - "region": "us-west-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:daff:4000::/40", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d07f:4000::/40", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f00:1000::/40", - "region": "ca-central-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d000:e000::/40", - "region": "me-south-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:da16::/36", - "region": "ap-northeast-3", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d07f:2000::/40", - "region": "eu-west-3", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d016::/36", - "region": "eu-north-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:da00:8000::/40", - "region": "ap-southeast-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f00:c000::/40", - "region": "us-west-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f12::/36", - "region": "us-gov-west-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d07f:6000::/40", - "region": "eu-north-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1fff:e000::/40", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:daff:c000::/40", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:da00:2000::/40", - "region": "ap-northeast-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1fff:8000::/40", - "region": "us-east-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f1c::/36", - "region": "us-west-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d012::/36", - "region": "eu-west-3", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f00:6000::/40", - "region": "us-east-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2620:108:700f::/64", - "region": "us-west-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f11::/36", - "region": "ca-central-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:daff:2000::/40", - "region": "ap-northeast-2", - "service": "EC2" - }, - { - "ipv6_prefix": "240f:8014::/36", - "region": "cn-northwest-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f00:8000::/40", - "region": "us-east-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f1e::/36", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d01e::/36", - "region": "me-south-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1fff:c000::/40", - "region": "us-west-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1fff:2000::/40", - "region": "us-gov-west-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2400:6700:ff00::/64", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2403:b300:ff00::/64", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f16::/36", - "region": "us-east-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:daff:a000::/40", - "region": "ap-south-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f00:4000::/40", - "region": "us-west-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f00:2000::/40", - "region": "us-gov-west-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d07f:8000::/40", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ipv6_prefix": "240f:8018::/36", - "region": "cn-north-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a01:578:13::/64", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:da00:a000::/40", - "region": "ap-south-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2620:107:4007::/64", - "region": "us-east-1", - "service": "EC2" - }, - { - "ipv6_prefix": "240f:8000:4000::/40", - "region": "cn-northwest-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f15::/32", - "region": "us-gov-east-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:da00:c000::/40", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:da14::/36", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d000:2000::/40", - "region": "eu-west-3", - "service": "EC2" - }, - { - "ipv6_prefix": "2620:108:d00f::/64", - "region": "us-gov-west-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:da1c::/36", - "region": "ap-southeast-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:da00:6000::/40", - "region": "ap-northeast-3", - "service": "EC2" - }, - { - "ipv6_prefix": "2804:800:ff00::/64", - "region": "sa-east-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:da00:4000::/40", - "region": "ap-northeast-1", - "service": "EC2" - }, - { - "ipv6_prefix": "240f:8000:8000::/40", - "region": "cn-north-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d018::/36", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ipv6_prefix": "240f:80ff:8000::/40", - "region": "cn-north-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d01c::/36", - "region": "eu-west-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d000:6000::/40", - "region": "eu-north-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2a05:d014::/36", - "region": "eu-central-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f18::/33", - "region": "us-east-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1fff:5000::/40", - "region": "us-gov-east-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:da12::/36", - "region": "ap-northeast-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:daff:6000::/40", - "region": "ap-northeast-3", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1f00:5000::/40", - "region": "us-gov-east-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2406:da00:ff00::/64", - "region": "us-east-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:1fff:6000::/40", - "region": "us-east-2", - "service": "EC2" - }, - { - "ipv6_prefix": "2a01:578:3::/64", - "region": "eu-west-1", - "service": "EC2" - }, - { - "ipv6_prefix": "2600:9000:eee::/48", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ipv6_prefix": "2600:9000:4000::/36", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ipv6_prefix": "2600:9000:3000::/36", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ipv6_prefix": "2600:9000:f000::/36", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ipv6_prefix": "2600:9000:fff::/48", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ipv6_prefix": "2600:9000:2000::/36", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ipv6_prefix": "2600:9000:1000::/36", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ipv6_prefix": "2600:9000:ddd::/48", - "region": "GLOBAL", - "service": "CLOUDFRONT" - }, - { - "ipv6_prefix": "2600:9000:5300::/40", - "region": "GLOBAL", - "service": "CLOUDFRONT" - } - ] -} \ No newline at end of file diff --git a/theHarvester/lib/output.py b/theHarvester/lib/output.py index 0857ba61..dec9dba9 100644 --- a/theHarvester/lib/output.py +++ b/theHarvester/lib/output.py @@ -59,9 +59,7 @@ def print_section(header: str, items: Iterable[str], separator: str) -> None: output_logger.info(item) -def print_linkedin_sections( - engines: Sequence[str], people: Sequence[str], links: Sequence[str], separator: str = '---------------------' -) -> None: +def print_linkedin_people(engines: Sequence[str], people: Sequence[str], separator: str = '---------------------') -> None: if len(people) == 0 and 'linkedin' in engines: output_logger.info('\n[*] No LinkedIn users found.\n\n') elif len(people) >= 1: @@ -69,9 +67,3 @@ def print_linkedin_sections( output_logger.info(separator) for usr in sorted_unique(people): output_logger.info(usr) - - if 'linkedin' in engines or 'rocketreach' in engines: - output_logger.info(f'\n[*] LinkedIn Links found: {len(links)}') - output_logger.info(separator) - for link in sorted_unique(links): - output_logger.info(link) diff --git a/theHarvester/lib/resolver_selection.py b/theHarvester/lib/resolver_selection.py new file mode 100644 index 00000000..7c2b684b --- /dev/null +++ b/theHarvester/lib/resolver_selection.py @@ -0,0 +1,24 @@ +from collections.abc import Iterable +from ipaddress import ip_address + +DEFAULT_DNS_RESOLVERS = ('1.1.1.1', '8.8.8.8', '9.9.9.9') + + +def normalize_resolver_addresses(values: Iterable[str]) -> list[str]: + """Return distinct canonical resolver IP addresses in operator order.""" + addresses: list[str] = [] + seen: set[str] = set() + for raw_value in values: + value = raw_value.strip() + if not value: + continue + try: + address = str(ip_address(value)) + except ValueError as error: + raise ValueError(f'Invalid DNS resolver address: {value}') from error + if address not in seen: + addresses.append(address) + seen.add(address) + if not addresses: + raise ValueError('Provide at least one DNS resolver IP address') + return addresses diff --git a/theHarvester/lib/resolvers.txt b/theHarvester/lib/resolvers.txt deleted file mode 100644 index d50a913d..00000000 --- a/theHarvester/lib/resolvers.txt +++ /dev/null @@ -1,2016 +0,0 @@ -1.0.0.1 -1.1.1.1 -141.1.27.249 -194.190.225.2 -194.225.16.5 -91.185.6.10 -194.2.0.50 -66.187.16.5 -83.222.161.130 -69.60.160.196 -194.150.118.3 -84.8.2.11 -195.175.39.40 -193.239.159.37 -205.152.6.20 -82.151.90.1 -144.76.202.253 -103.3.46.254 -5.144.17.119 -195.129.12.122 -211.35.96.6 -202.138.120.4 -209.130.139.2 -64.81.127.2 -202.199.160.206 -195.66.68.2 -103.3.76.7 -202.219.177.121 -216.143.135.12 -141.211.144.17 -101.203.168.123 -217.73.17.110 -205.242.187.234 -62.192.160.39 -187.115.52.101 -122.155.167.38 -203.229.169.69 -69.25.1.1 -121.52.87.38 -209.51.161.58 -80.72.146.2 -195.245.76.6 -149.156.64.210 -195.74.128.6 -81.15.197.10 -213.0.77.5 -212.89.130.180 -91.194.112.10 -203.146.237.222 -1.2.4.8 -200.118.2.88 -213.131.178.10 -203.63.8.27 -62.168.59.67 -200.175.3.232 -205.151.222.250 -213.115.244.69 -81.200.80.11 -195.206.7.98 -213.201.230.20 -63.146.122.11 -188.94.19.10 -114.114.114.119 -203.189.89.29 -190.9.57.2 -193.52.218.19 -62.183.50.230 -129.7.1.6 -202.248.37.74 -141.211.125.15 -91.195.202.131 -146.94.1.3 -35.8.2.41 -206.13.29.12 -63.218.44.186 -83.242.139.11 -217.117.111.1 -66.250.7.154 -213.157.176.3 -38.98.10.132 -84.21.31.230 -213.144.3.210 -89.140.140.8 -195.67.27.18 -200.62.64.1 -212.57.190.166 -82.115.163.2 -207.91.130.4 -213.235.248.245 -67.90.152.122 -79.140.66.38 -208.67.220.220 -195.189.131.1 -212.30.96.211 -202.14.67.4 -205.134.162.209 -213.169.55.10 -217.169.242.2 -212.24.98.97 -209.55.0.110 -15.227.128.50 -159.90.200.8 -216.244.192.3 -212.16.72.254 -195.54.152.2 -147.29.10.6 -69.67.254.2 -110.170.117.15 -217.76.240.2 -202.43.178.244 -101.255.64.74 -85.185.6.35 -72.37.141.91 -129.219.13.81 -204.95.160.2 -103.9.124.89 -210.248.255.82 -205.151.222.251 -212.214.82.198 -82.212.67.100 -108.61.213.134 -213.55.96.166 -121.194.2.2 -93.188.152.3 -198.6.1.3 -64.215.98.148 -193.252.247.52 -164.124.101.82 -82.182.37.49 -212.37.208.3 -213.184.242.6 -212.236.250.4 -193.89.221.2 -194.39.185.10 -70.36.0.5 -91.189.0.5 -217.71.105.254 -203.238.227.100 -203.109.129.68 -115.68.45.3 -193.109.4.5 -134.60.1.111 -78.143.192.10 -212.97.32.2 -212.57.190.166 -200.175.3.30 -193.27.80.34 -165.194.1.1 -194.25.0.60 -203.189.89.36 -216.66.22.2 -213.143.96.1 -213.184.0.42 -62.24.228.202 -91.214.72.34 -194.169.244.33 -192.116.16.26 -95.85.9.86 -91.188.0.5 -211.60.155.5 -209.145.176.20 -210.131.113.123 -217.113.48.1 -131.191.7.12 -64.105.163.106 -203.189.89.82 -69.7.192.2 -110.76.151.254 -212.9.160.1 -216.184.96.5 -61.63.0.66 -103.20.188.35 -195.234.101.234 -62.231.76.49 -208.72.120.204 -209.213.64.2 -213.211.50.2 -83.137.41.9 -195.113.144.194 -66.163.0.173 -109.69.8.34 -202.180.160.1 -216.81.128.132 -103.9.124.145 -92.43.224.1 -63.105.204.164 -212.96.1.70 -213.157.196.130 -81.173.113.30 -216.185.64.6 -212.26.6.11 -64.79.224.3 -62.243.190.9 -194.1.154.37 -193.186.162.3 -212.66.0.1 -195.175.39.39 -198.6.1.5 -62.77.85.100 -178.212.102.76 -217.151.0.50 -212.53.35.20 -101.255.64.62 -203.189.88.148 -213.157.0.193 -217.30.50.100 -178.151.86.169 -193.33.114.2 -193.228.86.5 -195.170.55.1 -148.160.20.195 -194.132.119.151 -64.181.43.34 -203.133.1.8 -83.233.78.163 -62.76.76.62 -64.105.202.138 -217.197.84.69 -212.34.194.211 -202.91.8.219 -122.0.0.13 -216.17.128.2 -195.166.192.1 -200.95.144.4 -202.116.128.1 -193.255.146.53 -202.65.159.4 -216.47.160.13 -117.102.224.26 -64.85.177.11 -168.88.66.6 -195.234.101.234 -83.177.163.51 -84.45.85.23 -101.255.64.114 -198.60.22.2 -66.165.173.235 -50.9.119.3 -195.177.240.3 -194.169.205.1 -151.236.6.156 -194.28.223.2 -195.158.239.4 -178.161.146.10 -64.94.1.33 -216.81.96.68 -63.251.161.33 -199.44.194.2 -159.90.200.7 -217.18.206.22 -101.255.64.227 -217.77.223.114 -122.155.167.8 -194.246.126.68 -93.91.146.150 -205.211.206.141 -82.99.212.18 -80.66.0.30 -212.37.208.4 -203.189.89.209 -209.252.33.101 -212.85.128.2 -196.29.40.3 -61.31.233.1 -213.157.0.194 -203.115.225.25 -195.140.236.250 -62.243.190.7 -193.232.69.22 -87.204.12.134 -209.183.48.21 -85.185.144.136 -206.126.32.101 -217.149.17.1 -111.223.252.193 -200.85.0.105 -194.145.147.195 -194.226.48.12 -216.186.27.15 -216.21.128.22 -77.241.112.23 -89.146.204.5 -207.190.94.129 -211.78.130.10 -210.23.64.1 -95.86.129.42 -200.85.44.70 -83.170.69.2 -193.231.173.2 -193.142.218.3 -157.157.90.193 -213.88.195.147 -83.97.97.3 -194.150.168.168 -212.42.165.37 -217.168.40.198 -66.216.18.222 -194.141.45.4 -198.82.247.34 -216.254.141.2 -213.241.193.250 -202.130.97.65 -193.33.236.1 -42.62.176.38 -195.186.4.110 -69.88.0.17 -69.26.129.2 -212.76.68.200 -210.23.129.34 -198.6.1.195 -202.203.192.33 -66.118.80.5 -213.233.161.69 -206.13.31.12 -84.241.98.36 -218.232.110.36 -67.17.215.132 -193.169.32.1 -78.38.253.138 -177.19.48.144 -188.114.194.2 -209.0.205.50 -139.130.4.4 -80.254.79.157 -202.46.1.2 -195.216.64.144 -201.163.145.101 -212.36.24.3 -210.29.96.33 -89.107.210.172 -194.113.160.68 -195.189.130.1 -213.178.66.111 -62.148.228.2 -216.47.160.12 -195.5.125.3 -186.107.119.118 -209.145.150.10 -209.195.95.95 -187.115.53.162 -62.243.190.8 -77.59.224.11 -91.189.0.2 -93.191.32.131 -62.3.32.17 -209.244.0.4 -212.31.253.69 -62.122.184.81 -213.144.108.117 -80.84.72.20 -208.112.89.187 -217.24.112.2 -206.51.143.55 -213.128.194.2 -212.118.241.1 -81.189.212.129 -81.222.80.2 -165.21.83.88 -87.105.250.3 -212.87.29.6 -68.179.203.94 -213.144.3.210 -180.211.129.42 -200.49.160.35 -38.119.98.220 -104.45.88.179 -219.96.224.90 -193.252.247.52 -82.145.163.1 -93.157.14.65 -212.181.124.8 -154.15.245.2 -200.35.174.126 -193.43.17.4 -204.174.120.45 -212.19.128.4 -203.130.2.3 -117.102.224.118 -213.152.142.12 -217.174.252.116 -202.43.176.14 -89.235.9.9 -194.20.0.24 -213.171.220.209 -203.130.2.4 -91.207.164.4 -84.200.69.80 -195.128.252.4 -119.160.208.252 -212.31.32.131 -204.119.0.2 -114.114.114.114 -62.58.3.11 -209.191.129.65 -202.141.224.34 -80.74.253.18 -212.18.15.3 -67.214.64.6 -193.43.108.3 -208.79.56.204 -208.70.22.22 -218.49.29.140 -195.189.72.2 -88.147.158.1 -66.9.182.1 -212.98.160.65 -213.88.151.150 -195.68.193.10 -203.112.2.5 -58.97.113.158 -203.119.36.106 -63.171.232.38 -194.52.202.98 -212.94.162.33 -195.137.189.203 -199.5.47.164 -114.114.115.115 -83.166.8.18 -202.14.67.14 -82.144.181.1 -195.149.104.186 -85.174.190.2 -212.58.111.1 -195.228.254.165 -205.152.37.23 -194.117.245.2 -91.98.110.15 -213.0.77.8 -212.122.224.10 -194.152.241.2 -85.158.50.50 -64.91.92.22 -202.43.178.245 -85.233.82.86 -210.44.112.66 -200.49.160.31 -217.8.180.98 -208.67.222.222 -217.159.0.17 -69.60.160.203 -207.241.160.34 -94.142.161.73 -151.164.1.8 -216.17.128.1 -217.15.17.2 -212.91.184.2 -63.251.161.1 -220.227.60.12 -202.120.111.3 -195.14.50.21 -209.87.64.70 -195.178.60.2 -41.211.233.10 -217.69.160.18 -217.64.163.1 -208.69.84.9 -81.17.66.14 -209.90.160.220 -200.175.3.68 -213.244.72.31 -95.128.246.2 -66.92.64.2 -217.22.209.254 -193.26.6.130 -200.66.96.1 -83.242.140.10 -153.19.1.254 -8.3.48.20 -152.99.78.136 -79.141.81.250 -206.165.6.11 -148.243.65.16 -213.159.193.54 -195.153.19.10 -8.8.4.4 -188.227.48.254 -80.79.179.2 -203.189.89.15 -203.90.78.65 -217.107.10.254 -218.49.29.141 -195.96.208.1 -207.248.224.71 -89.191.149.2 -213.151.109.1 -216.52.126.1 -212.66.129.98 -77.88.8.2 -8.8.8.8 -203.189.89.134 -61.199.193.162 -93.186.161.211 -83.143.8.220 -194.54.66.242 -82.202.131.1 -194.158.206.206 -62.16.86.100 -195.137.162.149 -193.89.221.124 -219.163.55.74 -62.37.228.20 -193.151.93.3 -193.22.119.195 -151.236.29.92 -217.30.49.100 -217.28.113.13 -78.159.224.224 -122.155.12.215 -212.66.1.1 -212.116.76.76 -64.13.115.12 -62.140.239.1 -82.96.193.12 -212.9.64.12 -213.183.57.55 -193.243.128.91 -212.51.17.1 -62.141.38.230 -206.248.95.194 -194.226.211.11 -74.82.46.6 -213.184.16.1 -216.66.80.98 -158.43.192.1 -195.244.25.3 -213.136.40.32 -217.28.98.62 -212.230.255.1 -213.135.67.1 -212.118.0.2 -141.211.125.17 -195.214.240.136 -202.83.20.101 -193.111.34.18 -217.149.155.180 -142.77.2.85 -130.180.228.2 -89.233.250.137 -106.51.255.133 -91.194.211.134 -195.42.215.17 -64.105.199.76 -202.91.8.234 -193.45.139.20 -213.128.216.115 -217.66.226.8 -211.67.112.1 -129.219.17.5 -217.72.1.2 -213.251.133.164 -202.30.143.11 -213.183.65.31 -208.3.14.1 -207.17.190.5 -94.25.63.2 -217.79.225.8 -83.234.220.253 -198.6.1.1 -87.204.12.130 -200.88.127.23 -81.209.202.46 -210.2.4.8 -195.35.110.4 -213.141.72.250 -24.154.1.5 -194.145.147.194 -95.215.150.15 -205.134.162.209 -83.170.64.2 -81.28.128.34 -202.86.8.100 -207.44.226.173 -89.248.162.3 -82.216.111.122 -187.115.52.91 -200.194.67.214 -203.109.129.67 -194.50.10.2 -88.82.105.19 -213.140.34.65 -200.123.192.244 -141.50.161.12 -217.31.160.30 -192.190.173.40 -82.96.81.10 -37.235.1.174 -187.115.52.78 -207.17.190.7 -209.172.128.2 -219.252.48.67 -62.149.132.2 -91.203.188.1 -82.209.190.82 -194.8.53.1 -198.6.1.4 -200.175.3.69 -212.40.5.51 -195.26.96.2 -203.115.81.38 -8.3.48.30 -194.158.206.205 -212.87.132.53 -194.169.244.34 -63.251.129.33 -69.16.169.11 -31.47.189.170 -190.11.32.42 -202.130.97.65 -203.189.88.211 -193.226.61.1 -204.117.214.10 -83.69.77.2 -81.199.3.7 -35.8.2.45 -84.55.62.75 -213.158.72.1 -94.247.200.3 -210.94.0.7 -89.160.27.232 -120.50.44.141 -201.217.16.89 -196.41.225.11 -62.196.2.70 -203.253.64.1 -148.233.151.8 -194.141.44.130 -62.8.96.38 -202.51.96.5 -46.246.94.136 -91.194.178.5 -212.112.39.25 -203.210.142.132 -213.73.14.227 -209.130.136.2 -149.250.222.22 -212.69.161.100 -91.202.12.10 -213.129.120.3 -88.80.64.200 -220.233.0.1 -216.184.96.6 -212.15.128.1 -211.41.128.71 -194.14.0.6 -212.94.34.34 -216.229.0.25 -216.143.135.11 -216.143.135.12 -203.189.89.1 -195.161.115.3 -195.166.192.8 -8.15.12.5 -202.62.124.238 -212.40.5.50 -216.254.95.2 -62.58.3.11 -217.219.236.8 -80.190.248.146 -89.186.66.6 -194.54.128.232 -194.145.240.6 -62.149.33.134 -69.28.148.102 -79.141.83.250 -203.41.44.20 -208.38.1.15 -82.76.253.115 -91.196.8.2 -205.152.144.23 -200.9.115.2 -62.33.47.253 -188.114.193.254 -202.248.0.34 -91.207.40.2 -210.131.113.123 -202.73.36.135 -142.47.133.81 -204.116.57.2 -185.46.7.100 -217.115.16.2 -66.92.159.2 -217.31.204.130 -185.16.40.143 -220.128.173.228 -212.51.17.1 -81.23.144.250 -193.28.97.130 -89.107.16.2 -88.82.84.129 -91.98.132.60 -194.169.239.10 -42.62.178.65 -199.166.6.2 -62.3.32.16 -193.33.200.22 -90.189.109.2 -213.33.82.1 -199.103.16.5 -141.85.128.1 -209.216.160.2 -110.76.151.1 -193.230.161.4 -213.253.137.17 -222.124.249.115 -81.24.128.146 -194.18.231.5 -5.144.19.8 -62.20.17.205 -194.98.65.165 -194.102.106.1 -4.2.2.6 -101.255.64.134 -158.43.128.1 -212.58.3.2 -89.233.43.71 -193.16.209.2 -77.88.8.8 -62.73.100.4 -81.189.214.162 -158.43.128.72 -115.68.100.103 -69.146.17.3 -200.85.39.206 -64.91.92.21 -200.40.230.36 -90.183.74.1 -84.1.240.34 -83.243.39.61 -202.248.20.133 -81.27.135.50 -195.84.194.3 -195.182.110.132 -203.189.88.213 -80.190.200.10 -207.178.128.21 -212.94.162.33 -195.170.97.254 -77.247.176.114 -82.145.160.140 -152.99.1.10 -212.192.128.3 -142.77.2.36 -42.62.176.30 -195.225.36.16 -84.241.100.31 -217.78.80.74 -166.70.25.18 -216.21.129.22 -205.171.2.65 -195.46.48.22 -147.235.250.2 -130.85.1.3 -91.203.177.4 -178.151.86.169 -201.217.19.225 -204.119.0.2 -88.255.242.6 -91.135.110.132 -190.22.34.170 -213.244.5.67 -117.102.224.154 -91.149.108.10 -194.246.127.11 -194.67.74.2 -64.119.60.9 -216.184.96.4 -216.52.169.1 -83.136.56.52 -194.239.164.25 -216.116.96.3 -84.32.80.20 -216.66.38.58 -206.253.194.65 -61.31.1.1 -217.21.96.1 -91.198.154.133 -212.5.218.3 -78.31.96.2 -194.225.128.22 -76.73.18.50 -129.250.35.251 -161.53.128.16 -203.189.88.54 -89.208.10.10 -87.104.254.39 -66.250.192.11 -218.223.32.1 -213.178.66.2 -82.199.102.38 -193.22.110.251 -212.19.149.226 -213.144.108.117 -199.249.18.1 -69.67.97.18 -8.2.208.2 -212.96.130.140 -217.199.217.200 -195.67.127.137 -212.203.33.12 -64.91.3.46 -213.178.0.33 -121.52.87.56 -216.116.96.2 -212.59.199.6 -216.185.192.1 -110.76.151.241 -203.156.104.21 -61.56.211.185 -194.72.9.61 -209.0.205.11 -93.158.117.138 -84.200.70.40 -101.255.64.154 -212.85.112.32 -211.78.130.11 -81.23.144.250 -84.237.112.3 -83.137.193.83 -193.111.200.191 -207.230.202.28 -80.94.48.254 -66.242.160.5 -79.137.227.122 -217.116.53.13 -200.58.161.25 -66.203.72.10 -212.51.16.1 -93.88.151.138 -200.12.63.10 -203.242.200.15 -203.189.88.152 -64.132.61.131 -81.92.96.22 -139.134.5.51 -89.223.7.242 -95.158.129.2 -62.133.163.171 -202.44.55.193 -91.144.248.227 -81.17.72.70 -193.110.157.2 -203.189.88.54 -193.230.161.3 -64.72.224.34 -85.115.224.18 -193.77.33.18 -203.189.88.214 -212.214.82.194 -216.66.80.30 -194.120.55.3 -81.199.48.244 -212.66.1.1 -83.97.97.2 -202.180.64.2 -67.214.159.198 -213.157.0.194 -77.241.24.5 -195.190.17.6 -217.77.176.10 -72.11.150.74 -66.252.170.3 -94.155.91.8 -200.175.3.59 -194.12.224.34 -213.147.64.1 -84.241.98.37 -207.178.128.20 -202.180.64.9 -187.73.241.67 -195.67.15.102 -78.133.155.218 -194.183.88.41 -212.9.160.1 -208.48.253.106 -193.242.114.129 -85.219.142.1 -101.255.64.42 -82.96.86.20 -200.62.64.65 -220.68.64.1 -216.52.254.33 -66.81.0.252 -193.151.32.40 -63.251.62.1 -203.133.1.7 -202.148.202.4 -193.95.93.243 -212.82.226.212 -212.58.3.7 -62.20.57.226 -216.58.97.20 -170.56.58.53 -193.201.185.3 -62.177.42.174 -212.69.161.100 -64.212.106.85 -83.243.39.59 -62.233.128.17 -204.52.135.2 -217.78.80.70 -213.164.38.66 -62.129.252.215 -50.116.23.211 -80.94.32.240 -200.85.35.158 -200.175.3.58 -129.250.35.250 -91.220.187.3 -202.136.162.11 -115.85.69.162 -212.11.191.72 -213.172.33.34 -213.30.253.65 -202.148.202.3 -213.27.209.8 -198.6.1.2 -160.44.1.4 -216.237.221.42 -194.88.202.11 -212.19.96.2 -212.233.128.1 -141.211.144.15 -93.99.200.1 -62.20.76.35 -201.217.17.74 -101.255.64.90 -80.64.32.2 -114.130.11.66 -122.255.96.132 -203.119.8.106 -69.7.192.1 -216.52.129.1 -194.6.216.5 -203.250.129.214 -103.9.124.154 -193.231.80.7 -85.249.45.253 -208.122.23.23 -210.80.58.66 -196.207.15.42 -217.69.169.25 -200.113.185.227 -63.238.52.1 -64.119.80.100 -204.9.123.122 -206.124.64.1 -193.232.65.2 -193.111.238.5 -209.161.175.30 -166.102.165.32 -212.94.32.32 -129.7.1.1 -160.220.137.2 -95.173.193.3 -139.0.27.186 -66.119.93.10 -103.22.248.62 -206.248.79.244 -121.52.87.128 -91.143.20.6 -82.99.211.195 -66.92.224.2 -193.254.232.1 -216.131.95.20 -115.85.69.162 -83.143.154.234 -206.124.1.254 -101.255.64.241 -207.164.234.193 -222.124.8.50 -147.29.37.19 -199.2.252.10 -194.152.248.42 -83.69.77.6 -174.34.129.34 -207.130.95.40 -193.175.51.10 -87.197.40.58 -193.6.10.1 -209.63.0.18 -212.50.131.153 -80.94.52.254 -62.95.15.107 -80.78.162.2 -67.17.215.133 -213.139.190.3 -213.129.120.6 -217.168.144.127 -66.51.206.100 -193.200.68.230 -217.196.1.5 -212.71.98.250 -64.13.48.12 -170.51.255.100 -194.242.50.66 -216.235.1.3 -173.44.32.2 -128.199.248.105 -195.167.98.3 -119.252.20.75 -212.111.28.5 -217.21.48.1 -62.91.2.20 -206.74.254.2 -81.199.3.7 -165.87.13.129 -194.8.53.1 -64.140.243.112 -147.235.251.3 -212.82.225.7 -187.115.52.83 -101.255.64.150 -216.254.141.13 -213.27.209.53 -79.141.82.250 -194.213.193.5 -148.233.151.6 -200.85.60.210 -193.231.236.25 -62.177.42.174 -190.11.32.199 -207.179.3.25 -202.130.97.66 -199.101.98.178 -91.185.2.10 -217.18.90.105 -195.182.224.11 -69.28.97.4 -209.97.224.3 -94.124.19.16 -194.169.235.2 -87.229.99.1 -88.80.64.201 -62.181.119.131 -147.29.10.55 -194.73.96.50 -84.32.80.20 -216.146.35.230 -190.146.118.41 -110.76.151.17 -58.96.3.34 -193.16.255.2 -61.19.252.238 -208.92.9.21 -85.88.19.11 -83.241.175.98 -203.146.237.237 -64.91.89.2 -194.141.12.1 -194.54.181.90 -193.41.252.146 -201.131.4.9 -62.33.183.254 -119.160.208.251 -217.18.80.105 -202.86.216.1 -62.109.182.2 -64.105.189.26 -72.52.104.74 -81.92.97.12 -87.255.68.242 -134.48.1.32 -216.218.226.238 -85.214.132.203 -62.97.84.4 -210.220.163.82 -103.239.165.34 -213.218.117.85 -203.248.252.2 -65.183.98.90 -168.95.1.1 -209.213.223.18 -200.88.127.22 -217.32.105.66 -62.20.15.234 -149.211.153.51 -193.111.144.145 -203.89.226.26 -203.80.96.10 -193.78.240.12 -109.69.8.51 -78.142.133.43 -212.94.162.1 -77.240.144.164 -213.234.128.211 -91.209.108.17 -64.207.64.5 -213.137.73.254 -205.172.19.79 -83.219.241.2 -88.82.105.18 -209.55.1.220 -193.58.251.251 -206.253.33.130 -141.56.31.3 -161.53.129.139 -158.39.46.248 -122.210.229.161 -203.253.31.1 -195.60.70.5 -202.38.128.58 -62.134.11.4 -207.178.128.21 -195.166.13.4 -192.43.161.22 -200.69.193.2 -203.153.214.14 -81.24.128.146 -208.78.24.238 -211.172.241.54 -185.46.7.110 -198.188.2.69 -66.93.87.2 -194.33.15.3 -193.34.129.253 -91.212.56.5 -81.90.168.3 -216.198.139.68 -193.231.249.1 -195.70.237.42 -65.74.130.6 -91.210.24.22 -65.163.107.11 -202.181.224.2 -195.70.248.1 -208.122.23.22 -210.227.119.194 -79.99.224.24 -168.243.165.225 -202.83.30.5 -212.24.98.98 -194.176.190.2 -77.59.224.10 -80.190.200.55 -91.135.230.231 -212.209.194.170 -65.220.16.14 -66.207.160.111 -66.28.0.45 -216.185.192.2 -216.54.201.11 -68.179.203.94 -216.52.94.1 -193.33.220.3 -194.145.198.226 -212.14.253.242 -62.108.161.200 -66.81.1.252 -217.65.192.1 -122.155.167.70 -195.170.96.2 -198.6.1.146 -168.213.3.10 -64.85.177.10 -66.165.177.69 -85.94.224.1 -193.111.144.161 -64.61.99.2 -85.235.199.199 -193.33.174.3 -149.156.64.210 -115.68.62.222 -119.160.208.252 -216.58.97.21 -194.158.230.53 -202.138.120.6 -218.192.240.2 -152.99.200.6 -202.152.162.66 -173.241.133.178 -194.132.32.32 -193.231.238.1 -195.182.192.10 -212.66.160.2 -89.255.99.131 -212.85.128.2 -65.74.130.5 -63.251.62.33 -200.56.224.11 -103.3.76.82 -212.108.200.77 -194.250.223.1 -194.172.160.4 -195.140.236.253 -209.142.182.250 -106.186.17.181 -58.150.55.34 -103.9.124.154 -206.123.64.245 -87.104.254.135 -64.13.131.34 -148.243.65.17 -103.226.55.129 -81.180.201.99 -50.21.174.18 -216.175.203.51 -66.163.0.161 -66.146.0.1 -216.162.32.20 -89.208.120.10 -202.43.176.13 -77.241.25.3 -212.40.0.10 -206.53.177.3 -75.94.255.12 -93.90.82.50 -64.187.29.134 -217.144.144.211 -195.46.48.21 -4.2.2.1 -62.165.33.250 -212.87.130.92 -205.151.69.200 -198.6.1.142 -66.63.192.2 -82.198.129.146 -209.142.152.253 -103.9.124.90 -213.211.50.1 -212.31.32.130 -64.105.179.138 -190.248.153.98 -94.247.200.3 -206.13.30.12 -92.42.200.66 -212.73.65.40 -64.135.2.250 -69.28.97.4 -195.110.17.40 -158.43.240.3 -82.96.40.83 -164.2.255.241 -206.124.0.254 -216.52.94.33 -200.221.11.101 -216.52.161.33 -198.100.146.51 -203.189.88.133 -193.7.169.9 -212.118.241.33 -200.175.0.91 -164.33.1.4 -89.160.63.190 -212.41.4.1 -198.6.1.122 -65.39.139.53 -64.254.99.13 -64.132.94.250 -195.182.192.2 -81.7.200.80 -202.45.84.59 -212.118.241.33 -91.206.72.2 -206.252.187.110 -164.124.101.51 -38.112.17.138 -195.24.228.3 -195.221.20.10 -87.204.28.12 -217.198.161.1 -146.185.134.104 -193.142.115.131 -203.99.253.1 -81.18.242.100 -66.165.164.250 -103.3.213.210 -80.67.169.12 -193.17.213.10 -159.230.4.130 -203.189.88.156 -199.80.64.202 -212.230.255.129 -194.102.93.2 -93.88.148.138 -201.217.18.178 -77.109.138.45 -41.221.5.11 -203.189.88.212 -216.66.80.26 -12.127.16.67 -202.44.204.63 -203.189.88.11 -218.44.242.98 -85.94.224.2 -193.231.112.1 -195.110.16.40 -77.87.152.9 -94.155.90.7 -193.89.248.1 -207.91.5.32 -149.6.140.30 -208.66.232.66 -91.206.213.2 -213.157.176.2 -62.105.17.252 -213.23.108.129 -205.162.201.2 -193.28.100.200 -203.193.139.150 -212.102.225.2 -220.233.0.3 -217.117.0.38 -194.6.240.1 -173.241.133.189 -193.205.136.1 -4.2.2.4 -212.245.158.66 -193.16.48.66 -193.201.185.2 -212.1.118.3 -82.198.129.138 -193.239.60.19 -212.53.34.1 -209.87.79.232 -213.88.195.146 -216.52.41.1 -78.159.232.232 -89.255.96.3 -195.251.119.23 -82.199.32.36 -165.166.142.42 -38.112.17.142 -62.91.2.20 -142.46.1.130 -81.12.49.100 -4.79.132.219 -91.197.164.11 -79.132.192.2 -203.189.88.11 -203.115.130.74 -202.62.224.2 -217.18.206.12 -206.124.64.253 -195.198.214.72 -69.28.239.8 -84.32.112.202 -83.166.8.18 -195.153.19.5 -203.189.89.241 -85.172.0.250 -77.239.96.2 -59.12.239.70 -203.189.89.131 -212.84.181.99 -82.96.65.2 -216.52.190.33 -202.174.131.19 -213.157.196.132 -37.221.170.105 -190.249.175.122 -64.79.224.27 -83.240.154.200 -216.147.131.34 -200.85.61.90 -216.106.184.6 -204.97.212.10 -194.146.136.1 -194.145.198.6 -81.180.206.137 -218.102.23.228 -194.158.230.54 -85.132.32.41 -212.28.34.90 -101.255.64.82 -67.214.64.27 -211.172.208.2 -81.92.226.181 -210.34.0.18 -163.152.1.1 -91.200.113.1 -195.177.223.3 -217.170.1.1 -77.88.8.88 -62.77.85.98 -67.100.88.27 -103.20.188.83 -198.6.1.6 -213.172.33.35 -206.80.254.4 -193.226.128.129 -62.108.161.161 -217.196.1.6 -66.112.235.200 -194.105.32.2 -122.155.13.155 -83.228.65.52 -66.118.80.4 -209.142.136.85 -74.222.30.2 -193.34.129.253 -168.243.165.226 -164.115.2.132 -80.80.111.254 -195.198.127.20 -188.34.0.4 -62.119.70.3 -194.242.50.65 -195.88.84.100 -217.65.100.7 -193.252.247.53 -82.96.193.10 -195.234.230.67 -218.232.110.37 -213.73.91.35 -119.18.159.222 -200.57.7.61 -64.105.199.74 -216.81.128.132 -195.206.96.47 -213.33.82.2 -93.188.152.3 -89.249.224.1 -195.66.89.4 -216.138.119.6 -89.19.193.1 -200.221.11.100 -91.188.0.35 -202.86.216.2 -199.249.19.2 -194.25.15.11 -204.101.45.5 -217.72.168.34 -78.47.34.12 -83.142.192.2 -193.204.192.2 -195.128.252.7 -195.12.4.247 -61.208.115.242 -194.187.164.20 -101.255.64.138 -91.98.128.112 -122.155.12.91 -212.49.128.65 -42.62.176.150 -213.88.195.148 -194.164.181.2 -193.95.93.77 -190.186.50.31 -142.46.128.130 -69.28.136.102 -194.113.160.68 -195.112.96.34 -203.153.214.26 -194.45.12.2 -101.255.64.58 -194.88.203.6 -212.5.220.252 -62.56.230.100 -194.237.202.250 -210.34.48.34 -195.20.193.11 -213.157.196.131 -203.198.7.66 -202.138.120.87 -62.22.102.5 -221.139.13.130 -69.25.1.33 -195.186.1.110 -212.233.128.2 -93.91.224.2 -80.149.86.20 -37.235.1.177 -194.2.0.20 -195.66.68.2 -209.68.1.11 -91.203.188.1 -216.54.2.11 -207.91.250.34 -203.189.89.65 -203.153.214.14 -80.88.171.16 -208.90.237.9 -216.81.96.67 -89.107.129.15 -194.1.148.1 -209.197.128.2 -77.246.144.5 -211.78.130.11 -192.43.161.22 -83.243.39.59 -62.40.32.34 -195.16.73.1 -166.70.25.18 -213.157.0.193 -62.77.94.72 -77.41.229.2 -203.112.2.4 -62.94.0.41 -81.21.112.130 -88.131.89.37 -62.36.225.150 -207.248.224.72 -200.95.144.3 -62.149.128.2 -216.218.221.6 -64.94.33.33 -101.203.168.123 -212.58.3.8 -81.200.5.165 -212.15.86.12 -115.68.45.3 -103.3.46.105 -216.147.131.33 -203.124.230.100 -61.8.0.113 -195.129.12.114 -205.236.148.130 -209.51.161.14 -12.127.17.72 -203.189.89.210 -164.115.2.132 -209.142.152.254 -194.102.44.130 -94.199.201.199 -217.115.16.3 -77.109.139.29 -202.43.160.50 -90.183.74.2 -164.124.101.47 -88.255.96.196 -203.112.194.243 -86.59.41.180 -82.141.136.2 -194.67.74.3 -115.68.62.210 -203.189.89.117 -91.192.56.2 -193.102.59.190 -216.136.95.2 -89.207.72.138 -208.196.63.2 -111.223.252.161 -193.16.208.114 -203.2.193.67 -207.230.192.254 -160.7.240.20 -195.22.192.252 -83.137.41.8 -194.187.148.1 -72.11.150.10 -60.32.112.42 -216.52.41.33 -212.54.160.7 -193.41.10.1 -202.125.132.154 -65.107.59.67 -194.73.96.62 -203.196.0.6 -69.28.104.5 -207.15.68.36 -66.80.130.18 -122.155.3.119 -209.244.0.53 -212.230.255.129 -212.41.3.147 -165.194.1.1 -216.37.1.19 -122.155.12.41 -213.253.136.17 -80.66.1.42 -195.186.1.111 -69.54.70.15 -198.32.2.10 -212.38.95.254 -187.110.170.74 -217.77.176.11 -201.131.4.5 -193.43.108.62 -211.61.13.227 -194.116.170.66 -5.144.12.202 -194.30.163.5 -213.178.66.112 -195.137.246.17 -78.143.192.20 -207.164.234.129 -95.215.149.5 -94.236.199.8 -82.209.213.60 -61.60.224.5 -94.23.222.19 -206.253.33.131 -211.61.13.126 -202.133.99.11 -213.253.193.2 -194.149.156.140 -193.78.240.12 -58.68.121.230 -210.180.98.69 -216.52.65.1 -216.27.175.2 -193.230.230.1 -211.41.128.70 -211.78.130.10 -62.37.225.56 -62.165.32.250 -211.161.46.84 -83.143.12.246 -220.110.92.202 -4.2.2.2 -209.216.160.131 -193.138.78.117 -209.143.22.182 -203.89.226.24 -217.29.16.250 -66.182.208.5 -201.217.51.45 -217.173.198.3 -147.29.37.20 -69.24.112.10 -88.82.84.129 -195.243.214.4 -195.54.152.3 -193.171.4.60 -81.20.240.34 -69.24.112.11 -93.88.16.66 -221.186.85.74 -80.254.77.39 -193.228.86.5 -194.25.0.52 -91.98.234.4 -89.187.240.60 -129.219.17.200 -194.77.8.1 -62.122.208.68 -74.84.4.139 -160.220.137.2 -203.189.88.151 -193.231.236.30 -63.238.52.2 -87.250.77.204 -91.98.30.222 -69.67.97.18 -168.215.165.186 -205.152.132.23 -119.252.20.75 -208.59.89.20 -208.54.220.20 -66.7.160.122 -61.63.0.66 -64.94.1.1 -85.114.105.3 -146.66.19.238 -217.77.223.114 -200.53.250.1 -66.232.139.10 -193.86.86.2 -121.52.206.130 -216.52.254.1 -115.68.100.102 -70.36.0.6 -212.65.160.43 -193.42.81.68 -212.112.39.22 -87.230.13.136 -194.126.181.47 -64.212.106.84 -193.47.72.17 -24.248.137.39 -83.149.244.194 -91.214.72.33 -111.223.252.225 -89.107.210.171 -141.1.1.1 -62.33.203.33 -194.218.25.250 -80.73.1.1 -23.226.230.72 -195.178.123.130 -165.194.128.1 -213.128.194.2 -95.158.128.2 -212.203.32.11 -208.71.147.74 -69.28.239.9 -210.80.58.3 -203.77.161.12 -202.28.162.1 -62.128.1.42 -46.163.72.207 -67.214.159.199 -202.62.31.18 -207.248.57.10 -24.154.1.4 -65.210.29.34 -192.76.144.66 -217.64.167.1 -14.139.223.100 -41.221.6.38 -66.218.245.13 -192.172.250.8 -194.44.211.194 -195.251.123.232 -213.0.76.5 -117.102.224.230 -212.4.96.22 -89.187.240.59 -64.135.1.20 -189.90.16.20 -201.161.6.46 -42.62.176.74 -203.242.200.5 -64.81.159.2 -208.67.220.222 -195.186.4.111 -80.94.32.240 -213.8.145.133 -194.187.100.2 -212.9.161.2 -194.126.130.6 -209.161.175.29 -66.203.66.203 -158.43.240.4 -91.239.100.100 -202.0.107.125 -211.78.130.3 -216.52.97.33 -212.67.131.4 -211.175.82.66 -203.124.230.21 -80.64.32.2 -193.230.183.201 -217.151.0.195 -208.67.222.220 -124.107.135.126 -103.20.188.82 -61.19.130.42 -64.119.60.5 -149.250.222.21 -195.69.65.98 -210.104.1.3 -213.235.248.228 -194.153.232.17 -164.124.101.2 -194.149.146.2 -83.143.12.249 -66.119.93.4 -62.37.225.57 -217.20.96.100 -91.211.16.6 -122.0.0.12 -64.91.3.60 -81.25.152.2 -205.236.148.131 -142.103.1.1 -193.178.124.1 -168.215.210.50 -80.74.160.11 -211.237.65.31 -173.241.133.190 -219.250.36.130 -203.189.88.10 -211.237.65.21 -216.131.94.5 -216.52.1.1 -103.20.184.62 -83.142.9.30 -195.145.22.37 -207.15.68.164 -200.57.2.108 -216.52.1.33 -217.27.240.20 -216.194.28.33 -213.241.193.250 -77.72.17.17 -220.233.0.4 -205.172.19.193 -85.119.72.2 -217.107.11.35 -195.114.173.153 -121.152.231.196 -194.149.133.11 -62.29.160.228 -206.80.254.68 -216.181.31.11 -208.86.117.40 -211.63.64.11 -202.180.64.9 -195.66.156.26 -189.38.95.96 -62.231.100.14 -208.48.253.106 -81.180.201.98 -219.252.2.100 -217.14.128.50 -212.216.172.222 -195.149.138.3 -193.58.204.59 -213.235.248.228 -213.16.104.61 -195.27.1.1 -50.116.28.138 -211.115.194.2 -217.144.6.6 -194.54.148.129 -212.85.32.3 -164.124.107.9 -61.70.87.96 -203.176.144.20 -168.213.3.11 -206.104.144.62 -85.88.19.10 -212.59.199.2 -111.223.252.27 -194.105.156.2 -81.90.168.3 -193.46.84.2 -207.15.68.36 -195.146.81.130 -82.216.111.121 -151.11.85.5 -217.20.82.4 -216.22.81.60 -62.94.0.42 -208.116.30.21 -94.247.200.2 -203.239.131.1 -211.115.194.3 -83.228.65.52 -193.95.93.77 -216.106.1.2 -72.52.104.74 -212.110.122.132 -64.105.97.90 -62.133.163.171 -204.9.122.102 -66.165.183.87 -194.20.8.1 -193.15.251.65 -62.128.1.53 -193.148.29.100 -212.85.32.2 -203.124.250.70 -72.46.0.2 -209.142.136.220 -193.148.29.103 -203.115.71.66 -217.156.106.1 -114.114.115.119 -213.159.0.55 -212.62.98.10 -193.7.168.1 -209.206.136.8 -217.148.122.40 -66.9.5.15 -42.62.176.125 -193.111.212.5 -196.29.40.4 -67.214.64.7 -63.171.232.39 -63.105.204.164 -212.73.209.34 -88.216.8.69 -80.78.208.2 -85.249.40.8 -203.113.11.37 -62.233.181.26 -187.115.53.163 -193.41.59.151 -202.62.120.4 -203.189.88.154 -139.175.55.244 -193.34.170.162 -210.204.251.22 -85.124.252.33 -213.158.72.44 -218.248.240.23 -89.186.66.7 -77.72.192.3 -77.73.104.3 -193.226.145.2 -64.56.129.2 -194.95.141.1 -77.72.178.77 -80.92.178.98 -63.246.63.142 -64.135.1.22 -213.211.50.2 -49.0.124.46 -213.27.209.55 -82.115.23.3 -216.52.65.33 -87.241.63.4 -178.254.21.113 -69.51.76.26 -195.138.160.3 -46.246.46.246 -81.27.133.50 -61.72.225.1 -65.203.109.2 -203.153.41.28 -194.183.88.40 -85.132.32.42 -192.121.170.170 -209.251.33.2 -74.207.242.213 -194.126.159.20 -193.189.114.254 -194.250.223.2 -103.20.188.82 -89.185.75.244 -213.133.224.2 -213.159.0.70 -190.41.153.24 -212.214.229.170 -66.218.44.5 -195.7.64.3 -195.18.161.132 -207.249.163.155 -203.176.144.12 -216.244.192.32 -213.146.65.11 -83.151.112.193 -66.92.64.2 -93.157.233.3 -77.88.8.1 -195.67.15.73 -121.52.87.65 -194.20.8.4 -217.20.240.5 -82.212.67.101 -203.189.89.210 -217.24.113.214 -193.254.22.13 -62.129.252.252 -76.10.192.201 -193.101.111.10 -62.192.128.60 -193.43.181.62 -194.242.50.65 -64.105.172.26 -193.109.53.2 -37.19.5.135 -94.153.224.74 -91.199.139.1 -101.255.64.86 -165.87.201.244 -217.159.1.126 -62.116.30.200 -195.129.12.83 -221.151.200.206 -119.252.167.229 -168.126.63.1 -200.85.61.90 -117.102.224.190 -195.67.160.3 -212.73.154.2 -131.155.140.130 -216.218.221.6 -208.38.1.15 -66.28.0.45 -212.9.64.11 -63.251.129.33 -35.8.98.43 -221.156.218.31 -94.155.91.4 -203.113.25.71 -211.61.13.227 -195.13.38.3 -80.78.66.66 -193.22.119.22 -194.42.108.135 -193.67.79.39 -62.72.87.4 -80.93.177.182 -206.13.28.12 -8.5.244.5 -209.183.52.21 -35.8.2.42 -81.18.97.50 -178.212.102.76 -213.239.204.35 -212.98.160.50 -194.126.130.7 -200.123.192.251 -87.103.133.167 -196.2.45.101 -212.24.97.97 -173.241.133.172 -212.211.132.4 -85.119.74.2 -101.255.64.210 -64.91.92.21 -85.119.136.158 -212.96.128.140 -207.230.202.29 -193.2.64.45 -187.115.52.142 -137.82.1.1 -101.255.64.34 -194.1.185.122 -194.179.109.10 -217.28.96.190 -217.17.34.68 -87.106.220.85 -12.173.168.201 -217.198.160.130 -194.179.1.100 -89.140.186.3 -195.99.66.220 -165.21.100.88 -149.211.153.50 -81.189.121.68 -209.142.152.253 -195.2.195.1 -203.229.169.1 -66.28.0.61 -69.16.170.11 -81.95.128.218 -209.143.0.10 -193.27.192.98 -194.75.147.212 -217.148.0.17 -81.196.170.20 -168.188.1.1 \ No newline at end of file diff --git a/theHarvester/lib/source_catalog.py b/theHarvester/lib/source_catalog.py index 8f705206..4d484726 100644 --- a/theHarvester/lib/source_catalog.py +++ b/theHarvester/lib/source_catalog.py @@ -1,4 +1,4 @@ -from collections.abc import Iterable +from collections.abc import Iterable, Mapping from dataclasses import dataclass from enum import Enum, StrEnum, auto from typing import Final @@ -13,12 +13,25 @@ class ActivityClass(StrEnum): ACTION_ACTIVITIES: Final = { 'dns-brute': ActivityClass.DNS, 'dns-lookup': ActivityClass.DNS, + 'dns-recursive': ActivityClass.DNS, 'dns-resolve': ActivityClass.DNS, 'shodan': ActivityClass.PASSIVE, 'api-scan': ActivityClass.DIRECT, 'screenshot': ActivityClass.DIRECT, - 'take-over': ActivityClass.DIRECT, + 'takeover': ActivityClass.DIRECT, } +ACTION_REQUEST_FIELDS: Final = { + **{name: name.replace('-', '_') for name in ACTION_ACTIVITIES}, + 'dns-recursive': 'dns_recursive_depth', +} + + +def selected_action_names(request: Mapping[str, object]) -> tuple[str, ...]: + def selected(name: str) -> bool: + value = request.get(ACTION_REQUEST_FIELDS[name]) + return isinstance(value, (int, float)) and value > 0 if name == 'dns-recursive' else bool(value) + + return tuple(name for name in ACTION_ACTIVITIES if selected(name)) class ResultRoute(Enum): @@ -34,9 +47,7 @@ class ResultRoute(Enum): IPS = auto() ASNS = auto() PEOPLE = auto() - LINKS = auto() URLS = auto() - INTERESTING_URLS = auto() BREACHES = auto() @@ -46,9 +57,7 @@ _ROUTE_CAPABILITIES = { ResultRoute.IPS: 'ips', ResultRoute.ASNS: 'asns', ResultRoute.PEOPLE: 'people', - ResultRoute.LINKS: 'urls', ResultRoute.URLS: 'urls', - ResultRoute.INTERESTING_URLS: 'urls', ResultRoute.BREACHES: 'breaches', } RESULT_CAPABILITIES = frozenset(_ROUTE_CAPABILITIES.values()) @@ -80,10 +89,10 @@ def _spec( _SPECS = ( _spec('arquivo', ResultRoute.SUBDOMAINS), _spec('baidu', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS), - _spec('bevigil', ResultRoute.SUBDOMAINS, ResultRoute.INTERESTING_URLS), + _spec('bevigil', ResultRoute.SUBDOMAINS, ResultRoute.URLS), _spec('brave', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS), _spec('bufferoverun', ResultRoute.SUBDOMAINS, ResultRoute.IPS), - _spec('builtwith', ResultRoute.SUBDOMAINS, ResultRoute.INTERESTING_URLS), + _spec('builtwith', ResultRoute.SUBDOMAINS, ResultRoute.URLS), _spec('censys', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS), _spec('certspotter', ResultRoute.SUBDOMAINS), _spec('chaos', ResultRoute.SUBDOMAINS), @@ -111,7 +120,7 @@ _SPECS = ( _spec('hudsonrock', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS, ResultRoute.IPS), _spec('hunter', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS), _spec('hunterhow', ResultRoute.SUBDOMAINS), - _spec('intelx', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS, ResultRoute.INTERESTING_URLS), + _spec('intelx', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS, ResultRoute.URLS), _spec('leakix', ResultRoute.SUBDOMAINS), _spec('leaklookup', ResultRoute.EMAILS, ResultRoute.BREACHES), _spec('mojeek', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS), @@ -122,7 +131,7 @@ _SPECS = ( _spec('projectdiscovery', ResultRoute.SUBDOMAINS), _spec('rapiddns', ResultRoute.SUBDOMAINS, ResultRoute.IPS), _spec('robtex', ResultRoute.SUBDOMAINS, ResultRoute.IPS), - _spec('rocketreach', ResultRoute.EMAILS, ResultRoute.LINKS), + _spec('rocketreach', ResultRoute.EMAILS, ResultRoute.URLS), _spec('securityTrails', ResultRoute.SUBDOMAINS, ResultRoute.IPS), _spec('securityscorecard', ResultRoute.SUBDOMAINS, ResultRoute.IPS), _spec('sherlockeye', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS, ResultRoute.IPS), @@ -138,7 +147,7 @@ _SPECS = ( _spec('subdomainfinderc99', ResultRoute.SUBDOMAINS, activity=ActivityClass.DNS), _spec('thc', ResultRoute.SUBDOMAINS), _spec('tomba', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS), - _spec('urlscan', ResultRoute.SUBDOMAINS, ResultRoute.IPS, ResultRoute.ASNS, ResultRoute.INTERESTING_URLS), + _spec('urlscan', ResultRoute.SUBDOMAINS, ResultRoute.IPS, ResultRoute.ASNS, ResultRoute.URLS), _spec('virustotal', ResultRoute.SUBDOMAINS), _spec('waybackarchive', ResultRoute.SUBDOMAINS), _spec('whoisxml', ResultRoute.SUBDOMAINS), @@ -156,7 +165,7 @@ _SPECS = ( ResultRoute.EMAILS, ResultRoute.IPS, ResultRoute.ASNS, - ResultRoute.INTERESTING_URLS, + ResultRoute.URLS, ), ) @@ -168,6 +177,20 @@ def get_source_spec(name: str) -> SourceSpec: return _CASEFOLDED_SOURCE_SPECS[name.casefold()] +def activity_classes_for_selection( + source_names: Iterable[str], + action_names: Iterable[str] = (), +) -> tuple[ActivityClass, ...]: + selected: set[ActivityClass] = set() + for name in source_names: + try: + selected.add(get_source_spec(name).activity) + except KeyError: + continue + selected.update(ACTION_ACTIVITIES[name] for name in action_names if name in ACTION_ACTIVITIES) + return tuple(activity for activity in ActivityClass if activity in selected) + + def resolve_sources(selection: str | Iterable[str]) -> list[str]: """Expand source and result-capability selectors into canonical source names.""" values = (selection,) if isinstance(selection, str) else selection diff --git a/theHarvester/restfulHarvest.py b/theHarvester/restfulHarvest.py index 295660d3..7629bab5 100644 --- a/theHarvester/restfulHarvest.py +++ b/theHarvester/restfulHarvest.py @@ -1,5 +1,4 @@ import argparse -import os import uvicorn @@ -32,17 +31,8 @@ def main(): help='Enable automatic reload used during development of the api', action='store_true', ) - parser.add_argument( - '--rate-limit', - default='5/minute', - help='Set API rate limit (e.g., "10/minute", "100/hour"), default is 5/minute', - ) - args: argparse.Namespace = parser.parse_args() - # Set environment variable for API rate limit - os.environ['API_RATE_LIMIT'] = args.rate_limit - uvicorn.run( 'theHarvester.lib.api.api:app', host=args.host, diff --git a/theHarvester/screenshot/screenshot.py b/theHarvester/screenshot/screenshot.py index 6b9eea83..6fd99a90 100644 --- a/theHarvester/screenshot/screenshot.py +++ b/theHarvester/screenshot/screenshot.py @@ -8,6 +8,9 @@ import ssl import sys from collections.abc import Collection from datetime import datetime +from ipaddress import ip_address +from pathlib import Path +from urllib.parse import urlsplit import aiohttp import certifi @@ -17,6 +20,18 @@ from playwright.async_api import async_playwright logger = logging.getLogger(__name__) +def _target_url(value: str, scheme: str = 'https') -> str: + if value.startswith(('http://', 'https://')): + return value + try: + address = ip_address(value) + except ValueError: + host = value + else: + host = f'[{address}]' if address.version == 6 else str(address) + return f'{scheme}://{host}' + + class ScreenShotter: def __init__(self, output) -> None: self.output = output @@ -57,7 +72,7 @@ class ScreenShotter: async def visit(url: str, proxy: str | None = None) -> tuple[str, str]: try: timeout = aiohttp.ClientTimeout(total=35) - urls = (url,) if url.startswith(('http://', 'https://')) else (f'https://{url}', f'http://{url}') + urls = (url,) if url.startswith(('http://', 'https://')) else (_target_url(url), _target_url(url, 'http')) sslcontext = ssl.create_default_context(cafile=certifi.where()) connector: ProxyConnector | aiohttp.TCPConnector proxy_param = None @@ -80,28 +95,35 @@ class ScreenShotter: logger.info(f'An exception has occurred while attempting to visit {url} : {e}') return '', '' - async def take_screenshot(self, url: str) -> str: - url = f'https://{url}' if not url.startswith(('http://', 'https://')) else url + async def take_screenshot(self, url: str, *, output_path: Path | None = None) -> str: + url = _target_url(url) logger.info(f'Attempting to take a screenshot of: {url}') async with async_playwright() as p: browser = await p.chromium.launch(headless=True) # New browser context context = await browser.new_context() page = await context.new_page() - path = rf'{self.output}{self.slash}{url.replace("http://", "").replace("https://", "")}.png' + path: Path | None = output_path or self.screenshot_path(url) date = str(datetime.now()) try: # Will fail if network idle or load event doesn't fire after # 35s which should be handled await page.goto(url, timeout=35000) await page.screenshot(path=path) - os.chmod(path, 0o600) + if path is not None: + os.chmod(path, 0o600) except Exception as e: logger.info(f'An exception has occurred attempting to screenshot: {url} : {e}') - path = '' + path = None finally: await page.close() await context.close() await browser.close() logger.info(f'{date} {url} {path}') return url if path else '' + + def screenshot_path(self, url: str) -> Path: + parsed = urlsplit(_target_url(url)) + hostname = (parsed.hostname or 'unknown-host').replace(':', '_') + port = f'_{parsed.port}' if parsed.port else '' + return Path(self.output) / f'{hostname}{port}.png' diff --git a/uv.lock b/uv.lock index 414dfe74..302bfbeb 100644 --- a/uv.lock +++ b/uv.lock @@ -484,18 +484,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, ] -[[package]] -name = "deprecated" -version = "1.3.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "wrapt" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/49/85/12f0a49a7c4ffb70572b6c2ef13c90c88fd190debda93b23f026b25f9634/deprecated-1.3.1.tar.gz", hash = "sha256:b1b50e0ff0c1fddaa5708a2c6b0a6588bb09b892825ab2b214ac9ea9d92a5223", size = 2932523, upload-time = "2025-10-30T08:19:02.757Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/84/d0/205d54408c08b13550c733c4b85429e7ead111c7f0014309637425520a9a/deprecated-1.3.1-py2.py3-none-any.whl", hash = "sha256:597bfef186b6f60181535a29fbe44865ce137a5079f295b479886c82729d5f3f", size = 11298, upload-time = "2025-10-30T08:19:00.758Z" }, -] - [[package]] name = "dnspython" version = "2.8.0" @@ -801,20 +789,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ce/62/b40b382fa0c66fee1478073eb8db352a4a6beda4a1adccf1df911d8c289c/librt-0.11.0-cp314-cp314t-win_arm64.whl", hash = "sha256:dee008f20b542e3cd162ba338a7f9ec0f6d23d395f66fe8aeeec3c9d067ea253", size = 102572, upload-time = "2026-05-10T18:17:06.809Z" }, ] -[[package]] -name = "limits" -version = "5.8.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "deprecated" }, - { name = "packaging" }, - { name = "typing-extensions" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/71/69/826a5d1f45426c68d8f6539f8d275c0e4fcaa57f0c017ec3100986558a41/limits-5.8.0.tar.gz", hash = "sha256:c9e0d74aed837e8f6f50d1fcebcf5fd8130957287206bc3799adaee5092655da", size = 226104, upload-time = "2026-02-05T07:17:35.859Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/b9/98/cb5ca20618d205a09d5bec7591fbc4130369c7e6308d9a676a28ff3ab22c/limits-5.8.0-py3-none-any.whl", hash = "sha256:ae1b008a43eb43073c3c579398bd4eb4c795de60952532dc24720ab45e1ac6b8", size = 60954, upload-time = "2026-02-05T07:17:34.425Z" }, -] - [[package]] name = "lxml" version = "6.1.1" @@ -1589,18 +1563,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, ] -[[package]] -name = "slowapi" -version = "0.1.10" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "limits" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/b9/52/24527cf25a8b508926aff53350b0136561dfe86c7125f61526653666e1b2/slowapi-0.1.10.tar.gz", hash = "sha256:d320d5bc04d9f171a77fb16700faf3036d85b00f420f22924c8a225f95bd14f9", size = 13841, upload-time = "2026-06-13T11:59:31.571Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/c4/8b/1d359f38706b4097d9a943bf8bd22599f537de4cbaff1e622d3e3936e164/slowapi-0.1.10-py3-none-any.whl", hash = "sha256:3acb61561dc9d687e3d3669362ff6a439de9ba44e2fed3a9c165da26b4b83e28", size = 14921, upload-time = "2026-06-13T11:59:30.485Z" }, -] - [[package]] name = "soupsieve" version = "2.8.4" @@ -1687,7 +1649,6 @@ dependencies = [ { name = "pyyaml" }, { name = "retrying" }, { name = "shodan" }, - { name = "slowapi" }, { name = "sqlalchemy" }, { name = "ujson" }, { name = "uvicorn" }, @@ -1732,7 +1693,6 @@ requires-dist = [ { name = "pyyaml", specifier = "==6.0.3" }, { name = "retrying", specifier = "==1.4.2" }, { name = "shodan", specifier = "==1.31.0" }, - { name = "slowapi", specifier = "==0.1.10" }, { name = "sqlalchemy", specifier = "==2.0.51" }, { name = "ujson", specifier = "==5.13.0" }, { name = "uvicorn", specifier = "==0.49.0" }, @@ -2010,70 +1970,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0f/8d/52eaa9187b88596b0a8b646874cfec5a5c3fce8c52b5182be1a0253203a3/winloop-0.6.3-cp314-cp314t-win_arm64.whl", hash = "sha256:447006f38f13827ff4600e7beeda70367370cb8dab8ea84042e8fa1749f32b1c", size = 576206, upload-time = "2026-04-27T16:08:07.007Z" }, ] -[[package]] -name = "wrapt" -version = "2.2.1" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/2d/9f/06263fcd8ad6c405f05a3905fd7a84dd3176eb5ad46e44bccc0cd16348bb/wrapt-2.2.1.tar.gz", hash = "sha256:6744f504375775d7609c82c8d3d94af1c9a6f05586984536905908ba905277b9", size = 127620, upload-time = "2026-05-22T14:49:43.056Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/89/0c/bfae7b9401583b6d05938cd16dedc43857d96da2f8a3d50d78cc515bf6ff/wrapt-2.2.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3ffad790d9d11d8ecf9f17c4bb671a5b4089e4d8b575c46c5129597f41f836b0", size = 81021, upload-time = "2026-05-22T14:48:00.313Z" }, - { url = "https://files.pythonhosted.org/packages/26/58/80f6a6599f933f4caecc1cb3ee88a04faf81e8b9bddbd6109c688dd63e0f/wrapt-2.2.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:628f5220c7a904d5fc78f7075c8d7871433eb6d035c94728a22fdf85f193d2a8", size = 81692, upload-time = "2026-05-22T14:48:01.49Z" }, - { url = "https://files.pythonhosted.org/packages/17/93/fb357cc7847c58a8ae790be718903afa81a28d23e642c843dc4129e8a0b2/wrapt-2.2.1-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:61acce4257a9883669703c525447c5b4c392edf0f987ae77ec32668440158f0e", size = 169364, upload-time = "2026-05-22T14:48:02.791Z" }, - { url = "https://files.pythonhosted.org/packages/aa/0b/76b601ee309a8bd556af0eecb184394c20b3c49aa9c8e085aa1ffacc2568/wrapt-2.2.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:727ab4244622cd6ad2390f322642090c877d2e83a608d2653a7643ae5368d926", size = 171079, upload-time = "2026-05-22T14:48:04.22Z" }, - { url = "https://files.pythonhosted.org/packages/cd/87/ee3f32d5658e3e26d3e0e457922b47a36dd3bfbdfee7f97bb3e802344a66/wrapt-2.2.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:03df9ebed4c73ab93fa8c07e3d41d818dfca1852b15731a3de59457b27814624", size = 160205, upload-time = "2026-05-22T14:48:05.553Z" }, - { url = "https://files.pythonhosted.org/packages/b1/d0/ae2fd64277a67f5d7bffcf2d05eea1e476263fb2a072baf0b0129ab85984/wrapt-2.2.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:0d9ff006f420b2ec8296aa56ade43ea7da3e997e85769f0aafc5e0661aacb710", size = 168922, upload-time = "2026-05-22T14:48:07.132Z" }, - { url = "https://files.pythonhosted.org/packages/b1/f3/2d541a060c5bbafb9400bca4917e4d78bfd1f239f404782c86831a8f6b29/wrapt-2.2.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:844c858fc3bb7eacc0ba8efa904935d16aac6a4470948ad1e7e55c9f5a2a665f", size = 158388, upload-time = "2026-05-22T14:48:08.629Z" }, - { url = "https://files.pythonhosted.org/packages/1d/68/8d92c8800c57e93cb116ae9e9d6cbafc34fade5ee9f9107b6f203fb4dc35/wrapt-2.2.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:87bacdaf225117a342a20d9c03438d701c02112f6e3f351ce9b7f32354f14797", size = 167682, upload-time = "2026-05-22T14:48:10.042Z" }, - { url = "https://files.pythonhosted.org/packages/30/72/83ea3790ea352439442349388e29ff07b76e0686265f9088bbb505d1608d/wrapt-2.2.1-cp312-cp312-win32.whl", hash = "sha256:2f8c90c8afde51969487be4e1343ae049b268854877d415c2510baf833775052", size = 77857, upload-time = "2026-05-22T14:48:11.782Z" }, - { url = "https://files.pythonhosted.org/packages/ef/cb/99450668dd3502d62a54a1c8aa56e44f34cb8c1261b381cfe2e7926c3b75/wrapt-2.2.1-cp312-cp312-win_amd64.whl", hash = "sha256:6ce32763ac31ce94fe9aada947e479b1975012bff166da409b4b9e4e376cf7e5", size = 80825, upload-time = "2026-05-22T14:48:13.046Z" }, - { url = "https://files.pythonhosted.org/packages/e6/3a/87512881be64e743f9ee4c66f4cbe8e884974bef2a5989af71f999653ac7/wrapt-2.2.1-cp312-cp312-win_arm64.whl", hash = "sha256:8d1b4d0e0c2119587a31f5c029abd547e0c81d93b89d394566fe1588659eb579", size = 79087, upload-time = "2026-05-22T14:48:14.323Z" }, - { url = "https://files.pythonhosted.org/packages/88/d1/a1b08f8f4fac8cbb156fa51cf64ee2c7f7f74f9875ba3cf70b3c58368694/wrapt-2.2.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:d2beb1c7cab10603aecdc42f8edd6ff013f9a32e4543474e38e6b77ce9975aeb", size = 80831, upload-time = "2026-05-22T14:48:15.598Z" }, - { url = "https://files.pythonhosted.org/packages/54/ce/57890814991446a845e09b3445ce8b694f27eb0577004f2c2a36a9772ed4/wrapt-2.2.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e0cb7e4dd71f4c32e5e84843cd3c4cd65dda034314004bbe1d7f99af2426ab80", size = 81375, upload-time = "2026-05-22T14:48:17.071Z" }, - { url = "https://files.pythonhosted.org/packages/38/65/08d7a6c76ac4493bdb668205ee9c1de1bd5daca61717c3e9aa49b4c01499/wrapt-2.2.1-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:95821352042722cd9f1108874579a47989d0a7e12a37d87d2fc4af20fd99ab8a", size = 167417, upload-time = "2026-05-22T14:48:18.303Z" }, - { url = "https://files.pythonhosted.org/packages/62/ce/f1ccbee7a1bfe5cdc6b3da6bab4b45713d628b9294da32a39f563d648140/wrapt-2.2.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:abd621552ede77c4c69be7fac44ba911225b0c812b6ba604e5964cf98085b474", size = 166948, upload-time = "2026-05-22T14:48:19.768Z" }, - { url = "https://files.pythonhosted.org/packages/86/2a/f85d48d1cd4869aee6704028d257d740a47c1c467b457ce396b4b5b55d07/wrapt-2.2.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e3677c7146ce694874941ba82b57092cc4875445aadf29d72807351023105143", size = 158148, upload-time = "2026-05-22T14:48:21.96Z" }, - { url = "https://files.pythonhosted.org/packages/fe/5c/93939ad11d4a12358ab1aab219a2ef5efa5612e0db6b9fc65af8af1a891b/wrapt-2.2.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:9a5934eaea872e17936b5f45501eba5ab0bce9a74122e172b663d7c28c459c4a", size = 165905, upload-time = "2026-05-22T14:48:23.373Z" }, - { url = "https://files.pythonhosted.org/packages/e0/22/b8c2aa89862ff58605934d7abf4b70e6a5a1c33df96656f49035ccdf1c8a/wrapt-2.2.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:f5b9daf6b629fce418e0cc3dd0436eac045188fa35deadb7a7f3941d5b8203f9", size = 156712, upload-time = "2026-05-22T14:48:24.767Z" }, - { url = "https://files.pythonhosted.org/packages/5d/78/bf00a7b02239c12bb02ddcc3c0b971bfcc36e578c5a44f1ccfef5b458545/wrapt-2.2.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:f53ac9f3ef573326d009ed809beff4efcac6451931c2b8132586da4b9e53ff31", size = 166560, upload-time = "2026-05-22T14:48:26.83Z" }, - { url = "https://files.pythonhosted.org/packages/fe/93/6390ca9c5b787683cef588d04f57c8d41b9a2323b5597a65f18638c90ef2/wrapt-2.2.1-cp313-cp313-win32.whl", hash = "sha256:1ffa9cfd4bdb581539951b14ae661ff20ed0c3599b3e911a131ee0ec5ac11337", size = 77817, upload-time = "2026-05-22T14:48:28.221Z" }, - { url = "https://files.pythonhosted.org/packages/97/73/ce10f0e71c0cfaa1a65faadb8efd4852028b3bb9ba28932b8889df769d38/wrapt-2.2.1-cp313-cp313-win_amd64.whl", hash = "sha256:368eac1e20fd0bb03dd3cc42bf9887154c3861b60989389ccb5fac032617d215", size = 80736, upload-time = "2026-05-22T14:48:30.139Z" }, - { url = "https://files.pythonhosted.org/packages/c7/4c/89f4a6818fafbbd840330e4fa3873073e1bfc166133a64cac7f8fde7a5e3/wrapt-2.2.1-cp313-cp313-win_arm64.whl", hash = "sha256:c754dafdf5aaf0b401b644a90a30046929a0dd1a536e0ff0ec959a59155d9c7f", size = 79099, upload-time = "2026-05-22T14:48:31.405Z" }, - { url = "https://files.pythonhosted.org/packages/bf/f2/9a8741c46f8c208ac0a45b25ba170bcb4fb72a2781d5fb97dbd7b6be73cb/wrapt-2.2.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:ed928d0fda15fc0adc8d13305c8b3c0f2fba5b0669950c9e6d019d9162a3b3e8", size = 82802, upload-time = "2026-05-22T14:48:33.307Z" }, - { url = "https://files.pythonhosted.org/packages/9c/0d/e9c855716a3705eef1416456bdf062b60620726fdc59428ff670fc3c60dc/wrapt-2.2.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:fafb4e739e43544d12cb4abd1605fd4683b6ca6a9ad682b7fd8f4d21973eafa8", size = 83329, upload-time = "2026-05-22T14:48:34.593Z" }, - { url = "https://files.pythonhosted.org/packages/3b/d6/a88f1c13112b7831adac75cea65d8310e0d696d570c8961844c90a57b865/wrapt-2.2.1-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:74d6a0c31472fe5d814917266b9f46495d7c61ed890af08b468acea92fb89a8d", size = 202937, upload-time = "2026-05-22T14:48:35.859Z" }, - { url = "https://files.pythonhosted.org/packages/42/65/e29d54aef06a4d898a5b8a25589a0b3769bde454f922fad8f6f89fbfb650/wrapt-2.2.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ab5be648d5a0b86b7438864f8df3c705a65cef35a2fd3e5561e3e203167e0f27", size = 209997, upload-time = "2026-05-22T14:48:38.153Z" }, - { url = "https://files.pythonhosted.org/packages/2a/91/e4454263516cf0e12640912fbca9a83654e424f0a6ddb79f5cd7ce14bf33/wrapt-2.2.1-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9d8f204c8e3a8bf9ece17e0a83d137fd807440977f8a5e762d59306795011440", size = 194856, upload-time = "2026-05-22T14:48:39.69Z" }, - { url = "https://files.pythonhosted.org/packages/de/d0/fe0ee202286afdf4a7f77dd29f195703145764d572aec209c5086e57d924/wrapt-2.2.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:d047f6498c973874ba08ac3f97c69a2c4b2211c8de6f4c205f75cb1c9522596e", size = 205654, upload-time = "2026-05-22T14:48:43.456Z" }, - { url = "https://files.pythonhosted.org/packages/23/b6/87d860dfc6460c246af70b1fd5c8b76df77571b42a493459423ded94fd7d/wrapt-2.2.1-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:7a4fdb9326aab4a5a477a1640e5ad786a8495901009d7e7b038371edd23a9d2b", size = 192206, upload-time = "2026-05-22T14:48:44.858Z" }, - { url = "https://files.pythonhosted.org/packages/df/46/3eea8cde077d985f239a38c0257087b8064fd9ee9b1a99e282d2c86da4ef/wrapt-2.2.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:c8cc5094b08abeae52da9c73c8a32003623be691a5193df2f4e3eac3d557c394", size = 198428, upload-time = "2026-05-22T14:48:46.319Z" }, - { url = "https://files.pythonhosted.org/packages/18/dc/b927ee9c7fc67adc3a5658f246a0d275425eb840ba36e7b702e70f18bde8/wrapt-2.2.1-cp313-cp313t-win32.whl", hash = "sha256:9907a4402ab6db12b7077a0ea5d7a4d028ecb22c8eee2b53527080d347cd1562", size = 79448, upload-time = "2026-05-22T14:48:47.901Z" }, - { url = "https://files.pythonhosted.org/packages/ec/b3/fd30b473fe498c70e6b9a5f328b8d3fbaf1b8c3c481465f59724bba8eb70/wrapt-2.2.1-cp313-cp313t-win_amd64.whl", hash = "sha256:5590d63f5243251641cf543009b4c9314a79d0598fdb8a8e4cfc918494536c53", size = 83021, upload-time = "2026-05-22T14:48:49.201Z" }, - { url = "https://files.pythonhosted.org/packages/ee/f3/96c39153a8737a6e9aa85adef254ac4195bea3f2d24efc60472ccc3c9e2e/wrapt-2.2.1-cp313-cp313t-win_arm64.whl", hash = "sha256:c318a64b53d97b841d7b5e637517e50a27be64bc695128422953d4b21710954e", size = 80295, upload-time = "2026-05-22T14:48:50.479Z" }, - { url = "https://files.pythonhosted.org/packages/0a/a3/11d7f34ebbf3231bc907a3e6d5ee051b14d034c1bc7b65a97d5cc00516df/wrapt-2.2.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:6f56a647e4eaf5f0ca40330fb070f566bdf9f7b0db89a1af20d71c28dcd7a0ab", size = 80879, upload-time = "2026-05-22T14:48:51.802Z" }, - { url = "https://files.pythonhosted.org/packages/13/3c/b74cfd984cef560b900fb1a727af20352d89e1f06bf2e1114dd3f00f5f5a/wrapt-2.2.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:64b7deeda4b70408e382328d8bbe52a256fe9bc63ae3db86d804608367e5422c", size = 81462, upload-time = "2026-05-22T14:48:53.18Z" }, - { url = "https://files.pythonhosted.org/packages/15/a3/7c8f704b8dc07dfe0a5d01c2edbfd88317aa8e5e3fa7c743eb7a085ae767/wrapt-2.2.1-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b9cf53ba90717db2e292401de290776c498d4bbfb0d4a559ca2895db8b9dcb5c", size = 167251, upload-time = "2026-05-22T14:48:54.562Z" }, - { url = "https://files.pythonhosted.org/packages/80/85/a34d1888d97247da6c2ff6118c3a721c73ed8cc4dd198c00208bb73b6f80/wrapt-2.2.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cf3638274ab9d9b724c9baa0b4c04e132cd6faefb78b4dd3dd1a02a4bdaad41e", size = 166316, upload-time = "2026-05-22T14:48:56.065Z" }, - { url = "https://files.pythonhosted.org/packages/e9/d7/72ffaeb01eebc704afe3fb99e840480f4bda45f0fa66e3381b6a39251c8f/wrapt-2.2.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:aed9658797d0b45d6c49adcfc6b41f66e6f2d0c6de3ec79e16cf4b1855df240f", size = 157952, upload-time = "2026-05-22T14:48:57.924Z" }, - { url = "https://files.pythonhosted.org/packages/24/5b/36f5d6b024e4edfdd90b140742d11ebcf7836daf5c9daf326c55c24db412/wrapt-2.2.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:1d676ee388bc42a04d56dd7deb5605244dac2e35cc2fadbb43c9fa25bbd93508", size = 166130, upload-time = "2026-05-22T14:48:59.384Z" }, - { url = "https://files.pythonhosted.org/packages/81/06/9296d9e97bfdef5483dfcc859d57b095b257144b2bc5300ab521e06f4bc7/wrapt-2.2.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:e395f7bc31851ef9b612050368cb446e9bc14cd7454b025018980349caf25ae5", size = 156604, upload-time = "2026-05-22T14:49:00.921Z" }, - { url = "https://files.pythonhosted.org/packages/53/37/16953929ed6776175720e58fc966e779926d8d71e2c7b2273230590ca71f/wrapt-2.2.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5f1845c2a8cc1180ccccfa45785dd06f562730d19ef75be180334254012b6283", size = 166007, upload-time = "2026-05-22T14:49:02.332Z" }, - { url = "https://files.pythonhosted.org/packages/b9/73/20ee58c0612dae7c31131a7095345812ed2c7b389019e175f68cde34e5b4/wrapt-2.2.1-cp314-cp314-win32.whl", hash = "sha256:436addbc4bb4fc0a88c702577f51195d7d73683a7f3e0e5b253d8404d7847243", size = 78327, upload-time = "2026-05-22T14:49:03.722Z" }, - { url = "https://files.pythonhosted.org/packages/22/b3/ef7c3295d02e0448a71c639a36a057f46d524d057c9486291a7a3039e65c/wrapt-2.2.1-cp314-cp314-win_amd64.whl", hash = "sha256:50972a1d974ea07725a7f6b1cec5f8759008afd030a0024843ebe7d52de47f2b", size = 81144, upload-time = "2026-05-22T14:49:05.093Z" }, - { url = "https://files.pythonhosted.org/packages/ac/dc/7bdf336953f99f4ceb0a584bb8870e42c8f26f93ea10c87834dad62f1668/wrapt-2.2.1-cp314-cp314-win_arm64.whl", hash = "sha256:1c9934ea5d92957e3cd0adbc0845539dccfd62710ebe16195a8c66c53954db36", size = 79569, upload-time = "2026-05-22T14:49:06.413Z" }, - { url = "https://files.pythonhosted.org/packages/6a/6d/6dfae80150ff1919c356d1dd528f049bcdfaae29b4d284bc957e022caef4/wrapt-2.2.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:17de18fc12cea55b8a9587314cb830573e37fb33b247a7515696350863714188", size = 82892, upload-time = "2026-05-22T14:49:07.925Z" }, - { url = "https://files.pythonhosted.org/packages/82/7b/4e34766a7d7804ffce9e71befe47e9b3225dc350c49c94493c4ab39fd3a5/wrapt-2.2.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:a9dec1aca52dddde7df94818310fa2fe79739c8f385b2014c4cb1035f5508199", size = 83333, upload-time = "2026-05-22T14:49:09.257Z" }, - { url = "https://files.pythonhosted.org/packages/9d/57/0b34db3e8de44ccfece62d7b337abd1631dd810f5adc5f3db571727836b5/wrapt-2.2.1-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:69f2e9244542cb34dd59c7f073445b9e54ad9f3fce8d93606c368a1b499fc413", size = 202899, upload-time = "2026-05-22T14:49:10.572Z" }, - { url = "https://files.pythonhosted.org/packages/e5/45/ac0c459f154b99d92789a6cba7ca727185b83513b986f8ec7fe2aacddcbf/wrapt-2.2.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2d83966dc7f4f45e8b97b5933685ac2e6e67fc0e19246ea314bceb9a8970c956", size = 209986, upload-time = "2026-05-22T14:49:12.229Z" }, - { url = "https://files.pythonhosted.org/packages/b7/e4/77e37ff33ad018fa81ade52c25fa327b80b56f81d734279a63614fcb4cbc/wrapt-2.2.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:78b0aa6bfb7be8deed0ab23e7aa028cc5210c29bc2d32a04d52b50e517a7307e", size = 194893, upload-time = "2026-05-22T14:49:14.139Z" }, - { url = "https://files.pythonhosted.org/packages/dd/9d/7ea651d1ab032fc5fa222fbec91d0f8a1397f6ae04ebb93fa7219aa921d7/wrapt-2.2.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:05d5cb74d1b232ec8cfa130a8f900708699ff2491d97b8f85a4cdc5996294b85", size = 205636, upload-time = "2026-05-22T14:49:15.714Z" }, - { url = "https://files.pythonhosted.org/packages/09/af/8e88031a701275b9085c54e64bc88c0b1cd55c77eadd400691c371cd76c4/wrapt-2.2.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:f6518b94edb9150452e9aba08027d4cc293433753ec1fbefb4629a21cbc74181", size = 192267, upload-time = "2026-05-22T14:49:17.283Z" }, - { url = "https://files.pythonhosted.org/packages/bf/a8/e657ca876b06710194f243d81c4b0896ade646e244bdbec2d87c8c56a8bd/wrapt-2.2.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ed55af48b3eb28f43228ca2306788892bcb629eb2b5c4876e2a3659872c2f17a", size = 198378, upload-time = "2026-05-22T14:49:18.785Z" }, - { url = "https://files.pythonhosted.org/packages/c8/59/822efe4ea722a3961331bfa35b7d90937790d2c20f0616de1997ccc3aebd/wrapt-2.2.1-cp314-cp314t-win32.whl", hash = "sha256:2e08688ab16525897da6589d56d0aebaf417bbe91c2d8e3b96203b1efa596e85", size = 80226, upload-time = "2026-05-22T14:49:20.264Z" }, - { url = "https://files.pythonhosted.org/packages/ab/31/2a7dc5f6abb2fca0b6e1610e120419f603650aceb4f1d3ac4cae0354e162/wrapt-2.2.1-cp314-cp314t-win_amd64.whl", hash = "sha256:fd0135d34387f5fd087d9be368ea77ea89cf2451dc1cd1c622d35021bcb3ab50", size = 83835, upload-time = "2026-05-22T14:49:21.634Z" }, - { url = "https://files.pythonhosted.org/packages/9f/c0/782b86e28d1ceebeb74cccea12d2cd3d2ba0bd68e3dec20b1bc5873f6127/wrapt-2.2.1-cp314-cp314t-win_arm64.whl", hash = "sha256:f70db64e8266d7c45d3b735f2e08eeb434b5e03da9a479ae42b2e2e486a21a00", size = 80722, upload-time = "2026-05-22T14:49:23.59Z" }, - { url = "https://files.pythonhosted.org/packages/53/46/29ac9daf11a86c22a8c38cd9236c62928ccae83f7ceb06bd3b0467cf9d05/wrapt-2.2.1-py3-none-any.whl", hash = "sha256:3aafea2975caef8ca49400640dde02cc7426e798f24870ed01f490bc3cffd32f", size = 61000, upload-time = "2026-05-22T14:49:41.593Z" }, -] - [[package]] name = "xlsxwriter" version = "3.2.9"