WEBDIENSTE/theHarvester
1
0
Fork 0
mirror of https://github.com/laramies/theHarvester.git synced 2026-08-17 19:35:40 +02:00
Code Issues Packages Projects Releases Wiki Activity

Add an authenticated durable run API v1 (#2511)

Browse Source 
* Model active evidence in result persistence

* Expose active action diagnostics

* Persist truthful DNS action evidence

* Persist direct action evidence

* feat: add authenticated durable run API v1

* Make API file interchange JSONL-only

* Harden API evidence boundaries

* Remove API rate limiter

* Unify API runs with result persistence

* Support target action runs

* Keep DNS resolver selection action-neutral

* Add action-neutral CLI resolver selection

* Preserve API evidence across JSONL round trips

* Complete HarvestView API run parity

* Harden HarvestView run API contracts

* Remove obsolete bundled network snapshots

Delete the unused bundled AWS IP-range and resolver snapshots while preserving operator-supplied resolver file input.

* refactor: canonicalize URL results

* docs: define a bounded test budget

* Standardize hostname and IP result names

* fix(api): avoid duplicate evidence conflicts
This commit is contained in:
Matt
2026-08-10 00:08:04 -04:00
committed by GitHub
parent e4d37b05a8
commit 1fd5749e52
82 changed files with 9022 additions and 13571 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+7
View File
@@ -28,3 +28,10 @@ Read [CONTEXT.md](CONTEXT.md) when changing discovery terminology, evidence clas
- Typing: `uv run mypy theHarvester`
Run focused checks first and expand according to risk. Report any skipped check and its reason.
### Test budget
- During implementation, run the narrowest test that covers the changed behavior. Do not rerun the full suite after every small edit.
- Run the full non-browser suite once at the publication head. Dependent stack layers do not need to repeat it unless they change Python behavior.
- Run the HarvestView browser suite once at the final UI head or rely on its GitHub workflow. Static UI edits should use focused UI tests and a JavaScript syntax check first.
- Before retrying a long-running test, confirm the previous process exited. Poll the existing command or stop only its exact owned process instead of starting an overlapping run.
CHANGELOG.md
+12 -2
View File
@@ -15,6 +15,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added bounded, keyless subdomain discovery through Arquivo.pt's public CDX API with offline response contracts.
- Added transactional SQLite storage and loading for completed full-pipeline runs without changing legacy result rows.
- Added deterministic JSONL report companions finalized after selected one-shot actions complete.
- Added a unified model and SQLite schema for active-action provenance and screenshot artifact metadata.
- Added authenticated bulk import for completed runs from validated theHarvester SQLite databases.
- Added bounded custom endpoint-path input for REST API scans without exposing server-side file paths.
- Recorded DNS resolution, recursive DNS, DNS brute force, and PTR lookup outcomes through the unified action model.
- Recorded takeover, Shodan, and API endpoint scan outcomes through the unified action model.
- Added normalized BuiltWith framework, language, server, CMS, and analytics findings to JSONL and completed-result SQLite output.
- Added DNSDB passive DNS discovery with API key configuration, shared transport handling, result parsing, and offline tests ([9b41b78e](https://github.com/laramies/theHarvester/commit/9b41b78e), [aba9fec6](https://github.com/laramies/theHarvester/commit/aba9fec6)).
- Added `--verbose` diagnostic logging while keeping normal operator output available at the default log level ([8a7b8b71](https://github.com/laramies/theHarvester/commit/8a7b8b71)).
@@ -22,6 +27,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added root contributor and security policies, structured issue forms, repository agent guidance, discovery terminology, and an operator-focused documentation wiki ([d090a29a](https://github.com/laramies/theHarvester/commit/d090a29a), [7c491ef5](https://github.com/laramies/theHarvester/commit/7c491ef5), [8b9d420b](https://github.com/laramies/theHarvester/commit/8b9d420b)).
### Changed
- Standardized SQLite, JSONL, API, and HarvestView result names on `hostname` and `ip` without a presentation alias.
- Standardized URL-producing adapters, JSON, JSONL, SQLite, and API evidence on one `url` result kind while preserving producer provenance.
- Fixed proxied POST requests so they retain the request method, body, and query parameters.
- Migrated Pentest-Tools discovery to its API v2 Bearer-authenticated scan, status, and output endpoints.
- Included HIBP verified-domain in `all` and matching capability selectors like every other P0 source, with REST operator authentication applied after source expansion when its provider key is configured.
@@ -35,20 +42,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Replaced deprecated hostname resolution with `getaddrinfo`-based handling ([6a847435](https://github.com/laramies/theHarvester/commit/6a847435)).
- Reworked routine CI to use read-only permissions, non-mutating Ruff checks, offline tests, and explicit opt-in live provider checks ([72e5820f](https://github.com/laramies/theHarvester/commit/72e5820f)).
- Grouped GitHub Actions, Python, and Docker Dependabot updates with a seven-day cooldown, and added a seven-day `uv` dependency freshness window ([7a947b66](https://github.com/laramies/theHarvester/commit/7a947b66), [52a79cdb](https://github.com/laramies/theHarvester/commit/52a79cdb)).
- Updated runtime dependencies: `aiohttp` to `3.14.1`, `beautifulsoup4` to `4.15.0`, `certifi` to `2026.6.17`, `fastapi` to `0.138.1`, `slowapi` to `0.1.10`, `ujson` to `5.13.0`, and `uvicorn` to `0.49.0`.
- Updated runtime dependencies: `aiohttp` to `3.14.1`, `beautifulsoup4` to `4.15.0`, `certifi` to `2026.6.17`, `fastapi` to `0.138.1`, `ujson` to `5.13.0`, and `uvicorn` to `0.49.0`.
- Updated development dependencies: `pytest` to `9.1.1`, `ruff` to `0.15.20`, and `ty` to `0.0.54`.
- Updated CI and container maintenance pins, including `actions/checkout`, `astral-sh/setup-uv`, `astral-sh/ruff-action`, `github/codeql-action`, StepSecurity Harden-Runner, Docker actions, and the Python base image.
- Expanded offline regression coverage for discovery providers, configuration contracts, logging, output, documentation, workflow policy, and scope boundaries.
### Removed
- Removed the obsolete bundled IP-range and resolver snapshots.
- Removed the REST API's built-in SlowAPI request limiter and its launcher option without adding a replacement.
- Removed Bitbucket domain discovery because its current REST APIs require workspace, repository, or user scope that the domain-only CLI contract cannot provide.
- Removed the nonfunctional ThreatCrowd source because its service hostnames terminate at deleted AWS load balancers and return NXDOMAIN; OTX remains available through its separate adapter.
### Fixed
- Kept API endpoint scan URLs canonical instead of prefixing targets onto already complete URLs.
- Made DeHashed pagination honor the CLI limit, retain only normalized email and IP evidence, and discard raw breach rows; aligned LeakIX with its authenticated subdomain endpoint and documented rate-limit retry.
- Added offline contracts for explicitly selected DNS and direct sources, retained normalized Pentest-Tools host and IP results, and hardened Shodan InternetDB, SubdomainFinder C99, and Windvane evidence boundaries.
- Retained relevant GitLab project, profile, and website URLs in consolidated JSONL and SQLite results while excluding unrelated user URLs.
- Removed BuiltWith's duplicate interesting-URL getter by allowing the shared collector to use either established getter spelling.
- Standardized BuiltWith and every other URL-producing adapter on `get_urls()`.
- Made no-filename REST `/query` executions reach completed-result construction and SQLite persistence without changing the legacy response fields.
- Made Chaos reject empty credentials, report HTTP and malformed-response failures, and preserve supported subdomain response shapes.
- Made Fofa reject incomplete credentials, report HTTP and malformed-response failures, normalize scoped hosts, and discard invalid IP values.
CONTEXT.md
+42 -2
View File
@@ -1,6 +1,6 @@
# theHarvester discovery context
This glossary is the source of truth for discussing subdomain discovery across code, issues, pull requests, output, and documentation. It separates current addressability from historical, indirect, or unresolved evidence.
This glossary is the source of truth for discussing subdomain discovery and HarvestView lifecycle behavior across code, issues, pull requests, output, and documentation. It separates current addressability from historical, indirect, or unresolved evidence.
The definitions state intended semantics; they do not imply that every discovery adapter already produces every evidence class. Update this glossary when a change alters a term's meaning or boundary.
@@ -50,13 +50,53 @@ _Avoid_: Result, duplicate, hit
A deduplicated operator-facing entity backed by one or more discovery observations and their retained provenance.
_Avoid_: Raw finding, source result
**Hostname result**:
One normalized DNS-name merged result. It can be the authorized target itself or a subordinate name and does not by itself imply current DNS addressability.
_Avoid_: Subdomain result, live host, resolved host
**IP result**:
One canonical IPv4 or IPv6 address merged result.
_Avoid_: IP-address result, resolved host
**URL result**:
One normalized URL merged result. Source and action origins identify how it was found; provider-specific URL categories are not separate result kinds.
_Avoid_: Interesting URL, LinkedIn link, API endpoint result
**DNS validation observation**:
One resolver vantage's time-bound DNS evidence about one in-scope candidate. It supports classifying the candidate as currently addressable or wildcard-indistinguishable without replacing its discovery observations.
_Avoid_: DNS result, resolved host, validation status
**Enumeration run**:
One finite execution of theHarvester against an explicit target and selected options, identified independently from every other execution.
_Avoid_: Scan, monitoring cycle, session
_Avoid_: Scan, monitoring cycle, session, job
**Action-only run**:
An enumeration run with no discovery sources that performs an explicitly selected DNS or direct action against an explicitly authorized target. It creates its own run record and never mutates the evidence of a parent run.
_Avoid_: Result action, parent-run update, inline scan
**Run record**:
The durable operator-facing record that begins when an enumeration is submitted or evidence is imported and retains lifecycle, authorization, and available evidence under one stable identifier.
_Avoid_: Task, worker job, scan record
**HarvestView**:
The browser-based analysis workspace for creating and inspecting run records, normalized evidence, source outcomes, and managed artifacts from theHarvester.
_Avoid_: Internal workflow names, operator app, console, dashboard
**Imported run**:
A run record created from an existing theHarvester result file. Import records evidence but never executes discovery or contacts a target.
_Avoid_: Uploaded scan, replayed run
**Lifecycle status**:
The durable state of a run record: queued, running, cancelling, cancelled, completed, or failed. It describes control flow, not evidence quality.
_Avoid_: Run result, provider status
**Terminal evidence status**:
The completeness classification reported by a finished enumeration result: complete, partial, or failed. It does not describe queue or cancellation state.
_Avoid_: Lifecycle status, completion state
**Cancellation request**:
The operator's durable request that the run worker prevent queued work from starting or ask the running child process to stop. A request is not itself proof that execution has ended.
_Avoid_: Cancelled run, process killed
**Source execution**:
One attempt to run one canonical discovery source within an enumeration run, with an explicit completion status and summary counts.
README.md
+28 -34
View File
@@ -79,11 +79,11 @@ uv run theHarvester -h
Options such as DNS brute force (`-c`), bounded recursive DNS (`--dns-recursive-depth`), reverse DNS lookup (`-n`), takeover checks (`-t`), API endpoint scanning (`-a`), DNS resolution (`-r`), and screenshots (`--screenshot`) generate additional network activity. Use them only within an explicitly authorized scope.
Recursive DNS requires exactly three distinct resolver IPs through `--dns-resolve`. It advances only names with two-vantage address consensus that are distinguishable from closest-encloser wildcard controls. Depth, DNS record query, and runtime limits are configurable through the three `--dns-recursive-*` options; the default query ceiling is 3,000 record queries across resolver vantages, and three consecutive zero-yield batches also stop recursion. PTR names for current addresses are retained as secondary evidence, but they do not establish current addressability or become recursion seeds. REST `/query` exposes the same options and requires the configured operator API key when recursion is enabled.
Recursive DNS requires exactly three distinct resolver IPs through `--dns-resolvers` or the compatible `--dns-resolve` value. It advances only names with two-vantage address consensus that are distinguishable from closest-encloser wildcard controls. Depth, DNS record query, and runtime limits are configurable through the three `--dns-recursive-*` options; the default query ceiling is 3,000 record queries across resolver vantages, and three consecutive zero-yield batches also stop recursion. PTR names for current addresses are retained as secondary evidence, but they do not establish current addressability or become recursion seeds. `POST /api/v1/runs` exposes the same controls.
Screenshot capture also requires a Playwright-compatible browser; see the installation guide for setup.
## Browser interface and REST API
## REST API
`restfulHarvest` starts a FastAPI service on `127.0.0.1:5000` by default:
@@ -95,33 +95,27 @@ Open [http://127.0.0.1:5000/docs](http://127.0.0.1:5000/docs) for interactive Sw
| Route | Purpose |
| --- | --- |
| `GET /sources` | List registered discovery sources. |
| `GET /query` | Return consolidated discovery results, including emails and breach names, as JSON. |
| `GET /dnsbrute` | Run DNS brute force for a domain. |
| `POST /additional/breaches` | Return Have I Been Pwned breach data. |
| `POST /additional/leaks` | Return Leak-Lookup data. |
| `POST /additional/security-score` | Return SecurityScorecard data. |
| `POST /additional/tech-stack` | Return BuiltWith technology data. |
| `POST /additional/all` | Run all additional API lookups. |
| `GET /api/v1/sources` | List registered discovery sources and capabilities. |
| `POST /api/v1/runs` | Submit a finite enumeration run. |
| `GET /api/v1/runs` | List durable run records. |
| `GET /api/v1/runs/{run_id}` | Retrieve lifecycle state, normalized results, and source outcomes. |
| `POST /api/v1/runs/{run_id}/cancel` | Cancel queued or running work. |
| `POST /api/v1/runs/import` | Import JSONL evidence without executing discovery. |
| `POST /api/v1/runs/import-database` | Import completed runs from a theHarvester SQLite database. |
| `GET /api/v1/runs/{run_id}/export` | Export normalized evidence as JSONL. |
The service rate limit defaults to five requests per minute and can be changed with `--rate-limit`. The `/additional/*` routes require `THEHARVESTER_API_KEY` on the server and the same value in the `X-API-Key` request header.
Every `/api/v1/*` route requires `THEHARVESTER_API_KEY` in the `X-API-Key` header. Provider credentials stay in server-side configuration and cannot be supplied in a request. Keep the service bound to localhost. If you require remote access, add network access controls and TLS.
The core `/query`, `/sources`, and `/dnsbrute` routes do not normally require authentication. When a `/query` selection includes `dehashed`, `hibpverified`, or `leaklookup` and that source's provider key is configured, the request requires `THEHARVESTER_API_KEY` in the `X-API-Key` header because these sources can access breach-account data. Keep the service bound to localhost. If you require remote access, add authentication, access controls, and TLS.
Docker Compose publishes port `5000` on every host interface unless you narrow the port mapping:
```bash
docker compose up --build
```
When `--proxies` and `--take-over` are combined, supported discovery and takeover requests use the configured proxies.
## Discovery sources
The table shows which result types each source can add to consolidated CLI results. Legacy JSON and XML keep their existing schemas; breach names are retained in JSONL and SQLite. Some adapters parse fields that the reports do not store.
The table shows which result types each source can add to consolidated CLI results. XML keeps its existing schema. Legacy JSON now consolidates `interesting_urls`, `linkedin_links`, and `trello_urls` into one `urls` field. Breach names are retained in JSONL and SQLite. Some adapters parse fields that the reports do not store.
The report groups findings by result type. It does not record which source found each item. Empty optional fields may be omitted.
JSON and XML group findings by result type without source attribution. JSONL and SQLite retain source attribution when the collection adapter provides it. Empty optional fields may be omitted.
BuiltWith's normalized frameworks, languages, servers, CMS products, and analytics products are retained in JSONL and completed-result SQLite rows.
A checkmark means the source can add that result type. The **Separate output** column lists REST endpoints and optional actions that return other data.
A checkmark means the source can add that result type. The **Additional action output** column lists optional actions that return other data.
Read the **API key** column as follows:
@@ -132,13 +126,13 @@ Read the **API key** column as follows:
<details>
<summary><strong>View the source and result matrix</strong></summary>
| Source | Subdomains | Emails | IPs | ASNs | URLs / links | People | Breaches | Separate REST/action output (not consolidated report) | API key |
| Source | Subdomains | Emails | IPs | ASNs | URLs | People | Breaches | Additional action output (not consolidated report) | API key |
| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | --- | :---: |
| `arquivo` | ✓ | No | No | No | No | No | No | No | No |
| `baidu` | ✓ | ✓ | No | No | No | No | No | No | No |
| `bevigil` | ✓ | No | No | No | ✓ | No | No | No | ✓ |
| `bufferoverun` | ✓ | No | ✓ | No | No | No | No | No | ✓ |
| `builtwith` | ✓ | No | No | No | ✓ | No | No | `POST /additional/tech-stack` response | ✓ |
| `builtwith` | ✓ | No | No | No | ✓ | No | No | No | ✓ |
| `brave` | ✓ | ✓ | No | No | No | No | No | No | ✓ |
| `censys` | ✓ | ✓ | No | No | No | No | No | No | ✓ |
| `certspotter` | ✓ | No | No | No | No | No | No | No | No |
@@ -156,14 +150,14 @@ Read the **API key** column as follows:
| `github-code` | ✓ | ✓ | No | No | No | No | No | No | ✓ |
| `gitlab` | ✓ | ✓ | No | No | ✓ | No | No | No | No |
| `hackertarget` | ✓ | No | No | No | No | No | No | No | Optional |
| `haveibeenpwned` | No | No | No | No | No | No | ✓ | `POST /additional/breaches` response | No |
| `haveibeenpwned` | No | No | No | No | No | No | ✓ | No | No |
| `hibpverified` | No | ✓ | No | No | No | No | ✓ | No | ✓ |
| `hudsonrock` | ✓ | ✓ | ✓ | No | No | No | No | No | No |
| `hunter` | ✓ | ✓ | No | No | No | No | No | No | ✓ |
| `hunterhow` | ✓ | No | No | No | No | No | No | No | ✓ |
| `intelx` | ✓ | ✓ | No | No | ✓ | No | No | No | ✓ |
| `leakix` | ✓ | No | No | No | No | No | No | No | ✓ |
| `leaklookup` | No | ✓ | No | No | No | No | ✓ | `POST /additional/leaks` response | ✓ |
| `leaklookup` | No | ✓ | No | No | No | No | ✓ | No | ✓ |
| `mojeek` | ✓ | ✓ | No | No | No | No | No | No | Optional |
| `netlas` | ✓ | No | No | No | No | No | No | No | ✓ |
| `onyphe` | ✓ | No | ✓ | ✓ | No | No | No | No | ✓ |
@@ -173,7 +167,7 @@ Read the **API key** column as follows:
| `rapiddns` | ✓ | No | ✓ | No | No | No | No | No | No |
| `robtex` | ✓ | No | ✓ | No | No | No | No | No | No |
| `rocketreach` | No | ✓ | No | No | ✓ | No | No | No | ✓ |
| `securityscorecard` | ✓ | No | ✓ | No | No | No | No | `POST /additional/security-score` response | ✓ |
| `securityscorecard` | ✓ | No | ✓ | No | No | No | No | No | ✓ |
| `securityTrails` | ✓ | No | ✓ | No | No | No | No | No | ✓ |
| `sherlockeye` | ✓ | ✓ | ✓ | No | No | No | No | No | ✓ |
| `shodan` | ✓ | No | No | No | No | No | No | `-s` / `--shodan` host-enrichment output | ✓ |
@@ -195,9 +189,9 @@ Read the **API key** column as follows:
Provider pricing is intentionally omitted because plans and quotas change frequently. See [Configuration and API Keys](docs/wiki/Configuration-and-API-Keys.md) and each provider's current documentation.
`haveibeenpwned` remains the keyless public breach catalogue. `hibpverified` is a separate authenticated source for HIBP's `breachedDomain` endpoint. It participates in `all` and matching capability selectors just like every other P0 source, and skips normally when its provider key is absent. REST selections that include it require the operator `X-API-Key` when the provider key is configured and return normalized emails plus stable breach names. A live run requires a user-owned paid HIBP API key and a user-owned domain verified in that account; routine tests use offline responses.
`haveibeenpwned` remains the keyless public breach catalogue. `hibpverified` is a separate authenticated source for HIBP's `breachedDomain` endpoint. It participates in `all` and matching capability selectors just like every other P0 source, and skips normally when its provider key is absent. API run requests can select it through the shared source contract and return normalized emails plus stable breach names. A live run requires a user-owned paid HIBP API key and a user-owned domain verified in that account; routine tests use offline responses.
The runtime registry also reports the legacy identifiers `linkedin`, `linkedin_links`, `netcraft`, `omnisint`, `sublist3r`, and `zoomeyeapi`. These identifiers have no active CLI handlers. The table does not present them as usable sources.
The runtime registry also reports the legacy identifiers `linkedin`, `netcraft`, `omnisint`, `sublist3r`, and `zoomeyeapi`. These identifiers have no active CLI handlers. The table does not present them as usable sources.
## Configuration
@@ -214,14 +208,14 @@ Never commit populated configuration files, API keys, account details, or provid
- `-f NAME` writes `NAME.json`, `NAME.xml`, and `NAME.jsonl`.
- Screenshots are written to the directory passed to `--screenshot`.
- Host, email, IP, and related scan records are stored in `~/.local/share/theHarvester/stash.sqlite`.
- Full-pipeline runs are also stored transactionally by run UUID with their completed, deduplicated findings. Early REST returns and DNS-brute utility requests are not recorded as completed runs.
- REST queries return JSON.
- Full CLI pipeline runs are also stored transactionally by run UUID with their completed, deduplicated findings.
- API executions use the same SQLite database as CLI results. Durable lifecycle rows stay separate from terminal evidence, while typed results and source or action origins remain queryable. JSONL handles individual run interchange, and the API can import completed runs from another theHarvester SQLite database.
Treat collected OSINT as potentially sensitive. Keep report files, screenshots, and the local database out of source control and share them only within the authorized engagement.
### Report formats
The JSON report is a single object that preserves the legacy automation contract. Host entries remain plain hostnames or `hostname:address[,address...]` values when DNS resolution is enabled. DNS resolution and DNS brute force retain candidates only when A, AAAA, or CNAME evidence is available; CNAME-only candidates remain plain hostnames in existing CLI, REST, JSON, and XML output.
The JSON report is a single object. Host entries remain plain hostnames or `hostname:address[,address...]` values when DNS resolution is enabled. DNS resolution and DNS brute force retain candidates only when A, AAAA, or CNAME evidence is available; CNAME-only candidates remain plain hostnames in existing CLI, REST, JSON, and XML output.
`Checker.check()` and `DnsForce.run()` retain their existing `(resolved, hosts, addresses)` return shape. Normalized A, AAAA, and CNAME values are available through each object's `records` mapping.
@@ -231,7 +225,7 @@ The JSON report is a single object that preserves the legacy automation contract
| `hosts` | Always | Discovered hosts; an empty array when none are found. |
| `shodan` | Always | Shodan enrichment rows; an empty array when Shodan is not used. |
| `ips`, `emails`, `vhosts`, `asns` | When non-empty | Network and contact findings. |
| `interesting_urls`, `trello_urls`, `linkedin_links` | When non-empty | Discovered links and URLs. |
| `urls` | When non-empty | Discovered URLs from every URL-producing source or action. |
| `people`, `twitter_people`, `linkedin_people` | When non-empty | People and profile findings. |
| `takeover_results` | When non-empty | Optional takeover-check results. |
@@ -240,11 +234,11 @@ The XML report contains the command, emails, hosts, and virtual hosts. Use JSON
The JSONL report is finalized after the selected one-shot actions finish. The first line identifies the run with its UUID, target, UTC timestamps, and result counts. Each later line is one sorted, deduplicated finding. When you concatenate report files, treat each summary line as the start of a new run.
```jsonl
{"completed_at":"2026-08-07T12:01:00Z","counts":{"hostname":1},"result_count":1,"run_id":"123e4567-e89b-12d3-a456-426614174000","started_at":"2026-08-07T12:00:00Z","target":"example.com","type":"summary"}
{"action_executions":[],"artifacts":[],"completed_at":"2026-08-07T12:01:00Z","counts":{"hostname":1},"evidence_status":"complete","result_count":1,"run_id":"123e4567-e89b-12d3-a456-426614174000","source_executions":[],"started_at":"2026-08-07T12:00:00Z","target":"example.com","type":"summary"}
{"sources":[],"type":"hostname","value":"api.example.com"}
```
JSONL is easy to stream for simple findings, but it is not uniformly self-describing. Finding lines inherit their run ID and target from the preceding summary. Structured result types, including recursive DNS records plus `person`, `infostealer`, `shodan`, and `takeover`, store a JSON object inside the string `value`. Parse those values a second time with `fromjson`. JSONL does not include source execution records. Finding records include source attribution when it is available.
JSONL is easy to stream one record at a time. The summary preserves the evidence status, source and action outcomes, and screenshot artifact metadata. Finding lines carry `sources` and, when applicable, `actions`; they inherit their run ID and target from the preceding summary. Hostnames, IP addresses, and URLs use the same `hostname`, `ip`, and `url` result kinds in JSONL, SQLite, the API, and HarvestView. Provenance identifies which source or action produced each finding. Structured result types, including recursive DNS records plus `person`, `infostealer`, `shodan`, and `takeover`, store a JSON object inside the string `value`. Parse those values a second time with `fromjson`.
Parse recursive DNS findings as JSON objects:
docs/adr/0003-run-worker-lifecycle.md
+19
View File
@@ -0,0 +1,19 @@
# Keep run records separate with one isolated worker
Status: proposed
## Decision
The HTTP application owns a durable run record separate from theHarvester's optional terminal `RunResult` evidence. A submission receives its stable ID while queued. One local worker claims one queued run at a time and executes the finite theHarvester core in an isolated child process.
Lifecycle transitions are `queued -> running -> completed|failed`, `queued -> cancelled`, and `running -> cancelling -> cancelled`. A fixed whole-run deadline applies to the child. Running cancellation first requests cooperative termination, waits a short grace period, and then forces termination if needed. Queued cancellation is an atomic transition that prevents the worker claim.
Evidence already persisted remains attached after failure or cancellation. Terminal evidence status (`complete`, `partial`, or `failed`) is reported independently from orchestration lifecycle status. On service restart, queued runs may resume; orphaned running or cancelling records become failed because their process ownership cannot be proven.
## Why
The existing core is a finite one-shot enumerator and its result object is created only when execution begins. Reusing it as queue state would conflate operator intent, process ownership, cancellation acknowledgement, and evidence quality. A single worker matches the local single-operator product, avoids concurrent output and credential contention, and can be widened later only if measured demand justifies it.
## Consequences
Run records need one small SQLite table and a lifecycle API. Child-process boundaries make deadline and forced cancellation reliable across blocking provider code. Work is serialized by design. Imported evidence enters as an already completed run record and never enters the queue.
docs/wiki/Configuration-and-API-Keys.md
+4 -4
View File
@@ -42,7 +42,7 @@ The [README source matrix](https://github.com/laramies/theHarvester/blob/dev/REA
Provider pricing, quotas, and terms change frequently. Check the provider's current documentation for these details.
`hibpverified` queries [HIBP's authenticated verified-domain endpoint](https://haveibeenpwned.com/API/v3#BreachedDomain) only when explicitly named, either alone or in a combination such as `breaches,hibpverified`. Capability selectors and `all` exclude it. Live use requires a user-owned paid HIBP API key and a user-owned domain verified in that account. REST queries selecting it also require the operator `X-API-Key`; the keyless `haveibeenpwned` source continues to query only the public breach catalogue.
`hibpverified` queries [HIBP's authenticated verified-domain endpoint](https://haveibeenpwned.com/API/v3#BreachedDomain). It is selected by its name, the `breaches` capability, and `all`. Without a configured HIBP API key it is skipped like other unavailable keyed sources. Live use requires a user-owned paid HIBP API key and a user-owned domain verified in that account. The keyless `haveibeenpwned` source continues to query only the public breach catalogue.
## Proxies
@@ -63,13 +63,13 @@ uv run theHarvester -d example.com -b crtsh -p
A proxy does not make an assessment anonymous and does not change the authorization boundary.
## REST API protection
## API protection
The `/additional/*` routes require a server-side key:
Every `/api/v1/*` route requires a server-side key:
```bash
export THEHARVESTER_API_KEY='replace-with-a-long-random-value'
uv run restfulHarvest
```
Clients send the same value in the `X-API-Key` header. This key protects only `/additional/*`; the core query routes remain unauthenticated.
API clients send the same value in the `X-API-Key` header. Provider credentials remain in `api-keys.yaml` and cannot be supplied through an API request.
docs/wiki/How-to-add-a-new-module.md
+1 -1
View File
@@ -21,7 +21,7 @@ An adapter normally provides:
- an initializer for the target and local result sets;
- an asynchronous `process()` method;
- only the getters it actually supports, such as `get_hostnames()`, `get_emails()`, `get_ips()`, `get_asns()`, `get_interesting_urls()`, or `get_results()`.
- only the getters it actually supports, such as `get_hostnames()`, `get_emails()`, `get_ips()`, `get_asns()`, `get_urls()`, or `get_results()`.
Do not return fields the provider did not supply. Normalize and deduplicate before returning results.
docs/wiki/Operator-Workflows.md
+2 -2
View File
@@ -25,14 +25,14 @@ AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -r
```
To control the resolvers used:
To control the resolvers used, create a resolver file with one IP address per line and pass its path:
```bash
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh -r resolvers.txt
```
Resolver files contain one IP address per line. DNS requests disclose candidate names to the selected resolver.
DNS requests disclose candidate names to each selected resolver.
## Shodan enrichment
docs/wiki/Quick-Start.md
+1 -1
View File
@@ -35,7 +35,7 @@ AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -r
```
Pass a resolver IP, comma-separated resolver IPs, or a file containing one resolver IP per line:
Pass a resolver IP, comma-separated resolver IPs, or a resolver file you create with one IP per line:
```bash
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
docs/wiki/Responsible-Use-and-Scope.md
+5 -3
View File
@@ -22,6 +22,8 @@ The following options require additional care:
| `--screenshot DIR` | Opens discovered web services in a browser. |
| `-a`, `--api-scan` | Requests common API paths from the target. |
Use `--dns-resolvers IPS_OR_FILE` to select resolver addresses for DNS brute force, reverse lookup, or recursive DNS without also enabling hostname resolution. The compatible `--dns-resolve` value still selects resolvers and enables hostname resolution.
Use an owned or explicitly authorized domain for active examples. Do not substitute universities, public companies, bounty targets, or reserved example domains for recurring active scans.
## Protect collected data
@@ -33,8 +35,8 @@ Results may contain private infrastructure, employee addresses, account identifi
- Redact credentials, private target data, account information, and unnecessary response content before filing an issue.
- Never publish raw provider responses merely to demonstrate a parsing or availability problem.
## Service exposure
## API exposure
The `restfulHarvest` core query routes do not require authentication. `THEHARVESTER_API_KEY` protects the optional `/additional/*` routes only. It does not protect `/query`, `/sources`, or `/dnsbrute`.
Every `/api/v1/*` route requires `THEHARVESTER_API_KEY`. Provider credentials remain server-side.
Keep the service on localhost. If you require remote access, add authentication, network controls, and TLS.
Keep the service on localhost. If you require remote access, add network controls and TLS in front of the existing API authentication.
docs/wiki/Rest-API.md
+108 -88
View File
@@ -1,25 +1,17 @@
# REST API
`restfulHarvest` runs a FastAPI service for local automation and interactive Swagger/ReDoc documentation.
`restfulHarvest` serves one versioned API for local automation.
## Start the service
Set a long random API key before startup:
```bash
export THEHARVESTER_API_KEY='replace-with-a-long-random-value'
uv run restfulHarvest
```
Defaults:
- host: `127.0.0.1`
- port: `5000`
- log level: `info`
- rate limit: `5/minute` per client address
Use `uv run restfulHarvest -h` for current launcher options. For example:
```bash
uv run restfulHarvest --rate-limit 10/minute
```
The service binds to `127.0.0.1:5000` by default. Use `uv run restfulHarvest -h` for launcher options.
Open:
@@ -28,104 +20,132 @@ Open:
Treat the runtime OpenAPI document as the exact request and response reference.
## Core routes
## Routes
| Route | Purpose |
| --- | --- |
| `GET /sources` | List current discovery sources. |
| `GET /query` | Run selected discovery sources and return consolidated JSON. |
| `GET /dnsbrute` | Run active DNS brute force for an authorized domain. |
| `GET /runs` | List recent completed enumeration runs. |
| `GET /runs/{run_id}` | Retrieve one completed run and its normalized evidence. |
| `GET /api/v1/sources` | List discovery sources, capabilities, activity classes, and credential names. |
| `POST /api/v1/runs` | Submit one finite enumeration run. |
| `GET /api/v1/runs` | List run records with `limit` and `offset` pagination. |
| `GET /api/v1/runs/{run_id}` | Retrieve lifecycle state, options, results, source outcomes, and artifacts. |
| `POST /api/v1/runs/{run_id}/cancel` | Cancel queued work or request cancellation of running work. |
| `POST /api/v1/runs/import` | Import a JSONL result file without executing discovery. |
| `POST /api/v1/runs/import-database` | Import completed runs from a theHarvester SQLite database. |
| `GET /api/v1/runs/{run_id}/export` | Export normalized results as JSONL. |
| `GET /api/v1/runs/{run_id}/screenshots/{name}` | Retrieve one managed screenshot. |
List sources:
There are no provider-specific routes. Sources such as `builtwith`, `haveibeenpwned`, `hibpverified`, `leaklookup`, and `securityscorecard` use the same run request as every other source.
The versioned API is asynchronous by design. A successful submission returns a durable run record instead of waiting for every provider and action to finish.
## Authentication
Every `/api/v1/*` route requires the configured key in `X-API-Key`:
```bash
curl -s http://127.0.0.1:5000/sources | jq -r '.sources[]'
```
Run a passive query:
```bash
curl -sG http://127.0.0.1:5000/query \
--data-urlencode 'domain=example.com' \
--data-urlencode 'source=crtsh' \
--data-urlencode 'source=certspotter' \
| jq
```
The `source` parameter also accepts the same capability selectors as the CLI:
`subdomains`, `emails`, `ips`, `asns`, `urls`, `people`, and `breaches`.
Repeat `source` to combine capabilities with explicit source names. Selection is
a union and does not filter fields returned by a selected source.
```bash
curl -sG http://127.0.0.1:5000/query \
--data-urlencode 'domain=example.com' \
--data-urlencode 'source=emails' \
--data-urlencode 'source=certspotter' \
| jq
```
A completed `/query` also retains its normalized terminal record in the local
SQLite database. No JSON, XML, or JSONL report file is written unless `filename`
is supplied.
HIBP verified-domain participates in `all`, `emails`, and `breaches` selections.
When its provider key is configured, any selection that includes it also requires
the operator API key:
```bash
curl -sG http://127.0.0.1:5000/query \
-H "X-API-Key: $THEHARVESTER_API_KEY" \
--data-urlencode "domain=$VERIFIED_DOMAIN" \
--data-urlencode 'source=hibpverified' \
| jq '{emails, breaches}'
```
Completed-run routes require the operator API key because retained evidence can
contain sensitive results:
```bash
curl -s http://127.0.0.1:5000/runs \
curl -s http://127.0.0.1:5000/api/v1/sources \
-H "X-API-Key: $THEHARVESTER_API_KEY" \
| jq
```
## Additional API routes
Provider credentials remain in theHarvester's server-side configuration. Requests cannot supply provider API keys.
The following `POST /additional/*` routes provide optional breach, leak, security-score, and technology-stack lookups:
## Submit and inspect a run
- `/additional/breaches`
- `/additional/leaks`
- `/additional/security-score`
- `/additional/tech-stack`
- `/additional/all`
Set a server key before startup:
Source names and capability selectors share the `sources` array. Multiple capabilities select the union of matching sources and do not filter fields returned by those sources.
```bash
export THEHARVESTER_API_KEY='replace-with-a-long-random-value'
uv run restfulHarvest
run_id="$(curl -s http://127.0.0.1:5000/api/v1/runs \
-X POST \
-H "X-API-Key: $THEHARVESTER_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
"target": "example.com",
"sources": ["emails", "crtsh"],
"limit": 500,
"deadline_seconds": 1800
}' \
| jq -r '.run_id')"
curl -s "http://127.0.0.1:5000/api/v1/runs/$run_id" \
-H "X-API-Key: $THEHARVESTER_API_KEY" \
| jq '{status, evidence_status, results, source_executions, action_executions, artifacts}'
```
Send that value in `X-API-Key`:
Run submission is asynchronous. Lifecycle status is `queued`, `running`, `cancelling`, `cancelled`, `completed`, or `failed`. Terminal evidence status is reported separately as `complete`, `partial`, or `failed` when evidence exists.
P1 DNS and P2 direct options are fields on the same run request. The OpenAPI schema shows their current defaults, limits, and descriptions. The server uses the operator-selected target and does not impose a public-only egress policy.
### Run an action against one result
Screenshots and DNS brute force can run directly against an authorized hostname without repeating discovery. Submit an empty `sources` array and select one action:
```bash
curl -s http://127.0.0.1:5000/additional/tech-stack \
curl -s http://127.0.0.1:5000/api/v1/runs \
-X POST \
-H "X-API-Key: $THEHARVESTER_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
"target": "subdomain.example.com",
"sources": [],
"screenshot": true
}' \
| jq
```
For DNS brute force, set `dns_brute` to `true`. You may also provide `dns_resolvers` as one or more distinct IPv4 or IPv6 addresses. Recursive DNS is the only action that requires exactly three resolver addresses.
The action catalog and run request use the same names. For example, set `takeover` to `true` for takeover checks. API endpoint scans can use the bundled paths or an explicit bounded list:
```json
{
"target": "api.example.com",
"sources": [],
"api_scan": true,
"api_scan_paths": ["/api/v2", "/health"]
}
```
Every custom API scan entry must be a URL path beginning with `/`. The API does not accept a server-side file path.
## Import and export
Import records existing evidence and never contacts the target. For one run, send the same JSONL written by `theHarvester -f NAME`:
```bash
curl -s "http://127.0.0.1:5000/api/v1/runs/import?filename=report.jsonl" \
-X POST \
-H "X-API-Key: $THEHARVESTER_API_KEY" \
-H 'Content-Type: application/json' \
-d '{"domain":"example.com"}' \
-H 'Content-Type: application/x-ndjson' \
--data-binary @report.jsonl \
| jq
```
These routes may also require provider credentials in the request body or local configuration. Consult `/docs` for the current schema.
JSONL is a terminal report, so an import is recorded as completed. The summary retains evidence status, source and action outcomes, and screenshot artifact metadata. Each finding's `sources` and `actions` arrays rebuild result attribution and must name an execution in the summary. Result kinds such as `hostname`, `ip`, and `url` use the same names in JSONL, SQLite, and the API.
To load every completed run from another theHarvester database:
```bash
curl -s "http://127.0.0.1:5000/api/v1/runs/import-database?filename=stash.sqlite" \
-X POST \
-H "X-API-Key: $THEHARVESTER_API_KEY" \
-H 'Content-Type: application/vnd.sqlite3' \
--data-binary @stash.sqlite \
| jq
```
The server checks the SQLite header, integrity, schema, and each completed run before copying it. Original run IDs are preserved. Exact duplicates are skipped, while a reused ID with different evidence is rejected. Close the source process or checkpoint its WAL before uploading the database. Screenshot metadata is imported, but screenshot files must be copied separately. The default upload ceiling is 1 GiB and can be changed with `THEHARVESTER_MAX_DATABASE_IMPORT_BYTES`.
Export one normalized result set in the same streamable format:
```bash
curl -s "http://127.0.0.1:5000/api/v1/runs/$run_id/export" \
-H "X-API-Key: $THEHARVESTER_API_KEY" \
-o results.jsonl
```
The first line is the `summary` record, including evidence status, source and action outcomes, and artifacts. Each remaining line is one normalized finding with `type`, `value`, `sources`, and optional `actions`. This keeps the file easy to stream with `jq -c` and makes API exports importable again without a format conversion. Lifecycle details and the submitted request remain available from `GET /api/v1/runs/{run_id}`.
## Security boundary
`THEHARVESTER_API_KEY` protects `/additional/*`, `/runs*`, and `/query` selections that include a configured `hibpverified` source. Other `/query` requests, `/sources`, and `/dnsbrute` remain unauthenticated.
Keep the default localhost binding. If you require remote access, add authentication, network allowlists, TLS, request logging, and an appropriate rate limit.
The supplied Docker Compose configuration binds host port `5000` on every interface unless you narrow the mapping.
Keep the default localhost binding. If remote access is required, add TLS, network access controls, request logging, and an appropriate rate limit. The supplied Docker Compose configuration publishes only to `127.0.0.1` by default.
docs/wiki/Results-and-Local-Data.md
+20 -6
View File
@@ -1,6 +1,6 @@
# Results and local data
theHarvester can print findings, write reports, retain selected records in SQLite, save screenshots, and return REST JSON. These outputs have different schemas and sensitivity.
theHarvester can print findings, write reports, retain selected records in SQLite, save screenshots, and expose durable run records through the API. These outputs have different schemas and sensitivity.
## Terminal output
@@ -34,17 +34,31 @@ Host, email, IP, and related records are stored at:
The database persists across runs. Account for it in engagement cleanup and retention procedures.
Completed CLI and REST `/query` executions also store one normalized terminal
record keyed by run UUID. REST keeps its existing response shape and does not
write report files unless a filename is requested.
Completed CLI executions store one normalized terminal record keyed by run UUID. API executions use the same database by default and may override its path with `THEHARVESTER_RUN_DB`. Lifecycle rows keep queue, cancellation, and worker state separate from terminal evidence. Imported JSONL is stored without executing discovery, and source attribution is rebuilt from each finding's `sources` array. A SQLite import copies every completed run after validating the database and keeps the original run IDs.
The normalized persistence model can represent active-action provenance and artifact metadata through five core tables:
- `runs`: one finite enumeration run;
- `executions`: each passive source or active action represented by the model;
- `results`: deduplicated hostnames, IPs, emails, URLs, and structured outputs;
- `result_origins`: which execution produced each result; and
- `artifacts`: files such as screenshots, linked to their creating action and subject result.
Current runtime collection populates passive source executions plus DNS, takeover, Shodan, and API endpoint scan executions and origins. Screenshot actions attach file metadata to their captured hostname or URL without creating fake screenshot findings.
Every discovered URL is stored as the `url` result kind. Its source or action origins identify whether it came from BuiltWith, GitLab, RocketReach, API scanning, or another producer; provider-specific URL kinds are not stored.
Hostname and IP evidence use the `hostname` and `ip` result kinds in SQLite, JSONL, the API, and HarvestView. A hostname may be the authorized target itself or a subordinate name, so the result kind does not claim that every value is a subdomain.
Two operational tables support the API without changing those five evidence concepts: `run_records` stores queue and lifecycle state, and `run_worker_leases` prevents two local workers from claiming the same queue. Older runless rows remain in `legacy_observations`. SQLite upgrades supported schemas automatically during normal initialization.
## Screenshots
`--screenshot DIR` writes browser captures to the selected directory. Screenshots may contain authentication pages, internal names, or other sensitive visual data even when no credentials were used.
## REST JSON
## API results
The REST `/query` response returns arrays for ASNs, interesting URLs, Twitter/LinkedIn data, Trello URLs, IPs, emails, and hosts. The corresponding normalized terminal record is retained in SQLite. Treat runtime `/docs`, `/redoc`, and OpenAPI as the exact request/response reference.
`GET /api/v1/runs/{run_id}` returns lifecycle state plus a normalized `results` array. Each result has `type`, `value`, `sources`, and `actions`. Run-level source and action outcomes remain available in `source_executions` and `action_executions`, while file metadata is returned through `artifacts`. JSONL imports or exports one run, and SQLite import loads completed runs in bulk. Treat runtime `/docs`, `/redoc`, and OpenAPI as the exact request and response reference.
## Handling and sharing
docs/wiki/Troubleshooting.md
+6 -6
View File
@@ -53,7 +53,7 @@ Do not post credentials, private targets, account details, or raw provider respo
## DNS resolution
`-r` accepts no value, a resolver IP, comma-separated resolver IPs, or a file with one IP per line:
`-r` accepts no value, a resolver IP, comma-separated resolver IPs, or a resolver file you create with one IP per line:
```bash
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
@@ -82,10 +82,10 @@ uv run restfulHarvest --log-level debug
Then open [http://127.0.0.1:5000/docs](http://127.0.0.1:5000/docs).
- `401` on `/additional/*`: the `X-API-Key` header is absent or does not match.
- `503` on `/additional/*`: `THEHARVESTER_API_KEY` was not configured before startup.
- `429`: the client exceeded the configured API rate limit.
- Core routes are intentionally not protected by that key; do not expose the service directly.
- `401` on `/api/v1/*`: the `X-API-Key` header does not match.
- `503` on `/api/v1/*`: `THEHARVESTER_API_KEY` was not configured before startup.
- `429`: a reverse proxy or remote provider applied its own rate limit. `restfulHarvest` has no built-in request limiter.
- `503` when creating a run: the execution worker is disabled or unavailable.
## Docker
@@ -94,7 +94,7 @@ docker compose ps
docker compose logs theharvester.svc.local
```
The container runs the REST API on container port `80`, published as host port `5000` by the supplied Compose file.
The container runs the REST API on container port `8000`, published as host port `5000` by the supplied Compose file.
## File an actionable issue
pyproject.toml
-1
View File
@@ -38,7 +38,6 @@ dependencies = [
"httpx==0.28.1",
"retrying==1.4.2",
"shodan==1.31.0",
"slowapi==0.1.10",
"sqlalchemy==2.0.51",
"ujson==5.13.0",
"uvicorn==0.49.0",
tests/discovery/test_api_endpoints.py
+103
View File
@@ -4,6 +4,7 @@ import aiohttp
import pytest
from theHarvester.discovery import api_endpoints
from theHarvester.lib.core import FetcherResponse
class FakeResponse:
@@ -62,6 +63,7 @@ async def test_api_endpoint_scan_uses_only_observational_http_methods(monkeypatc
methods = []
monkeypatch.setattr(search, '_load_wordlist', lambda: [])
async def detect_schema():
return 'https'
@@ -77,6 +79,107 @@ async def test_api_endpoint_scan_uses_only_observational_http_methods(monkeypatc
assert methods == ['GET', 'HEAD', 'OPTIONS']
@pytest.mark.asyncio
async def test_api_endpoint_scan_uses_only_the_configured_wordlist(monkeypatch, tmp_path) -> None:
wordlist = tmp_path / 'operator-paths.txt'
wordlist.write_text('/health\n', encoding='utf-8')
search = api_endpoints.SearchApiEndpoints('example.com', wordlist=str(wordlist), exact_paths=True)
detected_paths: list[str] = []
requested_urls: list[str] = []
async def detect_schema(path: str = '') -> str:
detected_paths.append(path)
return 'https'
async def fetch(url: str, *_args, **_kwargs):
requested_urls.append(url)
return ''
monkeypatch.setattr(search, '_detect_schema', detect_schema)
monkeypatch.setattr(api_endpoints.AsyncFetcher, 'fetch', fetch)
await search.do_search()
assert detected_paths == ['/health']
assert requested_urls == ['https://example.com/health'] * 3
@pytest.mark.asyncio
async def test_schema_detection_can_probe_an_exact_listed_path() -> None:
search = api_endpoints.SearchApiEndpoints('example.com', exact_paths=True)
session = FakeSession()
search._session = session
assert await search._detect_schema('/health') == 'https'
assert session.requests[0][0] == 'https://example.com/health'
@pytest.mark.asyncio
async def test_api_endpoint_scan_exposes_shared_fetcher_metadata(monkeypatch: pytest.MonkeyPatch) -> None:
search = api_endpoints.SearchApiEndpoints('example.com')
search.common_api_paths = ['/api']
monkeypatch.setattr(search, '_load_wordlist', lambda: [])
async def detect_schema() -> str:
return 'https'
async def fetch(*_args, **kwargs):
assert kwargs['include_metadata'] is True
return FetcherResponse(
body='{"status":"ok"}',
status=200,
headers={'content-type': 'application/json'},
)
monkeypatch.setattr(search, '_detect_schema', detect_schema)
monkeypatch.setattr(api_endpoints.AsyncFetcher, 'fetch', fetch)
await search.do_search()
result = search.get_found_endpoints()['https://example.com/api']
assert result.status_code == 200
assert result.method == 'GET'
assert result.content_type == 'application/json'
assert result.content_length == len('{"status":"ok"}')
assert result.content_preview == '{"status":"ok"}'
@pytest.mark.asyncio
async def test_api_endpoint_scan_counts_suppressed_request_failures(monkeypatch: pytest.MonkeyPatch) -> None:
search = api_endpoints.SearchApiEndpoints('example.com')
search.common_api_paths = ['/api']
monkeypatch.setattr(search, '_load_wordlist', lambda: [])
async def detect_schema() -> str:
return 'https'
async def fail_request(*_args, **kwargs):
assert kwargs['include_metadata'] is True
return None
monkeypatch.setattr(search, '_detect_schema', detect_schema)
monkeypatch.setattr(api_endpoints.AsyncFetcher, 'fetch', fail_request)
await search.do_search()
assert search.request_error_count == 3
assert search.request_error_types == {'TransportError'}
assert search.get_found_endpoints() == {}
@pytest.mark.asyncio
async def test_api_endpoint_scan_reports_suppressed_top_level_failure(monkeypatch: pytest.MonkeyPatch) -> None:
search = api_endpoints.SearchApiEndpoints('example.com')
async def fail_schema_detection() -> str:
raise RuntimeError('scan setup failed')
monkeypatch.setattr(search, '_detect_schema', fail_schema_detection)
assert await search.do_search() is None
assert search.scan_error_type == 'RuntimeError'
@pytest.mark.asyncio
async def test_api_endpoint_scan_allows_an_operator_selected_private_target(monkeypatch) -> None:
search = api_endpoints.SearchApiEndpoints('100.64.0.1')
tests/discovery/test_builtwith.py
+6 -6
View File
@@ -13,12 +13,12 @@ if 'aiohttp_socks' not in sys.modules:
def from_url(*_args, **_kwargs):
return None
setattr(aiohttp_socks_stub, 'ProxyConnector', _ProxyConnector)
aiohttp_socks_stub.ProxyConnector = _ProxyConnector # type: ignore[attr-defined]
sys.modules['aiohttp_socks'] = aiohttp_socks_stub
from theHarvester import __main__ as theharvester_main
from theHarvester.discovery import builtwith
from theHarvester.discovery.constants import MissingKey
from theHarvester import __main__ as theharvester_main
from theHarvester.lib.completed_result import CompletedResult
@@ -87,7 +87,7 @@ async def test_process_accepts_text_json_content_type(monkeypatch) -> None:
await search.process()
assert await search.get_hostnames() == {'sub.example.com'}
assert await search.get_interesting_urls() == {'https://example.com/login'}
assert await search.get_urls() == {'https://example.com/login'}
assert await search.get_frameworks() == {'Django'}
assert await search.get_languages() == {'Python'}
assert await search.get_servers() == {'nginx'}
@@ -157,7 +157,7 @@ async def test_normalized_builtwith_results_reach_completed_jsonl(
async def get_hostnames(self) -> set[str]:
return set()
async def get_interesting_urls(self) -> set[str]:
async def get_urls(self) -> set[str]:
return {'https://example.com/login'}
async def get_frameworks(self) -> set[str]:
@@ -188,12 +188,12 @@ async def test_normalized_builtwith_results_reach_completed_jsonl(
('analytics', 'Google Analytics'),
('cms', 'WordPress'),
('framework', 'Django'),
('interesting-url', 'https://example.com/login'),
('language', 'Python'),
('server', 'nginx'),
('url', 'https://example.com/login'),
)
records = [json.loads(line) for line in report.with_suffix('.jsonl').read_text().splitlines()]
assert {'type': 'interesting-url', 'value': 'https://example.com/login', 'sources': ['builtwith']} in records
assert {'type': 'url', 'value': 'https://example.com/login', 'sources': ['builtwith']} in records
assert {'type': 'framework', 'value': 'Django', 'sources': ['builtwith']} in records
assert {'type': 'language', 'value': 'Python', 'sources': ['builtwith']} in records
assert {'type': 'server', 'value': 'nginx', 'sources': ['builtwith']} in records
tests/discovery/test_haveibeenpwned.py
-16
View File
@@ -8,7 +8,6 @@ import pytest
from theHarvester.discovery import haveibeenpwned
from theHarvester import __main__ as theharvester_main
from theHarvester.lib.api import additional_endpoints
from theHarvester.lib.completed_result import CompletedResult
from theHarvester.lib.core import FetcherResponse
@@ -113,21 +112,6 @@ async def test_public_breach_catalog_attributes_http_failures(
assert 'HaveIBeenPwned request failed with HTTP 429' in caplog.text
@pytest.mark.asyncio
async def test_breach_rest_handler_does_not_initialize_unrelated_providers(monkeypatch: pytest.MonkeyPatch) -> None:
async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]:
return [FetcherResponse(body=[{'Domain': 'example.com'}], status=200, headers={})]
monkeypatch.setattr(haveibeenpwned.AsyncFetcher, 'fetch_all', fake_fetch_all)
result = await additional_endpoints.get_breaches(
additional_endpoints.DomainRequest(domain='example.com'),
_api_key='local-api-key',
)
assert result == {'status': 'success', 'data': [{'Domain': 'example.com'}]}
@pytest.mark.asyncio
async def test_public_breach_names_reach_completed_result_and_jsonl(
monkeypatch: pytest.MonkeyPatch,
tests/discovery/test_intelxsearch.py
+5 -7
View File
@@ -59,6 +59,8 @@ async def test_process_exposes_flat_normalized_in_scope_results(monkeypatch: pyt
{'selectorvalue': 'https://portal.example.com/path'},
{'selectorvalue': 'api.example.com.'},
{'selectorvalue': 'foo.example.com.evil'},
{'selectorvalue': 'https://foo.example.com.evil/path'},
{'selectorvalue': 'ftp://api.example.com/archive'},
{'selectorvalue': 'http://['},
{'selectorvalue': None},
None,
@@ -78,11 +80,7 @@ async def test_process_exposes_flat_normalized_in_scope_results(monkeypatch: pyt
assert await search.get_emails() == ['admin@example.com']
assert await search.get_hostnames() == ['api.example.com', 'portal.example.com']
assert await search.get_interestingurls() == [
'api.example.com.',
'foo.example.com.evil',
'https://portal.example.com/path',
]
assert await search.get_urls() == ['https://portal.example.com/path']
@pytest.mark.asyncio
@@ -112,7 +110,7 @@ async def test_process_treats_denied_and_malformed_responses_as_empty(
assert await search.get_emails() == []
assert await search.get_hostnames() == []
assert await search.get_interestingurls() == []
assert await search.get_urls() == []
@pytest.mark.asyncio
@@ -139,7 +137,7 @@ async def test_orchestrator_stores_intelx_subdomains_without_dns(monkeypatch: py
async def get_emails(self) -> list[str]:
return []
async def get_interestingurls(self) -> list[str]:
async def get_urls(self) -> list[str]:
return []
class _UnexpectedChecker:
tests/discovery/test_leaklookup.py
+1 -73
View File
@@ -6,10 +6,9 @@ from typing import Any
import pytest
from theHarvester.discovery import additional_apis, leaklookup
from theHarvester.discovery import leaklookup
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.completed_result import CompletedResult
from theHarvester.lib.api import additional_endpoints
from theHarvester.lib.core import FetcherResponse
from theHarvester import __main__ as theharvester_main
@@ -150,77 +149,6 @@ async def test_leaklookup_empty_success_returns_no_results(monkeypatch) -> None:
assert await search.get_breach_names() == set()
@pytest.mark.asyncio
async def test_additional_leaks_endpoint_uses_only_leaklookup(monkeypatch) -> None:
class FakeSearchLeakLookup:
def __init__(self, domain: str) -> None:
assert domain == 'example.test'
async def process(self) -> None:
return None
async def get_leaks(self) -> list[dict[str, str]]:
return [{'breach': 'Example Breach', 'email': 'alice@example.test'}]
class UnexpectedAdditionalAPIs:
def __init__(self, *_args: object, **_kwargs: object) -> None:
raise AssertionError('unrelated providers must not be initialized')
monkeypatch.setattr(additional_endpoints, 'SearchLeakLookup', FakeSearchLeakLookup, raising=False)
monkeypatch.setattr(additional_endpoints, 'AdditionalAPIs', UnexpectedAdditionalAPIs)
result = await additional_endpoints.get_leaks(
additional_endpoints.DomainRequest(domain='example.test'),
_api_key='local-api-key',
)
assert result == {
'status': 'success',
'data': [{'breach': 'Example Breach', 'email': 'alice@example.test'}],
}
@pytest.mark.asyncio
async def test_missing_leaklookup_key_does_not_break_security_score_endpoint(monkeypatch) -> None:
class PassiveProvider:
def __init__(self, _domain: str) -> None:
return None
class FakeSecurityScorecard(PassiveProvider):
score = 95
grades = {'network': 'A'}
issues: list[object] = []
recommendations: list[object] = []
hosts: set[str] = set()
async def process(self, _proxy: bool) -> None:
return None
class UnexpectedLeakLookup:
def __init__(self, _domain: str) -> None:
raise AssertionError('Leak-Lookup must be constructed only when requested')
monkeypatch.setattr(additional_apis, 'SearchHaveIBeenPwned', PassiveProvider)
monkeypatch.setattr(additional_apis, 'SearchBuiltWith', PassiveProvider)
monkeypatch.setattr(additional_apis, 'SearchSecurityScorecard', FakeSecurityScorecard)
monkeypatch.setattr(additional_apis, 'SearchLeakLookup', UnexpectedLeakLookup)
result = await additional_endpoints.get_security_score(
additional_endpoints.DomainRequest(domain='example.test'),
_api_key='local-api-key',
)
assert result == {
'status': 'success',
'data': {
'score': 95,
'grades': {'network': 'A'},
'issues': [],
'recommendations': [],
},
}
@pytest.mark.asyncio
async def test_leaklookup_emails_and_breaches_reach_completed_result_and_jsonl(monkeypatch, tmp_path: Path) -> None:
completed_results: list[CompletedResult] = []
tests/discovery/test_rapiddns.py
+44 -15
View File
@@ -154,15 +154,18 @@ async def test_rapiddns_evidence_reaches_existing_outputs(
def __init__(self, *, word: str, wordlist: str) -> None:
assert word == 'example.com'
assert wordlist.endswith('api_endpoints.txt')
self.scan_error_type = None
self.request_error_count = 0
self.request_error_types: set[str] = set()
async def do_search(self) -> None:
return None
def get_found_endpoints(self) -> dict[str, object]:
return {'/health': object()}
return {'https://example.com/health': object()}
def get_interesting_endpoints(self) -> dict[str, object]:
return {'/health': object()}
return {'https://example.com/health': object()}
def get_auth_required(self) -> dict[str, object]:
return {}
@@ -198,10 +201,14 @@ async def test_rapiddns_evidence_reaches_existing_outputs(
iprange: str,
callback: Any,
nameservers: list[str] | None = None,
error_types: set[str] | None = None,
) -> None:
assert iprange in {'192.0.2.0/24', '198.51.100.0/24', '2001:d00::/24'}
assert nameservers is None
callback('reverse.example.com')
if iprange == '198.51.100.0/24':
assert error_types is not None
error_types.add('TimeoutError')
report = tmp_path / 'rapiddns-report'
monkeypatch.setattr(rapiddns.AsyncFetcher, 'fetch_all', fake_fetch_all)
@@ -232,12 +239,15 @@ async def test_rapiddns_evidence_reaches_existing_outputs(
await theharvester_main.start(completed_result_checkpoint=capture_checkpoint)
assert exit_info.value.code == 0
assert stored.count(('api-endpoint', ('/health',), 'api_scan')) == 1
assert stored == []
report_json = json.loads(report.with_suffix('.json').read_text())
assert report_json['hosts'] == ['alias.example.com', 'api.example.com', 'broken.example.com']
assert report_json['ips'] == ['192.0.2.1', '198.51.100.9', '2001:db8::1']
assert report_json['urls'] == ['https://example.com/health']
assert 'interesting_urls' not in report_json
assert 'linkedin_links' not in report_json
assert 'trello_urls' not in report_json
jsonl_records = [json.loads(line) for line in report.with_suffix('.jsonl').read_text().splitlines()]
assert jsonl_records[0]['type'] == 'summary'
@@ -246,24 +256,43 @@ async def test_rapiddns_evidence_reaches_existing_outputs(
assert [str(result.run_id) for result in completed_results] == [jsonl_records[0]['run_id']]
assert checkpoints
assert {result.run_id for result in checkpoints} == {completed_results[0].run_id}
assert {'type': 'interesting-url', 'value': 'https://example.com/health', 'sources': []} in jsonl_records
assert {'type': 'url', 'value': 'https://example.com/health', 'sources': []} in jsonl_records
assert {'type': 'hostname', 'value': 'reverse.example.com', 'sources': []} in jsonl_records
assert {'type': 'ip-address', 'value': '198.51.100.9', 'sources': ['securityscorecard']} in jsonl_records
assert {'type': 'ip-address', 'value': '2001:db8::1', 'sources': ['securityscorecard']} in jsonl_records
assert {'type': 'url', 'value': 'https://example.com/health', 'sources': [], 'actions': ['api-scan']} in jsonl_records
assert {
'type': 'hostname',
'value': 'reverse.example.com',
'sources': [],
'actions': ['dns-lookup'],
} in jsonl_records
assert {'type': 'ip', 'value': '198.51.100.9', 'sources': ['securityscorecard']} in jsonl_records
assert {'type': 'ip', 'value': '2001:db8::1', 'sources': ['securityscorecard']} in jsonl_records
assert not any(record.get('value') == 'not-an-ip' for record in jsonl_records)
assert {(observation.source, observation.kind, observation.value) for observation in completed_results[0].observations} >= {
('rapiddns', 'hostname', 'api.example.com'),
('rapiddns', 'ip-address', '192.0.2.1'),
('securityscorecard', 'ip-address', '198.51.100.9'),
('securityscorecard', 'ip-address', '2001:db8::1'),
('rapiddns', 'ip', '192.0.2.1'),
('securityscorecard', 'ip', '198.51.100.9'),
('securityscorecard', 'ip', '2001:db8::1'),
}
securityscorecard_execution = next(
execution for execution in completed_results[0].source_executions if execution.source == 'securityscorecard'
)
assert securityscorecard_execution.status == 'completed'
assert securityscorecard_execution.result_count == 2
reverse_execution = next(
execution for execution in completed_results[0].active_evidence.executions if execution.action == 'dns-lookup'
)
assert reverse_execution.status == 'partial'
assert reverse_execution.error_type == 'TimeoutError'
assert reverse_execution.stop_reason == 'query-errors'
assert {(observation.kind, observation.value) for observation in reverse_execution.observations} == {
('hostname', 'reverse.example.com')
}
api_execution = next(
execution for execution in completed_results[0].active_evidence.executions if execution.action == 'api-scan'
)
assert api_execution.status == 'completed'
assert {(observation.kind, observation.value) for observation in api_execution.observations} == {
('url', 'https://example.com/health')
}
xml_hosts = {
(element.findtext('hostname') or (element.text or '').strip(), element.findtext('ip'))
for element in ElementTree.parse(report.with_suffix('.xml')).getroot().findall('host')
@@ -325,13 +354,13 @@ async def test_rapiddns_evidence_reaches_existing_outputs(
assert len(completed_results) == 2
assert FakeSecurityScorecard.created == 1
assert completed_results[1].target == 'example.com'
assert {'192.0.2.1', '198.51.100.2'} <= {value for kind, value in completed_results[1].results if kind == 'ip-address'}
assert {'192.0.2.1', '198.51.100.2'} <= {value for kind, value in completed_results[1].results if kind == 'ip'}
assert ('email', 'user@example.com') in completed_results[1].results
assert {(observation.source, observation.kind, observation.value) for observation in completed_results[1].observations} >= {
('dehashed', 'email', 'user@example.com'),
('dehashed', 'ip-address', '198.51.100.2'),
('dehashed', 'ip', '198.51.100.2'),
('rapiddns', 'hostname', 'api.example.com'),
('rapiddns', 'ip-address', '192.0.2.1'),
('rapiddns', 'ip', '192.0.2.1'),
}
monkeypatch.setattr(sys, 'argv', ['theHarvester', '-d', 'example.com', '-b', 'rapiddns'])
tests/discovery/test_rocketreach.py
+2 -2
View File
@@ -11,7 +11,7 @@ if 'aiohttp_socks' not in sys.modules:
def from_url(*_args, **_kwargs):
return None
setattr(aiohttp_socks_stub, 'ProxyConnector', _ProxyConnector)
aiohttp_socks_stub.ProxyConnector = _ProxyConnector # type: ignore[attr-defined]
sys.modules['aiohttp_socks'] = aiohttp_socks_stub
from theHarvester.discovery import rocketreach
@@ -84,7 +84,7 @@ async def test_do_search_uses_people_data_endpoint_and_start_pagination(monkeypa
assert first_data == {'query': {'current_employer_domain': ['example.com']}, 'start': 0, 'page_size': 100}
assert second_data == {'query': {'current_employer_domain': ['example.com']}, 'start': 100, 'page_size': 50}
links = await search.get_links()
links = await search.get_urls()
emails = await search.get_emails()
assert len(links) == 150
assert len(emails) == 150
tests/discovery/test_shodan_engine.py
+40 -3
View File
@@ -11,6 +11,25 @@ class TestShodanEngine:
async def test_shodan_provider_failure_returns_attributed_empty_evidence(self, monkeypatch, caplog):
from theHarvester.discovery import shodansearch
class FailingShodan:
def host(self, _ip):
raise shodansearch.exception.APIError('No information available for that IP.')
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: FailingShodan())
caplog.set_level(logging.INFO, logger=shodansearch.__name__)
search = shodansearch.SearchShodan()
result = await search.search_ip('203.0.113.1')
assert result == OrderedDict({'203.0.113.1': 'Not in Shodan'})
assert search.error_type is None
assert '203.0.113.1: Not in Shodan' in caplog.text
@pytest.mark.asyncio
async def test_shodan_api_failure_exposes_only_its_error_type(self, monkeypatch, caplog):
from theHarvester.discovery import shodansearch
class FailingShodan:
def host(self, _ip):
raise shodansearch.exception.APIError('provider-secret-payload')
@@ -19,12 +38,30 @@ class TestShodanEngine:
monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: FailingShodan())
caplog.set_level(logging.INFO, logger=shodansearch.__name__)
result = await shodansearch.SearchShodan().search_ip('203.0.113.1')
search = shodansearch.SearchShodan()
result = await search.search_ip('203.0.113.1')
assert result == OrderedDict({'203.0.113.1': 'Not in Shodan'})
assert '203.0.113.1: Not in Shodan' in caplog.text
assert result == OrderedDict({'203.0.113.1': 'Shodan request failed'})
assert search.error_type == 'APIError'
assert 'provider-secret-payload' not in caplog.text
@pytest.mark.asyncio
async def test_shodan_unexpected_failure_exposes_only_its_error_type(self, monkeypatch):
from theHarvester.discovery import shodansearch
class FailingShodan:
def host(self, _ip):
raise RuntimeError('provider-secret-payload')
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch, 'Shodan', lambda _key: FailingShodan())
search = shodansearch.SearchShodan()
result = await search.search_ip('203.0.113.1')
assert result == OrderedDict({'203.0.113.1': 'Shodan request failed'})
assert search.error_type == 'RuntimeError'
@pytest.mark.asyncio
async def test_shodan_engine_processes_without_work_item_error_and_yields_hostnames(self, monkeypatch, capsys):
# Import inside the test so monkeypatching affects the already-imported module namespace.
tests/discovery/test_takeover.py
+37
View File
@@ -0,0 +1,37 @@
import pytest
from theHarvester.discovery import takeover
from theHarvester.lib.core import FetcherResponse
@pytest.mark.asyncio
async def test_takeover_distinguishes_transport_failure_from_successful_empty_body(
monkeypatch: pytest.MonkeyPatch,
) -> None:
search = takeover.TakeOver(['api.example.com', 'timeout.example.com'])
monkeypatch.setattr(search, 'fingerprints', {'No such app': 'Heroku'})
async def fake_fetch_all(urls, **kwargs):
assert kwargs['include_metadata'] is True
assert set(urls) == {
'https://api.example.com',
'http://api.example.com',
'https://timeout.example.com',
'http://timeout.example.com',
}
return [
('https://api.example.com', FetcherResponse(body='No such app', status=200, headers={})),
('http://api.example.com', FetcherResponse(body='', status=204, headers={})),
('https://timeout.example.com', None),
('http://timeout.example.com', FetcherResponse(body='not vulnerable', status=200, headers={})),
]
monkeypatch.setattr(takeover.AsyncFetcher, 'fetch_all', fake_fetch_all)
assert await search.process() is None
assert search.request_count == 4
assert search.request_error_count == 1
assert search.request_error_types == {'TransportError'}
assert search.scan_error_type is None
assert await search.get_takeover_results() == {'https://api.example.com': [{'No such app': 'Heroku'}]}
tests/discovery/test_zoomeyesearch.py
+23
View File
@@ -0,0 +1,23 @@
import pytest
from theHarvester.discovery import zoomeyesearch
@pytest.mark.asyncio
async def test_banner_urls_are_absolute_http_and_scoped(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(zoomeyesearch.Core, 'zoomeye_key', staticmethod(lambda: 'test-key'))
search = zoomeyesearch.SearchZoomEye('example.com', 1)
banner = '\n'.join(
f'"{value}"'
for value in (
'https://api.example.com/v1',
'//example.com/path',
'/assets/example.com/config.js',
'https://evil.test/?target=example.com',
'ftp://api.example.com/archive',
)
)
_hostnames, _emails, _ips, _asns, urls = await search.parse_matches([{'service': {'banner': banner}}])
assert urls == {'https://api.example.com/v1'}
tests/lib/test_active_evidence.py
+110
View File
@@ -0,0 +1,110 @@
from datetime import UTC, datetime
import pytest
from theHarvester.lib.active_evidence import ActionExecution, ActionObservation, ActiveEvidence, ArtifactReference
def screenshot_artifact() -> ArtifactReference:
return ArtifactReference(
kind='screenshot',
subject_kind='hostname',
subject_value='api.example.com',
path='screenshots/api.example.com.png',
media_type='image/png',
size_bytes=3,
sha256='0' * 64,
created_at=datetime(2026, 8, 9, 12, 0, tzinfo=UTC),
)
def test_active_evidence_owns_action_results_and_artifacts() -> None:
evidence = ActiveEvidence(
executions=(
ActionExecution.finish(
action='dns-resolve',
status='completed',
duration_ms=12.5,
groups={'ip': ['192.0.2.10', '192.0.2.10']},
),
ActionExecution.finish(
action='screenshot',
status='completed',
duration_ms=4.0,
groups={},
artifacts=(screenshot_artifact(),),
),
ActionExecution.finish(action='takeover', status='completed', duration_ms=2.0, groups={}),
)
)
assert evidence.executions[0].result_count == 1
assert evidence.executions[0].observations == (ActionObservation('ip', '192.0.2.10'),)
assert evidence.executions[1].result_count == 0
assert evidence.executions[1].artifacts == (screenshot_artifact(),)
def test_active_evidence_rejects_duplicate_actions() -> None:
with pytest.raises(ValueError, match='action executions must be unique'):
ActiveEvidence(
executions=(
ActionExecution('dns-resolve', 'completed', 1.0),
ActionExecution('dns-resolve', 'failed', 2.0, error_type='RuntimeError'),
)
)
def test_action_execution_rejects_noncanonical_observations_and_artifacts() -> None:
observation = ActionObservation('hostname', 'api.example.com')
artifact = screenshot_artifact()
with pytest.raises(ValueError, match='action observations must be deduplicated and sorted'):
ActionExecution('dns-brute', 'completed', 1.0, observations=(observation, observation))
with pytest.raises(ValueError, match='artifacts must be deduplicated and sorted'):
ActionExecution('screenshot', 'completed', 1.0, artifacts=(artifact, artifact))
def test_active_evidence_rejects_screenshot_results() -> None:
with pytest.raises(ValueError, match='screenshots must be stored as artifacts'):
ActionExecution.finish(
action='screenshot',
status='completed',
duration_ms=1.0,
groups={'screenshot': ['https://api.example.com']},
)
with pytest.raises(ValueError, match='known non-empty result'):
ArtifactReference(
kind='screenshot',
subject_kind='screenshot',
subject_value='https://api.example.com',
path='screenshots/api.example.com.png',
media_type='image/png',
size_bytes=3,
sha256='0' * 64,
created_at=datetime(2026, 8, 9, 12, 0, tzinfo=UTC),
)
def test_artifact_reference_rejects_invalid_metadata() -> None:
with pytest.raises(ValueError, match='sha256'):
ArtifactReference(
kind='screenshot',
subject_kind='hostname',
subject_value='api.example.com',
path='screenshots/api.example.com.png',
media_type='image/png',
size_bytes=3,
sha256='not-a-hash',
created_at=datetime(2026, 8, 9, 12, 0, tzinfo=UTC),
)
with pytest.raises(ValueError, match='timezone-aware'):
ArtifactReference(
kind='screenshot',
subject_kind='hostname',
subject_value='api.example.com',
path='screenshots/api.example.com.png',
media_type='image/png',
size_bytes=3,
sha256='0' * 64,
created_at=datetime(2026, 8, 9, 12, 0),
)
tests/lib/test_api_v1.py
+871
View File
@@ -0,0 +1,871 @@
from __future__ import annotations
import asyncio
import json
import pytest
from fastapi.testclient import TestClient
def _jsonl_result(
*,
target: str = 'example.test',
finding_type: str = 'email',
value: str = 'a@example.test',
finding_fields: dict[str, object] | None = None,
summary_fields: dict[str, object] | None = None,
) -> str:
summary = {
'type': 'summary',
'run_id': '9f9b4383-6cc4-4f3f-80a4-c8d21930dc2d',
'target': target,
'started_at': '2026-08-08T01:00:00Z',
'completed_at': '2026-08-08T01:01:00Z',
'evidence_status': 'complete',
'result_count': 1,
'counts': {finding_type: 1},
}
summary.update(summary_fields or {})
return '\n'.join(
(
json.dumps(summary),
json.dumps({'type': finding_type, 'value': value, 'sources': [], **(finding_fields or {})}),
'',
)
)
@pytest.mark.parametrize(
('finding_type', 'finding_fields'),
[
('made-up-kind', {}),
('api-endpoint', {}),
('interesting-url', {}),
('ip-address', {}),
('linkedin-link', {}),
('subdomain', {}),
('hostname', {'dns_status': 'made-up-status'}),
],
)
def test_api_rejects_jsonl_findings_outside_the_contract(
tmp_path,
monkeypatch,
finding_type: str,
finding_fields: dict[str, object],
) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
with TestClient(api.app) as client:
response = client.post(
'/api/v1/runs/import',
params={'filename': 'invalid.jsonl'},
headers={'X-API-Key': 'test-key'},
content=_jsonl_result(finding_type=finding_type, finding_fields=finding_fields),
)
assert response.status_code == 400
def test_api_exposes_one_fresh_run_contract(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
with TestClient(api.app, base_url='http://127.0.0.1', client=('127.0.0.1', 50000)) as client:
schema = client.get('/openapi.json').json()
paths = set(schema['paths'])
old_responses = [
client.get('/'),
client.get('/query?domain=example.test&source=crtsh'),
client.get('/sources'),
client.get('/dnsbrute?domain=example.test'),
client.get('/runs'),
client.post('/additional/all', json={'domain': 'example.test'}),
]
assert paths == {
'/api/v1/sources',
'/api/v1/runs',
'/api/v1/runs/import',
'/api/v1/runs/import-database',
'/api/v1/runs/{run_id}',
'/api/v1/runs/{run_id}/cancel',
'/api/v1/runs/{run_id}/export',
'/api/v1/runs/{run_id}/screenshots/{name}',
}
assert all(response.status_code == 404 for response in old_responses)
def test_screenshot_route_serves_only_a_run_owned_png(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
headers = {'X-API-Key': 'test-key'}
with TestClient(api.app, client=('127.0.0.2', 50000)) as client:
imported = client.post(
'/api/v1/runs/import',
params={'filename': 'smoke.jsonl'},
headers=headers,
content=_jsonl_result(
finding_type='hostname',
value='owned.example.test',
summary_fields={
'action_executions': [
{
'action': 'screenshot',
'status': 'completed',
'duration_ms': 1,
'result_count': 0,
'error_type': None,
'stop_reason': None,
}
],
'artifacts': [
{
'action': 'screenshot',
'kind': 'screenshot',
'subject': {'kind': 'hostname', 'value': 'owned.example.test'},
'file': {
'path': 'screenshots/owned.example.test.png',
'media_type': 'image/png',
'size_bytes': 16,
'sha256': '0' * 64,
},
'created_at': '2026-08-08T01:01:00Z',
}
],
},
),
)
assert imported.status_code == 201
run_id = imported.json()['run_id']
screenshot_dir = tmp_path / 'artifacts' / run_id / 'screenshots'
screenshot_dir.mkdir(parents=True)
(screenshot_dir / 'owned.example.test.png').write_bytes(b'owned screenshot')
(screenshot_dir / 'unrecorded.png').write_bytes(b'unrecorded screenshot')
outside = tmp_path / 'outside.png'
outside.write_bytes(b'outside screenshot')
(screenshot_dir / 'linked.png').symlink_to(outside)
owned = client.get(f'/api/v1/runs/{run_id}/screenshots/owned.example.test.png', headers=headers)
unrecorded = client.get(f'/api/v1/runs/{run_id}/screenshots/unrecorded.png', headers=headers)
linked = client.get(f'/api/v1/runs/{run_id}/screenshots/linked.png', headers=headers)
traversal = client.get(f'/api/v1/runs/{run_id}/screenshots/%2E%2E%2Foutside.png', headers=headers)
assert owned.status_code == 200
assert owned.content == b'owned screenshot'
assert unrecorded.status_code == 404
assert linked.status_code == 404
assert traversal.status_code == 404
def test_openapi_names_the_public_response_shapes(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
with TestClient(api.app) as client:
schema = client.get('/openapi.json').json()
paths = schema['paths']
assert paths['/api/v1/sources']['get']['responses']['200']['content']['application/json']['schema'] == {
'$ref': '#/components/schemas/SourceCatalogResponse'
}
assert paths['/api/v1/runs']['get']['responses']['200']['content']['application/json']['schema']['items'] == {
'$ref': '#/components/schemas/RunSummary'
}
for path, method in (
('/api/v1/runs', 'post'),
('/api/v1/runs/import', 'post'),
('/api/v1/runs/{run_id}', 'get'),
('/api/v1/runs/{run_id}/cancel', 'post'),
):
assert paths[path][method]['responses']['201' if path in {'/api/v1/runs', '/api/v1/runs/import'} else '200']['content'][
'application/json'
]['schema'] == {'$ref': '#/components/schemas/RunDetail'}
def test_source_catalog_exposes_shared_action_activities(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
with TestClient(api.app) as client:
response = client.get('/api/v1/sources', headers={'X-API-Key': 'test-key'})
assert response.status_code == 200
catalog = response.json()
assert catalog['sources']
assert catalog['actions'] == [
{'name': 'api-scan', 'activity': 'P2'},
{'name': 'dns-brute', 'activity': 'P1'},
{'name': 'dns-lookup', 'activity': 'P1'},
{'name': 'dns-recursive', 'activity': 'P1'},
{'name': 'dns-resolve', 'activity': 'P1'},
{'name': 'screenshot', 'activity': 'P2'},
{'name': 'shodan', 'activity': 'P0'},
{'name': 'takeover', 'activity': 'P2'},
]
def test_openapi_explains_scope_and_execution_controls(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
with TestClient(api.app) as client:
schema = client.get('/openapi.json').json()
request_body = schema['paths']['/api/v1/runs']['post']['requestBody']
properties = request_body['content']['application/json']['schema']['properties']
assert request_body['required'] is True
assert 'union' in properties['sources']['description']
assert 'do not filter' in properties['sources']['description']
assert '/24' in properties['dns_lookup']['description']
assert 'whole run' in properties['deadline_seconds']['description']
assert 'three resolver' in properties['dns_recursive_query_limit']['description']
assert 'discovery sources' in properties['proxies']['description']
assert 'configured proxies' in properties['takeover']['description']
assert 'take_over' not in properties
assert 'endpoint paths' in properties['api_scan_paths']['description']
import_content = schema['paths']['/api/v1/runs/import']['post']['requestBody']['content']
assert set(import_content) == {'application/x-ndjson'}
export_content = schema['paths']['/api/v1/runs/{run_id}/export']['get']['responses']['200']['content']
assert set(export_content) == {'application/x-ndjson'}
assert set(schema['components']['schemas']['NormalizedResult']['properties']) == {
'type',
'value',
'sources',
'actions',
}
assert export_content['application/x-ndjson']['schema']['description'] == (
'UTF-8 JSONL with one summary followed by normalized findings.'
)
def references(value):
if isinstance(value, dict):
if '$ref' in value:
yield value['$ref']
for child in value.values():
yield from references(child)
elif isinstance(value, list):
for child in value:
yield from references(child)
components = schema['components']['schemas']
for reference in references(schema):
assert reference.startswith('#/components/schemas/')
assert reference.removeprefix('#/components/schemas/') in components
def test_run_detail_exposes_one_normalized_evidence_surface(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
with TestClient(api.app) as client:
imported = client.post(
'/api/v1/runs/import',
params={'filename': 'result.jsonl'},
headers={'X-API-Key': 'test-key'},
content=_jsonl_result(),
)
assert imported.status_code == 201
assert 'evidence' not in imported.json()
def test_api_scan_can_run_without_discovery_sources(tmp_path, monkeypatch) -> None:
from pydantic import ValidationError
from theHarvester.lib.api.run_models import RunRequest
request = RunRequest(target='example.test', sources=[], api_scan=True, api_scan_paths=['/api/v2', '/health'])
assert request.api_scan is True
assert request.api_scan_paths == ['/api/v2', '/health']
with pytest.raises(ValidationError):
RunRequest(target='example.test', sources=[], api_scan=True, api_scan_paths=['https://other.example/api'])
def test_fresh_api_uses_catalog_takeover_name_and_rejects_unknown_fields() -> None:
from pydantic import ValidationError
from theHarvester.lib.api.run_models import RunRequest
request = RunRequest(target='example.test', sources=[], takeover=True)
assert request.takeover is True
with pytest.raises(ValidationError):
RunRequest(target='example.test', sources=[], take_over=True)
@pytest.mark.parametrize(
('evidence_status', 'execution_status'),
[('complete', 'failed'), ('partial', 'completed')],
)
def test_api_rejects_evidence_status_that_disagrees_with_executions(
tmp_path,
monkeypatch,
evidence_status,
execution_status,
) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
payload = _jsonl_result(
summary_fields={
'evidence_status': evidence_status,
'source_executions': [
{
'source': 'crtsh',
'status': execution_status,
'duration_ms': 1,
'result_count': 0,
'error_type': 'RuntimeError',
'stop_reason': 'provider-error',
}
],
},
finding_fields={'sources': []},
)
with TestClient(api.app) as client:
response = client.post(
'/api/v1/runs/import',
params={'filename': 'inconsistent.jsonl'},
headers={'X-API-Key': 'test-key'},
content=payload,
)
assert response.status_code == 400
assert response.json()['detail'] == 'Evidence status does not match its execution outcomes'
def test_api_preserves_sparse_failed_status_without_executions(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
database = tmp_path / 'runs.sqlite'
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(database))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
payload = _jsonl_result(summary_fields={'evidence_status': 'failed'})
with TestClient(api.app) as client:
imported = client.post(
'/api/v1/runs/import',
params={'filename': 'failed.jsonl'},
headers={'X-API-Key': 'test-key'},
content=payload,
)
exported = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/export', headers={'X-API-Key': 'test-key'})
assert imported.status_code == 201
assert imported.json()['evidence_status'] == 'failed'
assert json.loads(exported.text.splitlines()[0])['evidence_status'] == 'failed'
def test_api_import_and_export_accept_only_jsonl(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
headers = {'X-API-Key': 'test-key'}
with TestClient(api.app, client=('127.0.0.3', 50000)) as client:
rejected = client.post(
'/api/v1/runs/import',
params={'filename': 'legacy.json'},
headers=headers,
content='{"target":"example.test"}',
)
imported = client.post(
'/api/v1/runs/import',
params={'filename': 'result.jsonl'},
headers={**headers, 'Content-Type': 'application/x-ndjson'},
content=_jsonl_result(),
)
exported = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/export', headers=headers)
reimported = client.post(
'/api/v1/runs/import',
params={'filename': 'round-trip.jsonl'},
headers={**headers, 'Content-Type': 'application/x-ndjson'},
content=exported.content,
)
old_json = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/exports/json', headers=headers)
old_csv = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/exports/csv', headers=headers)
assert rejected.status_code == 400
assert rejected.json()['detail'] == 'Choose a .jsonl result file'
assert imported.status_code == 201
assert exported.status_code == 200
assert exported.headers['content-type'] == 'application/x-ndjson'
assert exported.headers['content-disposition'].endswith('.jsonl"')
records = [json.loads(line) for line in exported.text.splitlines()]
assert 'schema' not in records[0]
assert 'schema_version' not in records[0]
assert records[0]['type'] == 'summary'
assert records[0]['target'] == 'example.test'
assert records[1] == {'sources': [], 'type': 'email', 'value': 'a@example.test'}
assert reimported.status_code == 201
assert reimported.json()['results'] == [{'type': 'email', 'value': 'a@example.test', 'sources': [], 'actions': []}]
assert old_json.status_code == 404
assert old_csv.status_code == 404
def test_api_jsonl_round_trip_preserves_canonical_url_sources(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
headers = {'X-API-Key': 'test-key', 'Content-Type': 'application/x-ndjson'}
sources = ['builtwith', 'gitlab', 'rocketreach']
executions = [
{
'source': source,
'status': 'completed',
'duration_ms': 1,
'result_count': 1,
'error_type': None,
'stop_reason': None,
}
for source in sources
]
payload = _jsonl_result(
finding_type='url',
value='https://example.test/profile',
finding_fields={'sources': sources},
summary_fields={'source_executions': executions},
)
with TestClient(api.app) as client:
imported = client.post(
'/api/v1/runs/import',
params={'filename': 'urls.jsonl'},
headers=headers,
content=payload,
)
run_id = imported.json()['run_id']
detail = client.get(f'/api/v1/runs/{run_id}', headers={'X-API-Key': 'test-key'})
exported = client.get(f'/api/v1/runs/{run_id}/export', headers={'X-API-Key': 'test-key'})
assert imported.status_code == 201
assert detail.json()['results'] == [
{
'type': 'url',
'value': 'https://example.test/profile',
'sources': sources,
'actions': [],
}
]
records = [json.loads(line) for line in exported.text.splitlines()]
assert records[1] == {'sources': sources, 'type': 'url', 'value': 'https://example.test/profile'}
def test_api_database_import_rejects_non_sqlite_content(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
with TestClient(api.app) as client:
response = client.post(
'/api/v1/runs/import-database',
params={'filename': 'results.sqlite'},
headers={'X-API-Key': 'test-key', 'Content-Type': 'application/vnd.sqlite3'},
content=b'not a sqlite database',
)
assert response.status_code == 400
assert response.json()['detail'] == 'Uploaded file is not a SQLite database'
def test_api_database_import_exposes_completed_cli_runs(tmp_path, monkeypatch) -> None:
from datetime import UTC, datetime
from theHarvester.lib.api import api
from theHarvester.lib.completed_result import CompletedResult
from theHarvester.lib.database import ResultStore, dispose_sqlite_databases
source_database = tmp_path / 'source.sqlite'
destination_database = tmp_path / 'destination.sqlite'
now = datetime.now(UTC)
completed = CompletedResult.finish(
target='imported.example.test',
started_at=now,
completed_at=now,
groups={'hostname': ['api.imported.example.test']},
)
async def seed() -> None:
store = ResultStore(source_database)
await store.initialize()
await store.save_run(completed)
await dispose_sqlite_databases()
asyncio.run(seed())
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(destination_database))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
with TestClient(api.app) as client:
imported = client.post(
'/api/v1/runs/import-database',
params={'filename': 'source.sqlite'},
headers={'X-API-Key': 'test-key', 'Content-Type': 'application/vnd.sqlite3'},
content=source_database.read_bytes(),
)
detail = client.get(
f'/api/v1/runs/{completed.run_id}',
headers={'X-API-Key': 'test-key'},
)
assert imported.status_code == 201
assert imported.json()['imported_run_ids'] == [str(completed.run_id)]
assert detail.status_code == 200
assert detail.json()['results'] == [{'type': 'hostname', 'value': 'api.imported.example.test', 'sources': [], 'actions': []}]
def test_api_jsonl_round_trip_preserves_source_attribution(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
source_execution = {
'source': 'crtsh',
'status': 'completed',
'duration_ms': 0,
'result_count': 1,
'error_type': None,
'stop_reason': None,
}
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
headers = {'X-API-Key': 'test-key'}
with TestClient(api.app, client=('127.0.0.4', 50000)) as client:
imported = client.post(
'/api/v1/runs/import',
params={'filename': 'complete.jsonl'},
headers=headers,
content=_jsonl_result(
finding_fields={'sources': ['crtsh']},
summary_fields={'source_executions': [source_execution]},
),
)
exported = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/export', headers=headers)
reimported = client.post(
'/api/v1/runs/import',
params={'filename': 'complete-round-trip.jsonl'},
headers=headers,
content=exported.content,
)
summary = json.loads(exported.text.splitlines()[0])
assert imported.json()['evidence_status'] == 'complete'
assert imported.json()['source_executions'] == [source_execution]
assert imported.json()['results'] == [{'type': 'email', 'value': 'a@example.test', 'sources': ['crtsh'], 'actions': []}]
assert summary['run_id'] == imported.json()['run_id']
assert summary['evidence_status'] == 'complete'
assert summary['source_executions'] == [source_execution]
assert json.loads(exported.text.splitlines()[1])['sources'] == ['crtsh']
assert reimported.json()['evidence_status'] == 'complete'
assert reimported.json()['request']['source_run_id'] == imported.json()['run_id']
assert reimported.json()['source_executions'] == [source_execution]
def test_api_jsonl_export_uses_evidence_timestamps_not_lifecycle_timestamps(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
store = RunStore()
queued = asyncio.run(store.create(RunRequest(target='example.test', sources=['crtsh'])))
asyncio.run(store.claim_next())
asyncio.run(
store.finish(
queued['run_id'],
{
'run_id': '3e7cf0c1-214b-4429-80ba-058b2cb68b06',
'target': 'example.test',
'status': 'complete',
'started_at': '2026-08-07T01:00:00Z',
'completed_at': '2026-08-07T01:01:00Z',
'results': [],
'source_executions': [],
},
'',
)
)
with TestClient(api.app, client=('127.0.0.16', 50000)) as client:
response = client.get(
f'/api/v1/runs/{queued["run_id"]}/export',
headers={'X-API-Key': 'test-key'},
)
summary = json.loads(response.text.splitlines()[0])
completed = asyncio.run(store.load_completed_result(queued['run_id']))
assert completed is not None
assert response.text == completed.jsonl()
assert summary['started_at'] == '2026-08-07T01:00:00Z'
assert summary['completed_at'] == '2026-08-07T01:01:00Z'
def test_api_jsonl_export_uses_lifecycle_timestamps_for_sparse_partial_evidence(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
store = RunStore()
queued = asyncio.run(store.create(RunRequest(target='example.test', sources=['crtsh'])))
asyncio.run(store.claim_next())
asyncio.run(
store.fail(
queued['run_id'],
'Provider process exited.',
'',
evidence={
'run_id': 'eb470313-d813-4d81-bd75-c1221a8bc00e',
'target': 'example.test',
'status': 'partial',
'results': [],
'source_executions': [],
},
)
)
with TestClient(api.app, client=('127.0.0.17', 50000)) as client:
exported = client.get(
f'/api/v1/runs/{queued["run_id"]}/export',
headers={'X-API-Key': 'test-key'},
)
reimported = client.post(
'/api/v1/runs/import',
params={'filename': 'partial.jsonl'},
headers={'X-API-Key': 'test-key'},
content=exported.content,
)
summary = json.loads(exported.text.splitlines()[0])
assert exported.status_code == 200
assert isinstance(summary['started_at'], str)
assert isinstance(summary['completed_at'], str)
assert reimported.status_code == 201
assert summary['evidence_status'] == 'partial'
assert reimported.json()['evidence_status'] == 'partial'
def test_api_jsonl_round_trip_uses_canonical_hostname_and_ip_kinds(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
headers = {'X-API-Key': 'test-key'}
for client_ip, finding_type, value, run_id in (
('127.0.0.5', 'hostname', 'www.example.test', '0f17b751-dd31-46da-968f-31580e233b72'),
('127.0.0.6', 'ip', '192.0.2.1', 'f7419165-d78c-4aef-9023-e9686f864ff0'),
):
with TestClient(api.app, client=(client_ip, 50000)) as client:
imported = client.post(
'/api/v1/runs/import',
params={'filename': f'{finding_type}.jsonl'},
headers=headers,
content=_jsonl_result(finding_type=finding_type, value=value, summary_fields={'run_id': run_id}),
)
exported = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/export', headers=headers)
reimported = client.post(
'/api/v1/runs/import',
params={'filename': f'{finding_type}-round-trip.jsonl'},
headers=headers,
content=exported.content,
)
assert imported.json()['results'] == [{'type': finding_type, 'value': value, 'sources': [], 'actions': []}]
assert json.loads(exported.text.splitlines()[1]) == {'sources': [], 'type': finding_type, 'value': value}
assert reimported.json()['results'] == [{'type': finding_type, 'value': value, 'sources': [], 'actions': []}]
def test_api_jsonl_round_trip_preserves_execution_outcomes_and_action_origins(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
headers = {'X-API-Key': 'test-key'}
payload = _jsonl_result(
finding_type='hostname',
value='api.example.test',
finding_fields={'sources': ['crtsh', 'crtsh'], 'actions': ['dns-brute', 'dns-brute']},
summary_fields={
'evidence_status': 'partial',
'source_executions': [
{
'source': 'crtsh',
'status': 'completed',
'duration_ms': 1,
'result_count': 1,
'error_type': None,
'stop_reason': None,
},
{
'source': 'certspotter',
'status': 'rate-limited',
'duration_ms': 2,
'result_count': 0,
'error_type': None,
'stop_reason': 'http-429',
},
],
'action_executions': [
{
'action': 'dns-brute',
'status': 'partial',
'duration_ms': 3,
'result_count': 1,
'error_type': 'TimeoutError',
'stop_reason': 'query-errors',
}
],
},
)
with TestClient(api.app, client=('127.0.0.18', 50000)) as client:
imported = client.post(
'/api/v1/runs/import',
params={'filename': 'attributed.jsonl'},
headers=headers,
content=payload,
)
exported = client.get(f'/api/v1/runs/{imported.json()["run_id"]}/export', headers=headers)
reimported = client.post(
'/api/v1/runs/import',
params={'filename': 'attributed-round-trip.jsonl'},
headers=headers,
content=exported.content,
)
assert imported.status_code == 201
assert exported.status_code == 200
assert reimported.status_code == 201
assert reimported.json()['evidence_status'] == 'partial'
assert reimported.json()['source_executions'] == imported.json()['source_executions']
assert reimported.json()['action_executions'] == imported.json()['action_executions']
assert reimported.json()['results'] == [
{
'type': 'hostname',
'value': 'api.example.test',
'sources': ['crtsh'],
'actions': ['dns-brute'],
}
]
def test_api_rejects_non_string_jsonl_timestamps(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
with TestClient(api.app, client=('127.0.0.7', 50000)) as client:
response = client.post(
'/api/v1/runs/import',
params={'filename': 'invalid.jsonl'},
headers={'X-API-Key': 'test-key'},
content=_jsonl_result(summary_fields={'completed_at': {'not': 'a timestamp'}}),
)
assert response.status_code == 400
assert response.json()['detail'] == 'JSONL summary must contain an ISO-8601 UTC completed_at'
def test_api_rejects_invalid_jsonl_summary_identity_and_timestamps(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
cases = (
({'run_id': None}, None, 'JSONL summary must contain a UUID run_id'),
({}, 'run_id', 'JSONL summary must contain a UUID run_id'),
({'run_id': 'not-a-uuid'}, None, 'JSONL summary must contain a UUID run_id'),
({}, 'started_at', 'JSONL summary must contain an ISO-8601 UTC started_at'),
({}, 'completed_at', 'JSONL summary must contain an ISO-8601 UTC completed_at'),
({'started_at': 'not-a-time'}, None, 'JSONL summary must contain an ISO-8601 UTC started_at'),
(
{'completed_at': '2026-08-08T02:01:00+01:00'},
None,
'JSONL summary must contain an ISO-8601 UTC completed_at',
),
(
{'started_at': '2026-08-08T03:00:00Z', 'completed_at': '2026-08-08T02:00:00Z'},
None,
'JSONL summary completed_at must not be earlier than started_at',
),
)
for index, (updates, removed_field, detail) in enumerate(cases, start=8):
records = [json.loads(line) for line in _jsonl_result().splitlines()]
records[0].update(updates)
if removed_field:
records[0].pop(removed_field)
content = ''.join(json.dumps(record) + '\n' for record in records)
with TestClient(api.app, client=(f'127.0.0.{index}', 50000)) as client:
response = client.post(
'/api/v1/runs/import',
params={'filename': 'invalid.jsonl'},
headers={'X-API-Key': 'test-key'},
content=content,
)
assert response.status_code == 400
assert response.json()['detail'] == detail
def test_api_refuses_to_export_before_evidence_exists(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
run = asyncio.run(RunStore().create(RunRequest(target='example.test', sources=['crtsh'])))
with TestClient(api.app) as client:
response = client.get(f'/api/v1/runs/{run["run_id"]}/export', headers={'X-API-Key': 'test-key'})
assert response.status_code == 409
assert response.json()['detail'] == 'No run evidence is available to export'
tests/lib/test_completed_persistence.py
+538 -12
View File
@@ -8,6 +8,7 @@ from uuid import UUID
import pytest
from theHarvester.lib import database as database_module
from theHarvester.lib.active_evidence import ActionExecution, ActiveEvidence, ArtifactReference
from theHarvester.lib.completed_result import CompletedResult, ResultObservation, SourceExecution
from theHarvester.lib.database import (
DuplicateRunError,
@@ -56,6 +57,78 @@ CREATE TABLE discovery_observations (
PRAGMA user_version = 1;
"""
SCHEMA_V2_RUN_PROVENANCE = """
CREATE TABLE runs (
run_id TEXT PRIMARY KEY,
target TEXT NOT NULL,
started_at TEXT NOT NULL,
completed_at TEXT NOT NULL
);
CREATE TABLE results (
run_id TEXT NOT NULL REFERENCES runs(run_id) ON DELETE CASCADE,
position INTEGER NOT NULL,
kind TEXT NOT NULL,
value TEXT NOT NULL,
PRIMARY KEY (run_id, position),
UNIQUE (run_id, kind, value)
);
CREATE TABLE executions (
run_id TEXT NOT NULL REFERENCES runs(run_id) ON DELETE CASCADE,
position INTEGER NOT NULL,
producer_kind TEXT NOT NULL,
name TEXT NOT NULL,
status TEXT NOT NULL,
duration_ms REAL NOT NULL,
result_count INTEGER NOT NULL,
error_type TEXT,
stop_reason TEXT,
PRIMARY KEY (run_id, position),
UNIQUE (run_id, producer_kind, name)
);
CREATE TABLE result_origins (
run_id TEXT NOT NULL,
result_position INTEGER NOT NULL,
execution_position INTEGER NOT NULL,
PRIMARY KEY (run_id, result_position, execution_position),
FOREIGN KEY (run_id, result_position) REFERENCES results(run_id, position) ON DELETE CASCADE,
FOREIGN KEY (run_id, execution_position) REFERENCES executions(run_id, position) ON DELETE CASCADE
);
CREATE TABLE legacy_observations (
id INTEGER PRIMARY KEY,
domain TEXT,
resource TEXT,
kind TEXT,
discovered_on DATE,
source TEXT
);
PRAGMA user_version = 2;
"""
SCHEMA_V4_URL_KINDS = (
SCHEMA_V2_RUN_PROVENANCE
+ """
CREATE TABLE artifacts (
run_id TEXT NOT NULL,
position INTEGER NOT NULL,
result_position INTEGER NOT NULL,
execution_position INTEGER NOT NULL,
kind TEXT NOT NULL,
path TEXT NOT NULL,
media_type TEXT NOT NULL,
size_bytes INTEGER NOT NULL,
sha256 TEXT NOT NULL,
created_at TEXT NOT NULL,
PRIMARY KEY (run_id, position),
FOREIGN KEY (run_id, result_position) REFERENCES results(run_id, position) ON DELETE CASCADE,
FOREIGN KEY (run_id, execution_position) REFERENCES executions(run_id, position) ON DELETE CASCADE
);
PRAGMA user_version = 4;
"""
)
SCHEMA_V5_RESULT_KINDS = SCHEMA_V4_URL_KINDS + 'PRAGMA user_version = 5;'
SCHEMA_V6_RESULT_KINDS = SCHEMA_V5_RESULT_KINDS + 'PRAGMA user_version = 6;'
def completed_result(run_id: str = 'f047261c-0afb-4e18-89d5-28a7d977f51f') -> CompletedResult:
return CompletedResult.finish(
@@ -71,12 +144,33 @@ def completed_result(run_id: str = 'f047261c-0afb-4e18-89d5-28a7d977f51f') -> Co
],
'dns-recursive-summary': ['{"depth_reached":1,"query_count":24,"stop_reason":"depth-limit","zero_yield_batches":0}'],
'hostname': ['api.example.com'],
'ip-address': ['192.0.2.1'],
'ip': ['192.0.2.1'],
'person': ['{"firstname":"Ada","lastname":"Lovelace"}'],
},
)
def screenshot_execution(completed_at: datetime) -> ActionExecution:
return ActionExecution.finish(
action='screenshot',
status='completed',
duration_ms=4.0,
groups={},
artifacts=(
ArtifactReference(
kind='screenshot',
subject_kind='hostname',
subject_value='api.example.com',
path='screenshots/api.example.com.png',
media_type='image/png',
size_bytes=3,
sha256='0' * 64,
created_at=completed_at,
),
),
)
@pytest.mark.asyncio
async def test_initialization_enables_wal_when_sqlite_contains_the_reset_fix(tmp_path) -> None:
database = tmp_path / 'stash.sqlite'
@@ -124,10 +218,10 @@ async def test_newer_schema_is_rejected_without_changing_journal_mode(tmp_path)
database = tmp_path / 'stash.sqlite'
store = ResultStore(database)
with sqlite3.connect(database) as db:
db.execute('PRAGMA user_version = 3')
db.execute('PRAGMA user_version = 8')
original_journal_mode = db.execute('PRAGMA journal_mode').fetchone()[0]
with pytest.raises(RuntimeError, match='schema version 3 is newer than supported version 2'):
with pytest.raises(RuntimeError, match='schema version 8 is newer than supported version 7'):
await store.initialize()
with sqlite3.connect(database) as db:
@@ -197,7 +291,13 @@ async def test_completed_result_round_trip_preserves_legacy_observations(tmp_pat
jsonl_items = {(record['type'], record['value']) for line in result.jsonl().splitlines()[1:] if (record := json.loads(line))}
assert stored == [('example.com', 'legacy.example.com', 'hostname', 'legacy-source')]
assert stored_items == jsonl_items
assert run_types == {'run_id': 'TEXT', 'target': 'TEXT', 'started_at': 'TEXT', 'completed_at': 'TEXT'}
assert run_types == {
'run_id': 'TEXT',
'target': 'TEXT',
'started_at': 'TEXT',
'completed_at': 'TEXT',
'evidence_status': 'TEXT',
}
assert result_types == {
'run_id': 'TEXT',
'position': 'INTEGER',
@@ -206,6 +306,26 @@ async def test_completed_result_round_trip_preserves_legacy_observations(tmp_pat
}
@pytest.mark.asyncio
@pytest.mark.parametrize('evidence_status', ['partial', 'failed'])
async def test_sparse_evidence_status_survives_result_store_round_trip(tmp_path, evidence_status: str) -> None:
store = ResultStore(tmp_path / 'stash.sqlite')
await store.initialize()
result = CompletedResult.finish(
target='example.com',
started_at=datetime(2026, 8, 5, 12, 0, tzinfo=UTC),
completed_at=datetime(2026, 8, 5, 12, 1, tzinfo=UTC),
groups={},
evidence_status=evidence_status,
)
await store.save_run(result)
loaded = await store.load_run(result.run_id)
assert loaded == result
assert loaded.status == evidence_status
@pytest.mark.asyncio
async def test_completed_result_round_trip_preserves_source_provenance(tmp_path) -> None:
database = tmp_path / 'stash.sqlite'
@@ -253,6 +373,124 @@ async def test_completed_result_round_trip_preserves_source_provenance(tmp_path)
]
@pytest.mark.asyncio
async def test_mixed_source_action_artifact_round_trip_uses_unified_tables(tmp_path) -> None:
database = tmp_path / 'stash.sqlite'
store = ResultStore(database)
await store.initialize()
completed_at = datetime(2026, 8, 9, 12, 1, tzinfo=UTC)
result = CompletedResult.finish(
run_id=UUID('d721f4c5-1c76-4e7a-904a-23c5d6755834'),
target='example.com',
started_at=datetime(2026, 8, 9, 12, 0, tzinfo=UTC),
completed_at=completed_at,
groups={'hostname': ['api.example.com']},
source_executions=(SourceExecution('shared-name', 'completed', 2.0, 1),),
observations=(ResultObservation('shared-name', 'hostname', 'api.example.com'),),
active_evidence=ActiveEvidence(
executions=(
ActionExecution.finish(
action='shared-name',
status='completed',
duration_ms=3.0,
groups={'ip': ['192.0.2.10']},
),
screenshot_execution(completed_at),
ActionExecution.finish(
action='takeover',
status='completed',
duration_ms=1.0,
groups={},
),
)
),
)
await store.save_run(result)
assert await store.load_run(result.run_id) == result
assert [item.to_dict() for item in await store.action_yields(result.run_id)] == [
{
'action': 'screenshot',
'observed_result_count': 0,
'unique_result_count': 0,
'shared_result_count': 0,
},
{
'action': 'shared-name',
'observed_result_count': 1,
'unique_result_count': 1,
'shared_result_count': 0,
},
{
'action': 'takeover',
'observed_result_count': 0,
'unique_result_count': 0,
'shared_result_count': 0,
},
]
with sqlite3.connect(database) as db:
tables = {
row[0] for row in db.execute("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'")
}
executions = db.execute('SELECT position, producer_kind, name, result_count FROM executions ORDER BY position').fetchall()
origins = db.execute(
'SELECT e.producer_kind, e.name, r.kind, r.value '
'FROM result_origins AS o '
'JOIN executions AS e ON e.run_id = o.run_id AND e.position = o.execution_position '
'JOIN results AS r ON r.run_id = o.run_id AND r.position = o.result_position '
'ORDER BY e.producer_kind, e.name'
).fetchall()
artifacts = db.execute(
'SELECT e.name, r.kind, r.value, a.kind, a.path, a.media_type, a.size_bytes, a.sha256, a.created_at '
'FROM artifacts AS a '
'JOIN executions AS e ON e.run_id = a.run_id AND e.position = a.execution_position '
'JOIN results AS r ON r.run_id = a.run_id AND r.position = a.result_position'
).fetchall()
assert tables == {
'runs',
'executions',
'results',
'result_origins',
'artifacts',
'legacy_observations',
'run_records',
'run_worker_leases',
}
assert executions == [
(0, 'source', 'shared-name', 1),
(1, 'action', 'shared-name', 1),
(2, 'action', 'screenshot', 0),
(3, 'action', 'takeover', 0),
]
assert origins == [
('action', 'shared-name', 'ip', '192.0.2.10'),
('source', 'shared-name', 'hostname', 'api.example.com'),
]
assert artifacts == [
(
'screenshot',
'hostname',
'api.example.com',
'screenshot',
'screenshots/api.example.com.png',
'image/png',
3,
'0' * 64,
completed_at.isoformat(),
)
]
with sqlite3.connect(database) as db:
db.execute('PRAGMA foreign_keys = ON')
db.execute('DELETE FROM runs WHERE run_id = ?', (str(result.run_id),))
db.commit()
assert db.execute('SELECT COUNT(*) FROM executions').fetchone()[0] == 0
assert db.execute('SELECT COUNT(*) FROM results').fetchone()[0] == 0
assert db.execute('SELECT COUNT(*) FROM result_origins').fetchone()[0] == 0
assert db.execute('SELECT COUNT(*) FROM artifacts').fetchone()[0] == 0
@pytest.mark.asyncio
async def test_source_yields_distinguish_unique_and_shared_results(tmp_path) -> None:
store = ResultStore(tmp_path / 'stash.sqlite')
@@ -346,6 +584,265 @@ async def test_current_schema_reopens_without_running_legacy_migration(tmp_path)
assert await reopened.load_run(existing.run_id) == existing
@pytest.mark.asyncio
async def test_schema_v2_upgrades_to_v7_without_rewriting_existing_rows(tmp_path) -> None:
database = tmp_path / 'stash.sqlite'
run_id = UUID('251d4047-190b-4a4d-9c4e-9eed3f23c8c7')
with sqlite3.connect(database) as db:
db.executescript(SCHEMA_V2_RUN_PROVENANCE)
db.execute(
'INSERT INTO runs (run_id, target, started_at, completed_at) VALUES (?, ?, ?, ?)',
(
str(run_id),
'example.com',
'2026-08-09T12:00:00+00:00',
'2026-08-09T12:01:00+00:00',
),
)
db.execute(
'INSERT INTO results (run_id, position, kind, value) VALUES (?, ?, ?, ?)',
(str(run_id), 0, 'hostname', 'api.example.com'),
)
db.execute(
'INSERT INTO executions '
'(run_id, position, producer_kind, name, status, duration_ms, result_count) '
'VALUES (?, ?, ?, ?, ?, ?, ?)',
(str(run_id), 0, 'source', 'crtsh', 'completed', 12.5, 1),
)
db.execute(
'INSERT INTO result_origins (run_id, result_position, execution_position) VALUES (?, ?, ?)',
(str(run_id), 0, 0),
)
store = ResultStore(database)
await store.initialize()
await store.initialize()
loaded = await store.load_run(run_id)
assert loaded == CompletedResult.finish(
run_id=run_id,
target='example.com',
started_at=datetime(2026, 8, 9, 12, 0, tzinfo=UTC),
completed_at=datetime(2026, 8, 9, 12, 1, tzinfo=UTC),
groups={'hostname': ['api.example.com']},
source_executions=(SourceExecution('crtsh', 'completed', 12.5, 1),),
observations=(ResultObservation('crtsh', 'hostname', 'api.example.com'),),
)
with sqlite3.connect(database) as db:
assert db.execute('PRAGMA user_version').fetchone()[0] == 7
assert db.execute('SELECT COUNT(*) FROM artifacts').fetchone()[0] == 0
@pytest.mark.asyncio
async def test_schema_v6_adds_nullable_evidence_status_without_rewriting_execution_status(tmp_path) -> None:
database = tmp_path / 'stash.sqlite'
run_id = UUID('cb34987b-9dd5-44f5-a58a-7ca7d34b0743')
with sqlite3.connect(database) as db:
db.executescript(SCHEMA_V6_RESULT_KINDS)
db.execute(
'INSERT INTO runs (run_id, target, started_at, completed_at) VALUES (?, ?, ?, ?)',
(
str(run_id),
'example.com',
'2026-08-09T12:00:00+00:00',
'2026-08-09T12:01:00+00:00',
),
)
db.execute(
'INSERT INTO executions '
'(run_id, position, producer_kind, name, status, duration_ms, result_count) '
'VALUES (?, ?, ?, ?, ?, ?, ?)',
(str(run_id), 0, 'source', 'crtsh', 'partial', 12.5, 0),
)
store = ResultStore(database)
await store.initialize()
loaded = await store.load_run(run_id)
with sqlite3.connect(database) as db:
run_columns = {row[1] for row in db.execute('PRAGMA table_info(runs)')}
stored_status = db.execute('SELECT evidence_status FROM runs WHERE run_id = ?', (str(run_id),)).fetchone()[0]
schema_version = db.execute('PRAGMA user_version').fetchone()[0]
assert loaded.status == 'partial'
assert stored_status is None
assert 'evidence_status' in run_columns
assert schema_version == 7
@pytest.mark.asyncio
async def test_schema_v4_merges_deprecated_url_kinds_without_losing_origins(tmp_path) -> None:
database = tmp_path / 'stash.sqlite'
run_id = UUID('d299651b-21c1-4511-8cac-63ba70f926f4')
target_url = 'https://portal.example.com/login'
with sqlite3.connect(database) as db:
db.executescript(SCHEMA_V4_URL_KINDS)
db.execute(
'INSERT INTO runs (run_id, target, started_at, completed_at) VALUES (?, ?, ?, ?)',
(str(run_id), 'example.com', '2026-08-09T12:00:00+00:00', '2026-08-09T12:01:00+00:00'),
)
db.executemany(
'INSERT INTO results (run_id, position, kind, value) VALUES (?, ?, ?, ?)',
[
(str(run_id), 0, 'api-endpoint', target_url),
(str(run_id), 1, 'hostname', 'portal.example.com'),
(str(run_id), 2, 'interesting-url', target_url),
(str(run_id), 3, 'linkedin-link', target_url),
(str(run_id), 4, 'url', target_url),
],
)
db.executemany(
'INSERT INTO executions '
'(run_id, position, producer_kind, name, status, duration_ms, result_count, error_type, stop_reason) '
'VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)',
[
(str(run_id), 0, 'source', 'builtwith', 'completed', 1.0, 1, None, None),
(str(run_id), 1, 'source', 'rocketreach', 'completed', 1.0, 1, None, None),
(str(run_id), 2, 'source', 'gitlab', 'completed', 1.0, 1, None, None),
(str(run_id), 3, 'action', 'api-scan', 'completed', 1.0, 3, None, None),
(str(run_id), 4, 'action', 'screenshot', 'completed', 1.0, 0, None, None),
],
)
db.executemany(
'INSERT INTO result_origins (run_id, result_position, execution_position) VALUES (?, ?, ?)',
[
(str(run_id), 2, 0),
(str(run_id), 3, 1),
(str(run_id), 4, 2),
(str(run_id), 0, 3),
(str(run_id), 2, 3),
(str(run_id), 4, 3),
],
)
db.execute(
'INSERT INTO artifacts '
'(run_id, position, result_position, execution_position, kind, path, media_type, size_bytes, sha256, created_at) '
'VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
(
str(run_id),
0,
2,
4,
'screenshot',
'screenshots/portal.png',
'image/png',
3,
'0' * 64,
'2026-08-09T12:01:00+00:00',
),
)
db.executemany(
'INSERT INTO legacy_observations (domain, resource, kind, discovered_on, source) VALUES (?, ?, ?, ?, ?)',
[
('example.com', target_url, 'interesting-url', '2026-08-09', 'builtwith'),
('example.com', target_url, 'linkedinlinks', '2026-08-09', 'rocketreach'),
],
)
store = ResultStore(database)
await store.initialize()
loaded = await store.load_run(run_id)
assert loaded.results == (('hostname', 'portal.example.com'), ('url', target_url))
assert {(item.source, item.kind, item.value) for item in loaded.observations} == {
('builtwith', 'url', target_url),
('gitlab', 'url', target_url),
('rocketreach', 'url', target_url),
}
api_scan = next(item for item in loaded.active_evidence.executions if item.action == 'api-scan')
assert api_scan.result_count == 1
assert {(item.kind, item.value) for item in api_scan.observations} == {('url', target_url)}
screenshot = next(item for item in loaded.active_evidence.executions if item.action == 'screenshot')
assert screenshot.artifacts[0].subject_kind == 'url'
assert screenshot.artifacts[0].subject_value == target_url
with sqlite3.connect(database) as db:
assert db.execute('PRAGMA user_version').fetchone()[0] == 7
assert db.execute('SELECT DISTINCT kind FROM legacy_observations').fetchall() == [('url',)]
assert db.execute('SELECT result_count FROM executions WHERE name = ?', ('api-scan',)).fetchone()[0] == 1
@pytest.mark.asyncio
async def test_schema_v5_merges_ip_address_into_ip_without_losing_provenance_or_artifacts(tmp_path) -> None:
database = tmp_path / 'stash.sqlite'
run_id = UUID('5b240ef2-e714-45de-b38f-174f20447f8b')
address = '192.0.2.1'
with sqlite3.connect(database) as db:
db.executescript(SCHEMA_V5_RESULT_KINDS)
db.execute(
'INSERT INTO runs (run_id, target, started_at, completed_at) VALUES (?, ?, ?, ?)',
(str(run_id), 'example.com', '2026-08-09T12:00:00+00:00', '2026-08-09T12:01:00+00:00'),
)
db.executemany(
'INSERT INTO results (run_id, position, kind, value) VALUES (?, ?, ?, ?)',
[
(str(run_id), 0, 'ip-address', address),
(str(run_id), 1, 'ip', address),
],
)
db.executemany(
'INSERT INTO executions '
'(run_id, position, producer_kind, name, status, duration_ms, result_count, error_type, stop_reason) '
'VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)',
[
(str(run_id), 0, 'source', 'dns', 'completed', 1.0, 2, None, None),
(str(run_id), 1, 'action', 'screenshot', 'completed', 2.0, 2, None, None),
],
)
db.executemany(
'INSERT INTO result_origins (run_id, result_position, execution_position) VALUES (?, ?, ?)',
[
(str(run_id), 0, 0),
(str(run_id), 1, 0),
(str(run_id), 0, 1),
(str(run_id), 1, 1),
],
)
db.execute(
'INSERT INTO artifacts '
'(run_id, position, result_position, execution_position, kind, path, media_type, size_bytes, sha256, created_at) '
'VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
(
str(run_id),
0,
0,
1,
'screenshot',
'screenshots/192.0.2.1.png',
'image/png',
3,
'0' * 64,
'2026-08-09T12:01:00+00:00',
),
)
db.executemany(
'INSERT INTO legacy_observations (domain, resource, kind, discovered_on, source) VALUES (?, ?, ?, ?, ?)',
[
('example.com', address, 'ip-address', '2026-08-09', 'dns'),
('example.com', address, 'ip', '2026-08-09', 'dns'),
],
)
store = ResultStore(database)
await store.initialize()
loaded = await store.load_run(run_id)
assert loaded.results == (('ip', address),)
assert [(item.source, item.kind, item.value) for item in loaded.observations] == [('dns', 'ip', address)]
assert loaded.source_executions[0].result_count == 1
screenshot = loaded.active_evidence.executions[0]
assert [(item.kind, item.value) for item in screenshot.observations] == [('ip', address)]
assert screenshot.artifacts[0].subject_kind == 'ip'
assert screenshot.artifacts[0].subject_value == address
with sqlite3.connect(database) as db:
assert db.execute('PRAGMA user_version').fetchone()[0] == 7
assert db.execute('SELECT kind, value FROM results').fetchall() == [('ip', address)]
assert db.execute('SELECT execution_position FROM result_origins ORDER BY execution_position').fetchall() == [
(0,),
(1,),
]
assert db.execute('SELECT result_count FROM executions ORDER BY position').fetchall() == [(1,), (1,)]
assert db.execute('SELECT DISTINCT kind FROM legacy_observations').fetchall() == [('ip',)]
@pytest.mark.asyncio
async def test_released_results_migrate_to_legacy_observations(tmp_path) -> None:
database = tmp_path / 'stash.sqlite'
@@ -376,15 +873,15 @@ async def test_released_results_migrate_to_legacy_observations(tmp_path) -> None
assert result_columns == ['run_id', 'position', 'kind', 'value']
assert observations == [
('api.example.com', 'hostname'),
('192.0.2.1', 'ip-address'),
('192.0.2.1', 'ip'),
('Ada Lovelace', 'person'),
('https://linkedin.test/ada', 'linkedin-link'),
('https://admin.example.com', 'interesting-url'),
('https://linkedin.test/ada', 'url'),
('https://admin.example.com', 'url'),
('AS64496', 'asn'),
('/api/v1', 'api-endpoint'),
('/api/v1', 'url'),
('admin@example.com', 'email'),
]
assert schema_version == 2
assert schema_version == 7
@pytest.mark.asyncio
@@ -456,6 +953,35 @@ async def test_completed_result_write_is_atomic_and_rejects_duplicate_run_id(tmp
result_count = db.execute('SELECT COUNT(*) FROM results').fetchone()[0]
assert (run_count, result_count) == (1, 7)
artifact_run_id = UUID('7ff120b6-4aec-4d27-b2db-d3ac9fd87340')
completed_at = datetime(2026, 8, 9, 12, 1, tzinfo=UTC)
failing_artifact = CompletedResult.finish(
run_id=artifact_run_id,
target='example.com',
started_at=datetime(2026, 8, 9, 12, 0, tzinfo=UTC),
completed_at=completed_at,
groups={'hostname': ['api.example.com']},
active_evidence=ActiveEvidence(executions=(screenshot_execution(completed_at),)),
)
with sqlite3.connect(database) as db:
db.execute(
f"""
CREATE TRIGGER fail_artifact
BEFORE INSERT ON artifacts
WHEN NEW.run_id = '{artifact_run_id}'
BEGIN
SELECT RAISE(ABORT, 'forced artifact failure');
END
"""
)
with pytest.raises(ResultStoreError, match='Could not save enumeration run'):
await store.save_run(failing_artifact)
with sqlite3.connect(database) as db:
assert db.execute('SELECT COUNT(*) FROM runs').fetchone()[0] == 1
assert db.execute('SELECT COUNT(*) FROM artifacts').fetchone()[0] == 0
@pytest.mark.asyncio
async def test_legacy_observations_keep_the_released_normalized_schema(tmp_path) -> None:
@@ -464,7 +990,7 @@ async def test_legacy_observations_keep_the_released_normalized_schema(tmp_path)
await store.initialize()
await store.record_observations('example.com', ['api.example.com', 'www.example.com'], 'hostname', 'crtsh')
await store.record_observations('example.com', ['admin@example.com'], 'email', 'hunter')
await store.record_observations('example.com', ['192.0.2.1'], 'ip-address', 'dns')
await store.record_observations('example.com', ['192.0.2.1'], 'ip', 'dns')
await store.record_observations('example.com', ['{"firstname":"Ada","lastname":"Lovelace"}'], 'person', 'hunter')
await store.record_observations('example.com', ['vhost.example.com'], 'vhost', 'virtual-host')
await store.record_observations('example.com', ['443'], 'shodan', 'shodan')
@@ -478,7 +1004,7 @@ async def test_legacy_observations_keep_the_released_normalized_schema(tmp_path)
('example.com', 'api.example.com', 'hostname', 'crtsh'),
('example.com', 'www.example.com', 'hostname', 'crtsh'),
('example.com', 'admin@example.com', 'email', 'hunter'),
('example.com', '192.0.2.1', 'ip-address', 'dns'),
('example.com', '192.0.2.1', 'ip', 'dns'),
('example.com', '{"firstname":"Ada","lastname":"Lovelace"}', 'person', 'hunter'),
('example.com', 'vhost.example.com', 'vhost', 'virtual-host'),
('example.com', '443', 'shodan', 'shodan'),
@@ -504,7 +1030,7 @@ async def test_schema_v1_observations_upgrade_without_losing_rows(tmp_path) -> N
schema_version = db.execute('PRAGMA user_version').fetchone()[0]
assert 'discovery_observations' not in tables
assert rows == [('example.com', 'api.example.com', 'hostname', 'crtsh')]
assert schema_version == 2
assert schema_version == 7
@pytest.mark.asyncio
tests/lib/test_completed_result.py
+117 -16
View File
@@ -5,9 +5,24 @@ from uuid import UUID
import pytest
from theHarvester.lib.active_evidence import ActionExecution, ActionObservation, ActiveEvidence, ArtifactReference
from theHarvester.lib.completed_result import CompletedResult, ResultObservation, SourceExecution
@pytest.mark.parametrize('evidence_status', ['partial', 'failed'])
def test_sparse_completed_result_retains_explicit_status(evidence_status: str) -> None:
result = CompletedResult.finish(
target='example.com',
started_at=datetime(2026, 8, 5, 12, 0, tzinfo=UTC),
completed_at=datetime(2026, 8, 5, 12, 1, tzinfo=UTC),
groups={},
evidence_status=evidence_status,
)
assert result.status == evidence_status
assert json.loads(result.jsonl().splitlines()[0])['evidence_status'] == evidence_status
def test_completed_result_is_deterministic_and_deduplicated() -> None:
result = CompletedResult.finish(
run_id=UUID('f047261c-0afb-4e18-89d5-28a7d977f51f'),
@@ -26,8 +41,12 @@ def test_completed_result_is_deterministic_and_deduplicated() -> None:
{
'completed_at': '2026-08-05T12:01:00Z',
'counts': {'email': 1, 'hostname': 2},
'evidence_status': 'complete',
'result_count': 3,
'run_id': 'f047261c-0afb-4e18-89d5-28a7d977f51f',
'source_executions': [],
'action_executions': [],
'artifacts': [],
'started_at': '2026-08-05T12:00:00Z',
'target': 'example.com',
'type': 'summary',
@@ -167,29 +186,111 @@ def test_completed_result_rejects_source_count_without_matching_origins() -> Non
)
def test_completed_result_keeps_terminal_action_evidence_in_jsonl() -> None:
def test_completed_result_merges_active_results_and_keeps_screenshot_as_an_artifact() -> None:
completed_at = datetime(2026, 8, 5, 12, 1, tzinfo=UTC)
artifact = ArtifactReference(
kind='screenshot',
subject_kind='hostname',
subject_value='api.example.com',
path='screenshots/api.example.com.png',
media_type='image/png',
size_bytes=3,
sha256='0' * 64,
created_at=completed_at,
)
result = CompletedResult.finish(
target='example.com',
started_at=completed_at,
completed_at=completed_at,
groups={'hostname': ['api.example.com']},
active_evidence=ActiveEvidence(
executions=(
ActionExecution.finish(
action='dns-resolve',
status='completed',
duration_ms=12.5,
groups={'ip': ['192.0.2.10']},
),
ActionExecution.finish(
action='screenshot',
status='completed',
duration_ms=4.0,
groups={},
artifacts=(artifact,),
),
)
),
)
assert result.results == (('hostname', 'api.example.com'), ('ip', '192.0.2.10'))
assert result.active_evidence.executions[0].observations == (ActionObservation('ip', '192.0.2.10'),)
assert result.active_evidence.executions[1].artifacts == (artifact,)
assert not any(kind == 'screenshot' for kind, _value in result.results)
assert result.evidence_dict()['results'] == [
{'type': 'hostname', 'value': 'api.example.com', 'sources': []},
{'type': 'ip', 'value': '192.0.2.10', 'sources': [], 'actions': ['dns-resolve']},
]
assert [json.loads(line) for line in result.jsonl().splitlines()][1:] == [
{'type': 'hostname', 'value': 'api.example.com', 'sources': []},
{'type': 'ip', 'value': '192.0.2.10', 'sources': [], 'actions': ['dns-resolve']},
]
def test_completed_result_rejects_artifact_without_a_real_subject_result() -> None:
completed_at = datetime(2026, 8, 5, 12, 1, tzinfo=UTC)
artifact = ArtifactReference(
kind='screenshot',
subject_kind='hostname',
subject_value='missing.example.com',
path='screenshots/missing.example.com.png',
media_type='image/png',
size_bytes=3,
sha256='0' * 64,
created_at=completed_at,
)
with pytest.raises(ValueError, match='artifact must reference a completed result'):
CompletedResult.finish(
target='example.com',
started_at=completed_at,
completed_at=completed_at,
groups={},
active_evidence=ActiveEvidence(
executions=(
ActionExecution.finish(
action='screenshot',
status='completed',
duration_ms=4.0,
groups={},
artifacts=(artifact,),
),
)
),
)
def test_action_status_contributes_to_completed_result_status() -> None:
completed_at = datetime(2026, 8, 5, 12, 1, tzinfo=UTC)
result = CompletedResult.finish(
target='example.com',
started_at=completed_at,
completed_at=completed_at,
groups={
'api-endpoint': ['/api/v1'],
'screenshot': ['https://api.example.com'],
'shodan': ['{"ip":"192.0.2.10","ports":[443]}'],
'takeover': ['{"matches":[{"No such app":"Heroku"}],"url":"https://old.example.com"}'],
},
groups={},
source_executions=(SourceExecution('crtsh', 'completed', 1.0, 0),),
active_evidence=ActiveEvidence(
executions=(
ActionExecution.finish(
action='takeover',
status='failed',
duration_ms=2.0,
groups={},
error_type='RuntimeError',
),
)
),
)
records = [json.loads(line) for line in result.jsonl().splitlines()]
assert records[0]['counts'] == {'api-endpoint': 1, 'screenshot': 1, 'shodan': 1, 'takeover': 1}
assert {(record['type'], record['value']) for record in records[1:]} == {
('api-endpoint', '/api/v1'),
('screenshot', 'https://api.example.com'),
('shodan', '{"ip":"192.0.2.10","ports":[443]}'),
('takeover', '{"matches":[{"No such app":"Heroku"}],"url":"https://old.example.com"}'),
}
assert result.evidence_dict()['status'] == 'partial'
@pytest.mark.parametrize('value', ['', ' ', 7])
tests/lib/test_core.py
+20 -1
View File
@@ -441,6 +441,7 @@ async def test_takeover_fetch_uses_the_shared_transport(
'url': url,
'proxy': proxy,
'request_timeout': 15,
'include_metadata': False,
}
]
@@ -462,7 +463,25 @@ async def test_takeover_fetch_all_falls_back_to_direct_when_proxy_pool_is_empty(
assert result == [('http://example.com', 'direct response')]
assert len(calls) == 1
assert calls[0][1] == {'proxy': None}
assert calls[0][1] == {'proxy': None, 'include_metadata': False}
@pytest.mark.asyncio
async def test_takeover_fetch_all_propagates_metadata_opt_in(monkeypatch) -> None:
reset_dummy_sessions()
seen: list[bool] = []
monkeypatch.setattr(core_module.aiohttp, 'ClientSession', DummySession)
async def fake_takeover_fetch(*_args, include_metadata: bool = False, **_kwargs):
seen.append(include_metadata)
return 'https://example.com', FetcherResponse(body='', status=204, headers={})
monkeypatch.setattr(AsyncFetcher, 'takeover_fetch', fake_takeover_fetch)
result = await AsyncFetcher.fetch_all(['https://example.com'], takeover=True, include_metadata=True)
assert seen == [True]
assert result[0][1].status == 204
@pytest.mark.asyncio
tests/lib/test_hostchecker.py
+92
View File
@@ -75,6 +75,8 @@ async def test_dns_force_preserves_legacy_result_and_typed_records(monkeypatch:
def __init__(self, _hosts: list[str], nameservers: list[str]) -> None:
assert nameservers == ['192.0.2.53']
self.records = records
self.query_error_count = 2
self.query_error_types = {'TimeoutError'}
async def check(self) -> tuple[list[str], list[str], list[str]]:
return ['www.example.com:192.0.2.10'], ['www.example.com'], ['192.0.2.10']
@@ -87,6 +89,15 @@ async def test_dns_force_preserves_legacy_result_and_typed_records(monkeypatch:
assert result == (['www.example.com:192.0.2.10'], ['www.example.com'], ['192.0.2.10'])
assert dns_force.records is records
assert dns_force.query_error_count == 2
assert dns_force.query_error_types == {'TimeoutError'}
def test_dns_force_preserves_selected_www_target() -> None:
dns_force = dnssearch.DnsForce('www.example.com', ['192.0.2.53'])
assert dns_force.domain == 'www.example.com'
assert all(candidate.endswith('.www.example.com') for candidate in dns_force.list)
@pytest.mark.asyncio
@@ -145,6 +156,87 @@ async def test_check_excludes_candidate_without_usable_evidence(
assert checker.records == {}
@pytest.mark.asyncio
async def test_check_distinguishes_expected_absence_from_query_failures(monkeypatch: pytest.MonkeyPatch) -> None:
not_found = hostchecker.aiodns.error.DNSError(hostchecker.aiodns.error.ARES_ENOTFOUND, 'not found')
no_data = hostchecker.aiodns.error.DNSError(hostchecker.aiodns.error.ARES_ENODATA, 'no data')
class FakeResolver:
async def query_dns(self, host: str, _record_type: str):
if host == 'missing.example.com':
raise not_found
if host == 'empty.example.com':
raise no_data
raise TimeoutError('resolver timed out')
monkeypatch.setattr(hostchecker.aiodns, 'DNSResolver', lambda **_kwargs: FakeResolver())
checker = hostchecker.Checker(
['missing.example.com', 'empty.example.com', 'timeout.example.com'],
nameservers=[],
)
assert await checker.check() == ([], [], [])
assert checker.query_error_count == 3
assert checker.query_error_types == {'TimeoutError'}
@pytest.mark.asyncio
async def test_dns_force_defaults_diagnostics_for_existing_checker_contract(monkeypatch: pytest.MonkeyPatch) -> None:
class ExistingChecker:
records: dict[str, hostchecker.HostDnsRecords] = {}
def __init__(self, _hosts: list[str], nameservers: list[str]) -> None:
assert nameservers == []
async def check(self) -> tuple[list[str], list[str], list[str]]:
return [], [], []
monkeypatch.setattr(dnssearch.hostchecker, 'Checker', ExistingChecker)
dns_force = dnssearch.DnsForce('example.com', [])
dns_force.list = []
assert await dns_force.run() == ([], [], [])
assert dns_force.query_error_count == 0
assert dns_force.query_error_types == set()
@pytest.mark.asyncio
async def test_reverse_single_ip_keeps_transport_failures_as_empty_results() -> None:
class FakeResolver:
async def gethostbyaddr(self, _ip: str):
raise TimeoutError('resolver timed out')
assert await dnssearch.reverse_single_ip('192.0.2.10', FakeResolver()) == ''
@pytest.mark.asyncio
async def test_reverse_range_reports_only_unexpected_ptr_errors(monkeypatch: pytest.MonkeyPatch) -> None:
not_found = hostchecker.aiodns.error.DNSError(hostchecker.aiodns.error.ARES_ENOTFOUND, 'not found')
class FakeResolver:
async def gethostbyaddr(self, ip: str):
if ip == '192.0.2.1':
return SimpleNamespace(name='api.example.com')
if ip == '192.0.2.2':
raise not_found
raise TimeoutError('resolver timed out')
monkeypatch.setattr(dnssearch, 'list_ips_in_network_range', lambda _range: ['192.0.2.1', '192.0.2.2', '192.0.2.3'])
monkeypatch.setattr(dnssearch, 'DNSResolver', lambda **_kwargs: FakeResolver())
monkeypatch.setattr(dnssearch, 'log_query', lambda _ip: None)
results: list[str] = []
error_types: set[str] = set()
await dnssearch.reverse_all_ips_in_range(
'192.0.2.0/24',
results.append,
error_types=error_types,
)
assert results == ['api.example.com', '', '']
assert error_types == {'TimeoutError'}
@pytest.mark.asyncio
async def test_check_propagates_cancellation(monkeypatch: pytest.MonkeyPatch) -> None:
class FakeResolver:
tests/lib/test_output.py
+10 -24
View File
@@ -1,39 +1,25 @@
from __future__ import annotations
from theHarvester.lib.output import configure_logging, print_linkedin_sections, sorted_unique
from theHarvester.lib.output import configure_logging, print_linkedin_people, sorted_unique
def test_sorted_unique_sorts_and_deduplicates() -> None:
assert sorted_unique(["b", "a", "b"]) == ["a", "b"]
assert sorted_unique(['b', 'a', 'b']) == ['a', 'b']
def test_print_linkedin_sections_prints_links_when_present(capsys) -> None:
# Regression coverage: the CLI previously never printed LinkedIn links when the list was non-empty.
def test_print_linkedin_people_reports_no_users(capsys) -> None:
configure_logging(verbose=False)
print_linkedin_sections(
engines=["linkedin"],
people=[],
links=["https://b.example", "https://a.example", "https://a.example"],
)
print_linkedin_people(engines=['linkedin'], people=[])
out = capsys.readouterr().out
assert "No LinkedIn users found" in out
assert "LinkedIn Links found: 3" in out
assert "https://a.example" in out
assert "https://b.example" in out
assert 'No LinkedIn users found' in out
def test_print_linkedin_sections_prints_people_and_links(capsys) -> None:
def test_print_linkedin_people_prints_people(capsys) -> None:
configure_logging(verbose=False)
print_linkedin_sections(
engines=["rocketreach"],
people=["bob", "alice", "bob"],
links=["https://z.example", "https://z.example"],
)
print_linkedin_people(engines=['rocketreach'], people=['bob', 'alice', 'bob'])
out = capsys.readouterr().out
assert "LinkedIn Users found: 3" in out
assert "alice" in out
assert "bob" in out
assert "LinkedIn Links found: 2" in out
assert "https://z.example" in out
assert 'LinkedIn Users found: 3' in out
assert 'alice' in out
assert 'bob' in out
tests/lib/test_run_backend.py
+984
View File
@@ -0,0 +1,984 @@
from __future__ import annotations
import asyncio
import inspect
import json
import os
import sqlite3
import sys
import time
from datetime import UTC, datetime
from pathlib import Path
from uuid import UUID
import pytest
from fastapi.testclient import TestClient
def test_run_paths_use_one_expanded_database_and_artifact_root(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api.run_store import RunStore
monkeypatch.setenv('HOME', str(tmp_path))
monkeypatch.setenv('THEHARVESTER_RUN_DB', '~/state/runs.sqlite')
monkeypatch.delenv('THEHARVESTER_RUN_ARTIFACTS', raising=False)
store = RunStore()
assert store.database == tmp_path / 'state' / 'runs.sqlite'
assert store.artifact_directory('run-id') == tmp_path / 'state' / 'run-artifacts' / 'run-id'
def test_run_store_does_not_change_caller_owned_directory_permissions(tmp_path) -> None:
from theHarvester.lib.api.run_store import RunStore
tmp_path.chmod(0o755)
asyncio.run(RunStore(tmp_path / 'runs.sqlite').initialize())
assert os.stat(tmp_path).st_mode & 0o777 == 0o755
def test_run_history_is_bounded_without_loading_completed_evidence(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
async def fail_load(*_args, **_kwargs):
raise AssertionError('run summaries must not hydrate terminal evidence')
async def scenario():
store = RunStore(tmp_path / 'runs.sqlite')
for index in range(4):
await store.create(RunRequest(target=f'{index}.example.test', sources=['crtsh']))
monkeypatch.setattr(store.results, 'load_run', fail_load)
return await store.list_runs(limit=2, offset=1)
history = asyncio.run(scenario())
assert len(history) == 2
assert [run['target'] for run in history] == ['2.example.test', '1.example.test']
assert all(run['result_count'] == 0 for run in history)
def test_run_history_breaks_timestamp_ties_by_run_id(tmp_path) -> None:
from theHarvester.lib.database import RunLifecycleStore
async def scenario() -> list[str]:
store = RunLifecycleStore(tmp_path / 'runs.sqlite')
await store.initialize()
for run_id in ('run-a', 'run-c', 'run-b'):
await store.create(
run_id=run_id,
target='example.test',
status='completed',
origin='imported',
created_at='2026-08-09T12:00:00+00:00',
request_json='{}',
)
first = await store.list_records(limit=2, offset=0)
second = await store.list_records(limit=2, offset=2)
return [str(run['run_id']) for run in first + second]
assert asyncio.run(scenario()) == ['run-c', 'run-b', 'run-a']
def test_api_lifespan_disposes_shared_sqlite_engines(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
disposed = False
async def no_op() -> None:
return None
async def dispose() -> None:
nonlocal disposed
disposed = True
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
monkeypatch.setattr(api, 'start_worker', no_op)
monkeypatch.setattr(api, 'stop_worker', no_op)
monkeypatch.setattr(api, 'dispose_sqlite_databases', dispose)
with TestClient(api.app):
pass
assert disposed is True
def test_static_assets_are_resolved_from_the_installed_module(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
monkeypatch.chdir(tmp_path)
assert api.STATIC_DIRECTORY == Path(api.__file__).resolve().parent / 'static'
def test_explicit_run_database_keeps_screenshot_artifacts_attached(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api.run_store import RunStore
monkeypatch.delenv('THEHARVESTER_RUN_DB', raising=False)
monkeypatch.delenv('THEHARVESTER_RUN_ARTIFACTS', raising=False)
async def scenario():
store = RunStore(tmp_path / 'state' / 'runs.sqlite')
imported = await store.import_evidence(
{
'run_id': '4a6e5a15-fae5-462c-a34b-122ced6bb86d',
'target': 'example.test',
'status': 'complete',
'started_at': '2026-08-09T12:00:00+00:00',
'completed_at': '2026-08-09T12:01:00+00:00',
'results': [{'type': 'hostname', 'value': 'owned.example.test', 'actions': []}],
'source_executions': [],
'action_executions': [
{
'action': 'screenshot',
'status': 'completed',
'duration_ms': 1,
'result_count': 0,
'error_type': None,
'stop_reason': None,
}
],
'artifacts': [
{
'action': 'screenshot',
'kind': 'screenshot',
'subject': {'kind': 'hostname', 'value': 'owned.example.test'},
'file': {
'path': 'screenshots/owned.example.test.png',
'media_type': 'image/png',
'size_bytes': 3,
'sha256': '0' * 64,
},
'created_at': '2026-08-09T12:01:00+00:00',
}
],
},
'evidence.jsonl',
)
screenshot_dir = store.artifact_directory(imported['run_id']) / 'screenshots'
screenshot_dir.mkdir(parents=True)
(screenshot_dir / 'owned.example.test.png').write_bytes(b'png')
return await store.get(imported['run_id'])
run = asyncio.run(scenario())
assert run is not None
assert run['evidence']['run_id'] == run['run_id']
assert run['request']['source_run_id'] == '4a6e5a15-fae5-462c-a34b-122ced6bb86d'
assert [screenshot['name'] for screenshot in run['screenshots']] == ['owned.example.test.png']
def test_api_lifecycle_and_terminal_evidence_share_the_sqlalchemy_database(tmp_path) -> None:
from theHarvester.lib.api import run_store as run_store_module
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
database = tmp_path / 'stash.sqlite'
async def scenario() -> str:
store = RunStore(database)
queued = await store.create(RunRequest(target='example.test', sources=['crtsh']))
await store.claim_next()
await store.finish(
queued['run_id'],
{
'run_id': '4a6e5a15-fae5-462c-a34b-122ced6bb86d',
'target': 'example.test',
'status': 'complete',
'started_at': '2026-08-09T12:00:00+00:00',
'completed_at': '2026-08-09T12:01:00+00:00',
'results': [{'type': 'hostname', 'value': 'api.example.test', 'sources': ['crtsh']}],
'source_executions': [
{
'source': 'crtsh',
'status': 'completed',
'duration_ms': 1,
'result_count': 1,
'error_type': None,
'stop_reason': None,
}
],
},
'',
)
return queued['run_id']
lifecycle_run_id = asyncio.run(scenario())
with sqlite3.connect(database) as db:
evidence_run_id = db.execute(
'SELECT evidence_run_id FROM run_records WHERE run_id = ?',
(lifecycle_run_id,),
).fetchone()[0]
stored_target = db.execute('SELECT target FROM runs WHERE run_id = ?', (evidence_run_id,)).fetchone()[0]
schema_version = db.execute('PRAGMA user_version').fetchone()[0]
assert evidence_run_id == lifecycle_run_id
assert stored_target == 'example.test'
assert schema_version == 7
assert 'import aiosqlite' not in inspect.getsource(run_store_module)
def test_run_store_distinguishes_no_evidence_from_a_broken_evidence_link(tmp_path) -> None:
from uuid import uuid4
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
from theHarvester.lib.database import ResultStoreError
database = tmp_path / 'runs.sqlite'
store = RunStore(database)
queued = asyncio.run(store.create(RunRequest(target='example.test', sources=['crtsh'])))
assert asyncio.run(store.load_completed_result(queued['run_id'])) is None
with sqlite3.connect(database) as db:
db.execute(
'UPDATE run_records SET evidence_run_id = ? WHERE run_id = ?',
(str(uuid4()), queued['run_id']),
)
with pytest.raises(ResultStoreError, match='Attached run evidence does not exist'):
asyncio.run(store.load_completed_result(queued['run_id']))
def test_child_execution_passes_the_configured_database_to_core(tmp_path, monkeypatch) -> None:
from theHarvester import __main__ as main_module
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
from theHarvester.lib.api.run_worker import _child_execute
from theHarvester.lib.completed_result import CompletedResult
database = tmp_path / 'state' / 'runs.sqlite'
seen_database = None
seen_run_id = None
async def fake_start(_args, **kwargs):
nonlocal seen_database, seen_run_id
seen_database = kwargs.get('result_database')
seen_run_id = kwargs.get('completed_run_id')
now = datetime.now(UTC)
result = CompletedResult.finish(
run_id=seen_run_id,
target='example.test',
started_at=now,
completed_at=now,
groups={},
)
return (result,)
async def scenario() -> None:
store = RunStore(database)
queued = await store.create(RunRequest(target='example.test', sources=['crtsh']))
await store.claim_next()
monkeypatch.setattr(main_module, 'start', fake_start)
await _child_execute(queued['run_id'], database)
asyncio.run(scenario())
assert seen_database == database
with sqlite3.connect(database) as db:
lifecycle_run_id = db.execute('SELECT run_id FROM run_records').fetchone()[0]
assert str(seen_run_id) == lifecycle_run_id
def test_child_screenshot_run_persists_downloadable_artifact_metadata(tmp_path, monkeypatch) -> None:
from theHarvester import __main__ as main_module
from theHarvester.lib.api.run_artifacts import read_child_evidence
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
from theHarvester.lib.api.run_worker import _child_execute
class FakeScreenShotter:
slash = '/'
def __init__(self, output: str) -> None:
self.output = output
def verify_path(self) -> bool:
return True
async def verify_installation(self) -> None:
return None
async def visit(self, host: str) -> tuple[str, str]:
return f'https://{host}', 'reachable'
@staticmethod
def chunk_list(values: list[str], _size: int) -> list[list[str]]:
return [values]
def screenshot_path(self, url: str) -> Path:
return Path(self.output) / f'{url.removeprefix("https://")}.png'
async def take_screenshot(self, url: str, *, output_path: Path | None = None) -> str:
captured_url = url if url.startswith('https://') else f'https://{url}'
(output_path or self.screenshot_path(captured_url)).write_bytes(b'png')
return captured_url
class FakePool:
def __init__(self, _workers: int) -> None:
pass
async def __aenter__(self):
return self
async def __aexit__(self, *_args) -> None:
return None
async def map(self, function, values):
return [await function(value) for value in values]
database = tmp_path / 'state' / 'runs.sqlite'
monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts'))
monkeypatch.setattr(main_module, 'ScreenShotter', FakeScreenShotter)
monkeypatch.setattr(main_module, 'Pool', FakePool)
async def scenario():
store = RunStore(database)
queued = await store.create(RunRequest(target='api.example.test', sources=[], screenshot=True))
await store.claim_next()
await _child_execute(queued['run_id'], database)
evidence, error = read_child_evidence(store.artifact_directory(queued['run_id']))
assert error is None
assert evidence is not None
await store.finish(queued['run_id'], evidence, '')
return await store.get(queued['run_id']), store.artifact_directory(queued['run_id'])
run, artifact_dir = asyncio.run(scenario())
assert run is not None
assert run['action_executions'][0]['action'] == 'screenshot'
assert run['action_executions'][0]['status'] == 'completed'
assert run['results'] == [{'type': 'hostname', 'value': 'api.example.test', 'sources': [], 'actions': []}]
assert [screenshot['name'] for screenshot in run['screenshots']] == ['api.example.test.png']
assert (artifact_dir / 'screenshots' / 'api.example.test.png').read_bytes() == b'png'
def test_child_screenshot_cancellation_reuses_the_checkpointed_evidence(tmp_path, monkeypatch) -> None:
from theHarvester import __main__ as main_module
from theHarvester.lib.api.run_artifacts import read_child_evidence
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
from theHarvester.lib.api.run_worker import _child_execute
first_captured = asyncio.Event()
class FakeScreenShotter:
slash = '/'
def __init__(self, output: str) -> None:
self.output = output
def verify_path(self) -> bool:
return True
async def verify_installation(self) -> None:
return None
async def visit(self, host: str) -> tuple[str, str]:
return f'https://{host}', 'reachable'
async def take_screenshot(self, url: str, *, output_path: Path | None = None) -> str:
assert output_path is not None
if 'first.' in url:
output_path.write_bytes(b'png') # noqa: ASYNC240 - tiny in-memory screenshot fixture
first_captured.set()
return url
await first_captured.wait()
raise asyncio.CancelledError
def screenshot_path(self, url: str) -> Path:
return Path(self.output) / f'{url.removeprefix("https://")}.png'
class TwoHostSource:
def __init__(self, _word: str) -> None:
pass
async def process(self, _proxy: bool) -> None:
return None
async def get_hostnames(self) -> set[str]:
return {'first.example.test', 'second.example.test'}
class FakePool:
def __init__(self, _workers: int) -> None:
pass
async def __aenter__(self):
return self
async def __aexit__(self, *_args) -> None:
return None
async def map(self, function, values):
return [await function(value) for value in values]
database = tmp_path / 'runs.sqlite'
monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts'))
monkeypatch.setattr(main_module.crtsh, 'SearchCrtsh', TwoHostSource)
monkeypatch.setattr(main_module, 'ScreenShotter', FakeScreenShotter)
monkeypatch.setattr(main_module, 'Pool', FakePool)
async def scenario():
store = RunStore(database)
queued = await store.create(RunRequest(target='example.test', sources=['crtsh'], screenshot=True))
await store.claim_next()
await _child_execute(queued['run_id'], database)
evidence, error = read_child_evidence(store.artifact_directory(queued['run_id']))
assert error is None
assert evidence is not None
await store.fail(queued['run_id'], 'cancelled', '', cancelled=True, evidence=evidence)
return await store.get(queued['run_id'])
run = asyncio.run(scenario())
assert run is not None
assert run['status'] == 'cancelled'
assert run['action_executions'][0]['status'] == 'partial'
assert [screenshot['target'] for screenshot in run['screenshots']] == ['first.example.test']
def test_sqlite_import_preserves_run_ids_and_is_idempotent(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import run_store
from theHarvester.lib.api.run_store import RunStore
from theHarvester.lib.completed_result import CompletedResult
from theHarvester.lib.database import ResultStore, dispose_sqlite_databases
source_database = tmp_path / 'source.sqlite'
destination_database = tmp_path / 'destination.sqlite'
now = datetime.now(UTC)
first = CompletedResult.finish(
target='first.example.test',
started_at=now,
completed_at=now,
groups={'hostname': ['api.first.example.test']},
)
second = CompletedResult.finish(
target='second.example.test',
started_at=now,
completed_at=now,
groups={'email': ['security@second.example.test']},
)
list_calls: list[tuple[int | None, int]] = []
original_list_runs = ResultStore.list_runs
async def track_source_batches(self, *, limit=50, offset=0):
if Path(self.database) == source_database.resolve():
list_calls.append((limit, offset))
return await original_list_runs(self, limit=limit, offset=offset)
monkeypatch.setattr(run_store, 'DATABASE_IMPORT_BATCH_SIZE', 1)
monkeypatch.setattr(ResultStore, 'list_runs', track_source_batches)
async def scenario():
source = ResultStore(source_database)
await source.initialize()
await source.save_run(first)
await source.save_run(second)
await dispose_sqlite_databases()
destination = RunStore(destination_database)
imported = await destination.import_database(source_database, 'source.sqlite')
first_import_calls = list(list_calls)
list_calls.clear()
repeated = await destination.import_database(source_database, 'source.sqlite')
history = await destination.list_runs()
return imported, repeated, history, first_import_calls
imported, repeated, history, first_import_calls = asyncio.run(scenario())
expected_ids = sorted((str(first.run_id), str(second.run_id)))
assert imported == {'filename': 'source.sqlite', 'imported_run_ids': expected_ids, 'skipped_run_ids': []}
assert repeated == {'filename': 'source.sqlite', 'imported_run_ids': [], 'skipped_run_ids': expected_ids}
assert sorted(run['run_id'] for run in history) == expected_ids
assert first_import_calls == [(1, 0), (1, 1), (1, 2), (1, 0), (1, 1), (1, 2)]
def test_orphan_recovery_reattaches_partial_checkpoint_and_leaves_queued_work(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api.run_artifacts import ensure_private_directory, write_child_evidence
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
from theHarvester.lib.completed_result import CompletedResult
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts'))
async def scenario():
store = RunStore()
for target in ('first.example', 'second.example', 'third.example'):
await store.create(RunRequest(target=target, sources=['crtsh']))
cancelling = await store.claim_next()
assert cancelling is not None
await store.cancel(cancelling['run_id'])
artifact_dir = store.artifact_directory(cancelling['run_id'])
ensure_private_directory(artifact_dir)
now = datetime.now(UTC)
checkpoint = CompletedResult.finish(
target='first.example',
started_at=now,
completed_at=now,
groups={'email': ['saved@first.example']},
)
write_child_evidence(artifact_dir, checkpoint, partial=True)
running = await store.claim_next()
assert running is not None
persisted = CompletedResult.finish(
run_id=UUID(running['run_id']),
target=str(running['target']),
started_at=now,
completed_at=now,
groups={'hostname': [f'api.{running["target"]}']},
)
await store.results.save_run(persisted)
await store.recover_orphans()
return await store.get(cancelling['run_id']), await store.get(running['run_id']), await store.list_runs()
cancelling, running, history = asyncio.run(scenario())
assert cancelling is not None
assert cancelling['status'] == 'failed'
assert cancelling['evidence_status'] == 'partial'
assert cancelling['results'] == [{'type': 'email', 'value': 'saved@first.example', 'sources': [], 'actions': []}]
assert running is not None
assert running['status'] == 'failed'
assert running['results'] == [{'type': 'hostname', 'value': f'api.{running["target"]}', 'sources': [], 'actions': []}]
assert {run['target']: run['status'] for run in history}['third.example'] == 'queued'
def test_orphan_recovery_does_not_attach_same_id_evidence_for_another_target(tmp_path) -> None:
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
from theHarvester.lib.completed_result import CompletedResult
async def scenario():
store = RunStore(tmp_path / 'runs.sqlite')
queued = await store.create(RunRequest(target='expected.example', sources=['crtsh']))
await store.claim_next()
now = datetime.now(UTC)
await store.results.save_run(
CompletedResult.finish(
run_id=UUID(queued['run_id']),
target='different.example',
started_at=now,
completed_at=now,
groups={'hostname': ['api.different.example']},
)
)
await store.recover_orphans()
return await store.get(queued['run_id'])
run = asyncio.run(scenario())
assert run is not None
assert run['status'] == 'failed'
assert run['results'] == []
def test_orphan_recovery_rejects_checkpoint_for_another_target(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api.run_artifacts import ensure_private_directory, write_child_evidence
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
from theHarvester.lib.completed_result import CompletedResult
monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts'))
async def scenario():
store = RunStore(tmp_path / 'runs.sqlite')
queued = await store.create(RunRequest(target='expected.example', sources=['crtsh']))
await store.claim_next()
artifact_dir = store.artifact_directory(queued['run_id'])
ensure_private_directory(artifact_dir)
now = datetime.now(UTC)
write_child_evidence(
artifact_dir,
CompletedResult.finish(
target='different.example',
started_at=now,
completed_at=now,
groups={'hostname': ['api.different.example']},
),
partial=True,
)
await store.recover_orphans()
return await store.get(queued['run_id'])
run = asyncio.run(scenario())
assert run is not None
assert run['status'] == 'failed'
assert run['results'] == []
assert 'does not match run target' in run['error']
@pytest.mark.parametrize('operation', ['finish', 'fail'])
def test_terminal_run_rejects_evidence_for_another_target(tmp_path, operation) -> None:
from fastapi import HTTPException
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
from theHarvester.lib.completed_result import CompletedResult
async def scenario():
store = RunStore(tmp_path / 'runs.sqlite')
queued = await store.create(RunRequest(target='expected.example', sources=['crtsh']))
await store.claim_next()
now = datetime.now(UTC)
evidence = CompletedResult.finish(
target='different.example',
started_at=now,
completed_at=now,
groups={'hostname': ['api.different.example']},
).evidence_dict()
terminal = (
store.finish(queued['run_id'], evidence, '')
if operation == 'finish'
else store.fail(queued['run_id'], 'failed', '', evidence=evidence)
)
with pytest.raises(HTTPException, match='Evidence target does not match run target'):
await terminal
with pytest.raises(LookupError):
await store.results.load_run(UUID(queued['run_id']))
asyncio.run(scenario())
def test_worker_lease_serializes_execution_owners(tmp_path) -> None:
from theHarvester.lib.api.run_store import RunStore
async def scenario() -> tuple[bool, bool, bool]:
store = RunStore(tmp_path / 'runs.sqlite')
first = await store.acquire_worker_lease('worker-a')
second = await store.acquire_worker_lease('worker-b')
await store.release_worker_lease('worker-a')
replacement = await store.acquire_worker_lease('worker-b')
return first, second, replacement
assert asyncio.run(scenario()) == (True, False, True)
def test_submission_fails_closed_when_worker_supervisor_stops(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api, run_worker
async def no_op() -> None:
return None
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setattr(api, 'start_worker', no_op)
monkeypatch.setattr(api, 'stop_worker', no_op)
monkeypatch.setattr(run_worker, 'worker_enabled', lambda: True)
monkeypatch.setattr(run_worker, '_worker_task', type('StoppedTask', (), {'done': lambda self: True})())
with TestClient(api.app) as client:
response = client.post(
'/api/v1/runs',
headers={'X-API-Key': 'test-key'},
json={'target': 'example.com', 'sources': ['crtsh']},
)
history = client.get('/api/v1/runs', headers={'X-API-Key': 'test-key'})
assert response.status_code == 503
assert response.json()['detail'] == 'theHarvester execution worker is unavailable'
assert history.json() == []
def test_authenticated_operator_can_queue_direct_activity_for_selected_target(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api, run_worker
async def no_op() -> None:
return None
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setattr(api, 'start_worker', no_op)
monkeypatch.setattr(api, 'stop_worker', no_op)
monkeypatch.setattr(run_worker, 'worker_enabled', lambda: True)
monkeypatch.setattr(run_worker, '_worker_task', type('RunningTask', (), {'done': lambda self: False})())
with TestClient(api.app) as client:
response = client.post(
'/api/v1/runs',
headers={'X-API-Key': 'test-key'},
json={'target': '192.0.2.8', 'sources': ['criminalip']},
)
assert response.status_code == 201
assert response.json()['target'] == '192.0.2.8'
assert response.json()['activities'] == ['P2']
def test_authenticated_operator_can_queue_screenshot_only_run(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api, run_worker
async def no_op() -> None:
return None
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setattr(api, 'start_worker', no_op)
monkeypatch.setattr(api, 'stop_worker', no_op)
monkeypatch.setattr(run_worker, 'worker_enabled', lambda: True)
monkeypatch.setattr(run_worker, '_worker_task', type('RunningTask', (), {'done': lambda self: False})())
with TestClient(api.app) as client:
response = client.post(
'/api/v1/runs',
headers={'X-API-Key': 'test-key'},
json={'target': 'api.example.com', 'sources': [], 'screenshot': True},
)
assert response.status_code == 201
assert response.json()['target'] == 'api.example.com'
assert response.json()['sources'] == []
assert response.json()['activities'] == ['P2']
def test_dns_brute_run_accepts_operator_resolver_list(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api, run_worker
async def no_op() -> None:
return None
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setattr(api, 'start_worker', no_op)
monkeypatch.setattr(api, 'stop_worker', no_op)
monkeypatch.setattr(run_worker, 'worker_enabled', lambda: True)
monkeypatch.setattr(run_worker, '_worker_task', type('RunningTask', (), {'done': lambda self: False})())
with TestClient(api.app) as client:
response = client.post(
'/api/v1/runs',
headers={'X-API-Key': 'test-key'},
json={
'target': 'dev.api.example.com',
'sources': [],
'dns_brute': True,
'dns_resolvers': ['192.0.2.53'],
},
)
assert response.status_code == 201
assert response.json()['activities'] == ['P1']
assert response.json()['request']['dns_resolvers'] == ['192.0.2.53']
def test_dns_brute_child_uses_operator_resolver_list(tmp_path, monkeypatch) -> None:
from theHarvester import __main__ as main_module
from theHarvester.lib.api import run_worker
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
from theHarvester.lib.completed_result import CompletedResult
received_options = []
async def fake_start(options, **_kwargs):
received_options.append(options)
now = datetime.now(UTC)
return (CompletedResult.finish(target=options.domain, started_at=now, completed_at=now, groups={}),)
monkeypatch.setattr(main_module, 'start', fake_start)
async def scenario() -> None:
store = RunStore(tmp_path / 'runs.sqlite')
created = await store.create(
RunRequest(
target='dev.api.example.com',
sources=[],
dns_brute=True,
dns_resolvers=['192.0.2.53'],
)
)
assert await store.claim_next() is not None
await run_worker._child_execute(created['run_id'], store.database)
asyncio.run(scenario())
assert received_options[0].source == ''
assert received_options[0].dns_brute is True
assert received_options[0].dns_resolve == ''
assert received_options[0].dns_resolvers == ('192.0.2.53',)
def test_api_scan_child_uses_operator_endpoint_paths(tmp_path, monkeypatch) -> None:
from theHarvester import __main__ as main_module
from theHarvester.lib.api import run_worker
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
from theHarvester.lib.completed_result import CompletedResult
received_wordlists: list[Path] = []
async def fake_start(options, **_kwargs):
received_wordlists.append(Path(options.wordlist))
now = datetime.now(UTC)
return (CompletedResult.finish(target=options.domain, started_at=now, completed_at=now, groups={}),)
monkeypatch.setattr(main_module, 'start', fake_start)
async def scenario() -> None:
store = RunStore(tmp_path / 'runs.sqlite')
created = await store.create(
RunRequest(
target='api.example.test',
sources=[],
api_scan=True,
api_scan_paths=['/api/v2', '/health'],
)
)
assert await store.claim_next() is not None
await run_worker._child_execute(created['run_id'], store.database)
asyncio.run(scenario())
assert received_wordlists[0].read_text(encoding='utf-8') == '/api/v2\n/health\n'
def test_running_cancellation_terminates_child_and_retains_partial_evidence(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api, run_worker
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'enabled')
async def slow_process(_run_id, _database, artifact_dir):
artifact_dir.mkdir(parents=True, exist_ok=True)
(artifact_dir / 'evidence.json').write_text(
json.dumps(
{
'target': 'example.test',
'status': 'partial',
'results': [{'type': 'email', 'value': 'saved@example.test'}],
}
),
encoding='utf-8',
)
return await asyncio.create_subprocess_exec(
sys.executable,
'-c',
'import time; time.sleep(60)',
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
monkeypatch.setattr(run_worker, '_process_factory', slow_process)
headers = {'X-API-Key': 'test-key'}
with TestClient(api.app) as client:
submitted = client.post(
'/api/v1/runs',
headers=headers,
json={'target': 'example.test', 'sources': ['crtsh'], 'deadline_seconds': 60},
).json()
deadline = time.monotonic() + 3
while time.monotonic() < deadline:
detail = client.get(f'/api/v1/runs/{submitted["run_id"]}', headers=headers).json()
if detail['status'] == 'running':
break
time.sleep(0.02)
requested = client.post(f'/api/v1/runs/{submitted["run_id"]}/cancel', headers=headers)
while time.monotonic() < deadline:
detail = client.get(f'/api/v1/runs/{submitted["run_id"]}', headers=headers).json()
if detail['status'] == 'cancelled':
break
time.sleep(0.02)
assert requested.json()['status'] == 'cancelling'
assert detail['status'] == 'cancelled'
assert detail['completed_at'] is not None
assert detail['evidence_status'] == 'partial'
assert detail['results'] == [{'type': 'email', 'value': 'saved@example.test', 'sources': [], 'actions': []}]
def test_whole_run_deadline_terminates_child_and_retains_partial_evidence(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import run_worker
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts'))
monkeypatch.setattr(run_worker, '_worker_stop', None)
async def slow_process(_run_id, _database, artifact_dir):
artifact_dir.mkdir(parents=True, exist_ok=True)
(artifact_dir / 'evidence.json').write_text(
json.dumps(
{
'target': 'example.test',
'status': 'partial',
'results': [{'type': 'email', 'value': 'saved@example.test'}],
}
),
encoding='utf-8',
)
return await asyncio.create_subprocess_exec(
sys.executable,
'-c',
'import time; time.sleep(60)',
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
monkeypatch.setattr(run_worker, '_process_factory', slow_process)
async def scenario():
store = RunStore()
await store.create(RunRequest(target='example.test', sources=['crtsh']))
run = await store.claim_next()
assert run is not None
run['request']['deadline_seconds'] = 0
await run_worker._execute_claimed(store, run)
return await store.get(run['run_id'])
detail = asyncio.run(scenario())
assert detail is not None
assert detail['status'] == 'failed'
assert 'deadline' in detail['error']
assert detail['evidence_status'] == 'partial'
assert detail['results'] == [{'type': 'email', 'value': 'saved@example.test', 'sources': [], 'actions': []}]
def test_worker_fails_run_without_attaching_child_evidence_for_another_target(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import run_worker
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
monkeypatch.setenv('THEHARVESTER_RUN_ARTIFACTS', str(tmp_path / 'artifacts'))
monkeypatch.setattr(run_worker, '_worker_stop', None)
async def mismatched_process(_run_id, _database, artifact_dir):
artifact_dir.mkdir(parents=True, exist_ok=True)
(artifact_dir / 'evidence.json').write_text(
json.dumps(
{
'target': 'different.example',
'status': 'complete',
'results': [{'type': 'email', 'value': 'saved@different.example'}],
}
),
encoding='utf-8',
)
return await asyncio.create_subprocess_exec(
sys.executable,
'-c',
'pass',
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
monkeypatch.setattr(run_worker, '_process_factory', mismatched_process)
async def scenario():
store = RunStore(tmp_path / 'runs.sqlite')
await store.create(RunRequest(target='expected.example', sources=['crtsh']))
run = await store.claim_next()
assert run is not None
await run_worker._execute_claimed(store, run)
return await store.get(run['run_id'])
detail = asyncio.run(scenario())
assert detail is not None
assert detail['status'] == 'failed'
assert detail['results'] == []
assert 'does not match run target' in detail['error']
tests/lib/test_source_catalog.py
+35 -8
View File
@@ -1,6 +1,7 @@
import ast
from pathlib import Path
from theHarvester.discovery import bevigil, builtwith, gitlabsearch, intelxsearch, rocketreach, urlscan, zoomeyesearch
from theHarvester.lib.core import Core
from theHarvester.lib.source_catalog import SOURCE_SPECS, ResultRoute, SourceSpec, get_source_spec
@@ -42,13 +43,7 @@ def test_invalid_bitbucket_domain_source_is_not_selectable() -> None:
def test_subdomain_route_drives_subdomain_capability() -> None:
spec = SourceSpec(
name='example',
routes=frozenset(
{
ResultRoute.SUBDOMAINS,
ResultRoute.LINKS,
ResultRoute.INTERESTING_URLS,
}
),
routes=frozenset({ResultRoute.SUBDOMAINS, ResultRoute.URLS}),
)
assert spec.capabilities == frozenset({'subdomains', 'urls'})
@@ -64,11 +59,43 @@ def test_source_specs_describe_consolidated_routes_not_getter_presence() -> None
ResultRoute.SUBDOMAINS,
ResultRoute.IPS,
ResultRoute.ASNS,
ResultRoute.INTERESTING_URLS,
ResultRoute.URLS,
}
)
def test_every_url_source_uses_one_route() -> None:
url_sources = {'bevigil', 'builtwith', 'gitlab', 'intelx', 'rocketreach', 'urlscan', 'zoomeye'}
assert {spec.name for spec in SOURCE_SPECS.values() if ResultRoute.URLS in spec.routes} == url_sources
assert {route.name for route in ResultRoute} == {
'SUBDOMAINS',
'EMAILS',
'IPS',
'ASNS',
'PEOPLE',
'URLS',
'BREACHES',
}
def test_every_url_adapter_uses_one_getter() -> None:
adapters = (
bevigil.SearchBeVigil,
builtwith.SearchBuiltWith,
gitlabsearch.SearchGitlab,
intelxsearch.SearchIntelx,
rocketreach.SearchRocketReach,
urlscan.SearchUrlscan,
zoomeyesearch.SearchZoomEye,
)
assert all(hasattr(adapter, 'get_urls') for adapter in adapters)
assert not any(hasattr(adapter, 'get_links') for adapter in adapters)
assert not any(hasattr(adapter, 'get_interestingurls') for adapter in adapters)
assert not any(hasattr(adapter, 'get_interesting_urls') for adapter in adapters)
def test_rapiddns_declares_separate_subdomain_and_ip_routes() -> None:
assert SOURCE_SPECS['rapiddns'].routes == frozenset({ResultRoute.SUBDOMAINS, ResultRoute.IPS})
tests/test_all_source_orchestration.py
+12 -12
View File
@@ -13,7 +13,6 @@ import pytest
from theHarvester import __main__ as theharvester_main
from theHarvester.lib.source_catalog import SOURCE_SPECS, ActivityClass
NON_PASSIVE_SOURCES = (
'criminalip',
'pentesttools',
@@ -234,18 +233,9 @@ async def test_all_schedules_each_passive_catalog_source_once_and_reports_result
async def get_people(self) -> list[dict[str, str]]:
return [{'name': 'Example Person'}]
async def get_links(self) -> set[str]:
return {'https://sub.example.test/profile'}
async def get_urls(self) -> set[str]:
return {'https://gitlab.com/example/project'}
async def get_interestingurls(self) -> set[str]:
return {'https://sub.example.test/evidence'}
async def get_interesting_urls(self) -> set[str]:
return await self.get_interestingurls()
async def get_host_ip_pairs(self) -> set[tuple[str, str]]:
return set()
@@ -296,18 +286,28 @@ async def test_all_schedules_each_passive_catalog_source_once_and_reports_result
json_report = json.loads(report.with_suffix('.json').read_text())
assert 'sub.example.test' in json_report['hosts']
assert 'user@example.test' in json_report['emails']
assert json_report['urls'] == ['https://sub.example.test/evidence']
assert not {'interesting_urls', 'linkedin_links', 'trello_urls'} & json_report.keys()
jsonl_records = [json.loads(line) for line in report.with_suffix('.jsonl').read_text().splitlines()]
findings = {(record['type'], record['value']): record for record in jsonl_records[1:]}
assert findings[('hostname', 'sub.example.test')]['sources']
assert findings[('email', 'user@example.test')]['sources']
assert findings[('url', 'https://gitlab.com/example/project')]['sources']
assert findings[('url', 'https://sub.example.test/evidence')]['sources'] == [
'bevigil',
'builtwith',
'gitlab',
'intelx',
'rocketreach',
'urlscan',
'zoomeye',
]
completed = await TestResultStore().load_run(UUID(jsonl_records[0]['run_id']))
assert completed.target == 'example.test'
assert ('hostname', 'sub.example.test') in completed.results
assert ('email', 'user@example.test') in completed.results
assert ('ip-address', '192.0.2.1') in completed.results
assert ('ip', '192.0.2.1') in completed.results
xml_hosts = {
(element.findtext('hostname') or (element.text or '').strip())
tests/test_logging.py
-55
View File
@@ -54,26 +54,6 @@ def test_operator_output_uses_stdout_without_verbose_logging() -> None:
assert result.stderr == ''
def test_api_example_entry_point_configures_output_and_diagnostics() -> None:
result = run_python(
"""
import logging
from theHarvester.lib.api import api_example
from theHarvester.lib.output import output_logger
async def fake_main():
output_logger.info('example result')
logging.getLogger(api_example.__name__).info('example diagnostic')
api_example.main = fake_main
api_example.entry_point()
"""
)
assert result.stdout == 'example result\n'
assert 'INFO theHarvester.lib.api.api_example: example diagnostic' in result.stderr
def test_diagnostics_use_stderr_only_when_verbose() -> None:
result = run_python(
"""
@@ -132,41 +112,6 @@ def test_verbose_logging_does_not_overwrite_a_later_host_level() -> None:
assert result.stdout == f'{logging.ERROR}\n'
def test_rest_errors_are_visible_with_uvicorn_logging() -> None:
result = run_python(
"""
import logging.config
import sys
from unittest.mock import AsyncMock, patch
from fastapi.testclient import TestClient
from uvicorn.config import LOGGING_CONFIG
logging.config.dictConfig(LOGGING_CONFIG)
from theHarvester.lib.api import api
client = TestClient(api.app)
statuses = []
with patch.object(api.__main__.Core, 'get_supportedengines', side_effect=RuntimeError('sources failure')):
statuses.append(client.get('/sources').status_code)
with patch.object(api.__main__, 'start', AsyncMock(side_effect=RuntimeError('dnsbrute failure'))):
statuses.append(client.get('/dnsbrute?domain=example.com').status_code)
with (
patch.object(api.__main__.Core, 'get_supportedengines', return_value=['baidu']),
patch.object(api.__main__, 'start', AsyncMock(side_effect=RuntimeError('query failure'))),
):
statuses.append(client.get('/query?domain=example.com&source=baidu').status_code)
sys.stdout.write(repr(statuses) + '\\n')
"""
)
assert result.stdout == '[500, 500, 500]\n'
assert 'Error in getsources endpoint' in result.stderr
assert 'Error in dnsbrute endpoint' in result.stderr
assert 'Error in query endpoint' in result.stderr
def test_verbose_enables_info_diagnostics(tmp_path: Path) -> None:
script = """import asyncio
import logging
tests/test_main.py
+1778 -25
View File
File diff suppressed because it is too large Load Diff
tests/test_readme.py
+5 -7
View File
@@ -7,15 +7,13 @@ import yaml
from theHarvester.lib.source_catalog import SOURCE_SPECS, ResultRoute
RESULT_COLUMNS = ('Subdomains', 'Emails', 'IPs', 'ASNs', 'URLs / links', 'People', 'Breaches')
RESULT_COLUMNS = ('Subdomains', 'Emails', 'IPs', 'ASNs', 'URLs', 'People', 'Breaches')
ROUTE_COLUMNS = {
ResultRoute.SUBDOMAINS: 'Subdomains',
ResultRoute.EMAILS: 'Emails',
ResultRoute.IPS: 'IPs',
ResultRoute.ASNS: 'ASNs',
ResultRoute.LINKS: 'URLs / links',
ResultRoute.URLS: 'URLs / links',
ResultRoute.INTERESTING_URLS: 'URLs / links',
ResultRoute.URLS: 'URLs',
ResultRoute.PEOPLE: 'People',
ResultRoute.BREACHES: 'Breaches',
}
@@ -80,7 +78,7 @@ def test_readme_matches_declared_source_contracts() -> None:
documented = _documented_source_contracts(readme)
declared = _declared_source_contracts()
assert '| Source | Subdomains | Emails | IPs | ASNs | URLs / links | People | Breaches |' in readme
assert '| Source | Subdomains | Emails | IPs | ASNs | URLs | People | Breaches |' in readme
assert len(declared) == 56
assert len(documented) == 56
assert documented == declared
@@ -133,6 +131,6 @@ def test_readme_explains_jsonl_record_and_structured_value_parsing() -> None:
assert '{"sources":[],"type":"hostname","value":"api.example.com"}' in readme
assert 'select(.type == "dns-recursive-finding") | .value | fromjson' in readme
assert 'JSONL is easy to stream for simple findings, but it is not uniformly self-describing.' in readme
assert 'JSONL is easy to stream one record at a time.' in readme
assert '`person`, `infostealer`, `shodan`, and `takeover`' in readme
assert 'JSONL does not include source execution records. Finding records include source attribution' in readme
assert 'The summary preserves the evidence status, source and action outcomes' in readme
tests/test_rest_api.py
-358
View File
@@ -1,358 +0,0 @@
from argparse import Namespace
import pytest
from fastapi.testclient import TestClient
from theHarvester.lib.api import api
from theHarvester.lib.core import Core
@pytest.fixture(autouse=True)
def reset_rate_limiter() -> None:
api.limiter.reset()
yield
api.limiter.reset()
def test_query_expands_source_capability(monkeypatch) -> None:
captured: list[tuple[Namespace, bool]] = []
async def fake_start(
args: Namespace,
*,
persist_completed_result: bool = False,
include_breaches: bool = False,
):
captured.append((args, persist_completed_result))
assert include_breaches is True
return ([], [], [], [], [], [], [], [], [], [])
monkeypatch.setattr(api.__main__, 'start', fake_start)
response = TestClient(api.app).get('/query?domain=example.test&source=subdomains')
assert response.status_code == 200
assert captured[0][0].source == ','.join(Core.expand_source_selection('subdomains'))
assert captured[0][1] is True
def test_query_allows_api_scan_of_operator_selected_private_target(monkeypatch) -> None:
captured: list[Namespace] = []
async def fake_start(
args: Namespace,
*,
persist_completed_result: bool = False,
include_breaches: bool = False,
):
captured.append(args)
return ([], [], [], [], [], [], [], [], [], [])
monkeypatch.setattr(api.__main__, 'start', fake_start)
response = TestClient(api.app).get('/query?domain=192.0.2.8&source=certspotter&api_scan=true')
assert response.status_code == 200
assert captured[0].domain == '192.0.2.8'
assert captured[0].api_scan is True
def test_query_forwards_bounded_recursive_dns_options(monkeypatch) -> None:
captured: list[Namespace] = []
async def fake_start(
args: Namespace,
*,
persist_completed_result: bool = False,
include_breaches: bool = False,
):
captured.append(args)
return ([], [], [], [], [], [], [], [], [], [])
monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret')
monkeypatch.setattr(api.__main__, 'start', fake_start)
response = TestClient(api.app).get(
'/query?domain=example.test&source=certspotter&dns_recursive_depth=2'
'&dns_recursive_query_limit=321&dns_recursive_runtime_seconds=4.5'
'&dns_resolve=192.0.2.53,192.0.2.54,192.0.2.55',
headers={'X-API-Key': 'operator-secret'},
)
assert response.status_code == 200
assert captured[0].dns_recursive_depth == 2
assert captured[0].dns_recursive_query_limit == 321
assert captured[0].dns_recursive_runtime_seconds == 4.5
def test_query_requires_operator_key_for_recursive_dns(monkeypatch) -> None:
async def unexpected_start(*_args, **_kwargs):
raise AssertionError('enumeration must not start')
monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False)
monkeypatch.setattr(api.__main__, 'start', unexpected_start)
response = TestClient(api.app).get(
'/query?domain=example.test&source=certspotter&dns_recursive_depth=1&dns_resolve=192.0.2.53,192.0.2.54,192.0.2.55'
)
assert response.status_code == 503
assert response.json()['detail'] == 'THEHARVESTER_API_KEY is not configured'
def test_query_documents_safe_recursive_dns_query_default() -> None:
query_operation = api.app.openapi()['paths']['/query']['get']
query_limit = next(
parameter for parameter in query_operation['parameters'] if parameter['name'] == 'dns_recursive_query_limit'
)
assert query_limit['schema']['default'] == 3_000
def test_query_documents_proxy_and_direct_action_scope() -> None:
parameters = {parameter['name']: parameter for parameter in api.app.openapi()['paths']['/query']['get']['parameters']}
assert parameters['proxies']['description'] == (
'Use proxies.yaml for supported discovery-source and takeover requests.'
)
assert 'using configured proxies when enabled' in parameters['take_over']['description']
assert parameters['api_scan']['description'] == (
'Check common API paths with GET, HEAD, and OPTIONS. Requests follow redirects.'
)
assert parameters['dns_server']['description'] == (
'Accepted for compatibility but currently unused; use dns_resolve to select resolvers.'
)
assert parameters['dns_lookup']['description'] == (
'Perform PTR lookups across the /24 network containing each discovered IPv4 address. '
'This sends active DNS queries.'
)
assert parameters['source']['description'] == (
'Source names or source capabilities to query. Multiple capabilities select the union of matching sources; '
'they do not filter returned fields.'
)
assert parameters['filename']['description'] == (
'Write uniquely prefixed server-side XML, JSON, and JSONL files using NAME as the filename suffix.'
)
@pytest.mark.parametrize('runtime_seconds', ['nan', 'inf'])
def test_query_rejects_non_finite_recursive_dns_runtime(monkeypatch, runtime_seconds: str) -> None:
async def unexpected_start(*_args, **_kwargs):
raise AssertionError('enumeration must not start')
monkeypatch.setattr(api.__main__, 'start', unexpected_start)
response = TestClient(api.app).get(
f'/query?domain=example.test&source=certspotter&dns_recursive_runtime_seconds={runtime_seconds}'
)
assert response.status_code == 422
def test_query_rejects_recursive_dns_without_three_distinct_resolvers(monkeypatch) -> None:
async def unexpected_start(*_args, **_kwargs):
raise AssertionError('enumeration must not start')
monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret')
monkeypatch.setattr(api.__main__, 'start', unexpected_start)
response = TestClient(api.app).get(
'/query?domain=example.test&source=certspotter&dns_recursive_depth=1&dns_resolve=192.0.2.53,192.0.2.54',
headers={'X-API-Key': 'operator-secret'},
)
assert response.status_code == 400
assert response.json()['detail'] == 'recursive DNS requires exactly three distinct resolver IPs'
def test_query_unions_capabilities_and_explicit_sources(monkeypatch) -> None:
captured: list[tuple[Namespace, bool]] = []
async def fake_start(
args: Namespace,
*,
persist_completed_result: bool = False,
include_breaches: bool = False,
):
captured.append((args, persist_completed_result))
assert include_breaches is True
return ([], [], [], [], [], [], [], [], [], [])
monkeypatch.setattr(api.__main__, 'start', fake_start)
response = TestClient(api.app).get('/query?domain=example.test&source=emails&source=certspotter')
assert response.status_code == 200
expected_sources = Core.expand_source_selection('emails,certspotter')
assert captured[0][0].source == ','.join(expected_sources)
assert captured[0][1] is True
def test_query_rejects_unknown_source_or_capability(monkeypatch) -> None:
async def unexpected_start(_args: Namespace):
raise AssertionError('enumeration must not start')
monkeypatch.setattr(api.__main__, 'start', unexpected_start)
response = TestClient(api.app).get('/query?domain=example.test&source=unknown')
assert response.status_code == 400
assert response.json()['detail'].startswith("Source 'unknown' is not supported")
@pytest.mark.parametrize('source', ['hibpverified', 'breaches', 'emails', 'all'])
def test_query_requires_operator_key_when_selection_includes_verified_hibp(monkeypatch, source: str) -> None:
async def unexpected_start(
_args: Namespace,
*,
persist_completed_result: bool = False,
include_breaches: bool = False,
):
raise AssertionError('enumeration must not start')
monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False)
monkeypatch.setattr(api.__main__.Core, 'hibpverified_key', lambda: 'provider-secret')
monkeypatch.setattr(api.__main__, 'start', unexpected_start)
response = TestClient(api.app).get(f'/query?domain=example.test&source={source}')
assert response.status_code == 503
assert response.json()['detail'] == 'THEHARVESTER_API_KEY is not configured'
def test_authenticated_query_returns_verified_hibp_emails_and_breaches(monkeypatch) -> None:
captured: list[Namespace] = []
async def fake_start(
args: Namespace,
*,
persist_completed_result: bool = False,
include_breaches: bool = False,
):
captured.append(args)
assert persist_completed_result is True
assert include_breaches is True
return ([], [], [], [], [], [], [], ['alice@example.test'], [], ['ExampleBreach'])
monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret')
monkeypatch.setattr(api.__main__.Core, 'hibpverified_key', lambda: 'provider-secret')
monkeypatch.setattr(api.__main__, 'start', fake_start)
response = TestClient(api.app).get(
'/query?domain=example.test&source=hibpverified',
headers={'X-API-Key': 'operator-secret'},
)
assert response.status_code == 200
assert captured[0].source == 'hibpverified'
assert response.json()['emails'] == ['alice@example.test']
assert response.json()['breaches'] == ['ExampleBreach']
def test_authenticated_query_includes_verified_hibp_from_capability_selection(monkeypatch) -> None:
captured: list[Namespace] = []
async def fake_start(
args: Namespace,
*,
persist_completed_result: bool = False,
include_breaches: bool = False,
):
captured.append(args)
assert include_breaches is True
return ([], [], [], [], [], [], [], [], [], [])
monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret')
monkeypatch.setattr(api.__main__.Core, 'hibpverified_key', lambda: 'provider-secret')
monkeypatch.setattr(api.__main__, 'start', fake_start)
response = TestClient(api.app).get(
'/query?domain=example.test&source=breaches',
headers={'X-API-Key': 'operator-secret'},
)
assert response.status_code == 200
assert captured[0].source == 'haveibeenpwned,hibpverified,leaklookup'
def test_query_requires_operator_auth_for_configured_leaklookup(monkeypatch) -> None:
async def unexpected_start(*_args, **_kwargs):
raise AssertionError('collection must not start without operator authentication')
monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret')
monkeypatch.setattr(api.__main__.Core, 'leaklookup_key', lambda: 'provider-secret')
monkeypatch.setattr(api.__main__, 'start', unexpected_start)
response = TestClient(api.app).get('/query?domain=example.test&source=leaklookup')
assert response.status_code == 401
@pytest.mark.parametrize('source', ['dehashed', 'emails'])
def test_query_requires_operator_auth_for_configured_dehashed(monkeypatch, source) -> None:
async def unexpected_start(*_args, **_kwargs):
raise AssertionError('collection must not start without operator authentication')
monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret')
monkeypatch.setattr(api.__main__.Core, 'dehashed_key', lambda: 'provider-secret')
monkeypatch.setattr(api.__main__, 'start', unexpected_start)
response = TestClient(api.app).get(f'/query?domain=example.test&source={source}')
assert response.status_code == 401
@pytest.mark.parametrize('dehashed_key', [None, '', ' '])
def test_query_skips_operator_auth_when_dehashed_key_is_blank(monkeypatch, dehashed_key) -> None:
captured: list[Namespace] = []
async def fake_start(
args: Namespace,
*,
persist_completed_result: bool = False,
include_breaches: bool = False,
):
captured.append(args)
return ([], [], [], [], [], [], [], [], [], [])
monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False)
monkeypatch.setattr(api.__main__.Core, 'dehashed_key', lambda: dehashed_key)
monkeypatch.setattr(api.__main__, 'start', fake_start)
response = TestClient(api.app).get('/query?domain=example.test&source=dehashed')
assert response.status_code == 200
assert captured[0].source == 'dehashed'
@pytest.mark.parametrize('leaklookup_key', [None, '', ' '])
def test_query_skips_operator_auth_when_credentialed_provider_keys_are_blank(monkeypatch, leaklookup_key) -> None:
captured: list[Namespace] = []
async def fake_start(
args: Namespace,
*,
persist_completed_result: bool = False,
include_breaches: bool = False,
):
captured.append(args)
return ([], [], [], [], [], [], [], [], [], [])
monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False)
monkeypatch.setattr(api.__main__.Core, 'hibpverified_key', lambda: None)
monkeypatch.setattr(api.__main__.Core, 'leaklookup_key', lambda: leaklookup_key)
monkeypatch.setattr(api.__main__, 'start', fake_start)
response = TestClient(api.app).get('/query?domain=example.test&source=breaches')
assert response.status_code == 200
assert captured[0].source == 'haveibeenpwned,hibpverified,leaklookup'
def test_sources_advertises_authenticated_verified_hibp(monkeypatch) -> None:
monkeypatch.setattr(api.__main__.Core, 'get_supportedengines', lambda: ['crtsh', 'hibpverified'])
response = TestClient(api.app).get('/sources')
assert response.status_code == 200
assert response.json() == {'sources': ['crtsh', 'hibpverified']}
tests/test_rest_completed_runs.py
-94
View File
@@ -1,94 +0,0 @@
from datetime import UTC, datetime, timedelta
from uuid import UUID
import pytest
from fastapi.testclient import TestClient
from theHarvester.lib.api import api
from theHarvester.lib.completed_result import CompletedResult
from theHarvester.lib.database import ResultStore
def completed_result(run_id: str, completed_at: datetime) -> CompletedResult:
return CompletedResult.finish(
run_id=UUID(run_id),
target='example.com',
started_at=completed_at - timedelta(minutes=1),
completed_at=completed_at,
groups={'hostname': ['www.example.com']},
)
@pytest.mark.asyncio
async def test_authenticated_operator_lists_recent_completed_runs(tmp_path, monkeypatch) -> None:
monkeypatch.setattr('theHarvester.lib.database._DEFAULT_DATABASE', tmp_path / 'stash.sqlite')
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-secret')
store = ResultStore()
await store.initialize()
older = completed_result('11111111-1111-4111-8111-111111111111', datetime(2026, 8, 6, 12, 0, tzinfo=UTC))
newer = completed_result('22222222-2222-4222-8222-222222222222', datetime(2026, 8, 6, 13, 0, tzinfo=UTC))
offset_older = completed_result(
'55555555-5555-4555-8555-555555555555', datetime.fromisoformat('2026-08-06T14:30:00+05:00')
)
await store.save_run(older)
await store.save_run(newer)
await store.save_run(offset_older)
response = TestClient(api.app).get('/runs?limit=1', headers={'X-API-Key': 'test-secret'})
assert response.status_code == 200
assert response.json() == [
{
'run_id': str(newer.run_id),
'target': 'example.com',
'started_at': '2026-08-06T12:59:00Z',
'completed_at': '2026-08-06T13:00:00Z',
'result_count': 1,
}
]
@pytest.mark.asyncio
async def test_authenticated_operator_gets_one_completed_run(tmp_path, monkeypatch) -> None:
monkeypatch.setattr('theHarvester.lib.database._DEFAULT_DATABASE', tmp_path / 'stash.sqlite')
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-secret')
store = ResultStore()
await store.initialize()
result = CompletedResult.finish(
run_id=UUID('33333333-3333-4333-8333-333333333333'),
target='example.com',
started_at=datetime(2026, 8, 6, 14, 0, tzinfo=UTC),
completed_at=datetime(2026, 8, 6, 14, 1, tzinfo=UTC),
groups={'email': ['security@example.com'], 'hostname': ['www.example.com']},
)
await store.save_run(result)
client = TestClient(api.app)
response = client.get(f'/runs/{result.run_id}', headers={'X-API-Key': 'test-secret'})
missing = client.get('/runs/44444444-4444-4444-8444-444444444444', headers={'X-API-Key': 'test-secret'})
assert response.status_code == 200
assert response.json() == {
'run_id': str(result.run_id),
'target': 'example.com',
'started_at': '2026-08-06T14:00:00Z',
'completed_at': '2026-08-06T14:01:00Z',
'result_count': 2,
'results': [
{'type': 'email', 'value': 'security@example.com'},
{'type': 'hostname', 'value': 'www.example.com'},
],
}
assert missing.status_code == 404
assert missing.json() == {'detail': 'Completed run not found'}
def test_completed_run_routes_fail_closed_without_operator_key(monkeypatch) -> None:
monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False)
client = TestClient(api.app)
list_response = client.get('/runs')
detail_response = client.get('/runs/33333333-3333-4333-8333-333333333333')
assert list_response.status_code == 503
assert detail_response.status_code == 503
tests/test_rest_terminal_evidence.py
-26
View File
@@ -1,26 +0,0 @@
from argparse import Namespace
from fastapi.testclient import TestClient
from theHarvester.lib.api import api
def test_query_requests_completed_result_persistence(monkeypatch) -> None:
persistence_flags: list[bool] = []
async def fake_start(
args: Namespace,
*,
persist_completed_result: bool = False,
include_breaches: bool = False,
):
persistence_flags.append(persist_completed_result)
assert include_breaches is True
return ([], [], [], [], [], [], [], [], [], [])
monkeypatch.setattr(api.__main__, 'start', fake_start)
response = TestClient(api.app).get('/query?domain=example.test&source=crtsh')
assert response.status_code == 200
assert persistence_flags == [True]
tests/test_restful_harvest.py
+15
View File
@@ -0,0 +1,15 @@
import sys
import pytest
from theHarvester import restfulHarvest
def test_help_does_not_offer_rate_limit_configuration(monkeypatch, capsys):
monkeypatch.setattr(sys, 'argv', ['restfulHarvest', '--help'])
with pytest.raises(SystemExit) as exit_info:
restfulHarvest.main()
assert exit_info.value.code == 0
assert '--rate-limit' not in capsys.readouterr().out
tests/test_screenshot.py
+74 -11
View File
@@ -24,10 +24,35 @@ def test_screenshot_output_separator_matches_platform(
assert ScreenShotter('screenshots').slash == expected_separator
@pytest.mark.parametrize(
('target', 'filename'),
[
('https://www.example.com:8443/login?next=/admin', 'www.example.com_8443.png'),
('2001:db8::1', '2001_db8__1.png'),
],
)
def test_screenshot_path_uses_the_target_host_and_port(tmp_path: Path, target: str, filename: str) -> None:
screenshotter = ScreenShotter(str(tmp_path))
assert screenshotter.screenshot_path(target) == tmp_path / filename
@pytest.mark.parametrize(
('target', 'response_url', 'request_url'),
[
('www.example.com', 'https://www.example.com/landing', 'https://www.example.com'),
('2001:db8::1', 'https://[2001:db8::1]/', 'https://[2001:db8::1]'),
],
)
@pytest.mark.asyncio
async def test_visit_prefers_https_and_returns_final_www_url(monkeypatch: pytest.MonkeyPatch) -> None:
async def test_visit_prefers_https_and_normalizes_the_target(
monkeypatch: pytest.MonkeyPatch,
target: str,
response_url: str,
request_url: str,
) -> None:
response = MagicMock()
response.url = 'https://www.example.com/landing'
response.url = response_url
response.__aenter__ = AsyncMock(return_value=response)
response.__aexit__ = AsyncMock(return_value=False)
response.text = AsyncMock(return_value='reachable')
@@ -38,10 +63,10 @@ async def test_visit_prefers_https_and_returns_final_www_url(monkeypatch: pytest
monkeypatch.setattr(screenshot_module.aiohttp, 'ClientSession', MagicMock(return_value=session))
monkeypatch.setattr(screenshot_module.aiohttp, 'TCPConnector', MagicMock())
monkeypatch.setattr(screenshot_module.ssl, 'create_default_context', MagicMock())
result = await ScreenShotter.visit('www.example.com')
result = await ScreenShotter.visit(target)
assert result == ('https://www.example.com/landing', 'reachable')
assert session.get.call_args.args[0] == 'https://www.example.com'
assert result == (response_url, 'reachable')
assert session.get.call_args.args[0] == request_url
@pytest.mark.asyncio
@@ -67,8 +92,44 @@ async def test_visit_falls_back_to_http_when_https_is_unreachable(monkeypatch: p
]
@pytest.mark.parametrize(
('target', 'normalized_url', 'filename'),
[
('www.example.com', 'https://www.example.com', 'www.example.com.png'),
('2001:db8::1', 'https://[2001:db8::1]', '2001_db8__1.png'),
],
)
@pytest.mark.asyncio
async def test_take_screenshot_preserves_www_hostname(
async def test_take_screenshot_normalizes_the_target(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
target: str,
normalized_url: str,
filename: str,
) -> None:
page = AsyncMock()
context = AsyncMock()
context.new_page.return_value = page
browser = AsyncMock()
browser.new_context.return_value = context
playwright = MagicMock()
playwright.chromium.launch = AsyncMock(return_value=browser)
manager = MagicMock()
manager.__aenter__ = AsyncMock(return_value=playwright)
manager.__aexit__ = AsyncMock(return_value=False)
monkeypatch.setattr(screenshot_module, 'async_playwright', MagicMock(return_value=manager))
monkeypatch.setattr(screenshot_module.os, 'chmod', MagicMock())
captured_url = await ScreenShotter(str(tmp_path)).take_screenshot(target)
assert captured_url == normalized_url
page.goto.assert_awaited_once_with(normalized_url, timeout=35000)
screenshot_path = page.screenshot.await_args.kwargs['path']
assert Path(screenshot_path) == tmp_path / filename
@pytest.mark.asyncio
async def test_take_screenshot_can_name_the_artifact_for_the_authorized_subject(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
@@ -84,13 +145,15 @@ async def test_take_screenshot_preserves_www_hostname(
manager.__aexit__ = AsyncMock(return_value=False)
monkeypatch.setattr(screenshot_module, 'async_playwright', MagicMock(return_value=manager))
monkeypatch.setattr(screenshot_module.os, 'chmod', MagicMock())
output_path = tmp_path / 'authorized.example.com.png'
captured_url = await ScreenShotter(str(tmp_path)).take_screenshot('www.example.com')
captured_url = await ScreenShotter(str(tmp_path)).take_screenshot(
'https://login.example.net/session',
output_path=output_path,
)
assert captured_url == 'https://www.example.com'
page.goto.assert_awaited_once_with('https://www.example.com', timeout=35000)
screenshot_path = page.screenshot.await_args.kwargs['path']
assert screenshot_path.endswith('www.example.com.png')
assert captured_url == 'https://login.example.net/session'
assert page.screenshot.await_args.kwargs['path'] == output_path
@pytest.mark.asyncio
tests/test_security.py
+44 -116
View File
@@ -1,7 +1,6 @@
import os
import re
import tempfile
from pathlib import Path
from unittest.mock import AsyncMock
import pytest
@@ -13,64 +12,10 @@ from theHarvester.__main__ import sanitize_filename, sanitize_for_xml
class TestCORSConfiguration:
"""Test CORS security configuration."""
def test_cors_does_not_allow_credentials_with_wildcard_origins(self):
"""
Security Test: CORS should not allow credentials with wildcard origins.
This prevents credential theft attacks where any origin can make
authenticated requests to the API.
"""
def test_api_does_not_enable_cross_origin_requests(self):
from theHarvester.lib.api.api import app
# Find CORS middleware in the app
cors_middleware = None
for middleware in app.user_middleware:
if 'CORSMiddleware' in str(middleware.cls):
cors_middleware = middleware
break
assert cors_middleware is not None, 'CORS middleware should be configured'
# Check that if allow_origins contains '*', allow_credentials must be False
# Access kwargs from the middleware
options = cors_middleware.kwargs
allow_origins = options.get('allow_origins', [])
allow_credentials = options.get('allow_credentials', False)
if isinstance(allow_origins, (list, tuple, set)) and '*' in allow_origins:
assert (
allow_credentials is False
), 'CRITICAL: CORS must not allow credentials with wildcard origins (CVE risk)'
def test_cors_restricts_http_methods(self):
"""
Security Test: CORS should restrict HTTP methods to only what's needed.
Reduces attack surface by limiting available methods.
"""
from theHarvester.lib.api.api import app
cors_middleware = None
for middleware in app.user_middleware:
if 'CORSMiddleware' in str(middleware.cls):
cors_middleware = middleware
break
assert cors_middleware is not None
options = cors_middleware.kwargs
allow_methods = options.get('allow_methods', [])
# Should not allow all methods
assert allow_methods != ['*'], 'CORS should restrict HTTP methods, not allow all (*)'
# Should only allow necessary methods (GET, POST for this API)
if isinstance(allow_methods, list):
dangerous_methods = {'DELETE', 'PUT', 'PATCH', 'TRACE', 'CONNECT'}
allowed_set = {m.upper() for m in allow_methods}
assert not (
allowed_set & dangerous_methods
), f'Unnecessary HTTP methods detected: {allowed_set & dangerous_methods}'
assert all('CORSMiddleware' not in str(middleware.cls) for middleware in app.user_middleware)
class TestXMLInjectionPrevention:
@@ -153,8 +98,7 @@ class TestInformationDisclosure:
Stack traces can reveal sensitive information about the system.
"""
# Test the /sources endpoint with a simulated error condition
response = client.get('/sources')
response = client.get('/api/v1/sources')
# Even if there's an error, traceback should not be in response
if response.status_code >= 400:
@@ -163,20 +107,19 @@ class TestInformationDisclosure:
assert 'Traceback' not in str(response_data), 'Traceback text found in response'
assert 'File "' not in str(response_data), 'File paths exposed in response'
def test_error_responses_do_not_leak_internal_paths(self, client, monkeypatch):
def test_error_responses_do_not_leak_internal_paths(self, client, tmp_path, monkeypatch):
"""
Security Test: Error messages should not reveal internal file paths.
"""
start = AsyncMock(return_value=([], [], [], [], [], [], [], [], []))
fetch_all = AsyncMock(side_effect=AssertionError('API security test attempted a provider request'))
monkeypatch.setattr('theHarvester.lib.api.api.__main__.start', start)
monkeypatch.setenv('THEHARVESTER_API_KEY', 'operator-secret')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setattr('theHarvester.lib.core.AsyncFetcher.fetch_all', fetch_all)
# Try various endpoints
endpoints = ['/sources', '/dnsbrute?domain=test', '/query?domain=test&source=baidu']
endpoints = ['/api/v1/sources', '/api/v1/runs/not-found']
for endpoint in endpoints:
response = client.get(endpoint)
response = client.get(endpoint, headers={'X-API-Key': 'operator-secret'})
response_text = str(response.json() if response.status_code != 200 else {})
# Check for common path leakage patterns
@@ -193,7 +136,6 @@ class TestInformationDisclosure:
matches = re.findall(pattern, response_text)
assert not matches, f'Internal path leaked in {endpoint}: {matches}'
assert start.await_count == 2
fetch_all.assert_not_awaited()
def test_debug_mode_does_not_expose_sensitive_info(self, client, monkeypatch):
@@ -204,7 +146,7 @@ class TestInformationDisclosure:
monkeypatch.setenv('DEBUG', '1')
# Make request that might trigger an error
response = client.get('/dnsbrute?domain=') # Invalid request
response = client.get('/api/v1/runs/not-found')
if response.status_code >= 400:
response_data = response.json()
@@ -212,8 +154,8 @@ class TestInformationDisclosure:
assert 'traceback' not in response_data, 'DEBUG mode exposes tracebacks to clients'
class TestAdditionalAPIAuthentication:
"""Test authentication and error handling for protected additional API routes."""
class TestAPIAuthentication:
"""Test authentication and error handling for the versioned API."""
@pytest.fixture
def client(self):
@@ -222,40 +164,48 @@ class TestAdditionalAPIAuthentication:
return TestClient(app)
def test_additional_endpoints_fail_closed_without_configured_api_key(self, client, monkeypatch):
def test_api_fails_closed_without_configured_api_key(self, client, monkeypatch):
monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False)
monkeypatch.delenv('THEHARVESTER_API_KEY_FILE', raising=False)
response = client.post('/additional/all', json={'domain': 'example.com'})
response = client.get('/api/v1/sources')
assert response.status_code == 503
def test_additional_endpoints_reject_missing_or_invalid_api_key(self, client, monkeypatch):
def test_api_key_can_be_read_from_a_docker_secret_file(self, client, tmp_path, monkeypatch):
secret = tmp_path / 'operator-api-key'
secret.write_text('test-secret\n', encoding='utf-8')
monkeypatch.delenv('THEHARVESTER_API_KEY', raising=False)
monkeypatch.setenv('THEHARVESTER_API_KEY_FILE', str(secret))
response = client.get('/api/v1/sources', headers={'X-API-Key': 'test-secret'})
assert response.status_code == 200
def test_api_rejects_missing_or_invalid_api_key(self, client, monkeypatch):
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-secret')
missing_response = client.post('/additional/all', json={'domain': 'example.com'})
invalid_response = client.post('/additional/all', headers={'X-API-Key': 'wrong'}, json={'domain': 'example.com'})
missing_response = client.get('/api/v1/sources')
invalid_response = client.get('/api/v1/sources', headers={'X-API-Key': 'wrong'})
assert missing_response.status_code == 401
assert invalid_response.status_code == 401
def test_additional_endpoints_do_not_expose_internal_errors(self, client, monkeypatch):
from theHarvester.lib.api import additional_endpoints
def test_api_does_not_expose_internal_errors(self, monkeypatch):
from theHarvester.lib.api import api
from theHarvester.lib.api.run_store import RunStore
class FailingAdditionalAPIs:
def __init__(self, domain, api_keys):
self.domain = domain
self.api_keys = api_keys
async def process(self):
raise RuntimeError('/home/user/project/secret.py:123 internal failure')
async def fail(_self):
raise RuntimeError('/home/user/project/secret.py:123 internal failure')
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-secret')
monkeypatch.setattr(additional_endpoints, 'AdditionalAPIs', FailingAdditionalAPIs)
monkeypatch.setattr(RunStore, 'list_runs', fail)
client = TestClient(api.app, raise_server_exceptions=False)
response = client.post('/additional/all', headers={'X-API-Key': 'test-secret'}, json={'domain': 'example.com'})
response = client.get('/api/v1/runs', headers={'X-API-Key': 'test-secret'})
assert response.status_code == 500
response_text = str(response.json())
response_text = response.text
assert 'internal failure' not in response_text
assert '/home/user/project' not in response_text
@@ -403,24 +353,11 @@ class TestSecurityBestPractices:
real_matches = [
m
for m in matches
if 'example' not in m.lower()
and 'your_' not in m.lower()
and '""' not in m
and "''" not in m
if 'example' not in m.lower() and 'your_' not in m.lower() and '""' not in m and "''" not in m
]
assert not real_matches, f'Potential hardcoded secret in {file_path}: {real_matches}'
def test_api_has_rate_limiting(self):
"""
Security Test: Verify API endpoints have rate limiting enabled.
"""
from theHarvester.lib.api.api import app
# Check that rate limiting is configured
assert hasattr(app.state, 'limiter'), 'Rate limiter not configured'
assert app.state.limiter is not None, 'Rate limiter is None'
def test_sensitive_endpoints_require_validation(self):
def test_sensitive_endpoints_require_validation(self, monkeypatch):
"""
Security Test: Ensure sensitive endpoints validate input.
"""
@@ -428,24 +365,15 @@ class TestSecurityBestPractices:
from theHarvester.lib.api.api import app
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-secret')
client = TestClient(app)
headers = {'X-API-Key': 'test-secret'}
# Test that endpoints reject invalid input
# Note: The /query endpoint requires 'source' as a list parameter
test_cases = [
('/dnsbrute?domain=', 400), # Empty domain should be rejected
('/dnsbrute?domain=a', 422), # Too short domain should be rejected by FastAPI validation
]
missing_target = client.post('/api/v1/runs', headers=headers, json={'sources': ['crtsh']})
empty_sources = client.post('/api/v1/runs', headers=headers, json={'target': 'example.test', 'sources': []})
for endpoint, expected_status in test_cases:
response = client.get(endpoint)
assert (
response.status_code >= 400
), f'Endpoint {endpoint} should reject invalid input (got {response.status_code})'
# Test query endpoint with proper parameter format but invalid domain
response = client.get('/query?domain=a&source=baidu') # Too short domain
assert response.status_code == 422
assert missing_target.status_code == 422
assert empty_sources.status_code == 422
if __name__ == '__main__':
theHarvester/__main__.py
+850 -307
View File
File diff suppressed because it is too large Load Diff
theHarvester/discovery/additional_apis.py
-183
View File
@@ -1,183 +0,0 @@
import asyncio
import logging
from typing import Any
from theHarvester.discovery.builtwith import SearchBuiltWith
from theHarvester.discovery.haveibeenpwned import SearchHaveIBeenPwned
from theHarvester.discovery.leaklookup import SearchLeakLookup
from theHarvester.discovery.securityscorecard import SearchSecurityScorecard
from theHarvester.discovery.shodansearch import SearchShodan
from theHarvester.lib.output import output_logger
logger = logging.getLogger(__name__)
class AdditionalAPIs:
"""Wrapper class for additional API services."""
def __init__(self, domain: str, api_keys: dict[str, str] | None = None):
self.domain = domain
self.api_keys = api_keys or {}
# Initialize API services
self.haveibeenpwned = SearchHaveIBeenPwned(domain)
self.leaklookup: SearchLeakLookup | None = None
self.securityscorecard = SearchSecurityScorecard(domain)
self.builtwith = SearchBuiltWith(domain)
self.shodan: SearchShodan | None = None # Will be initialized when needed
# Aggregated sets for results
self.hosts: set[str] = set()
self.emails: set[str] = set()
# Results storage
self.results: dict[str, Any] = {
'breaches': [],
'leaks': [],
'security_score': {},
'tech_stack': {},
'shodan_data': {},
'hosts': [],
'emails': [],
}
self.shodan_data: dict[str, Any] = {}
async def process(self, proxy: bool = False) -> dict[str, Any]:
"""Process all additional API services and return combined results."""
tasks = [
self._process_haveibeenpwned(proxy),
self._process_leaklookup(proxy),
self._process_securityscorecard(proxy),
self._process_builtwith(proxy),
self._process_shodan(proxy),
]
await asyncio.gather(*tasks, return_exceptions=True)
# Convert aggregated sets to lists for JSON serialization
self.results['hosts'] = list(self.hosts)
self.results['emails'] = list(self.emails)
self.results['shodan_data'] = self.shodan_data
return self.results
async def _process_haveibeenpwned(self, proxy: bool = False):
"""Process HaveIBeenPwned API."""
try:
await self.haveibeenpwned.process(proxy)
self.results['breaches'] = self.haveibeenpwned.breaches
self.hosts.update(self.haveibeenpwned.hosts)
self.emails.update(self.haveibeenpwned.emails)
except Exception as e:
logger.info(f'Error processing HaveIBeenPwned: {e}')
async def _process_leaklookup(self, proxy: bool = False):
"""Process Leak-Lookup API."""
try:
if self.leaklookup is None:
self.leaklookup = SearchLeakLookup(self.domain)
await self.leaklookup.process(proxy)
self.results['leaks'] = self.leaklookup.leaks
self.hosts.update(self.leaklookup.hosts)
self.emails.update(self.leaklookup.emails)
except Exception as e:
logger.info(f'Error processing Leak-Lookup: {e}')
async def _process_securityscorecard(self, proxy: bool = False):
"""Process SecurityScorecard API."""
try:
await self.securityscorecard.process(proxy)
self.results['security_score'] = {
'score': self.securityscorecard.score,
'grades': self.securityscorecard.grades,
'issues': self.securityscorecard.issues,
'recommendations': self.securityscorecard.recommendations,
}
self.hosts.update(self.securityscorecard.hosts)
except Exception as e:
logger.info(f'Error processing SecurityScorecard: {e}')
async def _process_builtwith(self, proxy: bool = False):
"""Process BuiltWith API."""
try:
await self.builtwith.process(proxy)
self.results['tech_stack'] = {
'frameworks': list(self.builtwith.frameworks),
'languages': list(self.builtwith.languages),
'servers': list(self.builtwith.servers),
'cms': list(self.builtwith.cms),
'analytics': list(self.builtwith.analytics),
'interesting_urls': list(self.builtwith.interesting_urls),
}
self.hosts.update(self.builtwith.hosts)
except Exception as e:
logger.info(f'Error processing BuiltWith: {e}')
async def _process_shodan(self, proxy: bool = False):
"""Process Shodan API for IP information."""
try:
# Initialize Shodan only when needed
if self.shodan is None:
self.shodan = SearchShodan()
# Get IPs from hosts for Shodan lookup
import socket
ips_to_search: set[str] = set()
# Try to resolve domain to IP
try:
ip = socket.gethostbyname(self.domain)
ips_to_search.add(ip)
except socket.gaierror as e:
logger.info(f"Failed to resolve domain '{self.domain}': {e}")
except Exception as e:
logger.info(f"Unexpected error while resolving domain '{self.domain}': {e}")
# Add any IPs from other results
for host in self.hosts:
if ':' in host:
# Extract IP from host:ip format
parts = host.split(':')
if len(parts) == 2 and self._is_valid_ip(parts[1]):
ips_to_search.add(parts[1])
elif self._is_valid_ip(host):
ips_to_search.add(host)
# Search each IP in Shodan
for ip in ips_to_search:
try:
output_logger.info(f'\tSearching Shodan for {ip}')
shodan_result = await self.shodan.search_ip(ip)
if ip in shodan_result and isinstance(shodan_result[ip], dict):
self.shodan_data[ip] = shodan_result[ip]
elif ip in shodan_result and isinstance(shodan_result[ip], str):
output_logger.info(f'{ip}: {shodan_result[ip]}')
await asyncio.sleep(2) # Rate limiting
except Exception as ip_error:
logger.info(f'Error searching Shodan for {ip}: {ip_error}')
continue
except Exception as e:
logger.info(f'Error processing Shodan: {e}')
@staticmethod
def _is_valid_ip(ip_str: str) -> bool:
"""Check if a string is a valid IP address."""
import ipaddress
try:
ipaddress.ip_address(ip_str)
return True
except ValueError:
return False
async def get_hosts(self) -> set[str]:
"""Get all discovered hosts."""
return self.hosts
async def get_emails(self) -> set[str]:
"""Get all discovered emails."""
return self.emails
theHarvester/discovery/api_endpoints.py
+38 -13
View File
@@ -55,6 +55,7 @@ class SearchApiEndpoints:
follow_redirects: bool = True,
verify_ssl: bool = True,
additional_headers: dict[str, str] | None = None,
exact_paths: bool = False,
) -> None:
"""Configure an API path scan.
@@ -68,6 +69,7 @@ class SearchApiEndpoints:
follow_redirects: Whether requests follow redirects.
verify_ssl: Whether to verify TLS certificates.
additional_headers: Extra HTTP headers to send.
exact_paths: Check only paths listed in the configured wordlist.
"""
self.word = word
@@ -92,12 +94,16 @@ class SearchApiEndpoints:
self.user_agent = user_agent or Core.get_user_agent()
self.additional_headers = additional_headers or {}
self._session: aiohttp.ClientSession | None = None
self.scan_error_type: str | None = None
self.request_error_count = 0
self.request_error_types: set[str] = set()
# Set default wordlist path
default_wordlist = os.path.join(
os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'wordlists', 'api_endpoints.txt'
)
self.wordlist = wordlist or default_wordlist
self.exact_paths = exact_paths
# Add comprehensive API paths categorized by functionality
self.common_api_paths = [
@@ -384,6 +390,9 @@ class SearchApiEndpoints:
async def do_search(self) -> None:
"""Check common paths with GET, HEAD, and OPTIONS."""
self.scan_error_type = None
self.request_error_count = 0
self.request_error_types.clear()
session: aiohttp.ClientSession | None = None
try:
session = aiohttp.ClientSession(
@@ -399,12 +408,14 @@ class SearchApiEndpoints:
self.logger.warning(f'No endpoints found in wordlist: {self.wordlist}')
endpoints = []
# Add common API paths that might not be in the wordlist
endpoints.extend(self.common_api_paths)
endpoints = list(set(endpoints)) # Remove duplicates
if not self.exact_paths:
endpoints.extend(self.common_api_paths)
endpoints = list(dict.fromkeys(endpoints))
if not endpoints:
return
# Detect base URL schema (http or https)
schema = await self._detect_schema()
schema = await self._detect_schema(endpoints[0]) if self.exact_paths else await self._detect_schema()
self.logger.info(f'Detected schema for {self.word}: {schema}')
# Generate batches of tasks to control concurrency
@@ -426,15 +437,16 @@ class SearchApiEndpoints:
await self._post_scan_analysis()
except Exception as e:
self.scan_error_type = type(e).__name__
self.logger.error(f'Error in API endpoint scan: {e!s}', exc_info=True)
finally:
self._session = None
if session is not None:
await session.close()
async def _detect_schema(self) -> str:
async def _detect_schema(self, path: str = '') -> str:
"""Detect if the domain supports HTTPS or fall back to HTTP."""
https_url = f'https://{self.word}'
https_url = f'https://{self.word}{path}'
if self._session is None:
raise RuntimeError('API endpoint session is not initialized')
try:
@@ -463,6 +475,9 @@ class SearchApiEndpoints:
# Ensure all paths start with /
endpoints = [line if line.startswith('/') else f'/{line}' for line in lines]
if self.exact_paths:
return list(dict.fromkeys(endpoints))
# Add some path variations (with and without trailing slash)
variations = []
for endpoint in endpoints:
@@ -507,21 +522,28 @@ class SearchApiEndpoints:
verify=self.verify_ssl,
follow_redirects=self.follow_redirects,
request_timeout=self.timeout,
include_metadata=True,
)
# Calculate response time
response_time = asyncio.get_event_loop().time() - start_time
# If we get a response, process it
if response:
result = self._process_response(url, method, response, response_time)
if result:
return result
if response is None:
self.request_error_count += 1
self.request_error_types.add('TransportError')
continue
result = self._process_response(url, method, response, response_time)
if result:
return result
except TimeoutError:
self.request_error_count += 1
self.request_error_types.add('TimeoutError')
self.logger.debug(f'Timeout for {method} {url}')
continue
except (aiohttp.ClientError, OSError, TypeError, ValueError, AttributeError) as e:
self.request_error_count += 1
self.request_error_types.add(type(e).__name__)
self.logger.debug(f'Error checking {method} {url}: {e!s}')
continue
@@ -567,7 +589,10 @@ class SearchApiEndpoints:
headers = {}
try:
content = getattr(response, 'content', b'')
content_value = getattr(response, 'body', getattr(response, 'content', b''))
content = content_value.encode() if isinstance(content_value, str) else content_value
if not isinstance(content, bytes):
content = b''
except (TypeError, AttributeError) as e:
self.logger.error(f'Failed to get content from response for URL {url}: {e}')
content = b''
@@ -576,7 +601,7 @@ class SearchApiEndpoints:
self.response_sizes[url] = content_length
# Try to get content type from headers
content_type = headers.get('Content-Type', '')
content_type = next((value for name, value in headers.items() if name.casefold() == 'content-type'), '')
# Try to create a preview of the response content (up to 200 characters)
content_preview = ''
theHarvester/discovery/bevigil.py
+4 -4
View File
@@ -6,7 +6,7 @@ class SearchBeVigil:
def __init__(self, word) -> None:
self.word = word
self.totalhosts: set = set()
self.interestingurls: set = set()
self.urls: set = set()
self.key = Core.bevigil_key()
if self.key is None:
self.key = ''
@@ -26,13 +26,13 @@ class SearchBeVigil:
responses = await AsyncFetcher.fetch_all([url_endpoint], json=True, proxy=self.proxy, headers=headers)
response = responses[0]
for url in response['urls']:
self.interestingurls.add(url)
self.urls.add(url)
async def get_hostnames(self) -> set:
return self.totalhosts
async def get_interestingurls(self) -> set:
return self.interestingurls
async def get_urls(self) -> set:
return self.urls
async def process(self, proxy: bool = False) -> None:
self.proxy = proxy
theHarvester/discovery/builtwith.py
+4 -4
View File
@@ -19,7 +19,7 @@ class SearchBuiltWith:
self.headers = {'Authorization': f'Bearer {self.api_key}', 'Content-Type': 'application/json'}
self.hosts: set[str] = set()
self.tech_stack: dict[str, Any] = {}
self.interesting_urls: set[str] = set()
self.urls: set[str] = set()
self.frameworks: set[str] = set()
self.languages: set[str] = set()
self.servers: set[str] = set()
@@ -51,7 +51,7 @@ class SearchBuiltWith:
if 'domains' in self.tech_stack:
self.hosts.update(self.tech_stack['domains'])
if 'paths' in self.tech_stack:
self.interesting_urls.update(self.tech_stack['paths'])
self.urls.update(self.tech_stack['paths'])
if 'technologies' in self.tech_stack:
for tech in self.tech_stack['technologies']:
if not isinstance(tech, dict):
@@ -80,8 +80,8 @@ class SearchBuiltWith:
async def get_tech_stack(self) -> dict:
return self.tech_stack
async def get_interesting_urls(self) -> set[str]:
return self.interesting_urls
async def get_urls(self) -> set[str]:
return self.urls
async def get_frameworks(self) -> set[str]:
return self.frameworks
theHarvester/discovery/criminalip.py
+1 -1
View File
@@ -181,7 +181,7 @@ class SearchCriminalIP:
async def parser(self, jlines):
# TODO when new scope field is added to parse lines for potential new scope!
# TODO map as_name to asn for asn data
# TODO determine if worth storing interesting urls
# TODO determine if returned URLs are useful
if not isinstance(jlines, dict) or 'data' not in jlines.keys() or not isinstance(jlines['data'], dict):
logger.info('CriminalIP report has an unexpected structure')
return
theHarvester/discovery/dnssearch.py
+19 -8
View File
@@ -32,12 +32,13 @@ class DnsForce:
self.subdo = False
self.verbose = verbose
self.records: dict[str, hostchecker.HostDnsRecords] = {}
self.query_error_count = 0
self.query_error_types: set[str] = set()
# self.dnsserver = [dnsserver] if isinstance(dnsserver, str) else dnsserver
# self.dnsserver = list(map(str, dnsserver.split(','))) if isinstance(dnsserver, str) else dnsserver
self.dnsserver = dnsserver
with DNS_NAMES.open('r') as file:
self.list = file.readlines()
self.domain = domain.replace('www.', '')
self.list = [f'{word.strip()}.{self.domain}' for word in self.list]
async def run(self):
@@ -45,6 +46,8 @@ class DnsForce:
checker = hostchecker.Checker(self.list, nameservers=self.dnsserver)
resolved_pair, hosts, ips = await checker.check()
self.records = checker.records
self.query_error_count = getattr(checker, 'query_error_count', 0)
self.query_error_types = set(getattr(checker, 'query_error_types', set()))
return resolved_pair, hosts, ips
@@ -112,7 +115,7 @@ def list_ips_in_network_range(iprange: str) -> list[str]:
return []
async def reverse_single_ip(ip: str, resolver: DNSResolver) -> str:
async def reverse_single_ip(ip: str, resolver: DNSResolver, error_types: set[str] | None = None) -> str:
"""Reverse a single IP and output the linked CNAME, if it exists.
Parameters
@@ -126,13 +129,20 @@ async def reverse_single_ip(ip: str, resolver: DNSResolver) -> str:
"""
try:
__host = await resolver.gethostbyaddr(ip)
return __host.name if __host else ''
except Exception:
host = await resolver.gethostbyaddr(ip)
return host.name if host else ''
except Exception as error:
if error_types is not None and not hostchecker.is_expected_dns_absence(error):
error_types.add(type(error).__name__)
return ''
async def reverse_all_ips_in_range(iprange: str, callback: Callable, nameservers: list[str] | None = None) -> None:
async def reverse_all_ips_in_range(
iprange: str,
callback: Callable,
nameservers: list[str] | None = None,
error_types: set[str] | None = None,
) -> None:
"""Reverse all the IPs stored in a network range.
All the queries are made concurrently.
@@ -146,7 +156,8 @@ async def reverse_all_ips_in_range(iprange: str, callback: Callable, nameservers
Arbitrary postprocessing function.
nameservers: List[str].
Optional list of DNS servers.
error_types: set[str].
Optional sink for unexpected resolver or transport error names.
Returns
-------
out: None.
@@ -156,7 +167,7 @@ async def reverse_all_ips_in_range(iprange: str, callback: Callable, nameservers
__resolver = DNSResolver(loop=loop, timeout=8, nameservers=nameservers)
for __ip in list_ips_in_network_range(iprange):
log_query(__ip)
__host = await reverse_single_ip(ip=__ip, resolver=__resolver)
__host = await reverse_single_ip(ip=__ip, resolver=__resolver, error_types=error_types)
callback(__host)
log_result(__host)
theHarvester/discovery/intelxsearch.py
+10 -7
View File
@@ -30,7 +30,7 @@ class SearchIntelx:
self.results: dict[str, Any] = {}
self.emails: list[str] = []
self.hostnames: list[str] = []
self.interesting_urls: list[str] = []
self.urls: list[str] = []
self.limit: int = 10000
self.proxy = False
self.offset = 0
@@ -80,7 +80,7 @@ class SearchIntelx:
intelx_parser = intelxparser.Parser()
raw_emails, raw_selectors = await intelx_parser.parse_dictionaries(self.results)
emails: set[str] = set()
interesting_urls: set[str] = set()
urls: set[str] = set()
hostnames: set[str] = set()
for email in raw_emails:
@@ -94,16 +94,19 @@ class SearchIntelx:
emails.add(f'{address.username}@{normalized_domain}')
for selector in raw_selectors:
selector = selector.strip()
try:
parsed = urlparse(selector if '://' in selector else f'//{selector}')
except ValueError:
continue
interesting_urls.add(selector)
if normalized_hostname := normalize_scoped_hostname(parsed.hostname, self.word):
normalized_hostname = normalize_scoped_hostname(parsed.hostname, self.word)
if normalized_hostname:
hostnames.add(normalized_hostname)
if parsed.scheme in {'http', 'https'} and parsed.netloc:
urls.add(selector)
self.emails = sorted(emails)
self.interesting_urls = sorted(interesting_urls)
self.urls = sorted(urls)
self.hostnames = sorted(hostnames)
async def get_emails(self) -> list[str]:
@@ -112,5 +115,5 @@ class SearchIntelx:
async def get_hostnames(self) -> list[str]:
return self.hostnames
async def get_interestingurls(self) -> list[str]:
return self.interesting_urls
async def get_urls(self) -> list[str]:
return self.urls
theHarvester/discovery/rocketreach.py
+4 -4
View File
@@ -17,7 +17,7 @@ class SearchRocketReach:
self.hosts: set = set()
self.proxy = False
self.baseurl = 'https://api.rocketreach.co/api/v2/person/search'
self.links: set = set()
self.urls: set = set()
self.emails: set = set()
self.limit = limit
@@ -64,7 +64,7 @@ class SearchRocketReach:
for profile in profiles:
if 'linkedin_url' in profile:
self.links.add(profile['linkedin_url'])
self.urls.add(profile['linkedin_url'])
if profile.get('emails'):
for email in profile['emails']:
if email.get('email'):
@@ -86,8 +86,8 @@ class SearchRocketReach:
except Exception as e:
logger.info(f'An exception has occurred rocketreach: {e}')
async def get_links(self):
return self.links
async def get_urls(self):
return self.urls
async def get_emails(self):
return self.emails
theHarvester/discovery/shodansearch.py
+11 -4
View File
@@ -17,8 +17,10 @@ class SearchShodan:
self.api = Shodan(self.key)
self.hostdatarow: list = []
self.tracker: OrderedDict = OrderedDict()
self.error_type: str | None = None
async def search_ip(self, ip) -> OrderedDict:
self.error_type = None
try:
ipaddress = ip
results = self.api.host(ipaddress)
@@ -108,10 +110,15 @@ class SearchShodan:
}
return self.tracker
except exception.APIError:
logger.info(f'{ip}: Not in Shodan')
self.tracker[ip] = 'Not in Shodan'
except exception.APIError as error:
if str(error).strip().rstrip('.').casefold() == 'no information available for that ip':
logger.info(f'{ip}: Not in Shodan')
self.tracker[ip] = 'Not in Shodan'
else:
self.error_type = type(error).__name__
self.tracker[ip] = 'Shodan request failed'
except Exception as e:
self.tracker[ip] = f'Error occurred in the Shodan IP search module: {e}'
self.error_type = type(e).__name__
self.tracker[ip] = 'Shodan request failed'
return self.tracker
theHarvester/discovery/takeover.py
+28 -4
View File
@@ -5,7 +5,7 @@ from random import shuffle
import ujson
from theHarvester.lib.core import AsyncFetcher, Core
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.output import output_logger
logger = logging.getLogger(__name__)
@@ -19,6 +19,10 @@ class TakeOver:
self.fingerprints: dict[str, str] = dict()
# https://stackoverflow.com/questions/33080869/python-how-to-create-a-dict-of-dict-of-list-with-defaultdict
self.results: defaultdict[str, list] = defaultdict(list)
self.request_count = 0
self.request_error_count = 0
self.request_error_types: set[str] = set()
self.scan_error_type: str | None = None
async def populate_fingerprints(self):
# Thank you to https://github.com/EdOverflow/can-i-take-over-xyz for these fingerprints
@@ -78,6 +82,10 @@ class TakeOver:
self.results[url].append({match: service})
async def do_take(self) -> None:
self.request_count = 0
self.request_error_count = 0
self.request_error_types.clear()
self.scan_error_type = None
try:
if len(self.hosts) > 0:
# Returns a list of tuples in this format: (url, response)
@@ -85,21 +93,37 @@ class TakeOver:
https_hosts = [f'https://{host}' for host in self.hosts]
http_hosts = [f'http://{host}' for host in self.hosts]
all_hosts = https_hosts + http_hosts
self.request_count = len(all_hosts)
shuffle(all_hosts)
resps: list = await AsyncFetcher.fetch_all(all_hosts, takeover=True, proxy=self.proxy)
for url, resp in tuple(resp for resp in resps if len(resp[1]) >= 1):
await self.check(url, resp)
responses: list[tuple[str, FetcherResponse | None]] = await AsyncFetcher.fetch_all(
all_hosts,
takeover=True,
proxy=self.proxy,
include_metadata=True,
)
for url, response in responses:
if response is None:
self.request_error_count += 1
self.request_error_types.add('TransportError')
continue
if response.body:
await self.check(url, response.body)
else:
return
except IndexError:
self.scan_error_type = 'IndexError'
logger.info('Response was empty: possible network error or invalid URL.')
except ujson.JSONDecodeError:
self.scan_error_type = 'JSONDecodeError'
logger.info('Failed to parse JSON: cert fingerprints might be unavailable.')
except KeyError as ke:
self.scan_error_type = 'KeyError'
logger.info(f'Missing expected field in fingerprint: {ke}')
except TypeError as te:
self.scan_error_type = 'TypeError'
logger.info(f'Invalid response structure: {te}')
except Exception as e:
self.scan_error_type = type(e).__name__
logger.info(f'Unexpected error: {e}')
async def process(self, proxy: bool = False) -> None:
theHarvester/discovery/urlscan.py
+4 -4
View File
@@ -6,7 +6,7 @@ class SearchUrlscan:
self.word = word
self.totalhosts: set = set()
self.totalips: set = set()
self.interestingurls: set = set()
self.urls: set = set()
self.totalasns: set = set()
self.proxy = False
@@ -16,7 +16,7 @@ class SearchUrlscan:
resp = response[0]
self.totalhosts = {f'{page["page"]["domain"]}' for page in resp['results']}
self.totalips = {f'{page["page"]["ip"]}' for page in resp['results'] if 'ip' in page['page']}
self.interestingurls = {
self.urls = {
f'{page["page"]["url"]}' for page in resp['results'] if self.word in page['page']['url'] and 'url' in page['page']
}
self.totalasns = {f'{page["page"]["asn"]}' for page in resp['results'] if 'asn' in page['page']}
@@ -27,8 +27,8 @@ class SearchUrlscan:
async def get_ips(self) -> set:
return self.totalips
async def get_interestingurls(self) -> set:
return self.interestingurls
async def get_urls(self) -> set:
return self.urls
async def get_asns(self) -> set:
return self.totalasns
theHarvester/discovery/zoomeyesearch.py
+25 -20
View File
@@ -4,9 +4,11 @@ import math
import re
from collections.abc import Iterable
from typing import Any
from urllib.parse import urlparse
from theHarvester.discovery.constants import MissingKey, get_delay
from theHarvester.lib.core import AsyncFetcher, Core
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.parsers import myparser
logger = logging.getLogger(__name__)
@@ -29,7 +31,7 @@ class SearchZoomEye:
self.proxy = False
self.totalasns: list = list()
self.totalhosts: list = list()
self.interestingurls: list = list()
self.urls: list = list()
self.totalips: list = list()
self.totalemails: list = list()
# Regex used is directly from: https://github.com/GerbenJavado/LinkFinder/blob/master/linkfinder.py#L29
@@ -62,7 +64,7 @@ class SearchZoomEye:
)
(?:"|') # End newline delimiter
"""
self.iurl_regex = re.compile(regex_str, re.VERBOSE)
self.url_regex = re.compile(regex_str, re.VERBOSE)
def _build_headers(self) -> dict[str, str]:
# API v2 uses API-KEY header
@@ -205,24 +207,24 @@ class SearchZoomEye:
if isinstance(payload, dict):
matches = extract_matches(payload)
if matches:
hostnames, emails, ips, asns, iurls = await self.parse_matches(matches)
hostnames, emails, ips, asns, urls = await self.parse_matches(matches)
self.totalhosts.extend(hostnames)
self.totalemails.extend(emails)
self.totalips.extend(ips)
self.totalasns.extend(asns)
self.interestingurls.extend(iurls)
self.urls.extend(urls)
return
# Parse first page then loop
if isinstance(payload, dict):
matches = extract_matches(payload)
if matches:
hostnames, emails, ips, asns, iurls = await self.parse_matches(matches)
hostnames, emails, ips, asns, urls = await self.parse_matches(matches)
self.totalhosts.extend(hostnames)
self.totalemails.extend(emails)
self.totalips.extend(ips)
self.totalasns.extend(asns)
self.interestingurls.extend(iurls)
self.urls.extend(urls)
for num in range(2, self.limit + 1):
params = (('query', f'site:{self.word}'), ('page', str(num)), ('size', str(size)))
@@ -244,9 +246,9 @@ class SearchZoomEye:
break
continue
hostnames, emails, ips, asns, iurls = await self.parse_matches(matches)
hostnames, emails, ips, asns, urls = await self.parse_matches(matches)
if len(hostnames) == 0 and len(emails) == 0 and len(ips) == 0 and len(asns) == 0 and len(iurls) == 0:
if len(hostnames) == 0 and len(emails) == 0 and len(ips) == 0 and len(asns) == 0 and len(urls) == 0:
nomatches_counter += 1
if nomatches_counter >= 5:
break
@@ -255,7 +257,7 @@ class SearchZoomEye:
self.totalemails.extend(emails)
self.totalips.extend(ips)
self.totalasns.extend(asns)
self.interestingurls.extend(iurls)
self.urls.extend(urls)
if num % 10 == 0:
await asyncio.sleep(get_delay() + 1)
@@ -263,7 +265,7 @@ class SearchZoomEye:
async def parse_matches(self, matches):
# Helper function to parse items from match json
ips: set[str] = set()
iurls: set[str] = set()
urls: set[str] = set()
hostnames: set[str] = set()
asns: set[str] = set()
emails: set[str] = set()
@@ -317,7 +319,7 @@ class SearchZoomEye:
if isinstance(v, str):
self._safe_add_hostname(hostnames, v)
# Banner/content extraction for emails, hostnames, iurls
# Banner/content extraction for emails, hostnames, and URLs
banners = []
portinfo = match.get('portinfo')
@@ -344,18 +346,21 @@ class SearchZoomEye:
temp_emails = set(await self.parse_emails(content_blob))
emails.update(temp_emails)
hostnames.update(set(await self.parse_hostnames(content_blob)))
found_urls = {
str(iurl.group(1)).replace('"', '')
for iurl in re.finditer(self.iurl_regex, content_blob)
if self.word in str(iurl.group(1))
}
iurls.update(found_urls)
for url_match in re.finditer(self.url_regex, content_blob):
candidate = str(url_match.group(1)).replace('"', '')
try:
parsed = urlparse(candidate)
hostname = normalize_scoped_hostname(parsed.hostname, self.word)
except ValueError:
continue
if parsed.scheme in {'http', 'https'} and parsed.netloc and hostname:
urls.add(candidate)
except Exception as e:
# Continue processing other matches instead of failing completely
logger.info(f'ZoomEye parsing error: {e}')
return hostnames, emails, ips, asns, iurls
return hostnames, emails, ips, asns, urls
async def process(self, proxy: bool = False) -> None:
self.proxy = proxy
@@ -381,5 +386,5 @@ class SearchZoomEye:
async def get_asns(self):
return set(self.totalasns)
async def get_interestingurls(self):
return set(self.interestingurls)
async def get_urls(self):
return set(self.urls)
theHarvester/lib/active_evidence.py
+169
View File
@@ -0,0 +1,169 @@
from collections.abc import Iterable, Mapping
from dataclasses import dataclass
from datetime import datetime
from typing import Self
from theHarvester.lib.evidence_types import EXECUTION_STATUSES, RESULT_KINDS, ExecutionStatus, ResultKind, format_utc
@dataclass(frozen=True, order=True, slots=True)
class ActionObservation:
kind: ResultKind
value: str
def __post_init__(self) -> None:
if self.kind not in RESULT_KINDS:
raise ValueError(f'unknown action observation kind: {self.kind}')
if self.kind == 'screenshot':
raise ValueError('screenshots must be stored as artifacts, not results')
if not isinstance(self.value, str) or not self.value.strip():
raise ValueError('action observation value must be a non-empty string')
@dataclass(frozen=True, order=True, slots=True)
class ArtifactReference:
kind: str
subject_kind: ResultKind
subject_value: str
path: str
media_type: str
size_bytes: int
sha256: str
created_at: datetime
def __post_init__(self) -> None:
if any(not isinstance(value, str) or not value.strip() for value in (self.kind, self.path, self.media_type)):
raise ValueError('artifact kind, path, and media type must not be empty')
if (
self.subject_kind not in RESULT_KINDS
or self.subject_kind == 'screenshot'
or not isinstance(self.subject_value, str)
or not self.subject_value.strip()
):
raise ValueError('artifact must reference a known non-empty result')
if self.size_bytes < 0:
raise ValueError('artifact size must not be negative')
if (
not isinstance(self.sha256, str)
or len(self.sha256) != 64
or any(character not in '0123456789abcdef' for character in self.sha256)
):
raise ValueError('artifact sha256 must be 64 lowercase hexadecimal characters')
if not isinstance(self.created_at, datetime) or self.created_at.tzinfo is None or self.created_at.utcoffset() is None:
raise ValueError('artifact created_at must be timezone-aware')
def to_dict(self) -> dict[str, object]:
return {
'kind': self.kind,
'subject': {'kind': self.subject_kind, 'value': self.subject_value},
'file': {
'path': self.path,
'media_type': self.media_type,
'size_bytes': self.size_bytes,
'sha256': self.sha256,
},
'created_at': format_utc(self.created_at),
}
@dataclass(frozen=True, slots=True)
class ActionExecution:
action: str
status: ExecutionStatus
duration_ms: float
observations: tuple[ActionObservation, ...] = ()
artifacts: tuple[ArtifactReference, ...] = ()
error_type: str | None = None
stop_reason: str | None = None
def __post_init__(self) -> None:
if not isinstance(self.action, str) or not self.action.strip():
raise ValueError('action must not be empty')
if self.status not in EXECUTION_STATUSES:
raise ValueError(f'unknown execution status: {self.status}')
if self.duration_ms < 0:
raise ValueError('execution duration must not be negative')
if self.observations != tuple(sorted(set(self.observations))):
raise ValueError('action observations must be deduplicated and sorted')
if self.artifacts != tuple(sorted(set(self.artifacts))):
raise ValueError('artifacts must be deduplicated and sorted')
@classmethod
def finish(
cls,
*,
action: str,
status: ExecutionStatus,
duration_ms: float,
groups: Mapping[ResultKind, Iterable[str]],
artifacts: Iterable[ArtifactReference] = (),
error_type: str | None = None,
stop_reason: str | None = None,
) -> Self:
observations: set[ActionObservation] = set()
for kind, values in groups.items():
if kind not in RESULT_KINDS:
raise ValueError(f'unknown action observation kind: {kind}')
if kind == 'screenshot':
raise ValueError('screenshots must be stored as artifacts, not results')
for value in values:
if not isinstance(value, str) or not value.strip():
raise ValueError('action observation value must be a non-empty string')
observations.add(ActionObservation(kind, value.strip()))
return cls(
action=action.strip(),
status=status,
duration_ms=duration_ms,
observations=tuple(sorted(observations)),
artifacts=tuple(sorted(set(artifacts))),
error_type=error_type,
stop_reason=stop_reason,
)
@property
def result_count(self) -> int:
return len(self.observations)
def to_dict(self) -> dict[str, str | float | int | None]:
return {
'action': self.action,
'status': self.status,
'duration_ms': self.duration_ms,
'result_count': self.result_count,
'error_type': self.error_type,
'stop_reason': self.stop_reason,
}
@dataclass(frozen=True, slots=True)
class ActiveEvidence:
executions: tuple[ActionExecution, ...] = ()
def __post_init__(self) -> None:
actions = [execution.action for execution in self.executions]
if len(actions) != len(set(actions)):
raise ValueError('action executions must be unique')
@property
def observations(self) -> tuple[tuple[str, ActionObservation], ...]:
return tuple((execution.action, observation) for execution in self.executions for observation in execution.observations)
@property
def artifacts(self) -> tuple[tuple[str, ArtifactReference], ...]:
return tuple((execution.action, artifact) for execution in self.executions for artifact in execution.artifacts)
@dataclass(frozen=True, slots=True)
class ActionYield:
action: str
observed_result_count: int
unique_result_count: int
shared_result_count: int
def to_dict(self) -> dict[str, str | int]:
return {
'action': self.action,
'observed_result_count': self.observed_result_count,
'unique_result_count': self.unique_result_count,
'shared_result_count': self.shared_result_count,
}
theHarvester/lib/api/additional_endpoints.py
-80
View File
@@ -1,80 +0,0 @@
import logging
from typing import Annotated, NoReturn
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from theHarvester.discovery.additional_apis import AdditionalAPIs
from theHarvester.discovery.haveibeenpwned import SearchHaveIBeenPwned
from theHarvester.discovery.leaklookup import SearchLeakLookup
from theHarvester.lib.api.auth import get_api_key
router = APIRouter()
logger = logging.getLogger(__name__)
class DomainRequest(BaseModel):
domain: str = Field(..., min_length=3)
api_keys: dict[str, str] | None = None
def _raise_processing_error(endpoint: str, exc: Exception) -> NoReturn:
logger.exception(f'Error processing additional API endpoint {endpoint}')
raise HTTPException(status_code=500, detail='An error occurred while processing your request') from exc
@router.post('/breaches')
async def get_breaches(request: DomainRequest, _api_key: Annotated[str, Depends(get_api_key)]):
"""Get breach information for a domain using HaveIBeenPwned."""
try:
search = SearchHaveIBeenPwned(request.domain)
await search.process()
return {'status': 'success', 'data': await search.get_breaches()}
except Exception as e:
_raise_processing_error('breaches', e)
@router.post('/leaks')
async def get_leaks(request: DomainRequest, _api_key: Annotated[str, Depends(get_api_key)]):
"""Get leaked credentials for a domain using Leak-Lookup."""
try:
search = SearchLeakLookup(request.domain)
await search.process()
return {'status': 'success', 'data': await search.get_leaks()}
except Exception as e:
_raise_processing_error('leaks', e)
@router.post('/security-score')
async def get_security_score(request: DomainRequest, _api_key: Annotated[str, Depends(get_api_key)]):
"""Get security scorecard for a domain."""
try:
apis = AdditionalAPIs(request.domain, request.api_keys or {})
await apis._process_securityscorecard()
results = apis.results['security_score']
return {'status': 'success', 'data': results}
except Exception as e:
_raise_processing_error('security-score', e)
@router.post('/tech-stack')
async def get_tech_stack(request: DomainRequest, _api_key: Annotated[str, Depends(get_api_key)]):
"""Get technology stack information for a domain using BuiltWith."""
try:
apis = AdditionalAPIs(request.domain, request.api_keys or {})
await apis._process_builtwith()
results = apis.results['tech_stack']
return {'status': 'success', 'data': results}
except Exception as e:
_raise_processing_error('tech-stack', e)
@router.post('/all')
async def get_all_info(request: DomainRequest, _api_key: Annotated[str, Depends(get_api_key)]):
"""Get all additional information for a domain."""
try:
apis = AdditionalAPIs(request.domain, request.api_keys or {})
results = await apis.process()
return {'status': 'success', 'data': results}
except Exception as e:
_raise_processing_error('all', e)
theHarvester/lib/api/api.py
+23 -510
View File
@@ -1,528 +1,41 @@
import argparse
import ipaddress
import logging
import os
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from datetime import datetime
from typing import Annotated, Any, cast
from uuid import UUID
from __future__ import annotations
from fastapi import Depends, FastAPI, Header, HTTPException, Query, Request, status
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse, Response
from pydantic import BaseModel, Field
from slowapi import Limiter, _rate_limit_exceeded_handler
from slowapi.errors import RateLimitExceeded
from slowapi.util import get_remote_address
from contextlib import asynccontextmanager
from pathlib import Path
from typing import TYPE_CHECKING
from fastapi import FastAPI
from starlette.staticfiles import StaticFiles
from theHarvester import __main__
from theHarvester.lib.api.additional_endpoints import router as additional_router
from theHarvester.lib.api.auth import get_api_key
from theHarvester.lib.completed_result import ResultKind
from theHarvester.lib.database import ResultStore, dispose_sqlite_databases
from theHarvester.lib.recursive_dns import DEFAULT_RECURSIVE_DNS_QUERY_LIMIT
from theHarvester import __version__
from theHarvester.lib.api.run_worker import start_worker, stop_worker
from theHarvester.lib.api.runs import router as api_router
from theHarvester.lib.database import dispose_sqlite_databases
logger = logging.getLogger(__name__)
API_RATE_LIMIT = os.getenv('API_RATE_LIMIT', '5/minute')
# Define Pydantic models for request and response validation
class QueryResponse(BaseModel):
asns: list[str] = Field(default_factory=list, description='List of ASNs')
interesting_urls: list[str] = Field(default_factory=list, description='List of interesting URLs')
twitter_people: list[str] = Field(default_factory=list, description='List of Twitter people')
linkedin_people: list[dict] = Field(default_factory=list, description='List of LinkedIn people')
linkedin_links: list[str] = Field(default_factory=list, description='List of LinkedIn links')
trello_urls: list[str] = Field(default_factory=list, description='List of discovered URLs (legacy field name)')
ips: list[str] = Field(default_factory=list, description='List of IPs')
emails: list[str] = Field(default_factory=list, description='List of emails')
hosts: list[str] = Field(default_factory=list, description='List of hosts')
breaches: list[str] = Field(default_factory=list, description='List of breach names')
class ErrorResponse(BaseModel):
detail: str = Field(..., description='Error message')
error_type: str | None = Field(None, description='Type of error')
traceback: str | None = Field(None, description='Error traceback')
limiter = Limiter(key_func=get_remote_address)
if TYPE_CHECKING:
from collections.abc import AsyncIterator
@asynccontextmanager
async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
manager = ResultStore()
await manager.initialize()
await start_worker()
try:
yield
finally:
await dispose_sqlite_databases()
try:
await stop_worker()
finally:
await dispose_sqlite_databases()
app = FastAPI(
title='Restful Harvest',
description='Rest API for theHarvester powered by FastAPI',
version='0.0.4',
title='theHarvester API',
description='Local API for finite theHarvester enumeration runs',
version=__version__,
docs_url='/docs',
redoc_url='/redoc',
lifespan=lifespan,
)
app.state.limiter = limiter
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) # type: ignore
# Add CORS middleware
app.add_middleware(
cast('Any', CORSMiddleware),
allow_origins=['*'],
allow_credentials=False,
allow_methods=['GET', 'POST'],
allow_headers=['*'],
)
# Include additional endpoints
app.include_router(additional_router, prefix='/additional', tags=['Additional APIs'])
# This is where we will host files that arise if the user specifies a filename
try:
app.mount('/static', StaticFiles(directory='theHarvester/lib/api/static/'), name='static')
except RuntimeError:
static_path = os.path.expanduser('~/.local/share/theHarvester/static/')
if not os.path.isdir(static_path):
os.makedirs(static_path)
app.mount(
'/static',
StaticFiles(directory=static_path),
name='static',
)
@app.get('/', response_class=HTMLResponse)
async def root(*, user_agent: Annotated[str | None, Header()] = None) -> Response:
"""Root endpoint that displays the theHarvester logo and links to the GitHub repository.
Also performs basic user agent filtering to redirect suspicious bots.
"""
# Very basic user agent filtering
if user_agent and ('gobuster' in user_agent or 'sqlmap' in user_agent or 'rustbuster' in user_agent):
response = RedirectResponse(app.url_path_for('bot'))
return response
return HTMLResponse(
"""
<!DOCTYPE html>
<html lang="en-US">
<head>
<title>theHarvester API</title>
<style>
.img-container {
text-align: center;
display: block;
}
.api-links {
text-align: center;
margin-top: 20px;
font-family: Arial, sans-serif;
}
.api-links a {
margin: 0 10px;
text-decoration: none;
color: #0366d6;
}
.api-links a:hover {
text-decoration: underline;
}
</style>
</head>
<body>
<br/>
<a href="https://github.com/laramies/theHarvester" target="_blank">
<span class="img-container">
<img src="https://raw.githubusercontent.com/laramies/theHarvester/master/theHarvester-logo.webp" alt="theHarvester logo"/>
</span>
</a>
<div class="api-links">
<a href="/docs">API Documentation</a> |
<a href="/redoc">ReDoc Documentation</a> |
<a href="/sources">Available Sources</a>
</div>
</body>
</html>
"""
)
# Define Pydantic model for bot response
class BotResponse(BaseModel):
bot: str = Field(..., description='Bot message')
@app.get('/nicebot', response_model=BotResponse)
async def bot() -> Response:
"""Easter egg endpoint for bots.
Returns a Star Wars reference when accessed.
"""
return JSONResponse({'bot': 'These are not the droids you are looking for'})
# Define Pydantic model for sources response
class SourcesResponse(BaseModel):
sources: list[str] = Field(..., description='List of supported data sources')
class CompletedRunSummary(BaseModel):
run_id: UUID
target: str
started_at: datetime
completed_at: datetime
result_count: int
class CompletedResultItem(BaseModel):
type: ResultKind
value: str
class CompletedRunDetail(CompletedRunSummary):
results: list[CompletedResultItem]
@app.get(
'/sources',
response_model=SourcesResponse,
responses={
status.HTTP_500_INTERNAL_SERVER_ERROR: {'model': ErrorResponse},
status.HTTP_429_TOO_MANY_REQUESTS: {'model': ErrorResponse},
},
)
@limiter.limit(API_RATE_LIMIT)
async def getsources(request: Request) -> Response:
"""Endpoint to query for available sources theHarvester supports.
Returns a list of all supported data sources that can be used with the query endpoint.
Rate limit is configurable via CLI argument (default: 5 requests per minute).
"""
try:
sources = __main__.Core.get_supportedengines()
return JSONResponse({'sources': sources})
except Exception as e:
logger.exception('Error in getsources endpoint')
return JSONResponse(
{
'detail': 'An error occurred while retrieving sources',
'error_type': type(e).__name__,
},
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
@app.get(
'/runs',
response_model=list[CompletedRunSummary],
responses={status.HTTP_429_TOO_MANY_REQUESTS: {'model': ErrorResponse}},
)
@limiter.limit(API_RATE_LIMIT)
async def list_runs(
request: Request,
_api_key: Annotated[str, Depends(get_api_key)],
limit: Annotated[int, Query(ge=1, le=500)] = 50,
) -> list[dict[str, object]]:
"""List recently completed enumeration runs."""
manager = ResultStore()
await manager.initialize()
return await manager.list_runs(limit=limit)
@app.get(
'/runs/{run_id}',
response_model=CompletedRunDetail,
responses={
status.HTTP_404_NOT_FOUND: {'model': ErrorResponse},
status.HTTP_429_TOO_MANY_REQUESTS: {'model': ErrorResponse},
},
)
@limiter.limit(API_RATE_LIMIT)
async def get_run(
request: Request,
run_id: UUID,
_api_key: Annotated[str, Depends(get_api_key)],
) -> dict[str, object]:
"""Retrieve one completed enumeration run with its normalized evidence."""
manager = ResultStore()
await manager.initialize()
try:
result = await manager.load_run(run_id)
except LookupError as error:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='Completed run not found') from error
return {
'run_id': str(result.run_id),
'target': result.target,
'started_at': result.started_at.isoformat(),
'completed_at': result.completed_at.isoformat(),
'result_count': len(result.results),
'results': [{'type': kind, 'value': value} for kind, value in result.results],
}
# Define Pydantic model for DNS brute force response
class DnsBruteResponse(BaseModel):
dns_bruteforce: list[str] = Field(default_factory=list, description='List of DNS brute force results')
@app.get(
'/dnsbrute',
response_model=DnsBruteResponse,
responses={
status.HTTP_500_INTERNAL_SERVER_ERROR: {'model': ErrorResponse},
status.HTTP_400_BAD_REQUEST: {'model': ErrorResponse},
status.HTTP_429_TOO_MANY_REQUESTS: {'model': ErrorResponse},
},
)
@limiter.limit(API_RATE_LIMIT)
async def dnsbrute(
request: Request,
domain: Annotated[str, Query(min_length=3, description='Domain to be brute forced')],
user_agent: Annotated[str | None, Header()] = None,
dns_resolve: Annotated[
str, Query(description='Perform DNS resolution on subdomains with a resolver list or passed in resolvers')
] = '',
) -> Response:
"""Endpoint for DNS brute forcing.
This endpoint performs DNS brute force on the specified domain and returns the results.
Rate limit is configurable via CLI argument (default: 5 requests per minute).
"""
# Basic user agent filtering
if user_agent and ('gobuster' in user_agent or 'sqlmap' in user_agent or 'rustbuster' in user_agent):
response = RedirectResponse(app.url_path_for('bot'))
return response
try:
# Validate domain
if not domain or len(domain) < 3:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail='Domain must be at least 3 characters long')
# Call the main function with the provided parameters
dns_bruteforce = await __main__.start(
argparse.Namespace(
dns_brute=True,
dns_lookup=False,
dns_server=False,
dns_tld=False,
domain=domain,
filename='',
google_dork=False,
limit=500,
proxies=False,
shodan=False,
source=','.join([]),
start=0,
take_over=False,
wordlist='',
api_scan=False,
dns_resolve=dns_resolve,
)
)
return JSONResponse({'dns_bruteforce': dns_bruteforce})
except HTTPException as e:
# Re-raise HTTP exceptions
raise e
except Exception as e:
logger.exception('Error in dnsbrute endpoint')
return JSONResponse(
{
'detail': 'An error occurred while processing your request',
'error_type': type(e).__name__,
},
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
@app.get(
'/query',
response_model=QueryResponse,
responses={
status.HTTP_500_INTERNAL_SERVER_ERROR: {'model': ErrorResponse},
status.HTTP_400_BAD_REQUEST: {'model': ErrorResponse},
status.HTTP_429_TOO_MANY_REQUESTS: {'model': ErrorResponse},
},
)
@limiter.limit(API_RATE_LIMIT)
async def query(
request: Request,
source: Annotated[
list[str],
Query(
description=(
'Source names or source capabilities to query. Multiple capabilities select the union of matching '
'sources; they do not filter returned fields.'
)
),
],
domain: Annotated[str, Query(min_length=3, description='Domain to be harvested')],
dns_server: Annotated[
str,
Query(description='Accepted for compatibility but currently unused; use dns_resolve to select resolvers.'),
] = '',
user_agent: Annotated[str | None, Header()] = None,
x_api_key: Annotated[str | None, Header(alias='X-API-Key')] = None,
dns_brute: Annotated[bool, Query(description='Perform a DNS brute force on the domain')] = False,
dns_lookup: Annotated[
bool,
Query(
description=(
'Perform PTR lookups across the /24 network containing each discovered IPv4 address. '
'This sends active DNS queries.'
)
),
] = False,
dns_resolve: Annotated[str, Query(description='Resolve discovered hostnames using resolver IPs or a resolver file')] = '',
dns_recursive_depth: Annotated[int, Query(ge=0, description='Maximum recursive DNS discovery depth. Zero disables it.')] = 0,
dns_recursive_query_limit: Annotated[
int, Query(gt=0, description='Hard cap on recursive DNS record queries across all resolver vantages')
] = DEFAULT_RECURSIVE_DNS_QUERY_LIMIT,
dns_recursive_runtime_seconds: Annotated[
float, Query(gt=0, allow_inf_nan=False, description='Hard runtime cap in seconds for recursive DNS discovery')
] = 60.0,
filename: Annotated[
str,
Query(description=('Write uniquely prefixed server-side XML, JSON, and JSONL files using NAME as the filename suffix.')),
] = '',
proxies: Annotated[
bool,
Query(description='Use proxies.yaml for supported discovery-source and takeover requests.'),
] = False,
shodan: Annotated[bool, Query(description='Use Shodan to query discovered hosts')] = False,
take_over: Annotated[
bool,
Query(description='Check discovered hosts for known takeover indicators, using configured proxies when enabled.'),
] = False,
wordlist: Annotated[str, Query(description='Path to the endpoint wordlist used by api_scan')] = '',
api_scan: Annotated[
bool,
Query(description='Check common API paths with GET, HEAD, and OPTIONS. Requests follow redirects.'),
] = False,
limit: Annotated[int, Query(description='Maximum results requested from each source that supports result limits')] = 500,
start: Annotated[int, Query(description='Result offset for sources that support pagination')] = 0,
) -> Response:
"""Query function that allows user to query theHarvester rest API.
This endpoint performs searches using the specified data sources and returns the results.
Rate limit is configurable via CLI argument (default: 5 requests per minute).
"""
# Basic user agent filtering
if user_agent and ('gobuster' in user_agent or 'sqlmap' in user_agent or 'rustbuster' in user_agent):
response = RedirectResponse(app.url_path_for('bot'))
return response
try:
# Validate sources
selected_sources = __main__.Core.expand_source_selection(','.join(source))
credentialed_source = any(
source_name in selected_sources and bool((key_getter() or '').strip())
for source_name, key_getter in (
('dehashed', __main__.Core.dehashed_key),
('hibpverified', __main__.Core.hibpverified_key),
('leaklookup', __main__.Core.leaklookup_key),
)
)
if credentialed_source:
get_api_key(x_api_key)
supported_engines = __main__.Core.get_supportedengines()
for s in selected_sources:
if s not in supported_engines:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Source '{s}' is not supported. Supported sources: {', '.join(supported_engines)}",
)
if dns_recursive_depth > 0:
get_api_key(x_api_key)
try:
recursive_resolvers = {
str(ipaddress.ip_address(value.strip())) for value in dns_resolve.split(',') if value.strip()
}
except ValueError as error:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail='recursive DNS requires exactly three distinct resolver IPs',
) from error
if len(recursive_resolvers) != 3:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail='recursive DNS requires exactly three distinct resolver IPs',
)
# Call the main function with the provided parameters
(
asns,
iurls,
twitter_people_list,
linkedin_people_list,
linkedin_links,
aurls,
aips,
aemails,
ahosts,
abreaches,
) = await __main__.start(
argparse.Namespace(
dns_brute=dns_brute,
dns_lookup=dns_lookup,
dns_server=dns_server,
domain=domain,
filename=filename,
limit=limit,
proxies=proxies,
shodan=shodan,
source=','.join(selected_sources),
start=start,
take_over=take_over,
wordlist=wordlist,
api_scan=api_scan,
dns_resolve=dns_resolve,
dns_recursive_depth=dns_recursive_depth,
dns_recursive_query_limit=dns_recursive_query_limit,
dns_recursive_runtime_seconds=dns_recursive_runtime_seconds,
quiet=False,
screenshot='',
),
persist_completed_result=True,
include_breaches=True,
)
# Return the results using the Pydantic model
return JSONResponse(
{
'asns': asns,
'interesting_urls': iurls,
'twitter_people': twitter_people_list,
'linkedin_people': linkedin_people_list,
'linkedin_links': linkedin_links,
'trello_urls': aurls,
'ips': aips,
'emails': aemails,
'hosts': ahosts,
'breaches': abreaches,
}
)
except HTTPException as e:
# Re-raise HTTP exceptions
raise e
except Exception as e:
logger.exception('Error in query endpoint')
return JSONResponse(
{
'detail': 'An error occurred while processing your request',
'error_type': type(e).__name__,
},
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
app.include_router(api_router)
STATIC_DIRECTORY = Path(__file__).resolve().parent / 'static'
app.mount('/static', StaticFiles(directory=STATIC_DIRECTORY), name='static')
theHarvester/lib/api/api_example.py
-127
View File
@@ -1,127 +0,0 @@
"""Example script to query theHarvester rest API, obtain results, and write out to stdout as well as an html"""
import asyncio
import logging
import aiohttp
import netaddr
from theHarvester.lib.output import configure_logging, output_logger, print_section, sorted_unique
logger = logging.getLogger(__name__)
async def fetch_json(session, url):
try:
async with session.get(url) as response:
response.raise_for_status() # Raise an exception for 4XX/5XX responses
return await response.json()
except Exception as e:
logger.info(f'Error fetching data from {url}: {e}')
return {}
async def fetch(session, url):
try:
async with session.get(url) as response:
response.raise_for_status() # Raise an exception for 4XX/5XX responses
return await response.text()
except Exception as e:
logger.info(f'Error fetching data from {url}: {e}')
return ''
async def main() -> None:
"""Just a simple example of how to interact with the rest api
you can use httpx instead of aiohttp or whatever you best see fit
"""
url = 'http://127.0.0.1:5000'
domain = 'netflix.com'
query_url = f'{url}/query?dns_brute=false&dns_lookup=false&dns_tld=false&proxies=false&shodan=false&take_over=false&virtual_host=false&api_scan=false&source=otx&source=subdomaincenter&limit=500&start=0&domain={domain}'
async with aiohttp.ClientSession() as session:
fetched_json = await fetch_json(session, query_url)
total_asns = fetched_json.get('asns', [])
interesting_urls = fetched_json.get('interesting_urls', [])
twitter_people_list_tracker = fetched_json.get('twitter_people', [])
linkedin_people_list_tracker = fetched_json.get('linkedin_people', [])
linkedin_links_tracker = fetched_json.get('linkedin_links', [])
trello_urls = fetched_json.get('trello_urls', [])
ips = fetched_json.get('ips', [])
emails = fetched_json.get('emails', [])
hosts = fetched_json.get('hosts', [])
if len(total_asns) > 0:
print_section(f'\n[*] ASNS found: {len(total_asns)}', total_asns, '--------------------')
total_asns = sorted_unique(total_asns)
if len(interesting_urls) > 0:
print_section(f'\n[*] Interesting Urls found: {len(interesting_urls)}', interesting_urls, '--------------------')
interesting_urls = sorted_unique(interesting_urls)
if len(twitter_people_list_tracker) == 0:
output_logger.info('\n[*] No Twitter users found.')
elif len(twitter_people_list_tracker) >= 1:
print_section(
'\n[*] Twitter Users found: ' + str(len(twitter_people_list_tracker)),
twitter_people_list_tracker,
'---------------------',
)
twitter_people_list_tracker = sorted_unique(twitter_people_list_tracker)
if len(linkedin_people_list_tracker) == 0:
output_logger.info('\n[*] No LinkedIn users found.')
elif len(linkedin_people_list_tracker) >= 1:
print_section(
'\n[*] LinkedIn Users found: ' + str(len(linkedin_people_list_tracker)),
linkedin_people_list_tracker,
'---------------------',
)
linkedin_people_list_tracker = sorted_unique(linkedin_people_list_tracker)
if len(linkedin_links_tracker) == 0:
output_logger.info('\n[*] No LinkedIn links found.')
else:
print_section(
f'\n[*] LinkedIn Links found: {len(linkedin_links_tracker)}', linkedin_links_tracker, '---------------------'
)
linkedin_links_tracker = sorted_unique(linkedin_links_tracker)
length_urls = len(trello_urls)
if length_urls == 0:
output_logger.info('\n[*] No Trello URLs found.')
else:
print_section('\n[*] Trello URLs found: ' + str(length_urls), trello_urls, '--------------------')
if len(ips) == 0:
output_logger.info('\n[*] No IPs found.')
else:
output_logger.info('\n[*] IPs found: ' + str(len(ips)))
output_logger.info('-------------------')
# use netaddr as the list may contain ipv4 and ipv6 addresses
ip_list = sorted([netaddr.IPAddress(ip.strip()) for ip in set(ips)])
output_logger.info('\n'.join(map(str, ip_list)))
if len(emails) == 0:
output_logger.info('\n[*] No emails found.')
else:
output_logger.info('\n[*] Emails found: ' + str(len(emails)))
output_logger.info('----------------------')
all_emails = sorted_unique(emails)
output_logger.info('\n'.join(all_emails))
if len(hosts) == 0:
output_logger.info('\n[*] No hosts found.\n\n')
else:
output_logger.info('\n[*] Hosts found: ' + str(len(hosts)))
output_logger.info('---------------------')
output_logger.info('\n'.join(hosts))
def entry_point() -> None:
configure_logging(verbose=True)
asyncio.run(main())
if __name__ == '__main__':
entry_point()
theHarvester/lib/api/auth.py
+20 -3
View File
@@ -1,15 +1,32 @@
import os
import secrets
from pathlib import Path
from typing import Annotated
from fastapi import Header, HTTPException, status
API_KEY_ENV_VAR = 'THEHARVESTER_API_KEY'
API_KEY_FILE_ENV_VAR = f'{API_KEY_ENV_VAR}_FILE'
def get_api_key(x_api_key: Annotated[str | None, Header(alias='X-API-Key')] = None) -> str:
"""Validate the API key used by protected API routes."""
def _configured_api_key() -> str | None:
configured_api_key = os.getenv(API_KEY_ENV_VAR)
if configured_api_key:
return configured_api_key
configured_api_key_file = os.getenv(API_KEY_FILE_ENV_VAR)
if not configured_api_key_file:
return None
try:
return Path(configured_api_key_file).read_text(encoding='utf-8').strip() or None
except OSError:
return None
def get_api_key(
x_api_key: Annotated[str | None, Header(alias='X-API-Key')] = None,
) -> str:
"""Validate the API key used by protected API routes."""
configured_api_key = _configured_api_key()
if not configured_api_key:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
@@ -22,4 +39,4 @@ def get_api_key(x_api_key: Annotated[str | None, Header(alias='X-API-Key')] = No
detail='Invalid API key',
)
return x_api_key
return configured_api_key
theHarvester/lib/api/run_artifacts.py
+67
View File
@@ -0,0 +1,67 @@
from __future__ import annotations
import json
import os
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from fastapi import HTTPException
from theHarvester.lib.database import ResultStore
from .run_evidence import validate_evidence
@dataclass(frozen=True, slots=True)
class RunPaths:
database: Path
artifacts: Path
@classmethod
def configured(cls, database: str | Path | None = None) -> RunPaths:
database_path = Path(database or os.getenv('THEHARVESTER_RUN_DB') or ResultStore().database)
database_path = database_path.expanduser()
configured_artifacts = os.getenv('THEHARVESTER_RUN_ARTIFACTS')
artifact_root = (
Path(configured_artifacts).expanduser() if configured_artifacts else database_path.parent / 'run-artifacts'
)
return cls(database=database_path, artifacts=artifact_root)
def artifact_directory(self, run_id: str) -> Path:
return self.artifacts / run_id
def ensure_private_directory(path: Path) -> None:
path.mkdir(parents=True, exist_ok=True, mode=0o700)
if path.is_symlink():
raise OSError(f'Refusing symlinked theHarvester directory: {path}')
path.chmod(0o700)
def read_child_evidence(
artifact_dir: Path,
expected_target: str | None = None,
) -> tuple[dict[str, Any] | None, str | None]:
evidence_path = artifact_dir / 'evidence.json'
if not evidence_path.is_file():
return None, None
try:
evidence = validate_evidence(json.loads(evidence_path.read_text(encoding='utf-8')))
if expected_target is not None and evidence.get('target') != expected_target:
return None, 'Child evidence target does not match run target'
return evidence, None
except (OSError, json.JSONDecodeError, HTTPException) as error:
return None, f'Child evidence is invalid: {error}'
def write_child_evidence(artifact_dir: Path, evidence: Any, *, partial: bool) -> None:
payload = evidence.evidence_dict()
if partial:
payload['status'] = 'partial'
temporary = artifact_dir / 'evidence.json.tmp'
temporary.write_text(json.dumps(payload), encoding='utf-8')
temporary.chmod(0o600)
evidence_path = artifact_dir / 'evidence.json'
temporary.replace(evidence_path)
evidence_path.chmod(0o600)
theHarvester/lib/api/run_evidence.py
+98
View File
@@ -0,0 +1,98 @@
from __future__ import annotations
from datetime import datetime, timedelta
from typing import Any
from uuid import UUID
from fastapi import HTTPException, status
from theHarvester.lib.completed_result import parse_result_jsonl
from theHarvester.lib.evidence_types import EVIDENCE_STATUSES
from .run_models import _normalize_target
def parse_jsonl_import(body: bytes) -> dict[str, Any]:
try:
summary, findings = parse_result_jsonl(body)
except ValueError as error:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(error)) from error
try:
raw_run_id = summary.get('run_id')
if not isinstance(raw_run_id, str):
raise ValueError
run_id = str(UUID(raw_run_id))
except ValueError as error:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail='JSONL summary must contain a UUID run_id',
) from error
timestamps: dict[str, datetime] = {}
for field in ('started_at', 'completed_at'):
value = summary.get(field)
try:
timestamp = datetime.fromisoformat(value) if isinstance(value, str) else None
except ValueError:
timestamp = None
if timestamp is None or timestamp.tzinfo is None or timestamp.utcoffset() != timedelta(0):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f'JSONL summary must contain an ISO-8601 UTC {field}',
)
timestamps[field] = timestamp
if timestamps['completed_at'] < timestamps['started_at']:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail='JSONL summary completed_at must not be earlier than started_at',
)
evidence = {
'run_id': run_id,
'target': summary.get('target'),
'status': summary.get('evidence_status'),
'started_at': summary.get('started_at'),
'completed_at': summary.get('completed_at'),
'results': [
{
'type': record['type'],
'value': record['value'],
'sources': record['sources'],
'actions': record['actions'],
}
for record in findings
],
'source_executions': summary.get('source_executions', []),
'action_executions': summary.get('action_executions', []),
'artifacts': summary.get('artifacts', []),
}
return validate_evidence(evidence)
def validate_evidence(evidence: dict[str, Any]) -> dict[str, Any]:
if not evidence.get('target'):
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail='Result file does not identify a target')
try:
evidence['target'] = _normalize_target(str(evidence['target']))
except ValueError as error:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(error)) from error
if evidence.get('status') not in EVIDENCE_STATUSES:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail='Evidence status must be complete, partial, or failed',
)
for field in ('started_at', 'completed_at'):
value = evidence.get(field)
if value is not None and not isinstance(value, str):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f'Evidence field {field} must be a string',
)
for field in ('results', 'source_executions', 'action_executions', 'artifacts'):
value = evidence.get(field)
if value is None:
evidence[field] = []
elif not isinstance(value, list):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f'Evidence field {field} must be an array',
)
return evidence
theHarvester/lib/api/run_models.py
+274
View File
@@ -0,0 +1,274 @@
from __future__ import annotations
import ipaddress
from datetime import UTC, datetime
from typing import Any, Literal
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
from theHarvester.lib.enumeration import (
DEFAULT_DNS_RECURSIVE_QUERY_LIMIT,
DEFAULT_DNS_RECURSIVE_RUNTIME_SECONDS,
DEFAULT_RESULT_START,
)
from theHarvester.lib.evidence_types import EvidenceStatus # noqa: TC001 - Pydantic resolves this annotation at runtime
from theHarvester.lib.resolver_selection import DEFAULT_DNS_RESOLVERS, normalize_resolver_addresses
from theHarvester.lib.source_catalog import SOURCE_SPECS, ActivityClass, selected_action_names
def utc_now() -> str:
return datetime.now(UTC).isoformat()
def _normalize_target(value: str) -> str:
target = value.strip().rstrip('.').lower()
if not target or len(target) > 253 or any(character in target for character in '/?#@'):
raise ValueError('Target must be a hostname or IP address')
try:
return str(ipaddress.ip_address(target))
except ValueError:
try:
target = target.encode('idna').decode('ascii')
except UnicodeError as error:
raise ValueError('Target must be a valid hostname') from error
labels = target.split('.')
if any(
not label
or len(label) > 63
or label.startswith('-')
or label.endswith('-')
or not all(character.isalnum() or character == '-' for character in label)
for label in labels
):
raise ValueError('Target must be a valid hostname')
return target
class RunRequest(BaseModel):
model_config = ConfigDict(extra='forbid')
target: str = Field(description='Authorized domain name or IP address to enumerate.')
sources: list[str] = Field(
max_length=len(SOURCE_SPECS),
description=(
'Discovery source names or source capabilities. Multiple capabilities select the union of matching '
'sources and do not filter result fields. May be empty when a target-only action is selected.'
),
)
limit: int = Field(
default=500,
ge=1,
le=10_000,
description='Maximum results requested from each source when that provider supports a limit.',
)
start: int = Field(
default=DEFAULT_RESULT_START,
ge=0,
description='Starting result offset for providers that support pagination.',
)
deadline_seconds: int = Field(
default=1800,
ge=30,
le=86_400,
description='Hard deadline in seconds for the whole run, including every selected source and action.',
)
proxies: bool = Field(
default=False,
description='Use configured proxies for supported discovery sources and takeover requests.',
)
dns_brute: bool = Field(default=False, description='Try wordlist candidates below the authorized target through DNS.')
dns_lookup: bool = Field(
default=False,
description="Perform reverse DNS lookup across each discovered IPv4 address's /24 network.",
)
dns_resolve: bool = Field(
default=False,
description='Validate discovered hostnames through the configured resolver addresses.',
)
dns_resolvers: list[str] = Field(
default_factory=lambda: list(DEFAULT_DNS_RESOLVERS),
min_length=1,
description=('Distinct resolver IPv4 or IPv6 addresses used by DNS actions. Recursive DNS requires exactly three.'),
)
dns_recursive_depth: int = Field(
default=0,
ge=0,
description='Maximum recursive label depth. Zero disables recursive DNS discovery.',
)
dns_recursive_query_limit: int = Field(
default=DEFAULT_DNS_RECURSIVE_QUERY_LIMIT,
gt=0,
description='Maximum DNS record queries shared across all three resolver vantages.',
)
dns_recursive_runtime_seconds: float = Field(
default=DEFAULT_DNS_RECURSIVE_RUNTIME_SECONDS,
gt=0,
allow_inf_nan=False,
description='Maximum wall-clock seconds spent in recursive DNS discovery.',
)
shodan: bool = Field(default=False, description='Enrich discovered hosts with configured Shodan access.')
screenshot: bool = Field(
default=False,
description='Capture discovered web services, or the authorized target when no discovery sources are selected.',
)
takeover: bool = Field(
default=False,
description='Check discovered hosts for takeover indicators, using configured proxies when enabled.',
)
api_scan: bool = Field(
default=False,
description='Request common API paths directly from the authorized target.',
)
api_scan_paths: list[str] = Field(
default_factory=list,
max_length=500,
description='Optional endpoint paths used by API scan instead of its bundled wordlist.',
)
@field_validator('target')
@classmethod
def normalize_target(cls, value: str) -> str:
return _normalize_target(value)
@field_validator('sources')
@classmethod
def validate_sources(cls, values: list[str]) -> list[str]:
if len(values) != len(set(values)):
raise ValueError('Sources must not contain duplicates')
return values
@field_validator('dns_resolvers')
@classmethod
def validate_dns_resolvers(cls, values: list[str]) -> list[str]:
return normalize_resolver_addresses(values)
@field_validator('api_scan_paths')
@classmethod
def validate_api_scan_paths(cls, values: list[str]) -> list[str]:
paths = [value.strip() for value in values]
if any(
not path
or not path.startswith('/')
or len(path) > 2048
or '://' in path
or any(character in path for character in '\r\n')
for path in paths
):
raise ValueError('API scan paths must be non-empty URL paths beginning with /')
if len(paths) != len(set(paths)):
raise ValueError('API scan paths must not contain duplicates')
return paths
@model_validator(mode='after')
def validate_selected_work(self) -> RunRequest:
if not self.sources and not selected_action_names(self.model_dump()):
raise ValueError('Select at least one discovery source or action')
if self.dns_recursive_depth > 0 and len(self.dns_resolvers) != 3:
raise ValueError('Recursive DNS requires exactly three distinct resolver IPs')
return self
class SourceResponse(BaseModel):
name: str
activity: ActivityClass
credentials: list[str]
capabilities: list[str]
class ActionResponse(BaseModel):
name: str
activity: ActivityClass
class SourceCatalogResponse(BaseModel):
sources: list[SourceResponse]
actions: list[ActionResponse]
class DatabaseImportResponse(BaseModel):
filename: str
imported_run_ids: list[str]
skipped_run_ids: list[str]
class NormalizedResult(BaseModel):
type: str
value: str
sources: list[str] = Field(default_factory=list)
actions: list[str] = Field(default_factory=list)
class ScreenshotRecord(BaseModel):
name: str
target: str
url: str
RunStatus = Literal['queued', 'running', 'cancelling', 'cancelled', 'completed', 'failed']
Activity = Literal['P0', 'P1', 'P2']
class ImportedRunRequest(BaseModel):
filename: str
source_run_id: str
sources: list[str]
activities: list[Activity]
class RunSummary(BaseModel):
run_id: str
target: str
status: RunStatus
origin: Literal['local', 'imported']
created_at: str
started_at: str | None
completed_at: str | None
cancellation_requested_at: str | None
error: str | None
sources: list[str]
activities: list[Activity]
evidence_status: EvidenceStatus | None
result_count: int
class RunDetail(RunSummary):
request: RunRequest | ImportedRunRequest
results: list[NormalizedResult]
source_executions: list[dict[str, Any]]
action_executions: list[dict[str, Any]]
artifacts: list[dict[str, Any]]
screenshots: list[ScreenshotRecord]
log: str
RUN_REQUEST_OPENAPI = {
'requestBody': {
'required': True,
'content': {'application/json': {'schema': RunRequest.model_json_schema()}},
}
}
IMPORT_REQUEST_OPENAPI = {
'requestBody': {
'required': True,
'content': {'application/x-ndjson': {'schema': {'type': 'string', 'format': 'binary'}}},
}
}
DATABASE_IMPORT_REQUEST_OPENAPI = {
'requestBody': {
'required': True,
'content': {'application/vnd.sqlite3': {'schema': {'type': 'string', 'format': 'binary'}}},
}
}
EXPORT_RESPONSES: dict[int | str, dict[str, Any]] = {
200: {
'description': 'Normalized run results as JSONL.',
'content': {
'application/x-ndjson': {
'schema': {
'type': 'string',
'description': 'UTF-8 JSONL with one summary followed by normalized findings.',
}
},
},
}
}
theHarvester/lib/api/run_projection.py
+97
View File
@@ -0,0 +1,97 @@
from __future__ import annotations
from pathlib import Path
from typing import Any
from theHarvester.lib.source_catalog import (
ActivityClass,
SourceSpec,
activity_classes_for_selection,
get_source_spec,
selected_action_names,
)
def activities_for_request(request: dict[str, Any]) -> list[str]:
if request.get('activities'):
return list(request['activities'])
actions = selected_action_names(request)
return [activity.value for activity in activity_classes_for_selection(request.get('sources', []), actions)]
def source_spec(name: str) -> SourceSpec | None:
try:
return get_source_spec(name)
except KeyError:
return None
def normalized_results(evidence: dict[str, Any] | None) -> list[dict[str, Any]]:
if not evidence:
return []
results: list[dict[str, Any]] = []
for item in evidence.get('results') or []:
if isinstance(item, dict) and item.get('type') != 'screenshot':
results.append(
{
'type': str(item.get('type', 'other')),
'value': str(item.get('value', '')),
'sources': sorted({str(source) for source in item.get('sources', [])}),
'actions': sorted({str(action) for action in item.get('actions', [])}),
}
)
return results
def source_executions(evidence: dict[str, Any] | None) -> list[dict[str, Any]]:
if not evidence:
return []
executions = evidence.get('source_executions') or []
return [dict(execution) for execution in executions if isinstance(execution, dict)]
def screenshots(evidence: dict[str, Any] | None, run_id: str, artifact_dir: Path) -> list[dict[str, Any]]:
screenshot_dir = artifact_dir / 'screenshots'
if not screenshot_dir.is_dir():
return []
allowed_names: set[str] = set()
targets: dict[str, str] = {}
for artifact in (evidence or {}).get('artifacts') or []:
if not isinstance(artifact, dict) or artifact.get('kind') != 'screenshot':
continue
file = artifact.get('file')
subject = artifact.get('subject')
if not isinstance(file, dict) or not isinstance(subject, dict):
continue
name = Path(str(file.get('path', ''))).name
if name and name.endswith('.png'):
allowed_names.add(name)
targets[name] = str(subject.get('value') or Path(name).stem)
return [
{
'name': path.name,
'target': targets.get(path.name, path.stem),
'url': f'/api/v1/runs/{run_id}/screenshots/{path.name}',
}
for path in sorted(screenshot_dir.glob('*.png'))
if path.name in allowed_names and path.is_file()
]
def activities_for_evidence(
source_executions: list[dict[str, Any]],
action_executions: list[dict[str, Any]],
) -> list[str]:
sources = [str(execution.get('source', '')) for execution in source_executions]
actions = [str(execution.get('action', '')) for execution in action_executions]
explicit: set[ActivityClass] = set()
for execution in (*source_executions, *action_executions):
activity = str(execution.get('activity') or '')
try:
explicit.add(ActivityClass(activity))
except ValueError:
pass
activities = set(activity_classes_for_selection(sources, actions)) | explicit
if not activities:
activities.add(ActivityClass.PASSIVE)
return [activity.value for activity in ActivityClass if activity in activities]
theHarvester/lib/api/run_store.py
+539
View File
@@ -0,0 +1,539 @@
from __future__ import annotations
import json
from collections import Counter, defaultdict
from datetime import datetime
from typing import TYPE_CHECKING, Any, cast
from uuid import UUID, uuid4
from fastapi import HTTPException, status
from theHarvester.lib.active_evidence import ActionExecution, ActiveEvidence, ArtifactReference
from theHarvester.lib.completed_result import CompletedResult, ResultObservation, SourceExecution
from theHarvester.lib.database import DuplicateRunError, ResultStore, ResultStoreError, RunLifecycleStore
from theHarvester.lib.evidence_types import EXECUTION_STATUSES, EvidenceStatus, ExecutionStatus, ResultKind
from .run_artifacts import RunPaths, read_child_evidence
from .run_models import RunRequest, _normalize_target, utc_now
from .run_projection import activities_for_evidence, activities_for_request, normalized_results, screenshots, source_executions
if TYPE_CHECKING:
from pathlib import Path
WORKER_LEASE_TIMEOUT_SECONDS = 30
DATABASE_IMPORT_BATCH_SIZE = 100
def _execution_status(value: object) -> ExecutionStatus:
normalized = str(value)
if normalized not in EXECUTION_STATUSES:
raise ValueError(f'unknown execution status: {normalized}')
return cast('ExecutionStatus', normalized)
def _completed_result(
evidence: dict[str, Any],
*,
run_id: UUID,
fallback_started_at: str,
fallback_completed_at: str,
) -> CompletedResult:
results = [item for item in evidence.get('results', []) if isinstance(item, dict)]
groups: dict[ResultKind, set[str]] = defaultdict(set)
source_origins: set[ResultObservation] = set()
source_counts: Counter[str] = Counter()
action_groups: dict[str, dict[ResultKind, set[str]]] = defaultdict(lambda: defaultdict(set))
for item in results:
kind = cast('ResultKind', str(item['type']))
value = str(item['value'])
groups[kind].add(value)
for source in set(item.get('sources', [])):
source_name = str(source)
source_origins.add(ResultObservation(source_name, kind, value))
source_counts[source_name] += 1
for action in set(item.get('actions', [])):
action_groups[str(action)][kind].add(value)
source_details = {
str(item['source']): item
for item in evidence.get('source_executions', [])
if isinstance(item, dict) and item.get('source')
}
if len(source_details) != len(evidence.get('source_executions', [])):
raise ValueError('source executions must have unique non-empty names')
if missing_sources := set(source_counts) - set(source_details):
raise ValueError(f'missing source execution: {sorted(missing_sources)[0]}')
source_names = sorted(source_details)
completed_sources = tuple(
SourceExecution(
source=name,
status=_execution_status(source_details.get(name, {}).get('status', 'completed')),
duration_ms=float(source_details.get(name, {}).get('duration_ms', 0)),
result_count=source_counts[name],
error_type=source_details.get(name, {}).get('error_type'),
stop_reason=source_details[name].get('stop_reason'),
)
for name in source_names
)
artifacts_by_action: dict[str, list[ArtifactReference]] = defaultdict(list)
for item in evidence.get('artifacts', []):
if not isinstance(item, dict) or not item.get('action'):
continue
subject = item.get('subject')
file = item.get('file')
if not isinstance(subject, dict) or not isinstance(file, dict):
continue
artifacts_by_action[str(item['action'])].append(
ArtifactReference(
kind=str(item['kind']),
subject_kind=cast('ResultKind', str(subject['kind'])),
subject_value=str(subject['value']),
path=str(file['path']),
media_type=str(file['media_type']),
size_bytes=int(file['size_bytes']),
sha256=str(file['sha256']),
created_at=datetime.fromisoformat(str(item.get('created_at') or fallback_completed_at)),
)
)
action_details = {
str(item['action']): item
for item in evidence.get('action_executions', [])
if isinstance(item, dict) and item.get('action')
}
if len(action_details) != len(evidence.get('action_executions', [])):
raise ValueError('action executions must have unique non-empty names')
missing_actions = (set(action_groups) | set(artifacts_by_action)) - set(action_details)
if missing_actions:
raise ValueError(f'missing action execution: {sorted(missing_actions)[0]}')
action_names = sorted(action_details)
active_evidence = ActiveEvidence(
executions=tuple(
ActionExecution.finish(
action=name,
status=_execution_status(action_details.get(name, {}).get('status', 'completed')),
duration_ms=float(action_details.get(name, {}).get('duration_ms', 0)),
groups=action_groups[name],
artifacts=artifacts_by_action[name],
error_type=action_details.get(name, {}).get('error_type'),
stop_reason=action_details.get(name, {}).get('stop_reason'),
)
for name in action_names
)
)
execution_status_is_authoritative = bool(completed_sources or action_names)
return CompletedResult.finish(
run_id=run_id,
target=str(evidence['target']),
started_at=datetime.fromisoformat(str(evidence.get('started_at') or fallback_started_at)),
completed_at=datetime.fromisoformat(str(evidence.get('completed_at') or fallback_completed_at)),
groups=groups,
source_executions=completed_sources,
observations=sorted(source_origins),
active_evidence=active_evidence,
evidence_status=(
cast('EvidenceStatus', str(evidence['status']))
if evidence.get('status') is not None and not execution_status_is_authoritative
else None
),
)
def _imported_request(completed: CompletedResult, filename: str, source_run_id: str) -> dict[str, object]:
evidence = completed.evidence_dict()
executions = source_executions(evidence)
raw_action_executions = evidence.get('action_executions', [])
action_executions = (
[dict(execution) for execution in raw_action_executions if isinstance(execution, dict)]
if isinstance(raw_action_executions, list)
else []
)
return {
'filename': filename,
'source_run_id': source_run_id,
'sources': sorted(
{
str(execution.get('source') or execution.get('name'))
for execution in executions
if execution.get('source') or execution.get('name')
}
),
'activities': activities_for_evidence(executions, action_executions),
}
class RunStore:
"""Join API lifecycle state with the canonical SQLAlchemy result store."""
def __init__(self, database: str | Path | None = None) -> None:
self.paths = RunPaths.configured(database)
self.database = self.paths.database
self.lifecycle = RunLifecycleStore(self.database)
self.results = ResultStore(self.database)
def artifact_directory(self, run_id: str) -> Path:
return self.paths.artifact_directory(run_id)
async def initialize(self) -> None:
self.database.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
await self.lifecycle.initialize()
self.database.chmod(0o600)
async def _row(self, record: dict[str, object], *, detail: bool = False) -> dict[str, Any]:
request = json.loads(str(record['request_json']))
evidence = None
if detail and record['evidence_run_id'] is not None:
evidence = (await self.results.load_run(UUID(str(record['evidence_run_id'])))).evidence_dict()
summary_result_count = record.get('result_count')
result = {
'run_id': record['run_id'],
'target': record['target'],
'status': record['status'],
'origin': record['origin'],
'created_at': record['created_at'],
'started_at': record['started_at'],
'completed_at': record['completed_at'],
'cancellation_requested_at': record['cancellation_requested_at'],
'error': record['error'],
'sources': request.get('sources', []),
'activities': activities_for_request(request),
'evidence_status': record['evidence_status'] or (evidence.get('status') if evidence else None),
'result_count': (
len(normalized_results(evidence))
if evidence
else summary_result_count
if isinstance(summary_result_count, int)
else 0
),
}
if detail:
result.update(
request=request,
evidence=evidence,
results=normalized_results(evidence),
source_executions=source_executions(evidence),
action_executions=evidence.get('action_executions', []) if evidence else [],
artifacts=evidence.get('artifacts', []) if evidence else [],
screenshots=screenshots(evidence, str(record['run_id']), self.artifact_directory(str(record['run_id']))),
log=record['log'],
)
return result
async def create(self, request: RunRequest) -> dict[str, Any]:
await self.initialize()
run_id = str(uuid4())
await self.lifecycle.create(
run_id=run_id,
target=request.target,
status='queued',
origin='local',
created_at=utc_now(),
request_json=request.model_dump_json(),
)
run = await self.get(run_id)
assert run is not None
return run
async def import_evidence(self, evidence: dict[str, Any], filename: str) -> dict[str, Any]:
await self.initialize()
created_at = utc_now()
target = _normalize_target(str(evidence['target']))
source_run_id = str(evidence['run_id'])
run_id = str(uuid4())
try:
completed = _completed_result(
evidence,
run_id=UUID(run_id),
fallback_started_at=created_at,
fallback_completed_at=created_at,
)
except (KeyError, TypeError, ValueError) as error:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f'Invalid run evidence: {error}',
) from error
if evidence['status'] != completed.status:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail='Evidence status does not match its execution outcomes',
)
await self._persist_completed(completed)
request = _imported_request(completed, filename, source_run_id)
await self.lifecycle.create(
run_id=run_id,
target=target,
status='completed',
origin='imported',
created_at=created_at,
started_at=completed.started_at.isoformat(),
completed_at=completed.completed_at.isoformat(),
request_json=json.dumps(request),
evidence_run_id=run_id,
evidence_status=completed.status,
)
run = await self.get(run_id)
assert run is not None
return run
async def import_database(self, source_path: Path, filename: str) -> dict[str, object]:
await self.initialize()
source = ResultStore(source_path)
imported_run_ids: list[str] = []
skipped_run_ids: set[str] = set()
reuse_evidence_ids: set[str] = set()
async def source_summaries():
offset = 0
while batch := await source.list_runs(limit=DATABASE_IMPORT_BATCH_SIZE, offset=offset):
for summary in batch:
yield summary
offset += len(batch)
try:
await source.validate_import_database()
await source.initialize()
async for summary in source_summaries():
completed = await source.load_run(UUID(str(summary['run_id'])))
run_id = str(completed.run_id)
record = await self.lifecycle.get(run_id)
existing = None
try:
existing = await self.results.load_run(completed.run_id)
except (LookupError, ResultStoreError):
pass
if record is not None or existing is not None:
if record is not None and existing == completed:
skipped_run_ids.add(run_id)
continue
if existing != completed:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail=f'Run ID conflicts with different evidence: {run_id}',
)
reuse_evidence_ids.add(run_id)
async for summary in source_summaries():
completed = await source.load_run(UUID(str(summary['run_id'])))
run_id = str(completed.run_id)
if run_id in skipped_run_ids:
continue
if run_id not in reuse_evidence_ids:
await self.results.save_run(completed)
await self.lifecycle.create(
run_id=run_id,
target=completed.target,
status='completed',
origin='imported',
created_at=completed.completed_at.isoformat(),
started_at=completed.started_at.isoformat(),
completed_at=completed.completed_at.isoformat(),
request_json=json.dumps(_imported_request(completed, filename, run_id)),
evidence_run_id=run_id,
evidence_status=completed.status,
)
imported_run_ids.append(run_id)
except (ResultStoreError, RuntimeError, ValueError) as error:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(error)) from error
finally:
await source.dispose()
return {
'filename': filename,
'imported_run_ids': sorted(imported_run_ids),
'skipped_run_ids': sorted(skipped_run_ids),
}
async def list_runs(self, *, limit: int = 100, offset: int = 0) -> list[dict[str, Any]]:
await self.initialize()
return [await self._row(record) for record in await self.lifecycle.list_records(limit=limit, offset=offset)]
async def get(self, run_id: str) -> dict[str, Any] | None:
await self.initialize()
record = await self.lifecycle.get(run_id)
return await self._row(record, detail=True) if record else None
async def load_completed_result(self, run_id: str) -> CompletedResult | None:
"""Load the canonical evidence attached to an API run."""
await self.initialize()
record = await self.lifecycle.get(run_id)
if record is None:
raise LookupError(run_id)
if record['evidence_run_id'] is None:
return None
try:
completed = await self.results.load_run(UUID(str(record['evidence_run_id'])))
except (LookupError, ValueError) as error:
raise ResultStoreError('Attached run evidence does not exist') from error
if completed.target != str(record['target']):
raise ResultStoreError('Attached run evidence target does not match its lifecycle run')
return completed
async def cancel(self, run_id: str) -> dict[str, Any] | None:
await self.initialize()
try:
record = await self.lifecycle.cancel(run_id, utc_now())
except ValueError as error:
raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail=f'Run is already {error.args[0]}') from error
return await self._row(record, detail=True) if record else None
async def recover_orphans(self) -> None:
await self.initialize()
recovered_at = utc_now()
for record in await self.lifecycle.running():
run_id = str(record['run_id'])
target = str(record['target'])
evidence, evidence_error = read_child_evidence(self.artifact_directory(run_id), target)
error = 'theHarvester restarted before child completion'
if evidence_error:
error += f'; {evidence_error}'
evidence_run_id = None
completed = None
if evidence:
completed = await self._save_evidence(
evidence,
str(record['started_at'] or record['created_at']),
recovered_at,
run_id=UUID(run_id),
expected_target=target,
)
evidence_run_id = str(completed.run_id)
else:
completed = await self._existing_evidence(run_id, target)
if completed is not None:
evidence_run_id = str(completed.run_id)
evidence_status = completed.status if completed is not None else None
if evidence is not None and completed is not None:
evidence_status = str(evidence.get('status', completed.status))
await self.lifecycle.fail(
run_id,
status='failed',
completed_at=recovered_at,
error=error,
log=str(record['log']),
evidence_run_id=evidence_run_id,
evidence_status=evidence_status,
)
async def acquire_worker_lease(self, owner_id: str) -> bool:
await self.initialize()
return await self.lifecycle.acquire_lease(owner_id, utc_now(), WORKER_LEASE_TIMEOUT_SECONDS)
async def heartbeat_worker_lease(self, owner_id: str) -> bool:
return await self.lifecycle.heartbeat_lease(owner_id, utc_now())
async def release_worker_lease(self, owner_id: str) -> None:
return await self.lifecycle.release_lease(owner_id)
async def claim_next(self) -> dict[str, Any] | None:
await self.initialize()
record = await self.lifecycle.claim_next(utc_now())
return await self._row(record, detail=True) if record else None
async def finish(self, run_id: str, evidence: dict[str, Any] | None, log: str) -> None:
record = await self.lifecycle.get(run_id)
if record is None:
return
evidence_run_id = None
if evidence:
completed = await self._save_evidence(
evidence,
str(record['started_at'] or record['created_at']),
utc_now(),
run_id=UUID(run_id),
expected_target=str(record['target']),
)
evidence_run_id = str(completed.run_id)
await self.lifecycle.finish(
run_id,
completed_at=utc_now(),
evidence_run_id=evidence_run_id,
evidence_status=str(evidence.get('status', completed.status)) if evidence else None,
log=log[-200_000:],
)
async def fail(
self,
run_id: str,
error: str,
log: str,
*,
cancelled: bool = False,
evidence: dict[str, Any] | None = None,
) -> None:
record = await self.lifecycle.get(run_id)
if record is None:
return
completed_at = utc_now()
evidence_run_id = None
completed = None
if evidence:
completed = await self._save_evidence(
evidence,
str(record['started_at'] or record['created_at']),
completed_at,
run_id=UUID(run_id),
expected_target=str(record['target']),
)
evidence_run_id = str(completed.run_id)
else:
completed = await self._existing_evidence(run_id, str(record['target']))
if completed is not None:
evidence_run_id = str(completed.run_id)
evidence_status = completed.status if completed is not None else None
if evidence is not None and completed is not None:
evidence_status = str(evidence.get('status', completed.status))
await self.lifecycle.fail(
run_id,
status='cancelled' if cancelled else 'failed',
completed_at=completed_at,
error=error,
log=log[-200_000:],
evidence_run_id=evidence_run_id,
evidence_status=evidence_status,
)
async def _existing_evidence(self, run_id: str, target: str) -> CompletedResult | None:
try:
completed = await self.results.load_run(UUID(run_id))
except (LookupError, ResultStoreError, ValueError):
return None
return completed if completed.target == target else None
async def _save_evidence(
self,
evidence: dict[str, Any],
fallback_started_at: str,
fallback_completed_at: str,
*,
run_id: UUID,
expected_target: str,
) -> CompletedResult:
try:
completed = _completed_result(
evidence,
run_id=run_id,
fallback_started_at=fallback_started_at,
fallback_completed_at=fallback_completed_at,
)
except (KeyError, TypeError, ValueError) as error:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f'Invalid run evidence: {error}',
) from error
if completed.target != expected_target:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail='Evidence target does not match run target',
)
await self._persist_completed(completed)
return completed
async def _persist_completed(self, completed: CompletedResult) -> None:
try:
await self.results.save_run(completed)
except DuplicateRunError:
existing = await self.results.load_run(completed.run_id)
if existing != completed:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail=f'Run evidence already exists with different contents: {completed.run_id}',
) from None
theHarvester/lib/api/run_worker.py
+353
View File
@@ -0,0 +1,353 @@
from __future__ import annotations
import asyncio
import os
import signal
import subprocess
import sys
from argparse import ArgumentParser
from pathlib import Path
from typing import TYPE_CHECKING, Any
from uuid import UUID, uuid4
from weakref import WeakKeyDictionary
import anyio
from theHarvester.lib.enumeration import (
DEFAULT_DNS_RECURSIVE_QUERY_LIMIT,
DEFAULT_DNS_RECURSIVE_RUNTIME_SECONDS,
DEFAULT_RESULT_START,
EnumerationOptions,
)
from theHarvester.lib.resolver_selection import DEFAULT_DNS_RESOLVERS
from .run_artifacts import ensure_private_directory, read_child_evidence, write_child_evidence
from .run_store import RunStore
if TYPE_CHECKING:
from collections.abc import Awaitable, Callable
_worker_task: asyncio.Task[None] | None = None
_worker_stop: asyncio.Event | None = None
_worker_wakeup: asyncio.Event | None = None
_worker_owner: str | None = None
_process_groups: WeakKeyDictionary[asyncio.subprocess.Process, int] = WeakKeyDictionary()
def worker_enabled() -> bool:
return os.getenv('THEHARVESTER_RUN_WORKER', 'enabled').casefold() != 'disabled'
def worker_available() -> bool:
return _worker_task is not None and not _worker_task.done()
async def _default_process_factory(run_id: str, database: Path, _artifact_dir_path: Path) -> asyncio.subprocess.Process:
process_options = (
{'creationflags': getattr(subprocess, 'CREATE_NEW_PROCESS_GROUP', 0)} if os.name == 'nt' else {'start_new_session': True}
)
process = await asyncio.create_subprocess_exec(
sys.executable,
'-m',
'theHarvester.lib.api.run_worker',
'--execute',
run_id,
'--database',
str(database),
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
**process_options,
)
if process.pid is not None:
_process_groups[process] = process.pid
return process
_process_factory: Callable[[str, Path, Path], Awaitable[asyncio.subprocess.Process]] = _default_process_factory
async def _process_output(process: asyncio.subprocess.Process) -> str:
async def read(stream: asyncio.StreamReader | None) -> bytes:
return await stream.read() if stream is not None else b''
stdout, stderr = await asyncio.gather(read(process.stdout), read(process.stderr))
return '\n'.join(part.decode('utf-8', errors='replace').strip() for part in (stdout, stderr) if part).strip()
async def _signal_process_tree(process: asyncio.subprocess.Process, *, force: bool) -> None:
process_group = _process_groups.get(process)
if process_group is not None and os.name != 'nt':
try:
os.killpg(process_group, signal.SIGKILL if force else signal.SIGTERM)
except ProcessLookupError:
pass
return
if process_group is not None and os.name == 'nt':
if force:
killer = await asyncio.create_subprocess_exec(
'taskkill',
'/PID',
str(process.pid),
'/T',
'/F',
stdout=asyncio.subprocess.DEVNULL,
stderr=asyncio.subprocess.DEVNULL,
)
await killer.wait()
else:
try:
process.send_signal(getattr(signal, 'CTRL_BREAK_EVENT', 1))
except ProcessLookupError:
pass
return
try:
process.kill() if force else process.terminate()
except ProcessLookupError:
pass
async def _stop_process(process: asyncio.subprocess.Process, wait_task: asyncio.Task[int]) -> None:
if process.returncode is None:
await _signal_process_tree(process, force=False)
try:
await asyncio.wait_for(asyncio.shield(wait_task), timeout=2)
except TimeoutError:
if process.returncode is None:
await _signal_process_tree(process, force=True)
await wait_task
async def _execute_claimed(store: RunStore, run: dict[str, Any], owner_id: str | None = None) -> None:
run_id = run['run_id']
artifact_dir = store.artifact_directory(run_id)
ensure_private_directory(artifact_dir)
try:
process = await _process_factory(run_id, store.database, artifact_dir)
except (OSError, RuntimeError) as error:
await store.fail(run_id, f'Could not start child process: {error}', '')
return
wait_task = asyncio.create_task(process.wait())
output_task = asyncio.create_task(_process_output(process))
deadline = asyncio.get_running_loop().time() + int(run['request']['deadline_seconds'])
next_heartbeat = 0.0
while not wait_task.done():
await asyncio.sleep(0.05)
if owner_id is not None and asyncio.get_running_loop().time() >= next_heartbeat:
if not await store.heartbeat_worker_lease(owner_id):
await _stop_process(process, wait_task)
await store.fail(run_id, 'Worker lost its execution lease', await output_task)
return
next_heartbeat = asyncio.get_running_loop().time() + 5
current = await store.get(run_id)
stopping = _worker_stop is not None and _worker_stop.is_set()
if current is not None and current['status'] == 'cancelling':
await _stop_process(process, wait_task)
evidence, evidence_error = read_child_evidence(artifact_dir, str(run['target']))
failure_message = 'Cancelled by operator' + (f'; {evidence_error}' if evidence_error else '')
await store.fail(run_id, failure_message, await output_task, cancelled=True, evidence=evidence)
return
if stopping:
await _stop_process(process, wait_task)
evidence, evidence_error = read_child_evidence(artifact_dir, str(run['target']))
failure_message = 'theHarvester stopped before child completion' + (f'; {evidence_error}' if evidence_error else '')
await store.fail(run_id, failure_message, await output_task, evidence=evidence)
return
if asyncio.get_running_loop().time() >= deadline:
await _stop_process(process, wait_task)
evidence, evidence_error = read_child_evidence(artifact_dir, str(run['target']))
failure_message = f'Run exceeded its {run["request"]["deadline_seconds"]} second deadline'
if evidence_error:
failure_message += f'; {evidence_error}'
await store.fail(
run_id,
failure_message,
await output_task,
evidence=evidence,
)
return
log = await output_task
current = await store.get(run_id)
evidence, evidence_error = read_child_evidence(artifact_dir, str(run['target']))
if evidence_error:
await store.fail(
run_id,
evidence_error,
log,
cancelled=current is not None and current['status'] == 'cancelling',
)
return
if current is not None and current['status'] == 'cancelling':
await store.finish(run_id, evidence, log)
elif process.returncode == 0 and evidence is not None:
await store.finish(run_id, evidence, log)
else:
await store.fail(
run_id, f'Child process exited with status {process.returncode} without terminal completion', log, evidence=evidence
)
async def _worker_loop(store: RunStore, owner_id: str) -> None:
assert _worker_stop is not None
assert _worker_wakeup is not None
while not _worker_stop.is_set():
if not await store.heartbeat_worker_lease(owner_id):
return
run = await store.claim_next()
if run is not None:
await _execute_claimed(store, run, owner_id)
continue
_worker_wakeup.clear()
try:
await asyncio.wait_for(_worker_wakeup.wait(), timeout=0.5)
except TimeoutError:
continue
async def _supervise_worker(store: RunStore, owner_id: str) -> None:
assert _worker_stop is not None
assert _worker_wakeup is not None
while not _worker_stop.is_set():
if await store.acquire_worker_lease(owner_id):
try:
await store.recover_orphans()
except BaseException:
await store.release_worker_lease(owner_id)
raise
await _worker_loop(store, owner_id)
return
_worker_wakeup.clear()
try:
await asyncio.wait_for(_worker_wakeup.wait(), timeout=0.5)
except TimeoutError:
continue
async def start_worker() -> None:
global _worker_owner, _worker_stop, _worker_task, _worker_wakeup
if not worker_enabled():
return
if _worker_task is not None and not _worker_task.done():
return
store = RunStore()
owner_id = str(uuid4())
_worker_owner = owner_id
_worker_stop = asyncio.Event()
_worker_wakeup = asyncio.Event()
_worker_task = asyncio.create_task(_supervise_worker(store, owner_id))
async def stop_worker() -> None:
global _worker_owner, _worker_stop, _worker_task, _worker_wakeup
task = _worker_task
owner_id = _worker_owner
try:
if task is not None:
if _worker_stop is not None:
_worker_stop.set()
if _worker_wakeup is not None:
_worker_wakeup.set()
await task
finally:
try:
if owner_id is not None:
await RunStore().release_worker_lease(owner_id)
finally:
_worker_task = None
_worker_owner = None
_worker_stop = None
_worker_wakeup = None
def wake_worker() -> None:
if _worker_wakeup is not None:
_worker_wakeup.set()
async def _child_execute(run_id: str, database: Path) -> None:
from theHarvester import __main__ as main_module
from theHarvester.lib.completed_result import CompletedResult
store = RunStore(database)
run = await store.get(run_id)
if run is None or run['status'] not in {'running', 'cancelling'}:
raise RuntimeError('theHarvester run is not executable')
request = run['request']
artifact_dir = store.artifact_directory(run_id)
ensure_private_directory(artifact_dir)
screenshot_dir = artifact_dir / 'screenshots'
if request.get('screenshot'):
ensure_private_directory(screenshot_dir)
recursive_depth = request.get('dns_recursive_depth', 0)
resolver_list = request.get('dns_resolvers', list(DEFAULT_DNS_RESOLVERS))
api_scan_wordlist = ''
if request.get('api_scan_paths'):
api_scan_wordlist_path = artifact_dir / 'api-scan-paths.txt'
await anyio.Path(api_scan_wordlist_path).write_text(
''.join(f'{path}\n' for path in request['api_scan_paths']),
encoding='utf-8',
)
api_scan_wordlist = str(api_scan_wordlist_path)
args = EnumerationOptions(
api_scan=request.get('api_scan', False),
dns_brute=request.get('dns_brute', False),
dns_lookup=request.get('dns_lookup', False),
dns_recursive_depth=recursive_depth,
dns_recursive_query_limit=request.get('dns_recursive_query_limit', DEFAULT_DNS_RECURSIVE_QUERY_LIMIT),
dns_recursive_runtime_seconds=request.get('dns_recursive_runtime_seconds', DEFAULT_DNS_RECURSIVE_RUNTIME_SECONDS),
dns_resolve=','.join(resolver_list) if request.get('dns_resolve') else '',
dns_resolvers=tuple(resolver_list),
dns_server=None,
domain=run['target'],
filename='',
limit=request['limit'],
proxies=request.get('proxies', False),
quiet=True,
screenshot=str(screenshot_dir) if request.get('screenshot') else '',
shodan=request.get('shodan', False),
source=','.join(request['sources']),
start=request.get('start', DEFAULT_RESULT_START),
take_over=request.get('takeover', False),
wordlist=api_scan_wordlist,
)
checkpoint_lock = asyncio.Lock()
async def checkpoint(evidence: CompletedResult) -> None:
async with checkpoint_lock:
write_child_evidence(artifact_dir, evidence, partial=True)
task = asyncio.create_task(
main_module.start(
args,
completed_result_checkpoint=checkpoint,
return_completed_result=True,
result_database=database,
completed_run_id=UUID(run_id),
)
)
loop = asyncio.get_running_loop()
signal_handler_installed = False
if os.name != 'nt':
try:
loop.add_signal_handler(signal.SIGTERM, task.cancel)
signal_handler_installed = True
except (NotImplementedError, RuntimeError):
pass
try:
response = await task
except asyncio.CancelledError:
return
finally:
if signal_handler_installed:
loop.remove_signal_handler(signal.SIGTERM)
evidence = response[-1]
if not isinstance(evidence, CompletedResult):
raise RuntimeError('theHarvester did not return terminal evidence')
write_child_evidence(artifact_dir, evidence, partial=False)
if __name__ == '__main__':
parser = ArgumentParser()
parser.add_argument('--execute', required=True)
parser.add_argument('--database', required=True, type=Path)
child_args = parser.parse_args()
asyncio.run(_child_execute(child_args.execute, child_args.database))
theHarvester/lib/api/runs.py
+275
View File
@@ -0,0 +1,275 @@
from __future__ import annotations
import os
import tempfile
from pathlib import Path
from typing import Annotated
import anyio
from fastapi import APIRouter, Depends, HTTPException, Query, Request, Response, status
from fastapi.responses import FileResponse
from pydantic import ValidationError
from theHarvester.lib.api.auth import get_api_key
from theHarvester.lib.source_catalog import ACTION_ACTIVITIES, SOURCE_SPECS, SourceSpec, get_source_spec, resolve_sources
from . import run_worker
from .run_evidence import parse_jsonl_import
from .run_models import (
DATABASE_IMPORT_REQUEST_OPENAPI,
EXPORT_RESPONSES,
IMPORT_REQUEST_OPENAPI,
RUN_REQUEST_OPENAPI,
ActionResponse,
DatabaseImportResponse,
RunDetail,
RunRequest,
RunSummary,
SourceCatalogResponse,
SourceResponse,
)
from .run_projection import source_spec
from .run_store import RunStore
router = APIRouter(prefix='/api/v1', tags=['Runs'])
MAX_IMPORT_BYTES = 10 * 1024 * 1024
MAX_RUN_REQUEST_BYTES = 64 * 1024
DEFAULT_MAX_DATABASE_IMPORT_BYTES = 1024 * 1024 * 1024
async def _read_limited_body(request: Request, limit: int, detail: str) -> bytes:
content_length = request.headers.get('content-length')
if content_length and content_length.isdigit() and int(content_length) > limit:
raise HTTPException(status_code=status.HTTP_413_CONTENT_TOO_LARGE, detail=detail)
body = bytearray()
async for chunk in request.stream():
if len(body) + len(chunk) > limit:
raise HTTPException(status_code=status.HTTP_413_CONTENT_TOO_LARGE, detail=detail)
body.extend(chunk)
return bytes(body)
async def _stream_limited_body(request: Request, path: Path, limit: int, detail: str) -> None:
content_length = request.headers.get('content-length')
if content_length and content_length.isdigit() and int(content_length) > limit:
raise HTTPException(status_code=status.HTTP_413_CONTENT_TOO_LARGE, detail=detail)
size = 0
async with await anyio.open_file(path, 'wb') as file:
async for chunk in request.stream():
size += len(chunk)
if size > limit:
raise HTTPException(status_code=status.HTTP_413_CONTENT_TOO_LARGE, detail=detail)
await file.write(chunk)
@router.get('/runs')
async def list_runs(
_api_key: Annotated[str, Depends(get_api_key)],
limit: Annotated[int, Query(ge=1, le=500, description='Maximum run summaries to return.')] = 100,
offset: Annotated[int, Query(ge=0, description='Number of newer run summaries to skip.')] = 0,
) -> list[RunSummary]:
return [RunSummary.model_validate(run) for run in await RunStore().list_runs(limit=limit, offset=offset)]
@router.get('/sources')
async def list_sources(_api_key: Annotated[str, Depends(get_api_key)]) -> SourceCatalogResponse:
from theHarvester.lib.core import Core
provider_aliases = {'chaos': 'projectDiscovery', 'github-code': 'github', 'pentesttools': 'pentestTools'}
api_key_fields = Core.api_key_fields()
provider_names = {provider.casefold(): provider for provider in api_key_fields}
def credentials(source: SourceSpec) -> list[str]:
provider = provider_aliases.get(source.name, provider_names.get(source.name.casefold()))
if provider is None:
return []
return [f'api-{field}' for field in api_key_fields.get(provider, ())]
return SourceCatalogResponse(
sources=[
SourceResponse(
name=source.name,
activity=source.activity,
credentials=credentials(source),
capabilities=sorted(source.capabilities),
)
for source in sorted(SOURCE_SPECS.values(), key=lambda item: item.name)
],
actions=[ActionResponse(name=name, activity=activity) for name, activity in sorted(ACTION_ACTIVITIES.items())],
)
@router.post(
'/runs',
status_code=status.HTTP_201_CREATED,
response_model_exclude_unset=True,
openapi_extra=RUN_REQUEST_OPENAPI,
)
async def create_run(
request: Request,
_api_key: Annotated[str, Depends(get_api_key)],
) -> RunDetail:
body = await _read_limited_body(request, MAX_RUN_REQUEST_BYTES, 'Run request exceeds the 64 KiB limit')
try:
run_request = RunRequest.model_validate_json(body)
except ValidationError as error:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail=error.errors(include_url=False, include_context=False),
) from error
selected_sources = resolve_sources(run_request.sources)
unsupported_sources = [source for source in selected_sources if source_spec(source) is None]
if unsupported_sources:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail=f'Unsupported sources: {", ".join(sorted(unsupported_sources))}',
)
run_request.sources = [get_source_spec(source).name for source in selected_sources]
if not run_worker.worker_enabled():
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail='theHarvester execution worker is disabled',
)
if not run_worker.worker_available():
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail='theHarvester execution worker is unavailable',
)
run = await RunStore().create(run_request)
run_worker.wake_worker()
return RunDetail.model_validate(run)
@router.post(
'/runs/import',
status_code=status.HTTP_201_CREATED,
response_model_exclude_unset=True,
openapi_extra=IMPORT_REQUEST_OPENAPI,
)
async def import_run(
request: Request,
_api_key: Annotated[str, Depends(get_api_key)],
filename: Annotated[
str,
Query(
min_length=1,
max_length=255,
description='Original .jsonl file name.',
),
],
) -> RunDetail:
safe_filename = Path(filename).name
if Path(safe_filename).suffix.casefold() != '.jsonl':
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail='Choose a .jsonl result file')
body = await _read_limited_body(request, MAX_IMPORT_BYTES, 'Result file exceeds the 10 MiB limit')
evidence = parse_jsonl_import(body)
return RunDetail.model_validate(await RunStore().import_evidence(evidence, safe_filename))
@router.post(
'/runs/import-database',
status_code=status.HTTP_201_CREATED,
openapi_extra=DATABASE_IMPORT_REQUEST_OPENAPI,
)
async def import_database(
request: Request,
_api_key: Annotated[str, Depends(get_api_key)],
filename: Annotated[
str,
Query(
min_length=1,
max_length=255,
description='Original .sqlite, .sqlite3, or .db file name.',
),
],
) -> DatabaseImportResponse:
safe_filename = Path(filename).name
if Path(safe_filename).suffix.casefold() not in {'.sqlite', '.sqlite3', '.db'}:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail='Choose a SQLite database file')
try:
maximum_size = int(os.getenv('THEHARVESTER_MAX_DATABASE_IMPORT_BYTES', DEFAULT_MAX_DATABASE_IMPORT_BYTES))
except ValueError as error:
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail='THEHARVESTER_MAX_DATABASE_IMPORT_BYTES must be an integer',
) from error
descriptor, temporary_name = tempfile.mkstemp(prefix='theharvester-import-', suffix='.sqlite')
os.close(descriptor)
temporary_path = Path(temporary_name)
await anyio.Path(temporary_path).chmod(0o600)
try:
await _stream_limited_body(
request,
temporary_path,
maximum_size,
'SQLite database exceeds the configured import limit',
)
async with await anyio.open_file(temporary_path, 'rb') as file:
header = await file.read(16)
if header != b'SQLite format 3\x00':
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail='Uploaded file is not a SQLite database')
return DatabaseImportResponse.model_validate(await RunStore().import_database(temporary_path, safe_filename))
finally:
await anyio.Path(temporary_path).unlink(missing_ok=True)
@router.get('/runs/{run_id}', response_model_exclude_unset=True)
async def get_run(run_id: str, _api_key: Annotated[str, Depends(get_api_key)]) -> RunDetail:
run = await RunStore().get(run_id)
if run is None:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='theHarvester run not found')
return RunDetail.model_validate(run)
@router.post('/runs/{run_id}/cancel', response_model_exclude_unset=True)
async def cancel_run(
run_id: str,
_api_key: Annotated[str, Depends(get_api_key)],
) -> RunDetail:
run = await RunStore().cancel(run_id)
if run is None:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='theHarvester run not found')
return RunDetail.model_validate(run)
@router.get('/runs/{run_id}/export', response_class=Response, responses=EXPORT_RESPONSES)
async def export_run(
run_id: str,
_api_key: Annotated[str, Depends(get_api_key)],
) -> Response:
store = RunStore()
try:
completed = await store.load_completed_result(run_id)
except LookupError:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='theHarvester run not found')
if completed is None:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail='No run evidence is available to export',
)
return Response(
completed.jsonl(),
media_type='application/x-ndjson',
headers={'Content-Disposition': f'attachment; filename="{completed.target}-{run_id}.jsonl"'},
)
@router.get('/runs/{run_id}/screenshots/{name}')
async def get_screenshot(
run_id: str,
name: str,
_api_key: Annotated[str, Depends(get_api_key)],
) -> FileResponse:
store = RunStore()
run = await store.get(run_id)
if run is None:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='Screenshot not found')
screenshot = next((item for item in run['screenshots'] if item['name'] == name), None)
if screenshot is None:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='Screenshot not found')
artifact_dir = store.artifact_directory(str(run['run_id']))
screenshot_dir = artifact_dir / 'screenshots'
path = screenshot_dir / screenshot['name']
if artifact_dir.is_symlink() or screenshot_dir.is_symlink() or path.is_symlink() or not path.is_file():
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail='Screenshot not found')
return FileResponse(path, media_type='image/png', filename=screenshot['name'])
theHarvester/lib/completed_result.py
+113 -55
View File
@@ -1,46 +1,62 @@
import json
from collections import Counter
from collections.abc import Iterable, Mapping
from dataclasses import dataclass
from datetime import UTC, datetime
from typing import Literal, Self, get_args
from dataclasses import dataclass, field
from datetime import datetime
from typing import Self
from uuid import UUID, uuid4
ResultKind = Literal[
'analytics',
'api-endpoint',
'asn',
'breach',
'cms',
'dns-recursive-classification',
'dns-recursive-finding',
'dns-recursive-summary',
'email',
'framework',
'hostname',
'infostealer',
'interesting-url',
'ip-address',
'language',
'linkedin-link',
'linkedin-person',
'person',
'server',
'screenshot',
'shodan',
'takeover',
'twitter-person',
'url',
'vhost',
]
ExecutionStatus = Literal['completed', 'partial', 'failed', 'rate-limited', 'skipped']
RESULT_KINDS: frozenset[str] = frozenset(get_args(ResultKind))
EXECUTION_STATUSES: frozenset[str] = frozenset(get_args(ExecutionStatus))
from theHarvester.lib.active_evidence import ActiveEvidence
from theHarvester.lib.evidence_types import (
EVIDENCE_STATUSES,
EXECUTION_STATUSES,
RESULT_KINDS,
EvidenceStatus,
ExecutionStatus,
ResultKind,
format_utc,
)
def _isoformat_utc(value: datetime) -> str:
return value.astimezone(UTC).isoformat().replace('+00:00', 'Z')
def encode_result_jsonl(
summary: Mapping[str, object],
findings: Iterable[Mapping[str, object]],
) -> str:
records = [{**summary, 'type': 'summary'}, *findings]
return ''.join(json.dumps(record, ensure_ascii=False, separators=(',', ':'), sort_keys=True) + '\n' for record in records)
def parse_result_jsonl(payload: bytes | str) -> tuple[dict[str, object], list[dict[str, object]]]:
try:
text = payload.decode('utf-8') if isinstance(payload, bytes) else payload
records = [json.loads(line) for line in text.splitlines() if line.strip()]
except (UnicodeDecodeError, json.JSONDecodeError) as error:
raise ValueError('result file is not valid JSONL') from error
if any(not isinstance(record, dict) for record in records):
raise ValueError('JSONL records must be objects')
summary = records[0] if records else None
if not summary or summary.get('type') != 'summary':
raise ValueError('JSONL must start with a summary record')
if 'schema' in summary or 'schema_version' in summary:
raise ValueError('JSONL must not contain a schema version')
findings = records[1:]
for record in findings:
sources = record.get('sources', [])
actions = record.get('actions', [])
if (
set(record) - {'type', 'value', 'sources', 'actions'}
or record.get('type') not in RESULT_KINDS
or not isinstance(record.get('value'), str)
or not record['value'].strip()
or not isinstance(sources, list)
or any(not isinstance(source, str) or not source.strip() for source in sources)
or not isinstance(actions, list)
or any(not isinstance(action, str) or not action.strip() for action in actions)
):
raise ValueError('JSONL findings must contain a known type, non-empty value, and producer names')
record['sources'] = sorted(set(sources))
record['actions'] = sorted(set(actions))
return summary, findings
@dataclass(frozen=True, order=True, slots=True)
@@ -118,6 +134,8 @@ class CompletedResult:
results: tuple[tuple[ResultKind, str], ...]
source_executions: tuple[SourceExecution, ...] = ()
observations: tuple[ResultObservation, ...] = ()
active_evidence: ActiveEvidence = field(default_factory=ActiveEvidence)
evidence_status: EvidenceStatus | None = None
def __post_init__(self) -> None:
if not self.target.strip():
@@ -148,6 +166,17 @@ class CompletedResult:
observation_counts = Counter(observation.source for observation in self.observations)
if any(execution.result_count != observation_counts[execution.source] for execution in self.source_executions):
raise ValueError('source execution result count must match its attributed observations')
if any(
(observation.kind, observation.value) not in result_set for _action, observation in self.active_evidence.observations
):
raise ValueError('every action observation must reference a completed result')
if any(
(artifact.subject_kind, artifact.subject_value) not in result_set
for _action, artifact in self.active_evidence.artifacts
):
raise ValueError('every artifact must reference a completed result')
if self.evidence_status is not None and self.evidence_status not in EVIDENCE_STATUSES:
raise ValueError('evidence status must be complete, partial, or failed')
@classmethod
def finish(
@@ -160,7 +189,10 @@ class CompletedResult:
groups: Mapping[ResultKind, Iterable[str]],
source_executions: Iterable[SourceExecution] = (),
observations: Iterable[ResultObservation] = (),
active_evidence: ActiveEvidence | None = None,
evidence_status: EvidenceStatus | None = None,
) -> Self:
completed_active_evidence = active_evidence if active_evidence is not None else ActiveEvidence()
results: set[tuple[ResultKind, str]] = set()
for kind, values in groups.items():
if kind not in RESULT_KINDS:
@@ -169,6 +201,8 @@ class CompletedResult:
if not isinstance(value, str) or not value.strip():
raise ValueError('results must contain non-empty string values')
results.add((kind, value.strip()))
for _action, observation in completed_active_evidence.observations:
results.add((observation.kind, observation.value))
return cls(
run_id=run_id or uuid4(),
target=target.strip(),
@@ -177,45 +211,69 @@ class CompletedResult:
results=tuple(sorted(results)),
source_executions=tuple(source_executions),
observations=tuple(sorted(set(observations))),
active_evidence=completed_active_evidence,
evidence_status=evidence_status,
)
def evidence_dict(self) -> dict[str, object]:
incomplete = {'partial', 'failed', 'rate-limited', 'skipped'}
status = 'complete'
if self.source_executions and all(execution.status == 'failed' for execution in self.source_executions):
status = 'failed'
elif any(execution.status in incomplete for execution in self.source_executions):
status = 'partial'
return {
'run_id': str(self.run_id),
'target': self.target,
'started_at': self.started_at.isoformat(),
'completed_at': self.completed_at.isoformat(),
'status': status,
'results': self._result_records(),
'status': self.status,
'results': self._result_records(include_actions=True),
'source_executions': [execution.to_dict() for execution in self.source_executions],
'action_executions': [execution.to_dict() for execution in self.active_evidence.executions],
'artifacts': [{'action': action, **artifact.to_dict()} for action, artifact in self.active_evidence.artifacts],
}
def jsonl(self) -> str:
counts = Counter(kind for kind, _value in self.results)
records = [
return encode_result_jsonl(
{
'completed_at': _isoformat_utc(self.completed_at),
'completed_at': format_utc(self.completed_at),
'counts': dict(sorted(counts.items())),
'evidence_status': self.status,
'result_count': len(self.results),
'run_id': str(self.run_id),
'started_at': _isoformat_utc(self.started_at),
'source_executions': [execution.to_dict() for execution in self.source_executions],
'action_executions': [execution.to_dict() for execution in self.active_evidence.executions],
'artifacts': [{'action': action, **artifact.to_dict()} for action, artifact in self.active_evidence.artifacts],
'started_at': format_utc(self.started_at),
'target': self.target,
'type': 'summary',
},
*self._result_records(),
]
return ''.join(json.dumps(record, ensure_ascii=False, separators=(',', ':'), sort_keys=True) + '\n' for record in records)
self._result_records(include_actions=True),
)
def _result_records(self) -> list[dict[str, object]]:
@property
def status(self) -> str:
incomplete = {'partial', 'failed', 'rate-limited', 'skipped'}
execution_statuses = [execution.status for execution in self.source_executions]
execution_statuses.extend(execution.status for execution in self.active_evidence.executions)
if execution_statuses and all(status == 'failed' for status in execution_statuses):
return 'failed'
if any(execution_status in incomplete for execution_status in execution_statuses):
return 'partial'
if execution_statuses:
return 'complete'
return self.evidence_status or 'complete'
def _result_records(self, *, include_actions: bool) -> list[dict[str, object]]:
sources_by_result: dict[tuple[ResultKind, str], list[str]] = {}
for observation in self.observations:
sources_by_result.setdefault((observation.kind, observation.value), []).append(observation.source)
return [
{'type': kind, 'value': value, 'sources': sources_by_result.get((kind, value), [])} for kind, value in self.results
]
actions_by_result: dict[tuple[ResultKind, str], list[str]] = {}
for action, action_observation in self.active_evidence.observations:
actions_by_result.setdefault((action_observation.kind, action_observation.value), []).append(action)
records: list[dict[str, object]] = []
for kind, value in self.results:
record: dict[str, object] = {
'type': kind,
'value': value,
'sources': sources_by_result.get((kind, value), []),
}
if include_actions and (actions := actions_by_result.get((kind, value))):
record['actions'] = actions
records.append(record)
return records
theHarvester/lib/core.py
+17 -3
View File
@@ -105,6 +105,10 @@ class Core:
keys = yaml.safe_load(Core._read_config('api-keys.yaml'))
return keys['apikeys']
@staticmethod
def api_key_fields() -> dict[str, tuple[str, ...]]:
return dict(Core._API_KEY_FIELDS)
@staticmethod
def _api_key_value(provider: str) -> Any:
provider_keys = Core.api_keys()[provider]
@@ -316,7 +320,6 @@ class Core:
'leakix',
'leaklookup',
'linkedin',
'linkedin_links',
'mojeek',
'netcraft',
'netlas',
@@ -710,6 +713,7 @@ class AsyncFetcher:
session,
url: str,
proxy: str | None = None,
include_metadata: bool = False,
) -> tuple[Any, Any] | str:
_, proxy_type = AsyncFetcher._resolve_proxy(proxy)
response = await AsyncFetcher.fetch(
@@ -717,6 +721,7 @@ class AsyncFetcher:
url=url,
proxy=proxy,
request_timeout=15,
include_metadata=include_metadata,
)
return url, response
@@ -745,13 +750,22 @@ class AsyncFetcher:
return list(
await asyncio.gather(
*[
AsyncFetcher.takeover_fetch(session, url, proxy=proxy_url)
AsyncFetcher.takeover_fetch(
session,
url,
proxy=proxy_url,
include_metadata=include_metadata,
)
for url, proxy_url in zip(urls, proxy_urls, strict=False)
]
)
)
else:
return list(await asyncio.gather(*[AsyncFetcher.takeover_fetch(session, url) for url in urls]))
return list(
await asyncio.gather(
*[AsyncFetcher.takeover_fetch(session, url, include_metadata=include_metadata) for url in urls]
)
)
if len(params) == 0:
async with aiohttp.ClientSession(headers=headers, timeout=timeout) as session:
theHarvester/lib/database.py
+615 -56
View File
@@ -5,17 +5,39 @@ import sqlite3
from collections import Counter
from collections.abc import AsyncIterator, Iterable
from contextlib import asynccontextmanager
from datetime import date
from datetime import date, timedelta
from pathlib import Path
from typing import Any, cast
from typing import TYPE_CHECKING, Any, cast
from uuid import UUID
from sqlalchemy import Date, Float, ForeignKey, ForeignKeyConstraint, Text, UniqueConstraint, event, func, select
from sqlalchemy import (
CheckConstraint,
Date,
Float,
ForeignKey,
ForeignKeyConstraint,
Text,
UniqueConstraint,
delete,
event,
func,
or_,
select,
update,
)
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from sqlalchemy.engine import URL
from sqlalchemy.exc import IntegrityError, SQLAlchemyError
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker, create_async_engine
from sqlalchemy.ext.asyncio import AsyncConnection, AsyncEngine, AsyncSession, async_sessionmaker, create_async_engine
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from theHarvester.lib.active_evidence import (
ActionExecution,
ActionObservation,
ActionYield,
ActiveEvidence,
ArtifactReference,
)
from theHarvester.lib.completed_result import (
CompletedResult,
ExecutionStatus,
@@ -25,11 +47,24 @@ from theHarvester.lib.completed_result import (
SourceYield,
)
if TYPE_CHECKING:
from theHarvester.lib.evidence_types import EvidenceStatus
logger = logging.getLogger(__name__)
SCHEMA_VERSION = 2
SCHEMA_VERSION = 7
_DEFAULT_DATABASE = Path('~/.local/share/theHarvester/stash.sqlite').expanduser()
_LEGACY_RESULT_KIND_RENAMES = {
'api-endpoint': 'url',
'api_endpoint': 'url',
'interesting-url': 'url',
'interestingurls': 'url',
'ip-address': 'ip',
'linkedin-link': 'url',
'linkedinlinks': 'url',
}
class ResultStoreError(RuntimeError):
"""The result store could not complete an operation."""
@@ -73,6 +108,7 @@ class _RunRow(_Base):
target: Mapped[str] = mapped_column(Text)
started_at: Mapped[str] = mapped_column(Text)
completed_at: Mapped[str] = mapped_column(Text)
evidence_status: Mapped[str | None] = mapped_column(Text)
class _ResultRow(_Base):
@@ -134,6 +170,67 @@ class _ResultOriginRow(_Base):
execution_position: Mapped[int] = mapped_column(primary_key=True)
class _ArtifactRow(_Base):
"""Metadata for a file created by an action and attached to one result."""
__tablename__ = 'artifacts'
__table_args__ = (
ForeignKeyConstraint(
('run_id', 'result_position'),
('results.run_id', 'results.position'),
ondelete='CASCADE',
),
ForeignKeyConstraint(
('run_id', 'execution_position'),
('executions.run_id', 'executions.position'),
ondelete='CASCADE',
),
CheckConstraint('size_bytes >= 0'),
CheckConstraint("length(sha256) = 64 AND sha256 NOT GLOB '*[^0-9a-f]*'"),
)
run_id: Mapped[str] = mapped_column(Text, primary_key=True)
position: Mapped[int] = mapped_column(primary_key=True)
result_position: Mapped[int]
execution_position: Mapped[int]
kind: Mapped[str] = mapped_column(Text)
path: Mapped[str] = mapped_column(Text)
media_type: Mapped[str] = mapped_column(Text)
size_bytes: Mapped[int]
sha256: Mapped[str] = mapped_column(Text)
created_at: Mapped[str] = mapped_column(Text)
class _RunRecordRow(_Base):
"""Lifecycle state for one API-submitted or imported run."""
__tablename__ = 'run_records'
run_id: Mapped[str] = mapped_column(Text, primary_key=True)
target: Mapped[str] = mapped_column(Text)
status: Mapped[str] = mapped_column(Text)
origin: Mapped[str] = mapped_column(Text)
created_at: Mapped[str] = mapped_column(Text)
started_at: Mapped[str | None] = mapped_column(Text)
completed_at: Mapped[str | None] = mapped_column(Text)
request_json: Mapped[str] = mapped_column(Text)
evidence_run_id: Mapped[str | None] = mapped_column(Text, ForeignKey('runs.run_id', ondelete='SET NULL'))
evidence_status: Mapped[str | None] = mapped_column(Text)
cancellation_requested_at: Mapped[str | None] = mapped_column(Text)
error: Mapped[str | None] = mapped_column(Text)
log: Mapped[str] = mapped_column(Text, default='')
class _WorkerLeaseRow(_Base):
"""The current owner of the single local API execution worker."""
__tablename__ = 'run_worker_leases'
name: Mapped[str] = mapped_column(Text, primary_key=True)
owner_id: Mapped[str] = mapped_column(Text)
heartbeat_at: Mapped[str] = mapped_column(Text)
def _configure_sqlite_connection(dbapi_connection: Any, _connection_record: Any) -> None:
dbapi_connection.isolation_level = None
cursor = dbapi_connection.cursor()
@@ -155,6 +252,102 @@ def _sqlite_engine(database: str | Path) -> AsyncEngine:
return engine
async def _canonicalize_result_kinds(connection: AsyncConnection) -> None:
"""Merge result-kind aliases without losing provenance or artifact references."""
aliases = ', '.join(f"'{kind}'" for kind in sorted(_LEGACY_RESULT_KIND_RENAMES))
run_rows = await connection.exec_driver_sql(f'SELECT DISTINCT run_id FROM results WHERE kind IN ({aliases})')
for (run_id,) in run_rows:
result_rows = list(
await connection.exec_driver_sql(
'SELECT position, kind, value FROM results WHERE run_id = ? ORDER BY position',
(run_id,),
)
)
origin_rows = list(
await connection.exec_driver_sql(
'SELECT result_position, execution_position FROM result_origins WHERE run_id = ?',
(run_id,),
)
)
artifact_rows = list(
await connection.exec_driver_sql(
'SELECT position, result_position, execution_position, kind, path, media_type, '
'size_bytes, sha256, created_at FROM artifacts WHERE run_id = ? ORDER BY position',
(run_id,),
)
)
canonical_results = sorted(
{(_LEGACY_RESULT_KIND_RENAMES.get(kind, kind), value) for _position, kind, value in result_rows}
)
new_positions = {result: position for position, result in enumerate(canonical_results)}
old_positions = {
position: new_positions[(_LEGACY_RESULT_KIND_RENAMES.get(kind, kind), value)] for position, kind, value in result_rows
}
canonical_origins = sorted(
{(old_positions[result_position], execution_position) for result_position, execution_position in origin_rows}
)
await connection.exec_driver_sql('DELETE FROM artifacts WHERE run_id = ?', (run_id,))
await connection.exec_driver_sql('DELETE FROM result_origins WHERE run_id = ?', (run_id,))
await connection.exec_driver_sql('DELETE FROM results WHERE run_id = ?', (run_id,))
if canonical_results:
await connection.exec_driver_sql(
'INSERT INTO results (run_id, position, kind, value) VALUES (?, ?, ?, ?)',
[(run_id, position, kind, value) for position, (kind, value) in enumerate(canonical_results)],
)
if canonical_origins:
await connection.exec_driver_sql(
'INSERT INTO result_origins (run_id, result_position, execution_position) VALUES (?, ?, ?)',
[(run_id, result_position, execution_position) for result_position, execution_position in canonical_origins],
)
if artifact_rows:
await connection.exec_driver_sql(
'INSERT INTO artifacts '
'(run_id, position, result_position, execution_position, kind, path, media_type, size_bytes, sha256, created_at) '
'VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
[
(
run_id,
position,
old_positions[result_position],
execution_position,
kind,
path,
media_type,
size_bytes,
sha256,
created_at,
)
for (
position,
result_position,
execution_position,
kind,
path,
media_type,
size_bytes,
sha256,
created_at,
) in artifact_rows
],
)
await connection.exec_driver_sql(
'UPDATE executions SET result_count = ('
'SELECT COUNT(*) FROM result_origins '
'WHERE result_origins.run_id = executions.run_id '
'AND result_origins.execution_position = executions.position'
') WHERE run_id = ?',
(run_id,),
)
for alias, canonical in _LEGACY_RESULT_KIND_RENAMES.items():
await connection.exec_driver_sql(
'UPDATE legacy_observations SET kind = ? WHERE kind = ?',
(canonical, alias),
)
class _SQLiteDatabase:
def __init__(self, database: str | Path) -> None:
self.database = str(Path(database).expanduser().resolve())
@@ -205,20 +398,21 @@ class _SQLiteDatabase:
if 'discovery_observations' in tables and 'legacy_observations' not in tables:
await connection.exec_driver_sql('ALTER TABLE discovery_observations RENAME TO legacy_observations')
await connection.run_sync(_Base.metadata.create_all)
run_column_rows = await connection.exec_driver_sql('PRAGMA table_info(runs)')
if 'evidence_status' not in {row[1] for row in run_column_rows}:
await connection.exec_driver_sql('ALTER TABLE runs ADD COLUMN evidence_status TEXT')
if has_legacy_results:
await connection.exec_driver_sql(
'INSERT INTO legacy_observations (domain, resource, kind, discovered_on, source) '
'SELECT domain, resource, CASE type '
"WHEN 'host' THEN 'hostname' "
"WHEN 'ip' THEN 'ip-address' "
"WHEN 'people' THEN 'person' "
"WHEN 'linkedinlinks' THEN 'linkedin-link' "
"WHEN 'interestingurls' THEN 'interesting-url' "
"WHEN 'asns' THEN 'asn' "
"WHEN 'api_endpoint' THEN 'api-endpoint' "
'ELSE type END, find_date, source FROM legacy_results'
)
await connection.exec_driver_sql('DROP TABLE legacy_results')
if schema_version < SCHEMA_VERSION:
await _canonicalize_result_kinds(connection)
await connection.exec_driver_sql(f'PRAGMA user_version = {SCHEMA_VERSION}')
await connection.commit()
except BaseException:
@@ -251,6 +445,240 @@ def _database_for(database: str | Path) -> _SQLiteDatabase:
return _databases[path]
def _row_count(result: Any) -> int:
return int(result.rowcount)
class RunLifecycleStore:
"""Persist API run state in the same SQLite database as terminal evidence."""
def __init__(self, database: str | Path | None = None) -> None:
self.database = str(Path(database or _DEFAULT_DATABASE).expanduser().resolve())
async def initialize(self) -> None:
Path(self.database).parent.mkdir(parents=True, exist_ok=True)
await _database_for(self.database).initialize()
async def create(
self,
*,
run_id: str,
target: str,
status: str,
origin: str,
created_at: str,
request_json: str,
started_at: str | None = None,
completed_at: str | None = None,
evidence_run_id: str | None = None,
evidence_status: str | None = None,
) -> None:
async with self._session() as session:
session.add(
_RunRecordRow(
run_id=run_id,
target=target,
status=status,
origin=origin,
created_at=created_at,
started_at=started_at,
completed_at=completed_at,
request_json=request_json,
evidence_run_id=evidence_run_id,
evidence_status=evidence_status,
cancellation_requested_at=None,
error=None,
log='',
)
)
await session.commit()
async def list_records(self, *, limit: int = 100, offset: int = 0) -> list[dict[str, object]]:
async with self._session() as session:
result_count = (
select(func.count(_ResultRow.position))
.where(_ResultRow.run_id == _RunRecordRow.evidence_run_id)
.correlate(_RunRecordRow)
.scalar_subquery()
)
rows = (
await session.execute(
select(_RunRecordRow, result_count.label('result_count'))
.order_by(_RunRecordRow.created_at.desc(), _RunRecordRow.run_id.desc())
.limit(limit)
.offset(offset)
)
).all()
return [self._record(row, result_count=count) for row, count in rows]
async def get(self, run_id: str) -> dict[str, object] | None:
async with self._session() as session:
row = await session.get(_RunRecordRow, run_id)
return self._record(row) if row is not None else None
async def cancel(self, run_id: str, requested_at: str) -> dict[str, object] | None:
async with self._session() as session:
queued = await session.execute(
update(_RunRecordRow)
.where(_RunRecordRow.run_id == run_id, _RunRecordRow.status == 'queued')
.values(status='cancelled', cancellation_requested_at=requested_at, completed_at=requested_at)
)
running = None
if _row_count(queued) != 1:
running = await session.execute(
update(_RunRecordRow)
.where(_RunRecordRow.run_id == run_id, _RunRecordRow.status == 'running')
.values(status='cancelling', cancellation_requested_at=requested_at)
)
await session.commit()
row = await self.get(run_id)
if row is None:
return None
if (
_row_count(queued) == 1
or (running is not None and _row_count(running) == 1)
or row['status'] in {'cancelling', 'cancelled'}
):
return row
raise ValueError(row['status'])
async def claim_next(self, started_at: str) -> dict[str, object] | None:
async with self._session() as session:
candidate = (
select(_RunRecordRow.run_id)
.where(_RunRecordRow.status == 'queued')
.order_by(_RunRecordRow.created_at)
.limit(1)
.scalar_subquery()
)
result = await session.execute(
update(_RunRecordRow)
.where(_RunRecordRow.run_id == candidate, _RunRecordRow.status == 'queued')
.values(status='running', started_at=started_at)
.returning(_RunRecordRow.run_id)
)
run_id = result.scalar_one_or_none()
if run_id is None:
await session.rollback()
return None
await session.commit()
return await self.get(run_id)
async def finish(
self,
run_id: str,
*,
completed_at: str,
evidence_run_id: str | None,
evidence_status: str | None,
log: str,
) -> None:
async with self._session() as session:
await session.execute(
update(_RunRecordRow)
.where(_RunRecordRow.run_id == run_id, _RunRecordRow.status.in_({'running', 'cancelling'}))
.values(
status=func.iif(_RunRecordRow.status == 'cancelling', 'cancelled', 'completed'),
completed_at=completed_at,
evidence_run_id=func.coalesce(evidence_run_id, _RunRecordRow.evidence_run_id),
evidence_status=func.coalesce(evidence_status, _RunRecordRow.evidence_status),
log=log,
)
)
await session.commit()
async def fail(
self,
run_id: str,
*,
status: str,
completed_at: str,
error: str,
log: str,
evidence_run_id: str | None,
evidence_status: str | None,
) -> None:
async with self._session() as session:
await session.execute(
update(_RunRecordRow)
.where(_RunRecordRow.run_id == run_id)
.values(
status=status,
completed_at=completed_at,
error=error,
log=log,
evidence_run_id=func.coalesce(evidence_run_id, _RunRecordRow.evidence_run_id),
evidence_status=func.coalesce(evidence_status, _RunRecordRow.evidence_status),
)
)
await session.commit()
async def running(self) -> list[dict[str, object]]:
async with self._session() as session:
rows = (await session.scalars(select(_RunRecordRow).where(_RunRecordRow.status.in_({'running', 'cancelling'})))).all()
return [self._record(row) for row in rows]
async def acquire_lease(self, owner_id: str, now: str, timeout_seconds: int) -> bool:
async with self._session() as session:
stale_before = (datetime.datetime.fromisoformat(now) - timedelta(seconds=timeout_seconds)).isoformat()
statement = sqlite_insert(_WorkerLeaseRow).values(name='executor', owner_id=owner_id, heartbeat_at=now)
statement = statement.on_conflict_do_update(
index_elements=[_WorkerLeaseRow.name],
set_={'owner_id': owner_id, 'heartbeat_at': now},
where=or_(_WorkerLeaseRow.owner_id == owner_id, _WorkerLeaseRow.heartbeat_at < stale_before),
)
result = await session.execute(statement)
await session.commit()
return _row_count(result) == 1
async def heartbeat_lease(self, owner_id: str, now: str) -> bool:
async with self._session() as session:
result = await session.execute(
update(_WorkerLeaseRow)
.where(_WorkerLeaseRow.name == 'executor', _WorkerLeaseRow.owner_id == owner_id)
.values(heartbeat_at=now)
)
await session.commit()
return _row_count(result) == 1
async def release_lease(self, owner_id: str) -> None:
async with self._session() as session:
await session.execute(
delete(_WorkerLeaseRow).where(
_WorkerLeaseRow.name == 'executor',
_WorkerLeaseRow.owner_id == owner_id,
)
)
await session.commit()
@asynccontextmanager
async def _session(self) -> AsyncIterator[AsyncSession]:
await self.initialize()
async with _database_for(self.database).session() as session:
yield session
@staticmethod
def _record(row: _RunRecordRow, *, result_count: int | None = None) -> dict[str, object]:
record: dict[str, object] = {
'run_id': row.run_id,
'target': row.target,
'status': row.status,
'origin': row.origin,
'created_at': row.created_at,
'started_at': row.started_at,
'completed_at': row.completed_at,
'request_json': row.request_json,
'evidence_run_id': row.evidence_run_id,
'evidence_status': row.evidence_status,
'cancellation_requested_at': row.cancellation_requested_at,
'error': row.error,
'log': row.log,
}
if result_count is not None:
record['result_count'] = int(result_count)
return record
class ResultStore:
"""Persist enumeration results without exposing SQLAlchemy to callers."""
@@ -274,6 +702,7 @@ class ResultStore:
target=result.target,
started_at=result.started_at.isoformat(),
completed_at=result.completed_at.isoformat(),
evidence_status=result.evidence_status,
)
)
await session.flush()
@@ -281,30 +710,58 @@ class ResultStore:
_ResultRow(run_id=run_id, position=position, kind=kind, value=value)
for position, (kind, value) in enumerate(result.results)
)
producers: list[tuple[str, str, SourceExecution | ActionExecution]] = [
('source', execution.source, execution) for execution in result.source_executions
]
producers.extend(('action', execution.action, execution) for execution in result.active_evidence.executions)
session.add_all(
_ExecutionRow(
run_id=run_id,
position=position,
producer_kind='source',
name=execution.source,
producer_kind=producer_kind,
name=name,
status=execution.status,
duration_ms=execution.duration_ms,
result_count=execution.result_count,
error_type=execution.error_type,
stop_reason=execution.stop_reason,
)
for position, execution in enumerate(result.source_executions)
for position, (producer_kind, name, execution) in enumerate(producers)
)
await session.flush()
result_positions = {item: position for position, item in enumerate(result.results)}
execution_positions = {execution.source: position for position, execution in enumerate(result.source_executions)}
execution_positions = {
(producer_kind, name): position for position, (producer_kind, name, _execution) in enumerate(producers)
}
origins: list[tuple[str, str, ResultKind, str]] = [
('source', observation.source, observation.kind, observation.value) for observation in result.observations
]
origins.extend(
('action', action, observation.kind, observation.value)
for action, observation in result.active_evidence.observations
)
session.add_all(
_ResultOriginRow(
run_id=run_id,
result_position=result_positions[(observation.kind, observation.value)],
execution_position=execution_positions[observation.source],
result_position=result_positions[(kind, value)],
execution_position=execution_positions[(producer_kind, name)],
)
for observation in result.observations
for producer_kind, name, kind, value in origins
)
session.add_all(
_ArtifactRow(
run_id=run_id,
position=position,
result_position=result_positions[(artifact.subject_kind, artifact.subject_value)],
execution_position=execution_positions[('action', action)],
kind=artifact.kind,
path=artifact.path,
media_type=artifact.media_type,
size_bytes=artifact.size_bytes,
sha256=artifact.sha256,
created_at=artifact.created_at.isoformat(),
)
for position, (action, artifact) in enumerate(result.active_evidence.artifacts)
)
await session.commit()
except IntegrityError as error:
@@ -326,14 +783,76 @@ class ResultStore:
).all()
execution_rows = (
await session.scalars(
select(_ExecutionRow)
.where(_ExecutionRow.run_id == str(run_id), _ExecutionRow.producer_kind == 'source')
.order_by(_ExecutionRow.position)
select(_ExecutionRow).where(_ExecutionRow.run_id == str(run_id)).order_by(_ExecutionRow.position)
)
).all()
origin_rows = (await session.scalars(select(_ResultOriginRow).where(_ResultOriginRow.run_id == str(run_id)))).all()
artifact_rows = (
await session.scalars(
select(_ArtifactRow).where(_ArtifactRow.run_id == str(run_id)).order_by(_ArtifactRow.position)
)
).all()
results_by_position = {row.position: row for row in rows}
executions_by_position = {row.position: row for row in execution_rows}
unknown_producer_kinds = {row.producer_kind for row in execution_rows} - {'source', 'action'}
if unknown_producer_kinds:
raise ResultStoreError(f'Unknown persisted producer kind: {sorted(unknown_producer_kinds)[0]}')
observations_by_execution: dict[int, list[ActionObservation]] = {}
source_observations: list[ResultObservation] = []
for origin in origin_rows:
execution = executions_by_position[origin.execution_position]
stored_result = results_by_position[origin.result_position]
if execution.producer_kind == 'source':
source_observations.append(
ResultObservation(
source=execution.name,
kind=cast('ResultKind', stored_result.kind),
value=stored_result.value,
)
)
else:
observations_by_execution.setdefault(execution.position, []).append(
ActionObservation(
kind=cast('ResultKind', stored_result.kind),
value=stored_result.value,
)
)
artifacts_by_execution: dict[int, list[ArtifactReference]] = {}
for artifact in artifact_rows:
execution = executions_by_position[artifact.execution_position]
if execution.producer_kind != 'action':
raise ResultStoreError('Persisted artifact must reference an action execution')
subject = results_by_position[artifact.result_position]
artifacts_by_execution.setdefault(execution.position, []).append(
ArtifactReference(
kind=artifact.kind,
subject_kind=cast('ResultKind', subject.kind),
subject_value=subject.value,
path=artifact.path,
media_type=artifact.media_type,
size_bytes=artifact.size_bytes,
sha256=artifact.sha256,
created_at=datetime.datetime.fromisoformat(artifact.created_at),
)
)
action_executions: list[ActionExecution] = []
for row in execution_rows:
if row.producer_kind != 'action':
continue
action_observations = tuple(sorted(observations_by_execution.get(row.position, [])))
if row.result_count != len(action_observations):
raise ResultStoreError(f'Persisted action result count does not match origins: {row.name}')
action_executions.append(
ActionExecution(
action=row.name,
status=cast('ExecutionStatus', row.status),
duration_ms=row.duration_ms,
observations=action_observations,
artifacts=tuple(sorted(artifacts_by_execution.get(row.position, []))),
error_type=row.error_type,
stop_reason=row.stop_reason,
)
)
return CompletedResult(
run_id=UUID(parent.run_id),
target=parent.target,
@@ -350,30 +869,25 @@ class ResultStore:
stop_reason=row.stop_reason,
)
for row in execution_rows
if row.producer_kind == 'source'
),
observations=tuple(
sorted(
ResultObservation(
source=executions_by_position[row.execution_position].name,
kind=cast('ResultKind', results_by_position[row.result_position].kind),
value=results_by_position[row.result_position].value,
)
for row in origin_rows
)
),
observations=tuple(sorted(source_observations)),
active_evidence=ActiveEvidence(executions=tuple(action_executions)),
evidence_status=cast('EvidenceStatus', parent.evidence_status) if parent.evidence_status is not None else None,
)
async def list_runs(self, *, limit: int = 50) -> list[dict[str, object]]:
async def list_runs(self, *, limit: int | None = 50, offset: int = 0) -> list[dict[str, object]]:
async with self._session() as session:
rows = (
await session.execute(
select(_RunRow, func.count(_ResultRow.position))
.outerjoin(_ResultRow, _ResultRow.run_id == _RunRow.run_id)
.group_by(_RunRow.run_id)
.order_by(func.julianday(_RunRow.completed_at).desc(), _RunRow.run_id.desc())
.limit(limit)
)
).all()
statement = (
select(_RunRow, func.count(_ResultRow.position))
.outerjoin(_ResultRow, _ResultRow.run_id == _RunRow.run_id)
.group_by(_RunRow.run_id)
.order_by(func.julianday(_RunRow.completed_at).desc(), _RunRow.run_id.desc())
)
if limit is not None:
statement = statement.limit(limit)
statement = statement.offset(offset)
rows = (await session.execute(statement)).all()
return [
{
'run_id': run.run_id,
@@ -385,6 +899,32 @@ class ResultStore:
for run, result_count in rows
]
async def validate_import_database(self) -> None:
engine = create_async_engine(URL.create('sqlite+aiosqlite', database=self.database))
try:
async with engine.connect() as connection:
quick_check = await connection.exec_driver_sql('PRAGMA quick_check')
if quick_check.scalar_one() != 'ok':
raise ResultStoreError('SQLite integrity check failed')
version = (await connection.exec_driver_sql('PRAGMA user_version')).scalar_one()
if version > SCHEMA_VERSION:
raise ResultStoreError(f'Database schema version {version} is newer than supported version {SCHEMA_VERSION}')
table_rows = await connection.exec_driver_sql("SELECT name FROM sqlite_master WHERE type = 'table'")
tables = {str(row[0]) for row in table_rows}
current_schema = {'runs', 'results'}.issubset(tables)
released_schema = {'completed_results', 'completed_result_items'}.issubset(tables)
if not current_schema and not released_schema:
raise ResultStoreError('SQLite database does not contain theHarvester completed runs')
except SQLAlchemyError as error:
raise ResultStoreError('Could not validate SQLite database') from error
finally:
await engine.dispose()
async def dispose(self) -> None:
database = _databases.pop(self.database, None)
if database is not None:
await database.dispose()
async def record_observations(
self,
target: str,
@@ -414,40 +954,59 @@ class ResultStore:
A result is unique when one source reported it and shared when more than one
source reported it. Sources that ran without results still appear with zero counts.
"""
yields = await self._producer_yields(run_id, 'source')
return [
SourceYield(
source=name,
observed_result_count=observed,
unique_result_count=unique,
shared_result_count=shared,
)
for name, observed, unique, shared in yields
]
async def action_yields(self, run_id: UUID) -> list[ActionYield]:
yields = await self._producer_yields(run_id, 'action')
return [
ActionYield(
action=name,
observed_result_count=observed,
unique_result_count=unique,
shared_result_count=shared,
)
for name, observed, unique, shared in yields
]
async def _producer_yields(self, run_id: UUID, producer_kind: str) -> list[tuple[str, int, int, int]]:
async with self._session() as session:
execution_rows = (
await session.scalars(
select(_ExecutionRow).where(
_ExecutionRow.run_id == str(run_id),
_ExecutionRow.producer_kind == 'source',
_ExecutionRow.producer_kind == producer_kind,
)
)
).all()
result_rows = (await session.scalars(select(_ResultRow).where(_ResultRow.run_id == str(run_id)))).all()
origin_rows = (await session.scalars(select(_ResultOriginRow).where(_ResultOriginRow.run_id == str(run_id)))).all()
source_by_position = {row.position: row.name for row in execution_rows}
producer_by_position = {row.position: row.name for row in execution_rows}
result_by_position = {row.position: (row.kind, row.value) for row in result_rows}
sources_by_result: dict[tuple[str, str], set[str]] = {}
producers_by_result: dict[tuple[str, str], set[str]] = {}
for origin in origin_rows:
source = source_by_position.get(origin.execution_position)
producer = producer_by_position.get(origin.execution_position)
result = result_by_position.get(origin.result_position)
if source is not None and result is not None:
sources_by_result.setdefault(result, set()).add(source)
if producer is not None and result is not None:
producers_by_result.setdefault(result, set()).add(producer)
observed_counts: Counter[str] = Counter()
unique_counts: Counter[str] = Counter()
shared_counts: Counter[str] = Counter()
for sources in sources_by_result.values():
for source in sources:
observed_counts[source] += 1
(unique_counts if len(sources) == 1 else shared_counts)[source] += 1
for producers in producers_by_result.values():
for producer in producers:
observed_counts[producer] += 1
(unique_counts if len(producers) == 1 else shared_counts)[producer] += 1
return [
SourceYield(
source=source,
observed_result_count=observed_counts[source],
unique_result_count=unique_counts[source],
shared_result_count=shared_counts[source],
)
for source in sorted(source_by_position.values())
(name, observed_counts[name], unique_counts[name], shared_counts[name])
for name in sorted(producer_by_position.values())
]
@asynccontextmanager
theHarvester/lib/enumeration.py
+4
View File
@@ -25,6 +25,8 @@ class EnumerationOptions:
dns_server: str | None = None
take_over: bool = False
dns_resolve: str | None = ''
dns_resolvers: tuple[str, ...] = ()
dns_resolver_input: str = ''
dns_lookup: bool = False
dns_brute: bool = False
dns_recursive_depth: int = 0
@@ -49,6 +51,8 @@ class EnumerationOptions:
dns_server=getattr(value, 'dns_server', None),
take_over=getattr(value, 'take_over', False),
dns_resolve=getattr(value, 'dns_resolve', ''),
dns_resolvers=tuple(getattr(value, 'dns_resolvers', ())),
dns_resolver_input=getattr(value, 'dns_resolver_input', ''),
dns_lookup=getattr(value, 'dns_lookup', False),
dns_brute=getattr(value, 'dns_brute', False),
dns_recursive_depth=getattr(value, 'dns_recursive_depth', 0),
theHarvester/lib/evidence_types.py
+37
View File
@@ -0,0 +1,37 @@
from datetime import UTC, datetime
from typing import Literal, get_args
ResultKind = Literal[
'analytics',
'asn',
'breach',
'cms',
'dns-recursive-classification',
'dns-recursive-finding',
'dns-recursive-summary',
'email',
'framework',
'hostname',
'infostealer',
'ip',
'language',
'linkedin-person',
'person',
'server',
'screenshot',
'shodan',
'takeover',
'twitter-person',
'url',
'vhost',
]
ExecutionStatus = Literal['completed', 'partial', 'failed', 'rate-limited', 'skipped']
EvidenceStatus = Literal['complete', 'partial', 'failed']
RESULT_KINDS: frozenset[str] = frozenset(get_args(ResultKind))
EXECUTION_STATUSES: frozenset[str] = frozenset(get_args(ExecutionStatus))
EVIDENCE_STATUSES: frozenset[str] = frozenset(get_args(EvidenceStatus))
def format_utc(value: datetime) -> str:
return value.astimezone(UTC).isoformat().replace('+00:00', 'Z')
theHarvester/lib/hostchecker.py
+14 -2
View File
@@ -28,6 +28,14 @@ class HostDnsRecords:
return self.ipv4 + self.ipv6
def is_expected_dns_absence(error: BaseException) -> bool:
return (
isinstance(error, aiodns.error.DNSError)
and bool(error.args)
and error.args[0] in {aiodns.error.ARES_ENODATA, aiodns.error.ARES_ENOTFOUND}
)
class Checker:
"""Resolve hosts while preserving the legacy ``check()`` return tuple.
@@ -41,6 +49,8 @@ class Checker:
self.addresses: set[str] = set()
self.records: dict[str, HostDnsRecords] = {}
self.nameservers: list[str] = nameservers
self.query_error_count = 0
self.query_error_types: set[str] = set()
# @staticmethod
# async def query(host, resolver) -> Tuple[str, Any]:
@@ -54,8 +64,7 @@ class Checker:
# except Exception:
# return f"{host}", tuple()
@staticmethod
async def resolve_host(host: str, resolver: aiodns.DNSResolver) -> tuple[str, HostDnsRecords] | None:
async def resolve_host(self, host: str, resolver: aiodns.DNSResolver) -> tuple[str, HostDnsRecords] | None:
record_types = ('A', 'AAAA', 'CNAME')
results = await asyncio.gather(
*(resolver.query_dns(host, record_type) for record_type in record_types),
@@ -65,6 +74,9 @@ class Checker:
for record_type, result in zip(record_types, results, strict=True):
if isinstance(result, BaseException):
if isinstance(result, Exception):
if not is_expected_dns_absence(result):
self.query_error_count += 1
self.query_error_types.add(type(result).__name__)
continue
raise result
for record in result.answer:
theHarvester/lib/ip-ranges.json
-8978
View File
File diff suppressed because it is too large Load Diff
theHarvester/lib/output.py
+1 -9
View File
@@ -59,9 +59,7 @@ def print_section(header: str, items: Iterable[str], separator: str) -> None:
output_logger.info(item)
def print_linkedin_sections(
engines: Sequence[str], people: Sequence[str], links: Sequence[str], separator: str = '---------------------'
) -> None:
def print_linkedin_people(engines: Sequence[str], people: Sequence[str], separator: str = '---------------------') -> None:
if len(people) == 0 and 'linkedin' in engines:
output_logger.info('\n[*] No LinkedIn users found.\n\n')
elif len(people) >= 1:
@@ -69,9 +67,3 @@ def print_linkedin_sections(
output_logger.info(separator)
for usr in sorted_unique(people):
output_logger.info(usr)
if 'linkedin' in engines or 'rocketreach' in engines:
output_logger.info(f'\n[*] LinkedIn Links found: {len(links)}')
output_logger.info(separator)
for link in sorted_unique(links):
output_logger.info(link)
theHarvester/lib/resolver_selection.py
+24
View File
@@ -0,0 +1,24 @@
from collections.abc import Iterable
from ipaddress import ip_address
DEFAULT_DNS_RESOLVERS = ('1.1.1.1', '8.8.8.8', '9.9.9.9')
def normalize_resolver_addresses(values: Iterable[str]) -> list[str]:
"""Return distinct canonical resolver IP addresses in operator order."""
addresses: list[str] = []
seen: set[str] = set()
for raw_value in values:
value = raw_value.strip()
if not value:
continue
try:
address = str(ip_address(value))
except ValueError as error:
raise ValueError(f'Invalid DNS resolver address: {value}') from error
if address not in seen:
addresses.append(address)
seen.add(address)
if not addresses:
raise ValueError('Provide at least one DNS resolver IP address')
return addresses
theHarvester/lib/resolvers.txt
-2016
View File
File diff suppressed because it is too large Load Diff
theHarvester/lib/source_catalog.py
+35 -12
View File
@@ -1,4 +1,4 @@
from collections.abc import Iterable
from collections.abc import Iterable, Mapping
from dataclasses import dataclass
from enum import Enum, StrEnum, auto
from typing import Final
@@ -13,12 +13,25 @@ class ActivityClass(StrEnum):
ACTION_ACTIVITIES: Final = {
'dns-brute': ActivityClass.DNS,
'dns-lookup': ActivityClass.DNS,
'dns-recursive': ActivityClass.DNS,
'dns-resolve': ActivityClass.DNS,
'shodan': ActivityClass.PASSIVE,
'api-scan': ActivityClass.DIRECT,
'screenshot': ActivityClass.DIRECT,
'take-over': ActivityClass.DIRECT,
'takeover': ActivityClass.DIRECT,
}
ACTION_REQUEST_FIELDS: Final = {
**{name: name.replace('-', '_') for name in ACTION_ACTIVITIES},
'dns-recursive': 'dns_recursive_depth',
}
def selected_action_names(request: Mapping[str, object]) -> tuple[str, ...]:
def selected(name: str) -> bool:
value = request.get(ACTION_REQUEST_FIELDS[name])
return isinstance(value, (int, float)) and value > 0 if name == 'dns-recursive' else bool(value)
return tuple(name for name in ACTION_ACTIVITIES if selected(name))
class ResultRoute(Enum):
@@ -34,9 +47,7 @@ class ResultRoute(Enum):
IPS = auto()
ASNS = auto()
PEOPLE = auto()
LINKS = auto()
URLS = auto()
INTERESTING_URLS = auto()
BREACHES = auto()
@@ -46,9 +57,7 @@ _ROUTE_CAPABILITIES = {
ResultRoute.IPS: 'ips',
ResultRoute.ASNS: 'asns',
ResultRoute.PEOPLE: 'people',
ResultRoute.LINKS: 'urls',
ResultRoute.URLS: 'urls',
ResultRoute.INTERESTING_URLS: 'urls',
ResultRoute.BREACHES: 'breaches',
}
RESULT_CAPABILITIES = frozenset(_ROUTE_CAPABILITIES.values())
@@ -80,10 +89,10 @@ def _spec(
_SPECS = (
_spec('arquivo', ResultRoute.SUBDOMAINS),
_spec('baidu', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS),
_spec('bevigil', ResultRoute.SUBDOMAINS, ResultRoute.INTERESTING_URLS),
_spec('bevigil', ResultRoute.SUBDOMAINS, ResultRoute.URLS),
_spec('brave', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS),
_spec('bufferoverun', ResultRoute.SUBDOMAINS, ResultRoute.IPS),
_spec('builtwith', ResultRoute.SUBDOMAINS, ResultRoute.INTERESTING_URLS),
_spec('builtwith', ResultRoute.SUBDOMAINS, ResultRoute.URLS),
_spec('censys', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS),
_spec('certspotter', ResultRoute.SUBDOMAINS),
_spec('chaos', ResultRoute.SUBDOMAINS),
@@ -111,7 +120,7 @@ _SPECS = (
_spec('hudsonrock', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS, ResultRoute.IPS),
_spec('hunter', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS),
_spec('hunterhow', ResultRoute.SUBDOMAINS),
_spec('intelx', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS, ResultRoute.INTERESTING_URLS),
_spec('intelx', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS, ResultRoute.URLS),
_spec('leakix', ResultRoute.SUBDOMAINS),
_spec('leaklookup', ResultRoute.EMAILS, ResultRoute.BREACHES),
_spec('mojeek', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS),
@@ -122,7 +131,7 @@ _SPECS = (
_spec('projectdiscovery', ResultRoute.SUBDOMAINS),
_spec('rapiddns', ResultRoute.SUBDOMAINS, ResultRoute.IPS),
_spec('robtex', ResultRoute.SUBDOMAINS, ResultRoute.IPS),
_spec('rocketreach', ResultRoute.EMAILS, ResultRoute.LINKS),
_spec('rocketreach', ResultRoute.EMAILS, ResultRoute.URLS),
_spec('securityTrails', ResultRoute.SUBDOMAINS, ResultRoute.IPS),
_spec('securityscorecard', ResultRoute.SUBDOMAINS, ResultRoute.IPS),
_spec('sherlockeye', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS, ResultRoute.IPS),
@@ -138,7 +147,7 @@ _SPECS = (
_spec('subdomainfinderc99', ResultRoute.SUBDOMAINS, activity=ActivityClass.DNS),
_spec('thc', ResultRoute.SUBDOMAINS),
_spec('tomba', ResultRoute.SUBDOMAINS, ResultRoute.EMAILS),
_spec('urlscan', ResultRoute.SUBDOMAINS, ResultRoute.IPS, ResultRoute.ASNS, ResultRoute.INTERESTING_URLS),
_spec('urlscan', ResultRoute.SUBDOMAINS, ResultRoute.IPS, ResultRoute.ASNS, ResultRoute.URLS),
_spec('virustotal', ResultRoute.SUBDOMAINS),
_spec('waybackarchive', ResultRoute.SUBDOMAINS),
_spec('whoisxml', ResultRoute.SUBDOMAINS),
@@ -156,7 +165,7 @@ _SPECS = (
ResultRoute.EMAILS,
ResultRoute.IPS,
ResultRoute.ASNS,
ResultRoute.INTERESTING_URLS,
ResultRoute.URLS,
),
)
@@ -168,6 +177,20 @@ def get_source_spec(name: str) -> SourceSpec:
return _CASEFOLDED_SOURCE_SPECS[name.casefold()]
def activity_classes_for_selection(
source_names: Iterable[str],
action_names: Iterable[str] = (),
) -> tuple[ActivityClass, ...]:
selected: set[ActivityClass] = set()
for name in source_names:
try:
selected.add(get_source_spec(name).activity)
except KeyError:
continue
selected.update(ACTION_ACTIVITIES[name] for name in action_names if name in ACTION_ACTIVITIES)
return tuple(activity for activity in ActivityClass if activity in selected)
def resolve_sources(selection: str | Iterable[str]) -> list[str]:
"""Expand source and result-capability selectors into canonical source names."""
values = (selection,) if isinstance(selection, str) else selection
theHarvester/restfulHarvest.py
-10
View File
@@ -1,5 +1,4 @@
import argparse
import os
import uvicorn
@@ -32,17 +31,8 @@ def main():
help='Enable automatic reload used during development of the api',
action='store_true',
)
parser.add_argument(
'--rate-limit',
default='5/minute',
help='Set API rate limit (e.g., "10/minute", "100/hour"), default is 5/minute',
)
args: argparse.Namespace = parser.parse_args()
# Set environment variable for API rate limit
os.environ['API_RATE_LIMIT'] = args.rate_limit
uvicorn.run(
'theHarvester.lib.api.api:app',
host=args.host,
theHarvester/screenshot/screenshot.py
+28 -6
View File
@@ -8,6 +8,9 @@ import ssl
import sys
from collections.abc import Collection
from datetime import datetime
from ipaddress import ip_address
from pathlib import Path
from urllib.parse import urlsplit
import aiohttp
import certifi
@@ -17,6 +20,18 @@ from playwright.async_api import async_playwright
logger = logging.getLogger(__name__)
def _target_url(value: str, scheme: str = 'https') -> str:
if value.startswith(('http://', 'https://')):
return value
try:
address = ip_address(value)
except ValueError:
host = value
else:
host = f'[{address}]' if address.version == 6 else str(address)
return f'{scheme}://{host}'
class ScreenShotter:
def __init__(self, output) -> None:
self.output = output
@@ -57,7 +72,7 @@ class ScreenShotter:
async def visit(url: str, proxy: str | None = None) -> tuple[str, str]:
try:
timeout = aiohttp.ClientTimeout(total=35)
urls = (url,) if url.startswith(('http://', 'https://')) else (f'https://{url}', f'http://{url}')
urls = (url,) if url.startswith(('http://', 'https://')) else (_target_url(url), _target_url(url, 'http'))
sslcontext = ssl.create_default_context(cafile=certifi.where())
connector: ProxyConnector | aiohttp.TCPConnector
proxy_param = None
@@ -80,28 +95,35 @@ class ScreenShotter:
logger.info(f'An exception has occurred while attempting to visit {url} : {e}')
return '', ''
async def take_screenshot(self, url: str) -> str:
url = f'https://{url}' if not url.startswith(('http://', 'https://')) else url
async def take_screenshot(self, url: str, *, output_path: Path | None = None) -> str:
url = _target_url(url)
logger.info(f'Attempting to take a screenshot of: {url}')
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
# New browser context
context = await browser.new_context()
page = await context.new_page()
path = rf'{self.output}{self.slash}{url.replace("http://", "").replace("https://", "")}.png'
path: Path | None = output_path or self.screenshot_path(url)
date = str(datetime.now())
try:
# Will fail if network idle or load event doesn't fire after
# 35s which should be handled
await page.goto(url, timeout=35000)
await page.screenshot(path=path)
os.chmod(path, 0o600)
if path is not None:
os.chmod(path, 0o600)
except Exception as e:
logger.info(f'An exception has occurred attempting to screenshot: {url} : {e}')
path = ''
path = None
finally:
await page.close()
await context.close()
await browser.close()
logger.info(f'{date} {url} {path}')
return url if path else ''
def screenshot_path(self, url: str) -> Path:
parsed = urlsplit(_target_url(url))
hostname = (parsed.hostname or 'unknown-host').replace(':', '_')
port = f'_{parsed.port}' if parsed.port else ''
return Path(self.output) / f'{hostname}{port}.png'
uv.lock
Generated
-104
View File
@@ -484,18 +484,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" },
]
[[package]]
name = "deprecated"
version = "1.3.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "wrapt" },
]
sdist = { url = "https://files.pythonhosted.org/packages/49/85/12f0a49a7c4ffb70572b6c2ef13c90c88fd190debda93b23f026b25f9634/deprecated-1.3.1.tar.gz", hash = "sha256:b1b50e0ff0c1fddaa5708a2c6b0a6588bb09b892825ab2b214ac9ea9d92a5223", size = 2932523, upload-time = "2025-10-30T08:19:02.757Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/84/d0/205d54408c08b13550c733c4b85429e7ead111c7f0014309637425520a9a/deprecated-1.3.1-py2.py3-none-any.whl", hash = "sha256:597bfef186b6f60181535a29fbe44865ce137a5079f295b479886c82729d5f3f", size = 11298, upload-time = "2025-10-30T08:19:00.758Z" },
]
[[package]]
name = "dnspython"
version = "2.8.0"
@@ -801,20 +789,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/ce/62/b40b382fa0c66fee1478073eb8db352a4a6beda4a1adccf1df911d8c289c/librt-0.11.0-cp314-cp314t-win_arm64.whl", hash = "sha256:dee008f20b542e3cd162ba338a7f9ec0f6d23d395f66fe8aeeec3c9d067ea253", size = 102572, upload-time = "2026-05-10T18:17:06.809Z" },
]
[[package]]
name = "limits"
version = "5.8.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "deprecated" },
{ name = "packaging" },
{ name = "typing-extensions" },
]
sdist = { url = "https://files.pythonhosted.org/packages/71/69/826a5d1f45426c68d8f6539f8d275c0e4fcaa57f0c017ec3100986558a41/limits-5.8.0.tar.gz", hash = "sha256:c9e0d74aed837e8f6f50d1fcebcf5fd8130957287206bc3799adaee5092655da", size = 226104, upload-time = "2026-02-05T07:17:35.859Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/b9/98/cb5ca20618d205a09d5bec7591fbc4130369c7e6308d9a676a28ff3ab22c/limits-5.8.0-py3-none-any.whl", hash = "sha256:ae1b008a43eb43073c3c579398bd4eb4c795de60952532dc24720ab45e1ac6b8", size = 60954, upload-time = "2026-02-05T07:17:34.425Z" },
]
[[package]]
name = "lxml"
version = "6.1.1"
@@ -1589,18 +1563,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" },
]
[[package]]
name = "slowapi"
version = "0.1.10"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "limits" },
]
sdist = { url = "https://files.pythonhosted.org/packages/b9/52/24527cf25a8b508926aff53350b0136561dfe86c7125f61526653666e1b2/slowapi-0.1.10.tar.gz", hash = "sha256:d320d5bc04d9f171a77fb16700faf3036d85b00f420f22924c8a225f95bd14f9", size = 13841, upload-time = "2026-06-13T11:59:31.571Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/c4/8b/1d359f38706b4097d9a943bf8bd22599f537de4cbaff1e622d3e3936e164/slowapi-0.1.10-py3-none-any.whl", hash = "sha256:3acb61561dc9d687e3d3669362ff6a439de9ba44e2fed3a9c165da26b4b83e28", size = 14921, upload-time = "2026-06-13T11:59:30.485Z" },
]
[[package]]
name = "soupsieve"
version = "2.8.4"
@@ -1687,7 +1649,6 @@ dependencies = [
{ name = "pyyaml" },
{ name = "retrying" },
{ name = "shodan" },
{ name = "slowapi" },
{ name = "sqlalchemy" },
{ name = "ujson" },
{ name = "uvicorn" },
@@ -1732,7 +1693,6 @@ requires-dist = [
{ name = "pyyaml", specifier = "==6.0.3" },
{ name = "retrying", specifier = "==1.4.2" },
{ name = "shodan", specifier = "==1.31.0" },
{ name = "slowapi", specifier = "==0.1.10" },
{ name = "sqlalchemy", specifier = "==2.0.51" },
{ name = "ujson", specifier = "==5.13.0" },
{ name = "uvicorn", specifier = "==0.49.0" },
@@ -2010,70 +1970,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/0f/8d/52eaa9187b88596b0a8b646874cfec5a5c3fce8c52b5182be1a0253203a3/winloop-0.6.3-cp314-cp314t-win_arm64.whl", hash = "sha256:447006f38f13827ff4600e7beeda70367370cb8dab8ea84042e8fa1749f32b1c", size = 576206, upload-time = "2026-04-27T16:08:07.007Z" },
]
[[package]]
name = "wrapt"
version = "2.2.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/2d/9f/06263fcd8ad6c405f05a3905fd7a84dd3176eb5ad46e44bccc0cd16348bb/wrapt-2.2.1.tar.gz", hash = "sha256:6744f504375775d7609c82c8d3d94af1c9a6f05586984536905908ba905277b9", size = 127620, upload-time = "2026-05-22T14:49:43.056Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/89/0c/bfae7b9401583b6d05938cd16dedc43857d96da2f8a3d50d78cc515bf6ff/wrapt-2.2.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3ffad790d9d11d8ecf9f17c4bb671a5b4089e4d8b575c46c5129597f41f836b0", size = 81021, upload-time = "2026-05-22T14:48:00.313Z" },
{ url = "https://files.pythonhosted.org/packages/26/58/80f6a6599f933f4caecc1cb3ee88a04faf81e8b9bddbd6109c688dd63e0f/wrapt-2.2.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:628f5220c7a904d5fc78f7075c8d7871433eb6d035c94728a22fdf85f193d2a8", size = 81692, upload-time = "2026-05-22T14:48:01.49Z" },
{ url = "https://files.pythonhosted.org/packages/17/93/fb357cc7847c58a8ae790be718903afa81a28d23e642c843dc4129e8a0b2/wrapt-2.2.1-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:61acce4257a9883669703c525447c5b4c392edf0f987ae77ec32668440158f0e", size = 169364, upload-time = "2026-05-22T14:48:02.791Z" },
{ url = "https://files.pythonhosted.org/packages/aa/0b/76b601ee309a8bd556af0eecb184394c20b3c49aa9c8e085aa1ffacc2568/wrapt-2.2.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:727ab4244622cd6ad2390f322642090c877d2e83a608d2653a7643ae5368d926", size = 171079, upload-time = "2026-05-22T14:48:04.22Z" },
{ url = "https://files.pythonhosted.org/packages/cd/87/ee3f32d5658e3e26d3e0e457922b47a36dd3bfbdfee7f97bb3e802344a66/wrapt-2.2.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:03df9ebed4c73ab93fa8c07e3d41d818dfca1852b15731a3de59457b27814624", size = 160205, upload-time = "2026-05-22T14:48:05.553Z" },
{ url = "https://files.pythonhosted.org/packages/b1/d0/ae2fd64277a67f5d7bffcf2d05eea1e476263fb2a072baf0b0129ab85984/wrapt-2.2.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:0d9ff006f420b2ec8296aa56ade43ea7da3e997e85769f0aafc5e0661aacb710", size = 168922, upload-time = "2026-05-22T14:48:07.132Z" },
{ url = "https://files.pythonhosted.org/packages/b1/f3/2d541a060c5bbafb9400bca4917e4d78bfd1f239f404782c86831a8f6b29/wrapt-2.2.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:844c858fc3bb7eacc0ba8efa904935d16aac6a4470948ad1e7e55c9f5a2a665f", size = 158388, upload-time = "2026-05-22T14:48:08.629Z" },
{ url = "https://files.pythonhosted.org/packages/1d/68/8d92c8800c57e93cb116ae9e9d6cbafc34fade5ee9f9107b6f203fb4dc35/wrapt-2.2.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:87bacdaf225117a342a20d9c03438d701c02112f6e3f351ce9b7f32354f14797", size = 167682, upload-time = "2026-05-22T14:48:10.042Z" },
{ url = "https://files.pythonhosted.org/packages/30/72/83ea3790ea352439442349388e29ff07b76e0686265f9088bbb505d1608d/wrapt-2.2.1-cp312-cp312-win32.whl", hash = "sha256:2f8c90c8afde51969487be4e1343ae049b268854877d415c2510baf833775052", size = 77857, upload-time = "2026-05-22T14:48:11.782Z" },
{ url = "https://files.pythonhosted.org/packages/ef/cb/99450668dd3502d62a54a1c8aa56e44f34cb8c1261b381cfe2e7926c3b75/wrapt-2.2.1-cp312-cp312-win_amd64.whl", hash = "sha256:6ce32763ac31ce94fe9aada947e479b1975012bff166da409b4b9e4e376cf7e5", size = 80825, upload-time = "2026-05-22T14:48:13.046Z" },
{ url = "https://files.pythonhosted.org/packages/e6/3a/87512881be64e743f9ee4c66f4cbe8e884974bef2a5989af71f999653ac7/wrapt-2.2.1-cp312-cp312-win_arm64.whl", hash = "sha256:8d1b4d0e0c2119587a31f5c029abd547e0c81d93b89d394566fe1588659eb579", size = 79087, upload-time = "2026-05-22T14:48:14.323Z" },
{ url = "https://files.pythonhosted.org/packages/88/d1/a1b08f8f4fac8cbb156fa51cf64ee2c7f7f74f9875ba3cf70b3c58368694/wrapt-2.2.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:d2beb1c7cab10603aecdc42f8edd6ff013f9a32e4543474e38e6b77ce9975aeb", size = 80831, upload-time = "2026-05-22T14:48:15.598Z" },
{ url = "https://files.pythonhosted.org/packages/54/ce/57890814991446a845e09b3445ce8b694f27eb0577004f2c2a36a9772ed4/wrapt-2.2.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e0cb7e4dd71f4c32e5e84843cd3c4cd65dda034314004bbe1d7f99af2426ab80", size = 81375, upload-time = "2026-05-22T14:48:17.071Z" },
{ url = "https://files.pythonhosted.org/packages/38/65/08d7a6c76ac4493bdb668205ee9c1de1bd5daca61717c3e9aa49b4c01499/wrapt-2.2.1-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:95821352042722cd9f1108874579a47989d0a7e12a37d87d2fc4af20fd99ab8a", size = 167417, upload-time = "2026-05-22T14:48:18.303Z" },
{ url = "https://files.pythonhosted.org/packages/62/ce/f1ccbee7a1bfe5cdc6b3da6bab4b45713d628b9294da32a39f563d648140/wrapt-2.2.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:abd621552ede77c4c69be7fac44ba911225b0c812b6ba604e5964cf98085b474", size = 166948, upload-time = "2026-05-22T14:48:19.768Z" },
{ url = "https://files.pythonhosted.org/packages/86/2a/f85d48d1cd4869aee6704028d257d740a47c1c467b457ce396b4b5b55d07/wrapt-2.2.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e3677c7146ce694874941ba82b57092cc4875445aadf29d72807351023105143", size = 158148, upload-time = "2026-05-22T14:48:21.96Z" },
{ url = "https://files.pythonhosted.org/packages/fe/5c/93939ad11d4a12358ab1aab219a2ef5efa5612e0db6b9fc65af8af1a891b/wrapt-2.2.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:9a5934eaea872e17936b5f45501eba5ab0bce9a74122e172b663d7c28c459c4a", size = 165905, upload-time = "2026-05-22T14:48:23.373Z" },
{ url = "https://files.pythonhosted.org/packages/e0/22/b8c2aa89862ff58605934d7abf4b70e6a5a1c33df96656f49035ccdf1c8a/wrapt-2.2.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:f5b9daf6b629fce418e0cc3dd0436eac045188fa35deadb7a7f3941d5b8203f9", size = 156712, upload-time = "2026-05-22T14:48:24.767Z" },
{ url = "https://files.pythonhosted.org/packages/5d/78/bf00a7b02239c12bb02ddcc3c0b971bfcc36e578c5a44f1ccfef5b458545/wrapt-2.2.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:f53ac9f3ef573326d009ed809beff4efcac6451931c2b8132586da4b9e53ff31", size = 166560, upload-time = "2026-05-22T14:48:26.83Z" },
{ url = "https://files.pythonhosted.org/packages/fe/93/6390ca9c5b787683cef588d04f57c8d41b9a2323b5597a65f18638c90ef2/wrapt-2.2.1-cp313-cp313-win32.whl", hash = "sha256:1ffa9cfd4bdb581539951b14ae661ff20ed0c3599b3e911a131ee0ec5ac11337", size = 77817, upload-time = "2026-05-22T14:48:28.221Z" },
{ url = "https://files.pythonhosted.org/packages/97/73/ce10f0e71c0cfaa1a65faadb8efd4852028b3bb9ba28932b8889df769d38/wrapt-2.2.1-cp313-cp313-win_amd64.whl", hash = "sha256:368eac1e20fd0bb03dd3cc42bf9887154c3861b60989389ccb5fac032617d215", size = 80736, upload-time = "2026-05-22T14:48:30.139Z" },
{ url = "https://files.pythonhosted.org/packages/c7/4c/89f4a6818fafbbd840330e4fa3873073e1bfc166133a64cac7f8fde7a5e3/wrapt-2.2.1-cp313-cp313-win_arm64.whl", hash = "sha256:c754dafdf5aaf0b401b644a90a30046929a0dd1a536e0ff0ec959a59155d9c7f", size = 79099, upload-time = "2026-05-22T14:48:31.405Z" },
{ url = "https://files.pythonhosted.org/packages/bf/f2/9a8741c46f8c208ac0a45b25ba170bcb4fb72a2781d5fb97dbd7b6be73cb/wrapt-2.2.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:ed928d0fda15fc0adc8d13305c8b3c0f2fba5b0669950c9e6d019d9162a3b3e8", size = 82802, upload-time = "2026-05-22T14:48:33.307Z" },
{ url = "https://files.pythonhosted.org/packages/9c/0d/e9c855716a3705eef1416456bdf062b60620726fdc59428ff670fc3c60dc/wrapt-2.2.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:fafb4e739e43544d12cb4abd1605fd4683b6ca6a9ad682b7fd8f4d21973eafa8", size = 83329, upload-time = "2026-05-22T14:48:34.593Z" },
{ url = "https://files.pythonhosted.org/packages/3b/d6/a88f1c13112b7831adac75cea65d8310e0d696d570c8961844c90a57b865/wrapt-2.2.1-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:74d6a0c31472fe5d814917266b9f46495d7c61ed890af08b468acea92fb89a8d", size = 202937, upload-time = "2026-05-22T14:48:35.859Z" },
{ url = "https://files.pythonhosted.org/packages/42/65/e29d54aef06a4d898a5b8a25589a0b3769bde454f922fad8f6f89fbfb650/wrapt-2.2.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ab5be648d5a0b86b7438864f8df3c705a65cef35a2fd3e5561e3e203167e0f27", size = 209997, upload-time = "2026-05-22T14:48:38.153Z" },
{ url = "https://files.pythonhosted.org/packages/2a/91/e4454263516cf0e12640912fbca9a83654e424f0a6ddb79f5cd7ce14bf33/wrapt-2.2.1-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9d8f204c8e3a8bf9ece17e0a83d137fd807440977f8a5e762d59306795011440", size = 194856, upload-time = "2026-05-22T14:48:39.69Z" },
{ url = "https://files.pythonhosted.org/packages/de/d0/fe0ee202286afdf4a7f77dd29f195703145764d572aec209c5086e57d924/wrapt-2.2.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:d047f6498c973874ba08ac3f97c69a2c4b2211c8de6f4c205f75cb1c9522596e", size = 205654, upload-time = "2026-05-22T14:48:43.456Z" },
{ url = "https://files.pythonhosted.org/packages/23/b6/87d860dfc6460c246af70b1fd5c8b76df77571b42a493459423ded94fd7d/wrapt-2.2.1-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:7a4fdb9326aab4a5a477a1640e5ad786a8495901009d7e7b038371edd23a9d2b", size = 192206, upload-time = "2026-05-22T14:48:44.858Z" },
{ url = "https://files.pythonhosted.org/packages/df/46/3eea8cde077d985f239a38c0257087b8064fd9ee9b1a99e282d2c86da4ef/wrapt-2.2.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:c8cc5094b08abeae52da9c73c8a32003623be691a5193df2f4e3eac3d557c394", size = 198428, upload-time = "2026-05-22T14:48:46.319Z" },
{ url = "https://files.pythonhosted.org/packages/18/dc/b927ee9c7fc67adc3a5658f246a0d275425eb840ba36e7b702e70f18bde8/wrapt-2.2.1-cp313-cp313t-win32.whl", hash = "sha256:9907a4402ab6db12b7077a0ea5d7a4d028ecb22c8eee2b53527080d347cd1562", size = 79448, upload-time = "2026-05-22T14:48:47.901Z" },
{ url = "https://files.pythonhosted.org/packages/ec/b3/fd30b473fe498c70e6b9a5f328b8d3fbaf1b8c3c481465f59724bba8eb70/wrapt-2.2.1-cp313-cp313t-win_amd64.whl", hash = "sha256:5590d63f5243251641cf543009b4c9314a79d0598fdb8a8e4cfc918494536c53", size = 83021, upload-time = "2026-05-22T14:48:49.201Z" },
{ url = "https://files.pythonhosted.org/packages/ee/f3/96c39153a8737a6e9aa85adef254ac4195bea3f2d24efc60472ccc3c9e2e/wrapt-2.2.1-cp313-cp313t-win_arm64.whl", hash = "sha256:c318a64b53d97b841d7b5e637517e50a27be64bc695128422953d4b21710954e", size = 80295, upload-time = "2026-05-22T14:48:50.479Z" },
{ url = "https://files.pythonhosted.org/packages/0a/a3/11d7f34ebbf3231bc907a3e6d5ee051b14d034c1bc7b65a97d5cc00516df/wrapt-2.2.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:6f56a647e4eaf5f0ca40330fb070f566bdf9f7b0db89a1af20d71c28dcd7a0ab", size = 80879, upload-time = "2026-05-22T14:48:51.802Z" },
{ url = "https://files.pythonhosted.org/packages/13/3c/b74cfd984cef560b900fb1a727af20352d89e1f06bf2e1114dd3f00f5f5a/wrapt-2.2.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:64b7deeda4b70408e382328d8bbe52a256fe9bc63ae3db86d804608367e5422c", size = 81462, upload-time = "2026-05-22T14:48:53.18Z" },
{ url = "https://files.pythonhosted.org/packages/15/a3/7c8f704b8dc07dfe0a5d01c2edbfd88317aa8e5e3fa7c743eb7a085ae767/wrapt-2.2.1-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b9cf53ba90717db2e292401de290776c498d4bbfb0d4a559ca2895db8b9dcb5c", size = 167251, upload-time = "2026-05-22T14:48:54.562Z" },
{ url = "https://files.pythonhosted.org/packages/80/85/a34d1888d97247da6c2ff6118c3a721c73ed8cc4dd198c00208bb73b6f80/wrapt-2.2.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cf3638274ab9d9b724c9baa0b4c04e132cd6faefb78b4dd3dd1a02a4bdaad41e", size = 166316, upload-time = "2026-05-22T14:48:56.065Z" },
{ url = "https://files.pythonhosted.org/packages/e9/d7/72ffaeb01eebc704afe3fb99e840480f4bda45f0fa66e3381b6a39251c8f/wrapt-2.2.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:aed9658797d0b45d6c49adcfc6b41f66e6f2d0c6de3ec79e16cf4b1855df240f", size = 157952, upload-time = "2026-05-22T14:48:57.924Z" },
{ url = "https://files.pythonhosted.org/packages/24/5b/36f5d6b024e4edfdd90b140742d11ebcf7836daf5c9daf326c55c24db412/wrapt-2.2.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:1d676ee388bc42a04d56dd7deb5605244dac2e35cc2fadbb43c9fa25bbd93508", size = 166130, upload-time = "2026-05-22T14:48:59.384Z" },
{ url = "https://files.pythonhosted.org/packages/81/06/9296d9e97bfdef5483dfcc859d57b095b257144b2bc5300ab521e06f4bc7/wrapt-2.2.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:e395f7bc31851ef9b612050368cb446e9bc14cd7454b025018980349caf25ae5", size = 156604, upload-time = "2026-05-22T14:49:00.921Z" },
{ url = "https://files.pythonhosted.org/packages/53/37/16953929ed6776175720e58fc966e779926d8d71e2c7b2273230590ca71f/wrapt-2.2.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5f1845c2a8cc1180ccccfa45785dd06f562730d19ef75be180334254012b6283", size = 166007, upload-time = "2026-05-22T14:49:02.332Z" },
{ url = "https://files.pythonhosted.org/packages/b9/73/20ee58c0612dae7c31131a7095345812ed2c7b389019e175f68cde34e5b4/wrapt-2.2.1-cp314-cp314-win32.whl", hash = "sha256:436addbc4bb4fc0a88c702577f51195d7d73683a7f3e0e5b253d8404d7847243", size = 78327, upload-time = "2026-05-22T14:49:03.722Z" },
{ url = "https://files.pythonhosted.org/packages/22/b3/ef7c3295d02e0448a71c639a36a057f46d524d057c9486291a7a3039e65c/wrapt-2.2.1-cp314-cp314-win_amd64.whl", hash = "sha256:50972a1d974ea07725a7f6b1cec5f8759008afd030a0024843ebe7d52de47f2b", size = 81144, upload-time = "2026-05-22T14:49:05.093Z" },
{ url = "https://files.pythonhosted.org/packages/ac/dc/7bdf336953f99f4ceb0a584bb8870e42c8f26f93ea10c87834dad62f1668/wrapt-2.2.1-cp314-cp314-win_arm64.whl", hash = "sha256:1c9934ea5d92957e3cd0adbc0845539dccfd62710ebe16195a8c66c53954db36", size = 79569, upload-time = "2026-05-22T14:49:06.413Z" },
{ url = "https://files.pythonhosted.org/packages/6a/6d/6dfae80150ff1919c356d1dd528f049bcdfaae29b4d284bc957e022caef4/wrapt-2.2.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:17de18fc12cea55b8a9587314cb830573e37fb33b247a7515696350863714188", size = 82892, upload-time = "2026-05-22T14:49:07.925Z" },
{ url = "https://files.pythonhosted.org/packages/82/7b/4e34766a7d7804ffce9e71befe47e9b3225dc350c49c94493c4ab39fd3a5/wrapt-2.2.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:a9dec1aca52dddde7df94818310fa2fe79739c8f385b2014c4cb1035f5508199", size = 83333, upload-time = "2026-05-22T14:49:09.257Z" },
{ url = "https://files.pythonhosted.org/packages/9d/57/0b34db3e8de44ccfece62d7b337abd1631dd810f5adc5f3db571727836b5/wrapt-2.2.1-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:69f2e9244542cb34dd59c7f073445b9e54ad9f3fce8d93606c368a1b499fc413", size = 202899, upload-time = "2026-05-22T14:49:10.572Z" },
{ url = "https://files.pythonhosted.org/packages/e5/45/ac0c459f154b99d92789a6cba7ca727185b83513b986f8ec7fe2aacddcbf/wrapt-2.2.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2d83966dc7f4f45e8b97b5933685ac2e6e67fc0e19246ea314bceb9a8970c956", size = 209986, upload-time = "2026-05-22T14:49:12.229Z" },
{ url = "https://files.pythonhosted.org/packages/b7/e4/77e37ff33ad018fa81ade52c25fa327b80b56f81d734279a63614fcb4cbc/wrapt-2.2.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:78b0aa6bfb7be8deed0ab23e7aa028cc5210c29bc2d32a04d52b50e517a7307e", size = 194893, upload-time = "2026-05-22T14:49:14.139Z" },
{ url = "https://files.pythonhosted.org/packages/dd/9d/7ea651d1ab032fc5fa222fbec91d0f8a1397f6ae04ebb93fa7219aa921d7/wrapt-2.2.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:05d5cb74d1b232ec8cfa130a8f900708699ff2491d97b8f85a4cdc5996294b85", size = 205636, upload-time = "2026-05-22T14:49:15.714Z" },
{ url = "https://files.pythonhosted.org/packages/09/af/8e88031a701275b9085c54e64bc88c0b1cd55c77eadd400691c371cd76c4/wrapt-2.2.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:f6518b94edb9150452e9aba08027d4cc293433753ec1fbefb4629a21cbc74181", size = 192267, upload-time = "2026-05-22T14:49:17.283Z" },
{ url = "https://files.pythonhosted.org/packages/bf/a8/e657ca876b06710194f243d81c4b0896ade646e244bdbec2d87c8c56a8bd/wrapt-2.2.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ed55af48b3eb28f43228ca2306788892bcb629eb2b5c4876e2a3659872c2f17a", size = 198378, upload-time = "2026-05-22T14:49:18.785Z" },
{ url = "https://files.pythonhosted.org/packages/c8/59/822efe4ea722a3961331bfa35b7d90937790d2c20f0616de1997ccc3aebd/wrapt-2.2.1-cp314-cp314t-win32.whl", hash = "sha256:2e08688ab16525897da6589d56d0aebaf417bbe91c2d8e3b96203b1efa596e85", size = 80226, upload-time = "2026-05-22T14:49:20.264Z" },
{ url = "https://files.pythonhosted.org/packages/ab/31/2a7dc5f6abb2fca0b6e1610e120419f603650aceb4f1d3ac4cae0354e162/wrapt-2.2.1-cp314-cp314t-win_amd64.whl", hash = "sha256:fd0135d34387f5fd087d9be368ea77ea89cf2451dc1cd1c622d35021bcb3ab50", size = 83835, upload-time = "2026-05-22T14:49:21.634Z" },
{ url = "https://files.pythonhosted.org/packages/9f/c0/782b86e28d1ceebeb74cccea12d2cd3d2ba0bd68e3dec20b1bc5873f6127/wrapt-2.2.1-cp314-cp314t-win_arm64.whl", hash = "sha256:f70db64e8266d7c45d3b735f2e08eeb434b5e03da9a479ae42b2e2e486a21a00", size = 80722, upload-time = "2026-05-22T14:49:23.59Z" },
{ url = "https://files.pythonhosted.org/packages/53/46/29ac9daf11a86c22a8c38cd9236c62928ccae83f7ceb06bd3b0467cf9d05/wrapt-2.2.1-py3-none-any.whl", hash = "sha256:3aafea2975caef8ca49400640dde02cc7426e798f24870ed01f490bc3cffd32f", size = 61000, upload-time = "2026-05-22T14:49:41.593Z" },
]
[[package]]
name = "xlsxwriter"
version = "3.2.9"