WEBDIENSTE/theHarvester
1
0
Fork 0
mirror of https://github.com/laramies/theHarvester.git synced 2026-08-17 19:35:40 +02:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #2548 from NotoriousRebel/codex/upstream-source-workers-default

Browse Source 
fix: restore conservative source-worker default
This commit is contained in:
Matt
2026-08-15 11:51:57 -04:00
committed by GitHub
parent f44807a046 4bcf2ceae9
commit fad5485de5
5 changed files with 6 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+1 -1
View File
@@ -36,7 +36,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed ### Changed
- Routed discovery sources through immutable source jobs with bounded `TaskGroup` ownership, typed outcomes, and native cancellation propagation instead of queuing live coroutine objects. - Routed discovery sources through immutable source jobs with bounded `TaskGroup` ownership, typed outcomes, and native cancellation propagation instead of queuing live coroutine objects.
- Discovery now uses a fixed pool of six source workers by default. CLI `-j` or `--source-workers`, REST - Discovery now uses a fixed pool of three source workers by default. CLI `-j` or `--source-workers`, REST
`source_workers`, and HarvestView can set another positive count without skipping sources or limiting their results. `source_workers`, and HarvestView can set another positive count without skipping sources or limiting their results.
- Replaced Shodan's synchronous Python SDK with cancellable async Host API requests that honor configured proxies, query every unique resolved IPv4, paginate target-bound hostname and TLS-certificate searches without an adapter-specific result cap, retain successful partial results, and add no source-local deadline. Shodan now stores one canonical `shodan-host` result per IP with every normalized TCP or UDP service and scoped certificate CN/SAN metadata in native JSONL, SQLite, API, and HarvestView details instead of an escaped JSON value. - Replaced Shodan's synchronous Python SDK with cancellable async Host API requests that honor configured proxies, query every unique resolved IPv4, paginate target-bound hostname and TLS-certificate searches without an adapter-specific result cap, retain successful partial results, and add no source-local deadline. Shodan now stores one canonical `shodan-host` result per IP with every normalized TCP or UDP service and scoped certificate CN/SAN metadata in native JSONL, SQLite, API, and HarvestView details instead of an escaped JSON value.
- Reworked screenshot scans to use one bounded aiohttp session and one shared browser, with isolated per-target contexts, status-based reachability, and deterministic async cleanup. - Reworked screenshot scans to use one bounded aiohttp session and one shared browser, with isolated per-target contexts, status-based reachability, and deterministic async cleanup.
README.md
+1 -1
View File
@@ -42,7 +42,7 @@ Query several passive sources:
uv run theHarvester -d example.com -b crtsh,certspotter,commoncrawl uv run theHarvester -d example.com -b crtsh,certspotter,commoncrawl
``` ```
Six discovery sources run at once by default. Use `-j` or `--source-workers` with a positive number to change that Three discovery sources run at once by default. Use `-j` or `--source-workers` with a positive number to change that
concurrency. The worker count never skips a selected source or limits its results, and it is automatically reduced when concurrency. The worker count never skips a selected source or limits its results, and it is automatically reduced when
fewer sources are selected. REST `source_workers` and HarvestView use the same setting. fewer sources are selected. REST `source_workers` and HarvestView use the same setting.
docs/wiki/Rest-API.md
+2 -2
View File
@@ -66,7 +66,7 @@ run_id="$(curl -s http://127.0.0.1:5000/api/v1/runs \
"target": "example.com", "target": "example.com",
"sources": ["emails", "crtsh"], "sources": ["emails", "crtsh"],
"limit": 500, "limit": 500,
"source_workers": 6, "source_workers": 3,
"deadline_seconds": 1800 "deadline_seconds": 1800
}' \ }' \
| jq -r '.run_id')" | jq -r '.run_id')"
@@ -79,7 +79,7 @@ curl -s "http://127.0.0.1:5000/api/v1/runs/$run_id" \
Run submission is asynchronous. Lifecycle status is `queued`, `running`, `cancelling`, `cancelled`, `completed`, or `failed`. Terminal evidence status is reported separately as `complete`, `partial`, or `failed` when evidence exists. Run submission is asynchronous. Lifecycle status is `queued`, `running`, `cancelling`, `cancelled`, `completed`, or `failed`. Terminal evidence status is reported separately as `complete`, `partial`, or `failed` when evidence exists.
`source_workers` is the same positive concurrency used by CLI `-j` or `--source-workers` and HarvestView. It defaults `source_workers` is the same positive concurrency used by CLI `-j` or `--source-workers` and HarvestView. It defaults
to six, is reduced when fewer sources are selected, and never skips sources or limits their results. to three, is reduced when fewer sources are selected, and never skips sources or limits their results.
P1 DNS and P2 direct options are fields on the same run request. The OpenAPI schema shows their current defaults, limits, and descriptions. The server uses the operator-selected target and does not impose a public-only egress policy. P1 DNS and P2 direct options are fields on the same run request. The OpenAPI schema shows their current defaults, limits, and descriptions. The server uses the operator-selected target and does not impose a public-only egress policy.
tests/lib/test_enumeration.py
+1 -1
View File
@@ -20,7 +20,7 @@ def test_enumeration_options_fill_the_shared_execution_defaults() -> None:
assert options.start == 0 assert options.start == 0
assert options.dns_recursive_query_limit == DEFAULT_DNS_RECURSIVE_QUERY_LIMIT is None assert options.dns_recursive_query_limit == DEFAULT_DNS_RECURSIVE_QUERY_LIMIT is None
assert options.dns_recursive_runtime_seconds == DEFAULT_DNS_RECURSIVE_RUNTIME_SECONDS is None assert options.dns_recursive_runtime_seconds == DEFAULT_DNS_RECURSIVE_RUNTIME_SECONDS is None
assert options.source_workers == DEFAULT_SOURCE_WORKERS == 6 assert options.source_workers == DEFAULT_SOURCE_WORKERS == 3
def test_enumeration_options_preserve_explicit_transport_values() -> None: def test_enumeration_options_preserve_explicit_transport_values() -> None:
theHarvester/lib/enumeration.py
+1 -1
View File
@@ -13,7 +13,7 @@ from theHarvester.lib.virtual_host import (
DEFAULT_RESULT_LIMIT = 500 DEFAULT_RESULT_LIMIT = 500
DEFAULT_RESULT_START = 0 DEFAULT_RESULT_START = 0
DEFAULT_SOURCE_WORKERS = 6 DEFAULT_SOURCE_WORKERS = 3
DEFAULT_DNS_RECURSIVE_QUERY_LIMIT = DEFAULT_RECURSIVE_DNS_QUERY_LIMIT DEFAULT_DNS_RECURSIVE_QUERY_LIMIT = DEFAULT_RECURSIVE_DNS_QUERY_LIMIT
DEFAULT_DNS_RECURSIVE_RUNTIME_SECONDS = DEFAULT_RECURSIVE_DNS_RUNTIME_SECONDS DEFAULT_DNS_RECURSIVE_RUNTIME_SECONDS = DEFAULT_RECURSIVE_DNS_RUNTIME_SECONDS