wayback, arquivo, certspotter, gitlab, hunter, tomba, mojeek, windvane,
yahoo, pentest-tools, and commoncrawl each built and tore down a fresh
ClientSession (connector, TLS context, proxy resolution) for every page,
poll, or README request. Each adapter now opens one session for its whole
conversation and passes session= to the shared fetchers, matching the
open_session pattern already used by the other paginating providers.
Also removes yahoo's unreachable empty-response guard, collects pages in
a list instead of O(n^2) string concatenation, and strips padded email
local parts with lstrip instead of per-character slicing. Contract tests
assert the proxy selection and session identity at the session owner.
- _child_execute now returns an exit code and the module entry maps it
to the process status, so a child terminated by an external signal
exits nonzero; the parent records the run as failed with the partial
checkpoint attached instead of mistaking leftover evidence for
terminal completion
- recover_orphans isolates each orphaned run: one unreadable checkpoint
(schema-passing but structurally broken) no longer aborts recovery of
the remaining runs or crashes every future startup - the affected run
is failed without evidence and recovery continues
- _process_output drains child streams to EOF but retains only a
bounded tail, so a verbose child cannot grow API memory unboundedly
- the execution poll reads the lifecycle row instead of the full detail
projection every 50 ms
- run-now maps a stale stored run template to a sanitized 503 instead
of leaking a raw pydantic ValidationError as a 500
A label such as _dmarc passes the recursive-DNS label normalizer (which
allows underscores) but is rejected by the shared hostname normalizer
inside validate_dns_candidates, whose empty result then crashed the
whole dns-recursive action with an unhandled ValueError from the strict
tuple unpack. Check the candidate with the same normalize_scoped_hostname
helper before querying and skip it, so one unusable wordlist entry no
longer aborts the action.
- _create_connector: an explicit ssl=False (verification disabled) was
silently replaced with a verifying context by 'ssl_context or default';
only an unset context now falls back to the certifi default
- fetch/post_fetch/fetch_all: caller-supplied headers were silently
dropped when a caller-owned session was passed; they are now forwarded
per request, matching _open_get_response and aiohttp merge semantics
(sessions that bake the same headers are unaffected)
- fetch: a borrowed session with default verification no longer rebuilds
a certifi SSL context on every request; it defers to the connector
context the session already owns
- _normalize_data: the post_fetch data='' default raised JSONDecodeError
internally and silently returned without sending any request; empty
bodies now send, and non-JSON strings pass through as raw bodies
Traced every owned-versus-borrowed branch; all post_fetch callers pass
json_body or valid JSON strings, and the only verify=False caller
(api_endpoints) supplies its own connector.
- hunter/tomba: replace the substring source-domain check that admitted
out-of-scope hosts (notexample.test, example.test.evil.net) with
normalize_scoped_hostname; tomba also tolerates full website URLs
- arquivo: a repeated page with zero collected hosts now reports failed
instead of partial, matching the module's own outcome convention
- pentesttools, rocketreach, shodan: blank or whitespace-only API keys
now raise MissingKey like peer providers instead of starting a doomed
authenticated run
- criminalip: inspect HTTP status via provider_http_error instead of
only the provider's embedded status field, so 401/403/429 with
non-JSON bodies are classified correctly; build the scan request with
json_body instead of an f-string that produced invalid JSON for
targets containing quotes or backslashes
- hudsonrock: remove the dead transport-error retry loop (the shared
transport already swallows those exceptions and returns None), drop a
redundant isinstance check, and describe the real 429-only retry
- mojeek: fix the finite API path off-by-one that fetched zero pages
for --limit 1 and dropped the final partial page, and request pages
sequentially instead of one concurrent burst
- githubcode: only HTTP 429 responses retry; a 403 now fails
immediately as access-denied instead of stalling ~3 minutes and being
mislabeled rate-limited
Transport errors, non-2xx responses, and malformed bodies were returned
as None, which the shared source runner maps to a completed run. Classify
them with the shared provider_http_error convention so failures surface
as failed/rate-limited outcomes and partial data is preserved:
- hunter/tomba: _fetch_json returns dict or SourceExecutionReport;
malformed-dict shapes still map to invalid-response
- gitlab: unreachable 'if not response' guard replaced; [None] transport
failures report transport-error instead of invalid-response
- waybackarchive: transport errors no longer masquerade as end-of-data;
_search_pattern returns explicit SourceExecutionReport outcomes and
HTTP failures use provider_http_error
- certspotter: transport failures distinguished from invalid responses;
HTTP failures classified per the shared convention
Contract tests updated to assert the resulting reports and to mock the
metadata-shaped responses the shared transport actually delivers.
Preserve provider outcome release notes alongside the HTTP proxy wording. Keep the dev domain glossary and carry the master HTTP-only proxy policy into its relocated architecture guide. Retain RocketReach cancellation coverage while removing the obsolete global dependency stub.
Align provider lifecycle regressions with the master session-construction boundary: construction ValueError is normalized to transport-error, while cancellation, lifecycle RuntimeError, and teardown ValueError still propagate.
* fix: preserve provider failures and decode POST evidence
Keep failed, rate-limited, malformed, and partial outcomes visible across ten provider adapters so saved hostname comparisons do not infer reliable absence. Restore DeHashed and Leak-Lookup JSON response decoding.
Refs #358, #359
* docs: document released 4.11 automation migrations
Address #361 with launcher, route, request, response, source, and DNS resolver migrations verified against the 4.11.1 tag.
* fix: own provider sessions across related requests
Provide harvest-report targets, contributions, and hostname-changes with consistent saved-evidence terminology across the CLI, REST API, and HarvestView. Preserve target scope, source-outcome handling, JSONL, and the stored evidence schema.
Update onboarding and operator documentation, keep the domain glossary in CONTEXT.md, record behavior rules in docs/architecture.md, and consolidate release preparation in CONTRIBUTING.md.
Validation: all eight GitHub checks passed at ed54542a; focused reporting and documentation checks passed. Tracks https://github.com/NotoriousRebel/theHarvester/issues/349.
Put IDNA and label validation in normalize_scoped_hostname so the
runner, parser, and storage path share one authorized-target boundary.
Drop IPv6 zone identifiers at collection instead of failing finish(),
and remove leftover BuiltWith and Shodan aliases now that www. is part
of the explicit target.
Use the standard asyncio event loop on Windows because winloop rejects the startupinfo argument Playwright passes when launching its driver subprocess. Remove the now-unused winloop dependency and cover the loop selection with a regression test.
L1ghtn1ng