wayback, arquivo, certspotter, gitlab, hunter, tomba, mojeek, windvane,
yahoo, pentest-tools, and commoncrawl each built and tore down a fresh
ClientSession (connector, TLS context, proxy resolution) for every page,
poll, or README request. Each adapter now opens one session for its whole
conversation and passes session= to the shared fetchers, matching the
open_session pattern already used by the other paginating providers.
Also removes yahoo's unreachable empty-response guard, collects pages in
a list instead of O(n^2) string concatenation, and strips padded email
local parts with lstrip instead of per-character slicing. Contract tests
assert the proxy selection and session identity at the session owner.
- _child_execute now returns an exit code and the module entry maps it
to the process status, so a child terminated by an external signal
exits nonzero; the parent records the run as failed with the partial
checkpoint attached instead of mistaking leftover evidence for
terminal completion
- recover_orphans isolates each orphaned run: one unreadable checkpoint
(schema-passing but structurally broken) no longer aborts recovery of
the remaining runs or crashes every future startup - the affected run
is failed without evidence and recovery continues
- _process_output drains child streams to EOF but retains only a
bounded tail, so a verbose child cannot grow API memory unboundedly
- the execution poll reads the lifecycle row instead of the full detail
projection every 50 ms
- run-now maps a stale stored run template to a sanitized 503 instead
of leaking a raw pydantic ValidationError as a 500
A label such as _dmarc passes the recursive-DNS label normalizer (which
allows underscores) but is rejected by the shared hostname normalizer
inside validate_dns_candidates, whose empty result then crashed the
whole dns-recursive action with an unhandled ValueError from the strict
tuple unpack. Check the candidate with the same normalize_scoped_hostname
helper before querying and skip it, so one unusable wordlist entry no
longer aborts the action.
- _create_connector: an explicit ssl=False (verification disabled) was
silently replaced with a verifying context by 'ssl_context or default';
only an unset context now falls back to the certifi default
- fetch/post_fetch/fetch_all: caller-supplied headers were silently
dropped when a caller-owned session was passed; they are now forwarded
per request, matching _open_get_response and aiohttp merge semantics
(sessions that bake the same headers are unaffected)
- fetch: a borrowed session with default verification no longer rebuilds
a certifi SSL context on every request; it defers to the connector
context the session already owns
- _normalize_data: the post_fetch data='' default raised JSONDecodeError
internally and silently returned without sending any request; empty
bodies now send, and non-JSON strings pass through as raw bodies
Traced every owned-versus-borrowed branch; all post_fetch callers pass
json_body or valid JSON strings, and the only verify=False caller
(api_endpoints) supplies its own connector.
- hunter/tomba: replace the substring source-domain check that admitted
out-of-scope hosts (notexample.test, example.test.evil.net) with
normalize_scoped_hostname; tomba also tolerates full website URLs
- arquivo: a repeated page with zero collected hosts now reports failed
instead of partial, matching the module's own outcome convention
- pentesttools, rocketreach, shodan: blank or whitespace-only API keys
now raise MissingKey like peer providers instead of starting a doomed
authenticated run
- criminalip: inspect HTTP status via provider_http_error instead of
only the provider's embedded status field, so 401/403/429 with
non-JSON bodies are classified correctly; build the scan request with
json_body instead of an f-string that produced invalid JSON for
targets containing quotes or backslashes
- hudsonrock: remove the dead transport-error retry loop (the shared
transport already swallows those exceptions and returns None), drop a
redundant isinstance check, and describe the real 429-only retry
- mojeek: fix the finite API path off-by-one that fetched zero pages
for --limit 1 and dropped the final partial page, and request pages
sequentially instead of one concurrent burst
- githubcode: only HTTP 429 responses retry; a 403 now fails
immediately as access-denied instead of stalling ~3 minutes and being
mislabeled rate-limited
Transport errors, non-2xx responses, and malformed bodies were returned
as None, which the shared source runner maps to a completed run. Classify
them with the shared provider_http_error convention so failures surface
as failed/rate-limited outcomes and partial data is preserved:
- hunter/tomba: _fetch_json returns dict or SourceExecutionReport;
malformed-dict shapes still map to invalid-response
- gitlab: unreachable 'if not response' guard replaced; [None] transport
failures report transport-error instead of invalid-response
- waybackarchive: transport errors no longer masquerade as end-of-data;
_search_pattern returns explicit SourceExecutionReport outcomes and
HTTP failures use provider_http_error
- certspotter: transport failures distinguished from invalid responses;
HTTP failures classified per the shared convention
Contract tests updated to assert the resulting reports and to mock the
metadata-shaped responses the shared transport actually delivers.
Preserve provider outcome release notes alongside the HTTP proxy wording. Keep the dev domain glossary and carry the master HTTP-only proxy policy into its relocated architecture guide. Retain RocketReach cancellation coverage while removing the obsolete global dependency stub.
Align provider lifecycle regressions with the master session-construction boundary: construction ValueError is normalized to transport-error, while cancellation, lifecycle RuntimeError, and teardown ValueError still propagate.
* fix: preserve provider failures and decode POST evidence
Keep failed, rate-limited, malformed, and partial outcomes visible across ten provider adapters so saved hostname comparisons do not infer reliable absence. Restore DeHashed and Leak-Lookup JSON response decoding.
Refs #358, #359
* docs: document released 4.11 automation migrations
Address #361 with launcher, route, request, response, source, and DNS resolver migrations verified against the 4.11.1 tag.
* fix: own provider sessions across related requests
Provide harvest-report targets, contributions, and hostname-changes with consistent saved-evidence terminology across the CLI, REST API, and HarvestView. Preserve target scope, source-outcome handling, JSONL, and the stored evidence schema.
Update onboarding and operator documentation, keep the domain glossary in CONTEXT.md, record behavior rules in docs/architecture.md, and consolidate release preparation in CONTRIBUTING.md.
Validation: all eight GitHub checks passed at ed54542a; focused reporting and documentation checks passed. Tracks https://github.com/NotoriousRebel/theHarvester/issues/349.
L1ghtn1ng