Normalize recognized provider names case-insensitively while preserving canonical-name precedence, unknown entries, and credential fields.
Verified locally with 121 focused tests and the full non-browser suite (2358 passed, 6 skipped), plus Ruff lint, formatting, and ty checks.
P1 describes DNS operations theHarvester runs itself, not work a provider does on its side (per review), so otilabs is passive and included in -b all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The API brute-forces common names and resolves what it finds through public resolvers, so the source is excluded from -b all and runs only when selected explicitly. Adds otilabs to the non-passive coverage in test_core and the CLI orchestration test, and asserts the activity class and selection in its contract module.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Classify SQLAlchemy access failures (locks, I/O, lost connections) as
ResultStoreUnavailableError so orphan recovery and existing-evidence
lookups propagate them instead of failing the run without evidence.
- Test the fault at the SQLite driver so ResultStore's translation runs,
then clear it and confirm the next recovery attaches the evidence.
- Derive GitHub code search retry waits from Retry-After and
x-ratelimit-reset, and report rate-limited instead of retrying when the
deadline exceeds the wait budget.
_process_response returned an EndpointResult for every status other
than 0, and _check_endpoint only treats None as "not found", so a path
answering 404 or 410 was stored in found_endpoints. The CLI then printed
it under "API Endpoints found" and added it to the URL results: against
a target that answers 404 for unknown paths, the whole wordlist was
reported as discovered.
- 404 and 410 now return None; the path still counts as checked and its
status stays in get_status_codes()
- a missing path is no longer re-probed with HEAD and OPTIONS, which
would otherwise turn it into a finding on servers that answer OPTIONS
for any path (common with CORS) and triple the requests per miss
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_fetch_http passed the raw status and Location header straight into
TakeoverHTTPOutcome, whose validation rejects them in normal situations,
and the resulting ValueError or LookupError escaped both TaskGroups:
process() stopped with scan-error and every other candidate was
cancelled. One host answering HTTP 999, sending an empty or oversized
Location, or declaring an unknown charset was enough.
- a status outside 100-599 or an undecodable response becomes an
InvalidResponseError outcome for that host only
- an empty, blank or over-long Location is dropped instead of raising
- Location bytes that are not UTF-8 (surrogateescape from aiohttp) are
decoded leniently, so the outcome can be persisted; previously
save_run failed with UnicodeEncodeError
- resolved hosts in host:addr1,addr2 or IPv6 form are accepted as
candidates; only the single-IPv4 form was recognized, so brute-forced
hosts with several A/AAAA records were silently skipped
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- hudsonrock: hoist re/ipaddress imports out of the per-record hot path
and precompile the email pattern at module level
- dnssearch: drop commented-out constructor code, replace str.format
with an f-string, and rename dunder-prefixed function locals
- api_endpoints: build the default wordlist path with pathlib and drop
the uncalled export_results (which shadowed the format builtin)
- duckduckgosearch: build the API URL with urlencode instead of blind
'x' substitution on the whole template
- baidusearch: use the new public AsyncFetcher.resolve_proxy instead of
reaching into the private _resolve_proxy
- twelve adapters: silent 'except Exception' backstops now log the
exception type so coding errors are distinguishable from transport
failures (no payloads are logged)
- fill in missing annotations on public constructors, parse helpers,
and result getters in otherwise-annotated modules
Every removal was verified against the full repo including tests:
- fullhuntsearch: drop the unused data-intelligence filter framework
(constants, add_filter/add_filters, advanced_search, nine search_*
helpers, get_all_results, the filters branch in do_search, and the
never-supplied process filters parameter); 528 -> 290 lines
- myparser: drop six never-called methods (url_clean, fileurls,
hostnames_all, set, urls, unique)
- shodan_internetdb: stop collecting ports/CVEs/tags/CPEs that no route
ever read; _has_results is unchanged because a valid payload always
records its IP
- githubcode: drop next_page_or_end and ErrorResult.body
- securityscorecard, sherlockeye, haveibeenpwned, leaklookup: drop
never-populated attributes and unwired getters (leaklookup keeps its
routed get_emails)
- windvane: drop never-called set_api_key and correct the docstring's
env-var claim; duckduckgosearch: drop four unused attributes
- mojeek, thc: unreachable returns; criminalip: a no-op wildcard filter
that could never match; api_endpoints: dead 'POST' membership test;
hunter/tomba: unused counter; takeover_evidence: no-op conditional
- __main__: drop the rest_filename machinery (always empty; its other
branch wrote into a nonexistent directory), unreachable NameError
guards, and a pointless else: pass; screenshot durations no longer
silently drop hours
- output: print_linkedin_people loses its CLI-unreachable
no-users-with-linkin-engine branch and its engines parameter
- takeover: HTTP probes ran with no time bound at all (session timeout
total=None, asserted by a test); a stalled endpoint could hang the
whole scan indefinitely. The probe session now carries a generous
30-second per-request bound.
- rocketreach: remove the trailing sleep after pagination that wasted
5.5-7.5 seconds on every successful run without any following request
- dehashed: drop the dead capitalized Retry-After lookup; the shared
transport always lowercases response headers
- subdomainfinderc99: pass the form payload directly instead of a
json.dumps round-trip that the shared transport immediately undid
wayback, arquivo, certspotter, gitlab, hunter, tomba, mojeek, windvane,
yahoo, pentest-tools, and commoncrawl each built and tore down a fresh
ClientSession (connector, TLS context, proxy resolution) for every page,
poll, or README request. Each adapter now opens one session for its whole
conversation and passes session= to the shared fetchers, matching the
open_session pattern already used by the other paginating providers.
Also removes yahoo's unreachable empty-response guard, collects pages in
a list instead of O(n^2) string concatenation, and strips padded email
local parts with lstrip instead of per-character slicing. Contract tests
assert the proxy selection and session identity at the session owner.
- _child_execute now returns an exit code and the module entry maps it
to the process status, so a child terminated by an external signal
exits nonzero; the parent records the run as failed with the partial
checkpoint attached instead of mistaking leftover evidence for
terminal completion
- recover_orphans isolates each orphaned run: one unreadable checkpoint
(schema-passing but structurally broken) no longer aborts recovery of
the remaining runs or crashes every future startup - the affected run
is failed without evidence and recovery continues
- _process_output drains child streams to EOF but retains only a
bounded tail, so a verbose child cannot grow API memory unboundedly
- the execution poll reads the lifecycle row instead of the full detail
projection every 50 ms
- run-now maps a stale stored run template to a sanitized 503 instead
of leaking a raw pydantic ValidationError as a 500
A label such as _dmarc passes the recursive-DNS label normalizer (which
allows underscores) but is rejected by the shared hostname normalizer
inside validate_dns_candidates, whose empty result then crashed the
whole dns-recursive action with an unhandled ValueError from the strict
tuple unpack. Check the candidate with the same normalize_scoped_hostname
helper before querying and skip it, so one unusable wordlist entry no
longer aborts the action.
- _create_connector: an explicit ssl=False (verification disabled) was
silently replaced with a verifying context by 'ssl_context or default';
only an unset context now falls back to the certifi default
- fetch/post_fetch/fetch_all: caller-supplied headers were silently
dropped when a caller-owned session was passed; they are now forwarded
per request, matching _open_get_response and aiohttp merge semantics
(sessions that bake the same headers are unaffected)
- fetch: a borrowed session with default verification no longer rebuilds
a certifi SSL context on every request; it defers to the connector
context the session already owns
- _normalize_data: the post_fetch data='' default raised JSONDecodeError
internally and silently returned without sending any request; empty
bodies now send, and non-JSON strings pass through as raw bodies
Traced every owned-versus-borrowed branch; all post_fetch callers pass
json_body or valid JSON strings, and the only verify=False caller
(api_endpoints) supplies its own connector.
- hunter/tomba: replace the substring source-domain check that admitted
out-of-scope hosts (notexample.test, example.test.evil.net) with
normalize_scoped_hostname; tomba also tolerates full website URLs
- arquivo: a repeated page with zero collected hosts now reports failed
instead of partial, matching the module's own outcome convention
- pentesttools, rocketreach, shodan: blank or whitespace-only API keys
now raise MissingKey like peer providers instead of starting a doomed
authenticated run
- criminalip: inspect HTTP status via provider_http_error instead of
only the provider's embedded status field, so 401/403/429 with
non-JSON bodies are classified correctly; build the scan request with
json_body instead of an f-string that produced invalid JSON for
targets containing quotes or backslashes
- hudsonrock: remove the dead transport-error retry loop (the shared
transport already swallows those exceptions and returns None), drop a
redundant isinstance check, and describe the real 429-only retry
- mojeek: fix the finite API path off-by-one that fetched zero pages
for --limit 1 and dropped the final partial page, and request pages
sequentially instead of one concurrent burst
- githubcode: only HTTP 429 responses retry; a 403 now fails
immediately as access-denied instead of stalling ~3 minutes and being
mislabeled rate-limited
Transport errors, non-2xx responses, and malformed bodies were returned
as None, which the shared source runner maps to a completed run. Classify
them with the shared provider_http_error convention so failures surface
as failed/rate-limited outcomes and partial data is preserved:
- hunter/tomba: _fetch_json returns dict or SourceExecutionReport;
malformed-dict shapes still map to invalid-response
- gitlab: unreachable 'if not response' guard replaced; [None] transport
failures report transport-error instead of invalid-response
- waybackarchive: transport errors no longer masquerade as end-of-data;
_search_pattern returns explicit SourceExecutionReport outcomes and
HTTP failures use provider_http_error
- certspotter: transport failures distinguished from invalid responses;
HTTP failures classified per the shared convention
Contract tests updated to assert the resulting reports and to mock the
metadata-shaped responses the shared transport actually delivers.
Preserve provider outcome release notes alongside the HTTP proxy wording. Keep the dev domain glossary and carry the master HTTP-only proxy policy into its relocated architecture guide. Retain RocketReach cancellation coverage while removing the obsolete global dependency stub.
Align provider lifecycle regressions with the master session-construction boundary: construction ValueError is normalized to transport-error, while cancellation, lifecycle RuntimeError, and teardown ValueError still propagate.
* fix: preserve provider failures and decode POST evidence
Keep failed, rate-limited, malformed, and partial outcomes visible across ten provider adapters so saved hostname comparisons do not infer reliable absence. Restore DeHashed and Leak-Lookup JSON response decoding.
Refs #358, #359
* docs: document released 4.11 automation migrations
Address #361 with launcher, route, request, response, source, and DNS resolver migrations verified against the 4.11.1 tag.
* fix: own provider sessions across related requests
Provide harvest-report targets, contributions, and hostname-changes with consistent saved-evidence terminology across the CLI, REST API, and HarvestView. Preserve target scope, source-outcome handling, JSONL, and the stored evidence schema.
Update onboarding and operator documentation, keep the domain glossary in CONTEXT.md, record behavior rules in docs/architecture.md, and consolidate release preparation in CONTRIBUTING.md.
Validation: all eight GitHub checks passed at ed54542a; focused reporting and documentation checks passed. Tracks https://github.com/NotoriousRebel/theHarvester/issues/349.
