Normalize recognized provider names case-insensitively while preserving canonical-name precedence, unknown entries, and credential fields.
Verified locally with 121 focused tests and the full non-browser suite (2358 passed, 6 skipped), plus Ruff lint, formatting, and ty checks.
- Classify SQLAlchemy access failures (locks, I/O, lost connections) as
ResultStoreUnavailableError so orphan recovery and existing-evidence
lookups propagate them instead of failing the run without evidence.
- Test the fault at the SQLite driver so ResultStore's translation runs,
then clear it and confirm the next recovery attaches the evidence.
- Derive GitHub code search retry waits from Retry-After and
x-ratelimit-reset, and report rate-limited instead of retrying when the
deadline exceeds the wait budget.
Every removal was verified against the full repo including tests:
- fullhuntsearch: drop the unused data-intelligence filter framework
(constants, add_filter/add_filters, advanced_search, nine search_*
helpers, get_all_results, the filters branch in do_search, and the
never-supplied process filters parameter); 528 -> 290 lines
- myparser: drop six never-called methods (url_clean, fileurls,
hostnames_all, set, urls, unique)
- shodan_internetdb: stop collecting ports/CVEs/tags/CPEs that no route
ever read; _has_results is unchanged because a valid payload always
records its IP
- githubcode: drop next_page_or_end and ErrorResult.body
- securityscorecard, sherlockeye, haveibeenpwned, leaklookup: drop
never-populated attributes and unwired getters (leaklookup keeps its
routed get_emails)
- windvane: drop never-called set_api_key and correct the docstring's
env-var claim; duckduckgosearch: drop four unused attributes
- mojeek, thc: unreachable returns; criminalip: a no-op wildcard filter
that could never match; api_endpoints: dead 'POST' membership test;
hunter/tomba: unused counter; takeover_evidence: no-op conditional
- __main__: drop the rest_filename machinery (always empty; its other
branch wrote into a nonexistent directory), unreachable NameError
guards, and a pointless else: pass; screenshot durations no longer
silently drop hours
- output: print_linkedin_people loses its CLI-unreachable
no-users-with-linkin-engine branch and its engines parameter
- _child_execute now returns an exit code and the module entry maps it
to the process status, so a child terminated by an external signal
exits nonzero; the parent records the run as failed with the partial
checkpoint attached instead of mistaking leftover evidence for
terminal completion
- recover_orphans isolates each orphaned run: one unreadable checkpoint
(schema-passing but structurally broken) no longer aborts recovery of
the remaining runs or crashes every future startup - the affected run
is failed without evidence and recovery continues
- _process_output drains child streams to EOF but retains only a
bounded tail, so a verbose child cannot grow API memory unboundedly
- the execution poll reads the lifecycle row instead of the full detail
projection every 50 ms
- run-now maps a stale stored run template to a sanitized 503 instead
of leaking a raw pydantic ValidationError as a 500
A label such as _dmarc passes the recursive-DNS label normalizer (which
allows underscores) but is rejected by the shared hostname normalizer
inside validate_dns_candidates, whose empty result then crashed the
whole dns-recursive action with an unhandled ValueError from the strict
tuple unpack. Check the candidate with the same normalize_scoped_hostname
helper before querying and skip it, so one unusable wordlist entry no
longer aborts the action.
- _create_connector: an explicit ssl=False (verification disabled) was
silently replaced with a verifying context by 'ssl_context or default';
only an unset context now falls back to the certifi default
- fetch/post_fetch/fetch_all: caller-supplied headers were silently
dropped when a caller-owned session was passed; they are now forwarded
per request, matching _open_get_response and aiohttp merge semantics
(sessions that bake the same headers are unaffected)
- fetch: a borrowed session with default verification no longer rebuilds
a certifi SSL context on every request; it defers to the connector
context the session already owns
- _normalize_data: the post_fetch data='' default raised JSONDecodeError
internally and silently returned without sending any request; empty
bodies now send, and non-JSON strings pass through as raw bodies
Traced every owned-versus-borrowed branch; all post_fetch callers pass
json_body or valid JSON strings, and the only verify=False caller
(api_endpoints) supplies its own connector.
Preserve provider outcome release notes alongside the HTTP proxy wording. Keep the dev domain glossary and carry the master HTTP-only proxy policy into its relocated architecture guide. Retain RocketReach cancellation coverage while removing the obsolete global dependency stub.
Align provider lifecycle regressions with the master session-construction boundary: construction ValueError is normalized to transport-error, while cancellation, lifecycle RuntimeError, and teardown ValueError still propagate.
Provide harvest-report targets, contributions, and hostname-changes with consistent saved-evidence terminology across the CLI, REST API, and HarvestView. Preserve target scope, source-outcome handling, JSONL, and the stored evidence schema.
Update onboarding and operator documentation, keep the domain glossary in CONTEXT.md, record behavior rules in docs/architecture.md, and consolidate release preparation in CONTRIBUTING.md.
Validation: all eight GitHub checks passed at ed54542a; focused reporting and documentation checks passed. Tracks https://github.com/NotoriousRebel/theHarvester/issues/349.
Put IDNA and label validation in normalize_scoped_hostname so the
runner, parser, and storage path share one authorized-target boundary.
Drop IPv6 zone identifiers at collection instead of failing finish(),
and remove leftover BuiltWith and Shodan aliases now that www. is part
of the explicit target.
