Covers hostname extraction, wildcard prefix stripping, multi-line
name_value splitting, numeric-prefix filtering, and the empty-response
and missing-key error paths. AsyncFetcher is mocked so the tests run
without network access.
Adds a new passive source that calls the Dymo data verifier endpoint
(https://api.tpeoficial.com/v1/private/secure/verify) with the target
domain. The response is stored on the source for downstream inspection
and any matching canonical domain or didYouMean suggestion is added to
the host set so the rest of the pipeline can resolve it.
The source follows the existing plugin shape: SearchDymo lives in
theHarvester/discovery/dymosearch.py, the API key is read from the
``dymo`` block in api-keys.yaml via Core.dymo_key(), and the engine is
registered in get_supportedengines() and the -b/--source dispatcher.
Tests cover the missing-key path, header/payload shape, the empty,
unrelated-suggestion and non-dict-response branches, and the
get_supportedengines() registration. Closes#2300.
The async worker that drains the engine queue catches `Exception` with
no `as` binding and prints the bare line
"A error occurred while processing a 'work item'."
That message both swallows the actual error class+text (so users
report bugs like #2227 with no actionable detail and the maintainer
has to ask them to repro under more verbosity) and reads "A error"
instead of the grammatical "An error".
Bind the exception, print its type and message, and fix the article.
The shodan integration test (`tests/discovery/test_shodan_engine.py`)
already asserts the message is not in output on the happy path; the
assertion is updated to match the new string.
Closes#2227.
BuiltWith API returns 'text/json; charset=utf-8' as the Content-Type
header instead of 'application/json'. aiohttp's response.json() raises
a ContentTypeError by default when the MIME type is not application/json.
Passing content_type=None tells aiohttp to skip the content-type check
and parse the response body as JSON regardless of the declared MIME type.
Fixes#2235
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Introduced comprehensive security tests, including CORS configuration, XML injection prevention, path traversal prevention, and information disclosure protection.
- Improved input sanitization for filenames and XML content to mitigate injection attacks and ensure safe handling.
- Updated CORS middleware to restrict HTTP methods and block credentials with wildcard origins.
- Removed traceback exposure in API error responses, even in debug mode.
- Verified rate limiting and input validation for sensitive API endpoints.
- Ensured no hardcoded secrets are present in the main codebase.
* feat(discovery): add THC subdomain enumeration module
Add new THC module integrating ip.thc.org for subdomain discovery.
Features:
- Free service, no API key required
- No rate limits
- Fast plain-text response parsing
Includes 24 comprehensive tests covering:
- API endpoint validation
- Subdomain search functionality
- Edge cases (empty, unicode, special chars)
- Proxy support
- Response format validation
- Integration with theHarvester framework
Closes#2217
* fix: translate test comments to English
Accidentally pushed the Spanish version of the test file.
This commit updates all docstrings and comments to English.
* fix: correct rate limit documentation and add rate limit header test
- Remove 'no rate limits' claim from README.md
- Add test_api_returns_rate_limit_headers to verify x-ratelimit-* headers
* feat: add explicit rate limit handling
Based on bravesearch.py pattern:
- Detect HTTP 429 status code
- Read x-ratelimit-remaining header
- Implement retry logic with backoff (3 retries, 2s base delay)
- Handle rate limit exceptions
Added test for rate limit settings initialization.
- Implemented unittests to validate `process` method behavior under error, retry, and pagination conditions.
- Introduced safeguards against infinite loops in pagination and retries within `SearchGithubCode`.
Introduce the `next_page_or_end` method in `githubcode.py` to handle pagination. Additionally, modify the test case and upgrade pytest settings in `pyproject.toml` to the latest versions, ensuring compatibility and updated configurations.
* Added call to api endpoint to explicitly gather subdomains from zoomeye, updated user agents, replaced orjson with ujson, and fixed substring not found error.
* Updated orjson to ujson.
* Fixed semantic error in html check in google workaround.
* Fixed flake8 errors.
* Fixed VT to use API.
* Fixed virustotal module.
* Fixed possible edge case that could possibly cause an infinite loop.
* Removed broken modules.
* Added 4 new sources: brave, criminalip, hunterhow, and subdomainfinderc99, added dnsresolve, and other misc changes.
* Added TODO comment.
* Fixed bin\theHarvester to allow python3.9
* Pep8 newline at end of file.
* Fixed error when passing in comma seperated resolvers and allow for user to pass in --dns-resolve flag with no arguments to use default resolvers that user has.
NotoriousRebel