Adds a new passive source that calls the Dymo data verifier endpoint
(https://api.tpeoficial.com/v1/private/secure/verify) with the target
domain. The response is stored on the source for downstream inspection
and any matching canonical domain or didYouMean suggestion is added to
the host set so the rest of the pipeline can resolve it.
The source follows the existing plugin shape: SearchDymo lives in
theHarvester/discovery/dymosearch.py, the API key is read from the
``dymo`` block in api-keys.yaml via Core.dymo_key(), and the engine is
registered in get_supportedengines() and the -b/--source dispatcher.
Tests cover the missing-key path, header/payload shape, the empty,
unrelated-suggestion and non-dict-response branches, and the
get_supportedengines() registration. Closes#2300.
The async worker that drains the engine queue catches `Exception` with
no `as` binding and prints the bare line
"A error occurred while processing a 'work item'."
That message both swallows the actual error class+text (so users
report bugs like #2227 with no actionable detail and the maintainer
has to ask them to repro under more verbosity) and reads "A error"
instead of the grammatical "An error".
Bind the exception, print its type and message, and fix the article.
The shodan integration test (`tests/discovery/test_shodan_engine.py`)
already asserts the message is not in output on the happy path; the
assertion is updated to match the new string.
Closes#2227.
BuiltWith API returns 'text/json; charset=utf-8' as the Content-Type
header instead of 'application/json'. aiohttp's response.json() raises
a ContentTypeError by default when the MIME type is not application/json.
Passing content_type=None tells aiohttp to skip the content-type check
and parse the response body as JSON regardless of the declared MIME type.
Fixes#2235
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Introduced comprehensive security tests, including CORS configuration, XML injection prevention, path traversal prevention, and information disclosure protection.
- Improved input sanitization for filenames and XML content to mitigate injection attacks and ensure safe handling.
- Updated CORS middleware to restrict HTTP methods and block credentials with wildcard origins.
- Removed traceback exposure in API error responses, even in debug mode.
- Verified rate limiting and input validation for sensitive API endpoints.
- Ensured no hardcoded secrets are present in the main codebase.
* feat(discovery): add THC subdomain enumeration module
Add new THC module integrating ip.thc.org for subdomain discovery.
Features:
- Free service, no API key required
- No rate limits
- Fast plain-text response parsing
Includes 24 comprehensive tests covering:
- API endpoint validation
- Subdomain search functionality
- Edge cases (empty, unicode, special chars)
- Proxy support
- Response format validation
- Integration with theHarvester framework
Closes#2217
* fix: translate test comments to English
Accidentally pushed the Spanish version of the test file.
This commit updates all docstrings and comments to English.
* fix: correct rate limit documentation and add rate limit header test
- Remove 'no rate limits' claim from README.md
- Add test_api_returns_rate_limit_headers to verify x-ratelimit-* headers
* feat: add explicit rate limit handling
Based on bravesearch.py pattern:
- Detect HTTP 429 status code
- Read x-ratelimit-remaining header
- Implement retry logic with backoff (3 retries, 2s base delay)
- Handle rate limit exceptions
Added test for rate limit settings initialization.
- Implemented unittests to validate `process` method behavior under error, retry, and pagination conditions.
- Introduced safeguards against infinite loops in pagination and retries within `SearchGithubCode`.
Introduce the `next_page_or_end` method in `githubcode.py` to handle pagination. Additionally, modify the test case and upgrade pytest settings in `pyproject.toml` to the latest versions, ensuring compatibility and updated configurations.
* Added call to api endpoint to explicitly gather subdomains from zoomeye, updated user agents, replaced orjson with ujson, and fixed substring not found error.
* Updated orjson to ujson.
* Fixed semantic error in html check in google workaround.
* Fixed flake8 errors.
* Fixed VT to use API.
* Fixed virustotal module.
* Fixed possible edge case that could possibly cause an infinite loop.
* Removed broken modules.
* Added 4 new sources: brave, criminalip, hunterhow, and subdomainfinderc99, added dnsresolve, and other misc changes.
* Added TODO comment.
* Fixed bin\theHarvester to allow python3.9
* Pep8 newline at end of file.
* Fixed error when passing in comma seperated resolvers and allow for user to pass in --dns-resolve flag with no arguments to use default resolvers that user has.
* add pytest.ini file and fix test warning in qwantsearch
* ignore unsed imports
* Remove n45ht as the api is down
* Remove modules that use google as google blocks you after one go
* Remove missed code from when removing modules that use google
* Remove missed code from when removing modules that use google
* Add new vt-py dep and update pytest
* WIP virustotal migration to api
* Remove test that fails due to how the api returns not found entries
* Remove entries from myparser re google
* update fastapi and starlette
* update version
* Update dockerfile to use dev ubuntu version
* remove spyse module
* remove spyse dep
* fix tests by removing un-needed tests and indentation fix
* fix some mypy errors
* remove spyse and remove color output and fix mypy error
* flake8 fixes
* bump version and set it a dev version
* Add new fullhunt module
* Remove python checks in the run script as they are not needed anymore
* Remove netcraft from ci as it is not needed
* Update Dockerfile to fix security issues
* Add new n45ht module, lots of results returned
Alisson Moretto