theHarvester/CONTRIBUTING.md

# Contributing to theHarvester

Thank you for helping improve theHarvester. This guide explains how to propose a focused change, test it safely, and submit evidence that makes review straightforward.

## Before you start

- Search the [issues](https://github.com/laramies/theHarvester/issues) and [pull requests](https://github.com/laramies/theHarvester/pulls) for existing work.
- Open an issue before investing in a large feature, dependency change, or user-visible behavior change.
- Keep each pull request to one logical change. Leave unrelated cleanup and reformatting for separate work.
- Base contribution branches on the current upstream `dev` branch.

For a bug report, include a minimal reproduction, expected and actual behavior, operating system, Python version, and only the output needed to diagnose the problem. Remove credentials, account information, private target data, and raw API responses.

## Set up a development environment

theHarvester requires Python 3.14 and uses [uv](https://docs.astral.sh/uv/) for dependency management. The repository's `.python-version` lets `uv` select it automatically.

Fork the repository, then clone your fork and add the upstream repository:

```bash
git clone https://github.com/YOUR-GITHUB-USERNAME/theHarvester.git
cd theHarvester
git remote add upstream https://github.com/laramies/theHarvester.git
git fetch upstream
git switch -c fix/short-description upstream/dev
uv sync --all-groups
```

Use a short branch name that describes the change, such as `fix/certspotter-pagination` or `feature/source-name`.

## Make a focused change

- Match the surrounding code and reuse existing configuration, transport, parsing, and result-normalization helpers.
- Tests are especially helpful for bug fixes and new behavior, but they are not required for every contribution. If you can add one, start with the nearest existing test.
- Use descriptive names and type annotations where they improve the changed code.
- Do not add dependencies or configuration options unless the change requires them.
- You are responsible for every submitted change, including AI-assisted code. Read it, understand it, and test it before opening a pull request.

### Discovery providers

Register an ordinary provider with one `SourceSpec` catalog entry and one `SOURCE_FACTORIES` entry. Do not edit CLI orchestration, persistence, or output code: the catalog drives selection/help/activity metadata and the shared runner owns construction, collection, and completed-result output.

If you can add coverage for a new or changed discovery provider, keep it focused and mock HTTP, DNS, and provider responses. Tests must not require API keys or external network access. [The Baidu discovery tests](tests/discovery/test_baidusearch.py) are a small example you can copy and adapt.

Useful cases include:

- missing credentials and configuration;
- non-success responses, timeouts, and malformed or empty data;
- pagination and retry termination;
- normalized, deduplicated results.

Every canonical provider has one offline contract module marked with
`pytest.mark.provider_contract("source-name")`. The catalog-derived coverage
gate fails when a catalog entry has no contract, when a contract names an
unknown source, or when two modules claim the same source. Add the marker to
the provider's deterministic contract module; do not maintain a second source
list in tests.

Use the shared transport when it can represent the request. If a provider requires behavior the shared transport does not support, keep the exception local and explain it in the pull request. Never log credentials, account information, private target data, or raw API responses.

## Test safely

Run the narrowest affected test first:

```bash
uv run pytest tests/path/to/test_file.py
```

Before submitting, run the non-mutating quality checks and full test suite:

```bash
uv run ruff check .
uv run ruff format --check .
uv run pytest
uv run mypy theHarvester
```

Before a release, manually dispatch the **Release validation** workflow against
the exact release branch or tag. For example:

```bash
gh workflow run provider-smoke.yml --ref dev -f run_live=false
```

This composes the Python, real-browser HarvestView, package, and container checks
on clean GitHub-hosted runners. Maintainers with explicit authorization may set
`run_live=true` to add bounded P0 provider checks against `mozilla.org`; the live
lane never enables DNS or direct target interaction.

Routine verification must use mocks, local fixtures, and reserved example domains. The test harness blocks external Python socket traffic unless a test is marked `live_network` and pytest is invoked with both `--run-live-network` and `-m live_network`. A live-marked test never satisfies the provider-contract coverage gate.

Do not run broad or active reconnaissance against third-party targets. If live verification is essential, use only a target you own or are explicitly authorized to test, limit the request scope, and keep collected data out of commits, issues, and pull requests. The manually dispatched provider workflow uses `mozilla.org` for small passive CLI crash smokes. Those runs can detect packaging, credential, or provider drift; they are not conformance tests and should not be retried merely to obtain more results.

CI runs additional source smoke tests, CodeQL, dependency review, and container checks. Contributors do not need to reproduce broad live-source checks locally.

## Open a pull request

Push the topic branch to your fork and open a pull request against `laramies/theHarvester:dev`.

The pull request should include:

- the problem and relevant issue;
- the behavior before and after the change;
- the exact tests and checks run;
- sanitized output or screenshots only when they help demonstrate the result;
- any compatibility, provider, rate-limit, or operational risk reviewers should know about.

Use a draft pull request when work is incomplete or verification is still pending. Keep the branch current, respond to review feedback, and ensure required checks pass before requesting a final review.

## Security-sensitive reports

Follow [SECURITY.md](SECURITY.md) when reporting a suspected vulnerability. Do not include vulnerability details, credentials, or sensitive collected data in a public issue or pull request.