WEBDIENSTE/theHarvester
1
0
Fork 0
mirror of https://github.com/laramies/theHarvester.git synced 2026-08-17 19:35:40 +02:00
Code Issues Packages Projects Releases Wiki Activity
4,706 Commits 3 Branches 38 Tags
f44807a0468fd75bfa42b00a02c303973108b4e2
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
MattandGitHub f44807a046 refactor: run discovery sources with TaskGroup (#2544) 
* refactor: run discovery sources with TaskGroup

* fix: preserve source runner compatibility diagnostics

* feat: add configurable source workers
 2026-08-15 01:48:33 -04:00
MattandGitHub 4c5e2df9a7 refactor: modernize event-loop bootstrap (#296) (#2538) 2026-08-14 23:16:47 -04:00
MattandGitHub 4fef2c5e3b refactor: bound screenshot lifecycle ownership (#2543) 
* refactor: bound screenshot lifecycle ownership (#293)

* fix: retain HTTP error screenshots (#293)
 2026-08-14 23:10:34 -04:00
MattandGitHub 38e32de023 fix: complete API endpoint scans by default (#2539) 
* fix: complete API endpoint scans by default

* refactor: reuse shared cancellation cleanup
 2026-08-14 23:03:28 -04:00
MattandGitHub 1a9b114a5d fix: complete DNS enumeration by default (#2542) 
* fix: complete DNS enumeration by default

* test: avoid URL substring sanitizer alert
 2026-08-14 22:54:25 -04:00
MattandGitHub bb77613a05 fix: call Censys Platform API directly (#2537) 
* fix: call Censys Platform API directly (#295)

(cherry picked from commit 7ce361413ad35471ab0b6878382f3548b0918dbe)

* docs: clarify Censys global search entitlement
 2026-08-14 22:44:45 -04:00
MattandGitHub 3e864d3984 fix: update BuiltWith to current Domain API (#2536) 
* fix: update BuiltWith to current Domain API (#281)

* refactor: move hostname normalization to shared module
 2026-08-14 22:04:02 -04:00
MattandGitHub b8a8f7c7ca fix: move Shodan discovery into adapter (#2535) 
* fix: move Shodan discovery into adapter (#283)

* docs: clarify Shodan transport and persistence

* fix: query every Shodan-resolved IPv4

* fix: call Shodan Host API directly

* fix: serve HarvestView assets locally

* Revert "fix: serve HarvestView assets locally"

This reverts commit 413e8b25ab.

* docs: clarify Shodan proxy transport

* docs: clarify Shodan changelog entry

* feat: persist structured Shodan host evidence

* feat: expand Shodan discovery with TLS search
 2026-08-14 21:11:12 -04:00
MattandGitHub 259455f1c1 fix: remove Windvane DNS guessing fallback (#2541) 2026-08-14 15:14:53 -04:00
MattandGitHub 7c4420150d Add route-aware no-host result filtering (#2534) 2026-08-13 11:14:55 -04:00
MattandGitHub 624066a3e8 Add RouteViews enrichment and sourced ASN organization attribution (#2528) 
* Add RouteViews network enrichment

* Add authenticated RouteViews API access

* Restrict RouteViews to explicit network pivots

* Restore discovered ASN RouteViews enrichment

* Retain sourced ASN organization attribution

* Avoid dangling and repeated ASN attributions

* Keep automatic RouteViews evidence target-relevant

* Complete target-relevant RouteViews enrichment

* Clarify automatic RouteViews IP pivots
 2026-08-13 10:46:54 -04:00
MattandGitHub 1db27309dd Remove duplicate Chaos discovery source (#2533) 2026-08-12 18:47:34 -04:00
MattandGitHub aed50e0b0d Add crt.name composite hostname discovery (#2531) 2026-08-12 17:05:12 -04:00
MattandGitHub 4643f6f0f9 fix: use explicit provider and browser user agents (#2530) 2026-08-12 17:01:57 -04:00
MattandGitHub 38524ab674 Add Sourcegraph hostname candidates from indexed code (#2524) 
* feat: add bounded Sourcegraph discovery

Use the documented anonymous streaming API with explicit keyword, result, timeout, and line-match bounds; retain only scoped hostname and email evidence with truthful source outcomes.

* Harden Sourcegraph hostname discovery

* Raise Sourcegraph recall budget to 5000

* docs: clarify Sourcegraph candidate semantics

* test: account for combined source catalog
 2026-08-12 12:14:02 -04:00
MattandGitHub 91ed6c1d9f Add APIs.guru hostname, email, and URL discovery (#2525) 
* feat: add bounded APIs.guru discovery

* docs: clarify APIs.guru result boundaries
 2026-08-12 12:06:04 -04:00
MattandGitHub f13280ddad Add typed ASN-prefix routing evidence (#2527) 
* Add typed network evidence foundation

* Deepen network evidence accumulation

* Bound network evidence accumulator limits

* Make nested JSON rejection test version tolerant

* Keep network evidence in unreleased schema v8

* Clarify structured result persistence dispatch
 2026-08-12 10:49:16 -04:00
MattandGitHub ac3c543768 Add bounded provider response streaming (#2526) 
* Add bounded provider response streaming

* Add bounded provider JSON fetching
 2026-08-12 09:41:13 -04:00
MattandGitHub dd93a0e63d fix: persist missing credential outcomes (#2523) 2026-08-12 09:40:37 -04:00
MattandGitHub 801460f835 Fix passive provider failure attribution and parsing (#2518) 
* fix: harden passive provider reliability

* fix: restore HackerTarget reverse DNS coverage
 2026-08-11 17:04:11 -04:00
MattandGitHub fa8afa5e46 Add bounded virtual-host discovery (#2517) 
* Add bounded virtual-host discovery

* Document virtual host probe boundary
 2026-08-11 00:43:04 -04:00
MattandGitHub d766c7c59a Package HarvestView for local deployment (#2513) 
* feat: package HarvestView for local deployment

* Align container smoke with JSONL API

* Align container smoke with unversioned JSONL

* refactor: rename web launcher to harvestview
 2026-08-10 00:19:00 -04:00
MattandGitHub 0d2ab5a0b3 Add the HarvestView web application (#2512) 
* feat: add HarvestView operator UI

* Use JSONL-only HarvestView file flows

* Drive HarvestView activities from the API catalog

* Add result action controls

* Render shared DNS resolver defaults

* Clarify resolver cardinality in HarvestView

* Load Tabulator from CDNjs

* Fix HarvestView wiki link

* Align HarvestView with execution status contract

* Add HarvestView import and action controls

* Harden HarvestView browser assertions

* Simplify HarvestView run selection

* Show truthful HarvestView execution outcomes

* refactor: show canonical URL results in HarvestView

* Show canonical hostname results in HarvestView

* test(harvestview): remove browser error race
 2026-08-10 00:08:05 -04:00
MattandGitHub 1fd5749e52 Add an authenticated durable run API v1 (#2511) 
* Model active evidence in result persistence

* Expose active action diagnostics

* Persist truthful DNS action evidence

* Persist direct action evidence

* feat: add authenticated durable run API v1

* Make API file interchange JSONL-only

* Harden API evidence boundaries

* Remove API rate limiter

* Unify API runs with result persistence

* Support target action runs

* Keep DNS resolver selection action-neutral

* Add action-neutral CLI resolver selection

* Preserve API evidence across JSONL round trips

* Complete HarvestView API run parity

* Harden HarvestView run API contracts

* Remove obsolete bundled network snapshots

Delete the unused bundled AWS IP-range and resolver snapshots while preserving operator-supplied resolver file input.

* refactor: canonicalize URL results

* docs: define a bounded test budget

* Standardize hostname and IP result names

* fix(api): avoid duplicate evidence conflicts
 2026-08-10 00:08:04 -04:00
MattandGitHub e4d37b05a8 Persist per-source run provenance (#2516) 
* Persist per-source run provenance

* Keep the first JSONL contract unversioned

* Explain source yield counts
 2026-08-09 11:50:08 -04:00
MattandGitHub c4af833da0 Use SQLAlchemy for async result persistence (#2515) 
* Use SQLAlchemy for async result persistence

* Explain persistence ORM records

* Clarify persisted run schema

* Clarify legacy schema migration

* Consolidate result persistence

* Preserve incomplete legacy observations
 2026-08-09 00:50:39 -04:00
MattandGitHub 6b02dbddc9 Share safe enumeration execution across transports (#2507) 
* feat: share safe enumeration execution across transports

* fix: clarify safe scan contracts

* fix: preserve public screenshot redirects

* fix: restore operator-controlled active requests

* fix: allow operator-selected API scan targets
 2026-08-07 16:02:16 -04:00
MattandGitHub 7a854eda95 Contract DeHashed and LeakIX evidence (#2506) 
* Contract DeHashed evidence handling

* Align LeakIX with its documented API

* Update provider capability selection

* Require REST auth for DeHashed results
 2026-08-07 00:25:54 -04:00
MattandGitHub 30ac342d9d Remove unavailable Venacus source (#2505) 2026-08-07 00:04:47 -04:00
MattandGitHub ce791dcaca Contract Leak-Lookup evidence handling (#2504) 
* Contract Leak-Lookup evidence

* Preserve additional API compatibility
 2026-08-07 00:01:26 -04:00
MattandGitHub d52e8b1293 Contract Hunter and Tomba provider responses (#2503) 
* Contract Hunter and Tomba pagination

* Honor Hunter and Tomba result windows

* Handle Hunter and Tomba response failures

* Preserve Tomba results across later failures

* Preserve Hunter results across later failures
 2026-08-06 23:36:33 -04:00
MattandGitHub eb7583227a Expose authenticated recursive DNS through REST (#2502) 
* Expose recursive DNS through REST

* Align REST recursive DNS limits and evidence

* Document hard recursive DNS query limits

* Reject non-finite recursive DNS REST runtimes

* Require operator authentication for REST recursion
 2026-08-06 22:53:02 -04:00
MattandGitHub d42322031c Expose bounded recursive DNS discovery in the CLI (#2501) 
* Integrate recursive DNS with CLI persistence

* Expose bounded PTR evidence in recursive DNS output

* Document the CLI hard query default

* Clarify recursive DNS query accounting

* Preserve dual-stack recursive DNS output

* Preserve recursive DNS CLI lifecycle contracts

* Deduplicate exact XML host address pairs
 2026-08-06 22:40:30 -04:00
MattandGitHub 2398bfd15b Add bounded recursive DNS discovery engine (#2500) 
* Add bounded recursive DNS engine

* Retain bounded PTR evidence in recursive DNS

* Count actual recursive DNS queries

* Type recursive DNS resolver capabilities

* Bound recursive DNS candidate generation

* Reject unbounded recursive DNS runtimes
 2026-08-06 22:35:14 -04:00
MattandGitHub 7baaf62874 Add exact DNS query budgets and PTR lookups (#2499) 
* Add exact PTR queries to DNS resolver vantages

* Enforce DNS query budgets across CNAME chains

* Harden DNS query budget contracts
 2026-08-06 22:27:17 -04:00
MattandGitHub 8d8bbdc060 Preserve POST semantics through proxies (#2498) 2026-08-06 21:32:02 -04:00
MattandGitHub 9ff9115dcf Migrate Pentest-Tools discovery to API v2 (#2497) 
* Migrate Pentest-Tools discovery to API v2

* Send Pentest scan payload as JSON
 2026-08-06 21:27:30 -04:00
MattandGitHub b6bd951c74 Add DNS consensus classification (#2495) 
* Add DNS consensus classification

* Document DNS consensus behavior
 2026-08-06 20:17:11 -04:00
MattandGitHub 4f76b93513 Preserve direct DNS source result contracts (#2494) 
* Harden explicit DNS direct source contracts

* Reject mismatched Shodan InternetDB evidence

* Preserve valid direct-source IP evidence

* Avoid ambiguous hostname output assertions

* Reference current Pentest-Tools scan documentation
 2026-08-06 20:16:23 -04:00
MattandGitHub a2c28bad36 Honor DNS resolution for RapidDNS hosts (#2496) 2026-08-06 19:03:59 -04:00
MattandGitHub d60afe12e1 Retain GitLab evidence URLs (#2492) 2026-08-06 15:39:26 -04:00
MattandGitHub 61d39dda3a feat: add Arquivo.pt passive source (#2490) 2026-08-06 14:47:33 -04:00
MattandGitHub 16c4866b49 fix: drive all from passive source catalog (#2489) 2026-08-06 14:47:33 -04:00
MattandGitHub a7f883e45c feat: add HIBP verified-domain source (#2488) 2026-08-06 14:47:33 -04:00
MattandGitHub ffd03b13ba fix: make FullHunt outcomes explicit (#2486) 2026-08-06 12:01:17 -04:00
MattandGitHub 41b9a2e28b Make Fofa provider outcomes explicit (#2485) 
* fix: make Fofa outcomes explicit

* fix: classify Fofa provider errors
 2026-08-06 11:59:07 -04:00
MattandGitHub 4ab4e88121 fix: make Chaos outcomes explicit (#2484) 2026-08-06 11:56:41 -04:00
MattandGitHub c8afaeefea feat: expose completed run history over REST (#2483) 2026-08-06 11:09:01 -04:00
MattandGitHub 873f649902 feat: add Shodan CT passive source (#2482) 2026-08-06 09:50:49 -04:00
MattandGitHub e1d2236685 fix: persist terminal evidence for REST queries (#2481) 2026-08-06 09:50:15 -04:00
MattandGitHub ab314b0274 feat: support REST source capability selectors (#2480) 2026-08-06 09:44:47 -04:00
MattandGitHub 2e233c4c35 Merge pull request #2479 from NotoriousRebel/codex/hudsonrock-public-contract 
Make Hudson Rock provider outcomes truthful
 2026-08-06 00:05:51 -04:00
NotoriousRebel 3a0bc74174 fix: isolate malformed Hudson Rock fields 2026-08-06 00:02:49 -04:00
NotoriousRebel 1b4268888e fix: retain Hudson Rock stealer results 2026-08-05 23:56:40 -04:00
NotoriousRebel 78babf5eee fix: make Hudson Rock outcomes truthful 2026-08-05 23:25:20 -04:00
MattandGitHub 3739ae7255 Merge pull request #2478 from NotoriousRebel/codex/hibp-public-breach-contract 
Fix public Have I Been Pwned breach catalogue contract
 2026-08-05 23:06:47 -04:00
NotoriousRebel 35b368b705 fix: retain HIBP breach names in completed results 2026-08-05 23:02:36 -04:00
NotoriousRebel 141c329c3a Fix public HIBP breach catalogue contract 2026-08-05 22:25:09 -04:00
MattandGitHub 037fe53068 Merge pull request #2477 from NotoriousRebel/codex/remove-invalid-bitbucket-source 
Remove invalid Bitbucket domain source
 2026-08-05 22:09:05 -04:00
NotoriousRebel e063c42a08 fix: remove invalid Bitbucket domain source (#100) 2026-08-05 21:45:08 -04:00
MattandGitHub ae009d87df Merge pull request #2476 from NotoriousRebel/codex/thc-bounded-rate-limit-contract 
Stop THC retries before the final sleep
 2026-08-05 21:33:55 -04:00
NotoriousRebel 09981d6ebf fix: stop THC retries before final sleep (#98) 2026-08-05 21:31:04 -04:00
MattandGitHub 5bf26fdf3b Merge pull request #2475 from NotoriousRebel/codex/remove-dead-threatcrowd-source 
Remove nonfunctional ThreatCrowd source
 2026-08-05 21:29:18 -04:00
MattandGitHub 4b14bca0b0 Merge pull request #2474 from NotoriousRebel/codex/public-passive-http-status 
Attribute public provider HTTP failures
 2026-08-05 21:28:22 -04:00
NotoriousRebel 7af23e4382 fix: remove nonfunctional ThreatCrowd source (#98) 2026-08-05 20:48:35 -04:00
NotoriousRebel b2c2923dbb fix: attribute public provider HTTP failures (#98) 2026-08-05 20:40:36 -04:00
MattandGitHub c8d4cbebaa Merge pull request #2473 from NotoriousRebel/codex/otx-bounded-rate-limit-retry 
Bound OTX rate-limit retries
 2026-08-05 19:10:12 -04:00
NotoriousRebel 74cf9730b7 fix: bound OTX rate-limit retries (#98) 2026-08-05 18:59:57 -04:00
MattandGitHub 8ba887f3ec Merge pull request #2472 from NotoriousRebel/codex/public-passive-contracts 
Harden public passive provider contracts
 2026-08-05 18:43:59 -04:00
MattandGitHub abb065c95a Merge pull request #2471 from NotoriousRebel/codex/http-response-metadata-v2 
Preserve shared HTTP response metadata
 2026-08-05 18:43:00 -04:00
NotoriousRebel 66e1beee7c feat: preserve shared HTTP response metadata 2026-08-05 18:31:36 -04:00
NotoriousRebel 0cab96c297 test: harden public passive provider contracts (#98) 2026-08-05 18:17:20 -04:00
MattandGitHub 38983bd76a Merge pull request #2470 from NotoriousRebel/codex/screenshot-cname-candidates 
fix: retain CNAME screenshot candidates
 2026-08-05 17:03:56 -04:00
MattandGitHub d9f097843d Merge pull request #2469 from NotoriousRebel/codex/completed-generic-people 
fix: preserve generic people in completed results
 2026-08-05 17:03:27 -04:00
MattandGitHub f156f757b5 Merge pull request #2468 from NotoriousRebel/codex/api-endpoint-https-fallback 
fix: narrow API endpoint HTTP fallback
 2026-08-05 17:02:52 -04:00
MattandGitHub 3694305683 Merge pull request #2467 from NotoriousRebel/codex/live-smoke-mozilla 
test: use Mozilla for live provider smoke
 2026-08-05 17:02:06 -04:00
NotoriousRebel e3f3584707 fix: retain CNAME screenshot candidates 2026-08-05 16:25:32 -04:00
NotoriousRebel 8d3876724e fix: preserve generic people in completed results 2026-08-05 16:25:27 -04:00
NotoriousRebel f30d655082 fix: narrow API endpoint HTTP fallback 2026-08-05 16:25:23 -04:00
NotoriousRebel 4e9f5f4212 test: use Mozilla for live provider smoke 2026-08-05 16:25:19 -04:00
MattandGitHub 5f8b299faf Merge pull request #2466 from NotoriousRebel/codex/upstream-dev-gitlab-public-bounds-v2 
fix: normalize GitLab public discovery
 2026-08-05 15:56:39 -04:00
NotoriousRebel eeff3816c3 fix: normalize GitLab public discovery (#243) 2026-08-05 15:51:38 -04:00
MattandGitHub 82aa420c33 Merge pull request #2465 from NotoriousRebel/codex/upstream-dev-github-fragment-bounds-v2 
fix: bound GitHub code-search fragments
 2026-08-05 15:49:28 -04:00
NotoriousRebel de025aefbe fix: bound GitHub code-search fragments (#242) 2026-08-05 13:53:14 -04:00
MattandGitHub 5b6863bc06 Merge pull request #2464 from NotoriousRebel/codex/upstream-dev-web-search-contracts-v2 
fix: harden web search provider contracts
 2026-08-05 13:26:04 -04:00
NotoriousRebel ec4a8d920f fix: harden web search provider contracts (#99) 2026-08-05 13:16:19 -04:00
MattandGitHub 2aec88eb2f Merge pull request #2463 from NotoriousRebel/codex/upstream-dev-completed-persistence 
feat: persist completed results
 2026-08-05 12:56:47 -04:00
NotoriousRebel 48d5df237d feat: persist completed results 
Store and load CompletedResult values transactionally without changing legacy rows or public output contracts.
 2026-08-05 12:06:15 -04:00
MattandGitHub 1fa7b46802 Merge pull request #2462 from NotoriousRebel/codex/upstream-dev-screenshot-www 
fix: preserve www and prefer HTTPS screenshots
 2026-08-05 12:05:07 -04:00
NotoriousRebel 3d5a89d064 fix: preserve www and prefer HTTPS screenshots 2026-08-05 11:29:16 -04:00
MattandGitHub 8d5a7cceea Merge pull request #2461 from NotoriousRebel/codex/upstream-dev-completed-jsonl 
feat: completed-result JSONL reports
 2026-08-05 11:14:46 -04:00
MattandGitHub 66ba185560 Merge pull request #2460 from NotoriousRebel/codex/upstream-dev-typed-dns-evidence-v2 
fix: retain typed DNS evidence
 2026-08-05 10:44:57 -04:00
MattandGitHub a2a5945d16 Merge pull request #2459 from NotoriousRebel/codex/upstream-dev-provider-ip-evidence 
fix: validate provider IP evidence
 2026-08-05 10:44:11 -04:00
MattandGitHub e153203aa6 Merge pull request #2458 from NotoriousRebel/codex/upstream-dev-provider-hostname-evidence 
fix: preserve hostname evidence
 2026-08-05 10:43:29 -04:00
NotoriousRebel 1142ed5172 fix: preserve www hostname evidence 2026-08-05 10:33:44 -04:00
NotoriousRebel 708405c882 feat: add completed-result JSONL reports 2026-08-05 01:59:08 -04:00
NotoriousRebel 24b36ba590 fix: retain typed DNS evidence 2026-08-05 01:29:18 -04:00
NotoriousRebel eb11e9be9f fix: validate provider IP evidence 2026-08-05 01:12:52 -04:00
NotoriousRebel 4db22f070d fix: preserve provider hostname evidence 2026-08-05 01:06:29 -04:00
MattandGitHub d5d9516b3d Merge pull request #2457 from NotoriousRebel/codex/upstream-dev-config-copy-logging 
test: isolate configuration copy logging
 2026-08-04 23:36:32 -04:00