WEBDIENSTE/maigret
1
0
Fork 0
mirror of https://github.com/soxoj/maigret.git synced 2026-08-17 19:25:41 +02:00
Code Issues Packages Projects Releases Wiki Activity
1,274 Commits 18 Branches 39 Tags
e962b8c693814bc7d375e2ee503b68f62b9037f6
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
SoxojandGitHub e962b8c693 Fix site checks: 5 fixed; readme fix (#2562) 
* Fix site checks: 5 fixed; readme fix

* Logging improvements

* Improve YouTube data extraction
 2026-04-25 18:15:38 +02:00
SoxojandGitHub 77c11df119 Fix Google Cloud Shell launch (#2557) 2026-04-23 21:45:27 +02:00
SoxojandGitHub 25026e21ea Fix site checks: 4 → ip_reputation, 9 fixed, 16 disabled, 3 dead dele… (#2555) 
* Fix site checks: 4 → ip_reputation, 9 fixed, 16 disabled, 3 dead deleted; clarify ip_reputation tag semantics

* Improved test coverage
 2026-04-23 21:17:07 +02:00
SoxojandGitHub b1004588af AI mode (#2529) 
* Add AI mode
 2026-04-23 12:12:54 +02:00
SoxojandGitHub 5e1cc45c17 Fix site checks: 12 fixed, 19 disabled; add new protection tags (#2550) 2026-04-22 20:25:41 +02:00
SoxojandGitHub d9b361b626 Fix site checks: 3 → ip_reputation, 10 fixed, 6 disabled, 2 dead deleted (#2549) 2026-04-22 12:46:53 +02:00
SoxojandGitHub 0131f0b64c Add OnlyFans with activation mechanism; updated site ranks (#2546) 2026-04-21 19:03:45 +02:00
SoxojandGitHub e8393bfce3 Fix site checks: 3 fixed, 2 → ip_reputation, 7 disabled, 1 dead deleted (#2543) 2026-04-21 16:02:36 +02:00
SoxojandGitHub 98f03c153b Add 3 crypto sites (Polymarket, Zora, Revolut.me), added crypto inves… (#2538) 
* Add 3 crypto sites (Polymarket, Zora, Revolut.me), added crypto investigation use case page in docs

* Added fintech tag

* Updated sites metadata
 2026-04-21 11:08:48 +02:00
SoxojandGitHub 1f823e8322 Fix site checks: 3 fixed, 2 → ip_reputation, 7 disabled, 1 dead deleted (#2539) 2026-04-21 10:58:45 +02:00
SoxojandGitHub d6905a8fd8 Fix site checks: 4 fixed, 14 → ip_reputation, 8 disabled, 5 dead deleted (#2537) 2026-04-21 00:40:24 +02:00
SoxojandGitHub 7d216638fa fix site checks: 14 sites → ip_reputation, 7 disabled, 5 dead deleted (#2536) 2026-04-20 23:51:18 +02:00
SoxojandGitHub fb71f26fd0 Fix site checks: recover 6 CF sites via tls_fingerprint, 500px GraphQL, delete 4 dead domains (#2535) 2026-04-20 22:41:51 +02:00
SoxojandGitHub 37ce4fe728 Update of Readme and documentation (#2514) 
* Big readme update

* Readme and documentation update

* Readme structure update

* Small fixes

* Changelog update
 2026-04-17 17:42:36 +02:00
SoxojandGitHub f74f82ee13 Fixed: Hack MD, DailyKos, Mywed, WikimapiaSearch, TikTok Online Viewer (#2526, #2522, #2523, #2500, #2496); Disabled: Radiokot, Lurkmore, Mylespaul, AppleDiscussions, Loveplanet (#2524, #2511, #2498) (#2528) 2026-04-17 17:04:50 +02:00
SoxojandGitHub 8267367bed Support Python 3.14 in tests (#2515) 2026-04-11 15:11:09 +02:00
SoxojandGitHub 7fd9bb3692 Update workflow to trigger on published releases (#2508) 2026-04-10 12:34:00 +02:00
SoxojandGitHub 385f9f5bb3 Bump to 0.6.0 (#2506) 2026-04-10 12:30:43 +02:00
SoxojandGitHub dc8751ac55 Added 3 sites, fixed 6, disabled 8 (#2505) 2026-04-10 12:26:41 +02:00
SoxojandGitHub 5e24117e93 Fix false positives (#2499) 
* Re-disable 29 false positives from #2478
 2026-04-09 17:48:45 +02:00
SoxojandGitHub 777e503e30 Re-enable 69 stale-disabled sites validated via self-check (#2478) 
Total: 2539 → 2608 enabled sites (+69).
 2026-04-09 12:27:48 +02:00
SoxojandGitHub b213f6e079 vBulletin cleanup, Flarum sites, engine stats, UA bump (#2476) 2026-04-09 01:17:24 +02:00
SoxojGitHubcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
 		6db1df2ddb Fix failing test for custom DB path resolution (#2468) 
* Fix `--db` bug

* Fix test_resolve_db_path_custom_file to create the file before testing

Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/3ea7b2e8-0565-4fca-8ec2-eff8eb4ee617

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
 2026-04-08 00:53:57 +02:00
SoxojandGitHub 6834483360 Fix Spotify, add Spotify Community forum (#2467) 2026-04-07 18:25:13 +02:00
SoxojandGitHub 3fd34afb77 Sites fixes (#2464) 2026-04-06 21:41:16 +02:00
SoxojandGitHub ad95302745 Add Markdown reports for LLM analysis (#2463) 2026-04-06 18:26:43 +02:00
SoxojandGitHub 6d0a22b738 False positive fixes (#2460) 
* Fix false positives: APClips, Taplink, gentoo, Discord.bio, ChaturBate; disable 7Cups, playtime, openriskmanual, reactos; update tags

* Fix db_meta.json regeneration in update_site_data.py (inline instead of module import)

* Fix false positives: disable Bit.ly, Firearmstalk, Needrom, Travelblog; fix gentoo, Discord.bio, brickimedia via API; remove dead sites dreamhost, typepad
 2026-04-04 19:08:51 +02:00
SoxojandGitHub abce3c9be4 Fix false positives (#2459) 
* Fix false positives: APClips, Taplink, gentoo, Discord.bio, ChaturBate; disable 7Cups, playtime, openriskmanual, reactos; update tags

* Fix db_meta.json regeneration in update_site_data.py (inline instead of module import)
 2026-04-04 18:22:21 +02:00
SoxojandGitHub 269d50eedc DB update mechanism (#2458) 
* Database update mechanism
 2026-04-04 18:00:50 +02:00
SoxojandGitHub e8f4318e5d Added Crypto/Web3 site checks (#2457) 2026-04-04 16:49:12 +02:00
SoxojandGitHub 75289c78bf Update of MIT License (#2455) 2026-04-03 18:02:54 +02:00
SoxojandGitHub d136014576 Multiple lint and types fixes (#2454) 2026-04-02 21:01:49 +02:00
SoxojGitHubcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
 		5d502eaef6 Add site protection tracking system, fix broken site checks (Instagra… (#2452) 
* Add site protection tracking system, fix broken site checks (Instagram, StackOverflow, LeetCode, Boosty, LiveLib), preserve unicode in data.json

* Update poetry.lock by running poetry lock

Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/14333f41-67d5-4e28-a782-9730b31fc667

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
 2026-04-02 20:28:20 +02:00
SoxojGitHubClaude Opus 4.6
 		5aa7f6429b Overhaul site tags and naming: add social tag to 33 networks, fill mi… (#2430) 
* Overhaul site tags and naming: add social tag to 33 networks, fill missing tags for 213 top-1000 sites, clean up false us/in country tags (~374 sites), normalize site names to Title Case, add tag validation tests, document tagging and naming rules
Remove LLM folder: ask @soxoj for the up-to-date version!

* Remove LLM/ from version control

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-28 19:48:16 +01:00
SoxojandGitHub a5d337b765 Tags and site names improvements (#2427) 
- Added social tag to social networks (33 sites)
- Fixed wrong tags (8 sites)
- Filled empty tags for 213 sites in top-1000
- Country tag cleanup (~374 sites)
- Site naming normalization (75 sites)
- New tests (3)
- Documentation updates
 2026-03-28 15:42:12 +01:00
SoxojandGitHub 51b452ad71 Add urlProbes (#2425) 2026-03-28 00:08:02 +01:00
SoxojandGitHub fa1a4d1b4a Sites re-check (#2423) 2026-03-27 22:41:55 +01:00
SoxojandGitHub b333a2e2b2 Readme update: commercial use (#2403) 2026-03-26 21:51:53 +01:00
SoxojandGitHub 656fe1df24 Added Max.ru check; --no-progressbar flag fixed (#2386) 2026-03-25 11:48:12 +01:00
SoxojandGitHub bc3d9faad9 Fix false-positive site checks reported by Maigret Bot (#2376) 2026-03-24 23:01:11 +01:00
SoxojGitHubcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
 		b145e7b26f feat(core): add POST request support, new sites, migrate to Majestic Million ranking (#2317) 
* feat(core): add POST request support, new sites, migrate to Majestic Million ranking
- Added native POST request support to the Maigret engine (requestMethod, requestPayload) to enable querying modern JSON registration endpoints.
- Replaced the discontinued Alexa rank API with the Majestic Million dataset for global popularity sorting and automated CI updates.
- Fixed multiple false positives among top 500 sites and bypassed standard anti-bot protections using custom User-Agents.
- Updated public documentation and internal playbooks to reflect the new features.

* feat(data): apply all data.json site check updates from main branch

- Added CTFtime and PentesterLab (new sites added in main)
- Removed forums.imore.com (deleted in main as dead site)
- Disabled 5 sites per main branch fixes: Librusec, MirTesen, amateurvoyeurforum.com, forums.stevehoffman.tv, vegalab
- Fixed 5 site checks per main branch: SoundCloud, Taplink, Setlist, RoyalCams, club.cnews.ru (switched from status_code to message checkType with proper markers)

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/a1d194d9-c0ff-4e2b-974c-c5e4b59548bf

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
 2026-03-24 22:08:42 +01:00
SoxojandGitHub e81b50ef61 Update site data workflow fix: remove ambiguous main tag (#2313) 
* feat(workflow): fix update site data workflow err

* feat(workflow): the final update side data workflow fix (hopefully)
 2026-03-22 22:37:48 +01:00
SoxojandGitHub 9ac0a65914 feat(workflow): fix update site data workflow err (#2312) 2026-03-22 22:31:55 +01:00
SoxojandGitHub a17e0c7a13 feat(workflow): fix update site data workflow dependency (#2306) 2026-03-22 21:34:30 +01:00
SoxojandGitHub b8ada1c818 Update sites list workflow (#2303) 2026-03-22 20:59:37 +01:00
SoxojandGitHub 959b2be136 feat(sites): fix false positives: disable 74 broken sites, fix 8 with API probes and better markers (#2302) 
- Disable 74 sites: Cloudflare/captcha blocks, identical responses,
    dead domains, vBulletin/phpBB engine failures
  - Fix Roblox, Salon24.pl, Planetaexcel → status_code (clear 404 signal)
  - Fix en.brickimedia.org → message with "noarticletext" absenceStr
  - Fix Arduino → narrower title-based presenseStrs/absenceStrs
  - Re-enable Fandom (3 wikis) via MediaWiki api.php urlProbe
  - Re-enable Substack via /api/v1/user/{}/public_profile urlProbe
  - Re-enable hashnode via GraphQL GET urlProbe (URL-encoded query)
  - Document lessons: engine template drift, search-by-author fragility,
    always-200 sites, TLS degradation, API bypassing Cloudflare,
    GraphQL GET support, URL-encoding for template safety
 2026-03-22 20:47:51 +01:00
SoxojandGitHub 97cc4b46d9 Improve site-check quality: fix broken site configs, add diagnostic utilities, and make self-check report-only by default with opt-in auto-disable. (#2301) 
- Fix VK and TradingView checkType; add Reddit and Microsoft Learn API-style probes where appropriate; adjust or disable entries that are unreliable under anti-bot protection.
- Self-check: stop aggressive auto-disable; default to reporting issues only; add --auto-disable and --diagnose for optional fixes and deeper output.
- Tooling: add utils/site_check.py and utils/check_top_n.py (and related helpers) to inspect and rank site behavior against the top-N list
- Scope: aligns with fixing top-traffic / high-impact sites and making diagnostics repeatable without silently flipping disabled flags
 2026-03-22 16:48:35 +01:00
SoxojandGitHub f3b741d283 Update Telegram bot link in README (#2300) 2026-03-22 12:23:35 +01:00
SoxojandGitHub 227a25bfa1 Twitter fixed, mirrors mechanism improvement (#2299) 2026-03-22 01:14:17 +01:00
SoxojandGitHub 5da4e78092 Pyinstaller GitHub workflow fix (#2298) 2026-03-22 00:59:17 +01:00
SoxojandGitHub e4d6b064df Update Telegram bot link in README (#2293) 2026-03-21 23:49:45 +01:00
SoxojandGitHub f99091f5f7 Fixed false positives in top-500 (#2292) 2026-03-21 23:35:59 +01:00
SoxojandGitHub f26976f1dd Dockerfile fix (#2290) 2026-03-21 20:02:35 +01:00
SoxojandGitHub 48ca13dc4d Make web interface accessible for Docker deployment by default (#2189) 2025-08-31 16:14:42 +02:00
SoxojandGitHub fb26ccd1f6 Disabled some sites giving false positive results (#2170) 2025-08-22 03:10:47 +02:00
SoxojandGitHub f869eb49ca Updated workflows: added 3.13 to test, updated pypi upload (#2111) 2025-08-10 14:10:06 +02:00
SoxojandGitHub bebadb0362 Bump to 0.5.0 (#2108) 2025-08-10 13:10:50 +02:00
SoxojandGitHub 5e88fd9ba8 Fixed test dialog_adds_site_negative (#2107) 2025-08-10 12:44:05 +02:00
SoxojandGitHub c0e60e25b8 upload-artifact action in python test workflow updated to v4 (#2024) 
* upload-artifact action in python test workflow updated to v4
* Upload artifacts for all jobs
 2025-06-28 23:08:56 +02:00
SoxojandGitHub 97e5f600d0 Async generator-executor for site checks (#1978) 2024-12-17 22:48:11 +01:00
SoxojandGitHub c3dfe9cb4d Small docs and parameters fixes for web interface mode (#1973) 2024-12-16 17:18:22 +01:00
SoxojandGitHub 4894a267d7 Added web interface docs (#1972) 2024-12-16 17:06:06 +01:00
SoxojandGitHub cb01535565 Preparation of 0.5.0 alpha version (#1966) 2024-12-13 12:51:31 +01:00
SoxojandGitHub c4af0a4df0 Fixed flaky tests to check cookies (#1965) 2024-12-13 12:37:58 +01:00
SoxojandGitHub f113c3d21a Merge pull request #1963 from soxoj/dependabot/pip/pytest-asyncio-0.25.0 
Bump pytest-asyncio from 0.24.0 to 0.25.0
 2024-12-13 11:25:15 +01:00
SoxojandGitHub b2283a5b04 Merge pull request #1961 from overcuriousity/main 
fix bad linux filename generation
 2024-12-12 22:07:21 +01:00
SoxojandGitHub 1ed0c61b56 Merge pull request #1962 from soxoj/site-fixes-121224 
Site check fixes
 2024-12-12 21:56:45 +01:00
SoxojandGitHub 2653c617f8 Merge pull request #1958 from soxoj/gravatar-pypi-fix 
Fixed Gravatar parsing (socid_extractor)
 2024-12-12 02:32:35 +01:00
SoxojandGitHub 33588ff090 Merge pull request #1957 from eltociear/patch-1 
chore: update submit.py
 2024-12-12 01:08:44 +01:00
SoxojandGitHub 2c39cd0646 Merge pull request #1956 from soxoj/submit-improvements-sitefixes 
* Fixed Vimeo, activation/probing mechanisms improvements
* Updated CNET, DailyMotion
 2024-12-11 01:23:01 +01:00
SoxojandGitHub 81a817a39f Improved "submit new site" mode, added tests, fixed top-500 sites (#1952) 2024-12-10 18:02:43 +01:00
SoxojandGitHub 51ab988e36 Fixed ProductHunt check (#1951) 2024-12-09 17:06:03 +01:00
SoxojandGitHub 5517636850 Updated OP.GG checks (#1950) 
* Updated OP.GG checks
* Finalized LoL, added Valorant, disabled Archive.org
 2024-12-09 15:59:19 +01:00
SoxojandGitHub 2be6e02800 Update README.md (#1949) 2024-12-09 13:01:31 +01:00
SoxojandGitHub 4eada16b94 Added a test for submitter (#1944) 2024-12-08 13:35:27 +01:00
SoxojandGitHub c66d776f8a Refactoring, test coverage increased to 60% (#1943) 2024-12-08 02:13:28 +01:00
SoxojandGitHub 4b1317789d Refactored self-check method, code formatting, small lint fixes (#1942) 2024-12-07 18:05:30 +01:00
SoxojandGitHub 8b7d8073d9 Fixed Linktr and discourse.mozilla.org (#1941) 2024-12-07 17:11:39 +01:00
SoxojandGitHub 2aa1ea39a0 Site fixes (#1940) 2024-12-06 14:27:38 +01:00
SoxojandGitHub cd789ed138 Fixed Ebay and BongaCams checks (#1939) 2024-12-06 13:32:51 +01:00
SoxojandGitHub 5641456ba0 Weibo site check fix, activation mechanism added (#1938) 2024-12-06 11:31:20 +01:00
SoxojandGitHub f4edab8946 Readme/docs update based on GH discussions (#1936) 2024-12-06 01:54:26 +01:00
SoxojandGitHub f04de78682 Activation mechanism documentation added (#1935) 
Few site checks fixed
 2024-12-06 01:35:19 +01:00
SoxojandGitHub cb9f01c106 Fixed Figma check (#1932) 
Fixed cookies bug
Improved self-check mode: don't disable sites because of check errors
 2024-12-04 19:21:27 +01:00
SoxojandGitHub e701c881a1 Documentation update (#1931) 2024-12-04 16:58:56 +01:00
SoxojandGitHub d78aa02833 Updated workflow (#1930) 2024-12-04 15:52:27 +01:00
SoxojandGitHub 4e54a9b496 Put Windows executable in Releases for each dev and main commit (#1929) 2024-12-04 15:27:28 +01:00
SoxojandGitHub 1cb25946dd Disabled Figma check (#1928) 2024-12-04 00:27:55 +01:00
SoxojandGitHub e982be4109 Installation docs update (#1927) 2024-12-03 20:23:49 +01:00
SoxojandGitHub 07a7a474f8 Documentation update (#1926) 2024-12-03 17:25:17 +01:00
SoxojandGitHub 2f93963a0a Refactored sites module, updated documentation (#1918) 2024-12-01 11:41:41 +01:00
SoxojandGitHub 5073ceff13 Update README.md (#1919) 2024-12-01 11:40:02 +01:00
SoxojandGitHub d15e12750b Sites fixes (#1917) 
* Some sites fixes

* Sites stats updated
 2024-12-01 03:19:36 +01:00
SoxojandGitHub 21a8459b18 An recursive search animation in README has been updated (#1915) 2024-11-29 15:15:52 +01:00
SoxojandGitHub e96d09dee7 Permutator output and documentation updates (#1914) 2024-11-29 13:15:03 +01:00
SoxojandGitHub 15702bd9f4 Fixed dateutil parsing error for CDT timezone (#1907) 2024-11-29 12:02:41 +01:00
SoxojandGitHub 909a7e6a91 A new logo added (#1906) 
* New cool logo
* Badges updated
* Increased the size of logo
 2024-11-27 15:30:41 +01:00
SoxojandGitHub 2e2a47a12b Close http connections (#1595) (#1905) 2024-11-27 15:28:10 +01:00
SoxojandGitHub 8a98aa9eaa Retries set to 0 by default, refactored code of executor with progress (#1899) 
* Retries set to 0 by default, refactored code of executor with progress
 2024-11-26 19:07:15 +01:00
SoxojandGitHub ee25c61fc2 Maigret bot support (custom progress function fixed) (#1898) 
* Fixed progress/close functions
* Fixed tests: execution time increased with alive_progressbar
 2024-11-26 15:54:26 +01:00