From c436174467f97e2c1b637a7b984a3e29f4b1fe69 Mon Sep 17 00:00:00 2001 From: Soxoj <31013580+soxoj@users.noreply.github.com> Date: Wed, 15 Jul 2026 17:23:53 +0200 Subject: [PATCH] Improve CONTRIBUTING.md (#2871) --- CONTRIBUTING.md | 10 ++++++++++ maigret/resources/data.json | 4 +++- maigret/resources/db_meta.json | 8 ++++---- sites.md | 21 +++++++++++---------- 4 files changed, 28 insertions(+), 15 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f720f57..262471c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -97,6 +97,16 @@ Each site entry uses one of three `checkType` modes to decide whether a profile Full reference for `checkType`, `urlProbe`, `engine`, and the rest of the `data.json` schema is in the [development guide](docs/source/development.rst), section *How to fix false-positives*. +### SPA / JS-rendered sites + +Modern sites often render the profile client-side, so a plain HTTP fetch returns an empty shell that trips both `message` and `status_code` checks. **Don't reach for a headless browser** — it's a heavy dependency for a problem that has three lighter answers. Work down the ladder in order: + +1. **XHR / GraphQL / RSC endpoint → `urlProbe`.** Open the profile in Chrome DevTools → Network, filter to `XHR`/`Fetch`, and look for the request that carries the actual profile JSON. Almost every SPA has one — that's what its own front-end calls. Put its URL in `urlProbe`; the human-readable profile URL stays in `url`. Watch for versioned APIs (`/v5/`, `/api/v2/`) — if the one you find starts returning 5xx or 404 for everyone, bump the version before disabling. +2. **JS-cookie / anti-bot challenge → `activation` handler.** If the first response is a challenge stub that sets a cookie via JavaScript and reloads, add a static method to `maigret/activation.py` that reproduces the token exchange with plain `aiohttp` (existing examples: `wikimapia`, `weibo`, `twitter`, `vimeo`, `onlyfans`, `proton`). Point `activation.method` at your function name and set `marks` to the challenge substring; Maigret retries the check once after `activation` fires. +3. **TLS-fingerprint block → `protection: ["tls_fingerprint"]`.** If the site 403's or serves a Cloudflare interstitial to plain `aiohttp` but works in a real browser, the check is being killed by JA3/JA4 fingerprinting. The tag routes the request through a Chrome-impersonating client — usually enough on its own, with no `disabled: true` needed. + +If none of these help — the site really does need a full JS runtime with DOM to render the profile signal — that's a case for issue [#579](https://github.com/soxoj/maigret/issues/579). Open a fresh issue naming the site so we can weigh a `checkType: browser` mode against its latency and dependency cost. + ### Editing `data.json` safely `data.json` is a single ~36 000-line JSON file. **Make surgical, line-level edits only.** Never rewrite it by reading it into a Python dict and dumping it back — `json.load` + `json.dump` reformats every entry and produces an unreviewable 70 000-line diff. The same rule applies to any helper script that touches the file: it must preserve the original formatting of untouched entries. diff --git a/maigret/resources/data.json b/maigret/resources/data.json index 886616c..20f3baf 100644 --- a/maigret/resources/data.json +++ b/maigret/resources/data.json @@ -27329,7 +27329,9 @@ "activation": { "url": "https://account.proton.me/api/auth/v4/sessions", "method": "proton", - "marks": ["Invalid access token"] + "marks": [ + "Invalid access token" + ] }, "urlMain": "https://proton.me/", "url": "https://account.proton.me/api/users/available?Name={username}", diff --git a/maigret/resources/db_meta.json b/maigret/resources/db_meta.json index 486b0fe..3f703e2 100644 --- a/maigret/resources/db_meta.json +++ b/maigret/resources/db_meta.json @@ -1,8 +1,8 @@ { "version": 1, - "updated_at": "2026-07-13T16:02:35Z", - "sites_count": 3187, + "updated_at": "2026-07-15T10:25:34Z", + "sites_count": 3188, "min_maigret_version": "0.5.0", - "data_sha256": "4eeed3b475a1ff4dce558bbca926d50adcb6b8e5372770f330e25baa4a252df0", + "data_sha256": "06b6f49f07344096e6ee827c6bc54c8391e8e97f51b350dbfe28c06075be40d7", "data_url": "https://raw.githubusercontent.com/soxoj/maigret/main/maigret/resources/data.json" -} \ No newline at end of file +} diff --git a/sites.md b/sites.md index 0fdebde..d5375d8 100644 --- a/sites.md +++ b/sites.md @@ -1,5 +1,5 @@ -## List of supported sites (search methods): total 3187 +## List of supported sites (search methods): total 3188 Rank data fetched from Majestic Million by domains. @@ -1494,6 +1494,7 @@ Rank data fetched from Majestic Million by domains. 1. ![](https://www.google.com/s2/favicons?domain=https://forum.f-droid.org) [F-droid (https://forum.f-droid.org)](https://forum.f-droid.org)*: top 100M, forum* 1. ![](https://www.google.com/s2/favicons?domain=http://forum.vn.ua) [forum.vn.ua (http://forum.vn.ua)](http://forum.vn.ua)*: top 100M*, search is disabled 1. ![](https://www.google.com/s2/favicons?domain=https://profile.ameba.jp) [Ameba (https://profile.ameba.jp)](https://profile.ameba.jp)*: top 100M, jp* +1. ![](https://www.google.com/s2/favicons?domain=https://amateur.tv/) [Amateur.tv (https://amateur.tv/)](https://amateur.tv/)*: top 100M, porn, webcam* 1. ![](https://www.google.com/s2/favicons?domain=https://forum.blackmagicdesign.com) [forum.blackmagicdesign.com (https://forum.blackmagicdesign.com)](https://forum.blackmagicdesign.com)*: top 100M, forum* 1. ![](https://www.google.com/s2/favicons?domain=https://hosted.weblate.org/) [Weblate (https://hosted.weblate.org/)](https://hosted.weblate.org/)*: top 100M* 1. ![](https://www.google.com/s2/favicons?domain=https://forum.sublimetext.com/) [SublimeForum (https://forum.sublimetext.com/)](https://forum.sublimetext.com/)*: top 100M, coding, forum* @@ -2266,7 +2267,7 @@ Rank data fetched from Majestic Million by domains. 1. ![](https://www.google.com/s2/favicons?domain=https://www.meendo.net) [Meendo (https://www.meendo.net)](https://www.meendo.net)*: top 100M, bg, kg, ru, ua* 1. ![](https://www.google.com/s2/favicons?domain=) [Megamodels.pl ()]()*: top 100M* 1. ![](https://www.google.com/s2/favicons?domain=http://megane2.ru/) [Megane2 (http://megane2.ru/)](http://megane2.ru/)*: top 100M, ru* -1. ![](https://www.google.com/s2/favicons?domain=) [ProtonMail ()]()*: top 100M* +1. ![](https://www.google.com/s2/favicons?domain=https://proton.me/) [ProtonMail (https://proton.me/)](https://proton.me/)*: top 100M* 1. ![](https://www.google.com/s2/favicons?domain=http://www.microchip.su) [Microchip (http://www.microchip.su)](http://www.microchip.su)*: top 100M, forum, ru*, search is disabled 1. ![](https://www.google.com/s2/favicons?domain=) [Minecraftlist ()]()*: top 100M* 1. ![](https://www.google.com/s2/favicons?domain=https://miped.ru) [Miped (https://miped.ru)](https://miped.ru)*: top 100M, ru* @@ -3191,24 +3192,24 @@ Rank data fetched from Majestic Million by domains. 1. ![](https://www.google.com/s2/favicons?domain=https://home.gamer.com.tw/) [Bahamut (https://home.gamer.com.tw/)](https://home.gamer.com.tw/)*: top 100M, cn, gaming, tw* 1. ![](https://www.google.com/s2/favicons?domain=https://www.hkitalk.net/) [hkitalk (https://www.hkitalk.net/)](https://www.hkitalk.net/)*: top 100M, cn, forum, hk* -The list was updated at (2026-07-13) +The list was updated at (2026-07-15) ## Statistics -Enabled/total sites: 2516/3187 = 78.95% +Enabled/total sites: 2517/3188 = 78.95% -Incomplete message checks: 315/2516 = 12.52% (false positive risks) +Incomplete message checks: 315/2517 = 12.51% (false positive risks) -Status code checks: 685/2516 = 27.23% (false positive risks) +Status code checks: 686/2517 = 27.25% (false positive risks) -False positive risk (total): 39.75% +False positive risk (total): 39.76% Sites with probing: 500px, Armchairgm, BinarySearch (disabled), BitBucket, BleachFandom, Bluesky, BongaCams, Boosty, BuyMeACoffee, Calendly, Cent, Chess, Code Sandbox (disabled), Code Snippet Wiki, DailyMotion, Discord, Diskusjon.no, Disqus, Docker Hub, Duolingo, F-droid, Faceit, FandomCommunityCentral, GitHub, GitLab, Golangbridge, Google Plus (archived), Gravatar, HackTheBox, HackerNews, HackerNoon, Hackerrank, Hashnode, Hey, Holopin, ITVDN Forum, Imgur, Instagram, Keybase, Kick, Kvinneguiden, LeetCode, Lemmy World, Lesswrong, Livejasmin, LocalCryptos (disabled), Mapillary Forum, Matrix, Medium, MetaDiscourse, MicrosoftLearn, Minds, MixCloud, Monkeytype, NPM, Niftygateway, ORCID, Omg.lol, OnlyFans, Paragraph, Picsart, Polarsteps, Rarible, Reddit, Reddit Search (Pushshift) (disabled), Revolut.me, RoyalCams, Scratch, Silver-collector, Soop, SportsTracker, Spotify, StackOverflow, Substack, TAP'D, Topcoder, Trello, Twitch, Twitter, Twitter Shadowban (disabled), UnstoppableDomains, Vimeo, Vivino, Warframe Market, Warpcast, Weibo, Wikipedia, Yapisal (disabled), YouNow, community.endlessos.com, community.getpostman.com, community.icons8.com, community.p2pu.org, discourse.haskell.org, discourse.jupyter.org, discuss.inventables.com, en.brickimedia.org, forum.audacityteam.org, forum.garudalinux.org, forum.ghost.org, forum.languagelearningwithnetflix.com, forum.shotcut.org, forum.zorin.com, forums.docker.com, forums.grandstream.com, forums.steinberg.net, hiveos.farm, iNaturalist, nightbot, notabug.org, openframeworks, programming.dev, qiwi.me (disabled), sourceruns, support.ilovegrowingmarijuana.com -Sites with activation: OnlyFans, Twitter, Vimeo, Weibo, WikimapiaSearch +Sites with activation: OnlyFans, ProtonMail, Twitter, Vimeo, Weibo, WikimapiaSearch Top 20 profile URLs: - (709) `{urlMain}/index/8-0-{username} (uCoz)` -- (319) `/{username}` +- (320) `/{username}` - (223) `{urlMain}{urlSubpath}/members/?username={username} (XenForo)` - (173) `/user/{username}` - (139) `/profile/{username}` @@ -3266,4 +3267,4 @@ Top 20 tags: - (22) `freelance` - (21) `art` - (18) `hobby` -- (17) `sport` +- (18) `porn`