WEBDIENSTE/maigret
1
0
Fork 0
mirror of https://github.com/soxoj/maigret.git synced 2026-08-17 19:25:41 +02:00
Code Issues Packages Projects Releases Wiki Activity

Improve CONTRIBUTING.md (#2871)

Browse Source
This commit is contained in:
Soxoj
2026-07-15 17:23:53 +02:00
committed by GitHub
parent 9d29061e2a
commit c436174467
4 changed files with 28 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTRIBUTING.md
+10
View File
@@ -97,6 +97,16 @@ Each site entry uses one of three `checkType` modes to decide whether a profile
Full reference for `checkType`, `urlProbe`, `engine`, and the rest of the `data.json` schema is in the [development guide](docs/source/development.rst), section *How to fix false-positives*.
### SPA / JS-rendered sites
Modern sites often render the profile client-side, so a plain HTTP fetch returns an empty shell that trips both `message` and `status_code` checks. **Don't reach for a headless browser** — it's a heavy dependency for a problem that has three lighter answers. Work down the ladder in order:
1. **XHR / GraphQL / RSC endpoint → `urlProbe`.** Open the profile in Chrome DevTools → Network, filter to `XHR`/`Fetch`, and look for the request that carries the actual profile JSON. Almost every SPA has one — that's what its own front-end calls. Put its URL in `urlProbe`; the human-readable profile URL stays in `url`. Watch for versioned APIs (`/v5/`, `/api/v2/`) — if the one you find starts returning 5xx or 404 for everyone, bump the version before disabling.
2. **JS-cookie / anti-bot challenge → `activation` handler.** If the first response is a challenge stub that sets a cookie via JavaScript and reloads, add a static method to `maigret/activation.py` that reproduces the token exchange with plain `aiohttp` (existing examples: `wikimapia`, `weibo`, `twitter`, `vimeo`, `onlyfans`, `proton`). Point `activation.method` at your function name and set `marks` to the challenge substring; Maigret retries the check once after `activation` fires.
3. **TLS-fingerprint block → `protection: ["tls_fingerprint"]`.** If the site 403's or serves a Cloudflare interstitial to plain `aiohttp` but works in a real browser, the check is being killed by JA3/JA4 fingerprinting. The tag routes the request through a Chrome-impersonating client — usually enough on its own, with no `disabled: true` needed.
If none of these help — the site really does need a full JS runtime with DOM to render the profile signal — that's a case for issue [#579](https://github.com/soxoj/maigret/issues/579). Open a fresh issue naming the site so we can weigh a `checkType: browser` mode against its latency and dependency cost.
### Editing `data.json` safely
`data.json` is a single ~36 000-line JSON file. **Make surgical, line-level edits only.** Never rewrite it by reading it into a Python dict and dumping it back — `json.load` + `json.dump` reformats every entry and produces an unreviewable 70 000-line diff. The same rule applies to any helper script that touches the file: it must preserve the original formatting of untouched entries.
maigret/resources/data.json
+3 -1
View File
@@ -27329,7 +27329,9 @@
"activation": {
"url": "https://account.proton.me/api/auth/v4/sessions",
"method": "proton",
"marks": ["Invalid access token"]
"marks": [
"Invalid access token"
]
},
"urlMain": "https://proton.me/",
"url": "https://account.proton.me/api/users/available?Name={username}",
maigret/resources/db_meta.json
+4 -4
View File
@@ -1,8 +1,8 @@
{
"version": 1,
"updated_at": "2026-07-13T16:02:35Z",
"sites_count": 3187,
"updated_at": "2026-07-15T10:25:34Z",
"sites_count": 3188,
"min_maigret_version": "0.5.0",
"data_sha256": "4eeed3b475a1ff4dce558bbca926d50adcb6b8e5372770f330e25baa4a252df0",
"data_sha256": "06b6f49f07344096e6ee827c6bc54c8391e8e97f51b350dbfe28c06075be40d7",
"data_url": "https://raw.githubusercontent.com/soxoj/maigret/main/maigret/resources/data.json"
}
}
sites.md
+11 -10
View File
@@ -1,5 +1,5 @@
## List of supported sites (search methods): total 3187
## List of supported sites (search methods): total 3188
Rank data fetched from Majestic Million by domains.
@@ -1494,6 +1494,7 @@ Rank data fetched from Majestic Million by domains.
1. ![](https://www.google.com/s2/favicons?domain=https://forum.f-droid.org) [F-droid (https://forum.f-droid.org)](https://forum.f-droid.org)*: top 100M, forum*
1. ![](https://www.google.com/s2/favicons?domain=http://forum.vn.ua) [forum.vn.ua (http://forum.vn.ua)](http://forum.vn.ua)*: top 100M*, search is disabled
1. ![](https://www.google.com/s2/favicons?domain=https://profile.ameba.jp) [Ameba (https://profile.ameba.jp)](https://profile.ameba.jp)*: top 100M, jp*
1. ![](https://www.google.com/s2/favicons?domain=https://amateur.tv/) [Amateur.tv (https://amateur.tv/)](https://amateur.tv/)*: top 100M, porn, webcam*
1. ![](https://www.google.com/s2/favicons?domain=https://forum.blackmagicdesign.com) [forum.blackmagicdesign.com (https://forum.blackmagicdesign.com)](https://forum.blackmagicdesign.com)*: top 100M, forum*
1. ![](https://www.google.com/s2/favicons?domain=https://hosted.weblate.org/) [Weblate (https://hosted.weblate.org/)](https://hosted.weblate.org/)*: top 100M*
1. ![](https://www.google.com/s2/favicons?domain=https://forum.sublimetext.com/) [SublimeForum (https://forum.sublimetext.com/)](https://forum.sublimetext.com/)*: top 100M, coding, forum*
@@ -2266,7 +2267,7 @@ Rank data fetched from Majestic Million by domains.
1. ![](https://www.google.com/s2/favicons?domain=https://www.meendo.net) [Meendo (https://www.meendo.net)](https://www.meendo.net)*: top 100M, bg, kg, ru, ua*
1. ![](https://www.google.com/s2/favicons?domain=) [Megamodels.pl ()]()*: top 100M*
1. ![](https://www.google.com/s2/favicons?domain=http://megane2.ru/) [Megane2 (http://megane2.ru/)](http://megane2.ru/)*: top 100M, ru*
1. ![](https://www.google.com/s2/favicons?domain=) [ProtonMail ()]()*: top 100M*
1. ![](https://www.google.com/s2/favicons?domain=https://proton.me/) [ProtonMail (https://proton.me/)](https://proton.me/)*: top 100M*
1. ![](https://www.google.com/s2/favicons?domain=http://www.microchip.su) [Microchip (http://www.microchip.su)](http://www.microchip.su)*: top 100M, forum, ru*, search is disabled
1. ![](https://www.google.com/s2/favicons?domain=) [Minecraftlist ()]()*: top 100M*
1. ![](https://www.google.com/s2/favicons?domain=https://miped.ru) [Miped (https://miped.ru)](https://miped.ru)*: top 100M, ru*
@@ -3191,24 +3192,24 @@ Rank data fetched from Majestic Million by domains.
1. ![](https://www.google.com/s2/favicons?domain=https://home.gamer.com.tw/) [Bahamut (https://home.gamer.com.tw/)](https://home.gamer.com.tw/)*: top 100M, cn, gaming, tw*
1. ![](https://www.google.com/s2/favicons?domain=https://www.hkitalk.net/) [hkitalk (https://www.hkitalk.net/)](https://www.hkitalk.net/)*: top 100M, cn, forum, hk*
The list was updated at (2026-07-13)
The list was updated at (2026-07-15)
## Statistics
Enabled/total sites: 2516/3187 = 78.95%
Enabled/total sites: 2517/3188 = 78.95%
Incomplete message checks: 315/2516 = 12.52% (false positive risks)
Incomplete message checks: 315/2517 = 12.51% (false positive risks)
Status code checks: 685/2516 = 27.23% (false positive risks)
Status code checks: 686/2517 = 27.25% (false positive risks)
False positive risk (total): 39.75%
False positive risk (total): 39.76%
Sites with probing: 500px, Armchairgm, BinarySearch (disabled), BitBucket, BleachFandom, Bluesky, BongaCams, Boosty, BuyMeACoffee, Calendly, Cent, Chess, Code Sandbox (disabled), Code Snippet Wiki, DailyMotion, Discord, Diskusjon.no, Disqus, Docker Hub, Duolingo, F-droid, Faceit, FandomCommunityCentral, GitHub, GitLab, Golangbridge, Google Plus (archived), Gravatar, HackTheBox, HackerNews, HackerNoon, Hackerrank, Hashnode, Hey, Holopin, ITVDN Forum, Imgur, Instagram, Keybase, Kick, Kvinneguiden, LeetCode, Lemmy World, Lesswrong, Livejasmin, LocalCryptos (disabled), Mapillary Forum, Matrix, Medium, MetaDiscourse, MicrosoftLearn, Minds, MixCloud, Monkeytype, NPM, Niftygateway, ORCID, Omg.lol, OnlyFans, Paragraph, Picsart, Polarsteps, Rarible, Reddit, Reddit Search (Pushshift) (disabled), Revolut.me, RoyalCams, Scratch, Silver-collector, Soop, SportsTracker, Spotify, StackOverflow, Substack, TAP'D, Topcoder, Trello, Twitch, Twitter, Twitter Shadowban (disabled), UnstoppableDomains, Vimeo, Vivino, Warframe Market, Warpcast, Weibo, Wikipedia, Yapisal (disabled), YouNow, community.endlessos.com, community.getpostman.com, community.icons8.com, community.p2pu.org, discourse.haskell.org, discourse.jupyter.org, discuss.inventables.com, en.brickimedia.org, forum.audacityteam.org, forum.garudalinux.org, forum.ghost.org, forum.languagelearningwithnetflix.com, forum.shotcut.org, forum.zorin.com, forums.docker.com, forums.grandstream.com, forums.steinberg.net, hiveos.farm, iNaturalist, nightbot, notabug.org, openframeworks, programming.dev, qiwi.me (disabled), sourceruns, support.ilovegrowingmarijuana.com
Sites with activation: OnlyFans, Twitter, Vimeo, Weibo, WikimapiaSearch
Sites with activation: OnlyFans, ProtonMail, Twitter, Vimeo, Weibo, WikimapiaSearch
Top 20 profile URLs:
- (709) `{urlMain}/index/8-0-{username} (uCoz)`
- (319) `/{username}`
- (320) `/{username}`
- (223) `{urlMain}{urlSubpath}/members/?username={username} (XenForo)`
- (173) `/user/{username}`
- (139) `/profile/{username}`
@@ -3266,4 +3267,4 @@ Top 20 tags:
- (22) `freelance`
- (21) `art`
- (18) `hobby`
- (17) `sport`
- (18) `porn`