Moves 77 Mastodon instances onto a shared engine instead of copying urlProbe into every entry. The lookup API answers 404 for a missing account, so twelve instances that until now returned 200 for any username stop reporting everyone as found.
Closes#3119. Supersedes #3115, #3116, #3117.
An engine is a template, but `update_from_engine` overwrites a site's scalar
fields with the engine's, and `strip_engine_data` then removes them on save. A
value written into an entry by hand is therefore ignored at runtime **and
deleted from `data.json`** the next time anything calls `save_to_file` — which
`utils/update_site_data.py` does on every regeneration. Nothing is printed: the
dict case already warns about a collision, the scalar case did not.
Found while giving MediaWiki entries a JSON `urlProbe`. The API path is not the
page path — a wiki with articles under `/wiki` keeps `api.php` under `/w` — so
a per-entry probe is the natural way to say it, and it turned out to be
impossible.
Now the entry wins and the override survives the round trip, with the same
warning the dict case prints.
Only fields the entry **stated** count. The constructor synthesises
`alexa_rank` for every site that does not carry one, so keying the rule off the
instance dict made seventeen op.gg sites keep that placeholder instead of the
rank their engine supplies; there is a test for exactly that, next to one for
the override winning and one for it surviving `strip_engine_data`.
The shipped database is unchanged — it round-trips byte for byte through load
and save, and no entry currently sets a field its engine also defines.
The display URL already goes through quote(), but urlProbe did a bare
format() with the raw username. A "#" in the username therefore ended the
URL and turned the rest into a fragment, so the probe was sent to a
different account's URL and the site was reported as claimed.
Encoding the username here keeps the two URLs consistent. Every username
in data.json is unaffected, since quote() is a no-op for them.
* fix: keep result lines from crashing on a non-UTF-8 stdout
#3050 stopped the startup banners from raising UnicodeEncodeError on a
Windows console, but the banner was not the only line the ANSI codepage can
break. Two ordinary things a run prints still went through a bare print():
- the --enrich notifications built in checking.py carry U+2192, which cp1252
has no slot for (checking.py:1847, 1851, 1859, 1899);
- any profile field maigret extracts carries the script the page was written
in, so a Cyrillic or CJK fullname breaks the result line that reports it.
Both arrive mid-scan, after the network work is done, so the crash discarded
results the run had already collected -- worse than failing at startup.
Route every write in notify.py through the existing _print_encodable helper.
Verified on a real cp1252 stdout, not a mock: both cases raise before this
change and print with replacement characters after it.
* fix: keep the AI spinner alive on a stderr that cannot encode braille
Second half of #3055. The spinner frames are braille, which cp1252 -- the
ANSI codepage of a stock Windows install -- cannot encode, so the first frame
raised UnicodeEncodeError inside the daemon thread: the thread died with a
traceback printed over the output and the animation stopped for the rest of
the run.
Encoding with replacement would only leave a row of '?' spinning, so pick the
frame set from what the stream can carry instead. A UTF-8 terminal keeps the
braille; anything that cannot hold it gets |/-\ and still animates.
print_streaming() in the same module writes model output the same way and
would raise on any non-Latin reply, but it has no callers anywhere in the
tree, so it is left alone rather than fixed as dead code.
`pyinstaller.yml` only runs on pushes to `main` and `dev`, so `maigret_standalone.exe` exists only on the moving `nightly-*` tags. Every stable release from v0.6.1 to v0.6.5 has zero assets, and winget and Scoop cannot pin a moving tag by hash.
- add the `release: types: [published]` trigger, the same one `python-publish.yml` uses. Checkout resolves `refs/tags/<tag>`, so the exe is built from the released code.
- guard the two nightly-only steps on `github.event_name == 'push'`, otherwise `NIGHTLY_TAG` becomes `nightly-v0.6.5` and creates a junk tag beside the real one.
- add a step attaching the binary to the published release with every `omit*DuringUpdate` set, so the title, body, prerelease and draft state are left alone.
- `tests/test_workflows.py`: the `release_step` fixture asserted there was exactly one `release-action` step, which a second one breaks. It now selects a step by the event it is guarded on, and the new step gets its own guards: every release step must be event specific, the published one must set all four omits, and it must target `github.event.release.tag_name` rather than `NIGHTLY_TAG`.
Existing releases are not backfilled. A release now costs two PyInstaller runs, one on the tag and one on the push to `main` that follows.
Every one of them is a site that answers the same way for any username, so the check said "claimed" for everyone.
Two of them are proof-of-work interstitials that come back with a 2xx, and those are worth a global marker rather than a per-site patch: joyreactor.cc now runs Anubis (`/.within.website/x/`, HTTP 200 on every path) and fixya.com serves an HTTP 202 with `window.POW_CHALLENGE_DATA`. Both are vendor products used far beyond these two sites, so they go into COMMON_ERRORS and the sites report bot protection instead of a hit.
The rest are per-site rot. championat.com answers every `/user/*` with the same 686-byte SberID auth stub, profiles are behind login now, disabled. forum.heroesworld.ru redirects to the forum index, which never contains the vBulletin absence marker, the forum has moved to `heroesworld.ru/user/{username}/` where a missing user is a clean 404. Codédex was a status_code check and flapped to 200 on a missing profile once in about fifteen requests, so it now matches the og:title of a real profile.
OP.GG search falls back to other regions when the requested one has no hit: `?q=soxoj®ion=ru` returns an Oceania summoner, and the shared engine marker `href="/lol/summoners/` matched it, so all seventeen region entries claimed the account. The marker moves from the engine to each site with its region baked in. Name matching is already exact on op.gg's side, `q=blaz` returns only summoners named exactly Blaz, so the region was the whole leak.
Completes subtask 2 of #2668. When manual feature detection sees a claimed profile return 2xx and the unclaimed profile return 4xx, --submit creates a status_code check. Redirects keep the message fallback.
* fix: don't treat refused or self-redirected responses as not-found
401 and 429 fell through to the checkType branch, where any non-2xx code means the username is free, so blocked and rate-limited requests were reported as confirmed absences; they now return errors, Vercel's Security Checkpoint joins the bot-protection markers, and a self-redirect that carries Set-Cookie is retried once instead of read as an absence.
* Fix site checks: 16 sites, 3 fixed, 12 disabled, 1 dead deleted
* docs: note that the joyreactor self-redirect is intermittent
save_graph_report handed the file off to pyvis's show(), which opens it with
open(name, "w+"), i.e. the locale encoding. On Windows that is the ANSI
codepage (cp1252 by default), and the vis-network bundle pyvis inlines carries
several hundred non-ASCII characters of its own, so every graph report died
with UnicodeEncodeError regardless of what was scanned. The CLI aborted
mid-run and left a 0-byte html file behind; the web interface builds a graph
for every scan, so each one ended as "Search failed" and discarded the
results.
Render the html and write it here with encoding="utf-8" instead. The debug.log
dump and the report template read had the same implicit encoding, so they get
the same treatment.
Three existing tests (test_build_reports_computes_found_count,
test_real_report_generation_does_not_crash, test_live_scan_streams_found_and_done)
fail on a non-utf-8 machine before this change and pass after it. The new tests
perform the writes under -X warn_default_encoding, so a return to the implicit
encoding fails on CI too, where the locale is utf-8 and the bug is invisible.
Rotating/residential proxies often drop a connection mid-request
(truncated body, closed socket, failed handshake) instead of failing
outright. These were previously lumped into "Unexpected" and never
retried, silently discarding otherwise-successful checks.
- Retry once on ClientPayloadError, ServerDisconnectedError, and
aiohttp_socks' ProxyConnectionError/ProxyTimeoutError (aiohttp
checker), and on CurlError (curl_cffi checker) — covers truncated
responses, dropped connections, CONNECT-tunnel 502s, and TLS
handshake failures.
- Fix the proxy-error except clause: it caught python_socks'
ProxyError, but aiohttp_socks (the connector actually in use)
raises its own unrelated same-named exceptions, so it never fired.
- Generic ProxyError (e.g. bad credentials) is classified but NOT
retried — it fails identically every attempt.
extract_ids_from_page() has the same bug #2907 fixed in
checking.py::parse_usernames(). "username" is itself in SUPPORTED_IDS, so
the `if k in SUPPORTED_IDS` loop stores a bare `username` value directly,
bypassing the is_plausible_username() guard that extract_usernames()
applies in its own separate pass. A URL or email returned by
socid_extractor under a bare `username` key therefore becomes a recursive
search target via `maigret --parse-url`, reproducing the #1403
false-error cascade.
Skip keys containing "username" in the SUPPORTED_IDS loop; those are
already owned (and validated) by extract_usernames(). Every other
SUPPORTED_ID (gaia_id, vk_id, orcid, ...) is unaffected since none of
those keys contain "username".
Add regression tests for the bare `username` key with a URL value and for
the unchanged handling of other supported IDs.
Closes#2911
* fix: block SSRF and local-file reads via report image URLs in PDF generation
save_pdf_report() rendered scraped profile image URLs (ids_data['image'])
straight into xhtml2pdf, which fetches <img src> while building the PDF.
The image field is attacker-influenced and pisaDocument ran with no
link_callback, so a profile carrying image = "file:///etc/passwd" or an
intranet/metadata URL turned report generation into a local file read or
an SSRF from the machine running maigret. In the web UI this is
server-side and fires on every search, since save_pdf_report is always
called.
Add a link_callback that only lets public http(s) images through and
diverts everything else (file://, data:, other schemes, and hosts that
resolve to loopback/private/link-local/reserved addresses) to a bundled
1x1 placeholder, so no fetch or read happens. Diverting rather than
raising keeps report generation working when a scanned profile carries a
hostile image URL.
Tests cover the URL classifier, the callback's placeholder diversion, and
an end-to-end check that PDF generation does not fetch an internal image.
* fix: use is_global to also block CGNAT (100.64.0.0/10) report image hosts
The flag chain missed 100.64.0.0/10, which is neither is_private nor
is_global and is routable inside many cloud and k8s networks. is_global
covers it along with private, loopback, link-local and unspecified.
Multicast and reserved stay explicit: both are still is_global on
CPython, and 64:ff9b::/96 reaches IPv4 through a NAT64 gateway.
parse_usernames() validates *_username fields with is_plausible_username
(the #1403 guard), but the trailing `if k in SUPPORTED_IDS` branch runs
unconditionally. Since "username" is itself in SUPPORTED_IDS, the bare
`username` key bypasses the guard: a URL/email/path value is dropped by
the plausibility check and then silently re-added, so it becomes a
recursive search target again and reproduces the #1403 false-error
cascade.
Make the branch an elif. The bare `username` key is already handled with
validation by the first branch; every other SUPPORTED_ID (gaia_id,
vk_id, orcid, ...) still gets added as before since none of those keys
contain "username".
Add a regression test for the bare `username` key with URL/email/path
values; the existing parse_usernames tests only covered *_username
variants, which aren't in SUPPORTED_IDS.
* refactor: drop dead 'if not dictionary' guards across report paths
Closes#2665.
Every site results entry flows through make_site_result (checking.py:788),
which initializes results_site = {} and then unconditionally populates it
(site/username/keywords/parsing_enabled/url_main/cookies up front, checker
at the end, plus status or url_user+future on every branch). It has no
return path that yields an empty dict. check_site_for_username wraps that
result and likewise never returns a falsy entry, and maigret.py does not
construct site-result dicts by hand.
So the four downstream 'if not dictionary: continue' guards (all tagged
'# TODO: fix no site data issue') were dead code — the entries they
guarded against cannot occur. Removed:
- maigret/maigret.py:101 (extract_ids_from_results)
- maigret/report.py:151 — 'if not dictionary or dictionary.get("is_similar")'
trimmed to just the is_similar check (real logic)
- maigret/report.py:467 (extended-report builder)
- maigret/report.py:609 (generate_txt_report)
- maigret/report.py:627 — 'if not site_result or not site_result.get("status")'
trimmed to just the status check (real logic)
- maigret/report.py:687 (generate_json_report)
plus the four '# TODO: fix no site data issue' comments.
No behavior change for well-formed inputs; downstream .get() calls are
safe because entries are always populated dicts. 292 tests pass.
* feat: add Chinese name false-positive fix + CSDN site
- #2633: When searching non-ASCII usernames, skip presence detection
if the response body doesn't contain the username at all, preventing
false positives on sites that return generic error pages.
- #2634: Add CSDN (blog.csdn.net) — major Chinese dev blog with
40M+ users, alexa rank #45 in China.
* fix: add tls_fingerprint to CSDN + add tests for non-ASCII false positive fix
Addresses PR #2876 review feedback from soxoj:
- Add tls_fingerprint to CSDN protection (curl_cffi required)
- Add 2 test cases for #2633 non-ASCII username false positive fix:
- non-ASCII username not in response → no false CLAIMED
- non-ASCII username in response → normal CLAIMED behavior
---------
Co-authored-by: aznikline <aznikline@users.noreply.github.com>
* Fix site checks: 82 fixed, 11 disabled + DiscourseJson engine
* Fix test_hackernews_requires_profile_marker: update test body to JSON format matching Firebase API probe
In MaigretDatabase.ranked_sites_dict, the include (whitelist) tag filter
compared the site's raw tags against the lowercased query tags:
is_tags_ok = lambda x: set(x.tags).intersection(set(normalized_tags))
while the exclude (blacklist) filter and every sibling lambda
(name/source/engine) lowercase the site-side value:
is_excluded_by_tag = lambda x: set(map(str.lower, x.tags)).intersection(
set(normalized_excluded_tags)
)
So a site stored with an upper/mixed-case tag (e.g. a custom or submitted
site tagged 'US') was excluded by `--exclude-tags us` but NOT found by
`--tags us` — an asymmetry between the two filters. Lowercase the site
tags in the include filter too, matching the rest of the method.
Adds a regression test asserting tags=['us'] finds a site tagged 'US'.
Serialize the maigret graph (the same one --graph builds) into an idempotent Cypher script importable into Neo4j. Reuses MaigretGraph via an extracted _build_maigret_graph() helper, leaving save_graph_report behavior unchanged; no new runtime dependency. Adds the --neo4j flag, the neo4j_report setting, docs, and a unit test.
Closes#2630
URLMatcher._HTTP_URL_RE_STR was "^https?://(www.|m.)?(.+)$". The dots in
the optional (www.|m.)? subdomain-prefix group were unescaped, so each `.`
matched ANY character. For a host that simply starts with `m` or `www`
(not the literal `m.`/`www.` prefix), the group greedily ate the first
characters:
extract_main_part('https://medium.com/alice') -> 'dium.com/alice'
extract_main_part('https://myspace.com/bob') -> 'space.com/bob'
extract_main_part feeds make_profile_url_regexp / get_url_template, so the
generated site-detection regexp was corrupted too (medium.com -> dium.com),
and the loose prefix could match bogus hosts like `xmedium.com`. 170+ hosts
in resources/data.json start with `m`.
Fix: escape the dots with a raw string, r"^https?://(www\.|m\.)?(.+)$".
Genuine `m.`/`www.` prefixes are still stripped
(m.wikipedia.org -> wikipedia.org). The two existing tests that pinned the
unescaped pattern string are updated to the corrected form, and a
regression test covers hosts beginning with m/www.
extract_and_group stored the per-error-type percentage already rounded to
2 decimals, and is_important() compares that stored value against the
threshold. Rounding before the comparison can push a rate that is strictly
below the threshold up to it:
8 Captcha errors / 267 sites = 2.99625%
round(2.99625, 2) == 3.0 -> is_important() True -> spurious
'Too many errors of type "Captcha" (3.0%)' warning
This is the residual of the same class fixed in #2788 (scale-then-round):
scaling to a percentage first stopped 2.5% from becoming 3.0%, but rounding
the percentage still feeds the threshold check. The display sites already
round (round(e["perc"], 2)), so storing the raw percentage keeps the shown
values unchanged while making the threshold comparison exact.
Genuine at-or-above-threshold rates still fire (3/100 = 3.0%, DNS 10%).
Adds a regression test for the 2.996% -> 3.0% rounding-up case.
`get_dict_ascii_tree(items, prepend="", new_line=True)` immediately reassigned
`new_line` to the horizontal box-drawing glyph ("─"), shadowing the boolean
parameter. The trailing `if not new_line: text = text[1:]` — meant to strip the
leading newline — therefore tested a non-empty string (always truthy) and never
ran. Callers passing `new_line=False` (e.g. `maigret.py` printing a user's
identity data) got a stray leading blank line.
Rename the glyph local to `h_line` so the `new_line` parameter is preserved and
its strip takes effect. The tree drawing is unchanged.
Adds a regression test asserting `new_line=False` drops the leading newline
while keeping the rest of the tree identical.
Closes#2666.
Issue #2666 tracked four paths flagged `# TODO: tests`. Two of them
(`executors.py` increment_progress / stop_progress) were removed by the
#2789 refactor and no longer exist. This PR covers the two that remain:
1. maigret/checking.py:247 — the ClientSession cookie_jar forwarding.
SimpleAiohttpChecker.check() builds ClientSession(cookie_jar=self.cookie_jar
if self.cookie_jar else None), but nothing asserted the jar handed to the
checker actually reached the outgoing session. Added two tests using the
existing constructor-capture pattern (_capture_clientsession): one asserts
a configured jar is forwarded verbatim, the other asserts None is passed
when no jar is configured (the else branch).
2. maigret/maigret.py:946 — extract_ids_from_results. The pre-existing
test_extract_ids_from_results was a bare expression with no `assert` (so
it silently passed regardless of return value) AND pinned its expectation
against test_db (tests/db.json), whose site set never matched the Reddit
URL and silently dropped the ids_links result. Fixed: real assertion against
default_db, plus three branch tests (cross-site username merge, empty-site
skip, empty input).
No new dependencies: aioresponses was suggested in the issue but is not in
the project — the existing monkeypatch capture pattern fits the codebase
better and adds zero deps.
Removed both `# TODO: tests` comments from the now-covered lines.
Co-authored-by: aznikline <aznikline@users.noreply.github.com>
`extract_and_group` computed the per-error-type percentage as
`round(count / len(search_res), 2) * 100` — it rounded the *fraction* to 2
decimals (i.e. to the nearest 1%) and only then scaled by 100, so `perc`
always had whole-percent granularity and was mis-rounded.
A genuine 2.5% error rate (1 error out of 40 sites) became
`round(0.025, 2) * 100 = 3.0%` and tripped the 3% "important" threshold in
`is_important()`, firing a spurious "Too many errors of type ..." warning.
Likewise 9.5% DNS errors rounded to 10.0% and crossed the DNS override.
Scale to a percentage *before* rounding: `round(count / len(search_res) * 100, 2)`.
The existing tests use integer-landing ratios (25/100, 5/100, 3/100) that are
identical under both formulas, so none of them change.
Adds a regression test for a sub-threshold non-integer rate (2.5%) staying silent.
Convert activation HTTP calls to aiohttp coroutines, await activation before retrying, and allocate independent protocol checkers per site check so concurrent retries do not overwrite shared checker state.
