resolve_db_path() runs before the first site check and fetched db_meta.json
and data.json with bare requests.get() calls, so a run started with --proxy
still reached raw.githubusercontent.com from the real address once every 24
hours. Auto-update is on by default, and only HTTP_PROXY / HTTPS_PROXY were
honoured, by accident of requests reading them itself.
Thread the proxy into both requests and into force_update(), normalising the
scheme for PySocks the way the other two transports already are. An explicit
proxies mapping also wins over the environment, so NO_PROXY cannot bypass it.
When the proxy is unreachable requests raises and the existing handler
reports the update as failed and keeps the local database. There is
deliberately no retry without the proxy: that would put the leak back on the
failure path.
Fixes#3108
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
self_check() has no cookies parameter, so the CLI cannot forward
args.cookie_file and every site check runs without the cookie jar. The
layer below already takes and plumbs it: site_self_check(cookies=None)
passes cookies through to maigret(), and submit.py passes
cookies=self.args.cookie_file into that same function. Self-check was the
one of the three paths that dropped it.
The effect is worse than a missing option. errors.py maps "Login required"
to the advice "Add authorization cookies through --cookies-jar-file", so a
login-walled site self-checked without the jar reports the claimed profile
as not detected, and with --auto-disable that writes disabled=True into
the sites database. CONTRIBUTING tells contributors to run
`maigret --self-check --site "SiteName" --diagnose`.
Same shape as #3026, which did this for --dns-resolver.
Co-authored-by: datrixlab <325650023+datrixlab@users.noreply.github.com>
The display URL already goes through quote(), but urlProbe did a bare
format() with the raw username. A "#" in the username therefore ended the
URL and turned the rest into a fragment, so the probe was sent to a
different account's URL and the site was reported as claimed.
Encoding the username here keeps the two URLs consistent. Every username
in data.json is unaffected, since quote() is a no-op for them.
* fix: don't treat refused or self-redirected responses as not-found
401 and 429 fell through to the checkType branch, where any non-2xx code means the username is free, so blocked and rate-limited requests were reported as confirmed absences; they now return errors, Vercel's Security Checkpoint joins the bot-protection markers, and a self-redirect that carries Set-Cookie is retried once instead of read as an absence.
* Fix site checks: 16 sites, 3 fixed, 12 disabled, 1 dead deleted
* docs: note that the joyreactor self-redirect is intermittent
save_graph_report handed the file off to pyvis's show(), which opens it with
open(name, "w+"), i.e. the locale encoding. On Windows that is the ANSI
codepage (cp1252 by default), and the vis-network bundle pyvis inlines carries
several hundred non-ASCII characters of its own, so every graph report died
with UnicodeEncodeError regardless of what was scanned. The CLI aborted
mid-run and left a 0-byte html file behind; the web interface builds a graph
for every scan, so each one ended as "Search failed" and discarded the
results.
Render the html and write it here with encoding="utf-8" instead. The debug.log
dump and the report template read had the same implicit encoding, so they get
the same treatment.
Three existing tests (test_build_reports_computes_found_count,
test_real_report_generation_does_not_crash, test_live_scan_streams_found_and_done)
fail on a non-utf-8 machine before this change and pass after it. The new tests
perform the writes under -X warn_default_encoding, so a return to the implicit
encoding fails on CI too, where the locale is utf-8 and the bug is invisible.
Rotating/residential proxies often drop a connection mid-request
(truncated body, closed socket, failed handshake) instead of failing
outright. These were previously lumped into "Unexpected" and never
retried, silently discarding otherwise-successful checks.
- Retry once on ClientPayloadError, ServerDisconnectedError, and
aiohttp_socks' ProxyConnectionError/ProxyTimeoutError (aiohttp
checker), and on CurlError (curl_cffi checker) — covers truncated
responses, dropped connections, CONNECT-tunnel 502s, and TLS
handshake failures.
- Fix the proxy-error except clause: it caught python_socks'
ProxyError, but aiohttp_socks (the connector actually in use)
raises its own unrelated same-named exceptions, so it never fired.
- Generic ProxyError (e.g. bad credentials) is classified but NOT
retried — it fails identically every attempt.
parse_usernames() validates *_username fields with is_plausible_username
(the #1403 guard), but the trailing `if k in SUPPORTED_IDS` branch runs
unconditionally. Since "username" is itself in SUPPORTED_IDS, the bare
`username` key bypasses the guard: a URL/email/path value is dropped by
the plausibility check and then silently re-added, so it becomes a
recursive search target again and reproduces the #1403 false-error
cascade.
Make the branch an elif. The bare `username` key is already handled with
validation by the first branch; every other SUPPORTED_ID (gaia_id,
vk_id, orcid, ...) still gets added as before since none of those keys
contain "username".
Add a regression test for the bare `username` key with URL/email/path
values; the existing parse_usernames tests only covered *_username
variants, which aren't in SUPPORTED_IDS.
* refactor: drop dead 'if not dictionary' guards across report paths
Closes#2665.
Every site results entry flows through make_site_result (checking.py:788),
which initializes results_site = {} and then unconditionally populates it
(site/username/keywords/parsing_enabled/url_main/cookies up front, checker
at the end, plus status or url_user+future on every branch). It has no
return path that yields an empty dict. check_site_for_username wraps that
result and likewise never returns a falsy entry, and maigret.py does not
construct site-result dicts by hand.
So the four downstream 'if not dictionary: continue' guards (all tagged
'# TODO: fix no site data issue') were dead code — the entries they
guarded against cannot occur. Removed:
- maigret/maigret.py:101 (extract_ids_from_results)
- maigret/report.py:151 — 'if not dictionary or dictionary.get("is_similar")'
trimmed to just the is_similar check (real logic)
- maigret/report.py:467 (extended-report builder)
- maigret/report.py:609 (generate_txt_report)
- maigret/report.py:627 — 'if not site_result or not site_result.get("status")'
trimmed to just the status check (real logic)
- maigret/report.py:687 (generate_json_report)
plus the four '# TODO: fix no site data issue' comments.
No behavior change for well-formed inputs; downstream .get() calls are
safe because entries are always populated dicts. 292 tests pass.
* feat: add Chinese name false-positive fix + CSDN site
- #2633: When searching non-ASCII usernames, skip presence detection
if the response body doesn't contain the username at all, preventing
false positives on sites that return generic error pages.
- #2634: Add CSDN (blog.csdn.net) — major Chinese dev blog with
40M+ users, alexa rank #45 in China.
* fix: add tls_fingerprint to CSDN + add tests for non-ASCII false positive fix
Addresses PR #2876 review feedback from soxoj:
- Add tls_fingerprint to CSDN protection (curl_cffi required)
- Add 2 test cases for #2633 non-ASCII username false positive fix:
- non-ASCII username not in response → no false CLAIMED
- non-ASCII username in response → normal CLAIMED behavior
---------
Co-authored-by: aznikline <aznikline@users.noreply.github.com>
* Fix site checks: 82 fixed, 11 disabled + DiscourseJson engine
* Fix test_hackernews_requires_profile_marker: update test body to JSON format matching Firebase API probe
Closes#2666.
Issue #2666 tracked four paths flagged `# TODO: tests`. Two of them
(`executors.py` increment_progress / stop_progress) were removed by the
#2789 refactor and no longer exist. This PR covers the two that remain:
1. maigret/checking.py:247 — the ClientSession cookie_jar forwarding.
SimpleAiohttpChecker.check() builds ClientSession(cookie_jar=self.cookie_jar
if self.cookie_jar else None), but nothing asserted the jar handed to the
checker actually reached the outgoing session. Added two tests using the
existing constructor-capture pattern (_capture_clientsession): one asserts
a configured jar is forwarded verbatim, the other asserts None is passed
when no jar is configured (the else branch).
2. maigret/maigret.py:946 — extract_ids_from_results. The pre-existing
test_extract_ids_from_results was a bare expression with no `assert` (so
it silently passed regardless of return value) AND pinned its expectation
against test_db (tests/db.json), whose site set never matched the Reddit
URL and silently dropped the ids_links result. Fixed: real assertion against
default_db, plus three branch tests (cross-site username merge, empty-site
skip, empty input).
No new dependencies: aioresponses was suggested in the issue but is not in
the project — the existing monkeypatch capture pattern fits the codebase
better and adds zero deps.
Removed both `# TODO: tests` comments from the now-covered lines.
Co-authored-by: aznikline <aznikline@users.noreply.github.com>
Convert activation HTTP calls to aiohttp coroutines, await activation before retrying, and allocate independent protocol checkers per site check so concurrent retries do not overwrite shared checker state.
