* refactor: drop dead 'if not dictionary' guards across report paths
Closes#2665.
Every site results entry flows through make_site_result (checking.py:788),
which initializes results_site = {} and then unconditionally populates it
(site/username/keywords/parsing_enabled/url_main/cookies up front, checker
at the end, plus status or url_user+future on every branch). It has no
return path that yields an empty dict. check_site_for_username wraps that
result and likewise never returns a falsy entry, and maigret.py does not
construct site-result dicts by hand.
So the four downstream 'if not dictionary: continue' guards (all tagged
'# TODO: fix no site data issue') were dead code — the entries they
guarded against cannot occur. Removed:
- maigret/maigret.py:101 (extract_ids_from_results)
- maigret/report.py:151 — 'if not dictionary or dictionary.get("is_similar")'
trimmed to just the is_similar check (real logic)
- maigret/report.py:467 (extended-report builder)
- maigret/report.py:609 (generate_txt_report)
- maigret/report.py:627 — 'if not site_result or not site_result.get("status")'
trimmed to just the status check (real logic)
- maigret/report.py:687 (generate_json_report)
plus the four '# TODO: fix no site data issue' comments.
No behavior change for well-formed inputs; downstream .get() calls are
safe because entries are always populated dicts. 292 tests pass.
* feat: add Chinese name false-positive fix + CSDN site
- #2633: When searching non-ASCII usernames, skip presence detection
if the response body doesn't contain the username at all, preventing
false positives on sites that return generic error pages.
- #2634: Add CSDN (blog.csdn.net) — major Chinese dev blog with
40M+ users, alexa rank #45 in China.
* fix: add tls_fingerprint to CSDN + add tests for non-ASCII false positive fix
Addresses PR #2876 review feedback from soxoj:
- Add tls_fingerprint to CSDN protection (curl_cffi required)
- Add 2 test cases for #2633 non-ASCII username false positive fix:
- non-ASCII username not in response → no false CLAIMED
- non-ASCII username in response → normal CLAIMED behavior
---------
Co-authored-by: aznikline <aznikline@users.noreply.github.com>
Two safe, mechanical changes:
- activation.py: replace `list(domain.values())[0]` with
`next(iter(domain.values()))` in import_aiohttp_cookies to avoid
materializing the entire iterable just to grab the first dict.
- checking.py: reorganize imports to PEP 8 order (stdlib, third-party,
local). `from unittest.mock import Mock` and the stdlib `quote`
import move into the stdlib section; `python_socks` and
`socid_extractor` move into third-party; `from .error_detection import
detect_error_page` and the `from .` local imports move into the local
section.
No behavioral change. Same diff shape as 2484509 -> HEAD for these two
files minus the else-removal work.
Pixwox's protection was tagged only ip_reputation, but its actual
failure mode is a Cloudflare "Just a moment" JS challenge. Since
build_cloudflare_bypass_config() only activates the bypass module for
sites whose protection matches trigger_protection
(cf_js_challenge/cf_firewall/webgate), --cloudflare-bypass was
silently never invoked for Pixwox even with a working FlareSolverr
instance configured.
Verified FlareSolverr solves the exact URL directly (200, real
profile HTML) — the gap was purely in the trigger match, not the
solver.
Fixes#2845
* Fix site checks: 82 fixed, 11 disabled + DiscourseJson engine
* Fix test_hackernews_requires_profile_marker: update test body to JSON format matching Firebase API probe
The opening <html> tag contains a malformed type="text/css" attribute
which makes the HTML invalid (leading > missing). In HTML5 the <html>
tag does not require any attributes.
Fixes the malformed HTML that appears in generated PDF reports.
Co-authored-by: yaqing <yyq1043-cloud@users.noreply.github.com>
In MaigretDatabase.ranked_sites_dict, the include (whitelist) tag filter
compared the site's raw tags against the lowercased query tags:
is_tags_ok = lambda x: set(x.tags).intersection(set(normalized_tags))
while the exclude (blacklist) filter and every sibling lambda
(name/source/engine) lowercase the site-side value:
is_excluded_by_tag = lambda x: set(map(str.lower, x.tags)).intersection(
set(normalized_excluded_tags)
)
So a site stored with an upper/mixed-case tag (e.g. a custom or submitted
site tagged 'US') was excluded by `--exclude-tags us` but NOT found by
`--tags us` — an asymmetry between the two filters. Lowercase the site
tags in the include filter too, matching the rest of the method.
Adds a regression test asserting tags=['us'] finds a site tagged 'US'.
Serialize the maigret graph (the same one --graph builds) into an idempotent Cypher script importable into Neo4j. Reuses MaigretGraph via an extracted _build_maigret_graph() helper, leaving save_graph_report behavior unchanged; no new runtime dependency. Adds the --neo4j flag, the neo4j_report setting, docs, and a unit test.
Closes#2630
URLMatcher._HTTP_URL_RE_STR was "^https?://(www.|m.)?(.+)$". The dots in
the optional (www.|m.)? subdomain-prefix group were unescaped, so each `.`
matched ANY character. For a host that simply starts with `m` or `www`
(not the literal `m.`/`www.` prefix), the group greedily ate the first
characters:
extract_main_part('https://medium.com/alice') -> 'dium.com/alice'
extract_main_part('https://myspace.com/bob') -> 'space.com/bob'
extract_main_part feeds make_profile_url_regexp / get_url_template, so the
generated site-detection regexp was corrupted too (medium.com -> dium.com),
and the loose prefix could match bogus hosts like `xmedium.com`. 170+ hosts
in resources/data.json start with `m`.
Fix: escape the dots with a raw string, r"^https?://(www\.|m\.)?(.+)$".
Genuine `m.`/`www.` prefixes are still stripped
(m.wikipedia.org -> wikipedia.org). The two existing tests that pinned the
unescaped pattern string are updated to the corrected form, and a
regression test covers hosts beginning with m/www.
extract_and_group stored the per-error-type percentage already rounded to
2 decimals, and is_important() compares that stored value against the
threshold. Rounding before the comparison can push a rate that is strictly
below the threshold up to it:
8 Captcha errors / 267 sites = 2.99625%
round(2.99625, 2) == 3.0 -> is_important() True -> spurious
'Too many errors of type "Captcha" (3.0%)' warning
This is the residual of the same class fixed in #2788 (scale-then-round):
scaling to a percentage first stopped 2.5% from becoming 3.0%, but rounding
the percentage still feeds the threshold check. The display sites already
round (round(e["perc"], 2)), so storing the raw percentage keeps the shown
values unchanged while making the threshold comparison exact.
Genuine at-or-above-threshold rates still fire (3/100 = 3.0%, DNS 10%).
Adds a regression test for the 2.996% -> 3.0% rounding-up case.
`get_dict_ascii_tree(items, prepend="", new_line=True)` immediately reassigned
`new_line` to the horizontal box-drawing glyph ("─"), shadowing the boolean
parameter. The trailing `if not new_line: text = text[1:]` — meant to strip the
leading newline — therefore tested a non-empty string (always truthy) and never
ran. Callers passing `new_line=False` (e.g. `maigret.py` printing a user's
identity data) got a stray leading blank line.
Rename the glyph local to `h_line` so the `new_line` parameter is preserved and
its strip takes effect. The tree drawing is unchanged.
Adds a regression test asserting `new_line=False` drops the leading newline
while keeping the rest of the tree identical.
Adds quick-access links to web archives (Wayback Machine and archive.is)
in profile URL report blocks, allowing users to see historical snapshots
of profile pages when they still exist but are no longer accessible.
Closes#247
Co-authored-by: yyq1043 <yyq1043@outlook.com>
