* refactor: drop dead 'if not dictionary' guards across report paths
Closes#2665.
Every site results entry flows through make_site_result (checking.py:788),
which initializes results_site = {} and then unconditionally populates it
(site/username/keywords/parsing_enabled/url_main/cookies up front, checker
at the end, plus status or url_user+future on every branch). It has no
return path that yields an empty dict. check_site_for_username wraps that
result and likewise never returns a falsy entry, and maigret.py does not
construct site-result dicts by hand.
So the four downstream 'if not dictionary: continue' guards (all tagged
'# TODO: fix no site data issue') were dead code — the entries they
guarded against cannot occur. Removed:
- maigret/maigret.py:101 (extract_ids_from_results)
- maigret/report.py:151 — 'if not dictionary or dictionary.get("is_similar")'
trimmed to just the is_similar check (real logic)
- maigret/report.py:467 (extended-report builder)
- maigret/report.py:609 (generate_txt_report)
- maigret/report.py:627 — 'if not site_result or not site_result.get("status")'
trimmed to just the status check (real logic)
- maigret/report.py:687 (generate_json_report)
plus the four '# TODO: fix no site data issue' comments.
No behavior change for well-formed inputs; downstream .get() calls are
safe because entries are always populated dicts. 292 tests pass.
* feat: add Chinese name false-positive fix + CSDN site
- #2633: When searching non-ASCII usernames, skip presence detection
if the response body doesn't contain the username at all, preventing
false positives on sites that return generic error pages.
- #2634: Add CSDN (blog.csdn.net) — major Chinese dev blog with
40M+ users, alexa rank #45 in China.
* fix: add tls_fingerprint to CSDN + add tests for non-ASCII false positive fix
Addresses PR #2876 review feedback from soxoj:
- Add tls_fingerprint to CSDN protection (curl_cffi required)
- Add 2 test cases for #2633 non-ASCII username false positive fix:
- non-ASCII username not in response → no false CLAIMED
- non-ASCII username in response → normal CLAIMED behavior
---------
Co-authored-by: aznikline <aznikline@users.noreply.github.com>
Two safe, mechanical changes:
- activation.py: replace `list(domain.values())[0]` with
`next(iter(domain.values()))` in import_aiohttp_cookies to avoid
materializing the entire iterable just to grab the first dict.
- checking.py: reorganize imports to PEP 8 order (stdlib, third-party,
local). `from unittest.mock import Mock` and the stdlib `quote`
import move into the stdlib section; `python_socks` and
`socid_extractor` move into third-party; `from .error_detection import
detect_error_page` and the `from .` local imports move into the local
section.
No behavioral change. Same diff shape as 2484509 -> HEAD for these two
files minus the else-removal work.
Pixwox's protection was tagged only ip_reputation, but its actual
failure mode is a Cloudflare "Just a moment" JS challenge. Since
build_cloudflare_bypass_config() only activates the bypass module for
sites whose protection matches trigger_protection
(cf_js_challenge/cf_firewall/webgate), --cloudflare-bypass was
silently never invoked for Pixwox even with a working FlareSolverr
instance configured.
Verified FlareSolverr solves the exact URL directly (200, real
profile HTML) — the gap was purely in the trigger match, not the
solver.
Fixes#2845
* Fix site checks: 82 fixed, 11 disabled + DiscourseJson engine
* Fix test_hackernews_requires_profile_marker: update test body to JSON format matching Firebase API probe
The opening <html> tag contains a malformed type="text/css" attribute
which makes the HTML invalid (leading > missing). In HTML5 the <html>
tag does not require any attributes.
Fixes the malformed HTML that appears in generated PDF reports.
Co-authored-by: yaqing <yyq1043-cloud@users.noreply.github.com>
In MaigretDatabase.ranked_sites_dict, the include (whitelist) tag filter
compared the site's raw tags against the lowercased query tags:
is_tags_ok = lambda x: set(x.tags).intersection(set(normalized_tags))
while the exclude (blacklist) filter and every sibling lambda
(name/source/engine) lowercase the site-side value:
is_excluded_by_tag = lambda x: set(map(str.lower, x.tags)).intersection(
set(normalized_excluded_tags)
)
So a site stored with an upper/mixed-case tag (e.g. a custom or submitted
site tagged 'US') was excluded by `--exclude-tags us` but NOT found by
`--tags us` — an asymmetry between the two filters. Lowercase the site
tags in the include filter too, matching the rest of the method.
Adds a regression test asserting tags=['us'] finds a site tagged 'US'.
Serialize the maigret graph (the same one --graph builds) into an idempotent Cypher script importable into Neo4j. Reuses MaigretGraph via an extracted _build_maigret_graph() helper, leaving save_graph_report behavior unchanged; no new runtime dependency. Adds the --neo4j flag, the neo4j_report setting, docs, and a unit test.
Closes#2630
