Two safe, mechanical changes:
- activation.py: replace `list(domain.values())[0]` with
`next(iter(domain.values()))` in import_aiohttp_cookies to avoid
materializing the entire iterable just to grab the first dict.
- checking.py: reorganize imports to PEP 8 order (stdlib, third-party,
local). `from unittest.mock import Mock` and the stdlib `quote`
import move into the stdlib section; `python_socks` and
`socid_extractor` move into third-party; `from .error_detection import
detect_error_page` and the `from .` local imports move into the local
section.
No behavioral change. Same diff shape as 2484509 -> HEAD for these two
files minus the else-removal work.
Pixwox's protection was tagged only ip_reputation, but its actual
failure mode is a Cloudflare "Just a moment" JS challenge. Since
build_cloudflare_bypass_config() only activates the bypass module for
sites whose protection matches trigger_protection
(cf_js_challenge/cf_firewall/webgate), --cloudflare-bypass was
silently never invoked for Pixwox even with a working FlareSolverr
instance configured.
Verified FlareSolverr solves the exact URL directly (200, real
profile HTML) — the gap was purely in the trigger match, not the
solver.
Fixes#2845
* Fix site checks: 82 fixed, 11 disabled + DiscourseJson engine
* Fix test_hackernews_requires_profile_marker: update test body to JSON format matching Firebase API probe
The opening <html> tag contains a malformed type="text/css" attribute
which makes the HTML invalid (leading > missing). In HTML5 the <html>
tag does not require any attributes.
Fixes the malformed HTML that appears in generated PDF reports.
Co-authored-by: yaqing <yyq1043-cloud@users.noreply.github.com>
In MaigretDatabase.ranked_sites_dict, the include (whitelist) tag filter
compared the site's raw tags against the lowercased query tags:
is_tags_ok = lambda x: set(x.tags).intersection(set(normalized_tags))
while the exclude (blacklist) filter and every sibling lambda
(name/source/engine) lowercase the site-side value:
is_excluded_by_tag = lambda x: set(map(str.lower, x.tags)).intersection(
set(normalized_excluded_tags)
)
So a site stored with an upper/mixed-case tag (e.g. a custom or submitted
site tagged 'US') was excluded by `--exclude-tags us` but NOT found by
`--tags us` — an asymmetry between the two filters. Lowercase the site
tags in the include filter too, matching the rest of the method.
Adds a regression test asserting tags=['us'] finds a site tagged 'US'.
Serialize the maigret graph (the same one --graph builds) into an idempotent Cypher script importable into Neo4j. Reuses MaigretGraph via an extracted _build_maigret_graph() helper, leaving save_graph_report behavior unchanged; no new runtime dependency. Adds the --neo4j flag, the neo4j_report setting, docs, and a unit test.
Closes#2630
URLMatcher._HTTP_URL_RE_STR was "^https?://(www.|m.)?(.+)$". The dots in
the optional (www.|m.)? subdomain-prefix group were unescaped, so each `.`
matched ANY character. For a host that simply starts with `m` or `www`
(not the literal `m.`/`www.` prefix), the group greedily ate the first
characters:
extract_main_part('https://medium.com/alice') -> 'dium.com/alice'
extract_main_part('https://myspace.com/bob') -> 'space.com/bob'
extract_main_part feeds make_profile_url_regexp / get_url_template, so the
generated site-detection regexp was corrupted too (medium.com -> dium.com),
and the loose prefix could match bogus hosts like `xmedium.com`. 170+ hosts
in resources/data.json start with `m`.
Fix: escape the dots with a raw string, r"^https?://(www\.|m\.)?(.+)$".
Genuine `m.`/`www.` prefixes are still stripped
(m.wikipedia.org -> wikipedia.org). The two existing tests that pinned the
unescaped pattern string are updated to the corrected form, and a
regression test covers hosts beginning with m/www.
extract_and_group stored the per-error-type percentage already rounded to
2 decimals, and is_important() compares that stored value against the
threshold. Rounding before the comparison can push a rate that is strictly
below the threshold up to it:
8 Captcha errors / 267 sites = 2.99625%
round(2.99625, 2) == 3.0 -> is_important() True -> spurious
'Too many errors of type "Captcha" (3.0%)' warning
This is the residual of the same class fixed in #2788 (scale-then-round):
scaling to a percentage first stopped 2.5% from becoming 3.0%, but rounding
the percentage still feeds the threshold check. The display sites already
round (round(e["perc"], 2)), so storing the raw percentage keeps the shown
values unchanged while making the threshold comparison exact.
Genuine at-or-above-threshold rates still fire (3/100 = 3.0%, DNS 10%).
Adds a regression test for the 2.996% -> 3.0% rounding-up case.
`get_dict_ascii_tree(items, prepend="", new_line=True)` immediately reassigned
`new_line` to the horizontal box-drawing glyph ("─"), shadowing the boolean
parameter. The trailing `if not new_line: text = text[1:]` — meant to strip the
leading newline — therefore tested a non-empty string (always truthy) and never
ran. Callers passing `new_line=False` (e.g. `maigret.py` printing a user's
identity data) got a stray leading blank line.
Rename the glyph local to `h_line` so the `new_line` parameter is preserved and
its strip takes effect. The tree drawing is unchanged.
Adds a regression test asserting `new_line=False` drops the leading newline
while keeping the rest of the tree identical.
Adds quick-access links to web archives (Wayback Machine and archive.is)
in profile URL report blocks, allowing users to see historical snapshots
of profile pages when they still exist but are no longer accessible.
Closes#247
Co-authored-by: yyq1043 <yyq1043@outlook.com>
Closes#2665.
Every site results entry flows through make_site_result (checking.py:788),
which initializes results_site = {} and then unconditionally populates it
(site/username/keywords/parsing_enabled/url_main/cookies up front, checker
at the end, plus status or url_user+future on every branch). It has no
return path that yields an empty dict. check_site_for_username wraps that
result and likewise never returns a falsy entry, and maigret.py does not
construct site-result dicts by hand.
So the four downstream 'if not dictionary: continue' guards (all tagged
'# TODO: fix no site data issue') were dead code — the entries they
guarded against cannot occur. Removed:
- maigret/maigret.py:101 (extract_ids_from_results)
- maigret/report.py:151 — 'if not dictionary or dictionary.get("is_similar")'
trimmed to just the is_similar check (real logic)
- maigret/report.py:467 (extended-report builder)
- maigret/report.py:609 (generate_txt_report)
- maigret/report.py:627 — 'if not site_result or not site_result.get("status")'
trimmed to just the status check (real logic)
- maigret/report.py:687 (generate_json_report)
plus the four '# TODO: fix no site data issue' comments.
No behavior change for well-formed inputs; downstream .get() calls are
safe because entries are always populated dicts. 292 tests pass.
Co-authored-by: aznikline <aznikline@users.noreply.github.com>
Closes#2666.
Issue #2666 tracked four paths flagged `# TODO: tests`. Two of them
(`executors.py` increment_progress / stop_progress) were removed by the
#2789 refactor and no longer exist. This PR covers the two that remain:
1. maigret/checking.py:247 — the ClientSession cookie_jar forwarding.
SimpleAiohttpChecker.check() builds ClientSession(cookie_jar=self.cookie_jar
if self.cookie_jar else None), but nothing asserted the jar handed to the
checker actually reached the outgoing session. Added two tests using the
existing constructor-capture pattern (_capture_clientsession): one asserts
a configured jar is forwarded verbatim, the other asserts None is passed
when no jar is configured (the else branch).
2. maigret/maigret.py:946 — extract_ids_from_results. The pre-existing
test_extract_ids_from_results was a bare expression with no `assert` (so
it silently passed regardless of return value) AND pinned its expectation
against test_db (tests/db.json), whose site set never matched the Reddit
URL and silently dropped the ids_links result. Fixed: real assertion against
default_db, plus three branch tests (cross-site username merge, empty-site
skip, empty input).
No new dependencies: aioresponses was suggested in the issue but is not in
the project — the existing monkeypatch capture pattern fits the codebase
better and adds zero deps.
Removed both `# TODO: tests` comments from the now-covered lines.
Co-authored-by: aznikline <aznikline@users.noreply.github.com>
`extract_and_group` computed the per-error-type percentage as
`round(count / len(search_res), 2) * 100` — it rounded the *fraction* to 2
decimals (i.e. to the nearest 1%) and only then scaled by 100, so `perc`
always had whole-percent granularity and was mis-rounded.
A genuine 2.5% error rate (1 error out of 40 sites) became
`round(0.025, 2) * 100 = 3.0%` and tripped the 3% "important" threshold in
`is_important()`, firing a spurious "Too many errors of type ..." warning.
Likewise 9.5% DNS errors rounded to 10.0% and crossed the DNS override.
Scale to a percentage *before* rounding: `round(count / len(search_res) * 100, 2)`.
The existing tests use integer-landing ratios (25/100, 5/100, 3/100) that are
identical under both formulas, so none of them change.
Adds a regression test for a sub-threshold non-integer rate (2.5%) staying silent.
