* Fix site checks: 82 fixed, 11 disabled + DiscourseJson engine
* Fix test_hackernews_requires_profile_marker: update test body to JSON format matching Firebase API probe
In MaigretDatabase.ranked_sites_dict, the include (whitelist) tag filter
compared the site's raw tags against the lowercased query tags:
is_tags_ok = lambda x: set(x.tags).intersection(set(normalized_tags))
while the exclude (blacklist) filter and every sibling lambda
(name/source/engine) lowercase the site-side value:
is_excluded_by_tag = lambda x: set(map(str.lower, x.tags)).intersection(
set(normalized_excluded_tags)
)
So a site stored with an upper/mixed-case tag (e.g. a custom or submitted
site tagged 'US') was excluded by `--exclude-tags us` but NOT found by
`--tags us` — an asymmetry between the two filters. Lowercase the site
tags in the include filter too, matching the rest of the method.
Adds a regression test asserting tags=['us'] finds a site tagged 'US'.
Serialize the maigret graph (the same one --graph builds) into an idempotent Cypher script importable into Neo4j. Reuses MaigretGraph via an extracted _build_maigret_graph() helper, leaving save_graph_report behavior unchanged; no new runtime dependency. Adds the --neo4j flag, the neo4j_report setting, docs, and a unit test.
Closes#2630
URLMatcher._HTTP_URL_RE_STR was "^https?://(www.|m.)?(.+)$". The dots in
the optional (www.|m.)? subdomain-prefix group were unescaped, so each `.`
matched ANY character. For a host that simply starts with `m` or `www`
(not the literal `m.`/`www.` prefix), the group greedily ate the first
characters:
extract_main_part('https://medium.com/alice') -> 'dium.com/alice'
extract_main_part('https://myspace.com/bob') -> 'space.com/bob'
extract_main_part feeds make_profile_url_regexp / get_url_template, so the
generated site-detection regexp was corrupted too (medium.com -> dium.com),
and the loose prefix could match bogus hosts like `xmedium.com`. 170+ hosts
in resources/data.json start with `m`.
Fix: escape the dots with a raw string, r"^https?://(www\.|m\.)?(.+)$".
Genuine `m.`/`www.` prefixes are still stripped
(m.wikipedia.org -> wikipedia.org). The two existing tests that pinned the
unescaped pattern string are updated to the corrected form, and a
regression test covers hosts beginning with m/www.
extract_and_group stored the per-error-type percentage already rounded to
2 decimals, and is_important() compares that stored value against the
threshold. Rounding before the comparison can push a rate that is strictly
below the threshold up to it:
8 Captcha errors / 267 sites = 2.99625%
round(2.99625, 2) == 3.0 -> is_important() True -> spurious
'Too many errors of type "Captcha" (3.0%)' warning
This is the residual of the same class fixed in #2788 (scale-then-round):
scaling to a percentage first stopped 2.5% from becoming 3.0%, but rounding
the percentage still feeds the threshold check. The display sites already
round (round(e["perc"], 2)), so storing the raw percentage keeps the shown
values unchanged while making the threshold comparison exact.
Genuine at-or-above-threshold rates still fire (3/100 = 3.0%, DNS 10%).
Adds a regression test for the 2.996% -> 3.0% rounding-up case.
`get_dict_ascii_tree(items, prepend="", new_line=True)` immediately reassigned
`new_line` to the horizontal box-drawing glyph ("─"), shadowing the boolean
parameter. The trailing `if not new_line: text = text[1:]` — meant to strip the
leading newline — therefore tested a non-empty string (always truthy) and never
ran. Callers passing `new_line=False` (e.g. `maigret.py` printing a user's
identity data) got a stray leading blank line.
Rename the glyph local to `h_line` so the `new_line` parameter is preserved and
its strip takes effect. The tree drawing is unchanged.
Adds a regression test asserting `new_line=False` drops the leading newline
while keeping the rest of the tree identical.
Closes#2666.
Issue #2666 tracked four paths flagged `# TODO: tests`. Two of them
(`executors.py` increment_progress / stop_progress) were removed by the
#2789 refactor and no longer exist. This PR covers the two that remain:
1. maigret/checking.py:247 — the ClientSession cookie_jar forwarding.
SimpleAiohttpChecker.check() builds ClientSession(cookie_jar=self.cookie_jar
if self.cookie_jar else None), but nothing asserted the jar handed to the
checker actually reached the outgoing session. Added two tests using the
existing constructor-capture pattern (_capture_clientsession): one asserts
a configured jar is forwarded verbatim, the other asserts None is passed
when no jar is configured (the else branch).
2. maigret/maigret.py:946 — extract_ids_from_results. The pre-existing
test_extract_ids_from_results was a bare expression with no `assert` (so
it silently passed regardless of return value) AND pinned its expectation
against test_db (tests/db.json), whose site set never matched the Reddit
URL and silently dropped the ids_links result. Fixed: real assertion against
default_db, plus three branch tests (cross-site username merge, empty-site
skip, empty input).
No new dependencies: aioresponses was suggested in the issue but is not in
the project — the existing monkeypatch capture pattern fits the codebase
better and adds zero deps.
Removed both `# TODO: tests` comments from the now-covered lines.
Co-authored-by: aznikline <aznikline@users.noreply.github.com>
`extract_and_group` computed the per-error-type percentage as
`round(count / len(search_res), 2) * 100` — it rounded the *fraction* to 2
decimals (i.e. to the nearest 1%) and only then scaled by 100, so `perc`
always had whole-percent granularity and was mis-rounded.
A genuine 2.5% error rate (1 error out of 40 sites) became
`round(0.025, 2) * 100 = 3.0%` and tripped the 3% "important" threshold in
`is_important()`, firing a spurious "Too many errors of type ..." warning.
Likewise 9.5% DNS errors rounded to 10.0% and crossed the DNS override.
Scale to a percentage *before* rounding: `round(count / len(search_res) * 100, 2)`.
The existing tests use integer-landing ratios (25/100, 5/100, 3/100) that are
identical under both formulas, so none of them change.
Adds a regression test for a sub-threshold non-integer rate (2.5%) staying silent.
Convert activation HTTP calls to aiohttp coroutines, await activation before retrying, and allocate independent protocol checkers per site check so concurrent retries do not overwrite shared checker state.
is_country_tag checks for a 2-letter alpha code. The field names
'country' and 'locale' are 6-7 characters and never match, so the
direct alpha_2 lookup branch was dead code and all country values fell
through to search_fuzzy unconditionally.
Change is_country_tag(k) -> is_country_tag(v) so that 2-letter ISO
codes (e.g. 'US', 'RU') use the fast direct lookup while full country
names continue to use fuzzy search.
Fixes#2752
update_site() used `s = site` inside a for-loop, which rebinds only the
local loop variable and leaves self._sites[i] unchanged. The method
returned self silently appearing to succeed while the database list
retained the original entry. This broke --auto-disable for any site
already present in the database.
Fix with enumerate so the actual list element is replaced.
Fixes#2750
* Make xhtml2pdf optional, fix install on Linux without libcairo
Move xhtml2pdf to the new [pdf] extra so default `pip install maigret`
no longer pulls pycairo (which has no Linux/macOS wheels and breaks the
build without libcairo2-dev). save_pdf_report now raises a clear
RuntimeError pointing to `pip install 'maigret[pdf]'`, and the CLI
turns it into a friendly warning instead of a crash. Adds tests
covering the missing-extra path, plus per-OS install docs.
Fix for #2657, #2534
* Make arabic-reshaper and python-bidi optional; idempotent update of db_meta.json and sites.md
* Regenerated poerty.lock
* Update CI workflow to cover minimal installation without PDF deps
The previous /reports/<path:filename> handler resolved the filename with
os.path.normpath and gated send_file on file_path.startswith(REPORTS_FOLDER).
Plain ../ traversal was rejected because the resolved path no longer started
with REPORTS_FOLDER, but a sibling-prefix variant slipped through: a request
of the form ..%2F<reports_root_basename>2/<file> resolves to a path like
/tmp/maigret_reports2/<file>, which still starts with /tmp/maigret_reports
and was served back to the caller.
Replace the manual normpath+startswith check with Flask's send_from_directory,
which delegates to werkzeug.security.safe_join. safe_join enforces a real
boundary against the resolved directory, rejects absolute paths, and refuses
.. segments that escape the root.
Tests: 4 new test_download_report_* cases in tests/test_web.py covering the
happy path, ../ traversal, the sibling-prefix bypass (regression test —
fails on the pre-fix code, passes on the new code), and absolute paths.
Detected by Aeon + manual review of maigret.web.app.
Severity: low (web UI defaults to FLASK_HOST=127.0.0.1; the Docker `web`
target binds 0.0.0.0; exploitation reads files from sibling /tmp directories,
which is bounded by who can place files there).
CWE-22.
Co-authored-by: aeonframework <aeon-bot@aaronjmars.com>
* Fix ID extraction crash when regex groups are optional
Handle None capture groups in username/id extraction and add regression coverage for optional trailing groups.
* Remove leftover line that overwrote safe _id in extract_id_from_url
the <0.3/<0.4/etc upper bounds don't leave room for darwin or
emulated/aarch64 runners, which have been seeing 0.7s+ on tests
that expected <0.3s.
bumped each upper bound by +0.7s. lower bounds unchanged — they
still validate that tasks ran in parallel rather than serially.
refs #679
Co-authored-by: Julio César Suástegui <juliosuas@users.noreply.github.com>
- Added social tag to social networks (33 sites)
- Fixed wrong tags (8 sites)
- Filled empty tags for 213 sites in top-1000
- Country tag cleanup (~374 sites)
- Site naming normalization (75 sites)
- New tests (3)
- Documentation updates
- Fix VK and TradingView checkType; add Reddit and Microsoft Learn API-style probes where appropriate; adjust or disable entries that are unreliable under anti-bot protection.
- Self-check: stop aggressive auto-disable; default to reporting issues only; add --auto-disable and --diagnose for optional fixes and deeper output.
- Tooling: add utils/site_check.py and utils/check_top_n.py (and related helpers) to inspect and rank site behavior against the top-N list
- Scope: aligns with fixing top-traffic / high-impact sites and making diagnostics repeatable without silently flipping disabled flags
