Closes#2666.
Issue #2666 tracked four paths flagged `# TODO: tests`. Two of them
(`executors.py` increment_progress / stop_progress) were removed by the
#2789 refactor and no longer exist. This PR covers the two that remain:
1. maigret/checking.py:247 — the ClientSession cookie_jar forwarding.
SimpleAiohttpChecker.check() builds ClientSession(cookie_jar=self.cookie_jar
if self.cookie_jar else None), but nothing asserted the jar handed to the
checker actually reached the outgoing session. Added two tests using the
existing constructor-capture pattern (_capture_clientsession): one asserts
a configured jar is forwarded verbatim, the other asserts None is passed
when no jar is configured (the else branch).
2. maigret/maigret.py:946 — extract_ids_from_results. The pre-existing
test_extract_ids_from_results was a bare expression with no `assert` (so
it silently passed regardless of return value) AND pinned its expectation
against test_db (tests/db.json), whose site set never matched the Reddit
URL and silently dropped the ids_links result. Fixed: real assertion against
default_db, plus three branch tests (cross-site username merge, empty-site
skip, empty input).
No new dependencies: aioresponses was suggested in the issue but is not in
the project — the existing monkeypatch capture pattern fits the codebase
better and adds zero deps.
Removed both `# TODO: tests` comments from the now-covered lines.
Co-authored-by: aznikline <aznikline@users.noreply.github.com>
`extract_and_group` computed the per-error-type percentage as
`round(count / len(search_res), 2) * 100` — it rounded the *fraction* to 2
decimals (i.e. to the nearest 1%) and only then scaled by 100, so `perc`
always had whole-percent granularity and was mis-rounded.
A genuine 2.5% error rate (1 error out of 40 sites) became
`round(0.025, 2) * 100 = 3.0%` and tripped the 3% "important" threshold in
`is_important()`, firing a spurious "Too many errors of type ..." warning.
Likewise 9.5% DNS errors rounded to 10.0% and crossed the DNS override.
Scale to a percentage *before* rounding: `round(count / len(search_res) * 100, 2)`.
The existing tests use integer-landing ratios (25/100, 5/100, 3/100) that are
identical under both formulas, so none of them change.
Adds a regression test for a sub-threshold non-integer rate (2.5%) staying silent.
Convert activation HTTP calls to aiohttp coroutines, await activation before retrying, and allocate independent protocol checkers per site check so concurrent retries do not overwrite shared checker state.
is_country_tag checks for a 2-letter alpha code. The field names
'country' and 'locale' are 6-7 characters and never match, so the
direct alpha_2 lookup branch was dead code and all country values fell
through to search_fuzzy unconditionally.
Change is_country_tag(k) -> is_country_tag(v) so that 2-letter ISO
codes (e.g. 'US', 'RU') use the fast direct lookup while full country
names continue to use fuzzy search.
Fixes#2752
update_site() used `s = site` inside a for-loop, which rebinds only the
local loop variable and leaves self._sites[i] unchanged. The method
returned self silently appearing to succeed while the database list
retained the original entry. This broke --auto-disable for any site
already present in the database.
Fix with enumerate so the actual list element is replaced.
Fixes#2750
* Make xhtml2pdf optional, fix install on Linux without libcairo
Move xhtml2pdf to the new [pdf] extra so default `pip install maigret`
no longer pulls pycairo (which has no Linux/macOS wheels and breaks the
build without libcairo2-dev). save_pdf_report now raises a clear
RuntimeError pointing to `pip install 'maigret[pdf]'`, and the CLI
turns it into a friendly warning instead of a crash. Adds tests
covering the missing-extra path, plus per-OS install docs.
Fix for #2657, #2534
* Make arabic-reshaper and python-bidi optional; idempotent update of db_meta.json and sites.md
* Regenerated poerty.lock
* Update CI workflow to cover minimal installation without PDF deps
The previous /reports/<path:filename> handler resolved the filename with
os.path.normpath and gated send_file on file_path.startswith(REPORTS_FOLDER).
Plain ../ traversal was rejected because the resolved path no longer started
with REPORTS_FOLDER, but a sibling-prefix variant slipped through: a request
of the form ..%2F<reports_root_basename>2/<file> resolves to a path like
/tmp/maigret_reports2/<file>, which still starts with /tmp/maigret_reports
and was served back to the caller.
Replace the manual normpath+startswith check with Flask's send_from_directory,
which delegates to werkzeug.security.safe_join. safe_join enforces a real
boundary against the resolved directory, rejects absolute paths, and refuses
.. segments that escape the root.
Tests: 4 new test_download_report_* cases in tests/test_web.py covering the
happy path, ../ traversal, the sibling-prefix bypass (regression test —
fails on the pre-fix code, passes on the new code), and absolute paths.
Detected by Aeon + manual review of maigret.web.app.
Severity: low (web UI defaults to FLASK_HOST=127.0.0.1; the Docker `web`
target binds 0.0.0.0; exploitation reads files from sibling /tmp directories,
which is bounded by who can place files there).
CWE-22.
Co-authored-by: aeonframework <aeon-bot@aaronjmars.com>
* Fix ID extraction crash when regex groups are optional
Handle None capture groups in username/id extraction and add regression coverage for optional trailing groups.
* Remove leftover line that overwrote safe _id in extract_id_from_url
the <0.3/<0.4/etc upper bounds don't leave room for darwin or
emulated/aarch64 runners, which have been seeing 0.7s+ on tests
that expected <0.3s.
bumped each upper bound by +0.7s. lower bounds unchanged — they
still validate that tasks ran in parallel rather than serially.
refs #679
Co-authored-by: Julio César Suástegui <juliosuas@users.noreply.github.com>
- Added social tag to social networks (33 sites)
- Fixed wrong tags (8 sites)
- Filled empty tags for 213 sites in top-1000
- Country tag cleanup (~374 sites)
- Site naming normalization (75 sites)
- New tests (3)
- Documentation updates
- Fix VK and TradingView checkType; add Reddit and Microsoft Learn API-style probes where appropriate; adjust or disable entries that are unreliable under anti-bot protection.
- Self-check: stop aggressive auto-disable; default to reporting issues only; add --auto-disable and --diagnose for optional fixes and deeper output.
- Tooling: add utils/site_check.py and utils/check_top_n.py (and related helpers) to inspect and rank site behavior against the top-N list
- Scope: aligns with fixing top-traffic / high-impact sites and making diagnostics repeatable without silently flipping disabled flags
The `Settings.load()` method iterates through multiple configuration file paths and updates the internal `__dict__`, intending to override earlier default settings with later user-specific ones. This cascading logic is a core configuration feature but lacks explicit tests to guarantee that dictionary merging and overriding behave exactly as documented (e.g., ensuring a setting in `~/.maigret/settings.json` correctly overrides `resources/settings.json` without wiping out other keys).
Affected files: test_settings.py
