settings.json ships web_interface_port and Settings declares it, but
nothing read it: --web had const=5000, and main() fell back to a second
hardcoded 5000 for --web 0. A port set in ~/.maigret/settings.json or
./settings.json was ignored.
Take both from settings.web_interface_port. The shipped value is 5000,
so the default does not change.
Co-authored-by: datrixlab <325650023+datrixlab@users.noreply.github.com>
self_check() has no cookies parameter, so the CLI cannot forward
args.cookie_file and every site check runs without the cookie jar. The
layer below already takes and plumbs it: site_self_check(cookies=None)
passes cookies through to maigret(), and submit.py passes
cookies=self.args.cookie_file into that same function. Self-check was the
one of the three paths that dropped it.
The effect is worse than a missing option. errors.py maps "Login required"
to the advice "Add authorization cookies through --cookies-jar-file", so a
login-walled site self-checked without the jar reports the claimed profile
as not detected, and with --auto-disable that writes disabled=True into
the sites database. CONTRIBUTING tells contributors to run
`maigret --self-check --site "SiteName" --diagnose`.
Same shape as #3026, which did this for --dns-resolver.
Co-authored-by: datrixlab <325650023+datrixlab@users.noreply.github.com>
save_xmind_report only set `category` when it created a tag section. For
a tag that already had one, the loop hit `continue` before assigning it,
so `category` stayed None and the site went to "Undefined". Only the
first site per tag ever reached that tag's section.
A real scan puts dozens of sites under tags like social, coding or
forum, so in practice nearly every account landed in "Undefined" and
each tag section held exactly one entry.
Assign the category for existing sections too. When all of a site's tags
are new the result is unchanged -- the last one still wins -- so this
only changes the case that was broken.
The existing XMind test uses a single site, which cannot tell "filed
under its tag" from "filed under its tag only if it was first"; the new
one uses two sites sharing a tag.
Co-authored-by: datrixlab <325650023+datrixlab@users.noreply.github.com>
os.fsync on Windows is FlushFileBuffers, which needs write access and
raises EBADF on a handle opened read-only. _normalize_xmind_archive
flushed the rewritten archive through open(..., 'rb'), so every --xmind
run on Windows died with OSError: [Errno 9] Bad file descriptor.
The reporting loop does not guard that call and XMind is the first
format written, so the run also lost the CSV, TXT and JSON reports asked
for alongside it and never reached the HTML/PDF stage.
Flush through 'rb+', and do it before the chmod: reopening the file for
writing after archive_mode has been applied would fail outright whenever
that mode is read-only.
The save at the end of Submitter.dialog read self.args.db, an attribute
argparse never creates: maigret.py declares --db with dest="db_file", and
the line right above it in the same block already reads self.args.db_file.
So `maigret --submit <url> --db custom.json` ran the whole interactive
dialog and then raised AttributeError on the final step. dialog() never
returned True, so main() never reached save_db_safely() either, and the
newly submitted site was lost after every question had been answered. The
default path was unaffected, because the block is guarded by
`db_file != settings.sites_db_path`.
Removing the block rather than renaming the attribute: main() already
persists the database after a successful submit, through
save_db_safely(db, db_file, logger). That call takes its path from
resolve_db_path(), which returns an absolute or module-relative path for a
custom --db, so the removed line could have written to a different file
than the rest of the run uses. save_db_safely also tolerates a read-only
installation, which db.save_to_file() does not.
The existing dialog tests pass a MagicMock, which answers to any attribute
name, so the typo was invisible to them. The new test builds its namespace
with the real parser.
Co-authored-by: datrixlab <325650023+datrixlab@users.noreply.github.com>
- Count probing sites in the statistics instead of listing them
- Add data quality, country and protection statistics
- Fix the presenceStrs typo that disabled 13 site checks
- Put the statistics first and format them as Markdown
- Add a title and a table of contents to sites.md
- Move the site count into the intro line
- Declare the mirrors field, it is read at runtime
Moves 77 Mastodon instances onto a shared engine instead of copying urlProbe into every entry. The lookup API answers 404 for a missing account, so twelve instances that until now returned 200 for any username stop reporting everyone as found.
Closes#3119. Supersedes #3115, #3116, #3117.
An engine is a template, but `update_from_engine` overwrites a site's scalar
fields with the engine's, and `strip_engine_data` then removes them on save. A
value written into an entry by hand is therefore ignored at runtime **and
deleted from `data.json`** the next time anything calls `save_to_file` — which
`utils/update_site_data.py` does on every regeneration. Nothing is printed: the
dict case already warns about a collision, the scalar case did not.
Found while giving MediaWiki entries a JSON `urlProbe`. The API path is not the
page path — a wiki with articles under `/wiki` keeps `api.php` under `/w` — so
a per-entry probe is the natural way to say it, and it turned out to be
impossible.
Now the entry wins and the override survives the round trip, with the same
warning the dict case prints.
Only fields the entry **stated** count. The constructor synthesises
`alexa_rank` for every site that does not carry one, so keying the rule off the
instance dict made seventeen op.gg sites keep that placeholder instead of the
rank their engine supplies; there is a test for exactly that, next to one for
the override winning and one for it surviving `strip_engine_data`.
The shipped database is unchanged — it round-trips byte for byte through load
and save, and no entry currently sets a field its engine also defines.
The display URL already goes through quote(), but urlProbe did a bare
format() with the raw username. A "#" in the username therefore ended the
URL and turned the rest into a fragment, so the probe was sent to a
different account's URL and the site was reported as claimed.
Encoding the username here keeps the two URLs consistent. Every username
in data.json is unaffected, since quote() is a no-op for them.
* fix: keep result lines from crashing on a non-UTF-8 stdout
#3050 stopped the startup banners from raising UnicodeEncodeError on a
Windows console, but the banner was not the only line the ANSI codepage can
break. Two ordinary things a run prints still went through a bare print():
- the --enrich notifications built in checking.py carry U+2192, which cp1252
has no slot for (checking.py:1847, 1851, 1859, 1899);
- any profile field maigret extracts carries the script the page was written
in, so a Cyrillic or CJK fullname breaks the result line that reports it.
Both arrive mid-scan, after the network work is done, so the crash discarded
results the run had already collected -- worse than failing at startup.
Route every write in notify.py through the existing _print_encodable helper.
Verified on a real cp1252 stdout, not a mock: both cases raise before this
change and print with replacement characters after it.
* fix: keep the AI spinner alive on a stderr that cannot encode braille
Second half of #3055. The spinner frames are braille, which cp1252 -- the
ANSI codepage of a stock Windows install -- cannot encode, so the first frame
raised UnicodeEncodeError inside the daemon thread: the thread died with a
traceback printed over the output and the animation stopped for the rest of
the run.
Encoding with replacement would only leave a row of '?' spinning, so pick the
frame set from what the stream can carry instead. A UTF-8 terminal keeps the
braille; anything that cannot hold it gets |/-\ and still animates.
print_streaming() in the same module writes model output the same way and
would raise on any non-Latin reply, but it has no callers anywhere in the
tree, so it is left alone rather than fixed as dead code.
`pyinstaller.yml` only runs on pushes to `main` and `dev`, so `maigret_standalone.exe` exists only on the moving `nightly-*` tags. Every stable release from v0.6.1 to v0.6.5 has zero assets, and winget and Scoop cannot pin a moving tag by hash.
- add the `release: types: [published]` trigger, the same one `python-publish.yml` uses. Checkout resolves `refs/tags/<tag>`, so the exe is built from the released code.
- guard the two nightly-only steps on `github.event_name == 'push'`, otherwise `NIGHTLY_TAG` becomes `nightly-v0.6.5` and creates a junk tag beside the real one.
- add a step attaching the binary to the published release with every `omit*DuringUpdate` set, so the title, body, prerelease and draft state are left alone.
- `tests/test_workflows.py`: the `release_step` fixture asserted there was exactly one `release-action` step, which a second one breaks. It now selects a step by the event it is guarded on, and the new step gets its own guards: every release step must be event specific, the published one must set all four omits, and it must target `github.event.release.tag_name` rather than `NIGHTLY_TAG`.
Existing releases are not backfilled. A release now costs two PyInstaller runs, one on the tag and one on the push to `main` that follows.
Every one of them is a site that answers the same way for any username, so the check said "claimed" for everyone.
Two of them are proof-of-work interstitials that come back with a 2xx, and those are worth a global marker rather than a per-site patch: joyreactor.cc now runs Anubis (`/.within.website/x/`, HTTP 200 on every path) and fixya.com serves an HTTP 202 with `window.POW_CHALLENGE_DATA`. Both are vendor products used far beyond these two sites, so they go into COMMON_ERRORS and the sites report bot protection instead of a hit.
The rest are per-site rot. championat.com answers every `/user/*` with the same 686-byte SberID auth stub, profiles are behind login now, disabled. forum.heroesworld.ru redirects to the forum index, which never contains the vBulletin absence marker, the forum has moved to `heroesworld.ru/user/{username}/` where a missing user is a clean 404. Codédex was a status_code check and flapped to 200 on a missing profile once in about fifteen requests, so it now matches the og:title of a real profile.
OP.GG search falls back to other regions when the requested one has no hit: `?q=soxoj®ion=ru` returns an Oceania summoner, and the shared engine marker `href="/lol/summoners/` matched it, so all seventeen region entries claimed the account. The marker moves from the engine to each site with its region baked in. Name matching is already exact on op.gg's side, `q=blaz` returns only summoners named exactly Blaz, so the region was the whole leak.
Completes subtask 2 of #2668. When manual feature detection sees a claimed profile return 2xx and the unclaimed profile return 4xx, --submit creates a status_code check. Redirects keep the message fallback.
* fix: don't treat refused or self-redirected responses as not-found
401 and 429 fell through to the checkType branch, where any non-2xx code means the username is free, so blocked and rate-limited requests were reported as confirmed absences; they now return errors, Vercel's Security Checkpoint joins the bot-protection markers, and a self-redirect that carries Set-Cookie is retried once instead of read as an absence.
* Fix site checks: 16 sites, 3 fixed, 12 disabled, 1 dead deleted
* docs: note that the joyreactor self-redirect is intermittent
save_graph_report handed the file off to pyvis's show(), which opens it with
open(name, "w+"), i.e. the locale encoding. On Windows that is the ANSI
codepage (cp1252 by default), and the vis-network bundle pyvis inlines carries
several hundred non-ASCII characters of its own, so every graph report died
with UnicodeEncodeError regardless of what was scanned. The CLI aborted
mid-run and left a 0-byte html file behind; the web interface builds a graph
for every scan, so each one ended as "Search failed" and discarded the
results.
Render the html and write it here with encoding="utf-8" instead. The debug.log
dump and the report template read had the same implicit encoding, so they get
the same treatment.
Three existing tests (test_build_reports_computes_found_count,
test_real_report_generation_does_not_crash, test_live_scan_streams_found_and_done)
fail on a non-utf-8 machine before this change and pass after it. The new tests
perform the writes under -X warn_default_encoding, so a return to the implicit
encoding fails on CI too, where the locale is utf-8 and the bug is invisible.
Rotating/residential proxies often drop a connection mid-request
(truncated body, closed socket, failed handshake) instead of failing
outright. These were previously lumped into "Unexpected" and never
retried, silently discarding otherwise-successful checks.
- Retry once on ClientPayloadError, ServerDisconnectedError, and
aiohttp_socks' ProxyConnectionError/ProxyTimeoutError (aiohttp
checker), and on CurlError (curl_cffi checker) — covers truncated
responses, dropped connections, CONNECT-tunnel 502s, and TLS
handshake failures.
- Fix the proxy-error except clause: it caught python_socks'
ProxyError, but aiohttp_socks (the connector actually in use)
raises its own unrelated same-named exceptions, so it never fired.
- Generic ProxyError (e.g. bad credentials) is classified but NOT
retried — it fails identically every attempt.
extract_ids_from_page() has the same bug #2907 fixed in
checking.py::parse_usernames(). "username" is itself in SUPPORTED_IDS, so
the `if k in SUPPORTED_IDS` loop stores a bare `username` value directly,
bypassing the is_plausible_username() guard that extract_usernames()
applies in its own separate pass. A URL or email returned by
socid_extractor under a bare `username` key therefore becomes a recursive
search target via `maigret --parse-url`, reproducing the #1403
false-error cascade.
Skip keys containing "username" in the SUPPORTED_IDS loop; those are
already owned (and validated) by extract_usernames(). Every other
SUPPORTED_ID (gaia_id, vk_id, orcid, ...) is unaffected since none of
those keys contain "username".
Add regression tests for the bare `username` key with a URL value and for
the unchanged handling of other supported IDs.
Closes#2911
* fix: block SSRF and local-file reads via report image URLs in PDF generation
save_pdf_report() rendered scraped profile image URLs (ids_data['image'])
straight into xhtml2pdf, which fetches <img src> while building the PDF.
The image field is attacker-influenced and pisaDocument ran with no
link_callback, so a profile carrying image = "file:///etc/passwd" or an
intranet/metadata URL turned report generation into a local file read or
an SSRF from the machine running maigret. In the web UI this is
server-side and fires on every search, since save_pdf_report is always
called.
Add a link_callback that only lets public http(s) images through and
diverts everything else (file://, data:, other schemes, and hosts that
resolve to loopback/private/link-local/reserved addresses) to a bundled
1x1 placeholder, so no fetch or read happens. Diverting rather than
raising keeps report generation working when a scanned profile carries a
hostile image URL.
Tests cover the URL classifier, the callback's placeholder diversion, and
an end-to-end check that PDF generation does not fetch an internal image.
* fix: use is_global to also block CGNAT (100.64.0.0/10) report image hosts
The flag chain missed 100.64.0.0/10, which is neither is_private nor
is_global and is routable inside many cloud and k8s networks. is_global
covers it along with private, loopback, link-local and unspecified.
Multicast and reserved stay explicit: both are still is_global on
CPython, and 64:ff9b::/96 reaches IPv4 through a NAT64 gateway.
parse_usernames() validates *_username fields with is_plausible_username
(the #1403 guard), but the trailing `if k in SUPPORTED_IDS` branch runs
unconditionally. Since "username" is itself in SUPPORTED_IDS, the bare
`username` key bypasses the guard: a URL/email/path value is dropped by
the plausibility check and then silently re-added, so it becomes a
recursive search target again and reproduces the #1403 false-error
cascade.
Make the branch an elif. The bare `username` key is already handled with
validation by the first branch; every other SUPPORTED_ID (gaia_id,
vk_id, orcid, ...) still gets added as before since none of those keys
contain "username".
Add a regression test for the bare `username` key with URL/email/path
values; the existing parse_usernames tests only covered *_username
variants, which aren't in SUPPORTED_IDS.
* refactor: drop dead 'if not dictionary' guards across report paths
Closes#2665.
Every site results entry flows through make_site_result (checking.py:788),
which initializes results_site = {} and then unconditionally populates it
(site/username/keywords/parsing_enabled/url_main/cookies up front, checker
at the end, plus status or url_user+future on every branch). It has no
return path that yields an empty dict. check_site_for_username wraps that
result and likewise never returns a falsy entry, and maigret.py does not
construct site-result dicts by hand.
So the four downstream 'if not dictionary: continue' guards (all tagged
'# TODO: fix no site data issue') were dead code — the entries they
guarded against cannot occur. Removed:
- maigret/maigret.py:101 (extract_ids_from_results)
- maigret/report.py:151 — 'if not dictionary or dictionary.get("is_similar")'
trimmed to just the is_similar check (real logic)
- maigret/report.py:467 (extended-report builder)
- maigret/report.py:609 (generate_txt_report)
- maigret/report.py:627 — 'if not site_result or not site_result.get("status")'
trimmed to just the status check (real logic)
- maigret/report.py:687 (generate_json_report)
plus the four '# TODO: fix no site data issue' comments.
No behavior change for well-formed inputs; downstream .get() calls are
safe because entries are always populated dicts. 292 tests pass.
* feat: add Chinese name false-positive fix + CSDN site
- #2633: When searching non-ASCII usernames, skip presence detection
if the response body doesn't contain the username at all, preventing
false positives on sites that return generic error pages.
- #2634: Add CSDN (blog.csdn.net) — major Chinese dev blog with
40M+ users, alexa rank #45 in China.
* fix: add tls_fingerprint to CSDN + add tests for non-ASCII false positive fix
Addresses PR #2876 review feedback from soxoj:
- Add tls_fingerprint to CSDN protection (curl_cffi required)
- Add 2 test cases for #2633 non-ASCII username false positive fix:
- non-ASCII username not in response → no false CLAIMED
- non-ASCII username in response → normal CLAIMED behavior
---------
Co-authored-by: aznikline <aznikline@users.noreply.github.com>
* Fix site checks: 82 fixed, 11 disabled + DiscourseJson engine
* Fix test_hackernews_requires_profile_marker: update test body to JSON format matching Firebase API probe
In MaigretDatabase.ranked_sites_dict, the include (whitelist) tag filter
compared the site's raw tags against the lowercased query tags:
is_tags_ok = lambda x: set(x.tags).intersection(set(normalized_tags))
while the exclude (blacklist) filter and every sibling lambda
(name/source/engine) lowercase the site-side value:
is_excluded_by_tag = lambda x: set(map(str.lower, x.tags)).intersection(
set(normalized_excluded_tags)
)
So a site stored with an upper/mixed-case tag (e.g. a custom or submitted
site tagged 'US') was excluded by `--exclude-tags us` but NOT found by
`--tags us` — an asymmetry between the two filters. Lowercase the site
tags in the include filter too, matching the rest of the method.
Adds a regression test asserting tags=['us'] finds a site tagged 'US'.
Serialize the maigret graph (the same one --graph builds) into an idempotent Cypher script importable into Neo4j. Reuses MaigretGraph via an extracted _build_maigret_graph() helper, leaving save_graph_report behavior unchanged; no new runtime dependency. Adds the --neo4j flag, the neo4j_report setting, docs, and a unit test.
Closes#2630
URLMatcher._HTTP_URL_RE_STR was "^https?://(www.|m.)?(.+)$". The dots in
the optional (www.|m.)? subdomain-prefix group were unescaped, so each `.`
matched ANY character. For a host that simply starts with `m` or `www`
(not the literal `m.`/`www.` prefix), the group greedily ate the first
characters:
extract_main_part('https://medium.com/alice') -> 'dium.com/alice'
extract_main_part('https://myspace.com/bob') -> 'space.com/bob'
extract_main_part feeds make_profile_url_regexp / get_url_template, so the
generated site-detection regexp was corrupted too (medium.com -> dium.com),
and the loose prefix could match bogus hosts like `xmedium.com`. 170+ hosts
in resources/data.json start with `m`.
Fix: escape the dots with a raw string, r"^https?://(www\.|m\.)?(.+)$".
Genuine `m.`/`www.` prefixes are still stripped
(m.wikipedia.org -> wikipedia.org). The two existing tests that pinned the
unescaped pattern string are updated to the corrected form, and a
regression test covers hosts beginning with m/www.
