In MaigretDatabase.ranked_sites_dict, the include (whitelist) tag filter
compared the site's raw tags against the lowercased query tags:
is_tags_ok = lambda x: set(x.tags).intersection(set(normalized_tags))
while the exclude (blacklist) filter and every sibling lambda
(name/source/engine) lowercase the site-side value:
is_excluded_by_tag = lambda x: set(map(str.lower, x.tags)).intersection(
set(normalized_excluded_tags)
)
So a site stored with an upper/mixed-case tag (e.g. a custom or submitted
site tagged 'US') was excluded by `--exclude-tags us` but NOT found by
`--tags us` — an asymmetry between the two filters. Lowercase the site
tags in the include filter too, matching the rest of the method.
Adds a regression test asserting tags=['us'] finds a site tagged 'US'.
URLMatcher._HTTP_URL_RE_STR was "^https?://(www.|m.)?(.+)$". The dots in
the optional (www.|m.)? subdomain-prefix group were unescaped, so each `.`
matched ANY character. For a host that simply starts with `m` or `www`
(not the literal `m.`/`www.` prefix), the group greedily ate the first
characters:
extract_main_part('https://medium.com/alice') -> 'dium.com/alice'
extract_main_part('https://myspace.com/bob') -> 'space.com/bob'
extract_main_part feeds make_profile_url_regexp / get_url_template, so the
generated site-detection regexp was corrupted too (medium.com -> dium.com),
and the loose prefix could match bogus hosts like `xmedium.com`. 170+ hosts
in resources/data.json start with `m`.
Fix: escape the dots with a raw string, r"^https?://(www\.|m\.)?(.+)$".
Genuine `m.`/`www.` prefixes are still stripped
(m.wikipedia.org -> wikipedia.org). The two existing tests that pinned the
unescaped pattern string are updated to the corrected form, and a
regression test covers hosts beginning with m/www.
update_site() used `s = site` inside a for-loop, which rebinds only the
local loop variable and leaves self._sites[i] unchanged. The method
returned self silently appearing to succeed while the database list
retained the original entry. This broke --auto-disable for any site
already present in the database.
Fix with enumerate so the actual list element is replaced.
Fixes#2750
* Fix ID extraction crash when regex groups are optional
Handle None capture groups in username/id extraction and add regression coverage for optional trailing groups.
* Remove leftover line that overwrote safe _id in extract_id_from_url
