- Count probing sites in the statistics instead of listing them
- Add data quality, country and protection statistics
- Fix the presenceStrs typo that disabled 13 site checks
- Put the statistics first and format them as Markdown
- Add a title and a table of contents to sites.md
- Move the site count into the intro line
- Declare the mirrors field, it is read at runtime
An engine is a template, but `update_from_engine` overwrites a site's scalar
fields with the engine's, and `strip_engine_data` then removes them on save. A
value written into an entry by hand is therefore ignored at runtime **and
deleted from `data.json`** the next time anything calls `save_to_file` — which
`utils/update_site_data.py` does on every regeneration. Nothing is printed: the
dict case already warns about a collision, the scalar case did not.
Found while giving MediaWiki entries a JSON `urlProbe`. The API path is not the
page path — a wiki with articles under `/wiki` keeps `api.php` under `/w` — so
a per-entry probe is the natural way to say it, and it turned out to be
impossible.
Now the entry wins and the override survives the round trip, with the same
warning the dict case prints.
Only fields the entry **stated** count. The constructor synthesises
`alexa_rank` for every site that does not carry one, so keying the rule off the
instance dict made seventeen op.gg sites keep that placeholder instead of the
rank their engine supplies; there is a test for exactly that, next to one for
the override winning and one for it surviving `strip_engine_data`.
The shipped database is unchanged — it round-trips byte for byte through load
and save, and no entry currently sets a field its engine also defines.
In MaigretDatabase.ranked_sites_dict, the include (whitelist) tag filter
compared the site's raw tags against the lowercased query tags:
is_tags_ok = lambda x: set(x.tags).intersection(set(normalized_tags))
while the exclude (blacklist) filter and every sibling lambda
(name/source/engine) lowercase the site-side value:
is_excluded_by_tag = lambda x: set(map(str.lower, x.tags)).intersection(
set(normalized_excluded_tags)
)
So a site stored with an upper/mixed-case tag (e.g. a custom or submitted
site tagged 'US') was excluded by `--exclude-tags us` but NOT found by
`--tags us` — an asymmetry between the two filters. Lowercase the site
tags in the include filter too, matching the rest of the method.
Adds a regression test asserting tags=['us'] finds a site tagged 'US'.
URLMatcher._HTTP_URL_RE_STR was "^https?://(www.|m.)?(.+)$". The dots in
the optional (www.|m.)? subdomain-prefix group were unescaped, so each `.`
matched ANY character. For a host that simply starts with `m` or `www`
(not the literal `m.`/`www.` prefix), the group greedily ate the first
characters:
extract_main_part('https://medium.com/alice') -> 'dium.com/alice'
extract_main_part('https://myspace.com/bob') -> 'space.com/bob'
extract_main_part feeds make_profile_url_regexp / get_url_template, so the
generated site-detection regexp was corrupted too (medium.com -> dium.com),
and the loose prefix could match bogus hosts like `xmedium.com`. 170+ hosts
in resources/data.json start with `m`.
Fix: escape the dots with a raw string, r"^https?://(www\.|m\.)?(.+)$".
Genuine `m.`/`www.` prefixes are still stripped
(m.wikipedia.org -> wikipedia.org). The two existing tests that pinned the
unescaped pattern string are updated to the corrected form, and a
regression test covers hosts beginning with m/www.
update_site() used `s = site` inside a for-loop, which rebinds only the
local loop variable and leaves self._sites[i] unchanged. The method
returned self silently appearing to succeed while the database list
retained the original entry. This broke --auto-disable for any site
already present in the database.
Fix with enumerate so the actual list element is replaced.
Fixes#2750
* Fix ID extraction crash when regex groups are optional
Handle None capture groups in username/id extraction and add regression coverage for optional trailing groups.
* Remove leftover line that overwrote safe _id in extract_id_from_url
