- Store each run's row, first-seen and retired counts (scan_deltas, scan_retired), valid while
the revision stamp matches scan_revisions; statement triggers on the seven result tables bump
it. Finalize warms, a beat backfill fills the rest, a live run answers from its last count and
queues a refresh. /scans 1.5 s -> 43 ms, dashboard overview 5-10 s -> 0.9 s.
- lead_cache holds a live scope for 20 s with a stale copy, and every build is single-flight;
search pages opt out.
- Leads: lowercase subdomain names (CHECK) and LIKE for host:, a guard for empty JSON arrays,
branches as MATERIALIZED CTEs read to the cap, every count in one statement.
- Addresses and services: the derived CTE as one grouped pass, rows and count in one execution
(page_rows), GIN lookups for page hosts, facets through grouping sets.
- Correlation carriers counted by EXISTS; takeover latest-sighting in SQL, no silent cap.
- Screenshot backfill partial index, nine unused low-cardinality indexes dropped, hostname
helpers memoised, vhost names normalised.
What's new lists only what changed: rescans that found new critical or
high findings, new programs, assets added to or leaving scope, bounty
table and rule changes, certificate hosts from watches, and targets a
scope or watch added. Events group by day on a rail, with a divider at
the Caught up mark. Each rescan names its worst new checks, and every
count opens the scan filtered to exactly those findings.
Program syncs now record payout, safe harbor and 2FA changes, and per
asset tier, bounty eligibility, severity cap and instruction changes.
The public feed sync diffs known programs for these too.
The search cache no longer stores a page that carries an error, so a
timed-out search no longer answers 0 rows for ten minutes.
Four implementations answered "what does this host share?" and two of them
could only answer it for one run: the project-wide page had no Correlation
tab at all, and its detail sheet passed the row's own scan_id, so "Related"
silently meant "related inside this one target's last run".
One engine now, shared/services/correlation/, parameterised by QueryScope.
kinds.py holds the vocabulary as data; hubs.py discovers hubs with a GROUP BY
over the whole scope instead of grouping a 3,000-row sample in Python, so a
hub is no longer missed for falling outside the cut. asset_query/groups.py
derives its dimensions from the same registry, and correlation-graph, related
and correlation take WebAssetScope. New /surface/correlation page.
The count is a promise, and three bugs broke it. Asset kinds joined
http_assets and counted rows, so a host answering on 80 and 443 counted twice
(cert.fingerprint said 80 where the search returned 41). asn excludes
CDN-fronted rows but its token did not, 8779 against 8815; KindSpec.narrows
appends -is:cdn. And carriers came from a window over the group set the
caller asked about, so one row's own value read as 100% of its carriers and
common swallowed it. All 28 hub counts across 14 kinds now equal their
searches.
The provider rules moved into the engine. generic_page, shared_edge and
cert_covers gated cross links alone, so estate-wide the graph drew 404 Not
Found across 5 targets, 403 Forbidden across 4 and a 548-byte 404 body across
5 as correlations. A hub is judged once its members load and is labelled with
who wrote it; hubs load MAX_HUBS_PER_KIND + 40 deep and re-rank.
Related takes the whole vocabulary, each kind carrying the phrase it is
announced with and the query its button opens, which retires relationDsl and
RELATION_LABELS. asn became a kind, out of the default set. A scope over
several targets opens on what crosses them: 3,000 nodes over 21,690
correlating hosts is a hairball, the 30 crossing hubs are the finding.
Measured on 63,844 host rows across 10 covering scans: ~1.1s cold for the
estate, 70-200ms for a scan, cached after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FwCCFgRBC8ut6SQpmzfGAa
Scan page froze on a duplicate each key; every tab body now sits in a
svelte:boundary. Composite discovered_at indexes on the five result
tables, leads counted in one FILTER pass, dashboard and scan-service
first-seen counts bounded to the scans in the window, ANALYZE at
finalize, shm_size on the db service, SQL_ECHO instead of DEBUG, an
entrypoint that serves with workers outside reload.
Settled-scope aggregates and search pages are cached by target
revision; triage, dismissal, interest evaluation, notes, certificate
rechecks and re-ranking bump it. Requests per page cut: single-flight
GETs, one schemas endpoint, batched whois correlations, a counts
endpoint for the dashboard, deduped store loads, trimmed coverage
payloads.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NWRq8utu7KXsVBVU3gYNi7
The comment rule in CLAUDE.md is one line maximum, never narration: no
multi-line blocks, no module-header explainers, no sentences that explain or
justify the code to the reader. The tree had drifted a long way from it, and
most of the drift is mine.
Removed:
232 lines of multi-line comment blocks
203 single-line comments that argued instead of stating
154 comments joined by so/because/otherwise/rather-than
143 narrating comments in the frontend
112 test docstrings (the test name already says what it asserts)
25 test module docstrings and 25 decorative section dividers
Collapsed:
324 multi-line docstrings to one factual line
271 one-line docstrings that had a justifying clause bolted on
Kept: functional directives, section dividers, short field labels, alembic
revision headers, and the eight FastAPI route docstrings that render in /docs.
Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs.
284 python tests and 34 frontend tests pass, ruff and ruff-format clean across
ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
Every result table asks for its lead counts on mount, on every facet change and on
every tab switch, and the answer is the same each time. After the earlier fixes the
host set costs 832 ms, endpoints 1,609 ms and the project-wide host set 1,653 ms.
Served from redis it is 4-6 ms, on all five dimensions.
Only a settled scope is cached. A live scan's counts change under the reader and a
count has to equal the rows its link opens, so a scope holding any scan that has not
reached a terminal status is always computed fresh.
The entry lives 60 seconds, and the reason is worth stating rather than assuming a
finished scan is immutable. Its rows are — but the counts drawn from them are not: a
reviewer suppressing a finding moves every vulnerability lead, the nightly EPSS and
KEV load moves is:kev, and an interest pass moves is:exposed. Plumbing an
invalidation into each of those and hoping none is ever missed trades a silent wrong
number for a slow right one; a short life serves the burst that this fix is about —
tab switches, facet changes, a second table on the same page — and bounds any count
at a minute stale.
The key is the dimension, the sorted scan ids and the facets, and facets_of drops
what cannot change a count: paging, sorting, and the search box, which the lead set
deliberately ignores. A lead set that did not compute is never stored, so a timeout
is not served for the next minute.
The statement is chunked at 16 branches. A fifty-branch union plans as one enormous
tree, and one that outruns the timeout takes every count in it down.
Verified against the live database: 33 host, 14 address and 18 service leads all
still equal the row count their query opens, served from cache; and two different
facet sets give two different answers rather than sharing an entry. Seven tests
cover the rules — computed once for a finished scan, never cached for a running one,
cached for a failed or cancelled one, distinct per filter, never cached when it did
not compute, and facets_of ignoring paging while keeping facets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
Yogesh Ojha