Yogesh Ojha 6949492e11 perf(read-path): stored scan deltas, live-scope cache and cheaper aggregates at millions of rows ...

- Store each run's row, first-seen and retired counts (scan_deltas, scan_retired), valid while
  the revision stamp matches scan_revisions; statement triggers on the seven result tables bump
  it. Finalize warms, a beat backfill fills the rest, a live run answers from its last count and
  queues a refresh. /scans 1.5 s -> 43 ms, dashboard overview 5-10 s -> 0.9 s.
- lead_cache holds a live scope for 20 s with a stale copy, and every build is single-flight;
  search pages opt out.
- Leads: lowercase subdomain names (CHECK) and LIKE for host:, a guard for empty JSON arrays,
  branches as MATERIALIZED CTEs read to the cap, every count in one statement.
- Addresses and services: the derived CTE as one grouped pass, rows and count in one execution
  (page_rows), GIN lookups for page hosts, facets through grouping sets.
- Correlation carriers counted by EXISTS; takeover latest-sighting in SQL, no silent cap.
- Screenshot backfill partial index, nine unused low-cardinality indexes dropped, hostname
  helpers memoised, vhost names normalised.

2026-09-25 05:40:01 +05:30

..

ct_stream

chore: production-readiness review across the tree

2026-09-13 18:03:29 +05:30

orchestrator

perf(read-path): stored scan deltas, live-scope cache and cheaper aggregates at millions of rows

2026-09-25 05:40:01 +05:30

tasks

perf(read-path): stored scan deltas, live-scope cache and cheaper aggregates at millions of rows

2026-09-25 05:40:01 +05:30

celery.py

perf(read-path): stored scan deltas, live-scope cache and cheaper aggregates at millions of rows

2026-09-25 05:40:01 +05:30

config.py

style: strip explanatory comments across the codebase

2026-09-11 14:11:23 +05:30

database.py

perf: make the read path scale with the estate

2026-09-12 08:49:41 +05:30