mirror of
https://github.com/yogeshojha/rengine.git
synced 2026-10-20 15:21:30 +02:00
Yogesh Ojha
6949492e11
- Store each run's row, first-seen and retired counts (scan_deltas, scan_retired), valid while the revision stamp matches scan_revisions; statement triggers on the seven result tables bump it. Finalize warms, a beat backfill fills the rest, a live run answers from its last count and queues a refresh. /scans 1.5 s -> 43 ms, dashboard overview 5-10 s -> 0.9 s. - lead_cache holds a live scope for 20 s with a stale copy, and every build is single-flight; search pages opt out. - Leads: lowercase subdomain names (CHECK) and LIKE for host:, a guard for empty JSON arrays, branches as MATERIALIZED CTEs read to the cap, every count in one statement. - Addresses and services: the derived CTE as one grouped pass, rows and count in one execution (page_rows), GIN lookups for page hosts, facets through grouping sets. - Correlation carriers counted by EXISTS; takeover latest-sighting in SQL, no silent cap. - Screenshot backfill partial index, nine unused low-cardinality indexes dropped, hostname helpers memoised, vhost names normalised.
21 lines
437 B
Python
21 lines
437 B
Python
MAX_TARGET_IMPORT = 1000
|
|
MAX_SCAN_BATCH = 500
|
|
|
|
MAX_COMMAND_OUTPUT = 200_000
|
|
|
|
SCANS_QUEUE = "scans"
|
|
SCAN_CONTROL_QUEUE = "scan_control"
|
|
SCAN_QUEUES = (SCANS_QUEUE, SCAN_CONTROL_QUEUE)
|
|
CRITICAL_QUEUE = "critical"
|
|
DEFAULT_QUEUE = "default"
|
|
|
|
MAX_RATE = 10000
|
|
MAX_THREADS = 1000
|
|
MAX_TIMEOUT = 3600
|
|
|
|
# bytes of a response body httpx keeps
|
|
HTTPX_RESPONSE_CAP = 131072
|
|
|
|
# entries each memoised pure helper keeps per process
|
|
PURE_CACHE = 1 << 16
|