mirror of
https://github.com/yogeshojha/rengine.git
synced 2026-10-20 15:21:30 +02:00
Yogesh OjhaandClaude Opus 5
3d310c173a
A running scan showed nothing. Caught live on an http_probe: pg_stat_user_tables reported 821 rows physically inserted into http_assets while count(*) returned 0, with one worker connection idle in transaction for six minutes. The stage streamed httpx row by row but only flushed inside savepoints and committed at the very end, so a 66-minute probe was one open transaction — invisible to every reader and pinning a connection against vacuum. Other long stages never streamed at all: subdomain_discovery wrote nothing until every provider and the resolver had finished, port_scan ran naabu unstreamed, url_discovery accumulated katana's whole crawl, screenshot collected every image first. And nothing told the browser: no rows-written event existed and no results table refreshed while a scan ran. stages/sink.py:ResultSink is now the one way to write incrementally — buffer, flush on size or age, commit inside the flush, announce on its own cadence. Stage.results_sink wires that to publish_results, which refreshes only the affected scans counters and emits RESULTS_FOUND. Nine stages convert. The write callback must commit; a flush that only flushes reproduces the original bug exactly, and clearing the previous attempt belongs inside the first flush or the table is empty for the length of the run. nuclei needed one more thing: findings are sparse, so _paced hands the caller back its own thread every two seconds, otherwise a three-finding burst sits unwritten for the rest of a two-hour run. The second half is when a derived fact is computed at all. ip_enrichment is 10.1 s of pure SQL — 0.09% of a 3 h 08 m gov.np run — yet sat at the last derived level, so 1,694 addresses carried a null ASN, operator and country for 2 h 27 m, and a run cancelled before that level lost the enrichment for good (scan d6532d88: 0 of 137 addresses, permanently). The lookup is now one definition, ip_asn.enrich_addresses, called from ip_inventory.materialize over the addresses it just wrote. The stage keeps what only a finished run can know: the union sweep, CDN adoption, the target-context fallback, the back-fills and the coverage number. Interest is judged twice for the same reason. A saved query matches a host on its own facts and is right at any point, so evaluate_live runs those on a debounce while the scan works; rarity needs a finished estate, so correlation and the model still wait for finalize. A correlation over a tenth of the estate is not a weaker judgement, it is a wrong one. Measured after: addresses enriched at level 2 with ip_enrichment not yet reached, that stage down from 10.1 s to 0.1 s, http_assets ASN filled at write time for free, 500 hosts flagged mid-discovery, and the browser's Web assets badge climbing 2,210 -> 13,530 with no reload. Eleven issues from an end-to-end review are folded in, four of them regressions from the above: the rules provider emits KEYWORD but declares RULE, so deriving the rewrite set from the class attribute dropped every keyword signal on the full pass; a live run reported itself stale, which made every read of the tab queue the unguarded correlation pass; SET LOCAL lock_timeout died with ensure_builtin's commit; and enrichment inside the insert transaction took scan-wide unordered locks that two same-level stages could deadlock on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0165iTiibhYePy1EDixwC2Vs