mirror of
https://github.com/yogeshojha/rengine.git
synced 2026-10-20 07:11:28 +02:00
Yogesh OjhaandClaude Opus 5
e821bd330d
A gov.np HTTP probe ran 66 minutes over 25,916 host and port pairs, then lost every row: a captured response body carried a NUL byte, postgres refuses NUL in text and in json alike, and the failing flush rolled back the whole stage transaction. Recording that failure then failed too — psycopg puts the offending parameters in its error message, so the NUL travelled into `scan_activities.error` and took the failure path down with it, leaving the activity RUNNING and the scan unable to finalize. `shared/utils/text.py` is now the one place that names what postgres will not store. `strip_nul` runs through every string, list and dict a tool record yields, applied in `tools/runner/fieldmap.py` so every tool that maps a record is covered rather than httpx alone. `strip_control` replaces the two identical local regexes in the nuclei parser and the template indexer, and now also guards the activity log, the activity timeline and the command recorder — the three places that write a tool's own words. The HTTP probe's batch flush runs in a savepoint: a row postgres refuses costs that row, retried one at a time, and the count comes back as a PARTIAL stage naming how many responses could not be stored. Recording a stage failure is wrapped too, so a second error while writing the first can never leave a scan running forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP