mirror of
https://github.com/yogeshojha/rengine.git
synced 2026-10-20 07:11:28 +02:00
Yogesh OjhaandClaude Opus 5
e821bd330d
A gov.np HTTP probe ran 66 minutes over 25,916 host and port pairs, then lost every row: a captured response body carried a NUL byte, postgres refuses NUL in text and in json alike, and the failing flush rolled back the whole stage transaction. Recording that failure then failed too — psycopg puts the offending parameters in its error message, so the NUL travelled into `scan_activities.error` and took the failure path down with it, leaving the activity RUNNING and the scan unable to finalize. `shared/utils/text.py` is now the one place that names what postgres will not store. `strip_nul` runs through every string, list and dict a tool record yields, applied in `tools/runner/fieldmap.py` so every tool that maps a record is covered rather than httpx alone. `strip_control` replaces the two identical local regexes in the nuclei parser and the template indexer, and now also guards the activity log, the activity timeline and the command recorder — the three places that write a tool's own words. The HTTP probe's batch flush runs in a savepoint: a row postgres refuses costs that row, retried one at a time, and the count comes back as a PARTIAL stage naming how many responses could not be stored. Recording a stage failure is wrapped too, so a second error while writing the first can never leave a scan running forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
31 lines
935 B
Python
31 lines
935 B
Python
"""Text stored in postgres: NUL is rejected outright, in text and in json alike."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
from typing import Any
|
|
|
|
_NUL = "\x00"
|
|
_CTRL = re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]")
|
|
|
|
|
|
def strip_nul(value: str) -> str:
|
|
"""Drop only what postgres refuses to store; a captured response body is not text."""
|
|
return value.replace(_NUL, "") if _NUL in value else value
|
|
|
|
|
|
def strip_control(value: str) -> str:
|
|
"""strip_nul plus the other control characters, for text a person reads back."""
|
|
return _CTRL.sub("", value)
|
|
|
|
|
|
def scrub(value: Any) -> Any:
|
|
"""strip_nul through strings, lists and dicts on their way to a column."""
|
|
if isinstance(value, str):
|
|
return strip_nul(value)
|
|
if isinstance(value, list):
|
|
return [scrub(item) for item in value]
|
|
if isinstance(value, dict):
|
|
return {scrub(k): scrub(v) for k, v in value.items()}
|
|
return value
|