mirror of
https://github.com/yogeshojha/rengine.git
synced 2026-10-20 15:21:30 +02:00
Yogesh OjhaandClaude Opus 5
e821bd330d
A gov.np HTTP probe ran 66 minutes over 25,916 host and port pairs, then lost every row: a captured response body carried a NUL byte, postgres refuses NUL in text and in json alike, and the failing flush rolled back the whole stage transaction. Recording that failure then failed too — psycopg puts the offending parameters in its error message, so the NUL travelled into `scan_activities.error` and took the failure path down with it, leaving the activity RUNNING and the scan unable to finalize. `shared/utils/text.py` is now the one place that names what postgres will not store. `strip_nul` runs through every string, list and dict a tool record yields, applied in `tools/runner/fieldmap.py` so every tool that maps a record is covered rather than httpx alone. `strip_control` replaces the two identical local regexes in the nuclei parser and the template indexer, and now also guards the activity log, the activity timeline and the command recorder — the three places that write a tool's own words. The HTTP probe's batch flush runs in a savepoint: a row postgres refuses costs that row, retried one at a time, and the count comes back as a PARTIAL stage naming how many responses could not be stored. Recording a stage failure is wrapped too, so a second error while writing the first can never leave a scan running forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
33 lines
998 B
Python
33 lines
998 B
Python
from __future__ import annotations
|
|
|
|
from collections.abc import Callable
|
|
from dataclasses import dataclass
|
|
from typing import Any
|
|
|
|
from shared.utils.text import scrub
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class F:
|
|
src: str | Callable[[dict], Any]
|
|
cast: Callable[[Any], Any] | None = None
|
|
max_len: int | None = None
|
|
|
|
def extract(self, record: dict) -> Any:
|
|
if callable(self.src):
|
|
value = self.src(record)
|
|
else:
|
|
value: Any = record
|
|
for part in self.src.split("."):
|
|
value = value.get(part) if isinstance(value, dict) else None
|
|
if value is not None and self.cast is not None:
|
|
value = self.cast(value)
|
|
value = scrub(value)
|
|
if self.max_len is not None and isinstance(value, str):
|
|
value = value[: self.max_len]
|
|
return value
|
|
|
|
|
|
def parse_record(record: dict, fieldmap: dict[str, F]) -> dict[str, Any]:
|
|
return {field: spec.extract(record) for field, spec in fieldmap.items()}
|