mirror of
https://github.com/yogeshojha/rengine.git
synced 2026-10-20 07:11:28 +02:00
Yogesh OjhaandClaude Sonnet 5
8fac962556
A nested chord canvas put the whole remaining pipeline into every stage message, duplicated per sibling per level: 33 MB per message, 39 of them filled a 1 GB Redis, and the worker was OOM-killed deserializing them. Every later launch then failed with "command not allowed when used memory > 'maxmemory'". shared/services/orchestrator/canvas.py now builds one step at a time; a step's chord body is app.tasks.scan.run_scan_step(steps, index+1), which dispatches the next step once the current one settles. Messages are ~1 KB now; tests/pipeline/test_canvas_shape.py caps them. Also, from an end-to-end review of the pipeline: - The reaper re-queues a PENDING scan with no launch anywhere in the queue, so a lost or failed send_task no longer strands it forever. api create() now dispatches off the event loop and records a failed dispatch the way create_batch already did. - Cancel no longer revokes queued stage tasks. A cancelled run used to walk every remaining step recording one aborted row per stage, and whether finalize ran depended on a race with the revoke broadcast. The step body now finalizes a cancelled run directly, and the api also dispatches finalize so a run with no live canvas still settles. - The reaper only skips scans that actually have a message queued, read off the scans queue, instead of skipping every scan whenever the queue holds anything. - Every beat tick carries expires, so a worker that comes back after being down does not replay hours of stacked ticks. - worker-default and worker-beat restart on their own. - _run_process kills the tool in a finally; a soft time limit raised inside proc.wait used to leave the tool running past the stage. - on_worker_ready crashed on a missing Consumer attribute, so the worker's IP-range and threat-feed warm-up never ran. - Worker logging now goes through the root logger; it was configured on a "rengine.worker" logger nothing writes to, so INFO logs (task starts, step hand-overs) never appeared in the container output. - _normalize_url no longer raises on an unparseable value, which had been killing the whole bounty feed sync on one bad scope entry. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0181JfVNatLVWnDbtGWRWPov