A nested chord canvas put the whole remaining pipeline into every
stage message, duplicated per sibling per level: 33 MB per message,
39 of them filled a 1 GB Redis, and the worker was OOM-killed
deserializing them. Every later launch then failed with "command not
allowed when used memory > 'maxmemory'".
shared/services/orchestrator/canvas.py now builds one step at a time;
a step's chord body is app.tasks.scan.run_scan_step(steps, index+1),
which dispatches the next step once the current one settles. Messages
are ~1 KB now; tests/pipeline/test_canvas_shape.py caps them.
Also, from an end-to-end review of the pipeline:
- The reaper re-queues a PENDING scan with no launch anywhere in the
queue, so a lost or failed send_task no longer strands it forever.
api create() now dispatches off the event loop and records a failed
dispatch the way create_batch already did.
- Cancel no longer revokes queued stage tasks. A cancelled run used to
walk every remaining step recording one aborted row per stage, and
whether finalize ran depended on a race with the revoke broadcast.
The step body now finalizes a cancelled run directly, and the api
also dispatches finalize so a run with no live canvas still settles.
- The reaper only skips scans that actually have a message queued,
read off the scans queue, instead of skipping every scan whenever
the queue holds anything.
- Every beat tick carries expires, so a worker that comes back after
being down does not replay hours of stacked ticks.
- worker-default and worker-beat restart on their own.
- _run_process kills the tool in a finally; a soft time limit raised
inside proc.wait used to leave the tool running past the stage.
- on_worker_ready crashed on a missing Consumer attribute, so the
worker's IP-range and threat-feed warm-up never ran.
- Worker logging now goes through the root logger; it was configured
on a "rengine.worker" logger nothing writes to, so INFO logs (task
starts, step hand-overs) never appeared in the container output.
- _normalize_url no longer raises on an unparseable value, which had
been killing the whole bounty feed sync on one bad scope entry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0181JfVNatLVWnDbtGWRWPov
A full static review of the 3.0 tree, run as two fan-out passes with every
finding verified against the running stack rather than taken on inspection.
Correctness and security
- instance settings encrypted the mask over the stored AI key: `!= MASK` never
matched `••••••••abcd`, so saving settings destroyed the key and `test_ai`
posted the mask as a bearer token
- the MFA token was replayable for its full five-minute life; the gate now
checks revocation and consumes the token after a successful exchange
- logout revoked nothing for a bearer client while answering "Logged out"
- `POST /api-keys/{id}/test` returned the plaintext key inside httpx's error
string, which embeds the whole URL
- an interest rule could be edited or deleted through another project's id
- redis ran the celery broker under `allkeys-lru`, free to evict a queued task
or a chord counter; now `noeviction`
- nuclei's stored command carried the OAST token and every auth header
- a finalize step could poison the session and leave a scan RUNNING for ever
- `is:issue` raised TypeError on every search
- offset paging repeated and skipped rows in four dimensions
- a cancelled run kept a stale `scans.*_found` rollup
- every PDF report raised `KeyError: 'software'`
- `version_key` truncated a wide numeric run and inverted release ordering
- psycopg raises a NUL as a bare ValueError, above StatementError, so every
savepoint guard was blind to the case it was written for
Performance
- endpoints.host had no trigram index: 35.9ms seq scan -> 0.389ms
- free text compiled to one OR, not the prescribed UNION ALL: 132ms -> 41ms
- `/http-assets/summary` and `/ips/summary` materialised whole tables
- the cross-link `ip` kind ran an estate query per page and drew nothing
Frontend
- `$effect` loaders that read the state they write, an unbounded request loop
on a failing endpoint
- error states that rendered as empty states
- 2FA backup codes were unusable while the copy said they worked
- a "Scan history" dialog that always reported none
- ~400 duplicated lines across five filter bars, now one component
Build and tooling
- every scanner installed at @latest; all 13 pinned to the measured versions
- CI built `web/`, gone since 2.x, with tags that cannot be valid
- the prettier pre-commit hook discarded its filenames and always passed
- the test suite destroyed itself when run twice
Also: subdomains from public code, via a pinned github-subdomains source and a
GitHub API key that subfinder receives too; sublist3r and oneforall removed
after measuring one unique host in 89 across four targets.
Verified: 853 tests, ruff clean over eleven packages, svelte-check 0 errors,
the count-is-a-promise gate at 131 checks and 0 mismatches, and 143 GET routes
with no 5xx. Four migrations, each with an exercised downgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVkdx5ynZ6aof5kLFczmhX
A target carries a list of host names, addresses and URLs an operator already
holds. Every scan of that target starts from them, so a list earned elsewhere
no longer has to be re-pasted or re-discovered.
Each pasted line is classified by its own shape and scope-checked against the
target; a refusal names the line and the reason. Seeds enter a run through the
asset_seed stage, which resolves the names it has no answer for and materialises
what they resolve to. Stages the engine has switched off still do not run.
Seeds no longer force a focused run: scope reads the new seed_only, which only
rescan, watch and connector runs set, so a seeded census scan still covers its
target. Seed count is a material run facet, so Compare states it before any
delta it would otherwise explain.
Two stages share the subdomains table for a run: discovery's clear no longer
drops rows asset_seed wrote, and the run's seeded hosts join its merge so the
imported source survives a name discovery also finds.
Closes#1538
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LYZfaNjq3Ym8GPvWYuHisS
Three holes in target validation, found by adding them all as targets.
A host name was stored as typed, so example.com, EXAMPLE.com and
Example.Com were three targets in one project and the unique constraint
never saw a collision. Host names now case-fold; a URL folds its scheme
and host and keeps its path, since a path is case-sensitive. The
migration folds existing rows and leaves a row whose folded value is
already taken for the operator to merge.
validate_asn was ^AS\d+$ with no bound, so AS0 and
AS99999999999999999999 were accepted and then failed enrichment with an
RDAP error nobody can act on. It now takes 1 to 4294967295.
0.0.0.0/0 and ::/0 were accepted, and expand_network samples a netblock
target evenly, so the scan would have swept the whole internet from this
instance's address. Loopback, link-local, multicast, reserved space and
the default route are refused; RFC1918 stays, corporate estates are the
reason it exists. Each refusal says which bound it crossed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XxW9yfg1SEchtg6HLGHRkK
Repo-wide sweep of every user-facing string: frontend components, routes,
config and stores; API details and route docs; shared definitions and
models; stage config help; report prose; MCP, toolbox, interest and
connector text; notification and worker messages.
Descriptions that explained, reassured, justified or restated their title
are deleted. Confirm dialogs name what is removed and stop. Failed actions
read "<Noun> not <verbed>", successes "<Noun> <verbed>". No reader address,
no product-as-subject, no em-dashes or parentheticals in prose.
Mirrors updated on both sides: EPSS band help, compare refusal reasons,
interest labels. Four tests asserting on old wording follow the new text.
The rules are condensed into CLAUDE.md under Writing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SK5o8xR96cbSnAfjYSTPSe
The comment rule in CLAUDE.md is one line maximum, never narration: no
multi-line blocks, no module-header explainers, no sentences that explain or
justify the code to the reader. The tree had drifted a long way from it, and
most of the drift is mine.
Removed:
232 lines of multi-line comment blocks
203 single-line comments that argued instead of stating
154 comments joined by so/because/otherwise/rather-than
143 narrating comments in the frontend
112 test docstrings (the test name already says what it asserts)
25 test module docstrings and 25 decorative section dividers
Collapsed:
324 multi-line docstrings to one factual line
271 one-line docstrings that had a justifying clause bolted on
Kept: functional directives, section dividers, short field labels, alembic
revision headers, and the eight FastAPI route docstrings that render in /docs.
Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs.
284 python tests and 34 frontend tests pass, ruff and ruff-format clean across
ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
Naming an engine or a scan context with more than 200 characters reached
Postgres and came back as a 500 — the column is bounded, the request
model was not. A name of spaces was accepted outright by engines,
contexts, schedules and projects, and a project created that way got an
empty slug, which is the segment its own URL is built from.
Proxies and notification channels already ran their names through
clean_name; every other named entity now does the same, on update as
well as on create, where none of them did. A rejected name says whether
it was empty or too long instead of failing at the database.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6LcH8uUddjJbKrahf34ys
Adding a target failed on the forms people actually paste. A value with
surrounding whitespace, a trailing slash, or a trailing root dot was
rejected outright, and "*.example.com" — how every bug bounty programme
writes a domain and its subdomains — was rejected too, though it names
exactly what a domain target already scans.
In the other direction any scheme validators.url accepts was taken as a
URL target, so ftp://x.com was stored, and the host was later read off
it as "ftp" and looked up in the registry under that name. A URL target
is now http or https, since that is what reNgine can scan.
A target value is normalised once, and that is what gets stored, so the
value on the page is the value that was validated.
The rejection said "Invalid target ID" for a value that is not an ID. It
now names the value and the accepted forms, in the same words the launch
dialog already uses. The two places that really were about a malformed
identifier keep saying so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6LcH8uUddjJbKrahf34ys
- Added `clean_name` function to `validation.py` for name validation with max length.
- Simplified docstrings in `dnsx` client, parser, and service files for clarity.
- Enhanced error handling in `ViewDNSService` to prevent disabling keys on rate limits.
- Improved WHOIS service to handle concurrent inserts gracefully with integrity checks.
- Updated `Settings` class in worker config to inherit from `BaseAppSettings`.
- Added new dependencies in `pyproject.toml` for `apprise` and `cryptography`.
- Added models for WHOIS responses including WhoisDomainResponse, WhoisIPResponse, and WhoisASNResponse.
- Created a parser for WHOIS data to convert raw responses into structured models.
- Developed an RDAPProvider class to handle RDAP lookups using the whoisit library.
- Implemented a WhoisService class for performing WHOIS lookups with optional database caching.
- Added methods for finding correlations between WHOIS records based on registrant, registrar, network, country, and nameservers.
- Updated Dockerfile to include tools directory and added rengine-tools dependency in pyproject.toml.
