A nested chord canvas put the whole remaining pipeline into every
stage message, duplicated per sibling per level: 33 MB per message,
39 of them filled a 1 GB Redis, and the worker was OOM-killed
deserializing them. Every later launch then failed with "command not
allowed when used memory > 'maxmemory'".
shared/services/orchestrator/canvas.py now builds one step at a time;
a step's chord body is app.tasks.scan.run_scan_step(steps, index+1),
which dispatches the next step once the current one settles. Messages
are ~1 KB now; tests/pipeline/test_canvas_shape.py caps them.
Also, from an end-to-end review of the pipeline:
- The reaper re-queues a PENDING scan with no launch anywhere in the
queue, so a lost or failed send_task no longer strands it forever.
api create() now dispatches off the event loop and records a failed
dispatch the way create_batch already did.
- Cancel no longer revokes queued stage tasks. A cancelled run used to
walk every remaining step recording one aborted row per stage, and
whether finalize ran depended on a race with the revoke broadcast.
The step body now finalizes a cancelled run directly, and the api
also dispatches finalize so a run with no live canvas still settles.
- The reaper only skips scans that actually have a message queued,
read off the scans queue, instead of skipping every scan whenever
the queue holds anything.
- Every beat tick carries expires, so a worker that comes back after
being down does not replay hours of stacked ticks.
- worker-default and worker-beat restart on their own.
- _run_process kills the tool in a finally; a soft time limit raised
inside proc.wait used to leave the tool running past the stage.
- on_worker_ready crashed on a missing Consumer attribute, so the
worker's IP-range and threat-feed warm-up never ran.
- Worker logging now goes through the root logger; it was configured
on a "rengine.worker" logger nothing writes to, so INFO logs (task
starts, step hand-overs) never appeared in the container output.
- _normalize_url no longer raises on an unparseable value, which had
been killing the whole bounty feed sync on one bad scope entry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0181JfVNatLVWnDbtGWRWPov
A full static review of the 3.0 tree, run as two fan-out passes with every
finding verified against the running stack rather than taken on inspection.
Correctness and security
- instance settings encrypted the mask over the stored AI key: `!= MASK` never
matched `••••••••abcd`, so saving settings destroyed the key and `test_ai`
posted the mask as a bearer token
- the MFA token was replayable for its full five-minute life; the gate now
checks revocation and consumes the token after a successful exchange
- logout revoked nothing for a bearer client while answering "Logged out"
- `POST /api-keys/{id}/test` returned the plaintext key inside httpx's error
string, which embeds the whole URL
- an interest rule could be edited or deleted through another project's id
- redis ran the celery broker under `allkeys-lru`, free to evict a queued task
or a chord counter; now `noeviction`
- nuclei's stored command carried the OAST token and every auth header
- a finalize step could poison the session and leave a scan RUNNING for ever
- `is:issue` raised TypeError on every search
- offset paging repeated and skipped rows in four dimensions
- a cancelled run kept a stale `scans.*_found` rollup
- every PDF report raised `KeyError: 'software'`
- `version_key` truncated a wide numeric run and inverted release ordering
- psycopg raises a NUL as a bare ValueError, above StatementError, so every
savepoint guard was blind to the case it was written for
Performance
- endpoints.host had no trigram index: 35.9ms seq scan -> 0.389ms
- free text compiled to one OR, not the prescribed UNION ALL: 132ms -> 41ms
- `/http-assets/summary` and `/ips/summary` materialised whole tables
- the cross-link `ip` kind ran an estate query per page and drew nothing
Frontend
- `$effect` loaders that read the state they write, an unbounded request loop
on a failing endpoint
- error states that rendered as empty states
- 2FA backup codes were unusable while the copy said they worked
- a "Scan history" dialog that always reported none
- ~400 duplicated lines across five filter bars, now one component
Build and tooling
- every scanner installed at @latest; all 13 pinned to the measured versions
- CI built `web/`, gone since 2.x, with tags that cannot be valid
- the prettier pre-commit hook discarded its filenames and always passed
- the test suite destroyed itself when run twice
Also: subdomains from public code, via a pinned github-subdomains source and a
GitHub API key that subfinder receives too; sublist3r and oneforall removed
after measuring one unique host in 89 across four targets.
Verified: 853 tests, ruff clean over eleven packages, svelte-check 0 errors,
the count-is-a-promise gate at 131 checks and 0 mismatches, and 143 GET routes
with no 5xx. Four migrations, each with an exercised downgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVkdx5ynZ6aof5kLFczmhX
The comment rule in CLAUDE.md is one line maximum, never narration: no
multi-line blocks, no module-header explainers, no sentences that explain or
justify the code to the reader. The tree had drifted a long way from it, and
most of the drift is mine.
Removed:
232 lines of multi-line comment blocks
203 single-line comments that argued instead of stating
154 comments joined by so/because/otherwise/rather-than
143 narrating comments in the frontend
112 test docstrings (the test name already says what it asserts)
25 test module docstrings and 25 decorative section dividers
Collapsed:
324 multi-line docstrings to one factual line
271 one-line docstrings that had a justifying clause bolted on
Kept: functional directives, section dividers, short field labels, alembic
revision headers, and the eight FastAPI route docstrings that render in /docs.
Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs.
284 python tests and 34 frontend tests pass, ruff and ruff-format clean across
ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
- Added `clean_name` function to `validation.py` for name validation with max length.
- Simplified docstrings in `dnsx` client, parser, and service files for clarity.
- Enhanced error handling in `ViewDNSService` to prevent disabling keys on rate limits.
- Improved WHOIS service to handle concurrent inserts gracefully with integrity checks.
- Updated `Settings` class in worker config to inherit from `BaseAppSettings`.
- Added new dependencies in `pyproject.toml` for `apprise` and `cryptography`.
- Update Dockerfile to copy shared directory and dependency files
- Create new README.md file
- Refactor database session management and security imports
- Implement user authentication routes and user management endpoints
- Add configuration settings for application and database
- Set up logging configuration for backend services
- Update docker-compose for improved service management
