Stage tasks hold a slot for their whole tool run, so step hand-overs and
finalize queued behind them and background jobs took scan slots.
- worker-scans consumes scans (run_scan_stage only), scalable with --scale
- worker-control consumes scan_control: launch, hand-over, resume, finalize
- worker-default consumes default and critical
- reaper reads both scan queues; readiness requires a consumer on every queue
- connection budget counts every worker; max_connections 250
The worker writes scan media as root, so `tar -cf scan_media.tar` run
as the host user failed on every scan directory. The failure went to
/dev/null and the `||` wrote an empty tar instead, and the dump
reported success: every archive taken so far holds no screenshots and
no captured responses. Measured on this instance — 10 KB before, 566
MB and 27,987 entries after. The tar and the restore both run in a
container now, which is also the only way the restore could remove the
old tree.
The volume names were guessed as <directory>_<volume>, which is wrong
whenever the compose project name is not the directory name, and the
guess was wrapped in `|| true`, so that also produced an empty archive
quietly. The project name comes from compose, and a missing volume is
an error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DafWxfoWZXVDgr9KroJBDZ
A full static review of the 3.0 tree, run as two fan-out passes with every
finding verified against the running stack rather than taken on inspection.
Correctness and security
- instance settings encrypted the mask over the stored AI key: `!= MASK` never
matched `••••••••abcd`, so saving settings destroyed the key and `test_ai`
posted the mask as a bearer token
- the MFA token was replayable for its full five-minute life; the gate now
checks revocation and consumes the token after a successful exchange
- logout revoked nothing for a bearer client while answering "Logged out"
- `POST /api-keys/{id}/test` returned the plaintext key inside httpx's error
string, which embeds the whole URL
- an interest rule could be edited or deleted through another project's id
- redis ran the celery broker under `allkeys-lru`, free to evict a queued task
or a chord counter; now `noeviction`
- nuclei's stored command carried the OAST token and every auth header
- a finalize step could poison the session and leave a scan RUNNING for ever
- `is:issue` raised TypeError on every search
- offset paging repeated and skipped rows in four dimensions
- a cancelled run kept a stale `scans.*_found` rollup
- every PDF report raised `KeyError: 'software'`
- `version_key` truncated a wide numeric run and inverted release ordering
- psycopg raises a NUL as a bare ValueError, above StatementError, so every
savepoint guard was blind to the case it was written for
Performance
- endpoints.host had no trigram index: 35.9ms seq scan -> 0.389ms
- free text compiled to one OR, not the prescribed UNION ALL: 132ms -> 41ms
- `/http-assets/summary` and `/ips/summary` materialised whole tables
- the cross-link `ip` kind ran an estate query per page and drew nothing
Frontend
- `$effect` loaders that read the state they write, an unbounded request loop
on a failing endpoint
- error states that rendered as empty states
- 2FA backup codes were unusable while the copy said they worked
- a "Scan history" dialog that always reported none
- ~400 duplicated lines across five filter bars, now one component
Build and tooling
- every scanner installed at @latest; all 13 pinned to the measured versions
- CI built `web/`, gone since 2.x, with tags that cannot be valid
- the prettier pre-commit hook discarded its filenames and always passed
- the test suite destroyed itself when run twice
Also: subdomains from public code, via a pinned github-subdomains source and a
GitHub API key that subfinder receives too; sublist3r and oneforall removed
after measuring one unique host in 89 across four targets.
Verified: 853 tests, ruff clean over eleven packages, svelte-check 0 errors,
the count-is-a-promise gate at 131 checks and 0 mismatches, and 143 GET routes
with no 5xx. Four migrations, each with an exercised downgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVkdx5ynZ6aof5kLFczmhX
Proxy observations are written as endpoint rows with source proxy into the
target's covering scan; a target without a census scan gets a Browsing run,
and finalize replays recorded shapes into a new census scan. "Seen by a
scan" is host plus path shape (new endpoints.shape column, backfilled),
with parameter differences reported as their own notice and unseen claimed
only when a settled scan covers the target.
Removed the scan trigger, sessions and coverage surfaces. The jar is served
from binaries/, the setup dialog derives the endpoint from the browser
origin, scans dispatch one run per target with the chosen scan context, and
manual findings mask credential headers. Endpoints page gains Browsed and
Not browsed filters; the Burp tab gains a host line, a notices table with
Open in reNgine, and a report dialog with notes and a result.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMmqqNaMMjCJWzGG49Xc1i
The comment rule in CLAUDE.md is one line maximum, never narration: no
multi-line blocks, no module-header explainers, no sentences that explain or
justify the code to the reader. The tree had drifted a long way from it, and
most of the drift is mine.
Removed:
232 lines of multi-line comment blocks
203 single-line comments that argued instead of stating
154 comments joined by so/because/otherwise/rather-than
143 narrating comments in the frontend
112 test docstrings (the test name already says what it asserts)
25 test module docstrings and 25 decorative section dividers
Collapsed:
324 multi-line docstrings to one factual line
271 one-line docstrings that had a justifying clause bolted on
Kept: functional directives, section dividers, short field labels, alembic
revision headers, and the eight FastAPI route docstrings that render in /docs.
Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs.
284 python tests and 34 frontend tests pass, ruff and ruff-format clean across
ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
The Caido connector had no client. Its setup steps pointed at the HAR importer
and then said "no native plugin yet", so the only thing it added over any other
proxy was a name in the picker and a promise reNgine could not keep.
Removed: connectors/caido/, the ConnectorKind value, the frontend union, and the
three UI strings that named it beside Burp. connectors.kind is a varchar rather
than a native pg enum, so no migration is needed, and the table holds no rows of
that kind. POST /connectors now answers "Unknown connector 'caido'." with a 400.
clients/har stays. It is not Caido code — it reads a HAR and posts it through the
ingest contract, which is the path for ZAP, mitmproxy, Burp and every browser's
devtools. Its prose named Caido first in that list; it does not now, and
connectors/README points at it for any proxy that only exports HAR.
311 python tests and 34 frontend tests still pass; svelte-check, eslint,
prettier and ruff clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
EndpointSource.FUZZ had a label, a rank and a SourceKind and no producer.
stages/content_discovery/ is it: ffuf against every live site with a ranked
wordlist, level 5 beside url_discovery, a capability a person chooses, and
enabled=False by default so an existing engine does not silently gain a run
that sends 25,000 requests.
tools/data/content.txt ships with the image (6,000 words, SecLists raft small,
directories interleaved before files at each rank, regenerated by
scripts/fetch_content_wordlist.py and committed like the port registry so an
offline box still has one). The order is the budget, exactly as the subdomain
list's is.
One ffuf process per site, never several sites through a second wordlist
keyword. The clusterbomb form looks like the efficient shape and silently
defeats the only filter that matters: measured on one soft-404 host,
`-w hosts:HOSTW -w words:FUZZ -u HOSTW/FUZZ -ac -ach` returned 278 of 300
guessed paths, while the same host asked in its own process returned 0. ffuf
cannot calibrate against a host it does not know until the payload is
substituted. Sites run over a small thread pool and only the stage thread
writes, the url_discovery rule.
A second belt sits behind calibration because no calibration is perfect: a site
answering to 25% or more of a ranked list is answering to anything, so its whole
run is dropped and the sites are named in a warning rather than quietly
filtered. -maxtime is the per-site budget, -sf gives up on a host returning 403
to everything, and a run that hit its budget, a site that was dropped and a site
that could not be reached are each counted and reported.
Rows store is_probed=True with the status ffuf observed — it made the request,
so the status is an observation, not an inference. `input.FUZZ` is base64 in
-json mode (plain in -of json, which is why vhost() is fine), so the word is
read off `url`.
Verified live: 3 sites x 300 words in 11.7s, 295 raw hits became 17 real ones —
/admin, /login, /media, /modules and /index.php among them.
29 tests, each proven to fail against 10 mutations.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
There was no backup path at all, and the connection arithmetic did not add up.
scripts/backup.sh dumps the four things an instance actually is: the database, the
scan media on disk, the docker volumes holding uploaded wordlists, nuclei templates,
report themes and fonts, and the settings a restore needs. SECRET_KEY travels with
it deliberately — API keys, proxy credentials and notification secrets are Fernet
encrypted with a key derived from it, so an archive without it restores a database
full of unreadable secrets. The header says so, and backups/ and *.tar are
gitignored, because an archive is a credential.
Restore is the half that is usually never tried, so it was: a 305 MB archive of this
instance was restored over it and came back at 9 scans, 59,858 subdomains, 535,000
endpoints, 29 findings and 7 targets — identical — with the migration head intact,
the API answering and an encrypted API key still decrypting. It stops the api and
both workers first so nothing writes underneath it, and asks for the database name
before it destroys anything.
The pools could ask for more connections than the server would give. The api takes
10 + 20, each celery prefork child takes 2 + 3, and beat takes another pool: at the
concurrency of 8 this box runs that is 75 against a max_connections of 100, and at
the 12 the compose file now defaults to it is 95 — five short of the ceiling, which
fails as a checkout timeout under load and reads like a slow query.
max_connections is now set from compose with the arithmetic beside it, and the api
checks it at startup: pool_demand() against max_connections less the superuser
reserve, warned once rather than discovered during an incident. The inputs moved to
shared settings because the sum spans both processes and neither could see the
other's numbers.
Worth recording: .env on this box pins CELERY_SCAN_CONCURRENCY=8, so raising the
compose default to 12 changes new installs and not this one. It also still carries
CELERY_DEFAULT_CONCURRENCY, which nothing has read since it was renamed.
Three tests: the demand counts every pool, the shipped defaults fit the shipped
server, and an oversized pool warns rather than failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
A connector is a pipe between a web proxy and reNgine. It records what a
person browses while testing, says what they missed without sending a
packet, and scans only what they agree to.
connectors/ is a top-level package mounted into the api, plug-and-play the
way stages, report sections and MCP tools are: one directory per proxy,
auto-discovered. Burp and Caido ship. Auth reuses the MCP service-token
model (rngconn_, SHA-256 at rest, project-bound, revocable) behind the
per-endpoint failure limiter as well as the global throttle.
Scope belongs to the moment of testing, not to the connection, so the
proxy carries a Working on picker filled from reNgine. A chosen target
wins over hostname matching; an unmatched request is kept and marked
unassigned rather than discarded, and adding the target later attaches
everything already recorded under it.
The bridge runs both ways. reNgine hands back queued work the proxy
collects and opens in Repeater, the scope rules for a target or a whole
bug bounty program, what it already knows about the host being tested,
and notices worth hearing while the tester is still testing. A finding
confirmed by hand is reported from the proxy and stored with its own
provenance, so triage, is:new, reports and compliance all apply to it.
Out-of-scope detection is the one that matters: any request to a host a
bug bounty program forbids is flagged loud, and reNgine refuses to offer
it as a target.
clients/burp is the Montoya extension, Java with no runtime dependencies,
in-tree while the ingest contract is still moving. Everything it uses is
on Burp's free tier.
Verified by scripts/connector_e2e.py (95 assertions) and walked as a
pentester by scripts/connector_journey.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017t7ysDm1v8bzyc9945cd3Z
The Google Fonts css2 API answers a modern user agent with one @font-face per
script, all sharing a family and weight and separated only by unicode-range.
WeasyPrint then shapes a run against one subset file and rasterises it from
another: the PDF's text layer read back correctly while the page showed
"ΨΗΦΙΑΚΗ ΥΠΗΡΕΣΙΑ" as "N ĂN ĐỀ ẤU NẤGĂNG". Every non-Latin page title in a
report was gibberish.
Ask the API with a user agent that predates unicode-range and it answers with
one whole file per family, weight and style; subset that here with fontTools
and save it as woff2. The vendoring script now refuses to write a face whose
block still carries a unicode-range, so the split cannot come back unnoticed.
131 files and 3.3 MB become 23 files and 1.2 MB — smaller as well as correct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AW5pitnosX8LEFEYxH8gFS
reNgine 2 could produce a PDF. This produces a deliverable, and it is built
the way scan engines are built: sections are plug-and-play directories under
reports/, each with a pydantic config model that IS the source of truth. That
model feeds GET /reports/catalog, the builder renders its own form from it, and
the API validates against it. Adding a section is a directory. Scan Engine is to
stages what Report Template is to sections.
WeasyPrint 69 renders it, chosen over Chromium and Typst after measuring what
each can actually do. Only real CSS Paged Media gives a table of contents with
true page numbers, running headers, a PDF outline and footnotes; SysReptor
reaches the same verdict and pipes Chromium's HTML into WeasyPrint for exactly
that reason. reNgine drops the Chromium hop because it templates in Jinja and
draws its charts as SVG in Python, so a browser buys nothing and costs 400 MB
per report. Typst was rejected because a user-uploadable theme would then have
to be Typst code rather than CSS.
A theme is data. shared/definitions/report_theme.py is the schema, six themes
ship as YAML, and a user uploads their own; there is no markup in a theme, so
there is no injection surface. Fonts are vendored so an offline instance still
renders in Latin, Greek and Cyrillic.
Everything the document claims is computed once, in reports/analysis/engine.py:
a posture score whose every deduction is named, weaknesses ranked by the signals
behind the rank rather than severity alone, attack paths that chain observations
into a route, a remediation plan ordered by risk removed, per-dimension change
against the previous run, and caveats that state what was never assessed. Two
narrators read that one brief. Without AI the report still argues its case; with
AI the prose is better and the model never sees a row, only the brief, capped and
cached per check. An AI failure falls back and the report still completes.
Generation is a job. POST /reports queues, the worker walks progress over SSE,
and four formats come out of one render: PDF, self-contained HTML, Markdown and
JSON. The estimate reads each section's own limits, so the page count it promises
is close to the one it delivers.
AI moves out of Settings into its own top-level page: connection, a model per
tier, what it may be used for, what it has cost, and the narrative cache.
Verified on a 1,379-host gov.cy scan: 124 pages in 19.5s across 19 sections,
all six themes, and a broken key still producing a complete report.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f2Mf2HGwFaTRnui4UxpRd
Discovery was passive-only — subfinder, ctfr, crtname, assetfinder and certificate
SANs — so anything absent from a public source was absent from the scan, and every
later stage inherited the miss. It now runs two more sources after the passive merge,
carrying SubdomainSource.BRUTEFORCE and PERMUTATION through the same scoping and
merge every other provider uses.
Bruteforce is `dnsx -d <apex> -w <list>` over a wordlist shipped in the repo:
4,860 labels ranked by real-world frequency, regenerated by
scripts/fetch_subdomain_wordlist.py and committed like the port registry, so an
install with no egress still has one. The word budget reads from the top of that
ranking, which is why a smaller budget is simply the first N lines. Permutation is
alterx over the names already found, seeded apex-first because a name one label from
the apex has more siblings worth guessing than a five-label one.
Two things this needed to be worth running at all:
- **A budget, not an idle watchdog.** A guessed name that does not exist produces
no output, so the stall detector that protects the resolver killed both runs at
exactly 90s the first time. Guessing is bounded by total time instead, floored at
5 names/s against a measured 9, and says so when it stops early.
- **Wildcard rejection.** The wildcard probe moved ahead of expansion: on a wildcard
zone every guess answers, and storing 5,000 fake names would be worse than not
guessing. Answers that resolve only to the wildcard address are dropped and
counted in the source's note.
Both are skipped at passive intensity and say why — the stage still touches no
target, but a wordlist run does query the target's own nameservers. Bruteforce
defaults on at 1,000 words (110s measured, 2 hits on vulnweb.com, both attributed
correctly alongside their passive sources); permutations default off, because every
variant is another DNS query.
ProviderResult gained `note` so a source can come up short without being reported as
failed. alterx reads seeds from stdin — `-l <file>` answers "no input found".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
An open port with no name is a dead end. The IANA Service Name and Transport
Protocol Port Number Registry (public domain, no key) fills the ~5,800 TCP
ports our curated map does not cover, committed as a generated data file
because it is small and slow-moving; scripts/fetch_port_registry.py rebuilds it.
IANA is the fallback vocabulary, never the authority. It still calls 9200
wap-wsp and 5601 esmagent while the internet runs Elasticsearch and Kibana
there, so WELL_KNOWN wins and gained ASM-framed descriptions of its own —
"Docker API without TLS, equivalent to root on the host" beats "Docker REST
API (plain text)". 80, 443, kibana, nessus, consul, kubelet and rabbitmq are
now named rather than falling through to a generic "http".
Provenance is carried, not hidden: describe() returns (text, from_registry)
and a registered name is presented as "IANA registration for port N. This scan
did not confirm it." — because IANA says port 12345 is a chat system. A name
the UI shows is always a name `service:<name>` can find, so the registry
writes through service_for_port into the stored column and a migration
backfills rows written before it existed.
Services also get an identity mark: the vendored Wappalyzer logo when the
service is a product, the class glyph when it is a protocol.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
TLDR; Script updated to abort uninstall process if anything other than the specified "y/Y/Yes/yes/YES" is entered. Fixes uninstall on user inputs like "no" or blank input.
---
The uninstall.sh script does not abort if any user input is provided.
The following user-input scenarios were tested, confirmed and fixed:
- no input
- "n"
- "N"
- "No"
- "1"
- "q"
- "
- (
- '
- <
- [
- {
- /
Because the `make` commands need to be executed in the same directory as where the Makefile is located, putting `update.sh` is not going to work. This commit moves the `update.sh` file back to the root directory.
Yogesh Ojha