The Caido connector had no client. Its setup steps pointed at the HAR importer
and then said "no native plugin yet", so the only thing it added over any other
proxy was a name in the picker and a promise reNgine could not keep.
Removed: connectors/caido/, the ConnectorKind value, the frontend union, and the
three UI strings that named it beside Burp. connectors.kind is a varchar rather
than a native pg enum, so no migration is needed, and the table holds no rows of
that kind. POST /connectors now answers "Unknown connector 'caido'." with a 400.
clients/har stays. It is not Caido code — it reads a HAR and posts it through the
ingest contract, which is the path for ZAP, mitmproxy, Burp and every browser's
devtools. Its prose named Caido first in that list; it does not now, and
connectors/README points at it for any proxy that only exports HAR.
311 python tests and 34 frontend tests still pass; svelte-check, eslint,
prettier and ruff clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
EndpointSource.FUZZ had a label, a rank and a SourceKind and no producer.
stages/content_discovery/ is it: ffuf against every live site with a ranked
wordlist, level 5 beside url_discovery, a capability a person chooses, and
enabled=False by default so an existing engine does not silently gain a run
that sends 25,000 requests.
tools/data/content.txt ships with the image (6,000 words, SecLists raft small,
directories interleaved before files at each rank, regenerated by
scripts/fetch_content_wordlist.py and committed like the port registry so an
offline box still has one). The order is the budget, exactly as the subdomain
list's is.
One ffuf process per site, never several sites through a second wordlist
keyword. The clusterbomb form looks like the efficient shape and silently
defeats the only filter that matters: measured on one soft-404 host,
`-w hosts:HOSTW -w words:FUZZ -u HOSTW/FUZZ -ac -ach` returned 278 of 300
guessed paths, while the same host asked in its own process returned 0. ffuf
cannot calibrate against a host it does not know until the payload is
substituted. Sites run over a small thread pool and only the stage thread
writes, the url_discovery rule.
A second belt sits behind calibration because no calibration is perfect: a site
answering to 25% or more of a ranked list is answering to anything, so its whole
run is dropped and the sites are named in a warning rather than quietly
filtered. -maxtime is the per-site budget, -sf gives up on a host returning 403
to everything, and a run that hit its budget, a site that was dropped and a site
that could not be reached are each counted and reported.
Rows store is_probed=True with the status ffuf observed — it made the request,
so the status is an observation, not an inference. `input.FUZZ` is base64 in
-json mode (plain in -of json, which is why vhost() is fine), so the word is
read off `url`.
Verified live: 3 sites x 300 words in 11.7s, 295 raw hits became 17 real ones —
/admin, /login, /media, /modules and /index.php among them.
29 tests, each proven to fail against 10 mutations.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
There was no backup path at all, and the connection arithmetic did not add up.
scripts/backup.sh dumps the four things an instance actually is: the database, the
scan media on disk, the docker volumes holding uploaded wordlists, nuclei templates,
report themes and fonts, and the settings a restore needs. SECRET_KEY travels with
it deliberately — API keys, proxy credentials and notification secrets are Fernet
encrypted with a key derived from it, so an archive without it restores a database
full of unreadable secrets. The header says so, and backups/ and *.tar are
gitignored, because an archive is a credential.
Restore is the half that is usually never tried, so it was: a 305 MB archive of this
instance was restored over it and came back at 9 scans, 59,858 subdomains, 535,000
endpoints, 29 findings and 7 targets — identical — with the migration head intact,
the API answering and an encrypted API key still decrypting. It stops the api and
both workers first so nothing writes underneath it, and asks for the database name
before it destroys anything.
The pools could ask for more connections than the server would give. The api takes
10 + 20, each celery prefork child takes 2 + 3, and beat takes another pool: at the
concurrency of 8 this box runs that is 75 against a max_connections of 100, and at
the 12 the compose file now defaults to it is 95 — five short of the ceiling, which
fails as a checkout timeout under load and reads like a slow query.
max_connections is now set from compose with the arithmetic beside it, and the api
checks it at startup: pool_demand() against max_connections less the superuser
reserve, warned once rather than discovered during an incident. The inputs moved to
shared settings because the sum spans both processes and neither could see the
other's numbers.
Worth recording: .env on this box pins CELERY_SCAN_CONCURRENCY=8, so raising the
compose default to 12 changes new installs and not this one. It also still carries
CELERY_DEFAULT_CONCURRENCY, which nothing has read since it was renamed.
Three tests: the demand counts every pool, the shipped defaults fit the shipped
server, and an oversized pool warns rather than failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
A connector is a pipe between a web proxy and reNgine. It records what a
person browses while testing, says what they missed without sending a
packet, and scans only what they agree to.
connectors/ is a top-level package mounted into the api, plug-and-play the
way stages, report sections and MCP tools are: one directory per proxy,
auto-discovered. Burp and Caido ship. Auth reuses the MCP service-token
model (rngconn_, SHA-256 at rest, project-bound, revocable) behind the
per-endpoint failure limiter as well as the global throttle.
Scope belongs to the moment of testing, not to the connection, so the
proxy carries a Working on picker filled from reNgine. A chosen target
wins over hostname matching; an unmatched request is kept and marked
unassigned rather than discarded, and adding the target later attaches
everything already recorded under it.
The bridge runs both ways. reNgine hands back queued work the proxy
collects and opens in Repeater, the scope rules for a target or a whole
bug bounty program, what it already knows about the host being tested,
and notices worth hearing while the tester is still testing. A finding
confirmed by hand is reported from the proxy and stored with its own
provenance, so triage, is:new, reports and compliance all apply to it.
Out-of-scope detection is the one that matters: any request to a host a
bug bounty program forbids is flagged loud, and reNgine refuses to offer
it as a target.
clients/burp is the Montoya extension, Java with no runtime dependencies,
in-tree while the ingest contract is still moving. Everything it uses is
on Burp's free tier.
Verified by scripts/connector_e2e.py (95 assertions) and walked as a
pentester by scripts/connector_journey.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017t7ysDm1v8bzyc9945cd3Z
The Google Fonts css2 API answers a modern user agent with one @font-face per
script, all sharing a family and weight and separated only by unicode-range.
WeasyPrint then shapes a run against one subset file and rasterises it from
another: the PDF's text layer read back correctly while the page showed
"ΨΗΦΙΑΚΗ ΥΠΗΡΕΣΙΑ" as "N ĂN ĐỀ ẤU NẤGĂNG". Every non-Latin page title in a
report was gibberish.
Ask the API with a user agent that predates unicode-range and it answers with
one whole file per family, weight and style; subset that here with fontTools
and save it as woff2. The vendoring script now refuses to write a face whose
block still carries a unicode-range, so the split cannot come back unnoticed.
131 files and 3.3 MB become 23 files and 1.2 MB — smaller as well as correct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AW5pitnosX8LEFEYxH8gFS
reNgine 2 could produce a PDF. This produces a deliverable, and it is built
the way scan engines are built: sections are plug-and-play directories under
reports/, each with a pydantic config model that IS the source of truth. That
model feeds GET /reports/catalog, the builder renders its own form from it, and
the API validates against it. Adding a section is a directory. Scan Engine is to
stages what Report Template is to sections.
WeasyPrint 69 renders it, chosen over Chromium and Typst after measuring what
each can actually do. Only real CSS Paged Media gives a table of contents with
true page numbers, running headers, a PDF outline and footnotes; SysReptor
reaches the same verdict and pipes Chromium's HTML into WeasyPrint for exactly
that reason. reNgine drops the Chromium hop because it templates in Jinja and
draws its charts as SVG in Python, so a browser buys nothing and costs 400 MB
per report. Typst was rejected because a user-uploadable theme would then have
to be Typst code rather than CSS.
A theme is data. shared/definitions/report_theme.py is the schema, six themes
ship as YAML, and a user uploads their own; there is no markup in a theme, so
there is no injection surface. Fonts are vendored so an offline instance still
renders in Latin, Greek and Cyrillic.
Everything the document claims is computed once, in reports/analysis/engine.py:
a posture score whose every deduction is named, weaknesses ranked by the signals
behind the rank rather than severity alone, attack paths that chain observations
into a route, a remediation plan ordered by risk removed, per-dimension change
against the previous run, and caveats that state what was never assessed. Two
narrators read that one brief. Without AI the report still argues its case; with
AI the prose is better and the model never sees a row, only the brief, capped and
cached per check. An AI failure falls back and the report still completes.
Generation is a job. POST /reports queues, the worker walks progress over SSE,
and four formats come out of one render: PDF, self-contained HTML, Markdown and
JSON. The estimate reads each section's own limits, so the page count it promises
is close to the one it delivers.
AI moves out of Settings into its own top-level page: connection, a model per
tier, what it may be used for, what it has cost, and the narrative cache.
Verified on a 1,379-host gov.cy scan: 124 pages in 19.5s across 19 sections,
all six themes, and a broken key still producing a complete report.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f2Mf2HGwFaTRnui4UxpRd
Discovery was passive-only — subfinder, ctfr, crtname, assetfinder and certificate
SANs — so anything absent from a public source was absent from the scan, and every
later stage inherited the miss. It now runs two more sources after the passive merge,
carrying SubdomainSource.BRUTEFORCE and PERMUTATION through the same scoping and
merge every other provider uses.
Bruteforce is `dnsx -d <apex> -w <list>` over a wordlist shipped in the repo:
4,860 labels ranked by real-world frequency, regenerated by
scripts/fetch_subdomain_wordlist.py and committed like the port registry, so an
install with no egress still has one. The word budget reads from the top of that
ranking, which is why a smaller budget is simply the first N lines. Permutation is
alterx over the names already found, seeded apex-first because a name one label from
the apex has more siblings worth guessing than a five-label one.
Two things this needed to be worth running at all:
- **A budget, not an idle watchdog.** A guessed name that does not exist produces
no output, so the stall detector that protects the resolver killed both runs at
exactly 90s the first time. Guessing is bounded by total time instead, floored at
5 names/s against a measured 9, and says so when it stops early.
- **Wildcard rejection.** The wildcard probe moved ahead of expansion: on a wildcard
zone every guess answers, and storing 5,000 fake names would be worse than not
guessing. Answers that resolve only to the wildcard address are dropped and
counted in the source's note.
Both are skipped at passive intensity and say why — the stage still touches no
target, but a wordlist run does query the target's own nameservers. Bruteforce
defaults on at 1,000 words (110s measured, 2 hits on vulnweb.com, both attributed
correctly alongside their passive sources); permutations default off, because every
variant is another DNS query.
ProviderResult gained `note` so a source can come up short without being reported as
failed. alterx reads seeds from stdin — `-l <file>` answers "no input found".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
An open port with no name is a dead end. The IANA Service Name and Transport
Protocol Port Number Registry (public domain, no key) fills the ~5,800 TCP
ports our curated map does not cover, committed as a generated data file
because it is small and slow-moving; scripts/fetch_port_registry.py rebuilds it.
IANA is the fallback vocabulary, never the authority. It still calls 9200
wap-wsp and 5601 esmagent while the internet runs Elasticsearch and Kibana
there, so WELL_KNOWN wins and gained ASM-framed descriptions of its own —
"Docker API without TLS, equivalent to root on the host" beats "Docker REST
API (plain text)". 80, 443, kibana, nessus, consul, kubelet and rabbitmq are
now named rather than falling through to a generic "http".
Provenance is carried, not hidden: describe() returns (text, from_registry)
and a registered name is presented as "IANA registration for port N. This scan
did not confirm it." — because IANA says port 12345 is a chat system. A name
the UI shows is always a name `service:<name>` can find, so the registry
writes through service_for_port into the stored column and a migration
backfills rows written before it existed.
Services also get an identity mark: the vendored Wappalyzer logo when the
service is a product, the class glyph when it is a protocol.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
TLDR; Script updated to abort uninstall process if anything other than the specified "y/Y/Yes/yes/YES" is entered. Fixes uninstall on user inputs like "no" or blank input.
---
The uninstall.sh script does not abort if any user input is provided.
The following user-input scenarios were tested, confirmed and fixed:
- no input
- "n"
- "N"
- "No"
- "1"
- "q"
- "
- (
- '
- <
- [
- {
- /
Because the `make` commands need to be executed in the same directory as where the Makefile is located, putting `update.sh` is not going to work. This commit moves the `update.sh` file back to the root directory.
