build.yml pushes yogeshojha/rengine-api and rengine-worker from the
development target on every master push and every release. That target
deliberately does not carry stages/, reports/, interest/, mcp/,
channels/, connectors/ or toolbox/ — compose mounts them from the host
— so the published api image holds app/ nothing and `import app.main`
answers ModuleNotFoundError. Both Dockerfiles said a prod stage must
copy them; neither had one.
Both now do, and the two build workflows target it. Verified by
building each and importing inside: the api loads app.main with
production settings, the worker loads its celery app and all 27 stages.
The ruff package list in the test workflow is written once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DafWxfoWZXVDgr9KroJBDZ
Four compose services declared their own build against worker/Dockerfile, so
buildkit shared the solve and then exported and unpacked the same image once per
tag. Compose does not dedupe identical targets, measured on a three-service
probe. Under the containerd snapshotter the unpack is a second full pass over
every byte, as expensive as the export.
The go module and build caches were baked into layers: `go install nuclei` made
a 2.66GB layer for a 190MB binary, and /root/go/pkg/mod (3.4GB) plus
/root/.cache/go-build (3.3GB) were half the 13.1GB image.
- the tools build in a throwaway stage with --mount=type=cache on GOCACHE and
GOMODCACHE; the runtime stage copies /out/bin alone
- worker-default builds rengine-worker:latest; worker-beat, ct-stream and
flower reference it. Building one of those alone now needs worker-default
built first
- the runtime stage drops the go toolchain, build-essential and libpcap-dev;
uv sync needs no compiler. libpcap0.8t64 stays, naabu dlopens it for SYN so
ldd shows nothing and the loss would only surface at scan time
- -ldflags=-s trims the binaries 650MB -> 495MB
- only shared/ and tools/ are copied, being file:// path deps; stages/,
reports/, interest/, toolbox/, mcp/, connectors/ and binaries/ are mounted
- the frontend fetches tech icons above COPY frontend/. ., so a source edit no
longer re-runs the tarball download
- .dockerignore excludes .git (271MB) and node_modules (320MB); COPY frontend/.
. had been copying the host's node_modules over the npm ci result
worker image 13.1GB -> 2.41GB, frontend 1.38GB -> 969MB
worker export+unpack 453s across three tags -> 55s across one
editing stages/ or reports/ now rebuilds nothing (1s), frontend/src 4s,
shared/ 19s; cold build 475s -> 414s, still dominated by go compilation
Tool versions diff identical against the old image, naabu SYN proven against a
live listener, 8/8 services up, 853 tests green, ruff clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015fWjAbvFBdmsmVeMcwaaR9
A full static review of the 3.0 tree, run as two fan-out passes with every
finding verified against the running stack rather than taken on inspection.
Correctness and security
- instance settings encrypted the mask over the stored AI key: `!= MASK` never
matched `••••••••abcd`, so saving settings destroyed the key and `test_ai`
posted the mask as a bearer token
- the MFA token was replayable for its full five-minute life; the gate now
checks revocation and consumes the token after a successful exchange
- logout revoked nothing for a bearer client while answering "Logged out"
- `POST /api-keys/{id}/test` returned the plaintext key inside httpx's error
string, which embeds the whole URL
- an interest rule could be edited or deleted through another project's id
- redis ran the celery broker under `allkeys-lru`, free to evict a queued task
or a chord counter; now `noeviction`
- nuclei's stored command carried the OAST token and every auth header
- a finalize step could poison the session and leave a scan RUNNING for ever
- `is:issue` raised TypeError on every search
- offset paging repeated and skipped rows in four dimensions
- a cancelled run kept a stale `scans.*_found` rollup
- every PDF report raised `KeyError: 'software'`
- `version_key` truncated a wide numeric run and inverted release ordering
- psycopg raises a NUL as a bare ValueError, above StatementError, so every
savepoint guard was blind to the case it was written for
Performance
- endpoints.host had no trigram index: 35.9ms seq scan -> 0.389ms
- free text compiled to one OR, not the prescribed UNION ALL: 132ms -> 41ms
- `/http-assets/summary` and `/ips/summary` materialised whole tables
- the cross-link `ip` kind ran an estate query per page and drew nothing
Frontend
- `$effect` loaders that read the state they write, an unbounded request loop
on a failing endpoint
- error states that rendered as empty states
- 2FA backup codes were unusable while the copy said they worked
- a "Scan history" dialog that always reported none
- ~400 duplicated lines across five filter bars, now one component
Build and tooling
- every scanner installed at @latest; all 13 pinned to the measured versions
- CI built `web/`, gone since 2.x, with tags that cannot be valid
- the prettier pre-commit hook discarded its filenames and always passed
- the test suite destroyed itself when run twice
Also: subdomains from public code, via a pinned github-subdomains source and a
GitHub API key that subfinder receives too; sublist3r and oneforall removed
after measuring one unique host in 89 across four targets.
Verified: 853 tests, ruff clean over eleven packages, svelte-check 0 errors,
the count-is-a-promise gate at 131 checks and 0 mismatches, and 143 GET routes
with no 5xx. Four migrations, each with an exercised downgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVkdx5ynZ6aof5kLFczmhX
Proxy observations are written as endpoint rows with source proxy into the
target's covering scan; a target without a census scan gets a Browsing run,
and finalize replays recorded shapes into a new census scan. "Seen by a
scan" is host plus path shape (new endpoints.shape column, backfilled),
with parameter differences reported as their own notice and unseen claimed
only when a settled scan covers the target.
Removed the scan trigger, sessions and coverage surfaces. The jar is served
from binaries/, the setup dialog derives the endpoint from the browser
origin, scans dispatch one run per target with the chosen scan context, and
manual findings mask credential headers. Endpoints page gains Browsed and
Not browsed filters; the Burp tab gains a host line, a notices table with
Open in reNgine, and a report dialog with notes and a result.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMmqqNaMMjCJWzGG49Xc1i
Scan page froze on a duplicate each key; every tab body now sits in a
svelte:boundary. Composite discovered_at indexes on the five result
tables, leads counted in one FILTER pass, dashboard and scan-service
first-seen counts bounded to the scans in the window, ANALYZE at
finalize, shm_size on the db service, SQL_ECHO instead of DEBUG, an
entrypoint that serves with workers outside reload.
Settled-scope aggregates and search pages are cached by target
revision; triage, dismissal, interest evaluation, notes, certificate
rechecks and re-ranking bump it. Requests per page cut: single-flight
GETs, one schemas endpoint, batched whois correlations, a counts
endpoint for the dashboard, deduped store loads, trimmed coverage
payloads.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NWRq8utu7KXsVBVU3gYNi7
reNgine 2 had a Tools section; 3.0 lost it, so a hunter left the product
for a whois or a dns lookup even though every capability was already here
inside a scan.
`toolbox/` is a top-level package mounted into api and worker, and a tool
is one file under `toolbox/tools/` — auto-discovered exactly like stages,
report sections and MCP tools, with a pydantic `Input` whose flattened
schema IS the form the UI renders. Six ship: WHOIS, DNS records, subdomain
finder, HTTP probe, IP address and CVE.
Where a tool runs is a declaration the registry enforces. The api image
carries no tool binaries, so `INLINE` answers from RDAP, the offline range
tables or the threat feeds on the request's AsyncSession, and `QUEUED`
shells out on the worker; `_validate` rejects a class whose `run` is async
or sync the wrong way round, so the two cannot drift.
A run is a question rather than inventory, so it lives in Redis with a
seven-day TTL and no migration, keyed by user id. A queued run nobody
collects fails after two minutes instead of spinning for ever.
A tool answers in blocks — facts, table, tags, code, note — so one
frontend renderer serves every tool and a new one costs no frontend work.
An empty block with nothing to say about being empty is dropped.
`require_public` refuses loopback, private and link-local before the HTTP
probe sends anything, because an authenticated user typing 169.254.169.254
is an SSRF.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CtRcxFCPSYxW5sXam3SUWB
A connector is a pipe between a web proxy and reNgine. It records what a
person browses while testing, says what they missed without sending a
packet, and scans only what they agree to.
connectors/ is a top-level package mounted into the api, plug-and-play the
way stages, report sections and MCP tools are: one directory per proxy,
auto-discovered. Burp and Caido ship. Auth reuses the MCP service-token
model (rngconn_, SHA-256 at rest, project-bound, revocable) behind the
per-endpoint failure limiter as well as the global throttle.
Scope belongs to the moment of testing, not to the connection, so the
proxy carries a Working on picker filled from reNgine. A chosen target
wins over hostname matching; an unmatched request is kept and marked
unassigned rather than discarded, and adding the target later attaches
everything already recorded under it.
The bridge runs both ways. reNgine hands back queued work the proxy
collects and opens in Repeater, the scope rules for a target or a whole
bug bounty program, what it already knows about the host being tested,
and notices worth hearing while the tester is still testing. A finding
confirmed by hand is reported from the proxy and stored with its own
provenance, so triage, is:new, reports and compliance all apply to it.
Out-of-scope detection is the one that matters: any request to a host a
bug bounty program forbids is flagged loud, and reNgine refuses to offer
it as a target.
clients/burp is the Montoya extension, Java with no runtime dependencies,
in-tree while the ingest contract is still moving. Everything it uses is
on Burp's free tier.
Verified by scripts/connector_e2e.py (95 assertions) and walked as a
pentester by scripts/connector_journey.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017t7ysDm1v8bzyc9945cd3Z
Recon MCP servers expose one verb: go and scan. reconFTW v5 ships seven tools
and not one of them reads a result, because a pipeline that writes files has
nothing to query. reNgine's value is the correlated corpus, so this server is
read-first — fourteen tools for asking questions, of which four can act and
only two reach a target. Wrapping the OpenAPI document instead would have cost
276 tools and 112,600 tokens of definitions, and would still not work, because
every result endpoint needs a scan id an agent has no way to choose.
Everything lives in mcp/, mounted into the api the way stages/ and reports/
are. The protocol is implemented directly rather than through an SDK: a
tools-only server is a small JSON-RPC surface, this keeps the package free of
dependencies, and protocol.py is the only module that knows the wire format.
Tools are plug-and-play like stages and report sections — one file under
mcp/tools/, a Tool subclass whose Input model is both the schema the model
plans against and the schema the server validates with. Nothing registers it.
mcp/dimensions.py is the second axis, one entry per result dimension, so
query_assets, group_assets, surface_brief and describe_query_language each stay
short and a sixth dimension is a single entry.
ToolResult carries the product's contracts structurally. Every count comes back
with the query token that reproduces it, so a number an agent reports equals
the rows a human lands on. A dimension nobody has scanned is reported as not
scanned rather than as zero, which is why resolve_target resolves the scan that
actually covered each dimension instead of taking the newest one. Rows carrying
text the scanned party wrote are flagged untrusted, so the model reports them
rather than following them.
Access is an opaque service token, hashed at rest, bound to a project and a
capability set, revocable by row. Read, plan, write and launch are separate;
launch is the only one that reaches a machine you do not own and is off at the
instance ceiling until an operator raises it. No token may exceed the ceiling,
lowering it narrows the tokens that already exist, and tools/list returns only
what a token may call. Start and stop live on the MCP page rather than in a
config file — the server is mounted in the api, so stopped means refusing
connections rather than a supervised daemon, and the dialog says so: stopping
does not cancel a scan an agent already started.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J6fcWDtvxTer9chPWAwKd2
reNgine 2 could produce a PDF. This produces a deliverable, and it is built
the way scan engines are built: sections are plug-and-play directories under
reports/, each with a pydantic config model that IS the source of truth. That
model feeds GET /reports/catalog, the builder renders its own form from it, and
the API validates against it. Adding a section is a directory. Scan Engine is to
stages what Report Template is to sections.
WeasyPrint 69 renders it, chosen over Chromium and Typst after measuring what
each can actually do. Only real CSS Paged Media gives a table of contents with
true page numbers, running headers, a PDF outline and footnotes; SysReptor
reaches the same verdict and pipes Chromium's HTML into WeasyPrint for exactly
that reason. reNgine drops the Chromium hop because it templates in Jinja and
draws its charts as SVG in Python, so a browser buys nothing and costs 400 MB
per report. Typst was rejected because a user-uploadable theme would then have
to be Typst code rather than CSS.
A theme is data. shared/definitions/report_theme.py is the schema, six themes
ship as YAML, and a user uploads their own; there is no markup in a theme, so
there is no injection surface. Fonts are vendored so an offline instance still
renders in Latin, Greek and Cyrillic.
Everything the document claims is computed once, in reports/analysis/engine.py:
a posture score whose every deduction is named, weaknesses ranked by the signals
behind the rank rather than severity alone, attack paths that chain observations
into a route, a remediation plan ordered by risk removed, per-dimension change
against the previous run, and caveats that state what was never assessed. Two
narrators read that one brief. Without AI the report still argues its case; with
AI the prose is better and the model never sees a row, only the brief, capped and
cached per check. An AI failure falls back and the report still completes.
Generation is a job. POST /reports queues, the worker walks progress over SSE,
and four formats come out of one render: PDF, self-contained HTML, Markdown and
JSON. The estimate reads each section's own limits, so the page count it promises
is close to the one it delivers.
AI moves out of Settings into its own top-level page: connection, a model per
tier, what it may be used for, what it has cost, and the narrative cache.
Verified on a 1,379-host gov.cy scan: 124 pages in 19.5s across 19 sections,
all six themes, and a broken key still producing a complete report.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f2Mf2HGwFaTRnui4UxpRd
- Implemented TargetEnrichmentStage for enriching targets with DNS, WHOIS, and BGP context.
- Added VhostStage for virtual host discovery using host-header fuzzing.
- Introduced WafDetectStage for fingerprinting web application firewalls.
- Created configuration files for each new stage with relevant parameters.
- Updated Dockerfile to copy stages instead of engines.
- Enhanced existing tools and services for better integration with new stages.
- Refactored task management in Celery to accommodate new stages and improve error handling.
- Added models for WHOIS responses including WhoisDomainResponse, WhoisIPResponse, and WhoisASNResponse.
- Created a parser for WHOIS data to convert raw responses into structured models.
- Developed an RDAPProvider class to handle RDAP lookups using the whoisit library.
- Implemented a WhoisService class for performing WHOIS lookups with optional database caching.
- Added methods for finding correlations between WHOIS records based on registrant, registrar, network, country, and nameservers.
- Updated Dockerfile to include tools directory and added rengine-tools dependency in pyproject.toml.
- Introduced a new endpoint to validate multiple target values in a single request.
- Added a constant for maximum target imports.
- Updated the target validation response model to include the target value.
- Modified Dockerfile CMD for graceful shutdown and reload directory.
- Commented out environment variable in docker-compose for polling.
