mirror of
https://github.com/yogeshojha/rengine.git
synced 2026-10-19 14:51:31 +02:00
72045c0a78e4cbec63bda2bf65a168aca4e18dd9
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Yogesh Ojha
|
a700bbed03 |
feat(scans): concurrent scan limit with a launch-order queue
Launches past the limit stay pending and start in launch order when a slot frees, so each running scan gets full speed and memory stays bounded. - run_scan admits a pending scan under an advisory lock: running plus waiting scans ahead of it must be under the limit - finalize, cancel, pause, delete and a limit change start the next scans; the reaper re-sends only scans whose turn has come - automatic limit: live worker-scans stage slots over the widest step, so --scale raises it; Settings sets a fixed limit from 1 to 50 - watch probes skip the queue, resume is never held - queued scans show how many are ahead |
||
|
Yogesh OjhaandClaude Opus 5
|
b0a47ba087 |
feat(oast): an instance-owned callback server, and a tier that pays the wait once
Out-of-band testing is an instance setting — `off`, `self_hosted` or `public` — with the server, the token (the INTERACTSH API key row) and the wait resolved once in `shared/services/oast.py` and read on the stage thread. A mode that cannot be used is off and says why: self-hosted with no server, or public without the disclosure acknowledged. A fresh instance is off. `POST /oast/test` asks a public address for `/poll` over HTTPS and reports that the server answered, never that OAST works. The engine keeps one switch. `FieldNeeds` puts the resource key on the field and `GET /engines/catalog` fills `unavailable_reason`, so the builder renders the instance reason beside a switch that stays settable. Measured on nuclei v3.11.1: with `-no-interactsh` a check that needs a callback still loads and the request is refused — 0 requests, exit 0. So the batch reported COMPLETED and every origin in it read `scanned` for checks that never touched the target. The selection now drops them when nothing is listening, one coverage row reports the count and the reason, and `Tier.OAST` is left out of the planned tiers. Membership is what a check references, not what it is tagged: 615 of the 12,586 official rows reference an interactsh placeholder and 233 of those carry no `oast` tag. `vuln_templates.needs_oast` is parsed at index time; a library resync fills the column. The wait is paid on out-of-band batches alone, which sweep for 30 minutes so it is paid fewer times, and the tier names the server on its own rows. A finding's callback is rendered on the sheet from one label map; the list path is bared to the metadata. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Yogesh OjhaandClaude Opus 5
|
0afe1a8de8 |
feat(nuclei): monitor targets against new checks, and a review pass on the pipeline
A target can follow the check library: once a day, when the library gains checks, one focused run tests every web asset the target's completed scans found, with those checks only. Per-target watermark, one sweep lock, a `daily.run` chain that registers future once-a-day jobs, two notifications, a What's new group, a "New" badge on the library, a monitored-target signal and a "New checks runs" scan filter. The pipeline review behind it, in three passes: Planner and scanner. The deep sweep is its own tier, outside ROOT_TIERS and after replay, so a cut costs the tail and cluster members are still confirmed; the one-request tier runs first, sized on its clustered cost; blind core ranks CISA then VulnCheck then severity then cost; the planned tiers come from the selection; skipped hosts are marked alone; library gaps are one row per run; the sweep cap is 8 requests. Output. Payload-free fingerprints for fuzzing, out-of-band and dalfox findings; suppression follows the replay edge; the digest baseline is every earlier scan; host and port read through split_host_port; interaction obeys store_evidence and redaction; the rollup excludes replays. Tools. nuclei's exit code, abort and budget are read; a tool arg cannot restate a stage flag under its short spelling; one refused row costs one row; a headless block decides the protocol. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B58eEM68a5qDFTQXPE5aoW |
||
|
Yogesh OjhaandClaude Opus 5
|
0b853e04b6 |
chore: production-readiness review across the tree
A full static review of the 3.0 tree, run as two fan-out passes with every
finding verified against the running stack rather than taken on inspection.
Correctness and security
- instance settings encrypted the mask over the stored AI key: `!= MASK` never
matched `••••••••abcd`, so saving settings destroyed the key and `test_ai`
posted the mask as a bearer token
- the MFA token was replayable for its full five-minute life; the gate now
checks revocation and consumes the token after a successful exchange
- logout revoked nothing for a bearer client while answering "Logged out"
- `POST /api-keys/{id}/test` returned the plaintext key inside httpx's error
string, which embeds the whole URL
- an interest rule could be edited or deleted through another project's id
- redis ran the celery broker under `allkeys-lru`, free to evict a queued task
or a chord counter; now `noeviction`
- nuclei's stored command carried the OAST token and every auth header
- a finalize step could poison the session and leave a scan RUNNING for ever
- `is:issue` raised TypeError on every search
- offset paging repeated and skipped rows in four dimensions
- a cancelled run kept a stale `scans.*_found` rollup
- every PDF report raised `KeyError: 'software'`
- `version_key` truncated a wide numeric run and inverted release ordering
- psycopg raises a NUL as a bare ValueError, above StatementError, so every
savepoint guard was blind to the case it was written for
Performance
- endpoints.host had no trigram index: 35.9ms seq scan -> 0.389ms
- free text compiled to one OR, not the prescribed UNION ALL: 132ms -> 41ms
- `/http-assets/summary` and `/ips/summary` materialised whole tables
- the cross-link `ip` kind ran an estate query per page and drew nothing
Frontend
- `$effect` loaders that read the state they write, an unbounded request loop
on a failing endpoint
- error states that rendered as empty states
- 2FA backup codes were unusable while the copy said they worked
- a "Scan history" dialog that always reported none
- ~400 duplicated lines across five filter bars, now one component
Build and tooling
- every scanner installed at @latest; all 13 pinned to the measured versions
- CI built `web/`, gone since 2.x, with tags that cannot be valid
- the prettier pre-commit hook discarded its filenames and always passed
- the test suite destroyed itself when run twice
Also: subdomains from public code, via a pinned github-subdomains source and a
GitHub API key that subfinder receives too; sublist3r and oneforall removed
after measuring one unique host in 89 across four targets.
Verified: 853 tests, ruff clean over eleven packages, svelte-check 0 errors,
the count-is-a-promise gate at 131 checks and 0 mismatches, and 143 GET routes
with no 5xx. Four migrations, each with an exercised downgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVkdx5ynZ6aof5kLFczmhX
|
||
|
Yogesh OjhaandClaude Opus 5
|
2cb8d73280 |
style: strip explanatory comments across the codebase
The comment rule in CLAUDE.md is one line maximum, never narration: no multi-line blocks, no module-header explainers, no sentences that explain or justify the code to the reader. The tree had drifted a long way from it, and most of the drift is mine. Removed: 232 lines of multi-line comment blocks 203 single-line comments that argued instead of stating 154 comments joined by so/because/otherwise/rather-than 143 narrating comments in the frontend 112 test docstrings (the test name already says what it asserts) 25 test module docstrings and 25 decorative section dividers Collapsed: 324 multi-line docstrings to one factual line 271 one-line docstrings that had a justifying clause bolted on Kept: functional directives, section dividers, short field labels, alembic revision headers, and the eight FastAPI route docstrings that render in /docs. Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs. 284 python tests and 34 frontend tests pass, ruff and ruff-format clean across ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
f3b406ec03 |
feat(freshness): re-check a certificate between scans, which is Class B's first job
The derivation rule names four classes and only three had anything implementing them. Class B is "needs a per-asset request, so a background job with its own freshness, never a stage that blocks", and a certificate is the clearest case: it expires and renews on a schedule no scan is aligned to. A dashboard built on a scan from six weeks ago reports every certificate that has auto-renewed since as expiring or expired — 548 hosts on this box sit inside a 30-day window right now, and on a 90-day ACME cycle every one of them would be wrong within a month. shared/services/cert_freshness.py + app.tasks.freshness.certificates (beat, 4h). The policy is the design: a certificate whose stored expiry is more than 30 days away is re-asked after 12h, one already inside its renewal window after 4h, nearest expiry first, 500 a pass, one tlsx handshake per host. Only the newest scan of each target is refreshed — an older scan's rows are a record of that run, not a claim about today. A host that did not answer keeps what the scan found and still gets its stamp moved: silence is not evidence the certificate changed, but an unreachable host must not be re-asked every four hours. It is opt-in, not opt-out (instance_settings.cert_recheck_enabled, default false, a switch in Settings). The threat feeds only download; this one handshakes the operator's targets outside a scan, and nobody should discover that after the fact. Two tool bugs, both the shape CLAUDE.md already lists four of. CLIToolRunner.run defaults to OutputFormat.PLAIN and only appends json_flag for JSONL, so passing json_flag without the format runs the tool with no -json at all — the first live pass answered 0 of 40. And tlsx writes every record it is going to write and then does not exit (40 hosts: 16s to emit, still running at 90s), so the run is bounded by budget(hosts) and Freshness.cut_short reports a pass that hit the ceiling instead of swallowing it. Together: 300.2s and 0 answers became 15.6s and 37 of 40. 19 tests, each proven to fail against 9 mutations of the policy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
163df52ad5 |
perf(db): a primary key is already an index, so the second one on id was write cost for nothing
SQLModel emitted index=True beside primary_key=True on every id field, so 41 tables carried ix_<table>_id alongside <table>_pkey: the same single-column btree twice. 12 MB of duplicate index, and every insert maintaining both — on the hot path of a scan writing 345,152 endpoints. The planner had been using them (72,160 scans across the set) simply because they were there and equivalent; with them gone it uses the primary key, verified on subdomains: Index Scan using subdomains_pkey. The migration finds them rather than naming them, because which tables exist depends on how far an install has migrated and a name already gone must not fail an upgrade. It drops only a non-unique, single-column btree on id belonging to a table that has a primary key — no other index can match that shape. There is no downgrade: the primary key serves every lookup they served. The 33 models are corrected too, so the next hand-written migration is not comparing against a schema that would want them back. Counted honestly: 41, not the 102 an earlier pass reported — that number included the 61 primary keys themselves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
fe5bc8094c |
feat(bounty-hub): Bugcrowd, Intigriti and YesWeHack from the public feed
Bounty Hub was empty until someone pasted a HackerOne token. It now ships with 464 public programs and 6,889 scope assets from arkadiyt/bounty-targets-data (MIT, republished every 30 minutes), so a fresh install is useful with no credentials at all and the token becomes an upgrade rather than a prerequisite. HackerOne is deliberately not read from the feed. The API gives us private programs and richer scope, so the feed would be a downgrade — and a new `source` column keeps that from happening by accident: an API row is authoritative and the feed never overwrites one. The three platforms do not agree on anything, so each gets its own entry in shared/definitions/bounty_feed.py: the asset field is `target` on two of them and `endpoint` on Intigriti; the type vocabularies are `website`/`network` versus `url`/`wildcard`/`iprange` versus `web-application`/`mobile-application-ios`; and only Intigriti and YesWeHack ship a handle, so Bugcrowd's is derived from its engagement URL. All of it lands on the 17 asset types reNgine already has, and target_for_scope needed no change to resolve 5,072 of them. What the platforms do not report is not invented. Bugcrowd publishes no submission state, so it stores `unknown` and the row shows no badge rather than guessing "open". What they report and HackerOne does not — the actual payout range — becomes a first-class field, a badge and a sort. Provenance is visible rather than implied: every row names its platform, the detail sheet says which source it came from and what that source can and cannot contain, and Platform and Source are faceted filters. The feed has its own schedule because it costs four small files rather than 640 requests: off, every 6 hours, daily or weekly, beside the HackerOne interval in Settings. The change-detection layer needed nothing — it was already platform-agnostic, so scope alerts work on all four platforms, and a first feed load of 464 programs correctly announced none of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu |
||
|
Yogesh OjhaandClaude Opus 5
|
992770478a |
feat(bounty-hub): a Settings tab for the sync schedule and change alerts
The interval belonged in Settings, not as a dropdown in a page header. There is now a Bounty Hub tab under Settings, gated on CAP_BOUNTY_PROGRAMS the same way the nav item is, holding two cards. Sync says how often reNgine asks HackerOne what changed — manual only, daily or weekly — and states the cost so the choice is informed: a full sync is about 640 requests against a 600-per-minute limit. Manual only says plainly that nothing leaves the instance on a timer. Notifications is a master switch plus one toggle per change kind, each with the sentence describing what it means. Turning a kind off does not stop the change being recorded: it still appears in the Updates tab, only the alert is suppressed, and the card says so. Preferences live in instance_settings .bounty_settings, following the mcp_settings precedent. The worker reads those preferences before it publishes, so the toggles change behaviour rather than only the UI. Verified end to end: with program_added switched off, a newly appearing program is recorded as an event and no notification is sent. An unknown change kind or interval is refused with the offending value named, and an empty kind list means no alerts rather than all of them — the same fail-closed rule the vulnerability selection axes use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu |
||
|
Yogesh OjhaandClaude Opus 5
|
d506f8be66 |
feat(bounty-hub): record what changed in a program and alert on it
The first person to see new scope wins, so Bounty Hub now watches for change rather than only listing what is there today. Every sync diffs against what was already stored and writes bounty_events: a program appearing, scope added or removed, an asset flipping in or out of scope, submissions reopening, a VDP starting to pay. An Updates tab reads that feed and the alerting kinds publish through the existing notification layer, so in-app and webhook channels both fire with no new delivery path. The rule that makes it trustworthy is the one used everywhere else: a first observation is not a change. A program read for the first time records a baseline and emits nothing, and a library synced for the first time does not announce 630 new programs. Verified against the live API — a re-sync with nothing changed upstream produces zero events. Sync is scheduled and the interval belongs to the user: off, daily or weekly, chosen in the header. Beat ticks hourly and the task decides whether the interval is due; Refresh from HackerOne passes force and always runs, the same rule the threat-intel feeds follow. Off means nothing leaves the box on a timer, and says so. Also fixes the same paginate() trap twice hit in this feature: a derived field on a …Read model has to carry a default, because paginate coerces the ORM row before the service fills it in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu |
||
|
Yogesh OjhaandClaude Opus 5
|
5bf2d0395c |
feat(vulnerabilities): rank findings by exploitation, re-scored nightly without rescanning
Severity alone is close to useless for ordering work: roughly 60% of CVEs
score 7.0 or above. Until now EPSS and KEV came from nuclei's template
metadata, frozen at authoring time, so a finding could never learn that the
world had changed around it.
Two keyless feeds now land nightly and re-score every finding already stored:
EPSS (368k CVEs) and the CISA KEV catalog (1,695). A feed that fails to
download leaves its table untouched, so an install with no egress keeps
ranking. CISA is authoritative for is_kev on anything carrying a CVE.
exploitation.py judges each finding the way interest/ judges hosts, and the
signals that matter cross dimensions no other tool joins:
ransom_path a KEV ransomware CVE on a host whose ports also expose
a service ransomware crews reach for
fresh_exploit a public exploit published after the scan that found it
untestable no scanner template covers this CVE, so no scan can
clear it -- BlueKeep, EternalBlue and HTTP/2 Rapid Reset
all report false here
reachable answering directly, with no CDN or WAF in front
vulnx is an optional ceiling, never the floor: per-CVE exploits and their
links, template coverage, internet-wide population and bounty activity,
cached forever. Measured at 10 requests a minute keyless with no Retry-After,
so the client paces off the remaining header and can only ever be a
background cache-fill.
Alerting is delta-only, on the same rule as new findings: intel_signals is
upserted so created_at is when a finding first earned a signal, and repeated
refreshes alert nothing.
The nightly download is opt-out (instance_settings.threat_intel_auto_sync).
It gates the beat task and the worker warm-up only -- Refresh now always
works, so nobody is stuck with a scheduled outbound request they did not ask
for. Bounty activity renders only under CAP_HACKERONE; the gate is
presentational, so switching mode never loses data.
Every count is a promise: coverage, the query it compiles to, and the rows
its sheet opens were verified equal across all five signals.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PwxTAkPFUXPHPvdYT8Rhtm
|
||
|
Yogesh OjhaandClaude Opus 5
|
0bb32328fa |
feat(mcp): agents can interrogate an attack surface, not just start one
Recon MCP servers expose one verb: go and scan. reconFTW v5 ships seven tools and not one of them reads a result, because a pipeline that writes files has nothing to query. reNgine's value is the correlated corpus, so this server is read-first — fourteen tools for asking questions, of which four can act and only two reach a target. Wrapping the OpenAPI document instead would have cost 276 tools and 112,600 tokens of definitions, and would still not work, because every result endpoint needs a scan id an agent has no way to choose. Everything lives in mcp/, mounted into the api the way stages/ and reports/ are. The protocol is implemented directly rather than through an SDK: a tools-only server is a small JSON-RPC surface, this keeps the package free of dependencies, and protocol.py is the only module that knows the wire format. Tools are plug-and-play like stages and report sections — one file under mcp/tools/, a Tool subclass whose Input model is both the schema the model plans against and the schema the server validates with. Nothing registers it. mcp/dimensions.py is the second axis, one entry per result dimension, so query_assets, group_assets, surface_brief and describe_query_language each stay short and a sixth dimension is a single entry. ToolResult carries the product's contracts structurally. Every count comes back with the query token that reproduces it, so a number an agent reports equals the rows a human lands on. A dimension nobody has scanned is reported as not scanned rather than as zero, which is why resolve_target resolves the scan that actually covered each dimension instead of taking the newest one. Rows carrying text the scanned party wrote are flagged untrusted, so the model reports them rather than following them. Access is an opaque service token, hashed at rest, bound to a project and a capability set, revocable by row. Read, plan, write and launch are separate; launch is the only one that reaches a machine you do not own and is off at the instance ceiling until an operator raises it. No token may exceed the ceiling, lowering it narrows the tokens that already exist, and tools/list returns only what a token may call. Start and stop live on the MCP page rather than in a config file — the server is mounted in the api, so stopped means refusing connections rather than a supervised daemon, and the dialog says so: stopping does not cancel a scan an agent already started. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J6fcWDtvxTer9chPWAwKd2 |
||
|
Yogesh OjhaandClaude Opus 5
|
0522444245 |
feat(reports): white-labelling, and a renderer that cannot be made to fetch
Uploaded typefaces and instance-wide branding defaults, which together make a report genuinely white-labellable: a client's own font, a logo set once, and a theme that names both. The security half came first, because the logo field was a live SSRF. It was a free-text string rendered into an <img src>, and WeasyPrint would have fetched whatever it pointed at from inside the worker. Nothing fetches now, at four layers. The load-bearing one is a WeasyPrint url_fetcher that allows data: URIs and files under three known roots and refuses everything else; it was checked against cloud metadata, internal https, loopback, /etc/passwd, a UNC host, ftp and protocol-relative references. Above it sit three input guards: a logo must be embedded data of a known image type, an uploaded theme may not use @import or a url() that is not data:, and authored markdown loses any image that is not embedded. A new feature that lets a report name a resource goes through the fetcher, not around it. A typeface is bytes, never a link. The family row records files named by us, so a hostile filename never reaches disk, and the family directory refuses anything resolving outside itself. A face is accepted only when the bytes agree: the signature is sniffed and then fontTools opens it and requires outlines, which is why fonttools is now an api dependency rather than only a WeasyPrint one. A 5-byte file with a valid TTF header and a 4 KB file of padding are both refused. The family name is banned from carrying quotes or braces because it is written into a quoted CSS font-family string. The standalone HTML export now embeds only the families the theme actually names, which halves it, and no longer carries file:// references that would have been dead outside the container. Instance defaults fill only what a template leaves blank, so setting a logo once never overrides a per-client deliverable that set its own. Verified: a real TTF family uploaded and embedded in a 53-page PDF under an uploaded theme with an uploaded logo, and every probe above refused. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017f2Mf2HGwFaTRnui4UxpRd |
||
|
Yogesh Ojha
|
69239333bc |
Refactor validation and DNS tools, implementation of onboarding wizard
- Added `clean_name` function to `validation.py` for name validation with max length. - Simplified docstrings in `dnsx` client, parser, and service files for clarity. - Enhanced error handling in `ViewDNSService` to prevent disabling keys on rate limits. - Improved WHOIS service to handle concurrent inserts gracefully with integrity checks. - Updated `Settings` class in worker config to inherit from `BaseAppSettings`. - Added new dependencies in `pyproject.toml` for `apprise` and `cryptography`. |