WEBDIENSTE/rengine
1
0
Fork 0
mirror of https://github.com/yogeshojha/rengine.git synced 2026-10-20 07:11:28 +02:00
Code Issues Packages Projects Releases Wiki Activity
4,248 Commits 11 Branches 28 Tags
fe5bc8094ccf3eae39ed2edc78fcfc7e457af192
Commit Graph
7 Commits
Author SHA1 Message Date
Yogesh OjhaandClaude Opus 5 fe5bc8094c feat(bounty-hub): Bugcrowd, Intigriti and YesWeHack from the public feed 
Bounty Hub was empty until someone pasted a HackerOne token. It now ships with
464 public programs and 6,889 scope assets from arkadiyt/bounty-targets-data
(MIT, republished every 30 minutes), so a fresh install is useful with no
credentials at all and the token becomes an upgrade rather than a prerequisite.

HackerOne is deliberately not read from the feed. The API gives us private
programs and richer scope, so the feed would be a downgrade — and a new
`source` column keeps that from happening by accident: an API row is
authoritative and the feed never overwrites one.

The three platforms do not agree on anything, so each gets its own entry in
shared/definitions/bounty_feed.py: the asset field is `target` on two of them
and `endpoint` on Intigriti; the type vocabularies are `website`/`network`
versus `url`/`wildcard`/`iprange` versus `web-application`/`mobile-application-ios`;
and only Intigriti and YesWeHack ship a handle, so Bugcrowd's is derived from
its engagement URL. All of it lands on the 17 asset types reNgine already has,
and target_for_scope needed no change to resolve 5,072 of them.

What the platforms do not report is not invented. Bugcrowd publishes no
submission state, so it stores `unknown` and the row shows no badge rather
than guessing "open". What they report and HackerOne does not — the actual
payout range — becomes a first-class field, a badge and a sort.

Provenance is visible rather than implied: every row names its platform, the
detail sheet says which source it came from and what that source can and
cannot contain, and Platform and Source are faceted filters.

The feed has its own schedule because it costs four small files rather than
640 requests: off, every 6 hours, daily or weekly, beside the HackerOne
interval in Settings. The change-detection layer needed nothing — it was
already platform-agnostic, so scope alerts work on all four platforms, and a
first feed load of 464 programs correctly announced none of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
 2026-09-07 16:22:05 +05:30
Yogesh OjhaandClaude Opus 5 992770478a feat(bounty-hub): a Settings tab for the sync schedule and change alerts 
The interval belonged in Settings, not as a dropdown in a page header. There
is now a Bounty Hub tab under Settings, gated on CAP_BOUNTY_PROGRAMS the same
way the nav item is, holding two cards.

Sync says how often reNgine asks HackerOne what changed — manual only, daily
or weekly — and states the cost so the choice is informed: a full sync is
about 640 requests against a 600-per-minute limit. Manual only says plainly
that nothing leaves the instance on a timer.

Notifications is a master switch plus one toggle per change kind, each with
the sentence describing what it means. Turning a kind off does not stop the
change being recorded: it still appears in the Updates tab, only the alert is
suppressed, and the card says so. Preferences live in instance_settings
.bounty_settings, following the mcp_settings precedent.

The worker reads those preferences before it publishes, so the toggles change
behaviour rather than only the UI. Verified end to end: with program_added
switched off, a newly appearing program is recorded as an event and no
notification is sent.

An unknown change kind or interval is refused with the offending value named,
and an empty kind list means no alerts rather than all of them — the same
fail-closed rule the vulnerability selection axes use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
 2026-09-07 15:01:34 +05:30
Yogesh OjhaandClaude Opus 5 d506f8be66 feat(bounty-hub): record what changed in a program and alert on it 
The first person to see new scope wins, so Bounty Hub now watches for change
rather than only listing what is there today.

Every sync diffs against what was already stored and writes bounty_events:
a program appearing, scope added or removed, an asset flipping in or out of
scope, submissions reopening, a VDP starting to pay. An Updates tab reads that
feed and the alerting kinds publish through the existing notification layer,
so in-app and webhook channels both fire with no new delivery path.

The rule that makes it trustworthy is the one used everywhere else: a first
observation is not a change. A program read for the first time records a
baseline and emits nothing, and a library synced for the first time does not
announce 630 new programs. Verified against the live API — a re-sync with
nothing changed upstream produces zero events.

Sync is scheduled and the interval belongs to the user: off, daily or weekly,
chosen in the header. Beat ticks hourly and the task decides whether the
interval is due; Refresh from HackerOne passes force and always runs, the same
rule the threat-intel feeds follow. Off means nothing leaves the box on a
timer, and says so.

Also fixes the same paginate() trap twice hit in this feature: a derived field
on a …Read model has to carry a default, because paginate coerces the ORM row
before the service fills it in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
 2026-09-07 14:44:18 +05:30
Yogesh OjhaandClaude Opus 5 5bf2d0395c feat(vulnerabilities): rank findings by exploitation, re-scored nightly without rescanning 
Severity alone is close to useless for ordering work: roughly 60% of CVEs
score 7.0 or above. Until now EPSS and KEV came from nuclei's template
metadata, frozen at authoring time, so a finding could never learn that the
world had changed around it.

Two keyless feeds now land nightly and re-score every finding already stored:
EPSS (368k CVEs) and the CISA KEV catalog (1,695). A feed that fails to
download leaves its table untouched, so an install with no egress keeps
ranking. CISA is authoritative for is_kev on anything carrying a CVE.

exploitation.py judges each finding the way interest/ judges hosts, and the
signals that matter cross dimensions no other tool joins:

  ransom_path     a KEV ransomware CVE on a host whose ports also expose
                  a service ransomware crews reach for
  fresh_exploit   a public exploit published after the scan that found it
  untestable      no scanner template covers this CVE, so no scan can
                  clear it -- BlueKeep, EternalBlue and HTTP/2 Rapid Reset
                  all report false here
  reachable       answering directly, with no CDN or WAF in front

vulnx is an optional ceiling, never the floor: per-CVE exploits and their
links, template coverage, internet-wide population and bounty activity,
cached forever. Measured at 10 requests a minute keyless with no Retry-After,
so the client paces off the remaining header and can only ever be a
background cache-fill.

Alerting is delta-only, on the same rule as new findings: intel_signals is
upserted so created_at is when a finding first earned a signal, and repeated
refreshes alert nothing.

The nightly download is opt-out (instance_settings.threat_intel_auto_sync).
It gates the beat task and the worker warm-up only -- Refresh now always
works, so nobody is stuck with a scheduled outbound request they did not ask
for. Bounty activity renders only under CAP_HACKERONE; the gate is
presentational, so switching mode never loses data.

Every count is a promise: coverage, the query it compiles to, and the rows
its sheet opens were verified equal across all five signals.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PwxTAkPFUXPHPvdYT8Rhtm
 2026-09-07 11:18:04 +05:30
Yogesh OjhaandClaude Opus 5 0bb32328fa feat(mcp): agents can interrogate an attack surface, not just start one 
Recon MCP servers expose one verb: go and scan. reconFTW v5 ships seven tools
and not one of them reads a result, because a pipeline that writes files has
nothing to query. reNgine's value is the correlated corpus, so this server is
read-first — fourteen tools for asking questions, of which four can act and
only two reach a target. Wrapping the OpenAPI document instead would have cost
276 tools and 112,600 tokens of definitions, and would still not work, because
every result endpoint needs a scan id an agent has no way to choose.

Everything lives in mcp/, mounted into the api the way stages/ and reports/
are. The protocol is implemented directly rather than through an SDK: a
tools-only server is a small JSON-RPC surface, this keeps the package free of
dependencies, and protocol.py is the only module that knows the wire format.
Tools are plug-and-play like stages and report sections — one file under
mcp/tools/, a Tool subclass whose Input model is both the schema the model
plans against and the schema the server validates with. Nothing registers it.
mcp/dimensions.py is the second axis, one entry per result dimension, so
query_assets, group_assets, surface_brief and describe_query_language each stay
short and a sixth dimension is a single entry.

ToolResult carries the product's contracts structurally. Every count comes back
with the query token that reproduces it, so a number an agent reports equals
the rows a human lands on. A dimension nobody has scanned is reported as not
scanned rather than as zero, which is why resolve_target resolves the scan that
actually covered each dimension instead of taking the newest one. Rows carrying
text the scanned party wrote are flagged untrusted, so the model reports them
rather than following them.

Access is an opaque service token, hashed at rest, bound to a project and a
capability set, revocable by row. Read, plan, write and launch are separate;
launch is the only one that reaches a machine you do not own and is off at the
instance ceiling until an operator raises it. No token may exceed the ceiling,
lowering it narrows the tokens that already exist, and tools/list returns only
what a token may call. Start and stop live on the MCP page rather than in a
config file — the server is mounted in the api, so stopped means refusing
connections rather than a supervised daemon, and the dialog says so: stopping
does not cancel a scan an agent already started.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J6fcWDtvxTer9chPWAwKd2
 2026-09-06 18:24:35 +05:30
Yogesh OjhaandClaude Opus 5 0522444245 feat(reports): white-labelling, and a renderer that cannot be made to fetch 
Uploaded typefaces and instance-wide branding defaults, which together make a
report genuinely white-labellable: a client's own font, a logo set once, and a
theme that names both.

The security half came first, because the logo field was a live SSRF. It was a
free-text string rendered into an <img src>, and WeasyPrint would have fetched
whatever it pointed at from inside the worker. Nothing fetches now, at four
layers. The load-bearing one is a WeasyPrint url_fetcher that allows data: URIs
and files under three known roots and refuses everything else; it was checked
against cloud metadata, internal https, loopback, /etc/passwd, a UNC host, ftp
and protocol-relative references. Above it sit three input guards: a logo must
be embedded data of a known image type, an uploaded theme may not use @import or
a url() that is not data:, and authored markdown loses any image that is not
embedded. A new feature that lets a report name a resource goes through the
fetcher, not around it.

A typeface is bytes, never a link. The family row records files named by us, so
a hostile filename never reaches disk, and the family directory refuses anything
resolving outside itself. A face is accepted only when the bytes agree: the
signature is sniffed and then fontTools opens it and requires outlines, which is
why fonttools is now an api dependency rather than only a WeasyPrint one. A
5-byte file with a valid TTF header and a 4 KB file of padding are both refused.
The family name is banned from carrying quotes or braces because it is written
into a quoted CSS font-family string.

The standalone HTML export now embeds only the families the theme actually
names, which halves it, and no longer carries file:// references that would have
been dead outside the container.

Instance defaults fill only what a template leaves blank, so setting a logo once
never overrides a per-client deliverable that set its own.

Verified: a real TTF family uploaded and embedded in a 53-page PDF under an
uploaded theme with an uploaded logo, and every probe above refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f2Mf2HGwFaTRnui4UxpRd
 2026-09-06 12:53:18 +05:30
Yogesh Ojha 69239333bc Refactor validation and DNS tools, implementation of onboarding wizard 
- Added `clean_name` function to `validation.py` for name validation with max length.
- Simplified docstrings in `dnsx` client, parser, and service files for clarity.
- Enhanced error handling in `ViewDNSService` to prevent disabling keys on rate limits.
- Improved WHOIS service to handle concurrent inserts gracefully with integrity checks.
- Updated `Settings` class in worker config to inherit from `BaseAppSettings`.
- Added new dependencies in `pyproject.toml` for `apprise` and `cryptography`.
 2026-06-21 21:56:20 +05:30