WEBDIENSTE/rengine
1
0
Fork 0
mirror of https://github.com/yogeshojha/rengine.git synced 2026-10-20 23:31:31 +02:00
Code Issues Packages Projects Releases Wiki Activity
4,544 Commits 11 Branches 28 Tags
a548afffa2e43ccc0bc7bbe7c2bb09dd424711b2
Commit Graph
10 Commits
Author SHA1 Message Date
Yogesh OjhaandClaude Opus 5.5 682a177e9b feat(bounty-hub): HackerOne reports and earnings with program cross-links 
- sync the account's reports, bounties and standing from the HackerOne
  hacker API into bounty_reports, bounty_awards and bounty_accounts;
  money comes from /payments/earnings, totals stay per currency
- /bounty-hub/<platform> as a Bounty Hub child in the sidebar, in the
  /scans idiom: stat strip, severity toggles, monthly chart with drag
  range, stage tabs with filtered counts, URL filters, expandable rows,
  keyboard navigation and a by-program view
- program names open the Bounty Hub program sheet, which links back to
  the reports on that program and fetches unfetched scope on open
- hourly report sync gated on bug bounty mode and the sync interval

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-24 17:51:30 +05:30
Yogesh OjhaandClaude Opus 5 0b853e04b6 chore: production-readiness review across the tree 
A full static review of the 3.0 tree, run as two fan-out passes with every
finding verified against the running stack rather than taken on inspection.

Correctness and security
- instance settings encrypted the mask over the stored AI key: `!= MASK` never
  matched `••••••••abcd`, so saving settings destroyed the key and `test_ai`
  posted the mask as a bearer token
- the MFA token was replayable for its full five-minute life; the gate now
  checks revocation and consumes the token after a successful exchange
- logout revoked nothing for a bearer client while answering "Logged out"
- `POST /api-keys/{id}/test` returned the plaintext key inside httpx's error
  string, which embeds the whole URL
- an interest rule could be edited or deleted through another project's id
- redis ran the celery broker under `allkeys-lru`, free to evict a queued task
  or a chord counter; now `noeviction`
- nuclei's stored command carried the OAST token and every auth header
- a finalize step could poison the session and leave a scan RUNNING for ever
- `is:issue` raised TypeError on every search
- offset paging repeated and skipped rows in four dimensions
- a cancelled run kept a stale `scans.*_found` rollup
- every PDF report raised `KeyError: 'software'`
- `version_key` truncated a wide numeric run and inverted release ordering
- psycopg raises a NUL as a bare ValueError, above StatementError, so every
  savepoint guard was blind to the case it was written for

Performance
- endpoints.host had no trigram index: 35.9ms seq scan -> 0.389ms
- free text compiled to one OR, not the prescribed UNION ALL: 132ms -> 41ms
- `/http-assets/summary` and `/ips/summary` materialised whole tables
- the cross-link `ip` kind ran an estate query per page and drew nothing

Frontend
- `$effect` loaders that read the state they write, an unbounded request loop
  on a failing endpoint
- error states that rendered as empty states
- 2FA backup codes were unusable while the copy said they worked
- a "Scan history" dialog that always reported none
- ~400 duplicated lines across five filter bars, now one component

Build and tooling
- every scanner installed at @latest; all 13 pinned to the measured versions
- CI built `web/`, gone since 2.x, with tags that cannot be valid
- the prettier pre-commit hook discarded its filenames and always passed
- the test suite destroyed itself when run twice

Also: subdomains from public code, via a pinned github-subdomains source and a
GitHub API key that subfinder receives too; sublist3r and oneforall removed
after measuring one unique host in 89 across four targets.

Verified: 853 tests, ruff clean over eleven packages, svelte-check 0 errors,
the count-is-a-promise gate at 131 checks and 0 mismatches, and 143 GET routes
with no 5xx. Four migrations, each with an exercised downgrade.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVkdx5ynZ6aof5kLFczmhX
 2026-09-13 18:03:29 +05:30
Yogesh OjhaandClaude Opus 5 dcdba5cfd4 feat(bounty-hub): read Intigriti through its researcher API 
A platform with a researcher API is now a provider: one module under
shared/services/bounty_providers implementing programs() and scopes(),
returning rows the sync engine stores. bounty_programs.py became the
platform-agnostic engine, so adding a platform is one module, one
APIProvider value, one PlatformSpec.api_provider and a key tester.

Intigriti is the second connected platform, on a personal access token
the hunter creates. On this instance: 220 programs, 54 of them private,
155 with scope and 65 the platform refuses, which accounts for all 220.

Measured against the live API, against the docs:

- Tier ids are unordered and undocumented. Id 4 is Tier 1, not No Bounty,
  so the first mapping stored the best-paying assets as paying nothing.
  Eligibility reads the label; out of scope is matched on id and label.
- A 403 carrying FORBID001 is a refusal, not the documented rate-limit
  overage, and it covers public programs the account has not joined.
  Retrying it cost 11s a program; AccessDeniedError ends it in one
  request and records scope_access, keeping the failure budget for
  real breakage.
- Membership is proved by the platform letting the token read a private
  program's scope. confidentialityLevel never reports "invite only", so
  joined was always false and its filter was dead.

Bounty Hub follows what the data actually says:

- source becomes sources, a JSONB array. A program is routinely in both
  a platform API and the public feed; the feed adds itself without
  taking an API row back, and facet counts overlap accordingly.
- The row hint read "Bookmarked on HackerOne" while rendering
  Intigriti's follow flag. It now reads PlatformSpec.follow_label.
- Refresh queued the sync and returned, so nothing changed on screen.
  It waits for the sweep, reloads, and says so.
- The dashed connect card per platform is one dismissible Alert.
- Connecting a platform no longer diffs API scope against feed scope,
  which would report every asset removed and re-added.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SiHx8Sm34hS1nanrsPdBaD
 2026-09-12 23:35:59 +05:30
Yogesh OjhaandClaude Fable 5.1 db030d733c feat(bounty-hub): watch a program 
Scope-true continuous discovery for a bounty program, bug bounty mode only
(capability `program_watches`).

Watching a program imports its in-scope targets under the program's
organization, writes out-of-scope entries as exclusions and the rate ceiling
on a `Watch · <program>` scan context, schedules an optional baseline, and
streams certificate transparency for the watched apexes. A new in-scope host
is resolved, written into the covering scan with source `ct_log`, probed by a
focused run with a screenshot, evaluated against the watch's alert query and
alerted once per fingerprint through the chosen channels.

- ct-stream compose service: certspotter 0.24.2 under a supervisor that
  regenerates the watchlist from active watches, spools events through a
  hook script that emits only certspotter fields, and reports over Redis
- worker tasks: certificate, settle, recheck, reconcile; finalize dispatches
  settle for any probe run; reconcile after every program or feed sync
- notifier: named channels bypass per-channel preferences; attachments;
  new `watch` notification type
- API under /bounty-programs/watches with preview, ledger, events, marks
- Bounty Hub: Watching tab, Watch dialog, watch sheet, program badge,
  certificate stream card in Settings
- schedules carry an intensity; scan_schedules.intensity

Two review rounds folded in: compare-and-set probe claim, resolver-down
handling, spool kept on failed dispatch, schedule cap, mute keeps prior
state, ledger-derived totals, event filters, mode lockout of worker and
stream in corporate mode.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ba2f7gwu2GXiETCKqRztyB
 2026-09-12 21:51:18 +05:30
Yogesh OjhaandClaude Opus 5 2cb8d73280 style: strip explanatory comments across the codebase 
The comment rule in CLAUDE.md is one line maximum, never narration: no
multi-line blocks, no module-header explainers, no sentences that explain or
justify the code to the reader. The tree had drifted a long way from it, and
most of the drift is mine.

Removed:
  232  lines of multi-line comment blocks
  203  single-line comments that argued instead of stating
  154  comments joined by so/because/otherwise/rather-than
  143  narrating comments in the frontend
  112  test docstrings (the test name already says what it asserts)
   25  test module docstrings and 25 decorative section dividers

Collapsed:
  324  multi-line docstrings to one factual line
  271  one-line docstrings that had a justifying clause bolted on

Kept: functional directives, section dividers, short field labels, alembic
revision headers, and the eight FastAPI route docstrings that render in /docs.
Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs.

284 python tests and 34 frontend tests pass, ruff and ruff-format clean across
ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
 2026-09-11 14:11:23 +05:30
Yogesh OjhaandClaude Opus 5 163df52ad5 perf(db): a primary key is already an index, so the second one on id was write cost for nothing 
SQLModel emitted index=True beside primary_key=True on every id field, so 41
tables carried ix_<table>_id alongside <table>_pkey: the same single-column btree
twice. 12 MB of duplicate index, and every insert maintaining both — on the hot
path of a scan writing 345,152 endpoints.

The planner had been using them (72,160 scans across the set) simply because
they were there and equivalent; with them gone it uses the primary key, verified
on subdomains: Index Scan using subdomains_pkey.

The migration finds them rather than naming them, because which tables exist
depends on how far an install has migrated and a name already gone must not fail
an upgrade. It drops only a non-unique, single-column btree on id belonging to a
table that has a primary key — no other index can match that shape. There is no
downgrade: the primary key serves every lookup they served.

The 33 models are corrected too, so the next hand-written migration is not
comparing against a schema that would want them back.

Counted honestly: 41, not the 102 an earlier pass reported — that number included
the 61 primary keys themselves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
 2026-09-10 21:24:43 +05:30
Yogesh OjhaandClaude Opus 5 fe5bc8094c feat(bounty-hub): Bugcrowd, Intigriti and YesWeHack from the public feed 
Bounty Hub was empty until someone pasted a HackerOne token. It now ships with
464 public programs and 6,889 scope assets from arkadiyt/bounty-targets-data
(MIT, republished every 30 minutes), so a fresh install is useful with no
credentials at all and the token becomes an upgrade rather than a prerequisite.

HackerOne is deliberately not read from the feed. The API gives us private
programs and richer scope, so the feed would be a downgrade — and a new
`source` column keeps that from happening by accident: an API row is
authoritative and the feed never overwrites one.

The three platforms do not agree on anything, so each gets its own entry in
shared/definitions/bounty_feed.py: the asset field is `target` on two of them
and `endpoint` on Intigriti; the type vocabularies are `website`/`network`
versus `url`/`wildcard`/`iprange` versus `web-application`/`mobile-application-ios`;
and only Intigriti and YesWeHack ship a handle, so Bugcrowd's is derived from
its engagement URL. All of it lands on the 17 asset types reNgine already has,
and target_for_scope needed no change to resolve 5,072 of them.

What the platforms do not report is not invented. Bugcrowd publishes no
submission state, so it stores `unknown` and the row shows no badge rather
than guessing "open". What they report and HackerOne does not — the actual
payout range — becomes a first-class field, a badge and a sort.

Provenance is visible rather than implied: every row names its platform, the
detail sheet says which source it came from and what that source can and
cannot contain, and Platform and Source are faceted filters.

The feed has its own schedule because it costs four small files rather than
640 requests: off, every 6 hours, daily or weekly, beside the HackerOne
interval in Settings. The change-detection layer needed nothing — it was
already platform-agnostic, so scope alerts work on all four platforms, and a
first feed load of 464 programs correctly announced none of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
 2026-09-07 16:22:05 +05:30
Yogesh OjhaandClaude Opus 5 992770478a feat(bounty-hub): a Settings tab for the sync schedule and change alerts 
The interval belonged in Settings, not as a dropdown in a page header. There
is now a Bounty Hub tab under Settings, gated on CAP_BOUNTY_PROGRAMS the same
way the nav item is, holding two cards.

Sync says how often reNgine asks HackerOne what changed — manual only, daily
or weekly — and states the cost so the choice is informed: a full sync is
about 640 requests against a 600-per-minute limit. Manual only says plainly
that nothing leaves the instance on a timer.

Notifications is a master switch plus one toggle per change kind, each with
the sentence describing what it means. Turning a kind off does not stop the
change being recorded: it still appears in the Updates tab, only the alert is
suppressed, and the card says so. Preferences live in instance_settings
.bounty_settings, following the mcp_settings precedent.

The worker reads those preferences before it publishes, so the toggles change
behaviour rather than only the UI. Verified end to end: with program_added
switched off, a newly appearing program is recorded as an event and no
notification is sent.

An unknown change kind or interval is refused with the offending value named,
and an empty kind list means no alerts rather than all of them — the same
fail-closed rule the vulnerability selection axes use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
 2026-09-07 15:01:34 +05:30
Yogesh OjhaandClaude Opus 5 d506f8be66 feat(bounty-hub): record what changed in a program and alert on it 
The first person to see new scope wins, so Bounty Hub now watches for change
rather than only listing what is there today.

Every sync diffs against what was already stored and writes bounty_events:
a program appearing, scope added or removed, an asset flipping in or out of
scope, submissions reopening, a VDP starting to pay. An Updates tab reads that
feed and the alerting kinds publish through the existing notification layer,
so in-app and webhook channels both fire with no new delivery path.

The rule that makes it trustworthy is the one used everywhere else: a first
observation is not a change. A program read for the first time records a
baseline and emits nothing, and a library synced for the first time does not
announce 630 new programs. Verified against the live API — a re-sync with
nothing changed upstream produces zero events.

Sync is scheduled and the interval belongs to the user: off, daily or weekly,
chosen in the header. Beat ticks hourly and the task decides whether the
interval is due; Refresh from HackerOne passes force and always runs, the same
rule the threat-intel feeds follow. Off means nothing leaves the box on a
timer, and says so.

Also fixes the same paginate() trap twice hit in this feature: a derived field
on a …Read model has to carry a default, because paginate coerces the ORM row
before the service fills it in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
 2026-09-07 14:44:18 +05:30
Yogesh OjhaandClaude Opus 5 09368a76ed feat(bounty-hub): browse HackerOne programs and import their scope as targets 
A new bug-bounty-mode surface at /bounty-hub, gated on CAP_BOUNTY_PROGRAMS —
the capability that already existed and gated nothing.

Programs and their structured scope are stored, not cached, so in-scope and
out-of-scope are real server-side filters rather than a modal accordion.
2.x deliberately discarded out-of-scope entries with a TODO; this lands it.

The vocabulary is one definition (shared/definitions/bounty_programs.py):
all 17 asset types across network/mobile/code/other, with the five reNgine
can reach mapped to a TargetType. 2.x carried two divergent allow-lists and
silently dropped CIDR assets the modal had shown.

Normalisation delegates rather than duplicates: an identifier is cleaned and
then handed to reNgine's own validate_target. The asset type decides whether
to try; validate_target decides whether it works. So *.example.com and
https://*.uber.com become domains, 10.0.0.0/8 stays a range, AS64500 is
recognised out of the free-text OTHER field, and *.example.* resolves to
nothing rather than a broken target.

Import is end to end: selected assets become targets grouped under an
organization named for the program and tagged with the platform's own tag,
both confirmed in a dialog where the name and tags can be edited. Out-of-scope
assets are never selectable without an explicit opt-in.

Measured against the live API (630 programs, 46,109 scope assets):

- state is public_mode or soft_launched; there is no private_mode. Matching
  the literal mislabelled every invite-only program as public.
- last_invitation_accepted_at_for_user separates programs you have joined
  from public ones you have not, so both are filters.
- profile_picture is a presigned URL up to 2,107 chars, so it is Text; the
  varchar(1000) it replaced failed one row and lost all 630.
- a program may publish no structured scope at all (200 with an empty array),
  which is a different fact from "not fetched yet" and reads as such.

Also fixes the API-key Test button, which could never have worked for a
provider needing a username: testers only ever received key_value, so they
now receive key_meta too, and HackerOne verifies through the same client
the sync uses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
 2026-09-07 14:26:31 +05:30