WEBDIENSTE/rengine
1
0
Fork 0
mirror of https://github.com/yogeshojha/rengine.git synced 2026-10-20 15:21:30 +02:00
Code Issues Packages Projects Releases Wiki Activity
4,544 Commits 11 Branches 28 Tags
a548afffa2e43ccc0bc7bbe7c2bb09dd424711b2
Commit Graph
11 Commits
Author SHA1 Message Date
Yogesh OjhaandClaude Opus 5.5 682a177e9b feat(bounty-hub): HackerOne reports and earnings with program cross-links 
- sync the account's reports, bounties and standing from the HackerOne
  hacker API into bounty_reports, bounty_awards and bounty_accounts;
  money comes from /payments/earnings, totals stay per currency
- /bounty-hub/<platform> as a Bounty Hub child in the sidebar, in the
  /scans idiom: stat strip, severity toggles, monthly chart with drag
  range, stage tabs with filtered counts, URL filters, expandable rows,
  keyboard navigation and a by-program view
- program names open the Bounty Hub program sheet, which links back to
  the reports on that program and fetches unfetched scope on open
- hourly report sync gated on bug bounty mode and the sync interval

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-24 17:51:30 +05:30
Yogesh OjhaandClaude Opus 5 0b853e04b6 chore: production-readiness review across the tree 
A full static review of the 3.0 tree, run as two fan-out passes with every
finding verified against the running stack rather than taken on inspection.

Correctness and security
- instance settings encrypted the mask over the stored AI key: `!= MASK` never
  matched `••••••••abcd`, so saving settings destroyed the key and `test_ai`
  posted the mask as a bearer token
- the MFA token was replayable for its full five-minute life; the gate now
  checks revocation and consumes the token after a successful exchange
- logout revoked nothing for a bearer client while answering "Logged out"
- `POST /api-keys/{id}/test` returned the plaintext key inside httpx's error
  string, which embeds the whole URL
- an interest rule could be edited or deleted through another project's id
- redis ran the celery broker under `allkeys-lru`, free to evict a queued task
  or a chord counter; now `noeviction`
- nuclei's stored command carried the OAST token and every auth header
- a finalize step could poison the session and leave a scan RUNNING for ever
- `is:issue` raised TypeError on every search
- offset paging repeated and skipped rows in four dimensions
- a cancelled run kept a stale `scans.*_found` rollup
- every PDF report raised `KeyError: 'software'`
- `version_key` truncated a wide numeric run and inverted release ordering
- psycopg raises a NUL as a bare ValueError, above StatementError, so every
  savepoint guard was blind to the case it was written for

Performance
- endpoints.host had no trigram index: 35.9ms seq scan -> 0.389ms
- free text compiled to one OR, not the prescribed UNION ALL: 132ms -> 41ms
- `/http-assets/summary` and `/ips/summary` materialised whole tables
- the cross-link `ip` kind ran an estate query per page and drew nothing

Frontend
- `$effect` loaders that read the state they write, an unbounded request loop
  on a failing endpoint
- error states that rendered as empty states
- 2FA backup codes were unusable while the copy said they worked
- a "Scan history" dialog that always reported none
- ~400 duplicated lines across five filter bars, now one component

Build and tooling
- every scanner installed at @latest; all 13 pinned to the measured versions
- CI built `web/`, gone since 2.x, with tags that cannot be valid
- the prettier pre-commit hook discarded its filenames and always passed
- the test suite destroyed itself when run twice

Also: subdomains from public code, via a pinned github-subdomains source and a
GitHub API key that subfinder receives too; sublist3r and oneforall removed
after measuring one unique host in 89 across four targets.

Verified: 853 tests, ruff clean over eleven packages, svelte-check 0 errors,
the count-is-a-promise gate at 131 checks and 0 mismatches, and 143 GET routes
with no 5xx. Four migrations, each with an exercised downgrade.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVkdx5ynZ6aof5kLFczmhX
 2026-09-13 18:03:29 +05:30
Yogesh OjhaandClaude Opus 5 dcdba5cfd4 feat(bounty-hub): read Intigriti through its researcher API 
A platform with a researcher API is now a provider: one module under
shared/services/bounty_providers implementing programs() and scopes(),
returning rows the sync engine stores. bounty_programs.py became the
platform-agnostic engine, so adding a platform is one module, one
APIProvider value, one PlatformSpec.api_provider and a key tester.

Intigriti is the second connected platform, on a personal access token
the hunter creates. On this instance: 220 programs, 54 of them private,
155 with scope and 65 the platform refuses, which accounts for all 220.

Measured against the live API, against the docs:

- Tier ids are unordered and undocumented. Id 4 is Tier 1, not No Bounty,
  so the first mapping stored the best-paying assets as paying nothing.
  Eligibility reads the label; out of scope is matched on id and label.
- A 403 carrying FORBID001 is a refusal, not the documented rate-limit
  overage, and it covers public programs the account has not joined.
  Retrying it cost 11s a program; AccessDeniedError ends it in one
  request and records scope_access, keeping the failure budget for
  real breakage.
- Membership is proved by the platform letting the token read a private
  program's scope. confidentialityLevel never reports "invite only", so
  joined was always false and its filter was dead.

Bounty Hub follows what the data actually says:

- source becomes sources, a JSONB array. A program is routinely in both
  a platform API and the public feed; the feed adds itself without
  taking an API row back, and facet counts overlap accordingly.
- The row hint read "Bookmarked on HackerOne" while rendering
  Intigriti's follow flag. It now reads PlatformSpec.follow_label.
- Refresh queued the sync and returned, so nothing changed on screen.
  It waits for the sweep, reloads, and says so.
- The dashed connect card per platform is one dismissible Alert.
- Connecting a platform no longer diffs API scope against feed scope,
  which would report every asset removed and re-added.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SiHx8Sm34hS1nanrsPdBaD
 2026-09-12 23:35:59 +05:30
Yogesh OjhaandClaude Fable 5.1 db030d733c feat(bounty-hub): watch a program 
Scope-true continuous discovery for a bounty program, bug bounty mode only
(capability `program_watches`).

Watching a program imports its in-scope targets under the program's
organization, writes out-of-scope entries as exclusions and the rate ceiling
on a `Watch · <program>` scan context, schedules an optional baseline, and
streams certificate transparency for the watched apexes. A new in-scope host
is resolved, written into the covering scan with source `ct_log`, probed by a
focused run with a screenshot, evaluated against the watch's alert query and
alerted once per fingerprint through the chosen channels.

- ct-stream compose service: certspotter 0.24.2 under a supervisor that
  regenerates the watchlist from active watches, spools events through a
  hook script that emits only certspotter fields, and reports over Redis
- worker tasks: certificate, settle, recheck, reconcile; finalize dispatches
  settle for any probe run; reconcile after every program or feed sync
- notifier: named channels bypass per-channel preferences; attachments;
  new `watch` notification type
- API under /bounty-programs/watches with preview, ledger, events, marks
- Bounty Hub: Watching tab, Watch dialog, watch sheet, program badge,
  certificate stream card in Settings
- schedules carry an intensity; scan_schedules.intensity

Two review rounds folded in: compare-and-set probe claim, resolver-down
handling, spool kept on failed dispatch, schedule cap, mute keeps prior
state, ledger-derived totals, event filters, mode lockout of worker and
stream in corporate mode.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ba2f7gwu2GXiETCKqRztyB
 2026-09-12 21:51:18 +05:30
Yogesh OjhaandClaude Opus 5 2cb8d73280 style: strip explanatory comments across the codebase 
The comment rule in CLAUDE.md is one line maximum, never narration: no
multi-line blocks, no module-header explainers, no sentences that explain or
justify the code to the reader. The tree had drifted a long way from it, and
most of the drift is mine.

Removed:
  232  lines of multi-line comment blocks
  203  single-line comments that argued instead of stating
  154  comments joined by so/because/otherwise/rather-than
  143  narrating comments in the frontend
  112  test docstrings (the test name already says what it asserts)
   25  test module docstrings and 25 decorative section dividers

Collapsed:
  324  multi-line docstrings to one factual line
  271  one-line docstrings that had a justifying clause bolted on

Kept: functional directives, section dividers, short field labels, alembic
revision headers, and the eight FastAPI route docstrings that render in /docs.
Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs.

284 python tests and 34 frontend tests pass, ruff and ruff-format clean across
ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
 2026-09-11 14:11:23 +05:30
Yogesh OjhaandClaude Opus 5 806eedc45d fix(bounty-hub): eleven issues from a second review 
The search box cleared itself on every keystroke. The effect that syncs the
input to the applied filter both read and wrote `draft`, so it was its own
dependency: typing a character reset the field, and only the last character of
each debounce window ever reached the query. The read is untracked now, and
typing "adobe" keeps "adobe" and returns the 2 rows the API returns.

Closing a deep-linked program reopened it. The guard was cleared synchronously
while goto() dropped ?program= asynchronously, so the effect re-fired with the
param still present. deepLinked is now cleared only once the param has actually
left the URL. Verified: the sheet stays closed and the query string empties.

sync_lock could mask the real exception and strand the advisory lock. Its
finally ran a statement on a possibly-poisoned transaction, which raises
PendingRollbackError over the original error and skips the unlock — and because
the engine is pooled, a session-level lock outlives session.close() and would
make every later sync answer "already_running" with nothing logged. It rolls
back first now; a deliberately poisoned transaction preserves its DataError and
the next run reacquires the lock.

target_value is varchar(500) and nothing capped what was written to it, so a
program listing a longer URL would abort a sync the way the profile picture and
the event detail already did. A resolved value that could never be stored — or
imported, since targets.target_value is also 500 — is no longer offered.

The per-program refresh took no lock while doing exactly the work the lock was
added for: on a feed platform it redownloads the file and rewrites every
program's scope. It now takes the same lock, and the endpoint is superuser like
its two siblings rather than letting any user drive that repeatedly.

Also: the program sheet said HackerOne in five places and linked feed programs
to a hackerone.com page that does not exist; formatPayout still announced a
minimum as "up to" and _organization_for still rolled back instead of using a
savepoint, both because an earlier unasserted string replace silently did not
apply; scope_ids=[] was read as "every scope" rather than none; and the feed
hardcoded "six_hours" and the "program_added" literal instead of their
definitions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
 2026-09-07 18:25:00 +05:30
Yogesh OjhaandClaude Opus 5 7cb5bcb16a fix(bounty-hub): twelve issues from an end-to-end review 
Two would have broken the feature outright.

A scope-change event carried the asset's instruction into bounty_events.detail,
which is varchar(500) while the instruction is capped at 4000. 334 stored scope
rows already exceed 500 characters, so the first time any of those programs
changed scope the insert would have raised 22001 and aborted the whole sync.
Detail is now cut to the column width — the same class of bug as the presigned
profile picture, and the same lesson: every column width is a cap something
upstream has to apply.

The public feed was unreachable without HackerOne credentials. status.configured
gated the entire page and the Settings cards, so an instance with no token saw a
credential prompt instead of the 464 programs the beat had already fetched —
which defeats the reason the feed was added. The prompt is now a banner over the
library, refresh always works, and without a token it refreshes the feed alone.

The rest: the Updates tab was silently filtered to HackerOne, so feed events
never appeared while the badge counted them; "Refresh scope" was a no-op on
every non-HackerOne program and could resync the wrong one on a handle
collision; every import was tagged hackerone and described as a HackerOne
program whatever the platform; formatPayout announced a minimum as "up to";
a program name over 100 characters made an import 500; skipped listed every
unreachable asset in the program rather than the ones selected; a slug race
rolled back the caller's uncommitted targets instead of a savepoint; the feed
stamped its schedule even when every download failed, suppressing the retry for
a full interval; and the search fired a paginated query with aggregates on
every keystroke.

The twelfth is one no static review could see. A no-change yeswehack pass
measured 0.5s alone and 120.5s against a concurrent sync — two runs deleting the
same scope rows block each other, and the hourly beat can fire while a manual
refresh is going. Both sync tasks now take a session-level advisory lock and
decline in 0.01s rather than queueing behind one another.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
 2026-09-07 16:55:17 +05:30
Yogesh OjhaandClaude Opus 5 fe5bc8094c feat(bounty-hub): Bugcrowd, Intigriti and YesWeHack from the public feed 
Bounty Hub was empty until someone pasted a HackerOne token. It now ships with
464 public programs and 6,889 scope assets from arkadiyt/bounty-targets-data
(MIT, republished every 30 minutes), so a fresh install is useful with no
credentials at all and the token becomes an upgrade rather than a prerequisite.

HackerOne is deliberately not read from the feed. The API gives us private
programs and richer scope, so the feed would be a downgrade — and a new
`source` column keeps that from happening by accident: an API row is
authoritative and the feed never overwrites one.

The three platforms do not agree on anything, so each gets its own entry in
shared/definitions/bounty_feed.py: the asset field is `target` on two of them
and `endpoint` on Intigriti; the type vocabularies are `website`/`network`
versus `url`/`wildcard`/`iprange` versus `web-application`/`mobile-application-ios`;
and only Intigriti and YesWeHack ship a handle, so Bugcrowd's is derived from
its engagement URL. All of it lands on the 17 asset types reNgine already has,
and target_for_scope needed no change to resolve 5,072 of them.

What the platforms do not report is not invented. Bugcrowd publishes no
submission state, so it stores `unknown` and the row shows no badge rather
than guessing "open". What they report and HackerOne does not — the actual
payout range — becomes a first-class field, a badge and a sort.

Provenance is visible rather than implied: every row names its platform, the
detail sheet says which source it came from and what that source can and
cannot contain, and Platform and Source are faceted filters.

The feed has its own schedule because it costs four small files rather than
640 requests: off, every 6 hours, daily or weekly, beside the HackerOne
interval in Settings. The change-detection layer needed nothing — it was
already platform-agnostic, so scope alerts work on all four platforms, and a
first feed load of 464 programs correctly announced none of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
 2026-09-07 16:22:05 +05:30
Yogesh OjhaandClaude Opus 5 992770478a feat(bounty-hub): a Settings tab for the sync schedule and change alerts 
The interval belonged in Settings, not as a dropdown in a page header. There
is now a Bounty Hub tab under Settings, gated on CAP_BOUNTY_PROGRAMS the same
way the nav item is, holding two cards.

Sync says how often reNgine asks HackerOne what changed — manual only, daily
or weekly — and states the cost so the choice is informed: a full sync is
about 640 requests against a 600-per-minute limit. Manual only says plainly
that nothing leaves the instance on a timer.

Notifications is a master switch plus one toggle per change kind, each with
the sentence describing what it means. Turning a kind off does not stop the
change being recorded: it still appears in the Updates tab, only the alert is
suppressed, and the card says so. Preferences live in instance_settings
.bounty_settings, following the mcp_settings precedent.

The worker reads those preferences before it publishes, so the toggles change
behaviour rather than only the UI. Verified end to end: with program_added
switched off, a newly appearing program is recorded as an event and no
notification is sent.

An unknown change kind or interval is refused with the offending value named,
and an empty kind list means no alerts rather than all of them — the same
fail-closed rule the vulnerability selection axes use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
 2026-09-07 15:01:34 +05:30
Yogesh OjhaandClaude Opus 5 d506f8be66 feat(bounty-hub): record what changed in a program and alert on it 
The first person to see new scope wins, so Bounty Hub now watches for change
rather than only listing what is there today.

Every sync diffs against what was already stored and writes bounty_events:
a program appearing, scope added or removed, an asset flipping in or out of
scope, submissions reopening, a VDP starting to pay. An Updates tab reads that
feed and the alerting kinds publish through the existing notification layer,
so in-app and webhook channels both fire with no new delivery path.

The rule that makes it trustworthy is the one used everywhere else: a first
observation is not a change. A program read for the first time records a
baseline and emits nothing, and a library synced for the first time does not
announce 630 new programs. Verified against the live API — a re-sync with
nothing changed upstream produces zero events.

Sync is scheduled and the interval belongs to the user: off, daily or weekly,
chosen in the header. Beat ticks hourly and the task decides whether the
interval is due; Refresh from HackerOne passes force and always runs, the same
rule the threat-intel feeds follow. Off means nothing leaves the box on a
timer, and says so.

Also fixes the same paginate() trap twice hit in this feature: a derived field
on a …Read model has to carry a default, because paginate coerces the ORM row
before the service fills it in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
 2026-09-07 14:44:18 +05:30
Yogesh OjhaandClaude Opus 5 09368a76ed feat(bounty-hub): browse HackerOne programs and import their scope as targets 
A new bug-bounty-mode surface at /bounty-hub, gated on CAP_BOUNTY_PROGRAMS —
the capability that already existed and gated nothing.

Programs and their structured scope are stored, not cached, so in-scope and
out-of-scope are real server-side filters rather than a modal accordion.
2.x deliberately discarded out-of-scope entries with a TODO; this lands it.

The vocabulary is one definition (shared/definitions/bounty_programs.py):
all 17 asset types across network/mobile/code/other, with the five reNgine
can reach mapped to a TargetType. 2.x carried two divergent allow-lists and
silently dropped CIDR assets the modal had shown.

Normalisation delegates rather than duplicates: an identifier is cleaned and
then handed to reNgine's own validate_target. The asset type decides whether
to try; validate_target decides whether it works. So *.example.com and
https://*.uber.com become domains, 10.0.0.0/8 stays a range, AS64500 is
recognised out of the free-text OTHER field, and *.example.* resolves to
nothing rather than a broken target.

Import is end to end: selected assets become targets grouped under an
organization named for the program and tagged with the platform's own tag,
both confirmed in a dialog where the name and tags can be edited. Out-of-scope
assets are never selectable without an explicit opt-in.

Measured against the live API (630 programs, 46,109 scope assets):

- state is public_mode or soft_launched; there is no private_mode. Matching
  the literal mislabelled every invite-only program as public.
- last_invitation_accepted_at_for_user separates programs you have joined
  from public ones you have not, so both are filters.
- profile_picture is a presigned URL up to 2,107 chars, so it is Text; the
  varchar(1000) it replaced failed one row and lost all 630.
- a program may publish no structured scope at all (200 with an empty array),
  which is a different fact from "not fetched yet" and reads as such.

Also fixes the API-key Test button, which could never have worked for a
provider needing a username: testers only ever received key_value, so they
now receive key_meta too, and HackerOne verifies through the same client
the sync uses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
 2026-09-07 14:26:31 +05:30