- Store each run's row, first-seen and retired counts (scan_deltas, scan_retired), valid while
the revision stamp matches scan_revisions; statement triggers on the seven result tables bump
it. Finalize warms, a beat backfill fills the rest, a live run answers from its last count and
queues a refresh. /scans 1.5 s -> 43 ms, dashboard overview 5-10 s -> 0.9 s.
- lead_cache holds a live scope for 20 s with a stale copy, and every build is single-flight;
search pages opt out.
- Leads: lowercase subdomain names (CHECK) and LIKE for host:, a guard for empty JSON arrays,
branches as MATERIALIZED CTEs read to the cap, every count in one statement.
- Addresses and services: the derived CTE as one grouped pass, rows and count in one execution
(page_rows), GIN lookups for page hosts, facets through grouping sets.
- Correlation carriers counted by EXISTS; takeover latest-sighting in SQL, no silent cap.
- Screenshot backfill partial index, nine unused low-cardinality indexes dropped, hostname
helpers memoised, vhost names normalised.
Launches past the limit stay pending and start in launch order when a slot
frees, so each running scan gets full speed and memory stays bounded.
- run_scan admits a pending scan under an advisory lock: running plus waiting
scans ahead of it must be under the limit
- finalize, cancel, pause, delete and a limit change start the next scans;
the reaper re-sends only scans whose turn has come
- automatic limit: live worker-scans stage slots over the widest step,
so --scale raises it; Settings sets a fixed limit from 1 to 50
- watch probes skip the queue, resume is never held
- queued scans show how many are ahead
Stage tasks hold a slot for their whole tool run, so step hand-overs and
finalize queued behind them and background jobs took scan slots.
- worker-scans consumes scans (run_scan_stage only), scalable with --scale
- worker-control consumes scan_control: launch, hand-over, resume, finalize
- worker-default consumes default and critical
- reaper reads both scan queues; readiness requires a consumer on every queue
- connection budget counts every worker; max_connections 250
- sync the account's reports, bounties and standing from the HackerOne
hacker API into bounty_reports, bounty_awards and bounty_accounts;
money comes from /payments/earnings, totals stay per currency
- /bounty-hub/<platform> as a Bounty Hub child in the sidebar, in the
/scans idiom: stat strip, severity toggles, monthly chart with drag
range, stage tabs with filtered counts, URL filters, expandable rows,
keyboard navigation and a by-program view
- program names open the Bounty Hub program sheet, which links back to
the reports on that program and fetches unfetched scope on open
- hourly report sync gated on bug bounty mode and the sync interval
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A target can follow the check library: once a day, when the library gains
checks, one focused run tests every web asset the target's completed scans
found, with those checks only. Per-target watermark, one sweep lock, a
`daily.run` chain that registers future once-a-day jobs, two notifications,
a What's new group, a "New" badge on the library, a monitored-target signal
and a "New checks runs" scan filter.
The pipeline review behind it, in three passes:
Planner and scanner. The deep sweep is its own tier, outside ROOT_TIERS and
after replay, so a cut costs the tail and cluster members are still
confirmed; the one-request tier runs first, sized on its clustered cost;
blind core ranks CISA then VulnCheck then severity then cost; the planned
tiers come from the selection; skipped hosts are marked alone; library gaps
are one row per run; the sweep cap is 8 requests.
Output. Payload-free fingerprints for fuzzing, out-of-band and dalfox
findings; suppression follows the replay edge; the digest baseline is every
earlier scan; host and port read through split_host_port; interaction obeys
store_evidence and redaction; the rollup excludes replays.
Tools. nuclei's exit code, abort and budget are read; a tool arg cannot
restate a stage flag under its short spelling; one refused row costs one
row; a headless block decides the protocol.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B58eEM68a5qDFTQXPE5aoW
A full static review of the 3.0 tree, run as two fan-out passes with every
finding verified against the running stack rather than taken on inspection.
Correctness and security
- instance settings encrypted the mask over the stored AI key: `!= MASK` never
matched `••••••••abcd`, so saving settings destroyed the key and `test_ai`
posted the mask as a bearer token
- the MFA token was replayable for its full five-minute life; the gate now
checks revocation and consumes the token after a successful exchange
- logout revoked nothing for a bearer client while answering "Logged out"
- `POST /api-keys/{id}/test` returned the plaintext key inside httpx's error
string, which embeds the whole URL
- an interest rule could be edited or deleted through another project's id
- redis ran the celery broker under `allkeys-lru`, free to evict a queued task
or a chord counter; now `noeviction`
- nuclei's stored command carried the OAST token and every auth header
- a finalize step could poison the session and leave a scan RUNNING for ever
- `is:issue` raised TypeError on every search
- offset paging repeated and skipped rows in four dimensions
- a cancelled run kept a stale `scans.*_found` rollup
- every PDF report raised `KeyError: 'software'`
- `version_key` truncated a wide numeric run and inverted release ordering
- psycopg raises a NUL as a bare ValueError, above StatementError, so every
savepoint guard was blind to the case it was written for
Performance
- endpoints.host had no trigram index: 35.9ms seq scan -> 0.389ms
- free text compiled to one OR, not the prescribed UNION ALL: 132ms -> 41ms
- `/http-assets/summary` and `/ips/summary` materialised whole tables
- the cross-link `ip` kind ran an estate query per page and drew nothing
Frontend
- `$effect` loaders that read the state they write, an unbounded request loop
on a failing endpoint
- error states that rendered as empty states
- 2FA backup codes were unusable while the copy said they worked
- a "Scan history" dialog that always reported none
- ~400 duplicated lines across five filter bars, now one component
Build and tooling
- every scanner installed at @latest; all 13 pinned to the measured versions
- CI built `web/`, gone since 2.x, with tags that cannot be valid
- the prettier pre-commit hook discarded its filenames and always passed
- the test suite destroyed itself when run twice
Also: subdomains from public code, via a pinned github-subdomains source and a
GitHub API key that subfinder receives too; sublist3r and oneforall removed
after measuring one unique host in 89 across four targets.
Verified: 853 tests, ruff clean over eleven packages, svelte-check 0 errors,
the count-is-a-promise gate at 131 checks and 0 mismatches, and 143 GET routes
with no 5xx. Four migrations, each with an exercised downgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVkdx5ynZ6aof5kLFczmhX
run_scan took the scan row FOR UPDATE, wrote one task id, committed, and
only then called apply_async. The commit released the lock, so a
redelivered launch landing inside that window saw a single task id, passed
the "already dispatched" guard — which needs two — and queued a second
canvas for the same scan. Two subdomain_discovery processes on one target
is the failure the run_epoch work was added to stop, reached by a
different route.
The dispatch now happens before the single commit, so a duplicate blocks
on the row and then sees the finished claim. A crash between apply_async
and the commit still leaves a queued canvas with the row unclaimed, the
same as before; reap_stalled remains the net for that.
dispatch_scan_run also takes the epoch rather than letting run_scan
default it to 0. It was correct only because a fresh scan's run_epoch
happens to be 0, which nothing stated or enforced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5roHYDYzSPRNaTkn8hSkJ
POST /exports saves what the table was showing — dimension, scope, the
filter body, the format — and queues a run. The row is the recipe, so
rerun runs it again against today's rows rather than handing back
yesterday's file, which is the point of keeping it.
The scope resolves when the run starts, not when it was saved: a scan
export reads that scan, a target or project export re-picks the newest
covering scan of each target, so a saved export follows the estate.
That needed a sync resolver, since surface_scope is async and the api is
not on the worker's path.
A bundle writes one file per dimension into a zip. Expiry is its own
status rather than a failure wearing the wrong label, and a reaper fails
a run the worker lost, which reports still lack.
Streaming takes a connection of its own. yield_per opens a server side
cursor, and the progress commit — or the per chunk join — executed on
the session would invalidate it mid-file; the first large export failed
with "named cursor isn't valid anymore" and nothing smaller ever would
have. The export therefore reads committed rows, so its promise test
moves to the durable fixture.
Proven end to end: a filtered CSV that matches the table, 7,784 URLs one
per line, a five file zip, a project wide export across 2,184 addresses,
and a rerun.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LYZfaNjq3Ym8GPvWYuHisS
Pause revoked with terminate=True and SIGKILL. A SIGKILLed celery child
runs no finally and no watchdog thread, so its subprocess.Popen tool was
reparented to init and kept hitting the target with its budget and idle
timer dead — an unbounded scan against a host the operator believes is
paused. Revoke now drops the queued tasks only; a task already running
stops through its own abort check, which kills the tool and reaps it.
tools/runner/abort.py binds the stage's check for the whole process, so a
tool stops even where the call site passed no should_stop. The blocking
run() path has no abort poll yet and still ends at the tool's own
timeout.
Resume was answered by the worker, so the api replied "paused" and the
page waited on a celery round trip — with the worker busy the control
looked dead. The api now takes the row lock, folds the pause, sets
RUNNING and bumps the epoch itself, then dispatches a task that only
builds the canvas. Pause and resume answer in ~20ms with the true status
and the stores hand the row back, so the page flips without a refetch.
run_scan read run_epoch at execution time, so a launch still queued when
a scan was paused and resumed adopted the new epoch and dispatched a
second canvas beside the resumed one: two subdomain_discovery processes
on one scan. It carries its epoch like every other signature now, and
superseded() is the one definition all four guards share.
Also: cancel kept the open pause out of paused_seconds, so a run
cancelled after sitting paused reported the pause as run time; pause and
cancel take the row lock, so five concurrent calls give one winner
instead of duplicate ledger entries; a watch probe refuses to pause,
since its host waits on a terminal status that pause never reaches; the
hero, ribbon and empty state get a paused arm rather than falling into
the settled one and claiming a delta for a run that has not looked yet;
isOpenStatus mirrors SCAN_OPEN_STATUSES so Compare, the timeline and bulk
cancel agree with the api; and the pause confirm is gone — it removes
nothing, so the cost is stated on the control instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rg2cefwCvz9f77A2SHEHDb
A long run could only be cancelled and started over. A scan now holds at
the stage boundary: pause settles the stages in flight as PAUSED rather
than ABORTED, and resume dispatches a fresh canvas from the first level
that has not finished.
run_epoch guards the handover. A paused stage's celery task may still be
in flight, so every stage and finalize signature carries the epoch it was
dispatched under and drops out when the scan has moved on — without it a
straggler from the paused canvas would write into the resumed one.
resume_point() reads the level and the finished set through stages_done(),
so pause, resume and the stall reaper all agree on what "finished" means.
Paused time is held in paused_seconds and taken out of the reported
duration, so a scan's elapsed figure stays the time it actually ran.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XxW9yfg1SEchtg6HLGHRkK
A platform with a researcher API is now a provider: one module under
shared/services/bounty_providers implementing programs() and scopes(),
returning rows the sync engine stores. bounty_programs.py became the
platform-agnostic engine, so adding a platform is one module, one
APIProvider value, one PlatformSpec.api_provider and a key tester.
Intigriti is the second connected platform, on a personal access token
the hunter creates. On this instance: 220 programs, 54 of them private,
155 with scope and 65 the platform refuses, which accounts for all 220.
Measured against the live API, against the docs:
- Tier ids are unordered and undocumented. Id 4 is Tier 1, not No Bounty,
so the first mapping stored the best-paying assets as paying nothing.
Eligibility reads the label; out of scope is matched on id and label.
- A 403 carrying FORBID001 is a refusal, not the documented rate-limit
overage, and it covers public programs the account has not joined.
Retrying it cost 11s a program; AccessDeniedError ends it in one
request and records scope_access, keeping the failure budget for
real breakage.
- Membership is proved by the platform letting the token read a private
program's scope. confidentialityLevel never reports "invite only", so
joined was always false and its filter was dead.
Bounty Hub follows what the data actually says:
- source becomes sources, a JSONB array. A program is routinely in both
a platform API and the public feed; the feed adds itself without
taking an API row back, and facet counts overlap accordingly.
- The row hint read "Bookmarked on HackerOne" while rendering
Intigriti's follow flag. It now reads PlatformSpec.follow_label.
- Refresh queued the sync and returned, so nothing changed on screen.
It waits for the sweep, reloads, and says so.
- The dashed connect card per platform is one dismissible Alert.
- Connecting a platform no longer diffs API scope against feed scope,
which would report every asset removed and re-added.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SiHx8Sm34hS1nanrsPdBaD
Scope-true continuous discovery for a bounty program, bug bounty mode only
(capability `program_watches`).
Watching a program imports its in-scope targets under the program's
organization, writes out-of-scope entries as exclusions and the rate ceiling
on a `Watch · <program>` scan context, schedules an optional baseline, and
streams certificate transparency for the watched apexes. A new in-scope host
is resolved, written into the covering scan with source `ct_log`, probed by a
focused run with a screenshot, evaluated against the watch's alert query and
alerted once per fingerprint through the chosen channels.
- ct-stream compose service: certspotter 0.24.2 under a supervisor that
regenerates the watchlist from active watches, spools events through a
hook script that emits only certspotter fields, and reports over Redis
- worker tasks: certificate, settle, recheck, reconcile; finalize dispatches
settle for any probe run; reconcile after every program or feed sync
- notifier: named channels bypass per-channel preferences; attachments;
new `watch` notification type
- API under /bounty-programs/watches with preview, ledger, events, marks
- Bounty Hub: Watching tab, Watch dialog, watch sheet, program badge,
certificate stream card in Settings
- schedules carry an intensity; scan_schedules.intensity
Two review rounds folded in: compare-and-set probe claim, resolver-down
handling, spool kept on failed dispatch, schedule cap, mute keeps prior
state, ledger-derived totals, event filters, mode lockout of worker and
stream in corporate mode.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ba2f7gwu2GXiETCKqRztyB
The comment rule in CLAUDE.md is one line maximum, never narration: no
multi-line blocks, no module-header explainers, no sentences that explain or
justify the code to the reader. The tree had drifted a long way from it, and
most of the drift is mine.
Removed:
232 lines of multi-line comment blocks
203 single-line comments that argued instead of stating
154 comments joined by so/because/otherwise/rather-than
143 narrating comments in the frontend
112 test docstrings (the test name already says what it asserts)
25 test module docstrings and 25 decorative section dividers
Collapsed:
324 multi-line docstrings to one factual line
271 one-line docstrings that had a justifying clause bolted on
Kept: functional directives, section dividers, short field labels, alembic
revision headers, and the eight FastAPI route docstrings that render in /docs.
Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs.
284 python tests and 34 frontend tests pass, ruff and ruff-format clean across
ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
reNgine 2 had a Tools section; 3.0 lost it, so a hunter left the product
for a whois or a dns lookup even though every capability was already here
inside a scan.
`toolbox/` is a top-level package mounted into api and worker, and a tool
is one file under `toolbox/tools/` — auto-discovered exactly like stages,
report sections and MCP tools, with a pydantic `Input` whose flattened
schema IS the form the UI renders. Six ship: WHOIS, DNS records, subdomain
finder, HTTP probe, IP address and CVE.
Where a tool runs is a declaration the registry enforces. The api image
carries no tool binaries, so `INLINE` answers from RDAP, the offline range
tables or the threat feeds on the request's AsyncSession, and `QUEUED`
shells out on the worker; `_validate` rejects a class whose `run` is async
or sync the wrong way round, so the two cannot drift.
A run is a question rather than inventory, so it lives in Redis with a
seven-day TTL and no migration, keyed by user id. A queued run nobody
collects fails after two minutes instead of spinning for ever.
A tool answers in blocks — facts, table, tags, code, note — so one
frontend renderer serves every tool and a new one costs no frontend work.
An empty block with nothing to say about being empty is dropped.
`require_public` refuses loopback, private and link-local before the HTTP
probe sends anything, because an authenticated user typing 169.254.169.254
is an SSRF.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CtRcxFCPSYxW5sXam3SUWB
A running scan showed nothing. Caught live on an http_probe:
pg_stat_user_tables reported 821 rows physically inserted into
http_assets while count(*) returned 0, with one worker connection idle in
transaction for six minutes. The stage streamed httpx row by row but only
flushed inside savepoints and committed at the very end, so a 66-minute
probe was one open transaction — invisible to every reader and pinning a
connection against vacuum. Other long stages never streamed at all:
subdomain_discovery wrote nothing until every provider and the resolver
had finished, port_scan ran naabu unstreamed, url_discovery accumulated
katana's whole crawl, screenshot collected every image first. And nothing
told the browser: no rows-written event existed and no results table
refreshed while a scan ran.
stages/sink.py:ResultSink is now the one way to write incrementally —
buffer, flush on size or age, commit inside the flush, announce on its
own cadence. Stage.results_sink wires that to publish_results, which
refreshes only the affected scans counters and emits RESULTS_FOUND. Nine
stages convert. The write callback must commit; a flush that only
flushes reproduces the original bug exactly, and clearing the previous
attempt belongs inside the first flush or the table is empty for the
length of the run. nuclei needed one more thing: findings are sparse, so
_paced hands the caller back its own thread every two seconds, otherwise
a three-finding burst sits unwritten for the rest of a two-hour run.
The second half is when a derived fact is computed at all. ip_enrichment
is 10.1 s of pure SQL — 0.09% of a 3 h 08 m gov.np run — yet sat at the
last derived level, so 1,694 addresses carried a null ASN, operator and
country for 2 h 27 m, and a run cancelled before that level lost the
enrichment for good (scan d6532d88: 0 of 137 addresses, permanently). The
lookup is now one definition, ip_asn.enrich_addresses, called from
ip_inventory.materialize over the addresses it just wrote. The stage
keeps what only a finished run can know: the union sweep, CDN adoption,
the target-context fallback, the back-fills and the coverage number.
Interest is judged twice for the same reason. A saved query matches a
host on its own facts and is right at any point, so evaluate_live runs
those on a debounce while the scan works; rarity needs a finished estate,
so correlation and the model still wait for finalize. A correlation over
a tenth of the estate is not a weaker judgement, it is a wrong one.
Measured after: addresses enriched at level 2 with ip_enrichment not yet
reached, that stage down from 10.1 s to 0.1 s, http_assets ASN filled at
write time for free, 500 hosts flagged mid-discovery, and the browser's
Web assets badge climbing 2,210 -> 13,530 with no reload.
Eleven issues from an end-to-end review are folded in, four of them
regressions from the above: the rules provider emits KEYWORD but declares
RULE, so deriving the rewrite set from the class attribute dropped every
keyword signal on the full pass; a live run reported itself stale, which
made every read of the tab queue the unguarded correlation pass; SET
LOCAL lock_timeout died with ensure_builtin's commit; and enrichment
inside the insert transaction took scan-wide unordered locks that two
same-level stages could deadlock on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0165iTiibhYePy1EDixwC2Vs
Two would have broken the feature outright.
A scope-change event carried the asset's instruction into bounty_events.detail,
which is varchar(500) while the instruction is capped at 4000. 334 stored scope
rows already exceed 500 characters, so the first time any of those programs
changed scope the insert would have raised 22001 and aborted the whole sync.
Detail is now cut to the column width — the same class of bug as the presigned
profile picture, and the same lesson: every column width is a cap something
upstream has to apply.
The public feed was unreachable without HackerOne credentials. status.configured
gated the entire page and the Settings cards, so an instance with no token saw a
credential prompt instead of the 464 programs the beat had already fetched —
which defeats the reason the feed was added. The prompt is now a banner over the
library, refresh always works, and without a token it refreshes the feed alone.
The rest: the Updates tab was silently filtered to HackerOne, so feed events
never appeared while the badge counted them; "Refresh scope" was a no-op on
every non-HackerOne program and could resync the wrong one on a handle
collision; every import was tagged hackerone and described as a HackerOne
program whatever the platform; formatPayout announced a minimum as "up to";
a program name over 100 characters made an import 500; skipped listed every
unreachable asset in the program rather than the ones selected; a slug race
rolled back the caller's uncommitted targets instead of a savepoint; the feed
stamped its schedule even when every download failed, suppressing the retry for
a full interval; and the search fired a paginated query with aggregates on
every keystroke.
The twelfth is one no static review could see. A no-change yeswehack pass
measured 0.5s alone and 120.5s against a concurrent sync — two runs deleting the
same scope rows block each other, and the hourly beat can fire while a manual
refresh is going. Both sync tasks now take a session-level advisory lock and
decline in 0.01s rather than queueing behind one another.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
Bounty Hub was empty until someone pasted a HackerOne token. It now ships with
464 public programs and 6,889 scope assets from arkadiyt/bounty-targets-data
(MIT, republished every 30 minutes), so a fresh install is useful with no
credentials at all and the token becomes an upgrade rather than a prerequisite.
HackerOne is deliberately not read from the feed. The API gives us private
programs and richer scope, so the feed would be a downgrade — and a new
`source` column keeps that from happening by accident: an API row is
authoritative and the feed never overwrites one.
The three platforms do not agree on anything, so each gets its own entry in
shared/definitions/bounty_feed.py: the asset field is `target` on two of them
and `endpoint` on Intigriti; the type vocabularies are `website`/`network`
versus `url`/`wildcard`/`iprange` versus `web-application`/`mobile-application-ios`;
and only Intigriti and YesWeHack ship a handle, so Bugcrowd's is derived from
its engagement URL. All of it lands on the 17 asset types reNgine already has,
and target_for_scope needed no change to resolve 5,072 of them.
What the platforms do not report is not invented. Bugcrowd publishes no
submission state, so it stores `unknown` and the row shows no badge rather
than guessing "open". What they report and HackerOne does not — the actual
payout range — becomes a first-class field, a badge and a sort.
Provenance is visible rather than implied: every row names its platform, the
detail sheet says which source it came from and what that source can and
cannot contain, and Platform and Source are faceted filters.
The feed has its own schedule because it costs four small files rather than
640 requests: off, every 6 hours, daily or weekly, beside the HackerOne
interval in Settings. The change-detection layer needed nothing — it was
already platform-agnostic, so scope alerts work on all four platforms, and a
first feed load of 464 programs correctly announced none of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
A new bug-bounty-mode surface at /bounty-hub, gated on CAP_BOUNTY_PROGRAMS —
the capability that already existed and gated nothing.
Programs and their structured scope are stored, not cached, so in-scope and
out-of-scope are real server-side filters rather than a modal accordion.
2.x deliberately discarded out-of-scope entries with a TODO; this lands it.
The vocabulary is one definition (shared/definitions/bounty_programs.py):
all 17 asset types across network/mobile/code/other, with the five reNgine
can reach mapped to a TargetType. 2.x carried two divergent allow-lists and
silently dropped CIDR assets the modal had shown.
Normalisation delegates rather than duplicates: an identifier is cleaned and
then handed to reNgine's own validate_target. The asset type decides whether
to try; validate_target decides whether it works. So *.example.com and
https://*.uber.com become domains, 10.0.0.0/8 stays a range, AS64500 is
recognised out of the free-text OTHER field, and *.example.* resolves to
nothing rather than a broken target.
Import is end to end: selected assets become targets grouped under an
organization named for the program and tagged with the platform's own tag,
both confirmed in a dialog where the name and tags can be edited. Out-of-scope
assets are never selectable without an explicit opt-in.
Measured against the live API (630 programs, 46,109 scope assets):
- state is public_mode or soft_launched; there is no private_mode. Matching
the literal mislabelled every invite-only program as public.
- last_invitation_accepted_at_for_user separates programs you have joined
from public ones you have not, so both are filters.
- profile_picture is a presigned URL up to 2,107 chars, so it is Text; the
varchar(1000) it replaced failed one row and lost all 630.
- a program may publish no structured scope at all (200 with an empty array),
which is a different fact from "not fetched yet" and reads as such.
Also fixes the API-key Test button, which could never have worked for a
provider needing a username: testers only ever received key_value, so they
now receive key_meta too, and HackerOne verifies through the same client
the sync uses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZEi41ZDRWtwYDPFwoCVnu
Severity alone is close to useless for ordering work: roughly 60% of CVEs
score 7.0 or above. Until now EPSS and KEV came from nuclei's template
metadata, frozen at authoring time, so a finding could never learn that the
world had changed around it.
Two keyless feeds now land nightly and re-score every finding already stored:
EPSS (368k CVEs) and the CISA KEV catalog (1,695). A feed that fails to
download leaves its table untouched, so an install with no egress keeps
ranking. CISA is authoritative for is_kev on anything carrying a CVE.
exploitation.py judges each finding the way interest/ judges hosts, and the
signals that matter cross dimensions no other tool joins:
ransom_path a KEV ransomware CVE on a host whose ports also expose
a service ransomware crews reach for
fresh_exploit a public exploit published after the scan that found it
untestable no scanner template covers this CVE, so no scan can
clear it -- BlueKeep, EternalBlue and HTTP/2 Rapid Reset
all report false here
reachable answering directly, with no CDN or WAF in front
vulnx is an optional ceiling, never the floor: per-CVE exploits and their
links, template coverage, internet-wide population and bounty activity,
cached forever. Measured at 10 requests a minute keyless with no Retry-After,
so the client paces off the remaining header and can only ever be a
background cache-fill.
Alerting is delta-only, on the same rule as new findings: intel_signals is
upserted so created_at is when a finding first earned a signal, and repeated
refreshes alert nothing.
The nightly download is opt-out (instance_settings.threat_intel_auto_sync).
It gates the beat task and the worker warm-up only -- Refresh now always
works, so nobody is stuck with a scheduled outbound request they did not ask
for. Bounty activity renders only under CAP_HACKERONE; the gate is
presentational, so switching mode never loses data.
Every count is a promise: coverage, the query it compiles to, and the rows
its sheet opens were verified equal across all five signals.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PwxTAkPFUXPHPvdYT8Rhtm
Rule changes dispatch refresh_project, which deliberately does not
notify — relabelling a past scan is not a discovery. But a scan is
relabelled lazily too: reading one whose interest signature has gone
stale dispatches evaluate_scan from the read path, and that helper
never passed the task's `notify` argument, so it took the default and
alerted.
Observed: adding one keyword and then opening an existing nmap.org scan
produced "14 new assets worth a look on nmap.org". Nothing was found;
the scan was hours old and the only change was a word the user had just
typed. It fires on read, so it would recur for each scan opened after a
rule change.
The dispatch helper now carries `notify` and the stale-read path passes
False, which is the same position refresh_project already takes. The
task already accepted the argument.
Verified both directions: after the change, editing a keyword and
reading the scan relabels it (keyword matches 1 to 11, bands shift) and
creates no notification, while calling evaluate_scan the way finalize
does still alerts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6LcH8uUddjJbKrahf34ys
The dispatch client in shared/services/celery_dispatch.py was built as
Celery(broker=...), and celery adopts a new app as the process-wide
current_app by default. That client carries no result backend, so the
moment finalize dispatched interest evaluation — or the beat tick
dispatched a scheduled run — current_app in that worker child became an
app whose backend is DisabledBackend.
The stage canvas is nested chords, and an unbound signature resolves its
app from current_app. From that point on every scan dispatched by the
same child raised "Starting chords requires a result backend to be
configured" out of run_scan, so the run either stopped part-way through
its levels or never started a stage at all.
The client no longer sets itself as current, and the canvas now carries
the configured app on every node so it cannot depend on ambient state.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6LcH8uUddjJbKrahf34ys
reNgine 2 answered this with one global keyword list matched by `icontains`, and
rebuilt that query once per table row — `get_is_interesting` was a
SerializerMethodField, so a 5,000-host page ran the whole unindexable OR-chain
5,000 times. It was also dead until the user saved the settings form once, since
`get_interesting_subdomains` returns none() with no custom_type row.
The rule survives; everything around it is rebuilt. An interest rule IS a saved
HOST_QUERY query, so there is no second matcher to keep in sync, and the keyword
box compiles to `host:[…] or title:[…]` behind the same grammar. Signals persist
to interest_signals, roll up in SQL onto subdomains.interest_score/band/kinds so
sorting and `is:interesting` stay index-backed, and the scan carries a signature
of the rules it was labelled against — a stale read refreshes in the background,
so editing a keyword still re-labels every past scan without a read-time query.
interest/ is a top-level package mounted into api and worker like stages/ and
reports/, with providers auto-discovered from disk. Three ship:
rules the keyword lists and 19 saved queries
correlation what stands out from the estate, which no keyword can see
ai a judgement, labelled as one everywhere it appears
Correlation is the half a rule cannot express: an ASN carrying under 5% of the
surface, software or a favicon on three hosts or fewer, the one asset answering
directly while the rest sits behind a CDN. All gated on a minimum estate size,
because rarity means nothing without a norm. On gov.cy: 226 flagged of 1,379 in
0.4s, 29 of them critical.
AI sees hostname, status, page title and technology for hosts that answered, and
nothing else — never a response body — deduped to three samples per identical
response shape. It may answer only in a closed schema, a host it was not given is
dropped rather than stored, and its sentence must state what was observed, never
what is exploitable. Page titles are attacker-controlled, so they travel as JSON
data with strip_control and a cap, never interpolated into the instruction, and
nothing it returns triggers an action. Every AI signal stores its model and
prompt version, and the UI never lets it read as an observation.
Then the judgement graduates: suggest.py shows the model what it flagged and asks
for the query that would catch it again without a model. Each proposal is
compiled, counted against the scan and dropped if it errors, matches nothing or
already exists; nothing is stored until a person presses Add, after which it runs
on every keyless install with no cost.
Alerts are delta-only — hosts the target has never flagged before, and only those
in a high band or matched by a rule the user asked to be notified about.
Four bugs found on the way. `dr` matched drones.gov.cy, so shipped hostname rules
are anchored on label boundaries and false positives went from 6 hosts to 0.
predicates.interest_signal aliased InterestSignal while its condition used the
unaliased class, uncorrelating the EXISTS so every interest: field returned the
unfiltered total. The correlation aggregates needed explicit output aliases or a
bare `t` collided with the CTE and printed a UUID inside a sentence. And dismiss
removed the signals while leaving the denormalised rank, so the row stayed in the
list.
The asset query compiler moves to shared/ because the worker has no api/ and has
to compile a rule at finalize; api keeps a shim that aliases every submodule into
sys.modules, so no existing import changes.
The rule library lives at /interest, not in the Arsenal. The Arsenal is tools.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NVh1CGtjDA33fupXjLGbk2
reNgine 2 could produce a PDF. This produces a deliverable, and it is built
the way scan engines are built: sections are plug-and-play directories under
reports/, each with a pydantic config model that IS the source of truth. That
model feeds GET /reports/catalog, the builder renders its own form from it, and
the API validates against it. Adding a section is a directory. Scan Engine is to
stages what Report Template is to sections.
WeasyPrint 69 renders it, chosen over Chromium and Typst after measuring what
each can actually do. Only real CSS Paged Media gives a table of contents with
true page numbers, running headers, a PDF outline and footnotes; SysReptor
reaches the same verdict and pipes Chromium's HTML into WeasyPrint for exactly
that reason. reNgine drops the Chromium hop because it templates in Jinja and
draws its charts as SVG in Python, so a browser buys nothing and costs 400 MB
per report. Typst was rejected because a user-uploadable theme would then have
to be Typst code rather than CSS.
A theme is data. shared/definitions/report_theme.py is the schema, six themes
ship as YAML, and a user uploads their own; there is no markup in a theme, so
there is no injection surface. Fonts are vendored so an offline instance still
renders in Latin, Greek and Cyrillic.
Everything the document claims is computed once, in reports/analysis/engine.py:
a posture score whose every deduction is named, weaknesses ranked by the signals
behind the rank rather than severity alone, attack paths that chain observations
into a route, a remediation plan ordered by risk removed, per-dimension change
against the previous run, and caveats that state what was never assessed. Two
narrators read that one brief. Without AI the report still argues its case; with
AI the prose is better and the model never sees a row, only the brief, capped and
cached per check. An AI failure falls back and the report still completes.
Generation is a job. POST /reports queues, the worker walks progress over SSE,
and four formats come out of one render: PDF, self-contained HTML, Markdown and
JSON. The estimate reads each section's own limits, so the page count it promises
is close to the one it delivers.
AI moves out of Settings into its own top-level page: connection, a model per
tier, what it may be used for, what it has cost, and the narrative cache.
Verified on a 1,379-host gov.cy scan: 124 pages in 19.5s across 19 sections,
all six themes, and a broken key still producing a complete report.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f2Mf2HGwFaTRnui4UxpRd
A Celery task that probes the unchecked, non-static URLs under one folder of a scan's
endpoints with httpx, applies the results through endpoint_inventory.verify (never
upsert, so no source is invented) and records an EndpointCoverage row whose cap_reason
names the branch. Dispatched from the api through celery_dispatch.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015KEfo6ipoWMYeHxThmTLRy
- Add aggregate functions for scan status and counts in orchestrator.
- Create ScanEventPublisher for emitting scan-related events.
- Introduce ScanActivityService for managing scan activities.
- Implement ScanCommandRecorder for logging command executions.
- Develop canvas building and finalization logic for scan tasks.
- Refactor scan task to utilize orchestrator functions for running stages and finalizing scans.
- Enhance scan notification system to dispatch relevant events.
- Integrate command recording into CLI tool execution for better tracking.
- Updated the default route for automation to redirect to engines instead of scans.
- Removed the scans page and replaced it with a new scans overview page that lists scan history for projects.
- Introduced a detailed scan view page to display individual scan results and subdomain discoveries.
- Added new enums for scan statuses and subdomain sources to improve code clarity and maintainability.
- Implemented a new scan task in Celery to handle scan execution and logging.
- Enhanced the database models for scans and subdomains to support new features and improve data integrity.
- Updated the worker Dockerfile to include necessary scanning tools and dependencies.
- Added utility functions for domain matching and pattern compilation to streamline scan operations.
- Added `clean_name` function to `validation.py` for name validation with max length.
- Simplified docstrings in `dnsx` client, parser, and service files for clarity.
- Enhanced error handling in `ViewDNSService` to prevent disabling keys on rate limits.
- Improved WHOIS service to handle concurrent inserts gracefully with integrity checks.
- Updated `Settings` class in worker config to inherit from `BaseAppSettings`.
- Added new dependencies in `pyproject.toml` for `apprise` and `cryptography`.
Yogesh Ojha