Settings has six tabs: General, API keys, Proxies, Notifications, AI and
Users. Each tab is one card whose controls save on change; list tabs are
tables with a row menu and their Add action in the tab row. A non-admin
sees General and AI, read-only.
Out-of-band testing is the Callback server sheet on the Arsenal Nuclei
tab, with a Need a callback count that filters the library to the same
rows. Bounty Hub settings are a sheet on the Bounty Hub page. The old
settings routes redirect to both.
- API keys store the last test result on the row and clear it when the
key changes. Providers report whether a test exists. The usage count
is gone: only ViewDNS ever incremented it.
- Notification channels store the last delivery, written by the
notifier on its own connection. A channel offers the seven event
types the product emits, program changes on by default, and levels
named All events, Warning and critical, Critical only.
- A default proxy applies to a scan launched without a context and is
preselected for new contexts. A pool hands each scan one endpoint,
proxy URLs bracket IPv6 hosts, and deleting a proxy clears it from
its contexts.
- Users can be added with a role, disabled and promoted. An account
cannot change itself, and one that owns records answers 409.
- Retention windows are validated and include keep forever. The
instance name cannot be blank and names every page title.
- The AI list drops the two switches nothing read; each feature states
what it sends. The out-of-band Off help says the checks are skipped.
Connectors, Bounty Hub and Remote control lose their extra page
padding. Migration 5d2e9a8c1b47 adds the test and delivery columns and
drops proxies.mode.
Claude-Session: https://claude.ai/code/session_01RVAHQxZ7JAkoEiTCDiHYue
An engine keeps its three intensity presets and now carries optional
per-tool rate and concurrency in transport_overrides. Any value that
differs from the preset shows as Custom; an empty field uses the preset.
A context rate caps the engine value rather than replacing it, and rate
is settable only for tools that have one. Edited through a Rates panel
on the engine builder, validated and clamped, carried through create,
update, duplicate, preview and launch.
Claude-Session: https://claude.ai/code/session_01Bb4u8p34euie1YFzb7qCPu
- Store each run's row, first-seen and retired counts (scan_deltas, scan_retired), valid while
the revision stamp matches scan_revisions; statement triggers on the seven result tables bump
it. Finalize warms, a beat backfill fills the rest, a live run answers from its last count and
queues a refresh. /scans 1.5 s -> 43 ms, dashboard overview 5-10 s -> 0.9 s.
- lead_cache holds a live scope for 20 s with a stale copy, and every build is single-flight;
search pages opt out.
- Leads: lowercase subdomain names (CHECK) and LIKE for host:, a guard for empty JSON arrays,
branches as MATERIALIZED CTEs read to the cap, every count in one statement.
- Addresses and services: the derived CTE as one grouped pass, rows and count in one execution
(page_rows), GIN lookups for page hosts, facets through grouping sets.
- Correlation carriers counted by EXISTS; takeover latest-sighting in SQL, no silent cap.
- Screenshot backfill partial index, nine unused low-cardinality indexes dropped, hostname
helpers memoised, vhost names normalised.
The summary bar draws one lane per event type over the last 30 days,
14 on a phone: a dot per day sized by its count, findings coloured by
the worst severity. Hover shows the day, a click or a drag picks the
days, the current period is tinted and Caught up is a dashed line.
Every event starts collapsed and expands in place. A rescan lists its
new critical and high findings, each opening its sheet. A bounty event
lists its rows with their actions. Hover is a background, never an
underline.
The period now picks whole runs, so a run's count is always what its
link opens. Days follow the viewer's time zone across the day filter,
the lanes and the headings. The badge counts events, a triaged finding
also hides its cluster replays, and a scan still running is dated by
its newest finding.
Claude-Session: https://claude.ai/code/session_015EWyHNJRpxkvhtapY3wNGx
Stage tasks hold a slot for their whole tool run, so step hand-overs and
finalize queued behind them and background jobs took scan slots.
- worker-scans consumes scans (run_scan_stage only), scalable with --scale
- worker-control consumes scan_control: launch, hand-over, resume, finalize
- worker-default consumes default and critical
- reaper reads both scan queues; readiness requires a consumer on every queue
- connection budget counts every worker; max_connections 250
What's new lists only what changed: rescans that found new critical or
high findings, new programs, assets added to or leaving scope, bounty
table and rule changes, certificate hosts from watches, and targets a
scope or watch added. Events group by day on a rail, with a divider at
the Caught up mark. Each rescan names its worst new checks, and every
count opens the scan filtered to exactly those findings.
Program syncs now record payout, safe harbor and 2FA changes, and per
asset tier, bounty eligibility, severity cap and instruction changes.
The public feed sync diffs known programs for these too.
The search cache no longer stores a page that carries an error, so a
timed-out search no longer answers 0 rows for ten minutes.
Only a capability stage's success covers a dimension; a support stage
covers through the rows it wrote. A cancelled run that finished takeover
and zone_transfer hid the previous run's findings on every page.
The targets list reads findings from each target's vulnerability
covering scan, and both strips gain a medium chip.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- sync the account's reports, bounties and standing from the HackerOne
hacker API into bounty_reports, bounty_awards and bounty_accounts;
money comes from /payments/earnings, totals stay per currency
- /bounty-hub/<platform> as a Bounty Hub child in the sidebar, in the
/scans idiom: stat strip, severity toggles, monthly chart with drag
range, stage tabs with filtered counts, URL filters, expandable rows,
keyboard navigation and a by-program view
- program names open the Bounty Hub program sheet, which links back to
the reports on that program and fetches unfetched scope on open
- hourly report sync gated on bug bounty mode and the sync interval
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/targets follows the scan history idiom: a strip of counts that filter
(targets, not scanned, stale, critical and high findings, enrichment
signals), type tabs and tools in one card, and rows that carry the latest
run's status, findings chips with previews, assets, change and a
findings trend, expanding into the run brief. Columns hide by the
table's container width and the list has the /scans keyboard.
GET /scans/latest returns the newest census run per target with the
history fields. Targets gain unscanned, stale, critical and high
signals whose counts equal the rows they filter; critical and high read
the latest run and skip suppressed findings.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AQDJwqjTD69CUKa95DjM8H
A scope bar under the header narrows every cell to chosen targets, an
organization or a tag, kept in the URL. The dimension endpoints, the
dashboard reads, threat intel, interest and domain posture accept
target_id, organization_id and tag_id through one resolver. A target row
in Surface against risk scopes the board on click, cell links carry
target=[...] so each count opens the same rows, and cells that cannot
narrow are marked All targets.
Dashboard copy reworded to state facts: no explanations, reassurance or
colourful labels, and failures read "<Noun> not loaded."
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each row carries open critical, high and medium counts, computed with the
Vulnerabilities page's suppression rules, and whether the vulnerability
stage ran. Medium is behind a toggle in the run brief; low and info are
not shown.
- Row brief with Findings (inline triage), Changes, Coverage and Engine tabs
- Latest per target view with earlier runs nested beneath
- Query grammar: target, engine, status, severity, is:partial, is:added
- 30-day runs chart with a date brush, strip counts equal to the rows opened
- Findings trend per target, inline compare sheet, keyboard navigation
- API: findings and target_runs on ScanRead, severity/short/added/date/latest
list filters, GET /scans/trends and /scans/daily
- Removed the unused /scans/targets grouping and the old list components
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuoZt3GW4VjRSF7FJeFtuV
GET /whats-new/visual pairs every host two consecutive settled census
runs both captured and keeps the ones whose screenshot dHash moved past
the cluster distance, so the pair is no longer the same page. Each pair
carries the hamming distance and the compare fields that moved; a pair
where none moved is a silent redeploy, its own badge and filter. The tab
shares the page's period, target and text filters, shows before and
after with hover previews, and opens the web asset sheet, Compare runs
or a scan from the card. The feed carries the pair count for the tab.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018XqZAsFckWFdr7EFiyZH9V
GET /whats-new reports what appeared in a project since the operator
last caught up, grouped by the run or program it came from. Ten kinds,
each noise-gated: web assets, services, findings and secrets from repeat
scans (a first scan sets a baseline and reports nothing, certificate
names stay with the watch), in-scope assets, programs, certificate hosts
and targets from Bounty Hub, and a Gone pair of out-of-scope assets and
retired web assets. A run group's "All N" opens that scan's tab with
is:new, so the count equals the rows. POST /whats-new/seen writes the
mark from the Caught up button alone; GET /whats-new/unseen feeds the
sidebar badge. Scope events count for programs the project watches or
whose scope covers a target; program events count library-wide, with a
ring switch for the whole library. The 13-week grid is per kind and
doubles as the period filter, by day or range.
The page carries kind tiles with facts and sparklines, a source switch
(All, My targets, Bounty Hub) that collapses in corporate mode, target
and program pickers, a text filter, rows that open the results tables'
detail sheets in place, inline Add target, Watch, Scan, Mute and Remove
target, card collapse with card-level Add N targets, shift-click range
selection, keyboard navigation, copy on hover, a sticky filter bar and
a reload when a run finishes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018XqZAsFckWFdr7EFiyZH9V
The feed reports new surface only: web assets a later scan was first to
find, scope a program opened, programs added or made public, hosts a
certificate revealed under a watch, and targets Bounty Hub brought in by
watch or scope. A first scan yields nothing, hand-added targets are the
activity log's business, and scope removals stay in Bounty Hub. Program
events count for programs the project watches or whose scope covers one
of its targets. The response carries per-day buckets per kind, drawn as
a week-column grid above the list; a lit day scrolls the list to its
header. Days are keyed in UTC on both sides. Windows are 7, 30 and 90
days, default 30.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XfLmNNiLk5VdePfq9CpugW
GET /changes reports what arrived in a project after a point in time: web
assets a scan was first to report (baseline-gated, certificate hosts left
to the watch ledger), in-scope hosts from program watches, scope and
program events for watched programs, and new targets with how they were
added. POST /changes/seen writes a per-user watermark keyed by project;
the page marks on open and keeps the returned cutoff for the visit, with
a 7-day window standing in on a first visit. Timeline mode reads a
24h to 90d window filtered by target or program. Linked from the
sidebar, the palette, the Bounty Hub header and the target page.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XfLmNNiLk5VdePfq9CpugW
_fold_layouts folds siblings that share a structure into one row, and
_emit_group wrote its dir token from the first 40 members while
subtree_count and the folder count covered them all. Past forty, the
row claimed more than the link returned — the promise every other
count in this codebase keeps.
The group now stops at the cap, ranked, and the folders past it stay
their own rows. Measured on the gov.cy tree: seven layout groups, each
naming exactly what it counts.
endpoint_tree.py had no test. Its pure parts have one now: the fold,
the cap, the two refusals to group, the anomaly sentence and the
archive-only test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DafWxfoWZXVDgr9KroJBDZ
The attack surface funnel is gone. The target page draws Reachability
instead: one composition bar over the four states a name can be in —
unresolved, resolved without an HTTP answer, answering, answering with
a finding — and the dashboard's Inventory row carries the count that
answered.
That count is is:web, not is:live. is:live compiles to 2xx and 3xx, so
a 401, 403 or 5xx responder answered and was still reported as having
dropped out; the funnel's "Answer HTTP" step had been reading the
wrong one, and the Inventory sub-line inherited it. Measured on this
instance: 3,418 answer where 2,814 are live, so 604 web assets — the
denied and error surface — were counted as not answering.
DashboardOverview.answering_hosts carries it and the predicate has one
definition, predicates.answered(), which the is:web flag compiles to.
subdomains insights re-derived live, web and resolved inline beside
the predicates module it already imports. It reads them from there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DafWxfoWZXVDgr9KroJBDZ
Covers what the .port guard changed: masking a stored URL never raises
whatever port it carries, and a port outside 1-65535 or with a letter
in it is refused when the channel is saved.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DafWxfoWZXVDgr9KroJBDZ
Per target, live web assets against open findings stacked by severity,
with actionable, act-now and KEV counts. The cell shows the six targets
with the most findings; a modal shows every target with organization and
tag filters, severity toggles and four sort orders.
GET /dashboard/surface-risk reads the same covering scans and suppression
as the overview, so its totals equal the board and the inventory.
The hero headline lost the space before "on N targets".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011h3F8L6bnjycZcXtWk4u6p
Every result and management table gains multi-row selection and the floating
action bar targets and scans already had: copy, export selection, delete, and
the dimension's own actions. Rescan and triage move off the inline strip, which
now only offers the rescan that acts on the whole query.
Delete is dimension-local. A row takes only what cannot stand without it: a web
asset its stored responses, a secret its sightings. Endpoints, services and
findings on a deleted host are kept, so every count stays equal to its table.
POST /surface/delete is the one route and the scope is a WHERE, so an id from
another scan deletes nothing. Vulnerabilities are addressable by template_id so
the Issues view deletes a whole check. Exposures dismiss rather than delete: an
exposure row is a web asset.
Selection export runs through the export pipeline. Every surface filter gains
ids, capped by MAX_SELECTED_ROWS.
Secrets and software become exportable, which fixes two breaks: a bundle export
raised KeyError on the secrets dimension, and mcp read a SURFACE_COLUMNS entry
for secrets that did not exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XyDpdVXL7YVsr5D7BVj2zt
The target overview now uses the same hairline cells as the dashboard: a
counts strip with per-run sparklines, the attack surface funnel, geography,
findings, posture, and the composition cells (hosting, services, technology,
hygiene, software CVEs, certificates, exposures) scoped to the covering scan
through ROUTES.results. Identity, monitoring and seeds are cells; the run
timeline is a cell with a surface-by-run chart once two runs completed.
components/cell.svelte is the shared primitive; the dashboard cell wraps it.
A target's estate is one engine (api/app/services/target_estate.py):
redirects and CNAMEs off the apex, certificates a host serves for another
organisation, names on the target's own certificates, cross-target relations
and shared addresses. A certificate naming more than five registrable domains
with a subject off the apex is a platform's and its names are neighbours, not
relatives. Names under the target's own zone are its own, evidence seen only
on an IP host is shared, and CNAME, NS, MX or SPF targets in PROVIDER_SUFFIXES
are providers. The target page, the targets page and the dashboard's untracked
domains read the same engine through GET /targets/{id}/estate and
GET /targets/estate, shown as a one-line strip with Review and Add.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0181ibJxKGBFiTbbHoXuhosP
Four bands in reading order: counts, the attack surface funnel beside
geography and the added/retired changes chart beside inventory, an
Act/Attend/Track findings board with severity per day, exploitation and
an evidence matrix, a mode band (platform events, watched programs and
browsing in bug bounty mode; certificates, hygiene and ownership in
corporate mode), scan activity and software CVEs, composition cells, and
an activity rail. The 7d/14d/30d window redraws every time chart; every
cell can be hidden and restored per mode from a catalog.
The overview gains the funnel, retired rows per day, findings per day by
severity, run outcomes per day, the severity by evidence matrix, queue
tiers and certificate expiry buckets. GET /dashboard/activity unions runs,
feeds, exploitation changes, connector syncs and, in bug bounty mode,
watch and platform events. GET /dashboard/programs carries the bug bounty
band.
Copy states a measure or nothing. Counts tiles open a per-target sheet
with a scan action, funnel steps hint and dim on hover, board cards open
a hover card.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HjHnxAPLMDSfQgVQDfDYBj
global_threads was multiplied by the scan context's thread multiplier and
then discarded; every stage carries its own threads field with its own
default, so the engine-level number had nothing to apply to.
global_http_crawl was stored, exported, imported and never read at all.
Both reached the engine form, the YAML, the launch preview and the run
comparison, where "Crawl responses" was a material facet: two runs that
differed on it were reported as able to find different things.
Gone from the table, the three schemas, ResolvedScanConfig, AdHocEngine,
the preview, both Compare facets and every frontend site, with a
migration to drop the columns. An engine exported before this still
imports: _RETIRED_KEYS accepts the two keys and ignores them, while an
unknown key is refused as before.
_MAX_ENGINE_THREADS goes with the validator that used it. It was declared
twice in the same file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5roHYDYzSPRNaTkn8hSkJ
export_yaml masks global_headers and tool_options, and import_yaml feeds
that document straight into create(), which had nothing to unmask it
against. Exporting an engine that carried auth and importing it back
produced an engine that sent `Authorization: ••••••••` to every target
and handed nuclei `-api-key ••••••••`.
Both validators now refuse a value carrying the mask, naming the header
or the tool. update() keeps working: it restores the stored value first
and validates what it is about to store, rather than what was submitted,
so the editor can keep sending masked values back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5roHYDYzSPRNaTkn8hSkJ
Four implementations answered "what does this host share?" and two of them
could only answer it for one run: the project-wide page had no Correlation
tab at all, and its detail sheet passed the row's own scan_id, so "Related"
silently meant "related inside this one target's last run".
One engine now, shared/services/correlation/, parameterised by QueryScope.
kinds.py holds the vocabulary as data; hubs.py discovers hubs with a GROUP BY
over the whole scope instead of grouping a 3,000-row sample in Python, so a
hub is no longer missed for falling outside the cut. asset_query/groups.py
derives its dimensions from the same registry, and correlation-graph, related
and correlation take WebAssetScope. New /surface/correlation page.
The count is a promise, and three bugs broke it. Asset kinds joined
http_assets and counted rows, so a host answering on 80 and 443 counted twice
(cert.fingerprint said 80 where the search returned 41). asn excludes
CDN-fronted rows but its token did not, 8779 against 8815; KindSpec.narrows
appends -is:cdn. And carriers came from a window over the group set the
caller asked about, so one row's own value read as 100% of its carriers and
common swallowed it. All 28 hub counts across 14 kinds now equal their
searches.
The provider rules moved into the engine. generic_page, shared_edge and
cert_covers gated cross links alone, so estate-wide the graph drew 404 Not
Found across 5 targets, 403 Forbidden across 4 and a 548-byte 404 body across
5 as correlations. A hub is judged once its members load and is labelled with
who wrote it; hubs load MAX_HUBS_PER_KIND + 40 deep and re-rank.
Related takes the whole vocabulary, each kind carrying the phrase it is
announced with and the query its button opens, which retires relationDsl and
RELATION_LABELS. asn became a kind, out of the default set. A scope over
several targets opens on what crosses them: 3,000 nodes over 21,690
correlating hosts is a hairball, the 30 crossing hubs are the finding.
Measured on 63,844 host rows across 10 covering scans: ~1.1s cold for the
estate, 70-200ms for a scan, cached after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FwCCFgRBC8ut6SQpmzfGAa
A platform with a researcher API is now a provider: one module under
shared/services/bounty_providers implementing programs() and scopes(),
returning rows the sync engine stores. bounty_programs.py became the
platform-agnostic engine, so adding a platform is one module, one
APIProvider value, one PlatformSpec.api_provider and a key tester.
Intigriti is the second connected platform, on a personal access token
the hunter creates. On this instance: 220 programs, 54 of them private,
155 with scope and 65 the platform refuses, which accounts for all 220.
Measured against the live API, against the docs:
- Tier ids are unordered and undocumented. Id 4 is Tier 1, not No Bounty,
so the first mapping stored the best-paying assets as paying nothing.
Eligibility reads the label; out of scope is matched on id and label.
- A 403 carrying FORBID001 is a refusal, not the documented rate-limit
overage, and it covers public programs the account has not joined.
Retrying it cost 11s a program; AccessDeniedError ends it in one
request and records scope_access, keeping the failure budget for
real breakage.
- Membership is proved by the platform letting the token read a private
program's scope. confidentialityLevel never reports "invite only", so
joined was always false and its filter was dead.
Bounty Hub follows what the data actually says:
- source becomes sources, a JSONB array. A program is routinely in both
a platform API and the public feed; the feed adds itself without
taking an API row back, and facet counts overlap accordingly.
- The row hint read "Bookmarked on HackerOne" while rendering
Intigriti's follow flag. It now reads PlatformSpec.follow_label.
- Refresh queued the sync and returned, so nothing changed on screen.
It waits for the sweep, reloads, and says so.
- The dashed connect card per platform is one dismissible Alert.
- Connecting a platform no longer diffs API scope against feed scope,
which would report every asset removed and re-added.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SiHx8Sm34hS1nanrsPdBaD
Clicking the DNS cell in the targets list opened the generic target dialog:
there was no DNS modal at all. There is one now, and it renders through the
same `dns-tab` the target page uses, so the records, the type filters, the
provider labels and the refresh are one component in two places.
DNS is also a relation. `dns_record` links two targets on a mail host, a zone
contact, an address or an SPF include - and only when the estate owns the
name. On the four unrelated estates here the only shared DNS values are
Google's MX hosts and an `include:mail.zendesk.com` that briefly linked
Shopify to HackerOne, which is the landlord problem in a new dress: a SaaS
sender is authorised by everyone. A mail host, a zone contact or an include
counts when its registrable domain is a target of the project, so
`mail.zendesk.com` says nothing and `serviceproviders.uber.com` says
everything. Measured: adding ubereats.com links it to uber.com on the SOA
contact alone, past nameservers (UltraDNS) and MX (Google) that say nothing.
An NS record is left to the WHOIS nameserver relation, so a fact appears once,
and an address that cdncheck flagged is not a shared machine.
`is_shared_host` folds the nameserver, shared-edge and mail-provider lists
into one judgement, so `aspmx.l.google.com`, `awsdns-hostmaster.amazon.com`
and `gold.foundationdns.com` are all recognised as rented.
The DNS and BGP modals carry a Shared with block through one
`related-targets` component: the other targets and the evidence, filtered to
the kinds that modal is about.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYop31s6BqDUt4TBs5mbwa
The scope link ran one way: a program's scope showed whether each entry was
already a target. The question a hunter asks is the other one - I have this
asset, who pays for it - and nothing answered it.
`ProgramCoverageService` matches a host against every synced program's scope
through the machinery a watch already uses (`plan_scope`, `WatchItem.matches`,
`excluded_by`), so a wildcard entry covers its subtree, an exact entry covers
itself, and a program's own out-of-scope rows mark the match excluded rather
than hiding it. Measured against 7,173 synced scope rows on this instance:
`foo.bugcrowd.com` and `deep.nested.bugcrowd.com` resolve to Bugcrowd through
`*.bugcrowd.com/auth/*`, `notacme.com` is not a subtree of `*.acme.com`, and
`allegro.cz` comes back uncovered because that program lists it out of scope.
The target page carries it in the rail as Bounty: the program, its platform
and the scope entry that covers the target, hidden outside bug bounty mode.
`for_host` takes a bare hostname so a discovered asset can be asked the same
question next.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYop31s6BqDUt4TBs5mbwa
A project's targets were never correlated with each other. WHOIS could link
two of them and only through registration; nothing read what the scans
already knew, so a shared certificate, a shared network and a shared favicon
went unnoticed.
`TargetRelationService` answers "which other targets are this one" from five
facts, strongest first: a certificate that names both (by fingerprint or by
SAN), one registrant after normalisation, a network the organisation runs, a
nameserver nobody rents, and a favicon. Each dimension is read from the scans
that covered it, so the answer follows the same scope rule as every other
project-wide view.
A landlord is not a relation. Correlating targets on the ASN alone links four
unrelated companies eight ways through AWS, Cloudflare, Google, Fastly and
SendGrid - measured on this instance. A network only counts when its holder
is a party to the target (`owns_network`: the ASN holder's name against the
target's apex label and its registrant), which keeps AS63086 Uber
Technologies and drops every provider above. A favicon on more than
`FAVICON_MAX_TARGETS` estates is a framework's and is dropped too.
On the target page, Related targets now names the target and the evidence
rather than the WHOIS group: shopify.com reads "shopifycloud.com ·
Certificate · Registrant", one certificate served by beta-help.shopify.com
naming the other domain and one registrant behind both. The WHOIS records
that are not targets stay where they were, as domains to add, so a fact
appears once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYop31s6BqDUt4TBs5mbwa
Target correlation returned nothing for every target in a real project, and
three of its four signals could not have worked.
A registrant is matched on what two records must share to be the same party:
case, punctuation and the company suffix are stripped by `registrant_key`,
whose SQL half sits beside it (`registrant_key_sql`) because a match invisible
to one half is a relation that never appears. Both halves refuse a redacted
name, so "Redacted for Privacy" cannot group every privacy-protected domain
in the instance into one false cluster. Measured on shopify.com and
shopifycloud.com: one relation, on the registrant, and nothing between the
four unrelated estates beside them.
A registrar is dropped as a correlation. MarkMonitor registers for thousands
of unrelated brands, so "same registrar" asserts nothing - shopify.com and
uber.com share it and are not related. It stays available as a deliberate
lookup in the correlation dialog, which is a question the operator asks, not
a relation reNgine claims. The case-sensitive exact match it used would have
missed those two anyway: `MarkMonitor Inc.` against `Markmonitor Inc.`.
A nameserver is the one strong signal here - `a.ns.hackerone.com` is
identifying, `gold.foundationdns.com` is Cloudflare's enterprise DNS and
identifies nothing. Fifteen managed providers join the suppression list,
foundationdns and markmonitor among them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYop31s6BqDUt4TBs5mbwa
Four scans against real estates (hackerone.com 40 names, bugcrowd.com 71,
uber.com 11,203, shopify.com 26,235) show the correlation graph drawing the
CDN's identities and the resolver reporting a fraction of the live surface as
all of it.
The resolver's silence is re-queried once. dnsx prints nothing both for a
name that does not exist and for a query it dropped, and the loss is uniform,
so `_retry_degraded`'s peer median never fires (and a single-batch estate has
no peers). uber.com reported 135 of 11,203 names resolved; a plain dnsx re-run
over a random 200 of the 11,068 it called dead answered 7, the same 7 three
times over, and a DoH check of another 25 found 2 more. `_retry_silent`
re-queries every unanswered name once (`dns_retry_silent`), writes through
`_write_resolution` so the live view keeps up, counts recoveries as
`recovered`, and reports partial when that pass still recovers names.
A hub that belongs to a provider is not a correlation. bugcrowd.com drew six
hubs and five of them were the CDN's: four Fastly anycast addresses and the
`j.sni.global.fastly.net` CNAME every Fastly tenant is aliased to - four hubs
over one pair of hosts, each restating what the CDN kind already says. Hubs
now carry `platform`/`platform_label` from `ip_addresses.is_cdn` and from
curated lists beside `is_shared_nameserver` (`SHARED_EDGE_HOSTS`,
`PUBLIC_CA_NAMES`), the graph hides them behind "Hide provider", and
`related()` drops them. Curated, never inferred from a suffix:
`d3tqr1od3k1eh2.cloudfront.net` is one distribution, `j.sni.global.fastly.net`
is the whole platform; a private issuer stays visible.
Three more corrections in the same build. A share is read against the hosts
carrying that kind, not every name in the scan, so shopify.com's 16-of-18
JARM cluster no longer reads as 0.5%, and `common` needs
`MIN_ESTATE_FOR_COMMON` carriers. The 3,000-host cut is ranked by "answered,
then resolved, then name" - alphabetically it held 19 of shopify.com's 116
web assets, and ranking took that scan from 35 hubs to 94 and its certificate
correlation from 6 hosts to 66. A body under `MIN_BODY_BYTES` is not an
identity: 11 hosts shared the SHA-256 of an empty body. Hidden hubs no longer
consume the per-kind cap, chip counts equal what they draw, and the endpoint
goes through `lead_cache` like its neighbours (3.3s to 18ms).
Enrichment, which feeds all of it:
- `sync_ranges` takes an advisory lock and a budget. Two concurrent scans on
a fresh instance each pulled the same 200 MB and each truncated and
reloaded 1.1M rows: 291s against 145s. A scan that finds the tables
missing reports partial, and `app.tasks.ip_asn.refresh` fills the blanks
through `backfill_addresses` once the feed lands.
- No address is attributed from its target. The last-resort fill stamped the
target's BGP holder and its WHOIS registrant country onto every address the
range tables did not match, private ones included - a registrant country is
not a geolocation and a target's network is not the operator of everything
its scan found. It fired wherever that data exists, which is IP, range and
ASN targets and any registry that publishes a country; only an ASN target
now states its own sweep.
- cdncheck sees every address, httpx only the ones that answered, so
`ip_enrichment` folds `ip_addresses.is_cdn` back onto the hosts that
resolve to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYop31s6BqDUt4TBs5mbwa
Scope-true continuous discovery for a bounty program, bug bounty mode only
(capability `program_watches`).
Watching a program imports its in-scope targets under the program's
organization, writes out-of-scope entries as exclusions and the rate ceiling
on a `Watch · <program>` scan context, schedules an optional baseline, and
streams certificate transparency for the watched apexes. A new in-scope host
is resolved, written into the covering scan with source `ct_log`, probed by a
focused run with a screenshot, evaluated against the watch's alert query and
alerted once per fingerprint through the chosen channels.
- ct-stream compose service: certspotter 0.24.2 under a supervisor that
regenerates the watchlist from active watches, spools events through a
hook script that emits only certspotter fields, and reports over Redis
- worker tasks: certificate, settle, recheck, reconcile; finalize dispatches
settle for any probe run; reconcile after every program or feed sync
- notifier: named channels bypass per-channel preferences; attachments;
new `watch` notification type
- API under /bounty-programs/watches with preview, ledger, events, marks
- Bounty Hub: Watching tab, Watch dialog, watch sheet, program badge,
certificate stream card in Settings
- schedules carry an intensity; scan_schedules.intensity
Two review rounds folded in: compare-and-set probe claim, resolver-down
handling, spool kept on failed dispatch, schedule cap, mute keeps prior
state, ledger-derived totals, event filters, mode lockout of worker and
stream in corporate mode.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ba2f7gwu2GXiETCKqRztyB
Proxy observations are written as endpoint rows with source proxy into the
target's covering scan; a target without a census scan gets a Browsing run,
and finalize replays recorded shapes into a new census scan. "Seen by a
scan" is host plus path shape (new endpoints.shape column, backfilled),
with parameter differences reported as their own notice and unseen claimed
only when a settled scan covers the target.
Removed the scan trigger, sessions and coverage surfaces. The jar is served
from binaries/, the setup dialog derives the endpoint from the browser
origin, scans dispatch one run per target with the chosen scan context, and
manual findings mask credential headers. Endpoints page gains Browsed and
Not browsed filters; the Burp tab gains a host line, a notices table with
Open in reNgine, and a report dialog with notes and a result.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMmqqNaMMjCJWzGG49Xc1i
reNgine could say what was new against a target's whole history, and 2.x could
say `[new, removed]` for subdomains. Neither answers the question an operator
actually asks after a scan finishes: what changed since that run, the named one
I am comparing against.
The hard part is not the diff. Two StandardRengine runs of vulnweb.com in this
instance, eighteen minutes apart, went from 14 findings to 0. A set diff reports
"14 findings resolved, 12 high". The truth is `vulnerability_scan.max_minutes`
went 0 to 2, and the coverage rows record 57,100 requests over 368s against four
hosts, then 27,582 over 120s against seven. The delta belongs to the run.
So comparability is stated before any delta, per dimension: like for like,
settings differ, runs differ in depth, not scanned. Stage settings and the runs'
own setup are both compared — engine, context, auth type, intensity, proxy
(as set/none, never the URL), scope exclusions, protocol, crawl. Facets that
change what a run can find raise the verdict; threads and rate limits do not.
Runs that ran between the pair are counted, a cancelled run in the pair lowers
the verdict, and when the chosen pair is not like for like the nearest run that
would be is offered.
Identity is read off the per-scan unique constraints already on each table —
`subdomains.name`, `endpoints.signature`, `ports.(ip, number, protocol)`,
`ip_addresses.ip`, `vulnerabilities.fingerprint` — so the diff key is not a new
invention. Four verbs: appeared, changed, disappeared, unchanged. Disappearance
is only asserted when the producing stage finished cleanly; dnsx drops 20-60% of
names silently, so a partial run's missing rows are unconfirmed, a bucket of
their own. A dimension one run did not produce reports not scanned, never
everything removed. EPSS and KEV movement is Class D and is counted apart.
A disappeared row does not exist in the later run, so a query scoped to it can
never return one. Rather than teach five grammars an `is:added` flag, the
service flips the scope per verb, and every column, sort key and detail sheet
keeps working. Rank is computed once as a SQL CASE and the signal name is
recovered from the rank, so the order and the label cannot disagree.
Nothing is stored: a 425k-row full outer join measures 66 ms, the whole endpoint
about 31 ms. No table, no migration.
Direction is a shape — solid, split, hollow rails — not green-and-red. New
attack surface is not good news. Diff mode keeps the git glyphs for the same
reason it colours them from the `--code-*` ramp: in a diff `+` is syntax. That
mode is generated once on the server and shown through `code-block`, so the
screen and the clipboard cannot drift.
Eligibility is explained, never hidden. `REFUSAL_REASON` is one definition
mirrored in `config/compare.ts`, so the disabled button's hint, the alert page
and the API error say the same sentence, and an impossible pair reached by URL
lands on an alert that lists the target's runs to pick from.
`recheck.py` now reads its host field list from the shared definitions, so the
product holds one answer to "did this host change".
Also fixes the scan hero, which said "No change since the previous scan" while
measuring only hosts. Beside a page reporting 21 changes that read as a
contradiction; it now says what it measured.
Vulnerability rows are one per finding rather than grouped by check, and
comparing against a run that is still going is not built.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KAVgTQe8K3EvGzPo5HVmKF
2.x had `recon_note.TodoNote` — a title, a body and an optional scan. 3.0
dropped it and replaced it with nothing, so the operator's own reasoning had
nowhere to live: the lead to come back to, the dead end not worth retesting,
the reason behind a conclusion.
A note anchors to a durable asset identity rather than to a scan row. A note
written on api.example.com during scan 2 is still on that host in scan 9,
because the anchor is the host name, not the `subdomains.id` scan 2 created.
Identity per dimension: host name, endpoint signature, `ip:port`, address, and
vulnerability fingerprint.
That gives the rollup for free. An asset sheet shows its own notes across every
scan; a scan shows notes for assets it observed, plus what was written on it; a
target shows everything under it; /notes holds the project. The scan reach is a
per-dimension EXISTS over the `TABLES` map `surface_scope` already holds.
Every note carries at least one tag, enforced by `min_length=1` on the create
model and a disabled Save in the composer. Tags are the project vocabulary from
`shared/models/tag.py`, so a tag means the same thing on a note as on a target.
`dimension` is a `SurfaceDimension` value, so notes inherit the labels, nouns
and ordering the rest of the product uses. No second asset-kind enum.
Surfaces: a Notes section in the web asset, endpoint, service and vulnerability
sheets; a Notes tab on a scan and on a target; and /notes with tag, dimension,
status and free-text filters. The IP sheet is left out — an address group folds
across targets, so it has no single rollup anchor.
Also gives the two ScrollAreas added with the focused-scan work a definite
height; `max-h` alone does not clip, and a short list now renders without a
scroller rather than showing an empty track.
Row markers on the result tables are not built.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01799buekQvDvH41ns8EdhdH
The comment rule in CLAUDE.md is one line maximum, never narration: no
multi-line blocks, no module-header explainers, no sentences that explain or
justify the code to the reader. The tree had drifted a long way from it, and
most of the drift is mine.
Removed:
232 lines of multi-line comment blocks
203 single-line comments that argued instead of stating
154 comments joined by so/because/otherwise/rather-than
143 narrating comments in the frontend
112 test docstrings (the test name already says what it asserts)
25 test module docstrings and 25 decorative section dividers
Collapsed:
324 multi-line docstrings to one factual line
271 one-line docstrings that had a justifying clause bolted on
Kept: functional directives, section dividers, short field labels, alembic
revision headers, and the eight FastAPI route docstrings that render in /docs.
Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs.
284 python tests and 34 frontend tests pass, ruff and ruff-format clean across
ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
Every other correlation in the codebase filters for noise — origin exposure decays a
fingerprint by its reach, the graph hides a hub carried by half the scan, rarity
signals refuse to fire below a 25-host estate. The per-host relations on the asset
sheet had only a cap, and a cap is not a filter.
Two things were wrong on a real host. "Same network AS45353" listed 300 hosts
because 300 is the cap; the network actually carries 7,163 of the scan's 24,028,
and the sheet reported the cap as the truth. And a value carried by the entire
estate — one CNAME in front of everything — was offered as a relation at all.
SubdomainRelation now carries the real total beside the capped list, so the sheet
reads "7,163 hosts sit in the same network" and shows the first 300 of them. A value
on half the estate or more is dropped, reusing COMMON_SHARE from the correlation
vocabulary rather than inventing a second threshold, and a scan below
MIN_ESTATE_FOR_COMMON suppresses nothing because rarity has no meaning without a
norm. The certificate relation went through its own hand-written query and now goes
through the same gate as the rest.
Four tests: a shared address is a relation, the count is the truth and not the
length of the list, a value the whole estate shares is not a relation, and a small
scan suppresses nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
There was no backup path at all, and the connection arithmetic did not add up.
scripts/backup.sh dumps the four things an instance actually is: the database, the
scan media on disk, the docker volumes holding uploaded wordlists, nuclei templates,
report themes and fonts, and the settings a restore needs. SECRET_KEY travels with
it deliberately — API keys, proxy credentials and notification secrets are Fernet
encrypted with a key derived from it, so an archive without it restores a database
full of unreadable secrets. The header says so, and backups/ and *.tar are
gitignored, because an archive is a credential.
Restore is the half that is usually never tried, so it was: a 305 MB archive of this
instance was restored over it and came back at 9 scans, 59,858 subdomains, 535,000
endpoints, 29 findings and 7 targets — identical — with the migration head intact,
the API answering and an encrypted API key still decrypting. It stops the api and
both workers first so nothing writes underneath it, and asks for the database name
before it destroys anything.
The pools could ask for more connections than the server would give. The api takes
10 + 20, each celery prefork child takes 2 + 3, and beat takes another pool: at the
concurrency of 8 this box runs that is 75 against a max_connections of 100, and at
the 12 the compose file now defaults to it is 95 — five short of the ceiling, which
fails as a checkout timeout under load and reads like a slow query.
max_connections is now set from compose with the arithmetic beside it, and the api
checks it at startup: pool_demand() against max_connections less the superuser
reserve, warned once rather than discovered during an incident. The inputs moved to
shared settings because the sum spans both processes and neither could see the
other's numbers.
Worth recording: .env on this box pins CELERY_SCAN_CONCURRENCY=8, so raising the
compose default to 12 changes new installs and not this one. It also still carries
CELERY_DEFAULT_CONCURRENCY, which nothing has read since it was renamed.
Three tests: the demand counts every pool, the shipped defaults fit the shipped
server, and an oversized pool warns rather than failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
GET /dashboard/overview was 842 ms and half of it was one statement: 493 ms of
row_number() OVER (PARTITION BY target_id, signature ORDER BY discovered_at)
across all 534,765 endpoints in the project, run on every load to answer which
scan first saw each thing. The same shape runs for all five dimensions.
DISTINCT ON asks the same question and can ride ix_endpoints_target_signature,
so Postgres sorts incrementally within each group instead of sorting the project
(measured 511 ms -> 360 ms on the raw statement, 842 ms -> 683 ms end to end,
byte-identical output). Rejected on the way: a covering index on (target_id,
signature, discovered_at, scan_id) removes the sort entirely but costs 72 MB for
another 1.4x, and raising work_mem does nothing because the plan already sorts
incrementally rather than spilling.
This is a floor, not a fix. The work is O(every row the project has ever
recorded) on every dashboard load, and it grows without bound. The honest answer
is a first-sighting flag written with the row — Class A in the project's own
derivation rule — or a cached per-scan count, since a finished scan's answer
never changes. Both are their own piece of work.
GET /surface/overview re-fetched the project's targets inside each of the five
coverage() calls and re-resolved the same scan picks for the exposures count.
The service is constructed per request, so the answers cannot go stale within
one: they are memoised on the instance. 213 ms -> 188 ms.
Four tests on what the number means rather than how it is computed: a first scan
contributes nothing new, only the scan that first saw a host is credited, an
unchanged host is never counted twice across three scans, and newness is judged
per target so a fresh target contributes nothing. Mutation-tested — crediting
the last sighting instead of the first fails three of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
