- Store each run's row, first-seen and retired counts (scan_deltas, scan_retired), valid while
the revision stamp matches scan_revisions; statement triggers on the seven result tables bump
it. Finalize warms, a beat backfill fills the rest, a live run answers from its last count and
queues a refresh. /scans 1.5 s -> 43 ms, dashboard overview 5-10 s -> 0.9 s.
- lead_cache holds a live scope for 20 s with a stale copy, and every build is single-flight;
search pages opt out.
- Leads: lowercase subdomain names (CHECK) and LIKE for host:, a guard for empty JSON arrays,
branches as MATERIALIZED CTEs read to the cap, every count in one statement.
- Addresses and services: the derived CTE as one grouped pass, rows and count in one execution
(page_rows), GIN lookups for page hosts, facets through grouping sets.
- Correlation carriers counted by EXISTS; takeover latest-sighting in SQL, no silent cap.
- Screenshot backfill partial index, nine unused low-cardinality indexes dropped, hostname
helpers memoised, vhost names normalised.
After the match, every inferred CVE row gets fixed_in: the nearest newer
release of the same product this scan saw that does not carry the CVE,
and how many assets run it. Each release's CVE set is read from the NVD
corpus, uncapped, so a scan at the match cap is still judged whole. A
version whose leading number sits far above the product's median is a
misread and never the fix. is:fixable filters on it, and the row and the
sheet name the release.
Every result and management table gains multi-row selection and the floating
action bar targets and scans already had: copy, export selection, delete, and
the dimension's own actions. Rescan and triage move off the inline strip, which
now only offers the rescan that acts on the whole query.
Delete is dimension-local. A row takes only what cannot stand without it: a web
asset its stored responses, a secret its sightings. Endpoints, services and
findings on a deleted host are kept, so every count stays equal to its table.
POST /surface/delete is the one route and the scope is a WHERE, so an id from
another scan deletes nothing. Vulnerabilities are addressable by template_id so
the Issues view deletes a whole check. Exposures dismiss rather than delete: an
exposure row is a web asset.
Selection export runs through the export pipeline. Every surface filter gains
ids, capped by MAX_SELECTED_ROWS.
Secrets and software become exportable, which fixes two breaks: a bundle export
raised KeyError on the secrets dimension, and mcp read a SURFACE_COLUMNS entry
for secrets that did not exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XyDpdVXL7YVsr5D7BVj2zt
A full static review of the 3.0 tree, run as two fan-out passes with every
finding verified against the running stack rather than taken on inspection.
Correctness and security
- instance settings encrypted the mask over the stored AI key: `!= MASK` never
matched `••••••••abcd`, so saving settings destroyed the key and `test_ai`
posted the mask as a bearer token
- the MFA token was replayable for its full five-minute life; the gate now
checks revocation and consumes the token after a successful exchange
- logout revoked nothing for a bearer client while answering "Logged out"
- `POST /api-keys/{id}/test` returned the plaintext key inside httpx's error
string, which embeds the whole URL
- an interest rule could be edited or deleted through another project's id
- redis ran the celery broker under `allkeys-lru`, free to evict a queued task
or a chord counter; now `noeviction`
- nuclei's stored command carried the OAST token and every auth header
- a finalize step could poison the session and leave a scan RUNNING for ever
- `is:issue` raised TypeError on every search
- offset paging repeated and skipped rows in four dimensions
- a cancelled run kept a stale `scans.*_found` rollup
- every PDF report raised `KeyError: 'software'`
- `version_key` truncated a wide numeric run and inverted release ordering
- psycopg raises a NUL as a bare ValueError, above StatementError, so every
savepoint guard was blind to the case it was written for
Performance
- endpoints.host had no trigram index: 35.9ms seq scan -> 0.389ms
- free text compiled to one OR, not the prescribed UNION ALL: 132ms -> 41ms
- `/http-assets/summary` and `/ips/summary` materialised whole tables
- the cross-link `ip` kind ran an estate query per page and drew nothing
Frontend
- `$effect` loaders that read the state they write, an unbounded request loop
on a failing endpoint
- error states that rendered as empty states
- 2FA backup codes were unusable while the copy said they worked
- a "Scan history" dialog that always reported none
- ~400 duplicated lines across five filter bars, now one component
Build and tooling
- every scanner installed at @latest; all 13 pinned to the measured versions
- CI built `web/`, gone since 2.x, with tags that cannot be valid
- the prettier pre-commit hook discarded its filenames and always passed
- the test suite destroyed itself when run twice
Also: subdomains from public code, via a pinned github-subdomains source and a
GitHub API key that subfinder receives too; sublist3r and oneforall removed
after measuring one unique host in 89 across four targets.
Verified: 853 tests, ruff clean over eleven packages, svelte-check 0 errors,
the count-is-a-promise gate at 131 checks and 0 mismatches, and 143 GET routes
with no 5xx. Four migrations, each with an exercised downgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVkdx5ynZ6aof5kLFczmhX
A search filter took any JSON object, so a client sending the wrong
field names got the whole scope back with no error and no sign anything
was ignored. The repo's own lead-cache test was built on it: it passed
page and size, which SubdomainFilter has never had. Every dimension
filter now forbids an unknown field, the way ScanCreate already does,
and the test uses the real offset and limit.
Two more while here: nuclei's _note_gaps had been copied inline into
_one and left behind unused, so the copy is gone; and the launch preview
now says when a target already has a run in flight, since nothing
stopped two scans pointing at one host from the same address.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XxW9yfg1SEchtg6HLGHRkK
One evidence axis runs across both findings dimensions
(shared/definitions/evidence.py): inferred < observed < corroborated <
proven. Software rows store inferred or corroborated, set in match_scan from
a host and CVE join to non-info, non-suppressed findings; vulnerabilities
store observed or proven, proven being an out-of-band interaction recorded at
write. Both grammars gain an evidence: field and the old is:proven becomes
is:recorded. GET /cves and GET /cves/{cve} answer "am I affected by X" across
both dimensions, with an MCP tool and a toolbox lookup beside them. Ladder
counts are count(*) FILTER over the same predicates the searches compile, so
a rung opens exactly the rows it counted; tests/grammar/test_evidence.py
asserts it. match_scan carries discovered_at forward through a
software_previous temp table and returns the newly exposed delta for the
nightly notification; nvd_corpus.load skips the download when the mirror
stamp is unchanged.
The Attack surface group had grown to eight items with three of them sharing
a word. Vulnerabilities is now one entry parenting three sibling routes
joined by a tab rail: Findings, Software and CVEs. The routes are unchanged,
so vuln_q and sw_q state, deep links and per-route caching survive, and
NavItem.match keeps the parent lit on the siblings. The Software dimension
label drops to "Software" while its rows keep the noun "software CVE".
The CVE index gets the layout its siblings have: coverage strip, a search bar
attached to the card, a severity rail, a filter row with sort and refresh,
ListHeader rows and pagination. components/search-bar.svelte is the
plain-text twin of QueryBar for surfaces with no grammar. GET /cves gained
page, size, sort, dir, severity, kev, ransomware and evidence and returns
severity_counts; severity and the top rung are resolved in SQL from
SEVERITY_RANK and EVIDENCE_RANK rather than in Python.
Counts verified against the rows they open: overview.cves 1,165 equals the
index total, the Software tab 91,039 equals coverage.findings, and the
severity rail 157/373/583/52 sums to 1,165. _INDEX_TOTAL now excludes the
triage-suppressed findings the listing already excluded, and the software
dimension is counted exactly rather than capped at COUNT_CAP, so the tab
matches the page. The distinct-CVE count is 10s with JIT on and 91ms under
SET LOCAL jit = off, leaving the overview at 137ms cold and 17ms warm.
Also fixes software-table, which passed a 1-based page to ResultsPagination
where every other table passes a 0-based one, so page one read
"Showing 51-100".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MirAGiQ2fSJJtikWi6YRNj
A sixth result dimension. Versions parsed from httpx tech strings and
Server headers match against a local NVD corpus, so an asset's known
CVEs are named without sending a request.
The stored CPE is not the input: 81 of 7,771 entries carry a version,
so matching on it returns every CVE ever filed against the product.
Versions come from tech and the Server banner, and the NVD pair comes
from a curated map rather than a name match — popper:popper is a 2005
webmail app, aos is a switch OS, and Apache's CVEs live under
apache:http_server, not apache:apache_http_server. An unmapped product
is reported as unmapped.
Ranges compare as zero-padded keys so the match is one SQL pass: a
letter suffix takes its own segment, trailing zeros drop, and numeric
and alphabetic schemes never order against each other. A criteria with
no version statement is discarded at load, because it means every
release. Measured against a reference implementation over 3,000,000
comparisons, residual disagreement is 0.07% and each case is the new
rule being the correct one.
Every match names what is unverified — conditional, backport,
fingerprint, coarse — and confidence is the count of those. Nothing
inferred enters the vulnerability count, triage or coverage a scanner
earned; software_cves keeps its own.
The corpus is a fourth feed, keyless, from the fkie-cad mirror of the
retired NVD JSON feeds: 2,197,146 ranges and 372,244 CVEs in 5m40s,
streamed rather than parsed whole (1,254 MB resident becomes 86 MB).
It rebuilds nightly and re-matches every covering scan, since the
corpus moves without a scan.
Exploitation coverage and the intel change feed leave Arsenal for the
dashboard; Arsenal keeps the feeds and the provider it configures.
Feed row counts, nouns and tables now come from FeedSpec — three
places had hardcoded two kinds, so a third feed rendered 0.
Port versions split out of the Server header in the migration: 223
rows become 3,037.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4NuBh4EsywioeryFZRNQR
Yogesh Ojha