Every web asset carries the analytics, tag manager and ads accounts its
page loads (Universal Analytics folded to its account, GA4, GTM, Google
Ads, AdSense, Meta pixel, Hotjar, Clarity, Yandex, HubSpot), read at
write and by the stored-response backfill. tracking is a correlation
kind, a host grammar field and the second cross link after the
certificate, judged by the page rule so a parked or default page never
vouches for its provider's account.
generic_page kept only ASCII, so a Devanagari or Chinese title read as
empty and every hub of a non-Latin estate was the server's. It keeps
Unicode word characters now.
A new disclosure check, internal_address, names the private addresses
and internal host names a response carries: any header, the pool member
a BIG-IP persistence cookie encodes, and links or quoted values in the
page. The body is read only where an address sits in a URL or a quoted
value, and every octet must be valid, so path data and version strings
never match. A migration clears hygiene_checked so the backfill
re-evaluates stored responses.
Three places built a probe target as f"{ip}:{port}". ip_addresses holds
IPv6 — seed_resolution writes both a and aaaa records — so http_probe,
origin_probe and the vulnerability surface handed httpx and nuclei
2001:db8::1:443, which is neither a host nor a host and port. The httpx
parser then made it worse: rsplit(":", 1) turned a bare v6 address into
2001:db8: before it was stored as the asset's host.
The endpoint layer already got this right in two places with the same
inline expression. That expression is now bracketed() in shared/utils/net
alongside host_port() and split_host_port(), and all five sites use it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5roHYDYzSPRNaTkn8hSkJ
A sixth result dimension. Versions parsed from httpx tech strings and
Server headers match against a local NVD corpus, so an asset's known
CVEs are named without sending a request.
The stored CPE is not the input: 81 of 7,771 entries carry a version,
so matching on it returns every CVE ever filed against the product.
Versions come from tech and the Server banner, and the NVD pair comes
from a curated map rather than a name match — popper:popper is a 2005
webmail app, aos is a switch OS, and Apache's CVEs live under
apache:http_server, not apache:apache_http_server. An unmapped product
is reported as unmapped.
Ranges compare as zero-padded keys so the match is one SQL pass: a
letter suffix takes its own segment, trailing zeros drop, and numeric
and alphabetic schemes never order against each other. A criteria with
no version statement is discarded at load, because it means every
release. Measured against a reference implementation over 3,000,000
comparisons, residual disagreement is 0.07% and each case is the new
rule being the correct one.
Every match names what is unverified — conditional, backport,
fingerprint, coarse — and confidence is the count of those. Nothing
inferred enters the vulnerability count, triage or coverage a scanner
earned; software_cves keeps its own.
The corpus is a fourth feed, keyless, from the fkie-cad mirror of the
retired NVD JSON feeds: 2,197,146 ranges and 372,244 CVEs in 5m40s,
streamed rather than parsed whole (1,254 MB resident becomes 86 MB).
It rebuilds nightly and re-matches every covering scan, since the
corpus moves without a scan.
Exploitation coverage and the intel change feed leave Arsenal for the
dashboard; Arsenal keeps the feeds and the provider it configures.
Feed row counts, nouns and tables now come from FeedSpec — three
places had hardcoded two kinds, so a third feed rendered 0.
Port versions split out of the Server header in the migration: 223
rows become 3,037.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4NuBh4EsywioeryFZRNQR
Seventeen hardening checks evaluated inside the write from response headers,
cookies, CSP and CORS: HSTS, HTTPS redirect, framing, nosniff, CSP presence,
enforcement, inline and wildcard script sources, cookie Secure and HttpOnly,
session cookie caching, CORS origin and credentials, server and runtime
disclosure, Referrer-Policy. Each check is pass, fail or not applicable, so
every count is measured against the hosts the check applies to.
Stored per asset on http_assets and folded as a union onto the host row by one
SQL statement shared by the stage and a beat backfill that evaluates responses
stored before the checks existed. Cookies are parsed from the Set-Cookie lines
of the raw header; httpx joins repeated headers with commas and cookie expiry
dates contain commas.
Surfaces: hygiene: query field with warning, info, any and none, nine counted
leads, group-by and facet, a Web hygiene panel on the scan Overview and the
project-wide Web assets page, a host table column and hover line, a sheet
section with an HTTP Hygiene pane, a compare watched field and a report
section. No score and no vulnerability rows.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JvutSbg8hHxUMCSTMpQctW
The comment rule in CLAUDE.md is one line maximum, never narration: no
multi-line blocks, no module-header explainers, no sentences that explain or
justify the code to the reader. The tree had drifted a long way from it, and
most of the drift is mine.
Removed:
232 lines of multi-line comment blocks
203 single-line comments that argued instead of stating
154 comments joined by so/because/otherwise/rather-than
143 narrating comments in the frontend
112 test docstrings (the test name already says what it asserts)
25 test module docstrings and 25 decorative section dividers
Collapsed:
324 multi-line docstrings to one factual line
271 one-line docstrings that had a justifying clause bolted on
Kept: functional directives, section dividers, short field labels, alembic
revision headers, and the eight FastAPI route docstrings that render in /docs.
Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs.
284 python tests and 34 frontend tests pass, ruff and ruff-format clean across
ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
- Introduced new utilities for scan insights in `scan-insights.ts`, including functions for certificate state, days until expiry, and query compilation.
- Added interfaces for web asset queries, subdomain filters, and insights structures.
- Enhanced `scan-status.ts` with new constants for pagination and search debounce.
- Updated the scan results page to utilize the new insights and remove unused subdomain and asset APIs.
- Modified the `HttpAsset` model to include raw request/response data and body previews.
- Created new models for scan correlation and insights, including surface stats and attention items.
- Implemented streaming JSON parsing in the HTTPX client for memory efficiency.
- Enhanced the executor to support streaming JSON output from subprocesses.
Rebuild the scan detail view into a tabbed results page: Overview
(tech/status/web-server distributions, TLS health, top networks and
services), Web assets (rich httpx table with screenshots, status,
tech, CDN/WAF/TLS-expired badges + a full per-asset detail drawer),
Subdomains (denormalized HTTP summary), Infrastructure (IP-centric
cards correlating open ports and the hosts resolving to each IP), and
Pipeline. Assets correlate by value within the scan (subdomain.name /
http_asset.host / ip), so one IP card surfaces every co-located host.
API: add per-scan http-assets, ports and ips list+summary endpoints
plus services mirroring the subdomains pattern, and serve screenshots
through an authenticated /media/screenshot route; mount scan_media
into the api container so it can read worker-written captures.
Fix two serializers that silently dropped fields: ScanRead omitted
http_assets_found (scan detail and list 500'd), and the subdomain
serializer never emitted the denormalized HTTP summary it stores.
Drop leftover docstrings and banner comments from the httpx client,
parser and field-map.
httpx is a goldmine and we were dropping most of it. Now:
- tools/runner/fieldmap.py: a declarative field map (single source of truth) —
adding/renaming a tool's output field is a one-line edit, reusable per tool
- httpx parser rewritten as a HTTPX_FIELDS map; captures jarm (was bugged),
cpe (CVE mapping), response_time, words/lines, body+header hashes, all A/AAAA
IPs, http2/pipeline, cdn_type, and a far richer TLS block (cipher, issuer_org,
serial, fingerprint, not_before, expired-derivation)
- HttpAsset gains ~21 columns for all of it
- the primary web service summary (status/title/webserver/response_time/
content_length/tech/cdn/screenshot) is denormalized onto the Subdomain row so
the subdomain table renders 2.x-style without joins; HttpAsset keeps full
per-port detail
HttpAsset.ip was storing httpx's request host (the hostname), not the
resolved IP, which broke the http-asset <-> IpAddress/Port value-join. Now:
- host = the probed host (from httpx `input`) -> exact match to Subdomain.name
- ip = the resolved IP (host_ip / A record) -> exact match to Port / IpAddress
- final_url = where a followed redirect landed, kept separate from the probed
url so the correlation key never drifts
Adds http_assets.final_url.
The host:port -> live HTTP asset convergence. After discovery, a scan now
runs expansion:
- port_scan: naabu (cdn-gated, well-known service names) -> Port
- cdn_check: cdncheck flags CDN/WAF/cloud IPs (ip family)
- http_probe: httpx probes the host:port matrix; one pass enriches each
service with status/title/tech/server/TLS/cdn/jarm/favicon/hash ->
HttpAsset, with asn backfilled from the resolved IP
httpx -tls-grab carries cert data inline (no separate tlsx pass); nmap is
dropped — naabu for ports, httpx for HTTP service id.
Also: every engine moved to its own folder; net_options() threads
proxy/headers/user-agent into every tool; rengine-tools no longer installs
top-level modules that shadowed libs (httpx); tool binaries resolve from the
go bin ahead of venv console scripts. New Port + HttpAsset models + migration
and the http_assets headline rollup.
Yogesh Ojha