- Introduced new utilities for scan insights in `scan-insights.ts`, including functions for certificate state, days until expiry, and query compilation.
- Added interfaces for web asset queries, subdomain filters, and insights structures.
- Enhanced `scan-status.ts` with new constants for pagination and search debounce.
- Updated the scan results page to utilize the new insights and remove unused subdomain and asset APIs.
- Modified the `HttpAsset` model to include raw request/response data and body previews.
- Created new models for scan correlation and insights, including surface stats and attention items.
- Implemented streaming JSON parsing in the HTTPX client for memory efficiency.
- Enhanced the executor to support streaming JSON output from subprocesses.
Rebuild the scan detail view into a tabbed results page: Overview
(tech/status/web-server distributions, TLS health, top networks and
services), Web assets (rich httpx table with screenshots, status,
tech, CDN/WAF/TLS-expired badges + a full per-asset detail drawer),
Subdomains (denormalized HTTP summary), Infrastructure (IP-centric
cards correlating open ports and the hosts resolving to each IP), and
Pipeline. Assets correlate by value within the scan (subdomain.name /
http_asset.host / ip), so one IP card surfaces every co-located host.
API: add per-scan http-assets, ports and ips list+summary endpoints
plus services mirroring the subdomains pattern, and serve screenshots
through an authenticated /media/screenshot route; mount scan_media
into the api container so it can read worker-written captures.
Fix two serializers that silently dropped fields: ScanRead omitted
http_assets_found (scan detail and list 500'd), and the subdomain
serializer never emitted the denormalized HTTP summary it stores.
Drop leftover docstrings and banner comments from the httpx client,
parser and field-map.
httpx is a goldmine and we were dropping most of it. Now:
- tools/runner/fieldmap.py: a declarative field map (single source of truth) —
adding/renaming a tool's output field is a one-line edit, reusable per tool
- httpx parser rewritten as a HTTPX_FIELDS map; captures jarm (was bugged),
cpe (CVE mapping), response_time, words/lines, body+header hashes, all A/AAAA
IPs, http2/pipeline, cdn_type, and a far richer TLS block (cipher, issuer_org,
serial, fingerprint, not_before, expired-derivation)
- HttpAsset gains ~21 columns for all of it
- the primary web service summary (status/title/webserver/response_time/
content_length/tech/cdn/screenshot) is denormalized onto the Subdomain row so
the subdomain table renders 2.x-style without joins; HttpAsset keeps full
per-port detail
HttpAsset.ip was storing httpx's request host (the hostname), not the
resolved IP, which broke the http-asset <-> IpAddress/Port value-join. Now:
- host = the probed host (from httpx `input`) -> exact match to Subdomain.name
- ip = the resolved IP (host_ip / A record) -> exact match to Port / IpAddress
- final_url = where a followed redirect landed, kept separate from the probed
url so the correlation key never drifts
Adds http_assets.final_url.
The host:port -> live HTTP asset convergence. After discovery, a scan now
runs expansion:
- port_scan: naabu (cdn-gated, well-known service names) -> Port
- cdn_check: cdncheck flags CDN/WAF/cloud IPs (ip family)
- http_probe: httpx probes the host:port matrix; one pass enriches each
service with status/title/tech/server/TLS/cdn/jarm/favicon/hash ->
HttpAsset, with asn backfilled from the resolved IP
httpx -tls-grab carries cert data inline (no separate tlsx pass); nmap is
dropped — naabu for ports, httpx for HTTP service id.
Also: every engine moved to its own folder; net_options() threads
proxy/headers/user-agent into every tool; rengine-tools no longer installs
top-level modules that shadowed libs (httpx); tool binaries resolve from the
go bin ahead of venv console scripts. New Port + HttpAsset models + migration
and the http_assets headline rollup.
Yogesh Ojha