Every HTTP service already carries five identities — TLS certificate, body
hash, favicon hash, header hash and JARM — and they were only ever used to
cluster hostnames. Pointed at ip:port instead, one index answers the question
pentesters ask first: which of these addresses answers directly as well as
behind the CDN?
A new origin_probe stage requests each address by IP with no hostname, so the
default virtual host is captured with the same fingerprinting as everything
else; screenshot moved to the level after it so both sides of the comparison
have an image. The correlation joins fingerprints across the CDN boundary and
reports the address, the hostnames it fronts for and what matched.
The noise filter is the whole trick, as it was for related domains. A
fingerprint's discriminating power falls off with its reach: a body hash on
two hosts identifies an application, a wildcard certificate on eighteen only
identifies the organisation, so weight drops with the number of hostnames
sharing it and the label says so. Error and near-empty bodies are excluded,
one weak signal is never a finding, and the sample shown as proof is the one
sharing the most identity rather than an arbitrary member of the group — which
is what makes the two screenshots line up.
Found on existing scan data with no re-scan: labs-prod-origin-<redacted-suffix>
answering on an AWS address with the same certificate, body and favicon as
eighteen CloudFront-fronted hostnames.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
An open port with no name is a dead end. The IANA Service Name and Transport
Protocol Port Number Registry (public domain, no key) fills the ~5,800 TCP
ports our curated map does not cover, committed as a generated data file
because it is small and slow-moving; scripts/fetch_port_registry.py rebuilds it.
IANA is the fallback vocabulary, never the authority. It still calls 9200
wap-wsp and 5601 esmagent while the internet runs Elasticsearch and Kibana
there, so WELL_KNOWN wins and gained ASM-framed descriptions of its own —
"Docker API without TLS, equivalent to root on the host" beats "Docker REST
API (plain text)". 80, 443, kibana, nessus, consul, kubelet and rabbitmq are
now named rather than falling through to a generic "http".
Provenance is carried, not hidden: describe() returns (text, from_registry)
and a registered name is presented as "IANA registration for port N. This scan
did not confirm it." — because IANA says port 12345 is a chat system. A name
the UI shows is always a name `service:<name>` can find, so the registry
writes through service_for_port into the stored column and a migration
backfills rows written before it existed.
Services also get an identity mark: the vendored Wappalyzer logo when the
service is a product, the class glyph when it is a protocol.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
An ASM is a change-detection engine, so the service inventory needs a
baseline. `is:new` compares (ip, port) against earlier scans of the same
target and only fires when a baseline exists — a target's first scan
reports zero new services rather than marking everything new, the same
contract the host grammar already keeps.
Rows carry the evidence as `is_new` (one page-scoped lookup, not a
correlated subquery per row), shown as a badge in the row and the detail
sheet, with a New quick filter and two counted leads under Change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
The unit of exposure is now a service — (ip, port) — not a bare open port.
Pipeline (EXPANSION): L0 produce hosts, L1 cdn_check + passive_ports,
L2 port_scan, L3 http_probe, L4 screenshot/waf_detect/service_fingerprint.
- cdn_check applies to every target type and records cdn_type, so a domain
scan finally knows which addresses are CDN-fronted. cdn/waf is an edge you
do not own (edge ports only); cloud is your own machine (scanned in full).
- port_scan decides and records a per-address scan_policy (full/web/skip)
honouring the scan context's excluded_ips, and offers port profiles,
exclusions, scan type, a tarpit threshold and an address budget.
- passive_ports reads Shodan internetdb through naabu -passive: keyless,
touches_target=False, so a passive-intensity scan still gets exposure data.
- http_probe seeds its host x port matrix from open ports and folds every
live response back onto the port (is_http, tls, product).
- service_fingerprint names the software behind non-web ports with a pure
Python banner prober (tools/banner/), including a SOCKS5 CONNECT so it
honours the context proxy. naabu -sV is unimplemented.
- shared/definitions/ports.py is the one vocabulary; ip_inventory and
port_inventory are the only writers of ip_addresses/ports.
- open_ports_found is measured from the table (DERIVED_COUNTS): four stages
write ports, so summing stage results double-counted.
- naabu failures no longer report zero open ports as success.
Surfaces: a Services tab (SERVICE_QUERY grammar, 16 fields, 18 counted
leads, 9 group dimensions, detail sheet) built on the shared results shell,
and an outlier-first Exposure panel on the Overview. Every count is a
promise — leads, groups and panel pivots were verified to equal the row
count they land on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
- Deleted `tech-stack-card.svelte` and `related-domains-card.svelte` as they are no longer needed.
- Consolidated scan overview components by integrating functionality into `scan-overview.svelte`, including the addition of `HeroPanel`, `AttentionPanel`, and `PosturePanel`.
- Removed `ScanActivityTimeline` and `ScanCommandLog` from the scan detail view, simplifying the UI.
- Updated scan insights handling to improve loading states and error management.
- Adjusted scan progress messages for clarity.
- Added `single_source` field to `SubdomainInsights` model to track source count.
- Added `newOnly` filter to `WebAssetQuery` and updated related functions.
- Introduced `FilterChip` interface for better query chip management.
- Implemented `activitySummary` function to summarize scan activity results.
- Enhanced `stageRows` function to include detailed activity states and summaries.
- Updated `SubdomainInsights` and `IpGroupRead` models to include new fields for better data representation.
- Added `SENSITIVE_PORTS` definition for sensitive port tracking.
- Improved scan page UI with new tab definitions and enhanced user interactions.
- Implemented history loading for scans to track previous results.
- Added functionality to cancel ongoing scans with user confirmation.
- Updated the engine management UI in `+page.svelte` to improve user experience with new controls for filtering stages, toggling side panels, and managing YAML configurations.
- Introduced new components such as `EngineSummaryBar` and `Resizable` for better layout handling.
- Added clipboard functionality for copying YAML configurations and formatting options.
- Enhanced the backend models in `scan_engine.py` to include a new `PreviewResolved` class for better handling of scan context and parameters.
- Updated presets in `presets.py` to clarify descriptions and improve user understanding of stage settings.
- Deleted the `scan-insights.svelte` component as it was no longer needed.
- Removed associated state management for scan changes and change window in the scans store.
- Updated the `scan-list-item.svelte` to include progress indicators for live scans.
- Introduced a new `scan-status-tabs.svelte` component for better status management.
- Enhanced the chart tooltip to support footer rendering.
- Updated utility functions for scan progress and status to accommodate new features.
- Modified the scan model to include a new field for counting new subdomains.
- Adjusted the main scans page layout to integrate the new activity chart and status tabs.
- Implemented TargetEnrichmentStage for enriching targets with DNS, WHOIS, and BGP context.
- Added VhostStage for virtual host discovery using host-header fuzzing.
- Introduced WafDetectStage for fingerprinting web application firewalls.
- Created configuration files for each new stage with relevant parameters.
- Updated Dockerfile to copy stages instead of engines.
- Enhanced existing tools and services for better integration with new stages.
- Refactored task management in Celery to accommodate new stages and improve error handling.
- Introduced new utilities for scan insights in `scan-insights.ts`, including functions for certificate state, days until expiry, and query compilation.
- Added interfaces for web asset queries, subdomain filters, and insights structures.
- Enhanced `scan-status.ts` with new constants for pagination and search debounce.
- Updated the scan results page to utilize the new insights and remove unused subdomain and asset APIs.
- Modified the `HttpAsset` model to include raw request/response data and body previews.
- Created new models for scan correlation and insights, including surface stats and attention items.
- Implemented streaming JSON parsing in the HTTPX client for memory efficiency.
- Enhanced the executor to support streaming JSON output from subprocesses.
Rebuild the scan detail view into a tabbed results page: Overview
(tech/status/web-server distributions, TLS health, top networks and
services), Web assets (rich httpx table with screenshots, status,
tech, CDN/WAF/TLS-expired badges + a full per-asset detail drawer),
Subdomains (denormalized HTTP summary), Infrastructure (IP-centric
cards correlating open ports and the hosts resolving to each IP), and
Pipeline. Assets correlate by value within the scan (subdomain.name /
http_asset.host / ip), so one IP card surfaces every co-located host.
API: add per-scan http-assets, ports and ips list+summary endpoints
plus services mirroring the subdomains pattern, and serve screenshots
through an authenticated /media/screenshot route; mount scan_media
into the api container so it can read worker-written captures.
Fix two serializers that silently dropped fields: ScanRead omitted
http_assets_found (scan detail and list 500'd), and the subdomain
serializer never emitted the denormalized HTTP summary it stores.
Drop leftover docstrings and banner comments from the httpx client,
parser and field-map.
httpx is a goldmine and we were dropping most of it. Now:
- tools/runner/fieldmap.py: a declarative field map (single source of truth) —
adding/renaming a tool's output field is a one-line edit, reusable per tool
- httpx parser rewritten as a HTTPX_FIELDS map; captures jarm (was bugged),
cpe (CVE mapping), response_time, words/lines, body+header hashes, all A/AAAA
IPs, http2/pipeline, cdn_type, and a far richer TLS block (cipher, issuer_org,
serial, fingerprint, not_before, expired-derivation)
- HttpAsset gains ~21 columns for all of it
- the primary web service summary (status/title/webserver/response_time/
content_length/tech/cdn/screenshot) is denormalized onto the Subdomain row so
the subdomain table renders 2.x-style without joins; HttpAsset keeps full
per-port detail
HttpAsset.ip was storing httpx's request host (the hostname), not the
resolved IP, which broke the http-asset <-> IpAddress/Port value-join. Now:
- host = the probed host (from httpx `input`) -> exact match to Subdomain.name
- ip = the resolved IP (host_ip / A record) -> exact match to Port / IpAddress
- final_url = where a followed redirect landed, kept separate from the probed
url so the correlation key never drifts
Adds http_assets.final_url.
The host:port -> live HTTP asset convergence. After discovery, a scan now
runs expansion:
- port_scan: naabu (cdn-gated, well-known service names) -> Port
- cdn_check: cdncheck flags CDN/WAF/cloud IPs (ip family)
- http_probe: httpx probes the host:port matrix; one pass enriches each
service with status/title/tech/server/TLS/cdn/jarm/favicon/hash ->
HttpAsset, with asn backfilled from the resolved IP
httpx -tls-grab carries cert data inline (no separate tlsx pass); nmap is
dropped — naabu for ports, httpx for HTTP service id.
Also: every engine moved to its own folder; net_options() threads
proxy/headers/user-agent into every tool; rengine-tools no longer installs
top-level modules that shadowed libs (httpx); tool binaries resolve from the
go bin ahead of venv console scripts. New Port + HttpAsset models + migration
and the http_assets headline rollup.
Every scan now runs a discovery phase, gated per seed type:
- seed_resolution: expand ip / cidr / asn seeds into IpAddress assets
(stdlib ipaddress, even-stride sampling + rfc1918 skip on big ranges)
- reverse_dns: dnsx -ptr over discovered IPs
- ip_enrichment: asn / org / country from cached bgp + whois
- target_enrichment: consolidate cached whois/dns/bgp, refresh on miss/stale
The engine framework grows a phase axis (discovery/expansion/depth) and
per-stage applies_to gating so a stage skips cleanly when it doesn't apply
to the seed; subdomain discovery is retagged as expansion. Adds the
IpAddress model + migration.
- Target delete failed when scan data existed: scans.target_id and
activity_logs.target_id FKs had no ON DELETE rule. Add ondelete CASCADE
(scans) / SET NULL (activity_logs) to the models + a migration that
alters the existing constraints. Deleting a target now cascades its
scans (and their subdomains/activities/commands) and nulls audit-log refs.
- Targets list: clicking the row checkbox bubbled to the row and navigated
to the target summary. Wrap the checkbox in a stopProp container.
- Scan history: add per-row + select-all checkboxes and a bulk action bar
(cancel running / delete selected) mirroring the targets pattern, backed
by scansStore.removeMany/cancelMany.
- Added new types and interfaces for scan scheduling in the frontend.
- Created a new Svelte page for managing scheduled scans, including listing, creating, editing, and deleting schedules.
- Implemented backend models and services for scan schedules, including handling different schedule types (one-off, interval, daily, cron).
- Introduced a Celery task to manage the execution of scheduled scans based on their defined timing.
- Enhanced notification system to include alerts for newly discovered subdomains during scans.
- Added utility functions for handling time zones and scheduling logic.
- Add aggregate functions for scan status and counts in orchestrator.
- Create ScanEventPublisher for emitting scan-related events.
- Introduce ScanActivityService for managing scan activities.
- Implement ScanCommandRecorder for logging command executions.
- Develop canvas building and finalization logic for scan tasks.
- Refactor scan task to utilize orchestrator functions for running stages and finalizing scans.
- Enhance scan notification system to dispatch relevant events.
- Integrate command recording into CLI tool execution for better tracking.
- Updated the default route for automation to redirect to engines instead of scans.
- Removed the scans page and replaced it with a new scans overview page that lists scan history for projects.
- Introduced a detailed scan view page to display individual scan results and subdomain discoveries.
- Added new enums for scan statuses and subdomain sources to improve code clarity and maintainability.
- Implemented a new scan task in Celery to handle scan execution and logging.
- Enhanced the database models for scans and subdomains to support new features and improve data integrity.
- Updated the worker Dockerfile to include necessary scanning tools and dependencies.
- Added utility functions for domain matching and pattern compilation to streamline scan operations.
- Added `clean_name` function to `validation.py` for name validation with max length.
- Simplified docstrings in `dnsx` client, parser, and service files for clarity.
- Enhanced error handling in `ViewDNSService` to prevent disabling keys on rate limits.
- Improved WHOIS service to handle concurrent inserts gracefully with integrity checks.
- Updated `Settings` class in worker config to inherit from `BaseAppSettings`.
- Added new dependencies in `pyproject.toml` for `apprise` and `cryptography`.
- Updated project slug generation to ensure uniqueness by checking active projects.
- Modified project retrieval functions to only fetch active projects.
- Introduced unique slug generation for tags associated with specific projects.
- Enhanced user authentication flow with improved error handling and session management.
- Added session expiration handling in the API client and SSE client for better user experience.
- Improved UI components for better accessibility and user feedback, including copy buttons and activity drawers.
- Refactored notification handling to ensure proper session management and state reset on logout.
- Removed unused enrichment components to streamline the codebase.
- Updated notification model to simplify JSONB handling and ensure proper expiration logic.
- Added migration commands to the new dev-README.md for creating and rolling back migrations.
- Updated docker-compose.yml to include additional volume mounts for alembic and build artifacts in development and worker services.
- Enhanced Target model in target.py to include a relationship with WhoisRecord and added WhoisRecordSummary to TargetRead for better data representation.
- Added models for WHOIS responses including WhoisDomainResponse, WhoisIPResponse, and WhoisASNResponse.
- Created a parser for WHOIS data to convert raw responses into structured models.
- Developed an RDAPProvider class to handle RDAP lookups using the whoisit library.
- Implemented a WhoisService class for performing WHOIS lookups with optional database caching.
- Added methods for finding correlations between WHOIS records based on registrant, registrar, network, country, and nameservers.
- Updated Dockerfile to include tools directory and added rengine-tools dependency in pyproject.toml.
Yogesh Ojha