Three findings from reading the feature against itself.
Wrong description on 8 curated ports. describe() consulted _BY_NAME before
the port's own entry, and _BY_NAME keeps one spec per name, so port 2376
reported "Docker API without TLS, equivalent to root on the host" — the
opposite of what it is, and the kind of claim an operator acts on. The
port's own entry now wins unless a banner named something the port does not
imply, which keeps 2222/ssh resolving to SSH. Checked exhaustively: all 73
curated ports now return their own description and class.
is:new was quadratic. service_has_baseline correlated per row on target_id
alone, giving Postgres nothing selective to join on: 41.5M rows removed by
join filter on a 6,400-service scan. Whether an earlier scan exists is a
scan-level fact, so it is now a non-correlated EXISTS evaluated once as an
InitPlan, and the API's page-level check shares the same predicate instead of
carrying a second copy in raw SQL. On 24,897 subdomains x 6,440 services with
a populated baseline: is:new 5.5s -> 0.72s, findings 4.6s -> 1.7s.
The naabu replace left a hole. clear() committed, naabu then ran for minutes,
and only then did the rows come back, so a scan in progress showed an empty
port table for the length of the scan. Delete and insert now share one
transaction inside upsert().
Also removed SERVICE_CLASS_HELP, PORT_PROFILE_KEYS and is_sensitive(), which
had no consumers.
Measured and rejected: replacing the hosts CTE with per-address GIN lookups
(44.8s vs 191ms), and an ix_ports(target_id, ip, number) index (Postgres picks
a hash anti-join either way, 5.4ms with and without).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
Conversational phrasing had crept into the strings this feature added.
Section labels are noun phrases again ("Description", not "What this is";
"Service classes", not "What is listening"), and the copy states the fact
rather than narrating it:
Not confirmed -> Unconfirmed
Known exposure -> External scanner
Named -> Identified
N web ports open with nothing answering -> N web ports with no HTTP response
services are not a web server -> services outside the web surface
web services answer off port 80 and 443 -> web services on non-standard ports
addresses that sit behind a CDN -> addresses fronted by a CDN
the packet rate, not this, is what the target feels
-> the packet rate governs load on the target
The query vocabulary gets the same treatment, since field, flag and example
descriptions render straight into the syntax sheet, autocomplete and the
findings dialog. Stage title "Known Exposure" is now "Passive Port Discovery",
which says what the stage does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
An open port with no name is a dead end. The IANA Service Name and Transport
Protocol Port Number Registry (public domain, no key) fills the ~5,800 TCP
ports our curated map does not cover, committed as a generated data file
because it is small and slow-moving; scripts/fetch_port_registry.py rebuilds it.
IANA is the fallback vocabulary, never the authority. It still calls 9200
wap-wsp and 5601 esmagent while the internet runs Elasticsearch and Kibana
there, so WELL_KNOWN wins and gained ASM-framed descriptions of its own —
"Docker API without TLS, equivalent to root on the host" beats "Docker REST
API (plain text)". 80, 443, kibana, nessus, consul, kubelet and rabbitmq are
now named rather than falling through to a generic "http".
Provenance is carried, not hidden: describe() returns (text, from_registry)
and a registered name is presented as "IANA registration for port N. This scan
did not confirm it." — because IANA says port 12345 is a chat system. A name
the UI shows is always a name `service:<name>` can find, so the registry
writes through service_for_port into the stored column and a migration
backfills rows written before it existed.
Services also get an identity mark: the vendored Wappalyzer logo when the
service is a product, the class glyph when it is a protocol.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
Detection without a delta is an inventory, not an ASM. Finalize now counts
the ports this scan is the first to see for its target and publishes a
notification, escalated to warning severity when any of them is an
administrative or datastore port.
Both delta notifications moved into `_notify_deltas` so finalize stays
under the statement budget.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
An ASM is a change-detection engine, so the service inventory needs a
baseline. `is:new` compares (ip, port) against earlier scans of the same
target and only fires when a baseline exists — a target's first scan
reports zero new services rather than marking everything new, the same
contract the host grammar already keeps.
Rows carry the evidence as `is_new` (one page-scoped lookup, not a
correlated subquery per row), shown as a badge in the row and the detail
sheet, with a New quick filter and two counted leads under Change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
The unit of exposure is now a service — (ip, port) — not a bare open port.
Pipeline (EXPANSION): L0 produce hosts, L1 cdn_check + passive_ports,
L2 port_scan, L3 http_probe, L4 screenshot/waf_detect/service_fingerprint.
- cdn_check applies to every target type and records cdn_type, so a domain
scan finally knows which addresses are CDN-fronted. cdn/waf is an edge you
do not own (edge ports only); cloud is your own machine (scanned in full).
- port_scan decides and records a per-address scan_policy (full/web/skip)
honouring the scan context's excluded_ips, and offers port profiles,
exclusions, scan type, a tarpit threshold and an address budget.
- passive_ports reads Shodan internetdb through naabu -passive: keyless,
touches_target=False, so a passive-intensity scan still gets exposure data.
- http_probe seeds its host x port matrix from open ports and folds every
live response back onto the port (is_http, tls, product).
- service_fingerprint names the software behind non-web ports with a pure
Python banner prober (tools/banner/), including a SOCKS5 CONNECT so it
honours the context proxy. naabu -sV is unimplemented.
- shared/definitions/ports.py is the one vocabulary; ip_inventory and
port_inventory are the only writers of ip_addresses/ports.
- open_ports_found is measured from the table (DERIVED_COUNTS): four stages
write ports, so summing stage results double-counted.
- naabu failures no longer report zero open ports as success.
Surfaces: a Services tab (SERVICE_QUERY grammar, 16 fields, 18 counted
leads, 9 group dimensions, detail sheet) built on the shared results shell,
and an outlier-first Exposure panel on the Overview. Every count is a
promise — leads, groups and panel pivots were verified to equal the row
count they land on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
- Added `newOnly` filter to `WebAssetQuery` and updated related functions.
- Introduced `FilterChip` interface for better query chip management.
- Implemented `activitySummary` function to summarize scan activity results.
- Enhanced `stageRows` function to include detailed activity states and summaries.
- Updated `SubdomainInsights` and `IpGroupRead` models to include new fields for better data representation.
- Added `SENSITIVE_PORTS` definition for sensitive port tracking.
- Improved scan page UI with new tab definitions and enhanced user interactions.
- Implemented history loading for scans to track previous results.
- Added functionality to cancel ongoing scans with user confirmation.
- Implemented TargetEnrichmentStage for enriching targets with DNS, WHOIS, and BGP context.
- Added VhostStage for virtual host discovery using host-header fuzzing.
- Introduced WafDetectStage for fingerprinting web application firewalls.
- Created configuration files for each new stage with relevant parameters.
- Updated Dockerfile to copy stages instead of engines.
- Enhanced existing tools and services for better integration with new stages.
- Refactored task management in Celery to accommodate new stages and improve error handling.
The host:port -> live HTTP asset convergence. After discovery, a scan now
runs expansion:
- port_scan: naabu (cdn-gated, well-known service names) -> Port
- cdn_check: cdncheck flags CDN/WAF/cloud IPs (ip family)
- http_probe: httpx probes the host:port matrix; one pass enriches each
service with status/title/tech/server/TLS/cdn/jarm/favicon/hash ->
HttpAsset, with asn backfilled from the resolved IP
httpx -tls-grab carries cert data inline (no separate tlsx pass); nmap is
dropped — naabu for ports, httpx for HTTP service id.
Also: every engine moved to its own folder; net_options() threads
proxy/headers/user-agent into every tool; rengine-tools no longer installs
top-level modules that shadowed libs (httpx); tool binaries resolve from the
go bin ahead of venv console scripts. New Port + HttpAsset models + migration
and the http_assets headline rollup.
- Added new types and interfaces for scan scheduling in the frontend.
- Created a new Svelte page for managing scheduled scans, including listing, creating, editing, and deleting schedules.
- Implemented backend models and services for scan schedules, including handling different schedule types (one-off, interval, daily, cron).
- Introduced a Celery task to manage the execution of scheduled scans based on their defined timing.
- Enhanced notification system to include alerts for newly discovered subdomains during scans.
- Added utility functions for handling time zones and scheduling logic.
- Add aggregate functions for scan status and counts in orchestrator.
- Create ScanEventPublisher for emitting scan-related events.
- Introduce ScanActivityService for managing scan activities.
- Implement ScanCommandRecorder for logging command executions.
- Develop canvas building and finalization logic for scan tasks.
- Refactor scan task to utilize orchestrator functions for running stages and finalizing scans.
- Enhance scan notification system to dispatch relevant events.
- Integrate command recording into CLI tool execution for better tracking.
- Added `clean_name` function to `validation.py` for name validation with max length.
- Simplified docstrings in `dnsx` client, parser, and service files for clarity.
- Enhanced error handling in `ViewDNSService` to prevent disabling keys on rate limits.
- Improved WHOIS service to handle concurrent inserts gracefully with integrity checks.
- Updated `Settings` class in worker config to inherit from `BaseAppSettings`.
- Added new dependencies in `pyproject.toml` for `apprise` and `cryptography`.
- Introduced a new endpoint to validate multiple target values in a single request.
- Added a constant for maximum target imports.
- Updated the target validation response model to include the target value.
- Modified Dockerfile CMD for graceful shutdown and reload directory.
- Commented out environment variable in docker-compose for polling.
Yogesh Ojha