A scan row whose execution_config was missing target_value/target_type
raised a pydantic ValidationError inside the pagination transformer, so
one bad row returned 500 for the entire /scans page. _resolved_config
now drops the offending keys, fills the identity fields and rebuilds,
logging a warning instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Wg6h3FMyybV7ZFGWrEou5
- Integrated Hint component to display tooltips for scan steps, IP addresses, service rows, and other elements across multiple Svelte components.
- Enhanced accessibility and usability by providing contextual information on hover for elements such as scan engines, private addresses, and filter options.
- Updated relevant components including live-scan-card, ip-hover-card, ip-row, origin-dialog, attention-panel, hero-panel, ranked-list, query-suggestions, service-row, group-list, web-asset-detail-sheet, asset-row, scan-filters, scan-list-item, and others.
bits-ui's scroll content is min-width:fit-content, so a row sizes to its
columns, not to the scrollport. Measured on the Services table at a 1078px
viewport: the row was 1264px and the ... cell sat at x=1497-1553, entirely
outside the visible area. And when the columns are narrower than the port, the
capped grow column leaves the slack *after* the last cell, so ... floated in
from the right instead of sitting on the edge.
The actions cell is now sticky right-0 with ml-auto, over an opaque bg-card
plate carrying the row's own tone, so it lands on the edge with free space and
stays there under scroll. Shared as ACTIONS_PIN/ACTIONS_BODY in table/columns
with rowTone/pinTone, replacing the tone ternary that was copied verbatim into
all three rows.
Measured after (header vs row, every column): drift 0, pin delta 0 on
web-assets, services and ips.
Services table, from the same review:
- sensitive is the endpoint's own colour now, not a badge beside it
- the service logo leads the row in a class-tinted 24px tile; the 6px dot it
replaces was carrying the same information invisibly
- Software gets w-56 and wraps instead of truncating, so a full version string
reads: OpenSSH_9.6p1 Ubuntu-3ubuntu13.18, MySQL 5.7.33-0ubuntu0.16.04.1
- one host prints no count; the hostname is already right there
- host chips size to the column instead of a 128px cap
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
The connector between the two screenshots was a constant, so a default-vhost
finding — whose entire claim is that the address returns different content —
was labelled "Same application" directly above the evidence row reading
"Response body differs". It is now per-kind, next to the title and summary it
has to agree with:
origin_exposed Same application
default_vhost Different content
Also gives the panel summary its noun, so both clauses count the same kind of
thing: "1 reachable origin · 2 addresses serving a different site".
Verified against procam.in: panel reads "1 address serving a different site",
dialog reads DIFFERENT CONTENT between accreds.procam.in and 34.196.18.76.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
Written fast, and it showed.
same app -> Same application
Possible -> Medium confidence
Address serves a different site than
its hostname -> Address serving a different site
"... so its protections can be bypassed
by requesting the address directly" -> "... Requests sent to the address
are not filtered by the CDN."
"... so another application is published
on the same port" -> dropped; the sentence states what
was observed and stops
N divergent addresses -> N serving a different site
Attribute CDNs and cloud -> Attribute CDN and cloud providers
live HTTP, tech and titles -> live HTTP, technologies and titles
to see what it serves -> and record what it serves
Finding titles are noun phrases like the ones already in the attention panel,
summaries state what was observed rather than narrating the consequence to the
reader, and four spellings of "stop after this many X" are now one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
Requesting an address with no hostname commonly returns 301, which is the
server asking for a hostname rather than publishing something else on the
port. Counting it as divergent produced two findings on the validation scan
that no operator would act on. The divergence rule now requires the address
to serve a page of its own, which leaves the real case: an address answering
200 with a directory listing its hostname does not serve.
Verified end to end on a live scan: origin_probe requested 159 address and
port pairs, 58 answered, 35 addresses probed, and the service table still
reports the hostname rather than the default virtual host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
The origin probe stores the address as a host, so the same origin surfaced
twice when it was also reachable by name. The named finding is the useful one;
the address-hosted duplicate is dropped when a name exists for that address.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
The origin probe stores assets whose host is the address itself, and web_top
picked its representative by URL, so an address-hosted row would have won on
sort order and the Services table would show the default virtual host's title
in place of the real one. Named hosts now sort first, and the count of names
served on a port ignores the address itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
Every HTTP service already carries five identities — TLS certificate, body
hash, favicon hash, header hash and JARM — and they were only ever used to
cluster hostnames. Pointed at ip:port instead, one index answers the question
pentesters ask first: which of these addresses answers directly as well as
behind the CDN?
A new origin_probe stage requests each address by IP with no hostname, so the
default virtual host is captured with the same fingerprinting as everything
else; screenshot moved to the level after it so both sides of the comparison
have an image. The correlation joins fingerprints across the CDN boundary and
reports the address, the hostnames it fronts for and what matched.
The noise filter is the whole trick, as it was for related domains. A
fingerprint's discriminating power falls off with its reach: a body hash on
two hosts identifies an application, a wildcard certificate on eighteen only
identifies the organisation, so weight drops with the number of hostnames
sharing it and the label says so. Error and near-empty bodies are excluded,
one weak signal is never a finding, and the sample shown as proof is the one
sharing the most identity rather than an arbitrary member of the group — which
is what makes the two screenshots line up.
Found on existing scan data with no re-scan: labs-prod-origin-<redacted-suffix>
answering on an AWS address with the same certificate, body and favicon as
eighteen CloudFront-fronted hostnames.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
cdn_check and passive_ports run at the same level and both call
ip_inventory.ensure(), which on a domain scan is the first thing to write
ip_addresses — so both insert the same rows at the same moment. The row order
came from a UNION with no ORDER BY, so the two statements could take row locks
in different sequences and deadlock, failing a stage for no reason. Both
inventories now insert in a deterministic key order.
A failed facets request also overwrote the tab badge with 0, which reads as
"this scan found nothing" rather than "the request failed". Only a successful
response may restate the count now, in the Services and IPs tables alike.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
Three findings from reading the feature against itself.
Wrong description on 8 curated ports. describe() consulted _BY_NAME before
the port's own entry, and _BY_NAME keeps one spec per name, so port 2376
reported "Docker API without TLS, equivalent to root on the host" — the
opposite of what it is, and the kind of claim an operator acts on. The
port's own entry now wins unless a banner named something the port does not
imply, which keeps 2222/ssh resolving to SSH. Checked exhaustively: all 73
curated ports now return their own description and class.
is:new was quadratic. service_has_baseline correlated per row on target_id
alone, giving Postgres nothing selective to join on: 41.5M rows removed by
join filter on a 6,400-service scan. Whether an earlier scan exists is a
scan-level fact, so it is now a non-correlated EXISTS evaluated once as an
InitPlan, and the API's page-level check shares the same predicate instead of
carrying a second copy in raw SQL. On 24,897 subdomains x 6,440 services with
a populated baseline: is:new 5.5s -> 0.72s, findings 4.6s -> 1.7s.
The naabu replace left a hole. clear() committed, naabu then ran for minutes,
and only then did the rows come back, so a scan in progress showed an empty
port table for the length of the scan. Delete and insert now share one
transaction inside upsert().
Also removed SERVICE_CLASS_HELP, PORT_PROFILE_KEYS and is_sensitive(), which
had no consumers.
Measured and rejected: replacing the hosts CTE with per-address GIN lookups
(44.8s vs 191ms), and an ix_ports(target_id, ip, number) index (Postgres picks
a hash anti-join either way, 5.4ms with and without).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
Conversational phrasing had crept into the strings this feature added.
Section labels are noun phrases again ("Description", not "What this is";
"Service classes", not "What is listening"), and the copy states the fact
rather than narrating it:
Not confirmed -> Unconfirmed
Known exposure -> External scanner
Named -> Identified
N web ports open with nothing answering -> N web ports with no HTTP response
services are not a web server -> services outside the web surface
web services answer off port 80 and 443 -> web services on non-standard ports
addresses that sit behind a CDN -> addresses fronted by a CDN
the packet rate, not this, is what the target feels
-> the packet rate governs load on the target
The query vocabulary gets the same treatment, since field, flag and example
descriptions render straight into the syntax sheet, autocomplete and the
findings dialog. Stage title "Known Exposure" is now "Passive Port Discovery",
which says what the stage does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
An open port with no name is a dead end. The IANA Service Name and Transport
Protocol Port Number Registry (public domain, no key) fills the ~5,800 TCP
ports our curated map does not cover, committed as a generated data file
because it is small and slow-moving; scripts/fetch_port_registry.py rebuilds it.
IANA is the fallback vocabulary, never the authority. It still calls 9200
wap-wsp and 5601 esmagent while the internet runs Elasticsearch and Kibana
there, so WELL_KNOWN wins and gained ASM-framed descriptions of its own —
"Docker API without TLS, equivalent to root on the host" beats "Docker REST
API (plain text)". 80, 443, kibana, nessus, consul, kubelet and rabbitmq are
now named rather than falling through to a generic "http".
Provenance is carried, not hidden: describe() returns (text, from_registry)
and a registered name is presented as "IANA registration for port N. This scan
did not confirm it." — because IANA says port 12345 is a chat system. A name
the UI shows is always a name `service:<name>` can find, so the registry
writes through service_for_port into the stored column and a migration
backfills rows written before it existed.
Services also get an identity mark: the vendored Wappalyzer logo when the
service is a product, the class glyph when it is a protocol.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
Detection without a delta is an inventory, not an ASM. Finalize now counts
the ports this scan is the first to see for its target and publishes a
notification, escalated to warning severity when any of them is an
administrative or datastore port.
Both delta notifications moved into `_notify_deltas` so finalize stays
under the statement budget.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
An ASM is a change-detection engine, so the service inventory needs a
baseline. `is:new` compares (ip, port) against earlier scans of the same
target and only fires when a baseline exists — a target's first scan
reports zero new services rather than marking everything new, the same
contract the host grammar already keeps.
Rows carry the evidence as `is_new` (one page-scoped lookup, not a
correlated subquery per row), shown as a badge in the row and the detail
sheet, with a New quick filter and two counted leads under Change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
The unit of exposure is now a service — (ip, port) — not a bare open port.
Pipeline (EXPANSION): L0 produce hosts, L1 cdn_check + passive_ports,
L2 port_scan, L3 http_probe, L4 screenshot/waf_detect/service_fingerprint.
- cdn_check applies to every target type and records cdn_type, so a domain
scan finally knows which addresses are CDN-fronted. cdn/waf is an edge you
do not own (edge ports only); cloud is your own machine (scanned in full).
- port_scan decides and records a per-address scan_policy (full/web/skip)
honouring the scan context's excluded_ips, and offers port profiles,
exclusions, scan type, a tarpit threshold and an address budget.
- passive_ports reads Shodan internetdb through naabu -passive: keyless,
touches_target=False, so a passive-intensity scan still gets exposure data.
- http_probe seeds its host x port matrix from open ports and folds every
live response back onto the port (is_http, tls, product).
- service_fingerprint names the software behind non-web ports with a pure
Python banner prober (tools/banner/), including a SOCKS5 CONNECT so it
honours the context proxy. naabu -sV is unimplemented.
- shared/definitions/ports.py is the one vocabulary; ip_inventory and
port_inventory are the only writers of ip_addresses/ports.
- open_ports_found is measured from the table (DERIVED_COUNTS): four stages
write ports, so summing stage results double-counted.
- naabu failures no longer report zero open ports as success.
Surfaces: a Services tab (SERVICE_QUERY grammar, 16 fields, 18 counted
leads, 9 group dimensions, detail sheet) built on the shared results shell,
and an outlier-first Exposure panel on the Overview. Every count is a
promise — leads, groups and panel pivots were verified to equal the row
count they land on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
- Deleted `tech-stack-card.svelte` and `related-domains-card.svelte` as they are no longer needed.
- Consolidated scan overview components by integrating functionality into `scan-overview.svelte`, including the addition of `HeroPanel`, `AttentionPanel`, and `PosturePanel`.
- Removed `ScanActivityTimeline` and `ScanCommandLog` from the scan detail view, simplifying the UI.
- Updated scan insights handling to improve loading states and error management.
- Adjusted scan progress messages for clarity.
- Added `single_source` field to `SubdomainInsights` model to track source count.
- Add a new component for displaying query examples in the query bar.
- Refactor query help to include sections for examples, grammar, fields, and flags.
- Improve query highlighting with better styling for key segments.
- Update query suggestions to include examples and a help button for syntax guidance.
- Modify field suggestions to prioritize based on user input and description relevance.
- Adjust caret context logic to handle edge cases in query input.
- Enhance the layout of the scan page to accommodate dynamic tab heights.
- Added `newOnly` filter to `WebAssetQuery` and updated related functions.
- Introduced `FilterChip` interface for better query chip management.
- Implemented `activitySummary` function to summarize scan activity results.
- Enhanced `stageRows` function to include detailed activity states and summaries.
- Updated `SubdomainInsights` and `IpGroupRead` models to include new fields for better data representation.
- Added `SENSITIVE_PORTS` definition for sensitive port tracking.
- Improved scan page UI with new tab definitions and enhanced user interactions.
- Implemented history loading for scans to track previous results.
- Added functionality to cancel ongoing scans with user confirmation.
- Updated the engine management UI in `+page.svelte` to improve user experience with new controls for filtering stages, toggling side panels, and managing YAML configurations.
- Introduced new components such as `EngineSummaryBar` and `Resizable` for better layout handling.
- Added clipboard functionality for copying YAML configurations and formatting options.
- Enhanced the backend models in `scan_engine.py` to include a new `PreviewResolved` class for better handling of scan context and parameters.
- Updated presets in `presets.py` to clarify descriptions and improve user understanding of stage settings.
- Deleted the `scan-insights.svelte` component as it was no longer needed.
- Removed associated state management for scan changes and change window in the scans store.
- Updated the `scan-list-item.svelte` to include progress indicators for live scans.
- Introduced a new `scan-status-tabs.svelte` component for better status management.
- Enhanced the chart tooltip to support footer rendering.
- Updated utility functions for scan progress and status to accommodate new features.
- Modified the scan model to include a new field for counting new subdomains.
- Adjusted the main scans page layout to integrate the new activity chart and status tabs.
- Implemented TargetEnrichmentStage for enriching targets with DNS, WHOIS, and BGP context.
- Added VhostStage for virtual host discovery using host-header fuzzing.
- Introduced WafDetectStage for fingerprinting web application firewalls.
- Created configuration files for each new stage with relevant parameters.
- Updated Dockerfile to copy stages instead of engines.
- Enhanced existing tools and services for better integration with new stages.
- Refactored task management in Celery to accommodate new stages and improve error handling.
