How is the surface hosted? One three-column flow on both Overviews answers it:
resolving web assets → fronting (CDN or WAF edge · cloud provider · direct to
origin) → network (the top nine AS organisations and "Other networks"). It sits
after Vulnerabilities on the scan page and after Needs attention on the target
page, renders only when something resolves, and replaces Composition's "Hosting
networks" ranked list, which said the same thing without the fronting
relationship.
GET /subdomains/hosting-flow aggregates Subdomain LEFT JOIN a distinct-on
HttpAsset per host. Every node and link carries its query token — is:resolved,
is:cdn, is:cloud (new on the host grammar: cdn_type == 'cloud'), asn:<n>, and
"is:resolved and not (asn:… or …)" for the remainder — and the count is a
promise: on gov.cy and hamropatro every figure equals the row count its query
lands on. "Resolving" uses the grammar's own resolved predicate rather than
is_active (they differed by two), and the one expression is shared between
SELECT and GROUP BY because a second call binds a second parameter Postgres
rejects. Unresolved hosts are not a node: they have no flow and they crushed
the scale; the head line says "444 of 1,379 resolve · 26 networks" instead.
The sankey is hand-rolled (no d3-sankey; node_modules is baked into the image):
a minimum node height so a one-asset network stays legible, centred columns,
labels clipped at 30 characters with the full name in the tooltip, and a halo
so mid-column labels read over ribbons. The whole row is the hit target and
every ribbon has a wide transparent twin; nodes are Tab stops with a focus ring
and an anchored tooltip; the tooltip carries the share and the query with a Web
Assets affordance; the traced path wears a moving dash that reduced motion
turns off; the draw-in starts when the panel scrolls into view. Click pivots
into Web Assets with the query.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012YK61xuAJGMhq1uniLhHX5
The summary rebuild was a card grid: a lot of space to see nothing, the same
fact three times, and a scan table pasted in from the history page. This is the
page rebuilt from an approved mockup as an entity page — Overview · Web Assets ·
DNS · WHOIS · BGP — where the rail is the fact sheet, the main column is the
story, and every fact has one home.
Overview: a five-pill surface strip (count plus one note — added/gone,
sensitive, critical, "so far" while live, dashed "Not scanned" — with a
sparkline over the last six completed runs once two carry a value); Needs
attention tiles ranked critical → warning from the summary's risk figures,
sensitive services and the derived checks; an activity timeline of runs on a
dotted rail whose latest chips read the summary's surface values, so the
timeline never contradicts the strip; and related targets from WHOIS
correlation and certificate names. The sticky rail opens with the globe (the
IPs-covering scan's countries, every dot a pivot), then Scan · Resolution ·
Registration · Routing, with NS, MX and registrar shown as provider name plus
vendored brand logo and a failed lookup's error carried in the group note next
to Refresh.
There is no Scans tab: "All runs" opens /scans?target=<id>, which the scan
history page now reads. DNS, WHOIS and BGP are evidence tabs sharing one
grammar (record-shell + record-group): DNS grouped by type with provider logos,
TXT purpose and private-range warnings; WHOIS as registration facts → EPP status
→ nameservers → contacts; BGP as routing facts → prefixes (family filter,
search, paging) → peers → related prefixes → abuse contacts. A tab renders only
when it has data or its lookup is still pending, so a deep link is not yanked
to Overview mid-load.
WhoisRecordSummary carries target_id so a correlated registrant or nameserver
links to its target. copy-button takes a class that overrides its size and the
org/tag popovers report their patch through onChange. Removes the enrichment/,
summary/ and scan-history components of the previous version.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012YK61xuAJGMhq1uniLhHX5
`/targets/[id]` was a header plus eight tabs, five of which rendered empty
states written before the scan pipeline existed. It is now four real tabs —
Summary, Web Assets, Intelligence, Scans — wearing the scan pages' clothes:
the same sticky counted tab strip, the same Card + PanelHead hairline-cell
panels, and a Scans tab that is the scan history page scoped to the target.
The five result dimensions become one vocabulary. shared/definitions/surface.py
and frontend config/surface.ts name what a scan produces the way the UI names
it — Web Assets, Endpoints, Services, IPs, Vulnerabilities. The scan page's tab
strip, scanCountPills, targetAssetNoun and the activity chart now read from that
one definition, so no screen says "subdomains" or "open ports" any more.
A target's surface is per-dimension "last observed", not "the latest scan".
GET /targets/{id}/summary reports what the most recent scan that actually ran
each dimension found. A dimension nothing has run reads "Not scanned", never 0 —
"we looked and found nothing" and "we never looked" are different facts. Rows
are proof a stage ran, because a cancelled run writes what it found before it is
stopped without recording success, and counts come from the tables rather than
the scans.*_found rollup, which is wrong for those runs. Every figure equals the
row count you land on: 1,379 / 7,982 / 656 / 182 / 626 against the result tabs,
and 37 sensitive, 20 critical, 19 KEV against their filters.
POST /targets/{id}/assets is a SQL rollup of the target's web assets across
every scan — first seen, last seen, scans seen in, and All/Current/Gone tabs.
It is the one asset view a scan page structurally cannot give. A running scan
can never retire an asset, and is_new needs a baseline, the same rule the
query grammars use. A 24k-asset target pages in ~90ms.
Also fixed: the headline read "No change since the previous scan" above a cell
reading "11,436" because two different comparisons shared a screen; formatSeconds
never rolled minutes into hours, so a 12-hour scan read "720m 55s" here and on
the scan history page; and truncate on an inline span in a block table cell does
not clip, so a long page title spilled across three columns.
Removes target-subdomains, subdomain-table and scan-tab-empty. Promotes
panel-head.svelte to a shared primitive.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M9Gw9RvaR2m6wkZZUdfb4w
Replace the tree rail beside a flat list with one outline: host rows, folder rows and
endpoint leaves in a single grid. Kind is a glyph on the leaf, folders get a glyph for
why they matter, a folder that is only its own index renders as a leaf, and the query
bar is the tree search with matched segments marked.
Defaults are measured: hosts come from a paginated SQL rollup and load their folders
when opened, nothing is open at rest, and a search opens branches only while they fit a
40-row screen budget. Hide static is a real server filter with the count carried by the
facets so the All tab equals the rows.
The tree now carries change and judgement: new and gone counts per node against the
previous scan of the target with a Gone lens, anomaly sentences for folders that are
mostly walled with one open door or mostly fine with a few 5xx, siblings that share a
skeleton folded into one group row whose dir:[...] token lands on exactly its count,
archive-only and robots.txt markers, copy paths as a wordlist, reveal by host= path=
tokens, sticky ancestor crumbs rendered in the sticky head, type-ahead, and a verify-this-
branch action wired to the worker task.
Also: class-aware path interests (a cPanel logo is not an administrative interface),
the tilde backup rule limited to trailing tildes, endpoint_count and a paths: field on
hosts, and a null-safe static predicate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015KEfo6ipoWMYeHxThmTLRy
- Updated highlight-text component to support multiple search terms.
- Modified results-pagination component to make onPageSize optional and conditionally render PageSizeSelector.
- Integrated HostStructure component into web-asset-detail-sheet for improved asset structure visualization.
- Added HostStructureDialog to manage host structure interactions.
- Enhanced web-assets-table to handle structure-related actions and dialogs.
- Introduced new folder glyphs and updated endpoint classification logic in endpoints configuration.
- Added endpoint_count to subdomain type for better tracking of discovered endpoints.
- Implemented new endpoint filtering capabilities in the backend to support static content visibility.
- Updated models and services to accommodate new endpoint structures and classifications.
A scan no longer needs a saved engine. The launch dialog has two
configuration modes: Scan Engine (default, runs a saved engine exactly
as configured, shown as a pipeline and its YAML) and Custom Scan (pick
the capabilities to run; dependent and supporting stages are included
automatically). Targets are created on launch from typed values, the
scan context is a first-class field, and "Run again" restores the
engine or the selection that ran.
Backend
- scans.engine_id is nullable; an absent engine resolves through
AdHocEngine and the run is named after its capabilities (plan_label)
- ScanCreate/ScanBatchCreate accept target_value(s) (create or reuse via
TargetService.ensure_targets), a per-run intensity, and preview works
for a target that does not exist yet
- every stage declares consumes/produces (AssetKind), group (StageGroup)
and role (capability | support); the registry validates them and the
catalog serves them with groups and seed_produces
- stage_effects walks levels so a producer only feeds later levels and
marks an unfed consumer skipped_no_input
Frontend
- components/scans/launch/: launch-dialog, launch-state, target-picker,
plan-picker (segmented Scan Engine | Custom Scan, pipeline + YAML
Config views, capability pills with full per-stage settings),
stage-pill, stage-config-popover, launch-summary + what-runs,
launch-context-field, launch-context-form, save-engine-dialog
- utilities/launch-plan.ts resolves implied producers and following
support stages; the last engine and last custom selection persist
- quick-scan-fields offers presets and engines in one select; the
command palette offers "Scan <value>"; scan list action is "Run again"
- effect-panel drops its explanatory header; NO_CONTEXT_LABEL is
"No scan context"; launch-modal and execution-preview are removed
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014erb6Jbm6v9sxPgYFBxB7E
A finding is corroborated when another check fired at the same matched_at,
names the same CVE or CWE, and neither is informational or suppressed. The
peer carries its basis, the shared identifiers and its scanner, so a second
scanner confirming the same weakness is distinguishable from a second nuclei
template. Co-location alone no longer counts: four unrelated TLS defects on
one endpoint used to vouch for each other.
Surface it the way the grammar surfaces everything else: an is:corroborated
flag, a Confirmed quick filter, two counted example leads, a Location group
dimension, and a tiebreaker in the risk sort after severity and KEV. The row
carries a confirmed badge beside new and KEV instead of trailing text on the
origin line; the detail sheet states each peer's basis and links to the other
findings at that location.
Add a btree over (scan_id, matched_at) - the equality lookup was probing the
trigram GIN index at 8.2ms a key - and disable JIT on the vulnerability query
paths, where the set-returning expansion's 100-rows-a-row estimate bought
267ms of compilation per request. On 60k findings: search 510ms to 240ms,
issues 1380ms to 440ms (830ms before the feature existed).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZKQQx39KeiDo8QE9XXzf3
A scan row whose execution_config was missing target_value/target_type
raised a pydantic ValidationError inside the pagination transformer, so
one bad row returned 500 for the entire /scans page. _resolved_config
now drops the offending keys, fills the identity fields and rebuilds,
logging a warning instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Wg6h3FMyybV7ZFGWrEou5
Requesting an address with no hostname commonly returns 301, which is the
server asking for a hostname rather than publishing something else on the
port. Counting it as divergent produced two findings on the validation scan
that no operator would act on. The divergence rule now requires the address
to serve a page of its own, which leaves the real case: an address answering
200 with a directory listing its hostname does not serve.
Verified end to end on a live scan: origin_probe requested 159 address and
port pairs, 58 answered, 35 addresses probed, and the service table still
reports the hostname rather than the default virtual host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
The origin probe stores the address as a host, so the same origin surfaced
twice when it was also reachable by name. The named finding is the useful one;
the address-hosted duplicate is dropped when a name exists for that address.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
The origin probe stores assets whose host is the address itself, and web_top
picked its representative by URL, so an address-hosted row would have won on
sort order and the Services table would show the default virtual host's title
in place of the real one. Named hosts now sort first, and the count of names
served on a port ignores the address itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
Every HTTP service already carries five identities — TLS certificate, body
hash, favicon hash, header hash and JARM — and they were only ever used to
cluster hostnames. Pointed at ip:port instead, one index answers the question
pentesters ask first: which of these addresses answers directly as well as
behind the CDN?
A new origin_probe stage requests each address by IP with no hostname, so the
default virtual host is captured with the same fingerprinting as everything
else; screenshot moved to the level after it so both sides of the comparison
have an image. The correlation joins fingerprints across the CDN boundary and
reports the address, the hostnames it fronts for and what matched.
The noise filter is the whole trick, as it was for related domains. A
fingerprint's discriminating power falls off with its reach: a body hash on
two hosts identifies an application, a wildcard certificate on eighteen only
identifies the organisation, so weight drops with the number of hostnames
sharing it and the label says so. Error and near-empty bodies are excluded,
one weak signal is never a finding, and the sample shown as proof is the one
sharing the most identity rather than an arbitrary member of the group — which
is what makes the two screenshots line up.
Found on existing scan data with no re-scan: labs-prod-origin-<redacted-suffix>
answering on an AWS address with the same certificate, body and favicon as
eighteen CloudFront-fronted hostnames.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
Three findings from reading the feature against itself.
Wrong description on 8 curated ports. describe() consulted _BY_NAME before
the port's own entry, and _BY_NAME keeps one spec per name, so port 2376
reported "Docker API without TLS, equivalent to root on the host" — the
opposite of what it is, and the kind of claim an operator acts on. The
port's own entry now wins unless a banner named something the port does not
imply, which keeps 2222/ssh resolving to SSH. Checked exhaustively: all 73
curated ports now return their own description and class.
is:new was quadratic. service_has_baseline correlated per row on target_id
alone, giving Postgres nothing selective to join on: 41.5M rows removed by
join filter on a 6,400-service scan. Whether an earlier scan exists is a
scan-level fact, so it is now a non-correlated EXISTS evaluated once as an
InitPlan, and the API's page-level check shares the same predicate instead of
carrying a second copy in raw SQL. On 24,897 subdomains x 6,440 services with
a populated baseline: is:new 5.5s -> 0.72s, findings 4.6s -> 1.7s.
The naabu replace left a hole. clear() committed, naabu then ran for minutes,
and only then did the rows come back, so a scan in progress showed an empty
port table for the length of the scan. Delete and insert now share one
transaction inside upsert().
Also removed SERVICE_CLASS_HELP, PORT_PROFILE_KEYS and is_sensitive(), which
had no consumers.
Measured and rejected: replacing the hosts CTE with per-address GIN lookups
(44.8s vs 191ms), and an ix_ports(target_id, ip, number) index (Postgres picks
a hash anti-join either way, 5.4ms with and without).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
An open port with no name is a dead end. The IANA Service Name and Transport
Protocol Port Number Registry (public domain, no key) fills the ~5,800 TCP
ports our curated map does not cover, committed as a generated data file
because it is small and slow-moving; scripts/fetch_port_registry.py rebuilds it.
IANA is the fallback vocabulary, never the authority. It still calls 9200
wap-wsp and 5601 esmagent while the internet runs Elasticsearch and Kibana
there, so WELL_KNOWN wins and gained ASM-framed descriptions of its own —
"Docker API without TLS, equivalent to root on the host" beats "Docker REST
API (plain text)". 80, 443, kibana, nessus, consul, kubelet and rabbitmq are
now named rather than falling through to a generic "http".
Provenance is carried, not hidden: describe() returns (text, from_registry)
and a registered name is presented as "IANA registration for port N. This scan
did not confirm it." — because IANA says port 12345 is a chat system. A name
the UI shows is always a name `service:<name>` can find, so the registry
writes through service_for_port into the stored column and a migration
backfills rows written before it existed.
Services also get an identity mark: the vendored Wappalyzer logo when the
service is a product, the class glyph when it is a protocol.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
An ASM is a change-detection engine, so the service inventory needs a
baseline. `is:new` compares (ip, port) against earlier scans of the same
target and only fires when a baseline exists — a target's first scan
reports zero new services rather than marking everything new, the same
contract the host grammar already keeps.
Rows carry the evidence as `is_new` (one page-scoped lookup, not a
correlated subquery per row), shown as a badge in the row and the detail
sheet, with a New quick filter and two counted leads under Change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
The unit of exposure is now a service — (ip, port) — not a bare open port.
Pipeline (EXPANSION): L0 produce hosts, L1 cdn_check + passive_ports,
L2 port_scan, L3 http_probe, L4 screenshot/waf_detect/service_fingerprint.
- cdn_check applies to every target type and records cdn_type, so a domain
scan finally knows which addresses are CDN-fronted. cdn/waf is an edge you
do not own (edge ports only); cloud is your own machine (scanned in full).
- port_scan decides and records a per-address scan_policy (full/web/skip)
honouring the scan context's excluded_ips, and offers port profiles,
exclusions, scan type, a tarpit threshold and an address budget.
- passive_ports reads Shodan internetdb through naabu -passive: keyless,
touches_target=False, so a passive-intensity scan still gets exposure data.
- http_probe seeds its host x port matrix from open ports and folds every
live response back onto the port (is_http, tls, product).
- service_fingerprint names the software behind non-web ports with a pure
Python banner prober (tools/banner/), including a SOCKS5 CONNECT so it
honours the context proxy. naabu -sV is unimplemented.
- shared/definitions/ports.py is the one vocabulary; ip_inventory and
port_inventory are the only writers of ip_addresses/ports.
- open_ports_found is measured from the table (DERIVED_COUNTS): four stages
write ports, so summing stage results double-counted.
- naabu failures no longer report zero open ports as success.
Surfaces: a Services tab (SERVICE_QUERY grammar, 16 fields, 18 counted
leads, 9 group dimensions, detail sheet) built on the shared results shell,
and an outlier-first Exposure panel on the Overview. Every count is a
promise — leads, groups and panel pivots were verified to equal the row
count they land on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
- Deleted `tech-stack-card.svelte` and `related-domains-card.svelte` as they are no longer needed.
- Consolidated scan overview components by integrating functionality into `scan-overview.svelte`, including the addition of `HeroPanel`, `AttentionPanel`, and `PosturePanel`.
- Removed `ScanActivityTimeline` and `ScanCommandLog` from the scan detail view, simplifying the UI.
- Updated scan insights handling to improve loading states and error management.
- Adjusted scan progress messages for clarity.
- Added `single_source` field to `SubdomainInsights` model to track source count.
- Added `newOnly` filter to `WebAssetQuery` and updated related functions.
- Introduced `FilterChip` interface for better query chip management.
- Implemented `activitySummary` function to summarize scan activity results.
- Enhanced `stageRows` function to include detailed activity states and summaries.
- Updated `SubdomainInsights` and `IpGroupRead` models to include new fields for better data representation.
- Added `SENSITIVE_PORTS` definition for sensitive port tracking.
- Improved scan page UI with new tab definitions and enhanced user interactions.
- Implemented history loading for scans to track previous results.
- Added functionality to cancel ongoing scans with user confirmation.
- Updated the engine management UI in `+page.svelte` to improve user experience with new controls for filtering stages, toggling side panels, and managing YAML configurations.
- Introduced new components such as `EngineSummaryBar` and `Resizable` for better layout handling.
- Added clipboard functionality for copying YAML configurations and formatting options.
- Enhanced the backend models in `scan_engine.py` to include a new `PreviewResolved` class for better handling of scan context and parameters.
- Updated presets in `presets.py` to clarify descriptions and improve user understanding of stage settings.
- Deleted the `scan-insights.svelte` component as it was no longer needed.
- Removed associated state management for scan changes and change window in the scans store.
- Updated the `scan-list-item.svelte` to include progress indicators for live scans.
- Introduced a new `scan-status-tabs.svelte` component for better status management.
- Enhanced the chart tooltip to support footer rendering.
- Updated utility functions for scan progress and status to accommodate new features.
- Modified the scan model to include a new field for counting new subdomains.
- Adjusted the main scans page layout to integrate the new activity chart and status tabs.
- Implemented TargetEnrichmentStage for enriching targets with DNS, WHOIS, and BGP context.
- Added VhostStage for virtual host discovery using host-header fuzzing.
- Introduced WafDetectStage for fingerprinting web application firewalls.
- Created configuration files for each new stage with relevant parameters.
- Updated Dockerfile to copy stages instead of engines.
- Enhanced existing tools and services for better integration with new stages.
- Refactored task management in Celery to accommodate new stages and improve error handling.
- Introduced new utilities for scan insights in `scan-insights.ts`, including functions for certificate state, days until expiry, and query compilation.
- Added interfaces for web asset queries, subdomain filters, and insights structures.
- Enhanced `scan-status.ts` with new constants for pagination and search debounce.
- Updated the scan results page to utilize the new insights and remove unused subdomain and asset APIs.
- Modified the `HttpAsset` model to include raw request/response data and body previews.
- Created new models for scan correlation and insights, including surface stats and attention items.
- Implemented streaming JSON parsing in the HTTPX client for memory efficiency.
- Enhanced the executor to support streaming JSON output from subprocesses.
Rebuild the scan detail view into a tabbed results page: Overview
(tech/status/web-server distributions, TLS health, top networks and
services), Web assets (rich httpx table with screenshots, status,
tech, CDN/WAF/TLS-expired badges + a full per-asset detail drawer),
Subdomains (denormalized HTTP summary), Infrastructure (IP-centric
cards correlating open ports and the hosts resolving to each IP), and
Pipeline. Assets correlate by value within the scan (subdomain.name /
http_asset.host / ip), so one IP card surfaces every co-located host.
API: add per-scan http-assets, ports and ips list+summary endpoints
plus services mirroring the subdomains pattern, and serve screenshots
through an authenticated /media/screenshot route; mount scan_media
into the api container so it can read worker-written captures.
Fix two serializers that silently dropped fields: ScanRead omitted
http_assets_found (scan detail and list 500'd), and the subdomain
serializer never emitted the denormalized HTTP summary it stores.
Drop leftover docstrings and banner comments from the httpx client,
parser and field-map.
- Added new types and interfaces for scan scheduling in the frontend.
- Created a new Svelte page for managing scheduled scans, including listing, creating, editing, and deleting schedules.
- Implemented backend models and services for scan schedules, including handling different schedule types (one-off, interval, daily, cron).
- Introduced a Celery task to manage the execution of scheduled scans based on their defined timing.
- Enhanced notification system to include alerts for newly discovered subdomains during scans.
- Added utility functions for handling time zones and scheduling logic.
- Add aggregate functions for scan status and counts in orchestrator.
- Create ScanEventPublisher for emitting scan-related events.
- Introduce ScanActivityService for managing scan activities.
- Implement ScanCommandRecorder for logging command executions.
- Develop canvas building and finalization logic for scan tasks.
- Refactor scan task to utilize orchestrator functions for running stages and finalizing scans.
- Enhance scan notification system to dispatch relevant events.
- Integrate command recording into CLI tool execution for better tracking.
- Updated the default route for automation to redirect to engines instead of scans.
- Removed the scans page and replaced it with a new scans overview page that lists scan history for projects.
- Introduced a detailed scan view page to display individual scan results and subdomain discoveries.
- Added new enums for scan statuses and subdomain sources to improve code clarity and maintainability.
- Implemented a new scan task in Celery to handle scan execution and logging.
- Enhanced the database models for scans and subdomains to support new features and improve data integrity.
- Updated the worker Dockerfile to include necessary scanning tools and dependencies.
- Added utility functions for domain matching and pattern compilation to streamline scan operations.
- Added `clean_name` function to `validation.py` for name validation with max length.
- Simplified docstrings in `dnsx` client, parser, and service files for clarity.
- Enhanced error handling in `ViewDNSService` to prevent disabling keys on rate limits.
- Improved WHOIS service to handle concurrent inserts gracefully with integrity checks.
- Updated `Settings` class in worker config to inherit from `BaseAppSettings`.
- Added new dependencies in `pyproject.toml` for `apprise` and `cryptography`.
- GET /targets/ids for select-all-matching across pages
- POST /targets/enrich/bulk (whois/dns/bgp), /tags/bulk, /organizations/bulk
- bulk bar: enrich/tag/org dropdowns; 'select all N matching' affordance
Move all targets-list filtering, signal filters and sorting into SQL so
they operate across the whole project instead of the loaded page.
- GET /targets gains search, organization_ids, tag_ids, signal, sort_by,
sort_dir (deterministic order with id tiebreaker, nulls last)
- new GET /targets/stats for per-signal KPI counts over the base set
- target_filters: signal predicates (expiring/attention/awaiting/enriched)
and sort columns (name/type/expiry/enrichment)
- frontend: KPI/smart-filter strip, sortable headers + sort menu, CSV/JSON
export, expiry exposure badge; store sends filters/sort to the server
WHOIS/DNS correlation was grouping unrelated targets on low-signal keys.
Add centralized privacy-redaction and shared-infrastructure detection,
drop country from auto-correlation, and bound every correlation query.
Backend:
- shared/utils/privacy.py: detect privacy-proxy/redacted registrant names
and emails (Withheld for Privacy, Domains By Proxy, WhoisGuard, GDPR
masked, etc.) so they never become correlation keys
- shared/utils/infra.py: detect shared/managed DNS (Cloudflare, Route 53,
GoDaddy, ...) so targets are not linked merely by DNS provider
- wire redaction into WHOIS entity extraction; guard correlation queries
defensively so legacy redacted rows do not correlate
- drop country from get_correlations_for_target (links whole regions)
- filter shared nameservers out of WHOIS nameserver correlation
- bound all WHOIS/DNS correlation queries (limit + distinct)
- exclude CDN-fronted IPs and shared nameservers from DNS correlation
- normalize country code casing at storage to fix missed matches
- return accurate cache-hit provenance from WhoisService.lookup
- reuse service finders from correlation API endpoints (DRY)
Frontend:
- remove country as a correlation type across types, API client, and the
related/lookup/overview WHOIS components (still shown as a record field)
- Updated project slug generation to ensure uniqueness by checking active projects.
- Modified project retrieval functions to only fetch active projects.
- Introduced unique slug generation for tags associated with specific projects.
- Enhanced user authentication flow with improved error handling and session management.
- Added session expiration handling in the API client and SSE client for better user experience.
- Improved UI components for better accessibility and user feedback, including copy buttons and activity drawers.
- Refactored notification handling to ensure proper session management and state reset on logout.
- Removed unused enrichment components to streamline the codebase.
- Updated notification model to simplify JSONB handling and ensure proper expiration logic.
