The in-app feed was a second copy of the activity log with a red badge on it.
A scan published up to six notifications (started, completed, new subdomains,
new services, new vulnerabilities, coverage), every one of them already an
activity_logs row and every one of them toasted for five seconds. A nightly
twenty-target schedule left forty to a hundred and twenty unread rows.
If it does not want a human to do something, it is not a notification.
Deleted scan_started, scan_completed and scan_cancelled (both the API-side echo
of the user's own Cancel click and the finalize one). whois/ripestat
enrichment_complete became enrichment_incomplete and return None unless
something actually failed.
One digest replaces the five scan templates. scan_digest(...) -> dict | None
takes a ScanDeltas and derives its severity, title, type and deep-link tab from
the worst thing inside it, and returns None when nothing changed, so a quiet
scheduled run is silent. Severity now encodes risk rather than process:
scan_failed is warning, not error, because the bell takes its colour from the
highest unread severity and one flaky run should not read like a KEV.
Toasts are interrupts only: warning and error toast, success and info land
silently in the bell. A toast with a url finally gets a View action; it used to
require an action_label no template ever set.
notifications.project_id scopes the bell to the active project (nullable and
indexed, no FK so a transient row can never block a project delete). The list,
stats, read-all and clear-all endpoints take ?project_id=, and the store drops
SSE events from other projects.
app.tasks.notifications.cleanup runs on beat every six hours. expires_at had
been decorative since nothing ever called cleanup_expired, and the inbox modal
fetched stats.total rows in a single request.
Three bugs this uncovered:
_count_new_subdomains had no baseline gate while services and vulnerabilities
did, so a target's first scan announced its whole inventory as new: the first
gov.np run would have reported 24022 new subdomains. Baselines are now per
dimension, and a run with none reports the census instead.
The delta counts ignored census_only, so a focused rescan both counted as a
baseline and had its own rows announced. Every comparison now joins
scans bs ... scope = 'full' AND bs.started_at < :started.
Measuring the deltas took 16.8s on a 13,528-host scan: the scan-level baseline
was a correlated EXISTS re-evaluated once per row and could not use
ix_subdomains_target_name_discovered, having no name predicate. Hoisting it to
one scan-level check and rewriting the per-row test as a hash anti-join over a
WITH seen AS (SELECT DISTINCT ...) CTE gives the identical answer in 52ms,
verified against a plain EXCEPT query.
Not done, deliberately: is_read is still one global flag, so on a shared
instance one analyst's Mark all as read still applies to everyone; that needs a
notification_reads table. And notifier.wants() still governs external channels
only, the bell having no preference layer, because volume reduction was the fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BCPTXCr5qPESCt3LxR3329
Uploaded typefaces and instance-wide branding defaults, which together make a
report genuinely white-labellable: a client's own font, a logo set once, and a
theme that names both.
The security half came first, because the logo field was a live SSRF. It was a
free-text string rendered into an <img src>, and WeasyPrint would have fetched
whatever it pointed at from inside the worker. Nothing fetches now, at four
layers. The load-bearing one is a WeasyPrint url_fetcher that allows data: URIs
and files under three known roots and refuses everything else; it was checked
against cloud metadata, internal https, loopback, /etc/passwd, a UNC host, ftp
and protocol-relative references. Above it sit three input guards: a logo must
be embedded data of a known image type, an uploaded theme may not use @import or
a url() that is not data:, and authored markdown loses any image that is not
embedded. A new feature that lets a report name a resource goes through the
fetcher, not around it.
A typeface is bytes, never a link. The family row records files named by us, so
a hostile filename never reaches disk, and the family directory refuses anything
resolving outside itself. A face is accepted only when the bytes agree: the
signature is sniffed and then fontTools opens it and requires outlines, which is
why fonttools is now an api dependency rather than only a WeasyPrint one. A
5-byte file with a valid TTF header and a 4 KB file of padding are both refused.
The family name is banned from carrying quotes or braces because it is written
into a quoted CSS font-family string.
The standalone HTML export now embeds only the families the theme actually
names, which halves it, and no longer carries file:// references that would have
been dead outside the container.
Instance defaults fill only what a template leaves blank, so setting a logo once
never overrides a per-client deliverable that set its own.
Verified: a real TTF family uploaded and embedded in a 53-page PDF under an
uploaded theme with an uploaded logo, and every probe above refused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f2Mf2HGwFaTRnui4UxpRd
reNgine 2 could produce a PDF. This produces a deliverable, and it is built
the way scan engines are built: sections are plug-and-play directories under
reports/, each with a pydantic config model that IS the source of truth. That
model feeds GET /reports/catalog, the builder renders its own form from it, and
the API validates against it. Adding a section is a directory. Scan Engine is to
stages what Report Template is to sections.
WeasyPrint 69 renders it, chosen over Chromium and Typst after measuring what
each can actually do. Only real CSS Paged Media gives a table of contents with
true page numbers, running headers, a PDF outline and footnotes; SysReptor
reaches the same verdict and pipes Chromium's HTML into WeasyPrint for exactly
that reason. reNgine drops the Chromium hop because it templates in Jinja and
draws its charts as SVG in Python, so a browser buys nothing and costs 400 MB
per report. Typst was rejected because a user-uploadable theme would then have
to be Typst code rather than CSS.
A theme is data. shared/definitions/report_theme.py is the schema, six themes
ship as YAML, and a user uploads their own; there is no markup in a theme, so
there is no injection surface. Fonts are vendored so an offline instance still
renders in Latin, Greek and Cyrillic.
Everything the document claims is computed once, in reports/analysis/engine.py:
a posture score whose every deduction is named, weaknesses ranked by the signals
behind the rank rather than severity alone, attack paths that chain observations
into a route, a remediation plan ordered by risk removed, per-dimension change
against the previous run, and caveats that state what was never assessed. Two
narrators read that one brief. Without AI the report still argues its case; with
AI the prose is better and the model never sees a row, only the brief, capped and
cached per check. An AI failure falls back and the report still completes.
Generation is a job. POST /reports queues, the worker walks progress over SSE,
and four formats come out of one render: PDF, self-contained HTML, Markdown and
JSON. The estimate reads each section's own limits, so the page count it promises
is close to the one it delivers.
AI moves out of Settings into its own top-level page: connection, a model per
tier, what it may be used for, what it has cost, and the narrative cache.
Verified on a 1,379-host gov.cy scan: 124 pages in 19.5s across 19 sections,
all six themes, and a broken key still producing a complete report.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f2Mf2HGwFaTRnui4UxpRd
- Added new definitions for dashboard windows and stale target management.
- Expanded domain definitions to include additional public second-level domains.
- Created rescan models and logic to handle asset rechecks based on previous scan results.
- Implemented new scan scopes (FULL and FOCUSED) to differentiate between complete scans and focused rescans.
- Developed asset seed stage to initiate scans from previously identified assets.
- Enhanced scan factory and resolution services to support new scanning configurations and seed assets.
- Updated existing scan models and services to accommodate new features and ensure compatibility.
- Added logging and error handling for focused rescans during scan finalization.
The wordlist field was a free-text absolute path, so the only lists anyone could
actually use were the two baked into the image. Changing that meant a bind mount or a
docker cp — a sysadmin action, not a product one — which in practice meant every
install guessed from the same list. Bruteforce quality is almost entirely wordlist
quality, so the knob was there and nobody could reach it.
Wordlists are now a library, on the same shape as the nuclei check library. Shipped
lists stay in the image and are indexed on read so the picker never starts empty;
uploads land under a `wordlists` volume shared by api and worker. A config field holds
a slug, never a path — `stages/config.py:wordlist(kind)` declares one the way
`threads()` and `rate()` declare theirs, and carries the kind the picker filters on so
a DNS stage is never offered a list of URL paths. `lookup()` still accepts the
absolute paths older engines stored, so an existing engine keeps working.
The UI is the Arsenal's second tab, which is why the Arsenal is now "what your scans
run with" rather than only scanners and their checks: upload, filter by what the words
are, preview the first 200 in the order a scan will read them, delete an upload. Every
list shows the name an engine refers to it by, because that is the thing you type into
a YAML document.
`resolve_path()` refuses anything resolving outside its origin's root and
`delete_custom()` refuses a shipped list — the guard the template library learned the
hard way. A stage that cannot find its list says so and reports PARTIAL rather than
quietly guessing from a different one; that is how the shared volume being absent was
caught during the end-to-end run.
Verified end to end: a seven-word list uploaded through the API, selected by slug,
every passive source disabled — the scan returned six hosts, all attributed to
bruteforce.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
The fieldmap scrubber covers every tool record that maps through it, which is not
how endpoints and ports are written: a sitemap entry, an archive record and a
service banner go straight from a provider dataclass into a row. gov.np's URL
discovery lost its whole run to one of them.
Both inventory row builders now scrub, at the same single-writer choke point the
fieldmap uses for tools. `endpoint_inventory._changes` scrubs too — the update path
writes the same untrusted text as the insert.
The savepoint guards added for exactly this caught nothing, because psycopg raises
the NUL complaint while *binding parameters*: SQLAlchemy wraps that above DBAPIError,
so `except DatabaseError` never fired. Both guards catch StatementError now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
`normalize_path` caps a path at MAX_PATH_LENGTH, but `split_path` takes the last
segment as the filename and that column is varchar(300) — a 1,400-character segment
with no slash in it failed the insert, poisoned the transaction and cost the whole
stage. gov.np hit it once the HTTP probe stopped truncating itself and handed URL
discovery 15,098 web assets instead of 2,687.
MAX_HOST_LENGTH and MAX_FILENAME_LENGTH join the caps already in
shared/definitions/endpoints.py, `split_path` applies the filename cap and
`parse_url` rejects an over-long host rather than storing a truncated one, and the
model's max_length values now read from those constants instead of repeating the
numbers. The batch insert runs in a savepoint and retries row by row, so a row
postgres refuses costs that row and lands in UpsertResult.rejected — the same rule
the HTTP probe now follows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
A gov.np HTTP probe ran 66 minutes over 25,916 host and port pairs, then lost every
row: a captured response body carried a NUL byte, postgres refuses NUL in text and in
json alike, and the failing flush rolled back the whole stage transaction. Recording
that failure then failed too — psycopg puts the offending parameters in its error
message, so the NUL travelled into `scan_activities.error` and took the failure path
down with it, leaving the activity RUNNING and the scan unable to finalize.
`shared/utils/text.py` is now the one place that names what postgres will not store.
`strip_nul` runs through every string, list and dict a tool record yields, applied in
`tools/runner/fieldmap.py` so every tool that maps a record is covered rather than
httpx alone. `strip_control` replaces the two identical local regexes in the nuclei
parser and the template indexer, and now also guards the activity log, the activity
timeline and the command recorder — the three places that write a tool's own words.
The HTTP probe's batch flush runs in a savepoint: a row postgres refuses costs that
row, retried one at a time, and the count comes back as a PARTIAL stage naming how
many responses could not be stored. Recording a stage failure is wrapped too, so a
second error while writing the first can never leave a scan running forever.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
A Celery task that probes the unchecked, non-static URLs under one folder of a scan's
endpoints with httpx, applies the results through endpoint_inventory.verify (never
upsert, so no source is invented) and records an EndpointCoverage row whose cap_reason
names the branch. Dispatched from the api through celery_dispatch.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015KEfo6ipoWMYeHxThmTLRy
- Updated highlight-text component to support multiple search terms.
- Modified results-pagination component to make onPageSize optional and conditionally render PageSizeSelector.
- Integrated HostStructure component into web-asset-detail-sheet for improved asset structure visualization.
- Added HostStructureDialog to manage host structure interactions.
- Enhanced web-assets-table to handle structure-related actions and dialogs.
- Introduced new folder glyphs and updated endpoint classification logic in endpoints configuration.
- Added endpoint_count to subdomain type for better tracking of discovered endpoints.
- Implemented new endpoint filtering capabilities in the backend to support static content visibility.
- Updated models and services to accommodate new endpoint structures and classifications.
A scan no longer needs a saved engine. The launch dialog has two
configuration modes: Scan Engine (default, runs a saved engine exactly
as configured, shown as a pipeline and its YAML) and Custom Scan (pick
the capabilities to run; dependent and supporting stages are included
automatically). Targets are created on launch from typed values, the
scan context is a first-class field, and "Run again" restores the
engine or the selection that ran.
Backend
- scans.engine_id is nullable; an absent engine resolves through
AdHocEngine and the run is named after its capabilities (plan_label)
- ScanCreate/ScanBatchCreate accept target_value(s) (create or reuse via
TargetService.ensure_targets), a per-run intensity, and preview works
for a target that does not exist yet
- every stage declares consumes/produces (AssetKind), group (StageGroup)
and role (capability | support); the registry validates them and the
catalog serves them with groups and seed_produces
- stage_effects walks levels so a producer only feeds later levels and
marks an unfed consumer skipped_no_input
Frontend
- components/scans/launch/: launch-dialog, launch-state, target-picker,
plan-picker (segmented Scan Engine | Custom Scan, pipeline + YAML
Config views, capability pills with full per-stage settings),
stage-pill, stage-config-popover, launch-summary + what-runs,
launch-context-field, launch-context-form, save-engine-dialog
- utilities/launch-plan.ts resolves implied producers and following
support stages; the last engine and last custom selection persist
- quick-scan-fields offers presets and engines in one select; the
command palette offers "Scan <value>"; scan list action is "Run again"
- effect-panel drops its explanatory header; NO_CONTEXT_LABEL is
"No scan context"; launch-modal and execution-preview are removed
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014erb6Jbm6v9sxPgYFBxB7E
cdn_check and passive_ports run at the same level and both call
ip_inventory.ensure(), which on a domain scan is the first thing to write
ip_addresses — so both insert the same rows at the same moment. The row order
came from a UNION with no ORDER BY, so the two statements could take row locks
in different sequences and deadlock, failing a stage for no reason. Both
inventories now insert in a deterministic key order.
A failed facets request also overwrote the tab badge with 0, which reads as
"this scan found nothing" rather than "the request failed". Only a successful
response may restate the count now, in the Services and IPs tables alike.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
Three findings from reading the feature against itself.
Wrong description on 8 curated ports. describe() consulted _BY_NAME before
the port's own entry, and _BY_NAME keeps one spec per name, so port 2376
reported "Docker API without TLS, equivalent to root on the host" — the
opposite of what it is, and the kind of claim an operator acts on. The
port's own entry now wins unless a banner named something the port does not
imply, which keeps 2222/ssh resolving to SSH. Checked exhaustively: all 73
curated ports now return their own description and class.
is:new was quadratic. service_has_baseline correlated per row on target_id
alone, giving Postgres nothing selective to join on: 41.5M rows removed by
join filter on a 6,400-service scan. Whether an earlier scan exists is a
scan-level fact, so it is now a non-correlated EXISTS evaluated once as an
InitPlan, and the API's page-level check shares the same predicate instead of
carrying a second copy in raw SQL. On 24,897 subdomains x 6,440 services with
a populated baseline: is:new 5.5s -> 0.72s, findings 4.6s -> 1.7s.
The naabu replace left a hole. clear() committed, naabu then ran for minutes,
and only then did the rows come back, so a scan in progress showed an empty
port table for the length of the scan. Delete and insert now share one
transaction inside upsert().
Also removed SERVICE_CLASS_HELP, PORT_PROFILE_KEYS and is_sensitive(), which
had no consumers.
Measured and rejected: replacing the hosts CTE with per-address GIN lookups
(44.8s vs 191ms), and an ix_ports(target_id, ip, number) index (Postgres picks
a hash anti-join either way, 5.4ms with and without).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
The unit of exposure is now a service — (ip, port) — not a bare open port.
Pipeline (EXPANSION): L0 produce hosts, L1 cdn_check + passive_ports,
L2 port_scan, L3 http_probe, L4 screenshot/waf_detect/service_fingerprint.
- cdn_check applies to every target type and records cdn_type, so a domain
scan finally knows which addresses are CDN-fronted. cdn/waf is an edge you
do not own (edge ports only); cloud is your own machine (scanned in full).
- port_scan decides and records a per-address scan_policy (full/web/skip)
honouring the scan context's excluded_ips, and offers port profiles,
exclusions, scan type, a tarpit threshold and an address budget.
- passive_ports reads Shodan internetdb through naabu -passive: keyless,
touches_target=False, so a passive-intensity scan still gets exposure data.
- http_probe seeds its host x port matrix from open ports and folds every
live response back onto the port (is_http, tls, product).
- service_fingerprint names the software behind non-web ports with a pure
Python banner prober (tools/banner/), including a SOCKS5 CONNECT so it
honours the context proxy. naabu -sV is unimplemented.
- shared/definitions/ports.py is the one vocabulary; ip_inventory and
port_inventory are the only writers of ip_addresses/ports.
- open_ports_found is measured from the table (DERIVED_COUNTS): four stages
write ports, so summing stage results double-counted.
- naabu failures no longer report zero open ports as success.
Surfaces: a Services tab (SERVICE_QUERY grammar, 16 fields, 18 counted
leads, 9 group dimensions, detail sheet) built on the shared results shell,
and an outlier-first Exposure panel on the Overview. Every count is a
promise — leads, groups and panel pivots were verified to equal the row
count they land on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
- Implemented TargetEnrichmentStage for enriching targets with DNS, WHOIS, and BGP context.
- Added VhostStage for virtual host discovery using host-header fuzzing.
- Introduced WafDetectStage for fingerprinting web application firewalls.
- Created configuration files for each new stage with relevant parameters.
- Updated Dockerfile to copy stages instead of engines.
- Enhanced existing tools and services for better integration with new stages.
- Refactored task management in Celery to accommodate new stages and improve error handling.
The host:port -> live HTTP asset convergence. After discovery, a scan now
runs expansion:
- port_scan: naabu (cdn-gated, well-known service names) -> Port
- cdn_check: cdncheck flags CDN/WAF/cloud IPs (ip family)
- http_probe: httpx probes the host:port matrix; one pass enriches each
service with status/title/tech/server/TLS/cdn/jarm/favicon/hash ->
HttpAsset, with asn backfilled from the resolved IP
httpx -tls-grab carries cert data inline (no separate tlsx pass); nmap is
dropped — naabu for ports, httpx for HTTP service id.
Also: every engine moved to its own folder; net_options() threads
proxy/headers/user-agent into every tool; rengine-tools no longer installs
top-level modules that shadowed libs (httpx); tool binaries resolve from the
go bin ahead of venv console scripts. New Port + HttpAsset models + migration
and the http_assets headline rollup.
- Added new types and interfaces for scan scheduling in the frontend.
- Created a new Svelte page for managing scheduled scans, including listing, creating, editing, and deleting schedules.
- Implemented backend models and services for scan schedules, including handling different schedule types (one-off, interval, daily, cron).
- Introduced a Celery task to manage the execution of scheduled scans based on their defined timing.
- Enhanced notification system to include alerts for newly discovered subdomains during scans.
- Added utility functions for handling time zones and scheduling logic.
- Add aggregate functions for scan status and counts in orchestrator.
- Create ScanEventPublisher for emitting scan-related events.
- Introduce ScanActivityService for managing scan activities.
- Implement ScanCommandRecorder for logging command executions.
- Develop canvas building and finalization logic for scan tasks.
- Refactor scan task to utilize orchestrator functions for running stages and finalizing scans.
- Enhance scan notification system to dispatch relevant events.
- Integrate command recording into CLI tool execution for better tracking.
- Updated the default route for automation to redirect to engines instead of scans.
- Removed the scans page and replaced it with a new scans overview page that lists scan history for projects.
- Introduced a detailed scan view page to display individual scan results and subdomain discoveries.
- Added new enums for scan statuses and subdomain sources to improve code clarity and maintainability.
- Implemented a new scan task in Celery to handle scan execution and logging.
- Enhanced the database models for scans and subdomains to support new features and improve data integrity.
- Updated the worker Dockerfile to include necessary scanning tools and dependencies.
- Added utility functions for domain matching and pattern compilation to streamline scan operations.
- Added `clean_name` function to `validation.py` for name validation with max length.
- Simplified docstrings in `dnsx` client, parser, and service files for clarity.
- Enhanced error handling in `ViewDNSService` to prevent disabling keys on rate limits.
- Improved WHOIS service to handle concurrent inserts gracefully with integrity checks.
- Updated `Settings` class in worker config to inherit from `BaseAppSettings`.
- Added new dependencies in `pyproject.toml` for `apprise` and `cryptography`.
- Updated project slug generation to ensure uniqueness by checking active projects.
- Modified project retrieval functions to only fetch active projects.
- Introduced unique slug generation for tags associated with specific projects.
- Enhanced user authentication flow with improved error handling and session management.
- Added session expiration handling in the API client and SSE client for better user experience.
- Improved UI components for better accessibility and user feedback, including copy buttons and activity drawers.
- Refactored notification handling to ensure proper session management and state reset on logout.
- Removed unused enrichment components to streamline the codebase.
- Updated notification model to simplify JSONB handling and ensure proper expiration logic.
- Added models for WHOIS responses including WhoisDomainResponse, WhoisIPResponse, and WhoisASNResponse.
- Created a parser for WHOIS data to convert raw responses into structured models.
- Developed an RDAPProvider class to handle RDAP lookups using the whoisit library.
- Implemented a WhoisService class for performing WHOIS lookups with optional database caching.
- Added methods for finding correlations between WHOIS records based on registrant, registrar, network, country, and nameservers.
- Updated Dockerfile to include tools directory and added rengine-tools dependency in pyproject.toml.
