The in-app feed was a second copy of the activity log with a red badge on it.
A scan published up to six notifications (started, completed, new subdomains,
new services, new vulnerabilities, coverage), every one of them already an
activity_logs row and every one of them toasted for five seconds. A nightly
twenty-target schedule left forty to a hundred and twenty unread rows.
If it does not want a human to do something, it is not a notification.
Deleted scan_started, scan_completed and scan_cancelled (both the API-side echo
of the user's own Cancel click and the finalize one). whois/ripestat
enrichment_complete became enrichment_incomplete and return None unless
something actually failed.
One digest replaces the five scan templates. scan_digest(...) -> dict | None
takes a ScanDeltas and derives its severity, title, type and deep-link tab from
the worst thing inside it, and returns None when nothing changed, so a quiet
scheduled run is silent. Severity now encodes risk rather than process:
scan_failed is warning, not error, because the bell takes its colour from the
highest unread severity and one flaky run should not read like a KEV.
Toasts are interrupts only: warning and error toast, success and info land
silently in the bell. A toast with a url finally gets a View action; it used to
require an action_label no template ever set.
notifications.project_id scopes the bell to the active project (nullable and
indexed, no FK so a transient row can never block a project delete). The list,
stats, read-all and clear-all endpoints take ?project_id=, and the store drops
SSE events from other projects.
app.tasks.notifications.cleanup runs on beat every six hours. expires_at had
been decorative since nothing ever called cleanup_expired, and the inbox modal
fetched stats.total rows in a single request.
Three bugs this uncovered:
_count_new_subdomains had no baseline gate while services and vulnerabilities
did, so a target's first scan announced its whole inventory as new: the first
gov.np run would have reported 24022 new subdomains. Baselines are now per
dimension, and a run with none reports the census instead.
The delta counts ignored census_only, so a focused rescan both counted as a
baseline and had its own rows announced. Every comparison now joins
scans bs ... scope = 'full' AND bs.started_at < :started.
Measuring the deltas took 16.8s on a 13,528-host scan: the scan-level baseline
was a correlated EXISTS re-evaluated once per row and could not use
ix_subdomains_target_name_discovered, having no name predicate. Hoisting it to
one scan-level check and rewriting the per-row test as a hash anti-join over a
WITH seen AS (SELECT DISTINCT ...) CTE gives the identical answer in 52ms,
verified against a plain EXCEPT query.
Not done, deliberately: is_read is still one global flag, so on a shared
instance one analyst's Mark all as read still applies to everyone; that needs a
notification_reads table. And notifier.wants() still governs external channels
only, the bell having no preference layer, because volume reduction was the fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BCPTXCr5qPESCt3LxR3329
reNgine 2 could produce a PDF. This produces a deliverable, and it is built
the way scan engines are built: sections are plug-and-play directories under
reports/, each with a pydantic config model that IS the source of truth. That
model feeds GET /reports/catalog, the builder renders its own form from it, and
the API validates against it. Adding a section is a directory. Scan Engine is to
stages what Report Template is to sections.
WeasyPrint 69 renders it, chosen over Chromium and Typst after measuring what
each can actually do. Only real CSS Paged Media gives a table of contents with
true page numbers, running headers, a PDF outline and footnotes; SysReptor
reaches the same verdict and pipes Chromium's HTML into WeasyPrint for exactly
that reason. reNgine drops the Chromium hop because it templates in Jinja and
draws its charts as SVG in Python, so a browser buys nothing and costs 400 MB
per report. Typst was rejected because a user-uploadable theme would then have
to be Typst code rather than CSS.
A theme is data. shared/definitions/report_theme.py is the schema, six themes
ship as YAML, and a user uploads their own; there is no markup in a theme, so
there is no injection surface. Fonts are vendored so an offline instance still
renders in Latin, Greek and Cyrillic.
Everything the document claims is computed once, in reports/analysis/engine.py:
a posture score whose every deduction is named, weaknesses ranked by the signals
behind the rank rather than severity alone, attack paths that chain observations
into a route, a remediation plan ordered by risk removed, per-dimension change
against the previous run, and caveats that state what was never assessed. Two
narrators read that one brief. Without AI the report still argues its case; with
AI the prose is better and the model never sees a row, only the brief, capped and
cached per check. An AI failure falls back and the report still completes.
Generation is a job. POST /reports queues, the worker walks progress over SSE,
and four formats come out of one render: PDF, self-contained HTML, Markdown and
JSON. The estimate reads each section's own limits, so the page count it promises
is close to the one it delivers.
AI moves out of Settings into its own top-level page: connection, a model per
tier, what it may be used for, what it has cost, and the narrative cache.
Verified on a 1,379-host gov.cy scan: 124 pages in 19.5s across 19 sections,
all six themes, and a broken key still producing a complete report.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f2Mf2HGwFaTRnui4UxpRd
The SSE route authenticated through CurrentUser, whose session dependency is
only released when the response ends — so every open stream left one connection
idle in transaction for as long as the tab stayed open. The engine was created
with no pool arguments at all, so the ceiling was SQLAlchemy's default 5+10:
fifteen tabs and every DB endpoint blocked for pool_timeout and then 500'd.
Reproduced exactly, and both pinned backends' last query was get_current_user's
SELECT users.
StreamUser opens and closes its own short session, so a stream holds nothing
pooled — 200 concurrent streams now report checked_out 1.
The pool is configured rather than defaulted (10+20, pre-ping, 30m recycle,
fail-fast timeout) and carries an application_name, because a blank one is why
this needed a live reproduction to diagnose. idle_in_transaction_session_timeout
is the backstop: any future leak from any path is reclaimed by Postgres instead
of accumulating, and pre-ping makes the next checkout succeed on a fresh socket.
A stream also no longer outlives its token. It re-checks expiry and revocation
on each heartbeat — JWT plus Redis, no DB — and emits `event: unauthorized`, so
logout actually ends the feed and the client refreshes at once instead of
burning three failed reconnects.
Worker pools are sized per prefork child (8 x 15 was over max_connections) and
the inherited engine is disposed on fork so a child never shares a parent socket.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015biqVXTng75nnfKXMqm61H
- Added new definitions for dashboard windows and stale target management.
- Expanded domain definitions to include additional public second-level domains.
- Created rescan models and logic to handle asset rechecks based on previous scan results.
- Implemented new scan scopes (FULL and FOCUSED) to differentiate between complete scans and focused rescans.
- Developed asset seed stage to initiate scans from previously identified assets.
- Enhanced scan factory and resolution services to support new scanning configurations and seed assets.
- Updated existing scan models and services to accommodate new features and ensure compatibility.
- Added logging and error handling for focused rescans during scan finalization.
Discovery was passive-only — subfinder, ctfr, crtname, assetfinder and certificate
SANs — so anything absent from a public source was absent from the scan, and every
later stage inherited the miss. It now runs two more sources after the passive merge,
carrying SubdomainSource.BRUTEFORCE and PERMUTATION through the same scoping and
merge every other provider uses.
Bruteforce is `dnsx -d <apex> -w <list>` over a wordlist shipped in the repo:
4,860 labels ranked by real-world frequency, regenerated by
scripts/fetch_subdomain_wordlist.py and committed like the port registry, so an
install with no egress still has one. The word budget reads from the top of that
ranking, which is why a smaller budget is simply the first N lines. Permutation is
alterx over the names already found, seeded apex-first because a name one label from
the apex has more siblings worth guessing than a five-label one.
Two things this needed to be worth running at all:
- **A budget, not an idle watchdog.** A guessed name that does not exist produces
no output, so the stall detector that protects the resolver killed both runs at
exactly 90s the first time. Guessing is bounded by total time instead, floored at
5 names/s against a measured 9, and says so when it stops early.
- **Wildcard rejection.** The wildcard probe moved ahead of expansion: on a wildcard
zone every guess answers, and storing 5,000 fake names would be worse than not
guessing. Answers that resolve only to the wildcard address are dropped and
counted in the source's note.
Both are skipped at passive intensity and say why — the stage still touches no
target, but a wordlist run does query the target's own nameservers. Bruteforce
defaults on at 1,000 words (110s measured, 2 hits on vulnweb.com, both attributed
correctly alongside their passive sources); permutations default off, because every
variant is another DNS query.
ProviderResult gained `note` so a source can come up short without being reported as
failed. alterx reads seeds from stdin — `-l <file>` answers "no input found".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
A gov.np HTTP probe ran 66 minutes over 25,916 host and port pairs, then lost every
row: a captured response body carried a NUL byte, postgres refuses NUL in text and in
json alike, and the failing flush rolled back the whole stage transaction. Recording
that failure then failed too — psycopg puts the offending parameters in its error
message, so the NUL travelled into `scan_activities.error` and took the failure path
down with it, leaving the activity RUNNING and the scan unable to finalize.
`shared/utils/text.py` is now the one place that names what postgres will not store.
`strip_nul` runs through every string, list and dict a tool record yields, applied in
`tools/runner/fieldmap.py` so every tool that maps a record is covered rather than
httpx alone. `strip_control` replaces the two identical local regexes in the nuclei
parser and the template indexer, and now also guards the activity log, the activity
timeline and the command recorder — the three places that write a tool's own words.
The HTTP probe's batch flush runs in a savepoint: a row postgres refuses costs that
row, retried one at a time, and the count comes back as a PARTIAL stage naming how
many responses could not be stored. Recording a stage failure is wrapped too, so a
second error while writing the first can never leave a scan running forever.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
Measured end to end against three targets on the default plan.
Execution order is now declared, not hand-numbered. `Stage.depends_on` names the
stages one may not start before and the registry derives `level` from it by
longest-path layering, rejecting an unknown name or a cycle. Eleven hand-assigned
barriers collapse to seven real ones: target enrichment no longer gates subdomain
discovery, and the whole post-probe fan-out (origin probe, service fingerprint,
WAF detection, URL discovery, vulnerability scan) is one parallel level instead of
five. The frontend plan solver groups by the same derived level.
Four tool invocations were wrong in ways that cost whole scans:
- naabu reads `-timeout` as a go duration, so a bare `-timeout 3000` pinned every
invocation at a fixed ~135s whatever the host and port count. `-timeout 3s`
returns the identical open-port set: 138s -> 5s on three addresses, and on a
1,610-address estate 7,428s (two 3,600s timeouts, stage failed) -> 240s.
- urlfinder has no `-t`; passing one made it exit 2, so the archive URL source had
never returned a row.
- amass v4 prints a relationship graph, not a host list, so its output was parsed
as hostnames and discarded — zero rows in every scan on record while costing the
full 360s tool budget. Parsed properly now, and out of the default source set.
- httpx's stream probe was capped at a 900s total, silently truncating a large HTTP
probe and reporting it as success. It is idle-bounded now, the way dnsx already
was, and reports PARTIAL when it is stopped.
Celery's redis visibility timeout (3600s default) sat below the 8h task limit, so any
stage running over an hour was delivered twice and the whole rest of the scan ran
again — one run duplicated its entire post-port-scan pipeline, another looped for ten
hours. It now sits above the hard limit.
Work inside stages that was serial and did not need to be: resolver batches
(dns_batch_concurrency, default 3), URL discovery providers, robots/sitemap fetches,
and wafw00f (which has no threads of its own) now run concurrently, with DB writes
kept on the stage thread. wafw00f, screenshots and the WAF stage report what they did
not cover instead of reporting a short run as complete. The cancellation flag is
polled at most every two seconds instead of once per resolved name.
Also: WAF detection and screenshots no longer load whole http_asset rows (stored
response bodies included) to write one column; tool output decodes with
errors="replace" so one non-UTF-8 byte cannot discard a run; the worker runs at
CELERY_SCAN_CONCURRENCY (8) because the widest level now holds five stages.
Measured: vulnweb.com 656s -> 218s with identical results. hamropatro.com completes
in 2,273s where it previously never finished. gov.np's port scan 7,428s -> 240s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
dnsx drops 20–60% of answers under load and exits 0, so a subdomain stage
could store half its hosts unresolved and still read Success. Resolution now
runs in shuffled batches (dns_batch_size) with the thread count clamped to the
range the resolver actually keeps up with; a batch answering far below the
median of its peers is retried, and one that stops producing is abandoned after
dns_idle_timeout instead of the whole run waiting on a total ceiling. What the
resolver managed — batches, retries, stalls — is reported, never assumed.
StageResult gains warnings + partial. The stage runner records a new PARTIAL
activity status carrying the notes, logs the stage outcome as a warning and
emits it live. The run ribbon paints a partial stage amber with a triangle, the
stage dialog shows the note in a warning box, and activityRan() is the one "did
it run" test so progress and summaries count partial as done. reverse_dns
reports the same way when dnsx times out or is missing.
stream_json yields a StreamOutcome (return code, timed_out, record count,
stderr) instead of a bare iterator, kills on idle as well as on total time, and
a timed-out run keeps the records that landed instead of discarding them.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012YK61xuAJGMhq1uniLhHX5
A Celery task that probes the unchecked, non-static URLs under one folder of a scan's
endpoints with httpx, applies the results through endpoint_inventory.verify (never
upsert, so no source is invented) and records an EndpointCoverage row whose cap_reason
names the branch. Dispatched from the api through celery_dispatch.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015KEfo6ipoWMYeHxThmTLRy
Detection without a delta is an inventory, not an ASM. Finalize now counts
the ports this scan is the first to see for its target and publishes a
notification, escalated to warning severity when any of them is an
administrative or datastore port.
Both delta notifications moved into `_notify_deltas` so finalize stays
under the statement budget.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
The unit of exposure is now a service — (ip, port) — not a bare open port.
Pipeline (EXPANSION): L0 produce hosts, L1 cdn_check + passive_ports,
L2 port_scan, L3 http_probe, L4 screenshot/waf_detect/service_fingerprint.
- cdn_check applies to every target type and records cdn_type, so a domain
scan finally knows which addresses are CDN-fronted. cdn/waf is an edge you
do not own (edge ports only); cloud is your own machine (scanned in full).
- port_scan decides and records a per-address scan_policy (full/web/skip)
honouring the scan context's excluded_ips, and offers port profiles,
exclusions, scan type, a tarpit threshold and an address budget.
- passive_ports reads Shodan internetdb through naabu -passive: keyless,
touches_target=False, so a passive-intensity scan still gets exposure data.
- http_probe seeds its host x port matrix from open ports and folds every
live response back onto the port (is_http, tls, product).
- service_fingerprint names the software behind non-web ports with a pure
Python banner prober (tools/banner/), including a SOCKS5 CONNECT so it
honours the context proxy. naabu -sV is unimplemented.
- shared/definitions/ports.py is the one vocabulary; ip_inventory and
port_inventory are the only writers of ip_addresses/ports.
- open_ports_found is measured from the table (DERIVED_COUNTS): four stages
write ports, so summing stage results double-counted.
- naabu failures no longer report zero open ports as success.
Surfaces: a Services tab (SERVICE_QUERY grammar, 16 fields, 18 counted
leads, 9 group dimensions, detail sheet) built on the shared results shell,
and an outlier-first Exposure panel on the Overview. Every count is a
promise — leads, groups and panel pivots were verified to equal the row
count they land on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
- Added `newOnly` filter to `WebAssetQuery` and updated related functions.
- Introduced `FilterChip` interface for better query chip management.
- Implemented `activitySummary` function to summarize scan activity results.
- Enhanced `stageRows` function to include detailed activity states and summaries.
- Updated `SubdomainInsights` and `IpGroupRead` models to include new fields for better data representation.
- Added `SENSITIVE_PORTS` definition for sensitive port tracking.
- Improved scan page UI with new tab definitions and enhanced user interactions.
- Implemented history loading for scans to track previous results.
- Added functionality to cancel ongoing scans with user confirmation.
- Implemented TargetEnrichmentStage for enriching targets with DNS, WHOIS, and BGP context.
- Added VhostStage for virtual host discovery using host-header fuzzing.
- Introduced WafDetectStage for fingerprinting web application firewalls.
- Created configuration files for each new stage with relevant parameters.
- Updated Dockerfile to copy stages instead of engines.
- Enhanced existing tools and services for better integration with new stages.
- Refactored task management in Celery to accommodate new stages and improve error handling.
Rounds out expansion (host discovery, virtual hosts, screenshots, WAF):
- host_discovery: naabu liveness sweep marks IpAddress.is_alive for
ip_range/asn; port_scan then deep-scans only live hosts
- vhost: ffuf Host-header bruteforce (domain, opt-in) -> vhost-only
Subdomains that http_probe then picks up
- screenshot: httpx headless chromium -> HttpAsset.screenshot_path under
a scan_media volume
- waf_detect: wafw00f fingerprints WAFs -> HttpAsset.waf
All config-driven (ExpansionConfig) with proxy / user-agent / headers +
per-tool custom args threaded through. Adds ffuf, chromium and wafw00f to
the worker image and a scan_media volume.
The host:port -> live HTTP asset convergence. After discovery, a scan now
runs expansion:
- port_scan: naabu (cdn-gated, well-known service names) -> Port
- cdn_check: cdncheck flags CDN/WAF/cloud IPs (ip family)
- http_probe: httpx probes the host:port matrix; one pass enriches each
service with status/title/tech/server/TLS/cdn/jarm/favicon/hash ->
HttpAsset, with asn backfilled from the resolved IP
httpx -tls-grab carries cert data inline (no separate tlsx pass); nmap is
dropped — naabu for ports, httpx for HTTP service id.
Also: every engine moved to its own folder; net_options() threads
proxy/headers/user-agent into every tool; rengine-tools no longer installs
top-level modules that shadowed libs (httpx); tool binaries resolve from the
go bin ahead of venv console scripts. New Port + HttpAsset models + migration
and the http_assets headline rollup.
Every scan now runs a discovery phase, gated per seed type:
- seed_resolution: expand ip / cidr / asn seeds into IpAddress assets
(stdlib ipaddress, even-stride sampling + rfc1918 skip on big ranges)
- reverse_dns: dnsx -ptr over discovered IPs
- ip_enrichment: asn / org / country from cached bgp + whois
- target_enrichment: consolidate cached whois/dns/bgp, refresh on miss/stale
The engine framework grows a phase axis (discovery/expansion/depth) and
per-stage applies_to gating so a stage skips cleanly when it doesn't apply
to the seed; subdomain discovery is retagged as expansion. Adds the
IpAddress model + migration.
- Added new types and interfaces for scan scheduling in the frontend.
- Created a new Svelte page for managing scheduled scans, including listing, creating, editing, and deleting schedules.
- Implemented backend models and services for scan schedules, including handling different schedule types (one-off, interval, daily, cron).
- Introduced a Celery task to manage the execution of scheduled scans based on their defined timing.
- Enhanced notification system to include alerts for newly discovered subdomains during scans.
- Added utility functions for handling time zones and scheduling logic.
- Add aggregate functions for scan status and counts in orchestrator.
- Create ScanEventPublisher for emitting scan-related events.
- Introduce ScanActivityService for managing scan activities.
- Implement ScanCommandRecorder for logging command executions.
- Develop canvas building and finalization logic for scan tasks.
- Refactor scan task to utilize orchestrator functions for running stages and finalizing scans.
- Enhance scan notification system to dispatch relevant events.
- Integrate command recording into CLI tool execution for better tracking.
- Updated the default route for automation to redirect to engines instead of scans.
- Removed the scans page and replaced it with a new scans overview page that lists scan history for projects.
- Introduced a detailed scan view page to display individual scan results and subdomain discoveries.
- Added new enums for scan statuses and subdomain sources to improve code clarity and maintainability.
- Implemented a new scan task in Celery to handle scan execution and logging.
- Enhanced the database models for scans and subdomains to support new features and improve data integrity.
- Updated the worker Dockerfile to include necessary scanning tools and dependencies.
- Added utility functions for domain matching and pattern compilation to streamline scan operations.
- Added `clean_name` function to `validation.py` for name validation with max length.
- Simplified docstrings in `dnsx` client, parser, and service files for clarity.
- Enhanced error handling in `ViewDNSService` to prevent disabling keys on rate limits.
- Improved WHOIS service to handle concurrent inserts gracefully with integrity checks.
- Updated `Settings` class in worker config to inherit from `BaseAppSettings`.
- Added new dependencies in `pyproject.toml` for `apprise` and `cryptography`.
- Added models for WHOIS responses including WhoisDomainResponse, WhoisIPResponse, and WhoisASNResponse.
- Created a parser for WHOIS data to convert raw responses into structured models.
- Developed an RDAPProvider class to handle RDAP lookups using the whoisit library.
- Implemented a WhoisService class for performing WHOIS lookups with optional database caching.
- Added methods for finding correlations between WHOIS records based on registrant, registrar, network, country, and nameservers.
- Updated Dockerfile to include tools directory and added rengine-tools dependency in pyproject.toml.
