Remove the justification tells the Writing rules ban: a fact followed by a
semicolon, colon or 'so' clause in the test comments, the contrastive and
colloquial reads in the echo-guard, wildcard, api-spec and pacing docstrings,
and the colon-and-reason form in the coverage-strip and GraphQL notes. Wording
only.
A source-map or schema sweep sends many requests to one host, and a fragile or
rate-limited host answers a burst with timeouts and 429s that read as no result.
Requests to one host now pass through an adaptive governor: the per-request delay
doubles when the host returns a timeout, 429 or 503 and eases back as it answers
cleanly, with a cap on how many requests are in flight at once. A healthy host
runs at full speed.
Finding titles become Third-party content on hostname and Dangling A
record, their descriptions become label and value lines, and impact and
remediation state one fact each. The internal address check, the
fixable flag, the tracking kind and the referenced hostname messages
drop their explanatory wording.
- Store each run's row, first-seen and retired counts (scan_deltas, scan_retired), valid while
the revision stamp matches scan_revisions; statement triggers on the seven result tables bump
it. Finalize warms, a beat backfill fills the rest, a live run answers from its last count and
queues a refresh. /scans 1.5 s -> 43 ms, dashboard overview 5-10 s -> 0.9 s.
- lead_cache holds a live scope for 20 s with a stale copy, and every build is single-flight;
search pages opt out.
- Leads: lowercase subdomain names (CHECK) and LIKE for host:, a guard for empty JSON arrays,
branches as MATERIALIZED CTEs read to the cap, every count in one statement.
- Addresses and services: the derived CTE as one grouped pass, rows and count in one execution
(page_rows), GIN lookups for page hosts, facets through grouping sets.
- Correlation carriers counted by EXISTS; takeover latest-sighting in SQL, no silent cap.
- Screenshot backfill partial index, nine unused low-cardinality indexes dropped, hostname
helpers memoised, vhost names normalised.
Every web asset carries the analytics, tag manager and ads accounts its
page loads (Universal Analytics folded to its account, GA4, GTM, Google
Ads, AdSense, Meta pixel, Hotjar, Clarity, Yandex, HubSpot), read at
write and by the stored-response backfill. tracking is a correlation
kind, a host grammar field and the second cross link after the
certificate, judged by the page rule so a parked or default page never
vouches for its provider's account.
generic_page kept only ASCII, so a Devanagari or Chinese title read as
empty and every hub of a non-Latin estate was the server's. It keeps
Unicode word characters now.
A passive stage after origin_probe judges every owned name against the
redirect, the certificate, page links and the bare-address response.
rengine-foreign-site (high) when the name serves another organisation's
page, rengine-unhosted-name (medium) when the server answers with its
default and someone else's certificate. The estate drops every domain a
claim names, so a tenant no longer reads as a related domain.
provider_of moves to shared/definitions/estate.py for the worker; sign-in
and code hosts join VENDOR_DOMAINS.
A nested chord canvas put the whole remaining pipeline into every
stage message, duplicated per sibling per level: 33 MB per message,
39 of them filled a 1 GB Redis, and the worker was OOM-killed
deserializing them. Every later launch then failed with "command not
allowed when used memory > 'maxmemory'".
shared/services/orchestrator/canvas.py now builds one step at a time;
a step's chord body is app.tasks.scan.run_scan_step(steps, index+1),
which dispatches the next step once the current one settles. Messages
are ~1 KB now; tests/pipeline/test_canvas_shape.py caps them.
Also, from an end-to-end review of the pipeline:
- The reaper re-queues a PENDING scan with no launch anywhere in the
queue, so a lost or failed send_task no longer strands it forever.
api create() now dispatches off the event loop and records a failed
dispatch the way create_batch already did.
- Cancel no longer revokes queued stage tasks. A cancelled run used to
walk every remaining step recording one aborted row per stage, and
whether finalize ran depended on a race with the revoke broadcast.
The step body now finalizes a cancelled run directly, and the api
also dispatches finalize so a run with no live canvas still settles.
- The reaper only skips scans that actually have a message queued,
read off the scans queue, instead of skipping every scan whenever
the queue holds anything.
- Every beat tick carries expires, so a worker that comes back after
being down does not replay hours of stacked ticks.
- worker-default and worker-beat restart on their own.
- _run_process kills the tool in a finally; a soft time limit raised
inside proc.wait used to leave the tool running past the stage.
- on_worker_ready crashed on a missing Consumer attribute, so the
worker's IP-range and threat-feed warm-up never ran.
- Worker logging now goes through the root logger; it was configured
on a "rengine.worker" logger nothing writes to, so INFO logs (task
starts, step hand-overs) never appeared in the container output.
- _normalize_url no longer raises on an unparseable value, which had
been killing the whole bounty feed sync on one bad scope entry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0181JfVNatLVWnDbtGWRWPov
Out-of-band testing is an instance setting — `off`, `self_hosted` or
`public` — with the server, the token (the INTERACTSH API key row) and the
wait resolved once in `shared/services/oast.py` and read on the stage
thread. A mode that cannot be used is off and says why: self-hosted with no
server, or public without the disclosure acknowledged. A fresh instance is
off. `POST /oast/test` asks a public address for `/poll` over HTTPS and
reports that the server answered, never that OAST works.
The engine keeps one switch. `FieldNeeds` puts the resource key on the
field and `GET /engines/catalog` fills `unavailable_reason`, so the builder
renders the instance reason beside a switch that stays settable.
Measured on nuclei v3.11.1: with `-no-interactsh` a check that needs a
callback still loads and the request is refused — 0 requests, exit 0. So
the batch reported COMPLETED and every origin in it read `scanned` for
checks that never touched the target. The selection now drops them when
nothing is listening, one coverage row reports the count and the reason,
and `Tier.OAST` is left out of the planned tiers.
Membership is what a check references, not what it is tagged: 615 of the
12,586 official rows reference an interactsh placeholder and 233 of those
carry no `oast` tag. `vuln_templates.needs_oast` is parsed at index time; a
library resync fills the column.
The wait is paid on out-of-band batches alone, which sweep for 30 minutes
so it is paid fewer times, and the tier names the server on its own rows.
A finding's callback is rendered on the sheet from one label map; the list
path is bared to the metadata.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
urllib.parse.SplitResult.port raises ValueError when the authority
carries a non-numeric port; nine call sites read it unguarded, so one
malformed URL (a stray quote, a client typo) turned into an unhandled
exception in the middle of a scan, a report, an interest rule or a
proxy dial. shared/utils/net.py:url_port makes the read itself the
failure boundary; unreadable_port names the bogus text so the naabu
and banner proxy paths can report what they refused rather than
retrying against a host string that dropped its port silently.
shared/definitions/ports.py:SCHEME_PORTS replaces four private copies
of the same http/https default-port map (endpoints.py, endpoint_judge,
scan_surface/normalize, nuclei's parser, dalfox's parser).
Also renames live_hosts to answered_hosts in the report source and the
interest context, and lifts the asset query's inline is:web predicate
into predicates.answered() — the existing distinction between is:live
(2xx) and is:web (answered at all) was already the rule; live_hosts
was the one name still claiming the stronger one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VckaDx6mBB4TsG76u61gvv
A full static review of the 3.0 tree, run as two fan-out passes with every
finding verified against the running stack rather than taken on inspection.
Correctness and security
- instance settings encrypted the mask over the stored AI key: `!= MASK` never
matched `••••••••abcd`, so saving settings destroyed the key and `test_ai`
posted the mask as a bearer token
- the MFA token was replayable for its full five-minute life; the gate now
checks revocation and consumes the token after a successful exchange
- logout revoked nothing for a bearer client while answering "Logged out"
- `POST /api-keys/{id}/test` returned the plaintext key inside httpx's error
string, which embeds the whole URL
- an interest rule could be edited or deleted through another project's id
- redis ran the celery broker under `allkeys-lru`, free to evict a queued task
or a chord counter; now `noeviction`
- nuclei's stored command carried the OAST token and every auth header
- a finalize step could poison the session and leave a scan RUNNING for ever
- `is:issue` raised TypeError on every search
- offset paging repeated and skipped rows in four dimensions
- a cancelled run kept a stale `scans.*_found` rollup
- every PDF report raised `KeyError: 'software'`
- `version_key` truncated a wide numeric run and inverted release ordering
- psycopg raises a NUL as a bare ValueError, above StatementError, so every
savepoint guard was blind to the case it was written for
Performance
- endpoints.host had no trigram index: 35.9ms seq scan -> 0.389ms
- free text compiled to one OR, not the prescribed UNION ALL: 132ms -> 41ms
- `/http-assets/summary` and `/ips/summary` materialised whole tables
- the cross-link `ip` kind ran an estate query per page and drew nothing
Frontend
- `$effect` loaders that read the state they write, an unbounded request loop
on a failing endpoint
- error states that rendered as empty states
- 2FA backup codes were unusable while the copy said they worked
- a "Scan history" dialog that always reported none
- ~400 duplicated lines across five filter bars, now one component
Build and tooling
- every scanner installed at @latest; all 13 pinned to the measured versions
- CI built `web/`, gone since 2.x, with tags that cannot be valid
- the prettier pre-commit hook discarded its filenames and always passed
- the test suite destroyed itself when run twice
Also: subdomains from public code, via a pinned github-subdomains source and a
GitHub API key that subfinder receives too; sublist3r and oneforall removed
after measuring one unique host in 89 across four targets.
Verified: 853 tests, ruff clean over eleven packages, svelte-check 0 errors,
the count-is-a-promise gate at 131 checks and 0 mismatches, and 143 GET routes
with no 5xx. Four migrations, each with an exercised downgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVkdx5ynZ6aof5kLFczmhX
Three places built a probe target as f"{ip}:{port}". ip_addresses holds
IPv6 — seed_resolution writes both a and aaaa records — so http_probe,
origin_probe and the vulnerability surface handed httpx and nuclei
2001:db8::1:443, which is neither a host nor a host and port. The httpx
parser then made it worse: rsplit(":", 1) turned a bare v6 address into
2001:db8: before it was stored as the asset's host.
The endpoint layer already got this right in two places with the same
inline expression. That expression is now bracketed() in shared/utils/net
alongside host_port() and split_host_port(), and all five sites use it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5roHYDYzSPRNaTkn8hSkJ
Three tools took the scan's network config and dropped part of it.
The banner prober spoke SOCKS5 to whatever proxy it was given. A scan
context holding an http:// proxy made every handshake fail, and the stage
reported "no banner returned by N ports" rather than a problem. It now
speaks SOCKS5 or HTTP CONNECT by scheme (tools/banner/proxy.py, tested
against a stub proxy), and refuses a scheme it cannot carry before any
port is touched instead of dialling direct.
naabu was handed the resolved URL whole. Its -proxy is socks5 only and
takes no inline credentials, so http:// or any password quietly did not
apply. proxy_args() renders -proxy host:port with -proxy-auth beside it,
and names the reason when the proxy cannot be expressed, which port_scan
and host_discovery report as partial. -passive is an HTTPS call to
Shodan, so it goes through proxy_env instead, which passive_ports was not
passing at all.
wafw00f got the proxy but never the headers, so an authenticated surface
was fingerprinted unauthenticated. It takes a file rather than a value,
so the run writes one 0600 temp file and removes it after.
proxy_env moves to shared/services/proxy_resolve.py now that naabu wants
it too, and host_port() lands in shared/utils/net.py for the bracketed
IPv6 authority CONNECT needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5roHYDYzSPRNaTkn8hSkJ
A stored secret that failed to decrypt was used as if it were plaintext.
unseal_headers handed the Fernet token back, so every run whose context
carried auth sent `Authorization: gAAAAAB...` to the target and reported
success. EncryptedJSON returned None, so a scan context's auth silently
vanished and the run went out unauthenticated. load_proxy_endpoints
returned an empty list, so a scan configured to go through a proxy went
direct instead. Rotating SECRET_KEY was enough to trigger all three.
decrypt_stored() now separates the two cases a lenient decrypt conflated.
A value that carries no Fernet token was written before sealing shipped
(2026-09-06) and is returned as it is. A value that carries one and will
not open raises SecretDecryptionError, naming the header, the column or
the proxy and what to do about it.
A stage turns that into a failed activity with the message on it; the api
answers 503 through a handler beside the existing DBAPIError one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5roHYDYzSPRNaTkn8hSkJ
Screenshots carry a perceptual hash, so hosts whose pages look alike are
grouped in the gallery and the table. An exact hash matches only identical
bytes, which leaves the same login across forty tenants in forty groups of
one, so this is a 64-bit dHash taken from a fixed viewport band.
Clustering follows the view's scope rather than being frozen at finalize:
one run on a scan page, every target on the project-wide view. A cluster is
every row within SCREENSHOT_DISTANCE of one representative, so its count and
the screenshot: filter it opens are the same set by construction.
Thresholds are measured over 9,890 stored screenshots on four estates.
Distance 2 collapses the cPanel port variants and a default page's one-bit
variants while 4 begins merging different skins. A hash setting fewer than
four bits is a blank render and never represents a cluster; a variance floor
let a one-bit hash sink 79 hosts into one group.
Screenshots are held as WebP at native size, 18% of PNG, converted by a
background task rather than the stage: httpx writes the PNG itself, so
converting inline saves no peak disk and costs every scan serial time. Only
settled scans are touched, or the path is rewritten under a stage still
writing it. The hash is taken before the encode and never recomputed, so
compression leaves the grouping untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U2BnU1s9wZg3KYgCHBxhuC
A target carries a list of host names, addresses and URLs an operator already
holds. Every scan of that target starts from them, so a list earned elsewhere
no longer has to be re-pasted or re-discovered.
Each pasted line is classified by its own shape and scope-checked against the
target; a refusal names the line and the reason. Seeds enter a run through the
asset_seed stage, which resolves the names it has no answer for and materialises
what they resolve to. Stages the engine has switched off still do not run.
Seeds no longer force a focused run: scope reads the new seed_only, which only
rescan, watch and connector runs set, so a seeded census scan still covers its
target. Seed count is a material run facet, so Compare states it before any
delta it would otherwise explain.
Two stages share the subdomains table for a run: discovery's clear no longer
drops rows asset_seed wrote, and the run's seeded hosts join its merge so the
imported source survives a name discovery also finds.
Closes#1538
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LYZfaNjq3Ym8GPvWYuHisS
Three holes in target validation, found by adding them all as targets.
A host name was stored as typed, so example.com, EXAMPLE.com and
Example.Com were three targets in one project and the unique constraint
never saw a collision. Host names now case-fold; a URL folds its scheme
and host and keeps its path, since a path is case-sensitive. The
migration folds existing rows and leaves a row whose folded value is
already taken for the operator to merge.
validate_asn was ^AS\d+$ with no bound, so AS0 and
AS99999999999999999999 were accepted and then failed enrichment with an
RDAP error nobody can act on. It now takes 1 to 4294967295.
0.0.0.0/0 and ::/0 were accepted, and expand_network samples a netblock
target evenly, so the scan would have swept the whole internet from this
instance's address. Loopback, link-local, multicast, reserved space and
the default route are refused; RFC1918 stays, corporate estates are the
reason it exists. Each refusal says which bound it crossed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XxW9yfg1SEchtg6HLGHRkK
A row states that a favicon, page title, body hash, certificate, CNAME
target or address is also held by a host under a different target, and
the popover names the peers and pivots to the query that returns them.
Read against the estate, so a row on a single-scan page still reaches
the other targets.
The refusal is the feature. Measured across four unrelated estates, all
five cross-target values were a provider's and none is drawn. A carrier
on a SHARED_EDGE_HOSTS CNAME vouches for no identity. generic_page()
refuses a title that restates the status of the rows carrying it, the
soft-404 phrases and the curated edge interstitials, and it gates the
three page-derived kinds. cert_covers() requires a certificate to name
the host presenting it, so *.okta.com on a tenant is not a link. JARM
spanned all four estates and is not a cross link.
Each value is judged once before the row loop and _crossing() refuses to
fetch carriers for a value one target holds: a wildcard CNAME with 8,679
carriers cost 4.6s a page, now ~80ms.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxAonr95Uj7Bdu4CTMCwx9
Clicking the DNS cell in the targets list opened the generic target dialog:
there was no DNS modal at all. There is one now, and it renders through the
same `dns-tab` the target page uses, so the records, the type filters, the
provider labels and the refresh are one component in two places.
DNS is also a relation. `dns_record` links two targets on a mail host, a zone
contact, an address or an SPF include - and only when the estate owns the
name. On the four unrelated estates here the only shared DNS values are
Google's MX hosts and an `include:mail.zendesk.com` that briefly linked
Shopify to HackerOne, which is the landlord problem in a new dress: a SaaS
sender is authorised by everyone. A mail host, a zone contact or an include
counts when its registrable domain is a target of the project, so
`mail.zendesk.com` says nothing and `serviceproviders.uber.com` says
everything. Measured: adding ubereats.com links it to uber.com on the SOA
contact alone, past nameservers (UltraDNS) and MX (Google) that say nothing.
An NS record is left to the WHOIS nameserver relation, so a fact appears once,
and an address that cdncheck flagged is not a shared machine.
`is_shared_host` folds the nameserver, shared-edge and mail-provider lists
into one judgement, so `aspmx.l.google.com`, `awsdns-hostmaster.amazon.com`
and `gold.foundationdns.com` are all recognised as rented.
The DNS and BGP modals carry a Shared with block through one
`related-targets` component: the other targets and the evidence, filtered to
the kinds that modal is about.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYop31s6BqDUt4TBs5mbwa
A project's targets were never correlated with each other. WHOIS could link
two of them and only through registration; nothing read what the scans
already knew, so a shared certificate, a shared network and a shared favicon
went unnoticed.
`TargetRelationService` answers "which other targets are this one" from five
facts, strongest first: a certificate that names both (by fingerprint or by
SAN), one registrant after normalisation, a network the organisation runs, a
nameserver nobody rents, and a favicon. Each dimension is read from the scans
that covered it, so the answer follows the same scope rule as every other
project-wide view.
A landlord is not a relation. Correlating targets on the ASN alone links four
unrelated companies eight ways through AWS, Cloudflare, Google, Fastly and
SendGrid - measured on this instance. A network only counts when its holder
is a party to the target (`owns_network`: the ASN holder's name against the
target's apex label and its registrant), which keeps AS63086 Uber
Technologies and drops every provider above. A favicon on more than
`FAVICON_MAX_TARGETS` estates is a framework's and is dropped too.
On the target page, Related targets now names the target and the evidence
rather than the WHOIS group: shopify.com reads "shopifycloud.com ·
Certificate · Registrant", one certificate served by beta-help.shopify.com
naming the other domain and one registrant behind both. The WHOIS records
that are not targets stay where they were, as domains to add, so a fact
appears once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYop31s6BqDUt4TBs5mbwa
Target correlation returned nothing for every target in a real project, and
three of its four signals could not have worked.
A registrant is matched on what two records must share to be the same party:
case, punctuation and the company suffix are stripped by `registrant_key`,
whose SQL half sits beside it (`registrant_key_sql`) because a match invisible
to one half is a relation that never appears. Both halves refuse a redacted
name, so "Redacted for Privacy" cannot group every privacy-protected domain
in the instance into one false cluster. Measured on shopify.com and
shopifycloud.com: one relation, on the registrant, and nothing between the
four unrelated estates beside them.
A registrar is dropped as a correlation. MarkMonitor registers for thousands
of unrelated brands, so "same registrar" asserts nothing - shopify.com and
uber.com share it and are not related. It stays available as a deliberate
lookup in the correlation dialog, which is a question the operator asks, not
a relation reNgine claims. The case-sensitive exact match it used would have
missed those two anyway: `MarkMonitor Inc.` against `Markmonitor Inc.`.
A nameserver is the one strong signal here - `a.ns.hackerone.com` is
identifying, `gold.foundationdns.com` is Cloudflare's enterprise DNS and
identifies nothing. Fifteen managed providers join the suppression list,
foundationdns and markmonitor among them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYop31s6BqDUt4TBs5mbwa
Four scans against real estates (hackerone.com 40 names, bugcrowd.com 71,
uber.com 11,203, shopify.com 26,235) show the correlation graph drawing the
CDN's identities and the resolver reporting a fraction of the live surface as
all of it.
The resolver's silence is re-queried once. dnsx prints nothing both for a
name that does not exist and for a query it dropped, and the loss is uniform,
so `_retry_degraded`'s peer median never fires (and a single-batch estate has
no peers). uber.com reported 135 of 11,203 names resolved; a plain dnsx re-run
over a random 200 of the 11,068 it called dead answered 7, the same 7 three
times over, and a DoH check of another 25 found 2 more. `_retry_silent`
re-queries every unanswered name once (`dns_retry_silent`), writes through
`_write_resolution` so the live view keeps up, counts recoveries as
`recovered`, and reports partial when that pass still recovers names.
A hub that belongs to a provider is not a correlation. bugcrowd.com drew six
hubs and five of them were the CDN's: four Fastly anycast addresses and the
`j.sni.global.fastly.net` CNAME every Fastly tenant is aliased to - four hubs
over one pair of hosts, each restating what the CDN kind already says. Hubs
now carry `platform`/`platform_label` from `ip_addresses.is_cdn` and from
curated lists beside `is_shared_nameserver` (`SHARED_EDGE_HOSTS`,
`PUBLIC_CA_NAMES`), the graph hides them behind "Hide provider", and
`related()` drops them. Curated, never inferred from a suffix:
`d3tqr1od3k1eh2.cloudfront.net` is one distribution, `j.sni.global.fastly.net`
is the whole platform; a private issuer stays visible.
Three more corrections in the same build. A share is read against the hosts
carrying that kind, not every name in the scan, so shopify.com's 16-of-18
JARM cluster no longer reads as 0.5%, and `common` needs
`MIN_ESTATE_FOR_COMMON` carriers. The 3,000-host cut is ranked by "answered,
then resolved, then name" - alphabetically it held 19 of shopify.com's 116
web assets, and ranking took that scan from 35 hubs to 94 and its certificate
correlation from 6 hosts to 66. A body under `MIN_BODY_BYTES` is not an
identity: 11 hosts shared the SHA-256 of an empty body. Hidden hubs no longer
consume the per-kind cap, chip counts equal what they draw, and the endpoint
goes through `lead_cache` like its neighbours (3.3s to 18ms).
Enrichment, which feeds all of it:
- `sync_ranges` takes an advisory lock and a budget. Two concurrent scans on
a fresh instance each pulled the same 200 MB and each truncated and
reloaded 1.1M rows: 291s against 145s. A scan that finds the tables
missing reports partial, and `app.tasks.ip_asn.refresh` fills the blanks
through `backfill_addresses` once the feed lands.
- No address is attributed from its target. The last-resort fill stamped the
target's BGP holder and its WHOIS registrant country onto every address the
range tables did not match, private ones included - a registrant country is
not a geolocation and a target's network is not the operator of everything
its scan found. It fired wherever that data exists, which is IP, range and
ASN targets and any registry that publishes a country; only an ASN target
now states its own sweep.
- cdncheck sees every address, httpx only the ones that answered, so
`ip_enrichment` folds `ip_addresses.is_cdn` back onto the hosts that
resolve to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYop31s6BqDUt4TBs5mbwa
Repo-wide sweep of every user-facing string: frontend components, routes,
config and stores; API details and route docs; shared definitions and
models; stage config help; report prose; MCP, toolbox, interest and
connector text; notification and worker messages.
Descriptions that explained, reassured, justified or restated their title
are deleted. Confirm dialogs name what is removed and stop. Failed actions
read "<Noun> not <verbed>", successes "<Noun> <verbed>". No reader address,
no product-as-subject, no em-dashes or parentheticals in prose.
Mirrors updated on both sides: EPSS band help, compare refusal reasons,
interest labels. Four tests asserting on old wording follow the new text.
The rules are condensed into CLAUDE.md under Writing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SK5o8xR96cbSnAfjYSTPSe
A sixth result dimension. Versions parsed from httpx tech strings and
Server headers match against a local NVD corpus, so an asset's known
CVEs are named without sending a request.
The stored CPE is not the input: 81 of 7,771 entries carry a version,
so matching on it returns every CVE ever filed against the product.
Versions come from tech and the Server banner, and the NVD pair comes
from a curated map rather than a name match — popper:popper is a 2005
webmail app, aos is a switch OS, and Apache's CVEs live under
apache:http_server, not apache:apache_http_server. An unmapped product
is reported as unmapped.
Ranges compare as zero-padded keys so the match is one SQL pass: a
letter suffix takes its own segment, trailing zeros drop, and numeric
and alphabetic schemes never order against each other. A criteria with
no version statement is discarded at load, because it means every
release. Measured against a reference implementation over 3,000,000
comparisons, residual disagreement is 0.07% and each case is the new
rule being the correct one.
Every match names what is unverified — conditional, backport,
fingerprint, coarse — and confidence is the count of those. Nothing
inferred enters the vulnerability count, triage or coverage a scanner
earned; software_cves keeps its own.
The corpus is a fourth feed, keyless, from the fkie-cad mirror of the
retired NVD JSON feeds: 2,197,146 ranges and 372,244 CVEs in 5m40s,
streamed rather than parsed whole (1,254 MB resident becomes 86 MB).
It rebuilds nightly and re-matches every covering scan, since the
corpus moves without a scan.
Exploitation coverage and the intel change feed leave Arsenal for the
dashboard; Arsenal keeps the feeds and the provider it configures.
Feed row counts, nouns and tables now come from FeedSpec — three
places had hardcoded two kinds, so a third feed rendered 0.
Port versions split out of the Server header in the migration: 223
rows become 3,037.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4NuBh4EsywioeryFZRNQR
The comment rule in CLAUDE.md is one line maximum, never narration: no
multi-line blocks, no module-header explainers, no sentences that explain or
justify the code to the reader. The tree had drifted a long way from it, and
most of the drift is mine.
Removed:
232 lines of multi-line comment blocks
203 single-line comments that argued instead of stating
154 comments joined by so/because/otherwise/rather-than
143 narrating comments in the frontend
112 test docstrings (the test name already says what it asserts)
25 test module docstrings and 25 decorative section dividers
Collapsed:
324 multi-line docstrings to one factual line
271 one-line docstrings that had a justifying clause bolted on
Kept: functional directives, section dividers, short field labels, alembic
revision headers, and the eight FastAPI route docstrings that render in /docs.
Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs.
284 python tests and 34 frontend tests pass, ruff and ruff-format clean across
ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
Every place that accepts user-supplied YAML — engine import, an
engine's stored yaml_source, nuclei template upload and report theme
upload — could be handed a billion-laughs document. Measured on engine
import: 379 bytes of nested aliases blocked the event loop for 36
seconds. A request issued alongside it timed out at 20s and the next
took 13s, against a 4ms baseline. Any authenticated user could do it,
repeatedly.
The cost is not in the parser. PyYAML shares an aliased node rather
than copying it, so safe_load returns in 2ms; the expansion happens in
whatever walks the result as a tree, which for us is pydantic — it has
no node cache, so nine levels of nine aliases is 43 million visits.
Bounding the parser would have missed this entirely.
load_document() now measures the document the way a consumer will walk
it and stops at 100,000 values, which is four times the largest thing
reNgine ships (a 24,133-node nuclei template, still indexed in 0.67s)
and far below the bomb. All four call sites use it and each reports the
limit in its own idiom — an HTTP 400, a template rejection reason, a
theme error.
Same payload after: 0.03s and refused, with API latency during the
attack at 36ms.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6LcH8uUddjJbKrahf34ys
Creating the tag "Web Servers" and then "web-servers" was a 500. The
name-to-slug mapping is many-to-one, but uniqueness was checked on the
name while the database constrains the slug, so any pair of names
differing only in case, spaces or punctuation collided on insert.
generate_slug also dropped every character outside a-z0-9, so a name in
any non-Latin script reduced to the empty string. The first such tag
was created with an empty slug, which made it unaddressable — the row
could never be read, updated or deleted through /tags/{project}/{slug}
— and the second one collided with it and failed.
Three routes and two services each derived slugs their own way; two had
a uniqueness loop and three did not, and the loop was a check followed
by an insert, so eight concurrent requests produced four 500s.
There is now one helper. generate_slug keeps its exact behaviour for
Latin names and falls back to Unicode word characters when that yields
nothing, so 東京支社 and Москва get real slugs. add_with_unique_slug
picks a free suffix, counting rows pending in the session as taken so a
CSV import carrying "Ops Team" and "ops-team" in one request lands two
tags, then inserts inside a savepoint and retries when the database
reports a slug conflict — re-raising anything else so an unrelated
constraint still surfaces its own error. Tag creation gained the
duplicate-name guard organizations already had.
Verified: eight concurrent creates over colliding names across tags,
organizations and projects, three rounds, no 5xx and one row per name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6LcH8uUddjJbKrahf34ys
Naming an engine or a scan context with more than 200 characters reached
Postgres and came back as a 500 — the column is bounded, the request
model was not. A name of spaces was accepted outright by engines,
contexts, schedules and projects, and a project created that way got an
empty slug, which is the segment its own URL is built from.
Proxies and notification channels already ran their names through
clean_name; every other named entity now does the same, on update as
well as on create, where none of them did. A rejected name says whether
it was empty or too long instead of failing at the database.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6LcH8uUddjJbKrahf34ys
Adding a target failed on the forms people actually paste. A value with
surrounding whitespace, a trailing slash, or a trailing root dot was
rejected outright, and "*.example.com" — how every bug bounty programme
writes a domain and its subdomains — was rejected too, though it names
exactly what a domain target already scans.
In the other direction any scheme validators.url accepts was taken as a
URL target, so ftp://x.com was stored, and the host was later read off
it as "ftp" and looked up in the registry under that name. A URL target
is now http or https, since that is what reNgine can scan.
A target value is normalised once, and that is what gets stored, so the
value on the page is the value that was validated.
The rejection said "Invalid target ID" for a value that is not an ID. It
now names the value and the accepted forms, in the same words the launch
dialog already uses. The two places that really were about a malformed
identifier keep saying so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6LcH8uUddjJbKrahf34ys
A registry only holds the registered name, but the lookup asked for the
target verbatim, so every subdomain target 400'd at the registry:
scanme.nmap.org went to the .org RDAP server, which has no such object.
Any target below the apex — api.example.com, www.bbc.co.uk — therefore
carried a permanent "WHOIS failed" badge. Lookups now ask for the
registrable domain, so bbc.co.uk is reached through the public-suffix
table rather than the last two labels.
Private and reserved address space has no registry record at all. That
was reported as a failure carrying the RDAP library's own wording
("IPv4Address 192.168.0.0/24 is private"), which put an error in the
activity feed and a red badge on the target for a fact about the
address. It is now recognised before any query is made and recorded as
not applicable, with a sentence that says why. The target page already
worded this correctly for a single private address; it now recognises a
private range too.
One lookup key is shared by the cache, the stored record and the API
read-back, so a domain and its subdomains resolve to one registry record
instead of re-querying on every miss.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6LcH8uUddjJbKrahf34ys
The first cut was rejected outright, and fairly. Four things were wrong.
The running head and foot sat 1.4mm from the physical sheet: vertical-align:top
on the @top-* margin boxes pins text to the paper, not to the text block. They
now hang off the block with RUNNING_CLEAR of air on the inside edge.
Severity was five saturated hues, restated by every theme and slightly
different in each. It is now one reserved ramp defined once, light rather than
heavy, with lightness carrying the rank so it survives deuteranopia; it clears
the CVD and normal-vision floors all-pairs on white paper. A dark page picks
up the dark restep automatically, so a dark theme carries no severity block at
all. shared/utils/color.py converts through OKLCH, so a theme names one colour
and the document derives the wash a badge sits on and the ink that sits on the
wash — every badge clears 7:1, including one from a theme a user uploads.
Badges are pills, never saturated slabs, and never inline-flex, which
WeasyPrint lays out as a block so the badge stretched its whole column.
Mono is for machine values now — hostnames, evidence, hashes. Labels, table
headers and the running head are the body face, and uppercase_labels finally
does something: the CSS had hard-coded uppercase and ignored the token.
Prose fills the text column, justified and hyphenated, instead of sitting in a
narrow measure with dead space beside it. Chapters still start a new page, but
a document may now ask them to run on, and the page estimate knows the
difference: 52 pages against 52, and 48 against 48.
Two defects the redesign surfaced: the risk summary never passed the variables
its own template reads, so that severity table had always printed empty; and
the standalone HTML export had no margins at all, because @page geometry does
not apply on screen.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AW5pitnosX8LEFEYxH8gFS
A gov.np HTTP probe ran 66 minutes over 25,916 host and port pairs, then lost every
row: a captured response body carried a NUL byte, postgres refuses NUL in text and in
json alike, and the failing flush rolled back the whole stage transaction. Recording
that failure then failed too — psycopg puts the offending parameters in its error
message, so the NUL travelled into `scan_activities.error` and took the failure path
down with it, leaving the activity RUNNING and the scan unable to finalize.
`shared/utils/text.py` is now the one place that names what postgres will not store.
`strip_nul` runs through every string, list and dict a tool record yields, applied in
`tools/runner/fieldmap.py` so every tool that maps a record is covered rather than
httpx alone. `strip_control` replaces the two identical local regexes in the nuclei
parser and the template indexer, and now also guards the activity log, the activity
timeline and the command recorder — the three places that write a tool's own words.
The HTTP probe's batch flush runs in a savepoint: a row postgres refuses costs that
row, retried one at a time, and the count comes back as a PARTIAL stage naming how
many responses could not be stored. Recording a stage failure is wrapped too, so a
second error while writing the first can never leave a scan running forever.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
- Implemented TargetEnrichmentStage for enriching targets with DNS, WHOIS, and BGP context.
- Added VhostStage for virtual host discovery using host-header fuzzing.
- Introduced WafDetectStage for fingerprinting web application firewalls.
- Created configuration files for each new stage with relevant parameters.
- Updated Dockerfile to copy stages instead of engines.
- Enhanced existing tools and services for better integration with new stages.
- Refactored task management in Celery to accommodate new stages and improve error handling.
Every scan now runs a discovery phase, gated per seed type:
- seed_resolution: expand ip / cidr / asn seeds into IpAddress assets
(stdlib ipaddress, even-stride sampling + rfc1918 skip on big ranges)
- reverse_dns: dnsx -ptr over discovered IPs
- ip_enrichment: asn / org / country from cached bgp + whois
- target_enrichment: consolidate cached whois/dns/bgp, refresh on miss/stale
The engine framework grows a phase axis (discovery/expansion/depth) and
per-stage applies_to gating so a stage skips cleanly when it doesn't apply
to the seed; subdomain discovery is retagged as expansion. Adds the
IpAddress model + migration.
- Added new types and interfaces for scan scheduling in the frontend.
- Created a new Svelte page for managing scheduled scans, including listing, creating, editing, and deleting schedules.
- Implemented backend models and services for scan schedules, including handling different schedule types (one-off, interval, daily, cron).
- Introduced a Celery task to manage the execution of scheduled scans based on their defined timing.
- Enhanced notification system to include alerts for newly discovered subdomains during scans.
- Added utility functions for handling time zones and scheduling logic.
- Updated the default route for automation to redirect to engines instead of scans.
- Removed the scans page and replaced it with a new scans overview page that lists scan history for projects.
- Introduced a detailed scan view page to display individual scan results and subdomain discoveries.
- Added new enums for scan statuses and subdomain sources to improve code clarity and maintainability.
- Implemented a new scan task in Celery to handle scan execution and logging.
- Enhanced the database models for scans and subdomains to support new features and improve data integrity.
- Updated the worker Dockerfile to include necessary scanning tools and dependencies.
- Added utility functions for domain matching and pattern compilation to streamline scan operations.
- Added `clean_name` function to `validation.py` for name validation with max length.
- Simplified docstrings in `dnsx` client, parser, and service files for clarity.
- Enhanced error handling in `ViewDNSService` to prevent disabling keys on rate limits.
- Improved WHOIS service to handle concurrent inserts gracefully with integrity checks.
- Updated `Settings` class in worker config to inherit from `BaseAppSettings`.
- Added new dependencies in `pyproject.toml` for `apprise` and `cryptography`.
WHOIS/DNS correlation was grouping unrelated targets on low-signal keys.
Add centralized privacy-redaction and shared-infrastructure detection,
drop country from auto-correlation, and bound every correlation query.
Backend:
- shared/utils/privacy.py: detect privacy-proxy/redacted registrant names
and emails (Withheld for Privacy, Domains By Proxy, WhoisGuard, GDPR
masked, etc.) so they never become correlation keys
- shared/utils/infra.py: detect shared/managed DNS (Cloudflare, Route 53,
GoDaddy, ...) so targets are not linked merely by DNS provider
- wire redaction into WHOIS entity extraction; guard correlation queries
defensively so legacy redacted rows do not correlate
- drop country from get_correlations_for_target (links whole regions)
- filter shared nameservers out of WHOIS nameserver correlation
- bound all WHOIS/DNS correlation queries (limit + distinct)
- exclude CDN-fronted IPs and shared nameservers from DNS correlation
- normalize country code casing at storage to fix missed matches
- return accurate cache-hit provenance from WhoisService.lookup
- reuse service finders from correlation API endpoints (DRY)
Frontend:
- remove country as a correlation type across types, API client, and the
related/lookup/overview WHOIS components (still shown as a record field)
- Added models for WHOIS responses including WhoisDomainResponse, WhoisIPResponse, and WhoisASNResponse.
- Created a parser for WHOIS data to convert raw responses into structured models.
- Developed an RDAPProvider class to handle RDAP lookups using the whoisit library.
- Implemented a WhoisService class for performing WHOIS lookups with optional database caching.
- Added methods for finding correlations between WHOIS records based on registrant, registrar, network, country, and nameservers.
- Updated Dockerfile to include tools directory and added rengine-tools dependency in pyproject.toml.
Yogesh Ojha