mirror of
https://github.com/yogeshojha/rengine.git
synced 2026-10-21 07:41:28 +02:00
695c4803bb2ab93f2dee2be2fa9430dc3afa71f4
189
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Yogesh OjhaandClaude Fable 5.1
|
695c4803bb |
refactor(engine): one intensity table, tiered fields, built-in Default engine
Every stage config field is a decision; rate, concurrency, timeout and retries come from a per-tool intensity profile the stage reads as its transport. 173 fields become 83: 41 transport values, 40 constants and the dead ones leave the models, and the builder groups by capability with support stages as checkboxes and advanced fields behind one disclosure. Aggressive now changes what a tool is handed. Every project carries a built-in Default engine, seeded on project creation and on first list, undeletable: stage defaults, subfinder as the only passive source, vulnerability scan on with nuclei. From the tracing review: tool args can no longer restate a flag the stage set, the session check obeys the probe scheme, excluded scope reaches every stage that reads addresses, hosts or paths, the sitemap and source-map providers throttle at the transport rate, the blind tier obeys its switch, the fuzzing lanes split the budget, and the removed private-address and evidence switches are back as advanced fields. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pq4xYJbgW5euUa9sCWBKFw |
||
|
Yogesh OjhaandClaude Opus 4.8
|
dd7ff0f6f1 |
feat(vuln): plan what each scanner is handed — typed inputs, origin clusters, tiers, dalfox
The vulnerability stage handed every http_assets row to nuclei, ordered by URL, with one 90-minute clock cutting wherever it landed. Template-spray then gave every host the first slice of the corpus in load order. On the two largest local scans that was 23.4M and 10.7M planned requests, of which 3.9% and 9% were sent, and no host recorded what it got. One GeoServer CVE was found 11 times on 11 hostnames of one server. Introduce one planner, shared/services/scan_surface/, that decides what every vulnerability scanner receives. It writes scan_surface_items per scan and is a pure function of the tables the scan already produced plus the context; nothing in it names a nuclei flag. Vocabulary lives in shared/definitions/scan_surface.py, mirrored by config/scan-surface.ts. - Inputs are typed by protocol, because nuclei rewrites input per protocol: roots for HTTP templates (any status), bare names for DNS, host:port for TLS and network, parameterised requests for fuzzing. Path and query are stripped from non-DAST input; values are normalised to one spelling. - Web assets that are one backend fold into one origin on exact agreement of address, certificate, body hash, status, title, canary and favicon — never a similarity score. Measured: 1,812 web assets to 1,607 origins on gov.np. - The corpus is spent in tiers: one-request checks on every asset, the universal set on each origin, product-matched checks through a curated tech map, the blind sweep last. Each tier is a ranked host-spray batch with its own coverage row; the budget cut lands between batches and every origin is scanned, covered or not scanned. - A finding on a representative is replayed onto each cluster member with one request, writing a real observed row or splitting the member out. Never an inferred row. - dast_scan is a new stage (off by default, after endpoint_probe): nuclei -dast over request items and exposure checks under directories, then dalfox behind a reflection probe. dalfox v3.2.2 is pinned in the worker image. Verified live on testasp.vulnweb.com: the tiers ran in order, nuclei -dast found two LFIs, dalfox executed under the reflection gate. All suites green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R7sBu1jBa4e2CXLS1KEsNr |
||
|
Yogesh OjhaandClaude Opus 5
|
de98b75947 |
refactor(hosting): replace the sankey with a fronting split over ranked networks
The three-column sankey's first column was always a single node, so two thirds of it was a stacked bar drawn as ribbons. On badssl.com — 15,477 web assets, one ASN — it degenerated to a 250px slab of one colour carrying one sentence, and 486 lines of layout, a linearGradient per link, an IntersectionObserver draw-in and a recursive hover walk paid for three numbers. GET /subdomains/hosting returns a HostingComposition: the fronting split, then the top MAX_NETWORKS AS orgs plus "Other networks" as ranked rows, each carrying its own fronting mix. hosting-body.svelte draws a stacked hairline with an inline legend and a ranked list whose meter is segmented by that mix, so "19 hosts in Salesforce AS14340, all direct to origin" is one glance. One body, used by hosting-panel.svelte on the scan Overview and hosting-section.svelte on the target Overview; the host passes its own padding. Rows reuse ranked-list.svelte, which gains segments, hint and a rendered meta, and the strip reuses composition-bar.svelte, which gains inline. Two count-is-a-promise breaks went with it. The service grouped on coalesce(Subdomain.asn, HttpAsset.asn) where the grammar's asn reads Subdomain.asn alone, and it read cloud off a DISTINCT ON (host) latest row where is:cloud is an EXISTS over every asset of the host, now bool_or. All 23 numbers on affirm.com equal their search/counts, checked across five scans. Unattributed hosts are a row, not a column: they appear as "Network not attributed" when some networks resolve, and when none do the list is dropped and the panel head says it instead of the network count. The dashboard widget showed two segments both reading "10,000 100%" — COUNT_CAP, with the denominator capped too. It now reads 10,000+ and drops the percentages when any count is capped. Its fronting colours come from config/hosting.ts, which the panel shares. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B6DocJiHaZQdMhgQcWddsb |
||
|
Yogesh OjhaandClaude Opus 5
|
06fa55824d |
perf(docker): build the worker image once and keep the go cache out of it
Four compose services declared their own build against worker/Dockerfile, so buildkit shared the solve and then exported and unpacked the same image once per tag. Compose does not dedupe identical targets, measured on a three-service probe. Under the containerd snapshotter the unpack is a second full pass over every byte, as expensive as the export. The go module and build caches were baked into layers: `go install nuclei` made a 2.66GB layer for a 190MB binary, and /root/go/pkg/mod (3.4GB) plus /root/.cache/go-build (3.3GB) were half the 13.1GB image. - the tools build in a throwaway stage with --mount=type=cache on GOCACHE and GOMODCACHE; the runtime stage copies /out/bin alone - worker-default builds rengine-worker:latest; worker-beat, ct-stream and flower reference it. Building one of those alone now needs worker-default built first - the runtime stage drops the go toolchain, build-essential and libpcap-dev; uv sync needs no compiler. libpcap0.8t64 stays, naabu dlopens it for SYN so ldd shows nothing and the loss would only surface at scan time - -ldflags=-s trims the binaries 650MB -> 495MB - only shared/ and tools/ are copied, being file:// path deps; stages/, reports/, interest/, toolbox/, mcp/, connectors/ and binaries/ are mounted - the frontend fetches tech icons above COPY frontend/. ., so a source edit no longer re-runs the tarball download - .dockerignore excludes .git (271MB) and node_modules (320MB); COPY frontend/. . had been copying the host's node_modules over the npm ci result worker image 13.1GB -> 2.41GB, frontend 1.38GB -> 969MB worker export+unpack 453s across three tags -> 55s across one editing stages/ or reports/ now rebuilds nothing (1s), frontend/src 4s, shared/ 19s; cold build 475s -> 414s, still dominated by go compilation Tool versions diff identical against the old image, naabu SYN proven against a live listener, 8/8 services up, 853 tests green, ruff clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015fWjAbvFBdmsmVeMcwaaR9 |
||
|
Yogesh OjhaandClaude Opus 5
|
0b853e04b6 |
chore: production-readiness review across the tree
A full static review of the 3.0 tree, run as two fan-out passes with every
finding verified against the running stack rather than taken on inspection.
Correctness and security
- instance settings encrypted the mask over the stored AI key: `!= MASK` never
matched `••••••••abcd`, so saving settings destroyed the key and `test_ai`
posted the mask as a bearer token
- the MFA token was replayable for its full five-minute life; the gate now
checks revocation and consumes the token after a successful exchange
- logout revoked nothing for a bearer client while answering "Logged out"
- `POST /api-keys/{id}/test` returned the plaintext key inside httpx's error
string, which embeds the whole URL
- an interest rule could be edited or deleted through another project's id
- redis ran the celery broker under `allkeys-lru`, free to evict a queued task
or a chord counter; now `noeviction`
- nuclei's stored command carried the OAST token and every auth header
- a finalize step could poison the session and leave a scan RUNNING for ever
- `is:issue` raised TypeError on every search
- offset paging repeated and skipped rows in four dimensions
- a cancelled run kept a stale `scans.*_found` rollup
- every PDF report raised `KeyError: 'software'`
- `version_key` truncated a wide numeric run and inverted release ordering
- psycopg raises a NUL as a bare ValueError, above StatementError, so every
savepoint guard was blind to the case it was written for
Performance
- endpoints.host had no trigram index: 35.9ms seq scan -> 0.389ms
- free text compiled to one OR, not the prescribed UNION ALL: 132ms -> 41ms
- `/http-assets/summary` and `/ips/summary` materialised whole tables
- the cross-link `ip` kind ran an estate query per page and drew nothing
Frontend
- `$effect` loaders that read the state they write, an unbounded request loop
on a failing endpoint
- error states that rendered as empty states
- 2FA backup codes were unusable while the copy said they worked
- a "Scan history" dialog that always reported none
- ~400 duplicated lines across five filter bars, now one component
Build and tooling
- every scanner installed at @latest; all 13 pinned to the measured versions
- CI built `web/`, gone since 2.x, with tags that cannot be valid
- the prettier pre-commit hook discarded its filenames and always passed
- the test suite destroyed itself when run twice
Also: subdomains from public code, via a pinned github-subdomains source and a
GitHub API key that subfinder receives too; sublist3r and oneforall removed
after measuring one unique host in 89 across four targets.
Verified: 853 tests, ruff clean over eleven packages, svelte-check 0 errors,
the count-is-a-promise gate at 131 checks and 0 mismatches, and 143 GET routes
with no 5xx. Four migrations, each with an exercised downgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVkdx5ynZ6aof5kLFczmhX
|
||
|
Yogesh OjhaandClaude Opus 5
|
689b0445b9 |
refactor(engines): remove the two engine settings no stage read
global_threads was multiplied by the scan context's thread multiplier and then discarded; every stage carries its own threads field with its own default, so the engine-level number had nothing to apply to. global_http_crawl was stored, exported, imported and never read at all. Both reached the engine form, the YAML, the launch preview and the run comparison, where "Crawl responses" was a material facet: two runs that differed on it were reported as able to find different things. Gone from the table, the three schemas, ResolvedScanConfig, AdHocEngine, the preview, both Compare facets and every frontend site, with a migration to drop the columns. An engine exported before this still imports: _RETIRED_KEYS accepts the two keys and ignores them, while an unknown key is refused as before. _MAX_ENGINE_THREADS goes with the validator that used it. It was declared twice in the same file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5roHYDYzSPRNaTkn8hSkJ |
||
|
Yogesh OjhaandClaude Opus 5
|
174b304c3d |
fix(scan): claim and dispatch a canvas under one row lock
run_scan took the scan row FOR UPDATE, wrote one task id, committed, and only then called apply_async. The commit released the lock, so a redelivered launch landing inside that window saw a single task id, passed the "already dispatched" guard — which needs two — and queued a second canvas for the same scan. Two subdomain_discovery processes on one target is the failure the run_epoch work was added to stop, reached by a different route. The dispatch now happens before the single commit, so a duplicate blocks on the row and then sees the finished claim. A crash between apply_async and the commit still leaves a queued canvas with the row unclaimed, the same as before; reap_stalled remains the net for that. dispatch_scan_run also takes the epoch rather than letting run_scan default it to 0. It was correct only because a fresh scan's run_epoch happens to be 0, which nothing stated or enforced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5roHYDYzSPRNaTkn8hSkJ |
||
|
Yogesh OjhaandClaude Opus 5
|
c88c4b5e00 |
fix(engines): refuse to store a masked credential
export_yaml masks global_headers and tool_options, and import_yaml feeds that document straight into create(), which had nothing to unmask it against. Exporting an engine that carried auth and importing it back produced an engine that sent `Authorization: ••••••••` to every target and handed nuclei `-api-key ••••••••`. Both validators now refuse a value carrying the mask, naming the header or the tool. update() keeps working: it restores the stored value first and validates what it is about to store, rather than what was submitted, so the editor can keep sending masked values back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5roHYDYzSPRNaTkn8hSkJ |
||
|
Yogesh OjhaandClaude Opus 5
|
d8122ae9c1 |
fix(secrets): refuse to run on a credential this instance cannot open
A stored secret that failed to decrypt was used as if it were plaintext. unseal_headers handed the Fernet token back, so every run whose context carried auth sent `Authorization: gAAAAAB...` to the target and reported success. EncryptedJSON returned None, so a scan context's auth silently vanished and the run went out unauthenticated. load_proxy_endpoints returned an empty list, so a scan configured to go through a proxy went direct instead. Rotating SECRET_KEY was enough to trigger all three. decrypt_stored() now separates the two cases a lenient decrypt conflated. A value that carries no Fernet token was written before sealing shipped (2026-09-06) and is returned as it is. A value that carries one and will not open raises SecretDecryptionError, naming the header, the column or the proxy and what to do about it. A stage turns that into a failed activity with the message on it; the api answers 503 through a handler beside the existing DBAPIError one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5roHYDYzSPRNaTkn8hSkJ |
||
|
Yogesh OjhaandClaude Opus 5
|
e21b9bc156 |
feat(targets): bring your seeds through bulk and CSV import, and say when an export lands
Seeds only reached a target added one at a time, which left the case #1538 actually asked about — a whole estate at once — needing the form fifty times. A bulk paste carries one scope list and every target keeps the lines that fall under its own apex: paste five domains and fifty subdomains and they sort themselves, with anything belonging to neither refused. A CSV row carries its own, under seeds, seed or subdomains, split on semicolons, commas or spaces. Its column lookup is now one helper rather than a fourth copy of the same eight lines. An export outlives the page it was started from, so it now posts the notification reports already post. Starting one from a scan, walking away, and finding it in the bell is the point — the toast only exists for whoever stayed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYZfaNjq3Ym8GPvWYuHisS |
||
|
Yogesh OjhaandClaude Opus 5
|
56eb5999c4 |
feat(correlation): correlate across every target, not just inside one scan
Four implementations answered "what does this host share?" and two of them could only answer it for one run: the project-wide page had no Correlation tab at all, and its detail sheet passed the row's own scan_id, so "Related" silently meant "related inside this one target's last run". One engine now, shared/services/correlation/, parameterised by QueryScope. kinds.py holds the vocabulary as data; hubs.py discovers hubs with a GROUP BY over the whole scope instead of grouping a 3,000-row sample in Python, so a hub is no longer missed for falling outside the cut. asset_query/groups.py derives its dimensions from the same registry, and correlation-graph, related and correlation take WebAssetScope. New /surface/correlation page. The count is a promise, and three bugs broke it. Asset kinds joined http_assets and counted rows, so a host answering on 80 and 443 counted twice (cert.fingerprint said 80 where the search returned 41). asn excludes CDN-fronted rows but its token did not, 8779 against 8815; KindSpec.narrows appends -is:cdn. And carriers came from a window over the group set the caller asked about, so one row's own value read as 100% of its carriers and common swallowed it. All 28 hub counts across 14 kinds now equal their searches. The provider rules moved into the engine. generic_page, shared_edge and cert_covers gated cross links alone, so estate-wide the graph drew 404 Not Found across 5 targets, 403 Forbidden across 4 and a 548-byte 404 body across 5 as correlations. A hub is judged once its members load and is labelled with who wrote it; hubs load MAX_HUBS_PER_KIND + 40 deep and re-rank. Related takes the whole vocabulary, each kind carrying the phrase it is announced with and the query its button opens, which retires relationDsl and RELATION_LABELS. asn became a kind, out of the default set. A scope over several targets opens on what crosses them: 3,000 nodes over 21,690 correlating hosts is a hairball, the 30 crossing hubs are the finding. Measured on 63,844 host rows across 10 covering scans: ~1.1s cold for the estate, 70-200ms for a scan, cached after. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FwCCFgRBC8ut6SQpmzfGAa |
||
|
Yogesh OjhaandClaude Opus 5
|
7781259c62 |
feat(exports): run an export as a job, and keep the recipe
POST /exports saves what the table was showing — dimension, scope, the filter body, the format — and queues a run. The row is the recipe, so rerun runs it again against today's rows rather than handing back yesterday's file, which is the point of keeping it. The scope resolves when the run starts, not when it was saved: a scan export reads that scan, a target or project export re-picks the newest covering scan of each target, so a saved export follows the estate. That needed a sync resolver, since surface_scope is async and the api is not on the worker's path. A bundle writes one file per dimension into a zip. Expiry is its own status rather than a failure wearing the wrong label, and a reaper fails a run the worker lost, which reports still lack. Streaming takes a connection of its own. yield_per opens a server side cursor, and the progress commit — or the per chunk join — executed on the session would invalidate it mid-file; the first large export failed with "named cursor isn't valid anymore" and nothing smaller ever would have. The export therefore reads committed rows, so its promise test moves to the durable fixture. Proven end to end: a filtered CSV that matches the table, 7,784 URLs one per line, a five file zip, a project wide export across 2,184 addresses, and a rerun. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYZfaNjq3Ym8GPvWYuHisS |
||
|
Yogesh OjhaandClaude Opus 5
|
3075ad3ddf |
refactor(surface): one definition of each dimension's filtered set
The five search services build their statement with pure, session-free helpers — _context, _apply_filter, _order, _scoped — and then enrich the page asynchronously. Only the enrichment needs the api. Moving the builders to shared/services/surface_query lets the worker compile the same filtered set the table shows, which an export has to do and could not reach from worker/app. Web assets gains the _scoped it never had: its base statement was inlined in search, leads, seeds and renders, so a predicate could drift between the rows a query returns and the rows its leads count. The four now read one definition. No behaviour change. Every route over the moved builders — search, leads, groups, renders and facets across all five dimensions — was exercised against a live scan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYZfaNjq3Ym8GPvWYuHisS |
||
|
Yogesh OjhaandClaude Opus 5
|
47148f2b2e |
feat(web-assets): group web assets that render the same page
Screenshots carry a perceptual hash, so hosts whose pages look alike are grouped in the gallery and the table. An exact hash matches only identical bytes, which leaves the same login across forty tenants in forty groups of one, so this is a 64-bit dHash taken from a fixed viewport band. Clustering follows the view's scope rather than being frozen at finalize: one run on a scan page, every target on the project-wide view. A cluster is every row within SCREENSHOT_DISTANCE of one representative, so its count and the screenshot: filter it opens are the same set by construction. Thresholds are measured over 9,890 stored screenshots on four estates. Distance 2 collapses the cPanel port variants and a default page's one-bit variants while 4 begins merging different skins. A hash setting fewer than four bits is a blank render and never represents a cluster; a variance floor let a one-bit hash sink 79 hosts into one group. Screenshots are held as WebP at native size, 18% of PNG, converted by a background task rather than the stage: httpx writes the PNG itself, so converting inline saves no peak disk and costs every scan serial time. Only settled scans are touched, or the path is rewritten under a stage still writing it. The hash is taken before the encode and never recomputed, so compression leaves the grouping untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2BnU1s9wZg3KYgCHBxhuC |
||
|
Yogesh OjhaandClaude Opus 5
|
1902be366b |
fix(scans): stop a paused scan's tools and answer both controls at once
Pause revoked with terminate=True and SIGKILL. A SIGKILLed celery child runs no finally and no watchdog thread, so its subprocess.Popen tool was reparented to init and kept hitting the target with its budget and idle timer dead — an unbounded scan against a host the operator believes is paused. Revoke now drops the queued tasks only; a task already running stops through its own abort check, which kills the tool and reaps it. tools/runner/abort.py binds the stage's check for the whole process, so a tool stops even where the call site passed no should_stop. The blocking run() path has no abort poll yet and still ends at the tool's own timeout. Resume was answered by the worker, so the api replied "paused" and the page waited on a celery round trip — with the worker busy the control looked dead. The api now takes the row lock, folds the pause, sets RUNNING and bumps the epoch itself, then dispatches a task that only builds the canvas. Pause and resume answer in ~20ms with the true status and the stores hand the row back, so the page flips without a refetch. run_scan read run_epoch at execution time, so a launch still queued when a scan was paused and resumed adopted the new epoch and dispatched a second canvas beside the resumed one: two subdomain_discovery processes on one scan. It carries its epoch like every other signature now, and superseded() is the one definition all four guards share. Also: cancel kept the open pause out of paused_seconds, so a run cancelled after sitting paused reported the pause as run time; pause and cancel take the row lock, so five concurrent calls give one winner instead of duplicate ledger entries; a watch probe refuses to pause, since its host waits on a terminal status that pause never reaches; the hero, ribbon and empty state get a paused arm rather than falling into the settled one and claiming a delta for a run that has not looked yet; isOpenStatus mirrors SCAN_OPEN_STATUSES so Compare, the timeline and bulk cancel agree with the api; and the pause confirm is gone — it removes nothing, so the cost is stated on the control instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rg2cefwCvz9f77A2SHEHDb |
||
|
Yogesh OjhaandClaude Opus 5
|
c41f0c860e |
feat(targets): store the assets you already know and seed every scan with them
A target carries a list of host names, addresses and URLs an operator already holds. Every scan of that target starts from them, so a list earned elsewhere no longer has to be re-pasted or re-discovered. Each pasted line is classified by its own shape and scope-checked against the target; a refusal names the line and the reason. Seeds enter a run through the asset_seed stage, which resolves the names it has no answer for and materialises what they resolve to. Stages the engine has switched off still do not run. Seeds no longer force a focused run: scope reads the new seed_only, which only rescan, watch and connector runs set, so a seeded census scan still covers its target. Seed count is a material run facet, so Compare states it before any delta it would otherwise explain. Two stages share the subdomains table for a run: discovery's clear no longer drops rows asset_seed wrote, and the run's seeded hosts join its merge so the imported source survives a name discovery also finds. Closes #1538 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYZfaNjq3Ym8GPvWYuHisS |
||
|
Yogesh OjhaandClaude Opus 5
|
c0151af343 |
feat(scans): pause a running scan and resume where it stopped
A long run could only be cancelled and started over. A scan now holds at the stage boundary: pause settles the stages in flight as PAUSED rather than ABORTED, and resume dispatches a fresh canvas from the first level that has not finished. run_epoch guards the handover. A paused stage's celery task may still be in flight, so every stage and finalize signature carries the epoch it was dispatched under and drops out when the scan has moved on — without it a straggler from the paused canvas would write into the resumed one. resume_point() reads the level and the finished set through stages_done(), so pause, resume and the stall reaper all agree on what "finished" means. Paused time is held in paused_seconds and taken out of the reported duration, so a scan's elapsed figure stays the time it actually ran. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XxW9yfg1SEchtg6HLGHRkK |
||
|
Yogesh OjhaandClaude Opus 5
|
873f8ef4c4 |
fix(search): reject a filter field that does not exist
A search filter took any JSON object, so a client sending the wrong field names got the whole scope back with no error and no sign anything was ignored. The repo's own lead-cache test was built on it: it passed page and size, which SubdomainFilter has never had. Every dimension filter now forbids an unknown field, the way ScanCreate already does, and the test uses the real offset and limit. Two more while here: nuclei's _note_gaps had been copied inline into _one and left behind unused, so the copy is gone; and the launch preview now says when a target already has a run in flight, since nothing stopped two scans pointing at one host from the same address. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XxW9yfg1SEchtg6HLGHRkK |
||
|
Yogesh OjhaandClaude Opus 5
|
dcdba5cfd4 |
feat(bounty-hub): read Intigriti through its researcher API
A platform with a researcher API is now a provider: one module under shared/services/bounty_providers implementing programs() and scopes(), returning rows the sync engine stores. bounty_programs.py became the platform-agnostic engine, so adding a platform is one module, one APIProvider value, one PlatformSpec.api_provider and a key tester. Intigriti is the second connected platform, on a personal access token the hunter creates. On this instance: 220 programs, 54 of them private, 155 with scope and 65 the platform refuses, which accounts for all 220. Measured against the live API, against the docs: - Tier ids are unordered and undocumented. Id 4 is Tier 1, not No Bounty, so the first mapping stored the best-paying assets as paying nothing. Eligibility reads the label; out of scope is matched on id and label. - A 403 carrying FORBID001 is a refusal, not the documented rate-limit overage, and it covers public programs the account has not joined. Retrying it cost 11s a program; AccessDeniedError ends it in one request and records scope_access, keeping the failure budget for real breakage. - Membership is proved by the platform letting the token read a private program's scope. confidentialityLevel never reports "invite only", so joined was always false and its filter was dead. Bounty Hub follows what the data actually says: - source becomes sources, a JSONB array. A program is routinely in both a platform API and the public feed; the feed adds itself without taking an API row back, and facet counts overlap accordingly. - The row hint read "Bookmarked on HackerOne" while rendering Intigriti's follow flag. It now reads PlatformSpec.follow_label. - Refresh queued the sync and returned, so nothing changed on screen. It waits for the sweep, reloads, and says so. - The dashed connect card per platform is one dismissible Alert. - Connecting a platform no longer diffs API scope against feed scope, which would report every asset removed and re-added. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SiHx8Sm34hS1nanrsPdBaD |
||
|
Yogesh OjhaandClaude Opus 5
|
e7a5359624 |
feat(web-assets): say when another target carries the same value
A row states that a favicon, page title, body hash, certificate, CNAME target or address is also held by a host under a different target, and the popover names the peers and pivots to the query that returns them. Read against the estate, so a row on a single-scan page still reaches the other targets. The refusal is the feature. Measured across four unrelated estates, all five cross-target values were a provider's and none is drawn. A carrier on a SHARED_EDGE_HOSTS CNAME vouches for no identity. generic_page() refuses a title that restates the status of the rows carrying it, the soft-404 phrases and the curated edge interstitials, and it gates the three page-derived kinds. cert_covers() requires a certificate to name the host presenting it, so *.okta.com on a tenant is not a link. JARM spanned all four estates and is not a cross link. Each value is judged once before the row loop and _crossing() refuses to fetch carriers for a value one target holds: a wildcard CNAME with 8,679 carriers cost 4.6s a page, now ~80ms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LxAonr95Uj7Bdu4CTMCwx9 |
||
|
Yogesh OjhaandClaude Opus 5
|
2ea061e713 |
feat(targets): a DNS modal that opens, and DNS as a relation
Clicking the DNS cell in the targets list opened the generic target dialog: there was no DNS modal at all. There is one now, and it renders through the same `dns-tab` the target page uses, so the records, the type filters, the provider labels and the refresh are one component in two places. DNS is also a relation. `dns_record` links two targets on a mail host, a zone contact, an address or an SPF include - and only when the estate owns the name. On the four unrelated estates here the only shared DNS values are Google's MX hosts and an `include:mail.zendesk.com` that briefly linked Shopify to HackerOne, which is the landlord problem in a new dress: a SaaS sender is authorised by everyone. A mail host, a zone contact or an include counts when its registrable domain is a target of the project, so `mail.zendesk.com` says nothing and `serviceproviders.uber.com` says everything. Measured: adding ubereats.com links it to uber.com on the SOA contact alone, past nameservers (UltraDNS) and MX (Google) that say nothing. An NS record is left to the WHOIS nameserver relation, so a fact appears once, and an address that cdncheck flagged is not a shared machine. `is_shared_host` folds the nameserver, shared-edge and mail-provider lists into one judgement, so `aspmx.l.google.com`, `awsdns-hostmaster.amazon.com` and `gold.foundationdns.com` are all recognised as rented. The DNS and BGP modals carry a Shared with block through one `related-targets` component: the other targets and the evidence, filtered to the kinds that modal is about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YYop31s6BqDUt4TBs5mbwa |
||
|
Yogesh OjhaandClaude Opus 5
|
a180a3020a |
feat(bounty-hub): say which program covers a target
The scope link ran one way: a program's scope showed whether each entry was already a target. The question a hunter asks is the other one - I have this asset, who pays for it - and nothing answered it. `ProgramCoverageService` matches a host against every synced program's scope through the machinery a watch already uses (`plan_scope`, `WatchItem.matches`, `excluded_by`), so a wildcard entry covers its subtree, an exact entry covers itself, and a program's own out-of-scope rows mark the match excluded rather than hiding it. Measured against 7,173 synced scope rows on this instance: `foo.bugcrowd.com` and `deep.nested.bugcrowd.com` resolve to Bugcrowd through `*.bugcrowd.com/auth/*`, `notacme.com` is not a subtree of `*.acme.com`, and `allegro.cz` comes back uncovered because that program lists it out of scope. The target page carries it in the rail as Bounty: the program, its platform and the scope entry that covers the target, hidden outside bug bounty mode. `for_host` takes a bare hostname so a discovered asset can be asked the same question next. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YYop31s6BqDUt4TBs5mbwa |
||
|
Yogesh OjhaandClaude Opus 5
|
570914c448 |
feat(targets): show which targets are the same estate
A project's targets were never correlated with each other. WHOIS could link two of them and only through registration; nothing read what the scans already knew, so a shared certificate, a shared network and a shared favicon went unnoticed. `TargetRelationService` answers "which other targets are this one" from five facts, strongest first: a certificate that names both (by fingerprint or by SAN), one registrant after normalisation, a network the organisation runs, a nameserver nobody rents, and a favicon. Each dimension is read from the scans that covered it, so the answer follows the same scope rule as every other project-wide view. A landlord is not a relation. Correlating targets on the ASN alone links four unrelated companies eight ways through AWS, Cloudflare, Google, Fastly and SendGrid - measured on this instance. A network only counts when its holder is a party to the target (`owns_network`: the ASN holder's name against the target's apex label and its registrant), which keeps AS63086 Uber Technologies and drops every provider above. A favicon on more than `FAVICON_MAX_TARGETS` estates is a framework's and is dropped too. On the target page, Related targets now names the target and the evidence rather than the WHOIS group: shopify.com reads "shopifycloud.com · Certificate · Registrant", one certificate served by beta-help.shopify.com naming the other domain and one registrant behind both. The WHOIS records that are not targets stay where they were, as domains to add, so a fact appears once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YYop31s6BqDUt4TBs5mbwa |
||
|
Yogesh OjhaandClaude Opus 5
|
913815acd4 |
fix(recon): correlate the estate, not the provider it rents
Four scans against real estates (hackerone.com 40 names, bugcrowd.com 71, uber.com 11,203, shopify.com 26,235) show the correlation graph drawing the CDN's identities and the resolver reporting a fraction of the live surface as all of it. The resolver's silence is re-queried once. dnsx prints nothing both for a name that does not exist and for a query it dropped, and the loss is uniform, so `_retry_degraded`'s peer median never fires (and a single-batch estate has no peers). uber.com reported 135 of 11,203 names resolved; a plain dnsx re-run over a random 200 of the 11,068 it called dead answered 7, the same 7 three times over, and a DoH check of another 25 found 2 more. `_retry_silent` re-queries every unanswered name once (`dns_retry_silent`), writes through `_write_resolution` so the live view keeps up, counts recoveries as `recovered`, and reports partial when that pass still recovers names. A hub that belongs to a provider is not a correlation. bugcrowd.com drew six hubs and five of them were the CDN's: four Fastly anycast addresses and the `j.sni.global.fastly.net` CNAME every Fastly tenant is aliased to - four hubs over one pair of hosts, each restating what the CDN kind already says. Hubs now carry `platform`/`platform_label` from `ip_addresses.is_cdn` and from curated lists beside `is_shared_nameserver` (`SHARED_EDGE_HOSTS`, `PUBLIC_CA_NAMES`), the graph hides them behind "Hide provider", and `related()` drops them. Curated, never inferred from a suffix: `d3tqr1od3k1eh2.cloudfront.net` is one distribution, `j.sni.global.fastly.net` is the whole platform; a private issuer stays visible. Three more corrections in the same build. A share is read against the hosts carrying that kind, not every name in the scan, so shopify.com's 16-of-18 JARM cluster no longer reads as 0.5%, and `common` needs `MIN_ESTATE_FOR_COMMON` carriers. The 3,000-host cut is ranked by "answered, then resolved, then name" - alphabetically it held 19 of shopify.com's 116 web assets, and ranking took that scan from 35 hubs to 94 and its certificate correlation from 6 hosts to 66. A body under `MIN_BODY_BYTES` is not an identity: 11 hosts shared the SHA-256 of an empty body. Hidden hubs no longer consume the per-kind cap, chip counts equal what they draw, and the endpoint goes through `lead_cache` like its neighbours (3.3s to 18ms). Enrichment, which feeds all of it: - `sync_ranges` takes an advisory lock and a budget. Two concurrent scans on a fresh instance each pulled the same 200 MB and each truncated and reloaded 1.1M rows: 291s against 145s. A scan that finds the tables missing reports partial, and `app.tasks.ip_asn.refresh` fills the blanks through `backfill_addresses` once the feed lands. - No address is attributed from its target. The last-resort fill stamped the target's BGP holder and its WHOIS registrant country onto every address the range tables did not match, private ones included - a registrant country is not a geolocation and a target's network is not the operator of everything its scan found. It fired wherever that data exists, which is IP, range and ASN targets and any registry that publishes a country; only an ASN target now states its own sweep. - cdncheck sees every address, httpx only the ones that answered, so `ip_enrichment` folds `ip_addresses.is_cdn` back onto the hosts that resolve to it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YYop31s6BqDUt4TBs5mbwa |
||
|
Yogesh OjhaandClaude Fable 5.1
|
db030d733c |
feat(bounty-hub): watch a program
Scope-true continuous discovery for a bounty program, bug bounty mode only (capability `program_watches`). Watching a program imports its in-scope targets under the program's organization, writes out-of-scope entries as exclusions and the rate ceiling on a `Watch · <program>` scan context, schedules an optional baseline, and streams certificate transparency for the watched apexes. A new in-scope host is resolved, written into the covering scan with source `ct_log`, probed by a focused run with a screenshot, evaluated against the watch's alert query and alerted once per fingerprint through the chosen channels. - ct-stream compose service: certspotter 0.24.2 under a supervisor that regenerates the watchlist from active watches, spools events through a hook script that emits only certspotter fields, and reports over Redis - worker tasks: certificate, settle, recheck, reconcile; finalize dispatches settle for any probe run; reconcile after every program or feed sync - notifier: named channels bypass per-channel preferences; attachments; new `watch` notification type - API under /bounty-programs/watches with preview, ledger, events, marks - Bounty Hub: Watching tab, Watch dialog, watch sheet, program badge, certificate stream card in Settings - schedules carry an intensity; scan_schedules.intensity Two review rounds folded in: compare-and-set probe claim, resolver-down handling, spool kept on failed dispatch, schedule cap, mute keeps prior state, ledger-derived totals, event filters, mode lockout of worker and stream in corporate mode. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ba2f7gwu2GXiETCKqRztyB |
||
|
Yogesh OjhaandClaude Opus 5
|
e3b310b5b1 |
feat(cve): CVEs become a surface of their own
One evidence axis runs across both findings dimensions
(shared/definitions/evidence.py): inferred < observed < corroborated <
proven. Software rows store inferred or corroborated, set in match_scan from
a host and CVE join to non-info, non-suppressed findings; vulnerabilities
store observed or proven, proven being an out-of-band interaction recorded at
write. Both grammars gain an evidence: field and the old is:proven becomes
is:recorded. GET /cves and GET /cves/{cve} answer "am I affected by X" across
both dimensions, with an MCP tool and a toolbox lookup beside them. Ladder
counts are count(*) FILTER over the same predicates the searches compile, so
a rung opens exactly the rows it counted; tests/grammar/test_evidence.py
asserts it. match_scan carries discovered_at forward through a
software_previous temp table and returns the newly exposed delta for the
nightly notification; nvd_corpus.load skips the download when the mirror
stamp is unchanged.
The Attack surface group had grown to eight items with three of them sharing
a word. Vulnerabilities is now one entry parenting three sibling routes
joined by a tab rail: Findings, Software and CVEs. The routes are unchanged,
so vuln_q and sw_q state, deep links and per-route caching survive, and
NavItem.match keeps the parent lit on the siblings. The Software dimension
label drops to "Software" while its rows keep the noun "software CVE".
The CVE index gets the layout its siblings have: coverage strip, a search bar
attached to the card, a severity rail, a filter row with sort and refresh,
ListHeader rows and pagination. components/search-bar.svelte is the
plain-text twin of QueryBar for surfaces with no grammar. GET /cves gained
page, size, sort, dir, severity, kev, ransomware and evidence and returns
severity_counts; severity and the top rung are resolved in SQL from
SEVERITY_RANK and EVIDENCE_RANK rather than in Python.
Counts verified against the rows they open: overview.cves 1,165 equals the
index total, the Software tab 91,039 equals coverage.findings, and the
severity rail 157/373/583/52 sums to 1,165. _INDEX_TOTAL now excludes the
triage-suppressed findings the listing already excluded, and the software
dimension is counted exactly rather than capped at COUNT_CAP, so the tab
matches the page. The distinct-CVE count is 10s with JIT on and 91ms under
SET LOCAL jit = off, leaving the overview at 137ms cold and 17ms warm.
Also fixes software-table, which passed a 1-based page to ResultsPagination
where every other table passes a 0-based one, so page one read
"Showing 51-100".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MirAGiQ2fSJJtikWi6YRNj
|
||
|
Yogesh OjhaandClaude Fable 5.1
|
91e9a6e3b8 |
feat(endpoints): drop junk URLs by rule before the insert and after the probe
URL rules run inside the inventory writer: static files, crawler artifacts, platform noise, and URLs past the per-family, per-language and per-folder caps are not stored. Tracking and cache-busting parameters, session path segments and index files are cleaned off every URL. Response rules run after the probe and delete rows the answer proves are not endpoints: not found against a three-request canary per root, archive rot, off-scope and fan-in redirects, the root's body at another path, identical and similar bodies. Every drop is counted per rule on the source's coverage row and shown in the Endpoints coverage popover. A family column sits beside the untouched signature (host, identifier- and language-folded shape, parameter names) and the probe partitions by it. Five knobs on the URL Discovery stage; thresholds stay as constants. Proxy, imported and fuzz rows are protected. katana runs with query parameters ignored and static extensions filtered. Measured on hamropatro.com: 21,430 URLs found, 694 stored, 20,448 dropped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkwdfHUdcUc4ToUoe55HMZ |
||
|
Yogesh OjhaandClaude Fable 5.1
|
ca32cc62f3 |
feat(connectors): browsing lands in the endpoints inventory
Proxy observations are written as endpoint rows with source proxy into the target's covering scan; a target without a census scan gets a Browsing run, and finalize replays recorded shapes into a new census scan. "Seen by a scan" is host plus path shape (new endpoints.shape column, backfilled), with parameter differences reported as their own notice and unseen claimed only when a settled scan covers the target. Removed the scan trigger, sessions and coverage surfaces. The jar is served from binaries/, the setup dialog derives the endpoint from the browser origin, scans dispatch one run per target with the chosen scan context, and manual findings mask credential headers. Endpoints page gains Browsed and Not browsed filters; the Burp tab gains a host line, a notices table with Open in reNgine, and a report dialog with notes and a result. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PMmqqNaMMjCJWzGG49Xc1i |
||
|
Yogesh OjhaandClaude Fable 5.1
|
0a69f3201c |
copy: state the fact, delete the explanation
Repo-wide sweep of every user-facing string: frontend components, routes, config and stores; API details and route docs; shared definitions and models; stage config help; report prose; MCP, toolbox, interest and connector text; notification and worker messages. Descriptions that explained, reassured, justified or restated their title are deleted. Confirm dialogs name what is removed and stop. Failed actions read "<Noun> not <verbed>", successes "<Noun> <verbed>". No reader address, no product-as-subject, no em-dashes or parentheticals in prose. Mirrors updated on both sides: EPSS band help, compare refusal reasons, interest labels. Four tests asserting on old wording follow the new text. The rules are condensed into CLAUDE.md under Writing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SK5o8xR96cbSnAfjYSTPSe |
||
|
Yogesh OjhaandClaude Opus 5
|
2b17f345af |
feat(software): infer CVEs from the versions assets report
A sixth result dimension. Versions parsed from httpx tech strings and Server headers match against a local NVD corpus, so an asset's known CVEs are named without sending a request. The stored CPE is not the input: 81 of 7,771 entries carry a version, so matching on it returns every CVE ever filed against the product. Versions come from tech and the Server banner, and the NVD pair comes from a curated map rather than a name match — popper:popper is a 2005 webmail app, aos is a switch OS, and Apache's CVEs live under apache:http_server, not apache:apache_http_server. An unmapped product is reported as unmapped. Ranges compare as zero-padded keys so the match is one SQL pass: a letter suffix takes its own segment, trailing zeros drop, and numeric and alphabetic schemes never order against each other. A criteria with no version statement is discarded at load, because it means every release. Measured against a reference implementation over 3,000,000 comparisons, residual disagreement is 0.07% and each case is the new rule being the correct one. Every match names what is unverified — conditional, backport, fingerprint, coarse — and confidence is the count of those. Nothing inferred enters the vulnerability count, triage or coverage a scanner earned; software_cves keeps its own. The corpus is a fourth feed, keyless, from the fkie-cad mirror of the retired NVD JSON feeds: 2,197,146 ranges and 372,244 CVEs in 5m40s, streamed rather than parsed whole (1,254 MB resident becomes 86 MB). It rebuilds nightly and re-matches every covering scan, since the corpus moves without a scan. Exploitation coverage and the intel change feed leave Arsenal for the dashboard; Arsenal keeps the feeds and the provider it configures. Feed row counts, nouns and tables now come from FeedSpec — three places had hardcoded two kinds, so a third feed rendered 0. Port versions split out of the Server header in the migration: 223 rows become 3,037. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H4NuBh4EsywioeryFZRNQR |
||
|
Yogesh OjhaandClaude Fable 5.1
|
97e5f7aa81 |
copy(glossary): a probed host name is a web asset
Target is what the user adds. The vulnerability coverage line said "checks against 10,199 targets" beside a sidebar listing seven; it now says assets. The Web assets row was also called host, web host and live host across the scan overview, the hygiene panel, the dashboard widgets, the issue rows and the report narratives; those read web asset now. Host and hostname stay where they mean the DNS name as a value, a proxy host:port or the scanner host. lib/glossary.test.ts fails on the two retired phrases. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VFwnNBRuLhL63kgpeoYfx8 |
||
|
Yogesh OjhaandClaude Fable 5.1
|
c0b0591ed0 |
feat(hygiene): derive web hygiene checks from stored responses
Seventeen hardening checks evaluated inside the write from response headers, cookies, CSP and CORS: HSTS, HTTPS redirect, framing, nosniff, CSP presence, enforcement, inline and wildcard script sources, cookie Secure and HttpOnly, session cookie caching, CORS origin and credentials, server and runtime disclosure, Referrer-Policy. Each check is pass, fail or not applicable, so every count is measured against the hosts the check applies to. Stored per asset on http_assets and folded as a union onto the host row by one SQL statement shared by the stage and a beat backfill that evaluates responses stored before the checks existed. Cookies are parsed from the Set-Cookie lines of the raw header; httpx joins repeated headers with commas and cookie expiry dates contain commas. Surfaces: hygiene: query field with warning, info, any and none, nine counted leads, group-by and facet, a Web hygiene panel on the scan Overview and the project-wide Web assets page, a host table column and hover line, a sheet section with an HTTP Hygiene pane, a compare watched field and a report section. No score and no vulnerability rows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JvutSbg8hHxUMCSTMpQctW |
||
|
Yogesh OjhaandClaude Fable 5.1
|
d4072b8efa |
perf: make the read path scale with the estate
Scan page froze on a duplicate each key; every tab body now sits in a svelte:boundary. Composite discovered_at indexes on the five result tables, leads counted in one FILTER pass, dashboard and scan-service first-seen counts bounded to the scans in the window, ANALYZE at finalize, shm_size on the db service, SQL_ECHO instead of DEBUG, an entrypoint that serves with workers outside reload. Settled-scope aggregates and search pages are cached by target revision; triage, dismissal, interest evaluation, notes, certificate rechecks and re-ranking bump it. Requests per page cut: single-flight GETs, one schemas endpoint, batched whois correlations, a counts endpoint for the dashboard, deduped store loads, trimmed coverage payloads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NWRq8utu7KXsVBVU3gYNi7 |
||
|
Yogesh OjhaandClaude Opus 5
|
00fb63638a |
refactor(contexts): drop the baseline fields that were never built
`scan_contexts.compare_baseline_scan_id` and `scan_only_new_assets` have been in the table since May and rejected at the API ever since with "Baseline comparison is not available yet." The scope section told the operator "Baseline comparison and new-asset-only scanning are chosen when a scan is launched" — neither was chosen at launch, or anywhere. Baseline comparison now exists as its own view, so half of what these fields reached for is delivered. The other half is not a context field: what counts as a new asset cannot be known before discovery runs, so "scan only new assets" means filtering assets between pipeline levels — every stage after discovery skipping anything present in the baseline. That needs a coverage story of its own, because a stage that silently skips an asset is exactly how a count starts lying. It should be scoped on its merits rather than sit as a column and a promise. A field the product advertises and the API refuses is worse than no field. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KAVgTQe8K3EvGzPo5HVmKF |
||
|
Yogesh OjhaandClaude Opus 5
|
3f6302870c |
feat(scans): compare two runs, and prove they are comparable first
reNgine could say what was new against a target's whole history, and 2.x could say `[new, removed]` for subdomains. Neither answers the question an operator actually asks after a scan finishes: what changed since that run, the named one I am comparing against. The hard part is not the diff. Two StandardRengine runs of vulnweb.com in this instance, eighteen minutes apart, went from 14 findings to 0. A set diff reports "14 findings resolved, 12 high". The truth is `vulnerability_scan.max_minutes` went 0 to 2, and the coverage rows record 57,100 requests over 368s against four hosts, then 27,582 over 120s against seven. The delta belongs to the run. So comparability is stated before any delta, per dimension: like for like, settings differ, runs differ in depth, not scanned. Stage settings and the runs' own setup are both compared — engine, context, auth type, intensity, proxy (as set/none, never the URL), scope exclusions, protocol, crawl. Facets that change what a run can find raise the verdict; threads and rate limits do not. Runs that ran between the pair are counted, a cancelled run in the pair lowers the verdict, and when the chosen pair is not like for like the nearest run that would be is offered. Identity is read off the per-scan unique constraints already on each table — `subdomains.name`, `endpoints.signature`, `ports.(ip, number, protocol)`, `ip_addresses.ip`, `vulnerabilities.fingerprint` — so the diff key is not a new invention. Four verbs: appeared, changed, disappeared, unchanged. Disappearance is only asserted when the producing stage finished cleanly; dnsx drops 20-60% of names silently, so a partial run's missing rows are unconfirmed, a bucket of their own. A dimension one run did not produce reports not scanned, never everything removed. EPSS and KEV movement is Class D and is counted apart. A disappeared row does not exist in the later run, so a query scoped to it can never return one. Rather than teach five grammars an `is:added` flag, the service flips the scope per verb, and every column, sort key and detail sheet keeps working. Rank is computed once as a SQL CASE and the signal name is recovered from the rank, so the order and the label cannot disagree. Nothing is stored: a 425k-row full outer join measures 66 ms, the whole endpoint about 31 ms. No table, no migration. Direction is a shape — solid, split, hollow rails — not green-and-red. New attack surface is not good news. Diff mode keeps the git glyphs for the same reason it colours them from the `--code-*` ramp: in a diff `+` is syntax. That mode is generated once on the server and shown through `code-block`, so the screen and the clipboard cannot drift. Eligibility is explained, never hidden. `REFUSAL_REASON` is one definition mirrored in `config/compare.ts`, so the disabled button's hint, the alert page and the API error say the same sentence, and an impossible pair reached by URL lands on an alert that lists the target's runs to pick from. `recheck.py` now reads its host field list from the shared definitions, so the product holds one answer to "did this host change". Also fixes the scan hero, which said "No change since the previous scan" while measuring only hosts. Beside a page reporting 21 changes that read as a contradiction; it now says what it measured. Vulnerability rows are one per finding rather than grouped by check, and comparing against a run that is still going is not built. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KAVgTQe8K3EvGzPo5HVmKF |
||
|
Yogesh OjhaandClaude Opus 5
|
0fd89332e2 |
feat(notes): recon notes anchored to an asset
2.x had `recon_note.TodoNote` — a title, a body and an optional scan. 3.0 dropped it and replaced it with nothing, so the operator's own reasoning had nowhere to live: the lead to come back to, the dead end not worth retesting, the reason behind a conclusion. A note anchors to a durable asset identity rather than to a scan row. A note written on api.example.com during scan 2 is still on that host in scan 9, because the anchor is the host name, not the `subdomains.id` scan 2 created. Identity per dimension: host name, endpoint signature, `ip:port`, address, and vulnerability fingerprint. That gives the rollup for free. An asset sheet shows its own notes across every scan; a scan shows notes for assets it observed, plus what was written on it; a target shows everything under it; /notes holds the project. The scan reach is a per-dimension EXISTS over the `TABLES` map `surface_scope` already holds. Every note carries at least one tag, enforced by `min_length=1` on the create model and a disabled Save in the composer. Tags are the project vocabulary from `shared/models/tag.py`, so a tag means the same thing on a note as on a target. `dimension` is a `SurfaceDimension` value, so notes inherit the labels, nouns and ordering the rest of the product uses. No second asset-kind enum. Surfaces: a Notes section in the web asset, endpoint, service and vulnerability sheets; a Notes tab on a scan and on a target; and /notes with tag, dimension, status and free-text filters. The IP sheet is left out — an address group folds across targets, so it has no single rollup anchor. Also gives the two ScrollAreas added with the focused-scan work a definite height; `max-h` alone does not clip, and a short list now renders without a scroller rather than showing an empty track. Row markers on the result tables are not built. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01799buekQvDvH41ns8EdhdH |
||
|
Yogesh OjhaandClaude Opus 5
|
2aac7a8d31 |
feat(scans): focused scans at estate scale
A focused scan was bound to one parent scan, so one target. Selection was a list of row ids, capped at 200 and pruned to the visible page. The project-wide result views could not start one at all. A selection is now a value the server resolves: `SeedSelection` carries either explicit picks or the query that matches them, and resolves into one group per target's covering scan. One action fans out into a focused scan per target, tied by `scans.run_group_id`. `parent_scan_id` + `assets` stays as the older one-scan form, normalized by a validator, so MCP and the row menu are unchanged. Resolution reuses the query grammar rather than a second matcher: a `seeds()` per results service over the existing `_scoped(scope, f, columns=...)` hook, projecting `(value, scan_id)`. Addresses are the exception, since the derived CTE folds them across targets and exposes `target_ids`; the resolver maps those through `SurfaceScopeService.scans_by_target`, as it does for a pick that names no scan. `MAX_SEED_ASSETS = 200` becomes `MAX_RUN_ASSETS = 5000` with `MAX_RUN_SCANS = 25`. One cap, not one per scan: a group is per target, and a lower per-scan cap would need intra-target chunking that `parent_scan_id` and `AssetRecheck` cannot express. `matched` is null once the probe limit is reached, so nothing reports the probe as a total. Frontend: - `RowSelection` replaces the per-table `SvelteSet` and the effect that pruned it to the current page on every page, sort and filter change - the `projectWide` guards are gone, so /surface/* can act on what it shows - the selection bar offers the search as the selection, and states the cap - `POST /scans/rescan/preview` feeds the dialog a per-target breakdown - the Endpoints tab gains a query-only entry point Verified against the dev estate: tech:cloudflare reports 3,484 across 5 targets in the bar, the dialog and the API; title:Census seeded 3 as 2+1 across gov.np and gov.ng under one run group. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01799buekQvDvH41ns8EdhdH |
||
|
Yogesh OjhaandClaude Opus 5
|
2cb8d73280 |
style: strip explanatory comments across the codebase
The comment rule in CLAUDE.md is one line maximum, never narration: no multi-line blocks, no module-header explainers, no sentences that explain or justify the code to the reader. The tree had drifted a long way from it, and most of the drift is mine. Removed: 232 lines of multi-line comment blocks 203 single-line comments that argued instead of stating 154 comments joined by so/because/otherwise/rather-than 143 narrating comments in the frontend 112 test docstrings (the test name already says what it asserts) 25 test module docstrings and 25 decorative section dividers Collapsed: 324 multi-line docstrings to one factual line 271 one-line docstrings that had a justifying clause bolted on Kept: functional directives, section dividers, short field labels, alembic revision headers, and the eight FastAPI route docstrings that render in /docs. Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs. 284 python tests and 34 frontend tests pass, ruff and ruff-format clean across ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
f3b406ec03 |
feat(freshness): re-check a certificate between scans, which is Class B's first job
The derivation rule names four classes and only three had anything implementing them. Class B is "needs a per-asset request, so a background job with its own freshness, never a stage that blocks", and a certificate is the clearest case: it expires and renews on a schedule no scan is aligned to. A dashboard built on a scan from six weeks ago reports every certificate that has auto-renewed since as expiring or expired — 548 hosts on this box sit inside a 30-day window right now, and on a 90-day ACME cycle every one of them would be wrong within a month. shared/services/cert_freshness.py + app.tasks.freshness.certificates (beat, 4h). The policy is the design: a certificate whose stored expiry is more than 30 days away is re-asked after 12h, one already inside its renewal window after 4h, nearest expiry first, 500 a pass, one tlsx handshake per host. Only the newest scan of each target is refreshed — an older scan's rows are a record of that run, not a claim about today. A host that did not answer keeps what the scan found and still gets its stamp moved: silence is not evidence the certificate changed, but an unreachable host must not be re-asked every four hours. It is opt-in, not opt-out (instance_settings.cert_recheck_enabled, default false, a switch in Settings). The threat feeds only download; this one handshakes the operator's targets outside a scan, and nobody should discover that after the fact. Two tool bugs, both the shape CLAUDE.md already lists four of. CLIToolRunner.run defaults to OutputFormat.PLAIN and only appends json_flag for JSONL, so passing json_flag without the format runs the tool with no -json at all — the first live pass answered 0 of 40. And tlsx writes every record it is going to write and then does not exit (40 hosts: 16s to emit, still running at 90s), so the run is bounded by budget(hosts) and Freshness.cut_short reports a pass that hit the ceiling instead of swallowing it. Together: 300.2s and 0 answers became 15.6s and 37 of 40. 19 tests, each proven to fail against 9 mutations of the policy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
4d0c0a2504 |
feat(origin_probe): the origin correlation is written down, not only drawn
origin_exposure computes a weighted, noise-filtered correlation across the CDN boundary — a TLS certificate, a body hash, a favicon or a header set shared between an address answering directly and a hostname behind a CDN — and rendered it as an attention tile and a dialog. Because it was never a row, it inherited none of what the findings model already provides: triage that carries across scans, is:new, delta-only alerting, the report sections, the compliance mapping. This is CLAUDE.md's own outstanding ask, and it was waiting on somewhere to put the provenance. Scanner.RENGINE, added for takeover, is that place. The computation moved to shared/services/origin_exposure.py and became sync, so the stage can run exactly what the API renders; api/app/services/origin_exposure.py is now a run_sync shim over it. One implementation, so the tile and the finding cannot disagree. Verified the API returns the same 20 findings, 256 probed addresses and 512 fronted assets on gov.ba as before the move. The finding says what was established and no more. Confidence maps to severity — two independent identities agreeing is medium, one is low, and a single weak signal was never a finding in the first place — and the description names the address, the hostname it matches and which identities matched: "web.bhansa.gov.ba answers directly and serves the same application as analytics.bhansa.gov.ba, which is fronted by a CDN. Shared: Response body matches; Favicon matches." The default-vhost case reads as itself rather than being folded into the same sentence. Seven tests, including that the fingerprint is stable across scans and specific to the pair, an unknown kind still produces a finding, and a finding with no recorded identity says so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
36b28badd9 |
feat(takeover): a dangling CNAME becomes a finding rather than a dashboard number
Subdomain takeover was a read-time heuristic living inside an API service file: a 33-entry substring list, an `if suffix in host`, and a count on the dashboard. It wrote nothing, so it got none of what the findings model already provides — triage that carries across scans, is:new, delta-only alerting, the reports, the compliance mapping, the query grammar. For a bug bounty tool that is the wrong side of the line for the highest-value thing it can find. TAKEOVER_FINGERPRINTS moves to shared/definitions/domains.py beside VENDOR_DOMAINS where it belongs, and a stages/takeover/ stage writes what it finds through vuln_inventory. Ten real candidates on gov.ng — dangling Azure and CloudFront names on cbn.gov.ng, firs.gov.ng and four others. Scanner gains RENGINE, because a finding reNgine made itself is not nuclei's and not a person's. That also gives cross-scanner corroboration its second scanner: the feature shipped and has been inert with only one. The fingerprint is sha256 of scanner, template, host and cname, so the same dangling name in next week's scan is the same finding — suppress it once and it stays suppressed, exactly as a nuclei finding does. It does not overclaim. The name resolving nowhere is a condition, not a confirmed takeover: the description says "If the resource behind it has been released, whoever registers it next serves content on this hostname", the severity is medium, and the stage sends no packet of its own. Confirming it would mean fetching the provider's claim-me page, which is a separate decision. A test caught the matcher being loose. "s3-website" has no dot, so `in host` matched mys3-website.evil.com — an attacker's hostname reported as an AWS bucket, which is the same trap the interest rules hit when icontains made "dr" match drones.gov.cy. A fingerprint now has to start a DNS label. Verified the anchoring loses nothing real: the same ten candidates. Nineteen tests, including that every one of the 33 fingerprints matches whole, that three plausible impostor hostnames do not, and that the two shapes of AWS website endpoint both do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
fc5788bf3f |
feat(correlation): the certificate and the header set become identities you can pivot on
http_assets has carried tls_fingerprint and header_hash all along. The fingerprint is the exact certificate — 15,191 assets across 4,091 distinct certificates on this data, so nearly four hosts per certificate — and it was used only inside origin exposure and the per-host relations: not a hub in the correlation graph, not a query field, not a group dimension. header_hash was weight-one evidence inside origin exposure and nothing else. Both are now full identities: a CorrelationKind with a label, help text, a hue and an icon; a query field, so cert.fingerprint: and header_hash: compile; and a group dimension, so a hub carries a drill-down token. The certificate draws at rest alongside the other strong identities, since two hosts presenting the same certificate is the strongest statement in the set. Verified on gov.ba: 80 certificate hubs and 25 header-set hubs, and every hub count checked equals the row count its query opens. The graph's asset query listed its columns by hand, so a new kind needed two edits and the second one failed at runtime rather than at import. It derives them from _ASSET_KINDS now. Rejected on the way: a separate serial-plus-issuer key. tls_serial has exactly the same coverage as tls_fingerprint (15,190 against 15,191) and the fingerprint already identifies the certificate, so it would be a second name for the same thing. Twenty-six tests, and the useful ones are structural: every kind must have a label, help, a hub source, a group dimension and a compilable field, must read a column that exists on its model, and must appear in the frontend mirror. The api now mounts frontend/src/lib/config read-only so that last one actually runs rather than skipping — mutation-proven by deleting a kind from the mirror. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
2bb9a61e1f |
fix(correlation): a host relation says how many, and drops what the whole estate shares
Every other correlation in the codebase filters for noise — origin exposure decays a fingerprint by its reach, the graph hides a hub carried by half the scan, rarity signals refuse to fire below a 25-host estate. The per-host relations on the asset sheet had only a cap, and a cap is not a filter. Two things were wrong on a real host. "Same network AS45353" listed 300 hosts because 300 is the cap; the network actually carries 7,163 of the scan's 24,028, and the sheet reported the cap as the truth. And a value carried by the entire estate — one CNAME in front of everything — was offered as a relation at all. SubdomainRelation now carries the real total beside the capped list, so the sheet reads "7,163 hosts sit in the same network" and shows the first 300 of them. A value on half the estate or more is dropped, reusing COMMON_SHARE from the correlation vocabulary rather than inventing a second threshold, and a scan below MIN_ESTATE_FOR_COMMON suppresses nothing because rarity has no meaning without a norm. The certificate relation went through its own hand-written query and now goes through the same gate as the rest. Four tests: a shared address is a relation, the count is the truth and not the length of the list, a value the whole estate shares is not a relation, and a small scan suppresses nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
1e2d16eed2 |
feat(ops): a backup that has been restored, and pools that cannot silently outgrow the server
There was no backup path at all, and the connection arithmetic did not add up. scripts/backup.sh dumps the four things an instance actually is: the database, the scan media on disk, the docker volumes holding uploaded wordlists, nuclei templates, report themes and fonts, and the settings a restore needs. SECRET_KEY travels with it deliberately — API keys, proxy credentials and notification secrets are Fernet encrypted with a key derived from it, so an archive without it restores a database full of unreadable secrets. The header says so, and backups/ and *.tar are gitignored, because an archive is a credential. Restore is the half that is usually never tried, so it was: a 305 MB archive of this instance was restored over it and came back at 9 scans, 59,858 subdomains, 535,000 endpoints, 29 findings and 7 targets — identical — with the migration head intact, the API answering and an encrypted API key still decrypting. It stops the api and both workers first so nothing writes underneath it, and asks for the database name before it destroys anything. The pools could ask for more connections than the server would give. The api takes 10 + 20, each celery prefork child takes 2 + 3, and beat takes another pool: at the concurrency of 8 this box runs that is 75 against a max_connections of 100, and at the 12 the compose file now defaults to it is 95 — five short of the ceiling, which fails as a checkout timeout under load and reads like a slow query. max_connections is now set from compose with the arithmetic beside it, and the api checks it at startup: pool_demand() against max_connections less the superuser reserve, warned once rather than discovered during an incident. The inputs moved to shared settings because the sum spans both processes and neither could see the other's numbers. Worth recording: .env on this box pins CELERY_SCAN_CONCURRENCY=8, so raising the compose default to 12 changes new installs and not this one. It also still carries CELERY_DEFAULT_CONCURRENCY, which nothing has read since it was renamed. Three tests: the demand counts every pool, the shipped defaults fit the shipped server, and an oversized pool warns rather than failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
0596a7b233 |
perf(asset-query): the lead set is computed once per burst, not once per tab switch
Every result table asks for its lead counts on mount, on every facet change and on every tab switch, and the answer is the same each time. After the earlier fixes the host set costs 832 ms, endpoints 1,609 ms and the project-wide host set 1,653 ms. Served from redis it is 4-6 ms, on all five dimensions. Only a settled scope is cached. A live scan's counts change under the reader and a count has to equal the rows its link opens, so a scope holding any scan that has not reached a terminal status is always computed fresh. The entry lives 60 seconds, and the reason is worth stating rather than assuming a finished scan is immutable. Its rows are — but the counts drawn from them are not: a reviewer suppressing a finding moves every vulnerability lead, the nightly EPSS and KEV load moves is:kev, and an interest pass moves is:exposed. Plumbing an invalidation into each of those and hoping none is ever missed trades a silent wrong number for a slow right one; a short life serves the burst that this fix is about — tab switches, facet changes, a second table on the same page — and bounds any count at a minute stale. The key is the dimension, the sorted scan ids and the facets, and facets_of drops what cannot change a count: paging, sorting, and the search box, which the lead set deliberately ignores. A lead set that did not compute is never stored, so a timeout is not served for the next minute. The statement is chunked at 16 branches. A fifty-branch union plans as one enormous tree, and one that outruns the timeout takes every count in it down. Verified against the live database: 33 host, 14 address and 18 service leads all still equal the row count their query opens, served from cache; and two different facet sets give two different answers rather than sharing an entry. Seven tests cover the rules — computed once for a finished scan, never cached for a running one, cached for a failed or cancelled one, distinct per filter, never cached when it did not compute, and facets_of ignoring paging while keeping facets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
69d75113de |
perf(endpoints): the host rollup reads the table too, and hide_static still narrows it
POST /endpoints/tree/hosts joined a scan-sized id subquery seven times over — the per-host aggregate, the totals, the parked-host count, the interest and source explodes, the class mix and the folder chips. It is the same cost the facets fix measured, seven times: 529 ms -> 310 ms, below the 484 ms it started at before the Postgres tuning moved its plan. _Reach now carries the decision as a FROM clause and a WHERE rather than a join, so a call site swaps two chained lines and nothing else changes shape. Every aggregate in the hosts path goes through it. The first cut of this was wrong in a way worth recording. It asked _narrowed(f), whose docstring says "anything but the static switch" — it exists to decide whether to compute unfiltered_count, not whether the base query differs from the scan. _apply_filter does apply hide_static, so with only that switch set the reach took the direct path and the static rows came back: total_endpoints 8 instead of 6 on the fixture, and 345,152 instead of 245,153 on the real scan. A differential run against the previous implementation across five filter shapes caught it; has_facets() is the predicate that actually mirrors _apply_filter. Verified identical output on all five shapes, and the mutation is now a test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
6e49f13818 |
perf(overview): the dashboard stops re-sorting the whole project to answer "what is new"
GET /dashboard/overview was 842 ms and half of it was one statement: 493 ms of row_number() OVER (PARTITION BY target_id, signature ORDER BY discovered_at) across all 534,765 endpoints in the project, run on every load to answer which scan first saw each thing. The same shape runs for all five dimensions. DISTINCT ON asks the same question and can ride ix_endpoints_target_signature, so Postgres sorts incrementally within each group instead of sorting the project (measured 511 ms -> 360 ms on the raw statement, 842 ms -> 683 ms end to end, byte-identical output). Rejected on the way: a covering index on (target_id, signature, discovered_at, scan_id) removes the sort entirely but costs 72 MB for another 1.4x, and raising work_mem does nothing because the plan already sorts incrementally rather than spilling. This is a floor, not a fix. The work is O(every row the project has ever recorded) on every dashboard load, and it grows without bound. The honest answer is a first-sighting flag written with the row — Class A in the project's own derivation rule — or a cached per-scan count, since a finished scan's answer never changes. Both are their own piece of work. GET /surface/overview re-fetched the project's targets inside each of the five coverage() calls and re-resolved the same scan picks for the exposures count. The service is constructed per request, so the answers cannot go stale within one: they are memoised on the instance. 213 ms -> 188 ms. Four tests on what the number means rather than how it is computed: a first scan contributes nothing new, only the scan that first saw a host is credited, an unchanged host is never counted twice across three scans, and newness is judged per target so a fresh target contributes nothing. Mutation-tested — crediting the last sighting instead of the first fails three of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
3260a09166 |
perf(endpoints): a facet counts the table, not a scan-sized list of its ids
GET /endpoints/facets took 1,271 ms on a 345,152-endpoint scan and refires on every five-second live tick. The obvious reading was the twelve serial round trips; measuring says otherwise. Each facet joined against `select(Endpoint.id)` over the whole scan, and that join is the cost: one facet is 158 ms through the id subquery and 52 ms filtering the table directly. Three times, six times over. _Reach carries that decision once. With nothing narrowing the rows — the state the page opens in — a facet filters on the scan directly; the moment a facet filter or a query narrows them it goes back through the subquery, because then the id list is doing real work. 1,271 ms -> 574 ms, byte-identical output. The round trips are collapsed too, from twelve to three: one row carrying both counts, one UNION ALL over the three column facets and the status buckets, one over the two array facets. Each branch keeps its own index, its own cap and its own ORDER BY, which is what makes the cap deterministic — dropping the secondary sort silently changed which thirty extensions came back. That was worth about nothing on its own and is kept for the shape, not the speed. Two regressions this introduced and the tests now hold shut: - status_class is an ordinal ramp and keeps its declared order. Sorting facets by count put 'none' (331,054 rows) ahead of 2xx and reshuffled the tabs. - a status outside every bucket is not "Not checked"; the CASE returns NULL and the row is dropped, as the old loop did. endpoints/summary's ten serial queries are collapsed into four the same way, and that one is honestly not faster: 661 ms before, 661 ms after. It is three aggregate passes over 345k rows — 343 ms for the counts, 225 ms for the source explode, 53 ms for the class breakdown — and merging them measured worse, since count(DISTINCT host) per group costs more than a second scan. Using count(*) instead of count(DISTINCT id) on the source explode would buy 120 ms on the strength of an invariant nothing enforces, so it is not taken. The real fix is denormalised counters written with the row, which is Class A in the project's own derivation rule and a separate piece of work. Six tests: the facet counts add up to the total, the two reach paths agree exactly, a filter actually narrows, static_total counts within the reach, the summary agrees with the facets, and status keeps its order. Mutation-tested — sorting status by count fails, and forcing the subquery path passes, which is the point. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |
||
|
Yogesh OjhaandClaude Opus 5
|
4b6ab5ecfd |
test: the invariants the product promises are now enforced by a suite
reNgine had no tests. All 102 test_*.py in the tree were inside .venv, and CI
built images and ran CodeQL without ever running a line of the code. Nothing
would have caught the is:new rewrite in
|
||
|
Yogesh OjhaandClaude Opus 5
|
a8b9304d10 |
perf(asset-query): the read path stops repeating work it already did
Measured against the seven scans on the dev box, worst case the gov.np run (24,028 hosts, 345,152 endpoints). Four separate causes, none of them touching how a scan writes. A scan-level fact was resolved per row: - has_baseline for hosts was the last of the four baseline predicates still written against the outer row. service_has_baseline, vuln_has_baseline and endpoint_has_baseline all resolve theirs once through _per_scan; hosts never took the fix. Postgres reported 192,448,261 rows removed by the join filter, and is:new took 15,202 ms. It is 26 ms now. - That one predicate was also what pushed the project-wide leads call past its 20s statement_timeout, so every findings count on the Attack surface pages came back computed:false — absent rather than wrong, with nothing saying so. An index that could never be reached: - port_match joined ports to subdomains with jsonb_exists(resolved_ips, ip), taking the probe value from the inner table, so ix_subdomains_ips_gin was unusable and is:sensitive, port: and service: degraded to a nested loop over 21k ports x 24k hosts. Resolving the address set once and probing with ?| returns the identical rows in 50 ms rather than 1,248 ms. Both now take the scope every other predicate takes, so a project view evaluates them against each row's own scan instead of correlating on target_id alone. A derived query rebuilt per reference: - the address and service subqueries are six- and four-CTE statements that scan subdomains, ports and http_assets. Attached with .subquery() they were inlined at every reference: twice in search, three times in facets and eighteen times in leads. As .cte() Postgres evaluates them once. ips/search/leads 8,360 ms -> 79 ms, ports/search/leads 1,683 ms -> 82 ms, ips/facets 2,379 ms -> 141 ms. A plan compiled instead of run: - NO_JIT already existed and was applied on two of the five dimensions. A leads statement is fifty count branches unioned together, which clears jit_above_cost easily, so Postgres spent seconds compiling a plan that runs in milliseconds. All twelve guarded query paths set it now; verified to carry the whole win on its own with the server default back at jit=on. Verified: is:new is the one predicate that only wakes up on a second scan of a target, and every target here had exactly one — so the counts-unchanged check was comparing zero to zero. Against a seeded baseline scan the old shape, the new shape and the ground truth all report 125 new hosts, and so do the compiler, the facet filter, the project-wide scope and the lead counts. Every non-zero lead in the host, address and service grammars still equals the row count its link opens. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C |