mirror of
https://github.com/yogeshojha/rengine.git
synced 2026-10-20 07:11:28 +02:00
Yogesh OjhaandClaude Opus 5
cb1412ce90
reNgine 2 answered this with one global keyword list matched by `icontains`, and rebuilt that query once per table row — `get_is_interesting` was a SerializerMethodField, so a 5,000-host page ran the whole unindexable OR-chain 5,000 times. It was also dead until the user saved the settings form once, since `get_interesting_subdomains` returns none() with no custom_type row. The rule survives; everything around it is rebuilt. An interest rule IS a saved HOST_QUERY query, so there is no second matcher to keep in sync, and the keyword box compiles to `host:[…] or title:[…]` behind the same grammar. Signals persist to interest_signals, roll up in SQL onto subdomains.interest_score/band/kinds so sorting and `is:interesting` stay index-backed, and the scan carries a signature of the rules it was labelled against — a stale read refreshes in the background, so editing a keyword still re-labels every past scan without a read-time query. interest/ is a top-level package mounted into api and worker like stages/ and reports/, with providers auto-discovered from disk. Three ship: rules the keyword lists and 19 saved queries correlation what stands out from the estate, which no keyword can see ai a judgement, labelled as one everywhere it appears Correlation is the half a rule cannot express: an ASN carrying under 5% of the surface, software or a favicon on three hosts or fewer, the one asset answering directly while the rest sits behind a CDN. All gated on a minimum estate size, because rarity means nothing without a norm. On gov.cy: 226 flagged of 1,379 in 0.4s, 29 of them critical. AI sees hostname, status, page title and technology for hosts that answered, and nothing else — never a response body — deduped to three samples per identical response shape. It may answer only in a closed schema, a host it was not given is dropped rather than stored, and its sentence must state what was observed, never what is exploitable. Page titles are attacker-controlled, so they travel as JSON data with strip_control and a cap, never interpolated into the instruction, and nothing it returns triggers an action. Every AI signal stores its model and prompt version, and the UI never lets it read as an observation. Then the judgement graduates: suggest.py shows the model what it flagged and asks for the query that would catch it again without a model. Each proposal is compiled, counted against the scan and dropped if it errors, matches nothing or already exists; nothing is stored until a person presses Add, after which it runs on every keyless install with no cost. Alerts are delta-only — hosts the target has never flagged before, and only those in a high band or matched by a rule the user asked to be notified about. Four bugs found on the way. `dr` matched drones.gov.cy, so shipped hostname rules are anchored on label boundaries and false positives went from 6 hosts to 0. predicates.interest_signal aliased InterestSignal while its condition used the unaliased class, uncorrelating the EXISTS so every interest: field returned the unfiltered total. The correlation aggregates needed explicit output aliases or a bare `t` collided with the CTE and printed a UUID inside a sentence. And dismiss removed the signals while leaving the denormalised rank, so the row stayed in the list. The asset query compiler moves to shared/ because the worker has no api/ and has to compile a rule at finalize; api keeps a shim that aliases every submodule into sys.modules, so no existing import changes. The rule library lives at /interest, not in the Arsenal. The Arsenal is tools. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVh1CGtjDA33fupXjLGbk2
34 lines
1.1 KiB
Python
34 lines
1.1 KiB
Python
from __future__ import annotations
|
|
|
|
from sqlalchemy.exc import DBAPIError
|
|
|
|
from shared.models.asset_query import QueryError
|
|
|
|
STATEMENT_TIMEOUT = "SET LOCAL statement_timeout = '20s'"
|
|
# a set-returning expansion is estimated at 100 rows a row, and the bogus cost buys 270ms of JIT
|
|
NO_JIT = "SET LOCAL jit = off"
|
|
|
|
QUERY_SQLSTATES = {
|
|
"2201B": (
|
|
"That regular expression is not valid here.",
|
|
"PostgreSQL regular expressions differ slightly from PCRE.",
|
|
),
|
|
"2201G": ("That regular expression is not valid here.", None),
|
|
"57014": (
|
|
"That search took too long to run.",
|
|
"Add a field filter to narrow it down.",
|
|
),
|
|
"22003": ("A number in that query is out of range.", None),
|
|
"22P02": ("A value in that query has the wrong shape.", None),
|
|
"54000": ("That search term is too large to index.", None),
|
|
}
|
|
|
|
|
|
def query_error_for(exc: DBAPIError) -> QueryError | None:
|
|
sqlstate = getattr(exc.orig, "sqlstate", None) or getattr(exc.orig, "pgcode", None)
|
|
known = QUERY_SQLSTATES.get(str(sqlstate))
|
|
if known is None:
|
|
return None
|
|
message, hint = known
|
|
return QueryError(message=message, hint=hint)
|