`normalize_path` caps a path at MAX_PATH_LENGTH, but `split_path` takes the last
segment as the filename and that column is varchar(300) — a 1,400-character segment
with no slash in it failed the insert, poisoned the transaction and cost the whole
stage. gov.np hit it once the HTTP probe stopped truncating itself and handed URL
discovery 15,098 web assets instead of 2,687.
MAX_HOST_LENGTH and MAX_FILENAME_LENGTH join the caps already in
shared/definitions/endpoints.py, `split_path` applies the filename cap and
`parse_url` rejects an over-long host rather than storing a truncated one, and the
model's max_length values now read from those constants instead of repeating the
numbers. The batch insert runs in a savepoint and retries row by row, so a row
postgres refuses costs that row and lands in UpsertResult.rejected — the same rule
the HTTP probe now follows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
Replace the tree rail beside a flat list with one outline: host rows, folder rows and
endpoint leaves in a single grid. Kind is a glyph on the leaf, folders get a glyph for
why they matter, a folder that is only its own index renders as a leaf, and the query
bar is the tree search with matched segments marked.
Defaults are measured: hosts come from a paginated SQL rollup and load their folders
when opened, nothing is open at rest, and a search opens branches only while they fit a
40-row screen budget. Hide static is a real server filter with the count carried by the
facets so the All tab equals the rows.
The tree now carries change and judgement: new and gone counts per node against the
previous scan of the target with a Gone lens, anomaly sentences for folders that are
mostly walled with one open door or mostly fine with a few 5xx, siblings that share a
skeleton folded into one group row whose dir:[...] token lands on exactly its count,
archive-only and robots.txt markers, copy paths as a wordlist, reveal by host= path=
tokens, sticky ancestor crumbs rendered in the sticky head, type-ahead, and a verify-this-
branch action wired to the worker task.
Also: class-aware path interests (a cPanel logo is not an administrative interface),
the tilde backup rule limited to trailing tildes, endpoint_count and a paths: field on
hosts, and a null-safe static predicate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015KEfo6ipoWMYeHxThmTLRy
- Updated highlight-text component to support multiple search terms.
- Modified results-pagination component to make onPageSize optional and conditionally render PageSizeSelector.
- Integrated HostStructure component into web-asset-detail-sheet for improved asset structure visualization.
- Added HostStructureDialog to manage host structure interactions.
- Enhanced web-assets-table to handle structure-related actions and dialogs.
- Introduced new folder glyphs and updated endpoint classification logic in endpoints configuration.
- Added endpoint_count to subdomain type for better tracking of discovered endpoints.
- Implemented new endpoint filtering capabilities in the backend to support static content visibility.
- Updated models and services to accommodate new endpoint structures and classifications.
