Uploaded typefaces and instance-wide branding defaults, which together make a
report genuinely white-labellable: a client's own font, a logo set once, and a
theme that names both.
The security half came first, because the logo field was a live SSRF. It was a
free-text string rendered into an <img src>, and WeasyPrint would have fetched
whatever it pointed at from inside the worker. Nothing fetches now, at four
layers. The load-bearing one is a WeasyPrint url_fetcher that allows data: URIs
and files under three known roots and refuses everything else; it was checked
against cloud metadata, internal https, loopback, /etc/passwd, a UNC host, ftp
and protocol-relative references. Above it sit three input guards: a logo must
be embedded data of a known image type, an uploaded theme may not use @import or
a url() that is not data:, and authored markdown loses any image that is not
embedded. A new feature that lets a report name a resource goes through the
fetcher, not around it.
A typeface is bytes, never a link. The family row records files named by us, so
a hostile filename never reaches disk, and the family directory refuses anything
resolving outside itself. A face is accepted only when the bytes agree: the
signature is sniffed and then fontTools opens it and requires outlines, which is
why fonttools is now an api dependency rather than only a WeasyPrint one. A
5-byte file with a valid TTF header and a 4 KB file of padding are both refused.
The family name is banned from carrying quotes or braces because it is written
into a quoted CSS font-family string.
The standalone HTML export now embeds only the families the theme actually
names, which halves it, and no longer carries file:// references that would have
been dead outside the container.
Instance defaults fill only what a template leaves blank, so setting a logo once
never overrides a per-client deliverable that set its own.
Verified: a real TTF family uploaded and embedded in a 53-page PDF under an
uploaded theme with an uploaded logo, and every probe above refused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f2Mf2HGwFaTRnui4UxpRd
reNgine 2 could produce a PDF. This produces a deliverable, and it is built
the way scan engines are built: sections are plug-and-play directories under
reports/, each with a pydantic config model that IS the source of truth. That
model feeds GET /reports/catalog, the builder renders its own form from it, and
the API validates against it. Adding a section is a directory. Scan Engine is to
stages what Report Template is to sections.
WeasyPrint 69 renders it, chosen over Chromium and Typst after measuring what
each can actually do. Only real CSS Paged Media gives a table of contents with
true page numbers, running headers, a PDF outline and footnotes; SysReptor
reaches the same verdict and pipes Chromium's HTML into WeasyPrint for exactly
that reason. reNgine drops the Chromium hop because it templates in Jinja and
draws its charts as SVG in Python, so a browser buys nothing and costs 400 MB
per report. Typst was rejected because a user-uploadable theme would then have
to be Typst code rather than CSS.
A theme is data. shared/definitions/report_theme.py is the schema, six themes
ship as YAML, and a user uploads their own; there is no markup in a theme, so
there is no injection surface. Fonts are vendored so an offline instance still
renders in Latin, Greek and Cyrillic.
Everything the document claims is computed once, in reports/analysis/engine.py:
a posture score whose every deduction is named, weaknesses ranked by the signals
behind the rank rather than severity alone, attack paths that chain observations
into a route, a remediation plan ordered by risk removed, per-dimension change
against the previous run, and caveats that state what was never assessed. Two
narrators read that one brief. Without AI the report still argues its case; with
AI the prose is better and the model never sees a row, only the brief, capped and
cached per check. An AI failure falls back and the report still completes.
Generation is a job. POST /reports queues, the worker walks progress over SSE,
and four formats come out of one render: PDF, self-contained HTML, Markdown and
JSON. The estimate reads each section's own limits, so the page count it promises
is close to the one it delivers.
AI moves out of Settings into its own top-level page: connection, a model per
tier, what it may be used for, what it has cost, and the narrative cache.
Verified on a 1,379-host gov.cy scan: 124 pages in 19.5s across 19 sections,
all six themes, and a broken key still producing a complete report.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f2Mf2HGwFaTRnui4UxpRd
The wordlist field was a free-text absolute path, so the only lists anyone could
actually use were the two baked into the image. Changing that meant a bind mount or a
docker cp — a sysadmin action, not a product one — which in practice meant every
install guessed from the same list. Bruteforce quality is almost entirely wordlist
quality, so the knob was there and nobody could reach it.
Wordlists are now a library, on the same shape as the nuclei check library. Shipped
lists stay in the image and are indexed on read so the picker never starts empty;
uploads land under a `wordlists` volume shared by api and worker. A config field holds
a slug, never a path — `stages/config.py:wordlist(kind)` declares one the way
`threads()` and `rate()` declare theirs, and carries the kind the picker filters on so
a DNS stage is never offered a list of URL paths. `lookup()` still accepts the
absolute paths older engines stored, so an existing engine keeps working.
The UI is the Arsenal's second tab, which is why the Arsenal is now "what your scans
run with" rather than only scanners and their checks: upload, filter by what the words
are, preview the first 200 in the order a scan will read them, delete an upload. Every
list shows the name an engine refers to it by, because that is the thing you type into
a YAML document.
`resolve_path()` refuses anything resolving outside its origin's root and
`delete_custom()` refuses a shipped list — the guard the template library learned the
hard way. A stage that cannot find its list says so and reports PARTIAL rather than
quietly guessing from a different one; that is how the shared volume being absent was
caught during the end-to-end run.
Verified end to end: a seven-word list uploaded through the API, selected by slug,
every passive source disabled — the scan returned six hosts, all attributed to
bruteforce.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
Measured end to end against three targets on the default plan.
Execution order is now declared, not hand-numbered. `Stage.depends_on` names the
stages one may not start before and the registry derives `level` from it by
longest-path layering, rejecting an unknown name or a cycle. Eleven hand-assigned
barriers collapse to seven real ones: target enrichment no longer gates subdomain
discovery, and the whole post-probe fan-out (origin probe, service fingerprint,
WAF detection, URL discovery, vulnerability scan) is one parallel level instead of
five. The frontend plan solver groups by the same derived level.
Four tool invocations were wrong in ways that cost whole scans:
- naabu reads `-timeout` as a go duration, so a bare `-timeout 3000` pinned every
invocation at a fixed ~135s whatever the host and port count. `-timeout 3s`
returns the identical open-port set: 138s -> 5s on three addresses, and on a
1,610-address estate 7,428s (two 3,600s timeouts, stage failed) -> 240s.
- urlfinder has no `-t`; passing one made it exit 2, so the archive URL source had
never returned a row.
- amass v4 prints a relationship graph, not a host list, so its output was parsed
as hostnames and discarded — zero rows in every scan on record while costing the
full 360s tool budget. Parsed properly now, and out of the default source set.
- httpx's stream probe was capped at a 900s total, silently truncating a large HTTP
probe and reporting it as success. It is idle-bounded now, the way dnsx already
was, and reports PARTIAL when it is stopped.
Celery's redis visibility timeout (3600s default) sat below the 8h task limit, so any
stage running over an hour was delivered twice and the whole rest of the scan ran
again — one run duplicated its entire post-port-scan pipeline, another looped for ten
hours. It now sits above the hard limit.
Work inside stages that was serial and did not need to be: resolver batches
(dns_batch_concurrency, default 3), URL discovery providers, robots/sitemap fetches,
and wafw00f (which has no threads of its own) now run concurrently, with DB writes
kept on the stage thread. wafw00f, screenshots and the WAF stage report what they did
not cover instead of reporting a short run as complete. The cancellation flag is
polled at most every two seconds instead of once per resolved name.
Also: WAF detection and screenshots no longer load whole http_asset rows (stored
response bodies included) to write one column; tool output decodes with
errors="replace" so one non-UTF-8 byte cannot discard a run; the worker runs at
CELERY_SCAN_CONCURRENCY (8) because the widest level now holds five stages.
Measured: vulnweb.com 656s -> 218s with identical results. hamropatro.com completes
in 2,273s where it previously never finished. gov.np's port scan 7,428s -> 240s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
- Implemented TargetEnrichmentStage for enriching targets with DNS, WHOIS, and BGP context.
- Added VhostStage for virtual host discovery using host-header fuzzing.
- Introduced WafDetectStage for fingerprinting web application firewalls.
- Created configuration files for each new stage with relevant parameters.
- Updated Dockerfile to copy stages instead of engines.
- Enhanced existing tools and services for better integration with new stages.
- Refactored task management in Celery to accommodate new stages and improve error handling.
Rebuild the scan detail view into a tabbed results page: Overview
(tech/status/web-server distributions, TLS health, top networks and
services), Web assets (rich httpx table with screenshots, status,
tech, CDN/WAF/TLS-expired badges + a full per-asset detail drawer),
Subdomains (denormalized HTTP summary), Infrastructure (IP-centric
cards correlating open ports and the hosts resolving to each IP), and
Pipeline. Assets correlate by value within the scan (subdomain.name /
http_asset.host / ip), so one IP card surfaces every co-located host.
API: add per-scan http-assets, ports and ips list+summary endpoints
plus services mirroring the subdomains pattern, and serve screenshots
through an authenticated /media/screenshot route; mount scan_media
into the api container so it can read worker-written captures.
Fix two serializers that silently dropped fields: ScanRead omitted
http_assets_found (scan detail and list 500'd), and the subdomain
serializer never emitted the denormalized HTTP summary it stores.
Drop leftover docstrings and banner comments from the httpx client,
parser and field-map.
Rounds out expansion (host discovery, virtual hosts, screenshots, WAF):
- host_discovery: naabu liveness sweep marks IpAddress.is_alive for
ip_range/asn; port_scan then deep-scans only live hosts
- vhost: ffuf Host-header bruteforce (domain, opt-in) -> vhost-only
Subdomains that http_probe then picks up
- screenshot: httpx headless chromium -> HttpAsset.screenshot_path under
a scan_media volume
- waf_detect: wafw00f fingerprints WAFs -> HttpAsset.waf
All config-driven (ExpansionConfig) with proxy / user-agent / headers +
per-tool custom args threaded through. Adds ffuf, chromium and wafw00f to
the worker image and a scan_media volume.
- Added new types and interfaces for scan scheduling in the frontend.
- Created a new Svelte page for managing scheduled scans, including listing, creating, editing, and deleting schedules.
- Implemented backend models and services for scan schedules, including handling different schedule types (one-off, interval, daily, cron).
- Introduced a Celery task to manage the execution of scheduled scans based on their defined timing.
- Enhanced notification system to include alerts for newly discovered subdomains during scans.
- Added utility functions for handling time zones and scheduling logic.
- Updated the default route for automation to redirect to engines instead of scans.
- Removed the scans page and replaced it with a new scans overview page that lists scan history for projects.
- Introduced a detailed scan view page to display individual scan results and subdomain discoveries.
- Added new enums for scan statuses and subdomain sources to improve code clarity and maintainability.
- Implemented a new scan task in Celery to handle scan execution and logging.
- Enhanced the database models for scans and subdomains to support new features and improve data integrity.
- Updated the worker Dockerfile to include necessary scanning tools and dependencies.
- Added utility functions for domain matching and pattern compilation to streamline scan operations.
- Updated project slug generation to ensure uniqueness by checking active projects.
- Modified project retrieval functions to only fetch active projects.
- Introduced unique slug generation for tags associated with specific projects.
- Enhanced user authentication flow with improved error handling and session management.
- Added session expiration handling in the API client and SSE client for better user experience.
- Improved UI components for better accessibility and user feedback, including copy buttons and activity drawers.
- Refactored notification handling to ensure proper session management and state reset on logout.
- Removed unused enrichment components to streamline the codebase.
- Updated notification model to simplify JSONB handling and ensure proper expiration logic.
- Added migration commands to the new dev-README.md for creating and rolling back migrations.
- Updated docker-compose.yml to include additional volume mounts for alembic and build artifacts in development and worker services.
- Enhanced Target model in target.py to include a relationship with WhoisRecord and added WhoisRecordSummary to TargetRead for better data representation.
- Introduced a new endpoint to validate multiple target values in a single request.
- Added a constant for maximum target imports.
- Updated the target validation response model to include the target value.
- Modified Dockerfile CMD for graceful shutdown and reload directory.
- Commented out environment variable in docker-compose for polling.
- Update Dockerfile to copy shared directory and dependency files
- Create new README.md file
- Refactor database session management and security imports
- Implement user authentication routes and user management endpoints
- Add configuration settings for application and database
- Set up logging configuration for backend services
- Update docker-compose for improved service management
* Add django debug toolbar
* Remove duplicate debug toolbar URLs
* Remove orphaned import in startScan
* Add DEBUG config commented in entrypoint
* Install debug toolbar into celery beat
* Create dev docker compose & entrypoints
* Move django toolbar install to dev entrypoints
* Install debug_toolbar only when debug env var is set
* Add dev command to MakeFile
* Use DEBUG from settings.py exclusively
* Add remote debug configuration in dev env
* Add VSCode debug configuration
* Celery log level to 1 when debug is on
* Remove hard coded celery debug port
* Add gunicorn in the production setup
* Add export var for celery log level
* Restrict debug port to 127.0.0.1
* Set remote debug to false by default
* Update doc on debug step for submitting issue
* Create dev docker compose & entrypoints
* Add django debug toolbar
* Remove duplicate debug toolbar URLs
* Remove orphaned import in startScan
* Add DEBUG config commented in entrypoint
* Install debug toolbar into celery beat
* Move django toolbar install to dev entrypoints
* Install debug_toolbar only when debug env var is set
* Add dev command to MakeFile
* Use DEBUG from settings.py exclusively
* Add remote debug configuration in dev env
* Add VSCode debug configuration
* Celery log level to 1 when debug is on
* Remove hard coded celery debug port
* Add gunicorn in the production setup
* Add export var for celery log level
* Restrict debug port to 127.0.0.1
* Set remote debug to false by default
* Update doc on debug step for submitting issue
---------
Co-authored-by: Raynald <raynald@easi-services.fr>
Co-authored-by: Psyray <psyray@users.noreply.github.com>
