mirror of
https://github.com/yogeshojha/rengine.git
synced 2026-10-21 07:41:28 +02:00
Yogesh OjhaandClaude Opus 4.8
dd7ff0f6f1
The vulnerability stage handed every http_assets row to nuclei, ordered by URL, with one 90-minute clock cutting wherever it landed. Template-spray then gave every host the first slice of the corpus in load order. On the two largest local scans that was 23.4M and 10.7M planned requests, of which 3.9% and 9% were sent, and no host recorded what it got. One GeoServer CVE was found 11 times on 11 hostnames of one server. Introduce one planner, shared/services/scan_surface/, that decides what every vulnerability scanner receives. It writes scan_surface_items per scan and is a pure function of the tables the scan already produced plus the context; nothing in it names a nuclei flag. Vocabulary lives in shared/definitions/scan_surface.py, mirrored by config/scan-surface.ts. - Inputs are typed by protocol, because nuclei rewrites input per protocol: roots for HTTP templates (any status), bare names for DNS, host:port for TLS and network, parameterised requests for fuzzing. Path and query are stripped from non-DAST input; values are normalised to one spelling. - Web assets that are one backend fold into one origin on exact agreement of address, certificate, body hash, status, title, canary and favicon — never a similarity score. Measured: 1,812 web assets to 1,607 origins on gov.np. - The corpus is spent in tiers: one-request checks on every asset, the universal set on each origin, product-matched checks through a curated tech map, the blind sweep last. Each tier is a ranked host-spray batch with its own coverage row; the budget cut lands between batches and every origin is scanned, covered or not scanned. - A finding on a representative is replayed onto each cluster member with one request, writing a real observed row or splitting the member out. Never an inferred row. - dast_scan is a new stage (off by default, after endpoint_probe): nuclei -dast over request items and exposure checks under directories, then dalfox behind a reflection probe. dalfox v3.2.2 is pinned in the worker image. Verified live on testasp.vulnweb.com: the tiers ran in order, nuclei -dast found two LFIs, dalfox executed under the reflection gate. All suites green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R7sBu1jBa4e2CXLS1KEsNr
108 lines
4.1 KiB
Docker
108 lines
4.1 KiB
Docker
FROM python:3.13-slim AS base
|
|
|
|
ENV PYTHONDONTWRITEBYTECODE=1 \
|
|
PYTHONUNBUFFERED=1 \
|
|
PYTHONPATH=/app \
|
|
C_FORCE_ROOT=1
|
|
|
|
WORKDIR /app
|
|
|
|
COPY --from=ghcr.io/astral-sh/uv:0.12.13 /uv /usr/local/bin/uv
|
|
|
|
|
|
##################################################################
|
|
# Tool build: only /out/bin reaches the runtime image
|
|
##################################################################
|
|
|
|
FROM base AS gotools
|
|
|
|
# libpcap-dev builds naabu
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
curl \
|
|
git \
|
|
build-essential \
|
|
libpcap-dev \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
RUN curl -fsSL https://go.dev/dl/go1.25.7.linux-amd64.tar.gz | tar -C /usr/local -xzf -
|
|
|
|
ENV PATH="/usr/local/go/bin:${PATH}" \
|
|
GOBIN=/out/bin \
|
|
GOCACHE=/cache/build \
|
|
GOMODCACHE=/cache/mod \
|
|
GOFLAGS=-ldflags=-s
|
|
|
|
RUN --mount=type=cache,target=/cache/build --mount=type=cache,target=/cache/mod \
|
|
go install -v github.com/projectdiscovery/dnsx/cmd/dnsx@v1.3.1 \
|
|
&& go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@v2.16.0 \
|
|
&& go install -v github.com/projectdiscovery/tlsx/cmd/tlsx@v1.4.0 \
|
|
&& go install -v github.com/projectdiscovery/alterx/cmd/alterx@v0.1.0 \
|
|
&& go install -v github.com/tomnomnom/assetfinder@v0.1.1 \
|
|
&& go install -v github.com/gwen001/github-subdomains@v1.2.2 \
|
|
&& go install -v github.com/owasp-amass/amass/v4/...@v4.2.0
|
|
|
|
RUN --mount=type=cache,target=/cache/build --mount=type=cache,target=/cache/mod \
|
|
go install -v github.com/projectdiscovery/naabu/v2/cmd/naabu@v2.6.1 \
|
|
&& go install -v github.com/projectdiscovery/httpx/cmd/httpx@v1.12.0 \
|
|
&& go install -v github.com/projectdiscovery/cdncheck/cmd/cdncheck@v1.3.0
|
|
|
|
RUN --mount=type=cache,target=/cache/build --mount=type=cache,target=/cache/mod \
|
|
go install -v github.com/ffuf/ffuf/v2@v2.3.0 \
|
|
&& go install -v github.com/projectdiscovery/katana/cmd/katana@v1.7.0 \
|
|
&& go install -v github.com/projectdiscovery/urlfinder/cmd/urlfinder@v0.0.3
|
|
|
|
# nuclei, with templates synced at runtime onto the vuln-templates volume
|
|
RUN --mount=type=cache,target=/cache/build --mount=type=cache,target=/cache/mod \
|
|
go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@v3.11.1
|
|
|
|
# certificate transparency stream for program watches
|
|
RUN --mount=type=cache,target=/cache/build --mount=type=cache,target=/cache/mod \
|
|
go install -v software.sslmate.com/src/certspotter/cmd/certspotter@v0.24.2
|
|
|
|
# dalfox v3 is a Rust binary, shipped as a release tarball rather than go install
|
|
ARG DALFOX_VERSION=v3.2.2
|
|
RUN set -eux; \
|
|
arch="$(dpkg --print-architecture)"; \
|
|
case "$arch" in \
|
|
amd64) slug="linux-x86_64" ;; \
|
|
arm64) slug="linux-aarch64" ;; \
|
|
*) echo "unsupported arch $arch" >&2; exit 1 ;; \
|
|
esac; \
|
|
curl -fsSL -o /tmp/dalfox.tar.gz "https://github.com/hahwul/dalfox/releases/download/${DALFOX_VERSION}/dalfox-${DALFOX_VERSION}-${slug}.tar.gz"; \
|
|
tar -xzf /tmp/dalfox.tar.gz -C /tmp; \
|
|
mv "/tmp/dalfox-${DALFOX_VERSION}-${slug}/dalfox" /out/bin/dalfox; \
|
|
rm -rf /tmp/dalfox.tar.gz "/tmp/dalfox-${DALFOX_VERSION}-${slug}"; \
|
|
chmod +x /out/bin/dalfox; \
|
|
/out/bin/dalfox --version
|
|
|
|
|
|
##################################################################
|
|
# Development
|
|
##################################################################
|
|
|
|
FROM base AS development
|
|
|
|
# chromium screenshots, naabu's SYN scan dlopens libpcap, WeasyPrint text layout
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
chromium \
|
|
libpcap0.8t64 \
|
|
libpango-1.0-0 libpangoft2-1.0-0 libharfbuzz-subset0 fontconfig \
|
|
&& rm -rf /var/lib/apt/lists/* \
|
|
&& python3 -m pip install --no-cache-dir --break-system-packages wafw00f
|
|
|
|
COPY --from=gotools /out/bin/ /root/go/bin/
|
|
ENV PATH="/root/go/bin:${PATH}"
|
|
|
|
COPY shared /app/shared
|
|
COPY tools /app/tools
|
|
|
|
COPY worker/pyproject.toml worker/uv.lock* ./
|
|
|
|
RUN uv sync --no-install-project
|
|
|
|
CMD ["uv", "run", "celery", "-A", "app.celery", "worker", "--loglevel=INFO"]
|
|
|
|
|
|
# stages/, reports/, interest/ and toolbox/ are mounted in dev; a prod stage must
|
|
# COPY them and keep PYTHONPATH=/app
|