WEBDIENSTE/rengine
1
0
Fork 0
mirror of https://github.com/yogeshojha/rengine.git synced 2026-10-20 15:21:30 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
53cdca4e4e9c7bab33cce49521e6600152ef54a0
rengine/docker-compose.yml
T
Yogesh OjhaandClaude Opus 5 d67bbb773d feat(connectors): a Caido path that works today, through HAR 
connectors/caido/ existed and told the operator "no plugin is published yet",
leaving them a raw curl contract to drive by hand. A native Caido plugin is
TypeScript against Caido's own SDK, comparable in size to the 2,264 lines of
Java in clients/burp/, and is deliberately not built here.

What is built is the path that makes Caido usable now and serves every other
proxy at the same time: clients/har/rengine_har.py, stdlib-only Python that
reads an exported history and posts it through the same /connectors/ingest
contract the Burp extension uses. Caido, ZAP, mitmproxy, Burp and every
browser's devtools export HAR, so one importer covers all of them.

It sends no more than the Burp client does: url, method, status, content type
and length, page title, the NAMES of body parameters, and whether the request
carried a session. Headers, cookies and bodies never leave the machine, and
there is deliberately no request_sample — the Java client sends none either, and
a HAR is full of credentials.

The client-side rules the extension keeps, it keeps: drop anything that is not
an http url, collapse exact repeats to keep the wire small (reNgine does the
real shaping), batch at 500, and stop after a single 401, because the endpoint
sits behind a per-IP failure limiter and retrying a rotated token locks the
operator out. An out-of-scope host is still sent — the server decides scope and
keeps an unmatched request as unassigned, so adding the target later attaches
everything already recorded.

Verified end to end against a live instance: 3 accepted, 3 new shapes, body
parameter names and page titles extracted, authenticated set from a Cookie
header whose value never travelled, hosts recorded for the Discovered tab.
./clients is mounted read-only into the api container so the tests exercise the
shipped file rather than a copy.

24 tests. One mutation survived the first cut — a 401 being retried — because
the test asserted an exception class instead of the behaviour. It drives post()
against a stubbed 401 now, and separates that from a 500, which is the server's
problem and must not send the user rotating a good token.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
2026-09-11 09:24:35 +05:30

235 lines
6.4 KiB
YAML
Raw Blame History

services:
db:
image: postgres:16-alpine
env_file:
- .env
volumes:
- postgres_data:/var/lib/postgresql/data
ports:
- "5432:5432"
command: >
postgres
-c shared_buffers=${POSTGRES_SHARED_BUFFERS:-2GB}
-c effective_cache_size=${POSTGRES_EFFECTIVE_CACHE_SIZE:-6GB}
-c work_mem=${POSTGRES_WORK_MEM:-32MB}
-c maintenance_work_mem=${POSTGRES_MAINTENANCE_WORK_MEM:-512MB}
-c random_page_cost=1.1
-c effective_io_concurrency=200
-c max_parallel_workers_per_gather=4
-c jit=off
-c max_connections=${POSTGRES_MAX_CONNECTIONS:-150}
-c shared_preload_libraries=pg_stat_statements
-c pg_stat_statements.max=5000
-c pg_stat_statements.track=top
-c track_io_timing=on
-c log_min_duration_statement=${POSTGRES_LOG_MIN_DURATION:-2000}
healthcheck:
test: ["CMD-SHELL", "pg_isready -U rengine"]
interval: 5s
timeout: 5s
retries: 5
redis:
image: redis:8-alpine
ports:
- "6379:6379"
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 5s
retries: 5
command: >
redis-server
--appendonly yes
--maxmemory 256mb
--maxmemory-policy allkeys-lru
api:
build:
context: .
dockerfile: api/Dockerfile
target: development
volumes:
- ./api/app:/app/app
- ./shared:/app/shared
- ./tools:/app/tools
- ./stages:/app/stages
- ./reports:/app/reports
- ./interest:/app/interest
- ./mcp:/app/mcp
- ./connectors:/app/connectors
- ./toolbox:/app/toolbox
- ./alembic:/app/alembic
- ./alembic.ini:/app/alembic.ini
- ./tests:/app/tests
- ./pyproject.toml:/app/pyproject.toml
# read-only, so the tests can check a frontend mirror has not drifted
- ./frontend/src/lib/config:/app/frontend-config:ro
# read-only, so the tests exercise the shipped client rather than a copy
- ./clients:/app/clients:ro
- ./scan_media:/app/scan_media
- vuln_templates:/app/vuln-templates
- wordlists:/app/wordlists
- reports_out:/app/reports-out
- report_fonts:/app/report-fonts
- /app/shared/build
- /app/shared/rengine_shared.egg-info
- /app/tools/build
- /app/tools/rengine_tools.egg-info
ports:
- "8000:8000"
env_file:
- .env
environment:
- WATCHFILES_FORCE_POLLING=true
depends_on:
db:
condition: service_healthy
redis:
condition: service_healthy
# default workers for handling regular tasks like imports, whois etc
worker-default:
init: true
build:
context: .
dockerfile: worker/Dockerfile
target: development
volumes:
- ./worker/app:/app/app
- ./shared:/app/shared
- ./tools:/app/tools
- ./stages:/app/stages
- ./reports:/app/reports
- ./interest:/app/interest
- ./toolbox:/app/toolbox
- ./scan_media:/app/scan_media
- vuln_templates:/app/vuln-templates
- wordlists:/app/wordlists
- reports_out:/app/reports-out
- report_fonts:/app/report-fonts
- /app/shared/build
- /app/shared/rengine_shared.egg-info
- /app/tools/build
- /app/tools/rengine_tools.egg-info
env_file:
- .env
# one scan now holds up to 5 slots at its widest — 3 gating stages plus the 2 that
# run beside them — so 8 covered a single scan and queued the moment two overlapped
command: >
uv run celery -A app.celery worker
--loglevel=DEBUG
--concurrency=${CELERY_SCAN_CONCURRENCY:-12}
--queues=critical,default,scans
--hostname=worker-default@%h
--max-tasks-per-child=100
depends_on:
db:
condition: service_healthy
redis:
condition: service_healthy
healthcheck:
test: ["CMD-SHELL", "/app/.venv/bin/celery -A app.celery inspect ping -d worker-default@$$HOSTNAME || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
# beat scheduler: emits the schedule tick that fires due scan schedules
worker-beat:
init: true
build:
context: .
dockerfile: worker/Dockerfile
target: development
volumes:
- ./worker/app:/app/app
- ./shared:/app/shared
- ./tools:/app/tools
- ./stages:/app/stages
- ./reports:/app/reports
- ./interest:/app/interest
- ./toolbox:/app/toolbox
- ./scan_media:/app/scan_media
- vuln_templates:/app/vuln-templates
- wordlists:/app/wordlists
- reports_out:/app/reports-out
- report_fonts:/app/report-fonts
- /app/shared/build
- /app/shared/rengine_shared.egg-info
- /app/tools/build
- /app/tools/rengine_tools.egg-info
env_file:
- .env
command: >
uv run celery -A app.celery beat
--loglevel=INFO
--schedule=/tmp/celerybeat-schedule
depends_on:
db:
condition: service_healthy
redis:
condition: service_healthy
flower:
build:
context: .
dockerfile: worker/Dockerfile
target: development
volumes:
- ./worker/app:/app/app
- ./shared:/app/shared
- ./tools:/app/tools
- ./stages:/app/stages
- ./reports:/app/reports
- ./interest:/app/interest
- ./toolbox:/app/toolbox
- ./scan_media:/app/scan_media
- vuln_templates:/app/vuln-templates
- wordlists:/app/wordlists
- reports_out:/app/reports-out
- report_fonts:/app/report-fonts
- /app/shared/build
- /app/shared/rengine_shared.egg-info
- /app/tools/build
- /app/tools/rengine_tools.egg-info
ports:
- "5555:5555"
env_file:
- .env
environment:
- FLOWER_UNAUTHENTICATED_API=true
command: >
uv run celery -A app.celery flower
--port=5555
--broker=redis://${REDIS_HOST:-redis}:${REDIS_PORT:-6379}/${REDIS_DB:-0}
--basic-auth=${FLOWER_USER:-admin}:${FLOWER_PASSWORD:-admin}
depends_on:
redis:
condition: service_healthy
frontend:
build:
context: .
dockerfile: frontend/Dockerfile.dev
volumes:
- ./frontend/src:/app/src
- ./frontend/vite.config.ts:/app/vite.config.ts
ports:
- "5173:5173"
environment:
- NODE_ENV=development
depends_on:
- api
volumes:
postgres_data:
redis_data:
celery_beat_data:
vuln_templates:
wordlists:
reports_out:
report_fonts:
Reference in New Issue View Git Blame Copy Permalink