WEBDIENSTE/lasuite-messages
1
0
Fork 0
mirror of https://github.com/suitenumerique/messages.git synced 2026-08-17 21:25:41 +02:00
Code Issues Packages Projects Releases Wiki Activity

📝(spam) add full documentation on spam processing

Browse Source
This commit is contained in:
Sylvain Zimmer
2026-07-22 16:11:19 +02:00
committed by jbpenrath
parent 07e906a390
commit 31144b1a88
2 changed files with 341 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+1 -1
View File
@@ -109,7 +109,7 @@
- DKIM signing for outbound messages
- SPF and DMARC policy enforcement
- Anti-spam filtering via rspamd
- Anti-spam filtering via rspamd — see [spam.md](spam.md) for the inbound spam & sender-authentication pipeline
### Data Protection
docs/spam.md
+340
View File
@@ -0,0 +1,340 @@
# Spam & sender-authentication processing
This document describes how Messages classifies inbound mail as spam or forged,
how that classification is configured (globally or per mail domain), how it can
be extended (native rspamd, header rules, custom webhooks, or an upstream MX
gateway), and how the results are surfaced to users.
## Overview
Every externally-received message runs through an **inbound pipeline** before it
is delivered. The pipeline is assembled per recipient mailbox and, for external
mail, runs these steps in order:
1. **Before-spam webhooks** (`message.inbound`) — user/integration webhooks that
may drop, defer, or pre-decide the spam verdict.
2. **Hardcoded header rules** — deterministic `header_match` rules from config.
3. **rspamd** — native `/checkv2` scan.
4. **Inbound authentication** — DKIM/DMARC verdict (SPF indirectly).
5. **After-spam webhooks** (`message.delivering`, then `message.delivered`).
Each step returns a `Decision` (`CONTINUE` / `RETRY` / `DROP`) and may set the
spam verdict. The verdict is a tri-state `ctx.is_spam`: `None` (undecided) until
a step decides it; **the last decisive step wins**, and the hardcoded-rules and
rspamd steps are skipped entirely once a verdict already exists. This lets a
before-spam webhook, an internal-origin short-circuit, or a header rule pre-empt
rspamd.
Two distinct outcomes are produced:
- **`is_spam` (boolean)** — routes the message to the **Junk** view and
suppresses auto-reply and push notifications. Set only by high-confidence
signals (rspamd `quarantine`/`reject`, a matching `spam` rule, or a webhook
override).
- **Graded UI markers** (`postmark["spam"]` = `"possible"` / `"likely"`) — the
message still lands in the inbox but shows a "may be / likely spam" banner.
Internal mailbox-to-mailbox mail and selfcheck probes are trusted: the task
pre-sets `is_spam = False` and the pipeline omits the spam and auth steps
(webhooks still fire, so consumers can't tell the difference).
> Implementation: `core/mda/inbound_pipeline.py` (pipeline), `core/mda/spam.py`
> (rspamd + rules), `core/mda/inbound_auth.py` (DKIM/DMARC),
> `core/mda/inbound_tasks.py` (task wrapper). Webhooks are documented in
> [webhooks.md](./webhooks.md).
## rspamd native integration
Messages talks to rspamd over HTTP. The reference deployment ships an **MPA**
container (rspamd engine + nginx proxy, see `src/mpa/`) that exposes the
`/checkv2` endpoint.
On each external message, `call_rspamd()` POSTs the raw RFC-822 bytes to
`{rspamd_url}/checkv2` (`Content-Type: message/rfc822`, 10 s timeout), optionally
with an `Authorization` header, and forwards the SMTP envelope as rspamd scan
headers (`From`→`mail_from`, `Rcpt`→`rcpt_to`, `IP`→`ip`, `Helo`→`helo`,
`Hostname`→`hostname`) so rspamd can evaluate SPF, reputation, etc.
Messages **does not interpret the rspamd score**. It reads back rspamd's
`action` and maps it to a delivery outcome. This is the single source of truth
for that mapping:
| rspamd `action` | Outcome |
|----------------------------|---------------------------------------------------------------|
| `no action` | Deliver to inbox, not spam |
| `greylist`, `soft reject` | `RETRY` — held and re-tried (temporary defer, see below) |
| `add header` | Deliver to inbox, UI marker `spam = "possible"` |
| `rewrite subject` | Deliver to inbox, UI marker `spam = "likely"` |
| `quarantine`, `reject` | `is_spam = True` → **Junk** (reject can't be honored at SMTP time, so it lands in Junk) |
| `discard` | `DROP` — accepted and silently blackholed, no bounce |
| unknown / unmapped | Deliver to inbox |
If rspamd is **not configured** (`rspamd_url` absent), the step is a no-op and
the verdict is left to other steps. If rspamd is configured but **errors or is
unreachable**, the message is **held for `RETRY` (fail-closed, never
fail-open)** rather than delivered unchecked — see deferral below.
DKIM/DMARC symbols from the same `/checkv2` response are reused by the inbound
authentication step (below) when it runs in `rspamd` mode, so a message is only
scanned once.
## Configuration: global and per-domain
Spam behavior is driven by a single `SPAM_CONFIG` dictionary.
- **Global** — the `SPAM_CONFIG` Django setting (env var `SPAM_CONFIG`, a
JSON/dict value, default `{}`).
- **Per mail domain** — a `MailDomain` may override any subset of keys via
`MailDomain.custom_settings["SPAM_CONFIG"]`. Resolution is a shallow
key-by-key merge over the global config:
```python
spam_config = settings.SPAM_CONFIG.copy()
if maildomain.custom_settings and "SPAM_CONFIG" in maildomain.custom_settings:
spam_config.update(maildomain.custom_settings["SPAM_CONFIG"])
```
(`MailDomain.get_spam_config()`, resolved at delivery time from the recipient
mailbox's domain.)
There is **no per-mailbox spam configuration** — the effective scope is
global → mail domain. (A future enhancement could add a per-mailbox layer.)
### `SPAM_CONFIG` keys
| Key | Type | Meaning |
|------------------|--------|---------|
| `rspamd_url` | string | Base URL of the rspamd HTTP endpoint (`/checkv2` is appended). Omit to disable rspamd. |
| `rspamd_auth` | string | Optional value for the `Authorization` header sent to rspamd. |
| `inbound_auth` | string | Sender-auth backend: `native`, `rspamd`, or `authentication-results`. Omit/empty to disable DKIM/DMARC checks. |
| `trusted_relays` | int | Number of sender-side `Received`/`Authentication-Results` blocks to trust, counting from the boundary our own MTA prepends. Default `0` (trust only our own hop). Raise this when a fixed upstream gateway sits in front. |
| `rules` | list | Ordered hardcoded header-match rules (see below). |
### Related settings
| Setting | Default | Meaning |
|-------------------------------------|----------------|---------|
| `SPAM_CONFIG` | `{}` | Global spam config dict (above). |
| `MESSAGES_INBOUND_DEFERRAL_MAX_AGE` | `172800` (48h) | Max time a message may be held on `RETRY` before it is force-delivered flagged (see deferral). |
> Note: there are no dedicated `RSPAMD_*` env vars — the rspamd URL and auth
> live inside `SPAM_CONFIG` (`rspamd_url`, `rspamd_auth`).
## Sender authentication (DKIM / DMARC)
The inbound-auth step produces a **sender-auth verdict** independent of the spam
verdict, stored in `postmark["auth"]` and surfaced in the UI (below):
- `fail` → the message is a likely **forgery** (DMARC fail).
- `none` → the sender's identity **could not be verified** (DKIM not passing,
no enforceable DMARC).
- verified → nothing recorded (no banner).
The backend is selected by `SPAM_CONFIG["inbound_auth"]`:
- **`native`** — verify the DKIM signature locally (crypto + DNS) and require
strict alignment between the signing `d=` domain and the `From:` domain. Never
returns `fail` (no DMARC policy lookup); worst case is `none`.
- **`rspamd`** — read DKIM/DMARC **symbols** from the rspamd `/checkv2` result
(reusing the spam-step scan). Verdict precedence: `fail` > `pass` > `none`.
- **`authentication-results`** — parse `dkim=`/`dmarc=` from the
`Authentication-Results` header(s) added by trusted upstream relays (bounded
by `trusted_relays`). Use this when an upstream MX gateway already does
authentication.
### rspamd symbol → outcome (mode `rspamd`)
| Check | Symbols → `pass` | Symbols → `fail` | Symbols → `none` |
|-------|-------------------------|------------------------------------------------------------------------------|---------------------------|
| DKIM | `R_DKIM_ALLOW` | `R_DKIM_REJECT`, `R_DKIM_PERMFAIL`, `R_DKIM_TEMPFAIL`, `DKIM_INVALID` | `R_DKIM_NA`, `DKIM_NA` |
| DMARC | `DMARC_POLICY_ALLOW` | `DMARC_POLICY_REJECT`, `DMARC_POLICY_QUARANTINE`, `DMARC_BAD_POLICY` | `DMARC_NA` |
Final rule: a DMARC `fail` yields **forged** (`auth = "fail"`); otherwise if DKIM
is not `pass`, **unverified** (`auth = "none"`); otherwise verified.
> **SPF is not surfaced as a standalone verdict.** It only influences
> classification indirectly through rspamd scoring (the envelope is forwarded to
> rspamd). The user-facing auth verdict is DKIM + DMARC only.
## Hardcoded header rules
`SPAM_CONFIG["rules"]` is an ordered list of deterministic header-match rules,
evaluated before rspamd. The first matching rule decides the verdict. Each rule:
| Field | Meaning |
|----------------------|---------|
| `header_match` | Literal `Header-Name: value` (case-insensitive). Must contain a colon. |
| `header_match_regex` | Regex alternative, full-match, case-insensitive. |
| `action` | `spam` / `reject` → mark spam; `ham` / `no action` → mark not-spam. Default `spam`. |
Rules honor `trusted_relays`: only headers within the trusted window (the most
recent `trusted_relays + 1` header blocks, newest first) are considered, so a
spammer can't forge a header that an upstream you trust would have stripped or
overwritten. The `Return-Path` header is always ignored (spoofable envelope
value).
This is the primary mechanism for **honoring the verdict of an upstream filter**
(next section).
## Upstream / edge MX filtering ("en amont")
Many deployments put a dedicated anti-spam gateway **in front of** Messages at
the MX edge — it scans mail before it ever reaches a mailbox and typically
stamps its verdict into a header (e.g. `X-Spam-Flag: YES`) and/or adds its own
`Authentication-Results`. Messages accommodates this without any native scanning
of its own:
1. **Trust the gateway's position.** Set `trusted_relays` to the number of hops
the gateway adds, so its headers fall inside the trusted window and forged
copies from further upstream are ignored.
2. **Honor its spam verdict** with a hardcoded rule, e.g.:
```json
{
"SPAM_CONFIG": {
"trusted_relays": 1,
"rules": [
{ "header_match_regex": "X-Spam-Flag:\\s*YES", "action": "spam" }
]
}
}
```
(Set globally or per mail domain, e.g. only for domains whose MX points at the
gateway.)
3. **Honor its authentication results** by setting
`inbound_auth: "authentication-results"` so the gateway's DKIM/DMARC checks
drive the sender-auth verdict instead of re-checking locally.
You can run an upstream gateway **and** rspamd together (defense in depth):
header rules run first and short-circuit rspamd when they match, so the gateway's
decision takes precedence and rspamd only scores what the gateway passed through.
## Custom spam processor via webhooks
Webhooks can fully **replace or override** the built-in spam decision — see
[webhooks.md](./webhooks.md) for the general webhook contract; this section
covers the spam-specific behavior.
A **blocking** webhook (`message.inbound` or `message.delivering`) may return a
`2xx` JSON body containing:
```json
{ "is_spam": true }
```
which sets `ctx.is_spam` to that boolean. Only a real JSON boolean is honored;
any other value means "no opinion".
Phase determines the semantics:
- **`message.inbound` (before-spam)** runs *before* the header-rules and rspamd
steps. Because those steps skip once a verdict exists, a before-spam webhook
that sets `is_spam` **pre-empts rspamd entirely** — i.e. it becomes your
spam processor.
- **`message.delivering` (after-spam)** runs *after* rspamd, so it **overrides**
the verdict rspamd produced.
- **`message.delivered` (after-spam, fire-and-forget)** is non-blocking; its
response is ignored and it cannot influence the verdict. It receives the final
`is_spam` value for logging/sync.
The webhook payload carries the pending verdict in the `X-StMsg-Is-Spam` header
(`pending` while undecided during the before-spam phase, else `true`/`false`).
The response body may also drive other actions (`action: "drop"`, labels,
assignment, `skip_autoreply`, etc.) — see [webhooks.md](./webhooks.md).
Webhook channels are scoped **global / maildomain / mailbox** and must have
`Channel.is_active = True` to fire. Blocking-webhook results (including the
`is_spam` override) are cached for the deferral window and replayed on retry, so
a rspamd outage that forces a `RETRY` doesn't re-invoke an already-successful
before-spam webhook.
## What happens to a spam message
Once `is_spam = True`:
- **Foldering** — spam is a boolean flag on the message/thread, not a separate
folder. The thread list excludes spam by default; the **Junk** view is the
same list with `is_spam=1`. The thread's `is_spam` follows its first message,
and spam messages are excluded from "active" counters/timestamps. `is_spam` is
indexed for search and filtering. It is independent of `is_trashed`.
- **Auto-reply suppressed** — `should_send_autoreply()` returns early for spam
(never vacation-reply to spam).
- **Push notifications suppressed** — no push is enqueued for a spam message.
### Deferral & "processing failed"
`RETRY` outcomes (rspamd `greylist`/`soft reject`, rspamd/webhook errors, or a
non-2xx blocking webhook) hold the message and re-try it on the inbound queue
(roughly every 5 minutes). If a message is still failing after
`MESSAGES_INBOUND_DEFERRAL_MAX_AGE` (default 48 h), it is **force-delivered**
with `is_spam = False`, auto-reply skipped, and stamped
`postmark["processing"] = "fail"` — which surfaces the "delivered without our
usual safety checks" banner (below). This guarantees mail is never lost to a
persistently failing rspamd/webhook dependency, at the cost of one un-checked
delivery.
## User-facing warnings in the reading UI
The reading view shows up to four banners, derived from the message's
`stmsg_headers` (a serializer projection of the internal `Message.postmark`
JSONField; legacy `X-StMsg-*` MIME headers are also merged for old messages).
> Naming note: "postmark" here is an **internal per-delivery pipeline record**
> (`Message.postmark`), not the Postmark email SaaS.
| Banner | Severity | `stmsg_headers` trigger | Text |
|--------|----------|-------------------------|------|
| **Forged sender** | error | `sender-auth == "fail"` | "This message failed sender authentication and is likely a forgery. Do not trust it." |
| **Unverified sender** | warning | `sender-auth == "none"` | "This contact's identity could not be verified. Proceed with caution." |
| **Processing failed** | error | `processing-failed` truthy | "This message was delivered without our usual safety checks. Please review it with caution." |
| **Suspected spam** | warning | `spam == "likely"` / `"possible"` | "This message is likely spam…" / "This message may be spam. Review it with caution." |
The sender chip also shows an icon + tooltip: a `warning` icon for unverified
and a `gpp_bad` icon for forged senders.
### How each warning maps to the pipeline
| Banner | Source signal | Produced by |
|-------------------|---------------|-------------|
| Forged / Unverified sender | `postmark["auth"]` = `fail` / `none` | Inbound-auth step (DKIM/DMARC) — for mode `rspamd`, from the symbol table above |
| Suspected spam | `postmark["spam"]` = `possible` / `likely` | rspamd **action** `add header` → possible, `rewrite subject` → likely |
| Processing failed | `postmark["processing"]` = `fail` | Deferral-window expiry force-delivery (any persistently failing step) |
Note the two-tier design: rspamd's mid-confidence actions (`add header`,
`rewrite subject`) produce an **inbox banner**, while its high-confidence actions
(`quarantine`, `reject`) set `is_spam` and route to **Junk** with no banner.
## User "report as spam" action
Users with edit rights can flag a message/thread as spam (or un-flag it) via the
flag API (`spam` flag). This sets the `is_spam` boolean (cascading to draft
children) and recomputes thread stats, moving the thread in/out of the Junk view.
> **Gap:** this action is purely local foldering — it does **not** send any
> feedback to rspamd (no `learnspam`/`learnham`/`fuzzy` training call exists in
> the backend). Wiring the report-as-spam / not-spam actions into rspamd's learn
> endpoints is tracked as a future enhancement (issue #509).
## Not yet implemented / future work
- **rspamd learn feedback** from the report-as-spam button (#509).
- **Per-mailbox** spam configuration (currently global → maildomain only).
- **SPF** as a standalone user-visible verdict (today it only feeds rspamd
scoring; the surfaced auth verdict is DKIM + DMARC).
## Implementation map
| Area | File |
|------|------|
| Pipeline assembly, rspamd action mapping, deferral constants | `core/mda/inbound_pipeline.py` |
| rspamd `/checkv2` client, hardcoded header rules | `core/mda/spam.py` |
| DKIM/DMARC verdict, rspamd symbol table, AR parsing | `core/mda/inbound_auth.py` |
| Task wrapper, internal/selfcheck short-circuit, force-delivery | `core/mda/inbound_tasks.py` |
| Webhook dispatch, phases, `is_spam` override, result cache | `core/mda/dispatch_webhooks.py` |
| `SPAM_CONFIG` resolution (`get_spam_config`) | `core/models.py` (`MailDomain`) |
| `is_spam` flag, `postmark` field, `stmsg_headers` projection | `core/models.py` (`Message`/`Thread`) |
| Spam / Junk view filtering | `core/api/viewsets/thread.py` |
| Report-as-spam flag action | `core/api/viewsets/flag.py` |
| UI warning banners | `src/frontend/.../thread-message/thread-message-header.tsx` |
| rspamd engine + proxy container | `src/mpa/` |