# Tiered blob storage Blobs (raw RFC822 email bodies and attachments) live in PostgreSQL by default. Once a blob is older than `MESSAGES_BLOBS_OFFLOAD_DELAY`, a periodic celery task moves its bytes to S3 and clears the PG row's `raw_content`. Reads transparently fetch from whichever location the row points at — application code only ever calls `blob.get_content()`. ## Architecture - **Storage path**: `blobs/{key_id}/{sha[:3]}/{sha}`. The leading `key_id` segment lets blobs encrypted with different keys coexist (essential for crash-safe online key rotation). The 3-char sha prefix shards each key into 4096 sub-prefixes for S3 request-rate balance. - **Deduplication**: identical content always lands as exactly ONE ``Blob`` row, regardless of how many mailboxes / messages / attachments reference it. ``BlobManager.create_blob`` hashes the input first and short-circuits on a sha-match — no compress, no encrypt, no insert. Multiple ``Message`` / ``Attachment`` / ``MessageTemplate`` rows can FK the same Blob; cleanup is governed by the reference graph + GC sweep below. After offload, the single Blob row maps to one S3 object. The DB is the sole source of truth; drift between DB and bucket (external deletion, lifecycle expiry) is detected offline by ``verify_blobs --mode=db-to-storage``, not on the hot path. - **Concurrency**: a Postgres transaction-scoped advisory lock keyed on the first 8 bytes of sha256 serializes offload, cleanup, and re-encrypt for any one content. Different shas run in parallel. - **Encryption**: optional AEAD (currently AES-256-GCM; the algo is named per-key in config so future additions don't require a format change). Configured secrets are hashed to 32 bytes via SHA-256 (same pattern as `encrypted-fields`'s `SALT_KEY`). The hash adds no entropy — its strength is whatever entropy the operator put into the input string. Storage layout per object: `nonce(12) || ct+tag(16)`, total overhead 28 bytes, no base64. The blob's SHA-256 is bound as AAD on the auth tag, so ciphertext is non-portable: copying bytes between blob paths fails decrypt with `InvalidTag`. - **Read-time hash verification (optional)**: set `MESSAGES_BLOBS_VERIFY_HASH=True` to re-hash decompressed plaintext on every read. Adds one SHA-256 per read. Most useful for `key_id=0` (plaintext-stored) blobs — encrypted blobs already get the AAD-bound auth tag for free. ## Blob lifetime: reference graph + GC sweep Blobs are **not owned by a Mailbox/MailDomain** at the schema level. A Blob is alive as long as any of these references it: - ``Message.blob`` (the raw RFC822 MIME) or ``Message.draft_blob`` (the body of a draft being composed) - ``Attachment.blob`` (per-attachment during draft composition) - ``MessageTemplate.blob`` (signatures, autoreply bodies) Plus a short-lived **upload reservation** in the form of a ``MailboxBlob`` row carrying an explicit ``expires_at`` timestamp. The JMAP upload endpoint creates one alongside the ``Blob`` row so the blob_id survives until the follow-up attach call lands; the attach flow drops it once the ``Attachment`` row exists. When a reference source is deleted (Message, Attachment, MessageTemplate ``post_delete``), the affected blob_id is pushed into a Redis candidate set. A periodic Celery task — ``gc_orphan_blobs_task`` in ``core/services/blob_gc.py`` — drains the set, re-checks the reference graph under the per-sha advisory lock, deletes the row if no references remain, and cleans up the S3 object inline. No per-blob celery fan-out; one task processes the whole backlog within a 55-minute wall-clock budget per hourly tick. Two modes: - ``mode="fast"`` (default, beat-scheduled hourly): drain the Redis candidate set, GC anything that's actually orphaned. - ``mode="full"``: walk every Blob row. Use as a periodic safety net (weekly cron) to catch anything dropped by a Redis outage or a signal that didn't fire. Invoke manually with ``python manage.py run_task core.services.blob_gc.gc_orphan_blobs_task --kwargs '{"mode": "full"}'``. Pass ``"dry_run": true`` to preview without deleting: ``python manage.py run_task core.services.blob_gc.gc_orphan_blobs_task --kwargs '{"mode": "full", "dry_run": true}'``. The task logs one INFO line per would-be-deleted blob (id, sha256, storage_location, size, created_at) and returns counts with ``would_delete`` instead of ``deleted``. Nothing is locked or mutated. In ``fast`` mode, dry-run peeks at the Redis candidate set rather than popping it, so a follow-up real run still sees the same ids. The result is informational: a concurrent reference insert between dry-run and a real run could turn a would-delete into a skip. The GC-driven model fixes a latent bug from the FK-cascade era: when two mailboxes shared a thread, deleting one mailbox would CASCADE through ``Blob.mailbox`` and break content access for the other mailbox. Without the FK, the blob is alive as long as any thread access still references it. ## Enabling offload By default everything runs in PostgreSQL with `raw_content` populated. To start moving cold blobs to S3: 1. Provision a bucket and credentials. 2. Set the `STORAGE_MESSAGE_BLOBS_*` env vars (see [env.md](env.md)). 3. `make create-buckets` (or the production equivalent). 4. (Optional) Preview what the next tick would offload: ``python manage.py run_task core.services.tiered_storage_tasks.offload_blobs_task --kwargs '{"dry_run": true}'``. The task logs one INFO line per eligible blob (id, size, stored, content_type, created_at) and returns counts with ``would_offload`` plus ``bytes_plain`` / ``bytes_stored``. Bypasses both the ``MESSAGES_BLOBS_OFFLOAD_ENABLED`` and ``service.enabled`` gates so you can preview before configuring the bucket. 5. Set `MESSAGES_BLOBS_OFFLOAD_ENABLED=True`. A single celery beat task fires hourly and processes eligible blobs sequentially within a 55-minute wall-clock budget — no per-blob fan-out. Each row is offloaded under the per-sha advisory lock, so the row flip is atomic. Whatever isn't done in one tick is picked up by the next. ## Encryption To enable encryption-at-rest for blobs: 1. Generate a random secret string. Use `openssl rand` or equivalent so the input is genuinely high-entropy: ```sh openssl rand -base64 32 ``` Startup emits a warning if any configured secret is shorter than 32 characters. The warning is a length check, not an entropy measurement — `"a" * 32` passes silently. Treat the floor as a tripwire for typos, not a security guarantee. 2. Add it to `MESSAGES_BLOBS_ENCRYPT_KEYS` as a JSON dict. Every entry must spell out `algo` and `secret`. Add `"active": true` to the entry whose key new blobs should be encrypted with: ```sh MESSAGES_BLOBS_ENCRYPT_KEYS='{"1": {"algo": "aes-gcm", "secret": "", "active": true}}' ``` Entries without `active` (or with `active: false`) stay readable for legacy ciphertext but no longer encrypt new blobs. At most one entry may be active. The `algo` value is a complete spec — picking a new cipher in the future means adding a new algo identifier, not changing what the current one means. Unknown algos are rejected at boot and at use time. 3. Restart. New blobs are encrypted; existing unencrypted blobs (`encryption_key_id=0`) keep working until they're rotated. `manage.py check` validates the config at startup and refuses to boot if more than one entry is active, or any algo is unknown. **Lose the key dict, lose the data.** Back it up alongside your DB. **Rotate before you hit AES-GCM's safety limit.** With a random 96-bit nonce per encryption, the standard guidance is to retire a key after 2^32 (≈ 4 billion) encryptions to keep the nonce-collision probability negligible. Plan a rotation at-or-below that count even if no key compromise is suspected; the [`re_store_blobs` runbook](#key-rotation-runbook) below walks through the steps. (One blob = one encryption, regardless of how many DB rows reference it via dedup.) ## Reconciling state: `re_store_blobs` `re_store_blobs` makes every blob's state match the current configuration. Two operations, applied based on settings: - **Key rotation** (always). Every row whose `encryption_key_id` differs from the entry currently flagged `active=true` in `MESSAGES_BLOBS_ENCRYPT_KEYS` is re-encrypted under that key. - **Restore** (only when `MESSAGES_BLOBS_OFFLOAD_ENABLED=False`). Every `OBJECT_STORAGE` row is pulled back into PostgreSQL. Two runbooks below use this command. ### Key rotation runbook To rotate from key 1 to key 2 while keeping tiered storage enabled: 1. Add the new key alongside the old one, with `active=true` only on the new entry: ```sh MESSAGES_BLOBS_ENCRYPT_KEYS='{"1": {"algo": "aes-gcm", "secret": ""}, "2": {"algo": "aes-gcm", "secret": "", "active": true}}' ``` New blobs encrypt under key 2 immediately; existing blobs stay on key 1 until rotated. 2. **Pause offload to avoid a race**: ```sh MESSAGES_BLOBS_OFFLOAD_ENABLED=False ``` Wait for in-flight tasks to drain. ⚠️ Note: with offload disabled, `re_store_blobs` will *also* pull OBJECT_STORAGE blobs back into PostgreSQL. If you only want to rotate keys and keep blobs in S3, leave `MESSAGES_BLOBS_OFFLOAD_ENABLED=True` and accept that an offload task could race the rotation (both take the per-sha advisory lock, so the worst case is a few re-tries — no corruption). 3. Run the rotation: ```sh python manage.py re_store_blobs ``` For each blob with `encryption_key_id != 2`, the command - writes the new ciphertext to `blobs/2/...` (atomic S3), - flips the dedup cohort's `encryption_key_id` to 2 (atomic DB), - best-effort deletes the old `blobs/1/...` path. Each step is independently atomic, so a crash at any point leaves the blob readable from one consistent path. 4. Confirm key 1's prefix is empty: ```sh aws s3 ls s3://msg-blobs/blobs/1/ ``` If anything remains, re-run the command (it's idempotent) or use `verify_blobs --mode=storage-to-db` to list orphans for manual cleanup. 5. Re-enable offload, then drop key 1 from the dict in a follow-up deploy: ```sh MESSAGES_BLOBS_OFFLOAD_ENABLED=True MESSAGES_BLOBS_ENCRYPT_KEYS='{"2": {"algo": "aes-gcm", "secret": "", "active": true}}' ``` ### Rolling tiered storage back To bring every offloaded blob back into PostgreSQL: 1. Disable offload so no new blobs leave PG: ```sh MESSAGES_BLOBS_OFFLOAD_ENABLED=False ``` Wait for in-flight offload tasks to drain (next beat tick is the kill switch — already-running tasks finish their current blob, then the loop's `disabled` check makes the next tick a no-op). 2. Verify your PG has room. Each row that comes back from S3 carries its own copy of the (compressed, encrypted) bytes — dedup that was sharing a single S3 object becomes one PG row per blob. Check `pg_total_relation_size('messages_blob')` ahead of time. 3. Run the restore. Bucket creds must still be configured so ciphertext can be read on the way home: ```sh python manage.py re_store_blobs ``` For every `OBJECT_STORAGE` row the command: - downloads the ciphertext, decrypts under its current key, re-encrypts under the active key, - writes `raw_content` and flips `storage_location=POSTGRES` (atomic DB), - opportunistically deletes the S3 object once the cohort empties out. 4. Confirm the bucket is empty: ```sh aws s3 ls s3://msg-blobs/blobs/ --recursive ``` Anything still there is an orphan — `verify_blobs --mode=storage-to-db` lists them. 5. (Optional) Tear down bucket creds. Once ```sh SELECT count(*) FROM messages_blob WHERE storage_location=2; ``` returns 0, you can safely unset `STORAGE_MESSAGE_BLOBS_*` — no read path will need them. ## Operator commands Two commands cover the operational surface; they are deliberately split so the read-only audit can never touch state and the mutating reconciliation can never be run by accident as part of "I just want to see what's going on": - **`python manage.py verify_blobs`** — read-only audit. Nothing is mutated; output drives manual recovery. - **`python manage.py re_store_blobs`** — mutating reconciliation (key rotation + optional pull-back from S3 → PG). See [the runbook section](#reconciling-state-re_store_blobs) above. ### `verify_blobs` (read-only) Choose a mode with `--mode=`. The default is `full` which runs both checks. - **`--mode=db-to-storage`** — for every blob row marked `OBJECT_STORAGE`, HEAD the expected S3 object. Reports `MISSING` for any drift (DB says present, bucket says absent). - **`--mode=storage-to-db`** — LIST every object under `blobs/` and look up the matching DB row. Reports `ORPHAN` for any S3 object not referenced by any row, and `INVALID PATH` for objects whose key doesn't match the `blobs/{key_id}/{sha[:3]}/{sha}` shape. - **`--mode=full`** (default) — runs `db-to-storage` then `storage-to-db` back-to-back. - **`--verify-hashes`** — additional flag. When combined with `storage-to-db` or `full`, downloads each S3 object, decrypts, decompresses, and recomputes SHA-256. Reports `HASH MISMATCH` on divergence. Slow; use after a storage incident or as periodic paranoia. Silently no-op when used with `--mode=db-to-storage`. - **`--limit=N`** — caps items checked (useful for sampling). - **`--start-after-key=`** — resumes a `storage-to-db` listing past the given key. Pair with `--limit` to chunk a multi-million- object bucket across runs. ### `re_store_blobs` (mutating) Reads each blob, decrypts under its current key, re-encrypts under the active key, and writes the result to the target storage location implied by `MESSAGES_BLOBS_OFFLOAD_ENABLED`. The PG → S3 direction (offload of cold blobs) is owned by the periodic `offload_blobs_task` celery beat task — not exposed here. - **`--dry-run`** — prints what would happen without writing. - **`--limit=N`** — caps the worklist; rerun until empty. ## Recovery scenarios | Symptom | Detection | Fix | |---|---|---| | DB row says `OBJECT_STORAGE` but S3 object missing | `verify --mode=db-to-storage` reports `MISSING` | Restore from backup (re-upload the bytes to the bucket and re-run `verify --mode=db-to-storage` to confirm). If no backup exists, re-ingest the original content from upstream (the producing source — IMAP, MTA, importer — or application logs that captured it). If neither is possible the blob is unrecoverable; mark the row with an incident annotation rather than fabricating bytes from another row. | | S3 object with no DB row | `verify --mode=storage-to-db` reports `ORPHAN` | After confirming, `aws s3 rm` the object. | | Hash mismatch | `verify --verify-hashes` reports `HASH MISMATCH` | The S3 object is corrupted (or AAD-bound integrity has been broken — same effect). Treat as data loss; restore from backup. | | Old key_id prefix non-empty after rotation | `aws s3 ls blobs//` | Re-run `re_store_blobs`; it's idempotent. | | `re_store_blobs` reports errors on a subset of blobs | command exit summary lists `Errors: N` | The loop already skipped them and continued; failed blobs stay in their pre-run state. Re-run after fixing the underlying cause (key missing, S3 5xx, etc.); the command is idempotent. | | Suspect orphan Blob rows accumulating (e.g. after a Redis outage) | `SELECT count(*) FROM messages_blob` significantly higher than the sum of ``Message.blob``, ``Message.draft_blob``, ``Attachment.blob``, ``MessageTemplate.blob`` distinct ids | Run the full GC: ``python manage.py run_task core.services.blob_gc.gc_orphan_blobs_task --kwargs '{"mode": "full"}'``. It walks every Blob row and deletes anything without a remaining reference. Idempotent. | ## Schema migration is one-way Migration `0027` drops `Blob.mailbox` / `Blob.maildomain` and the `blob_has_owner` constraint, and flips every Blob-referencing FK (`Message.blob`, `Message.draft_blob`, `Attachment.blob`, `MessageTemplate.blob`, `MailboxBlob.blob`) to `on_delete=PROTECT`. The FK drop is intentionally **not reversible**: once those columns are gone, there is no way to repopulate them from the reference graph (a blob shared across mailboxes via dedup has no single "owner" to write back). The `on_delete=PROTECT` flips are reversible at the SQL level (just metadata), but rolling back re-introduces the data-loss races the change fixed. Treat the migration as one-way for production. ## Operational notes - **Initial rollout on a populated DB.** The offload task processes eligible blobs sequentially within a 55-minute wall-clock budget per hourly tick. There is no per-blob celery fan-out, so a backlog of millions doesn't queue-bomb the broker. Ramping `MESSAGES_BLOBS_OFFLOAD_DELAY` down gradually (e.g. 365d → 90d → 30d → 1d, expressed in seconds) still helps spread the load over several days, especially if bandwidth to the bucket is the bottleneck. Run `--mode=db-to-storage` periodically afterwards to confirm everything that flipped to `OBJECT_STORAGE` is actually in the bucket. - **Reading offloaded blobs** triggers a synchronous S3 GET in the request path. Old messages are rarely re-opened, so this is generally invisible — but expect 50–500 ms latency when it does happen. - **Bulk deletes** (e.g. mailbox cascade, `QuerySet.delete()`) push the affected blob_ids into the GC candidate set via post_delete signals on the *reference sources* (Message, Attachment, Template). No per-blob celery task is enqueued — the periodic GC drains the set in a single bounded task per tick. Cascade of 100k attachments produces 100k Redis SADDs (~1-2s total), not 100k Celery messages. - **Drift between DB and bucket.** Offload trusts the DB on dedup (no S3 HEAD), so a missing-S3-but-DB-says-OBJECT_STORAGE row will not auto-repair. Run `verify_blobs --mode=db-to-storage` on a schedule (weekly, monthly — workload-dependent) to catch drift; the recovery scenarios table above maps each symptom to a fix. - **Postgres disk growth.** Offload nulls `raw_content`, it doesn't delete the row, so TOAST stabilizes around the un-offloaded backlog while heap and non-partial indexes (`pkey`, `sha256`) grow with the cumulative row count. The offload `UPDATE` changes `storage_location` (an indexed column), so HOT updates are disabled and each offload writes into every index on the table. At high write rates default autovacuum can fall behind on index and TOAST bloat. Monitor `dead_pct` and `pg_total_relation_size` for `messages_blob` over time; if growth or dead-tuple ratio trend upward after autovacuum catches up, tighten per-table autovacuum and add a periodic online compaction (e.g. `pg_repack`) to the runbook. Partitioning by `created_at` is the lever to keep in mind once row count gets large.