⬆️(collaboration) upgrade yhub to 0.9.0 and really delete superseded blobs

The S3 persistence plugin records the version id of the object it wrote
and names that version when it deletes it. On a versioned bucket - what
a deployment runs - a delete that names no version deletes nothing: it
writes a delete marker and keeps every version underneath. Each
compaction supersedes the blobs of the one before, so what was kept was
every version of every document ever written, a document someone asked
to erase included, still readable by anyone who can list versions. On
AWS this needs s3:DeleteObjectVersion, which a policy granting
s3:DeleteObject alone does not cover.

Blobs are written to the bucket for every branch of a document.

YHUB_S3_PERSISTENCE now governs only whether new blobs are written
there. The plugin itself is attached whenever the YHUB_S3_* settings
name a bucket, on or off, because reading is the half that must never
be taken away: a row pointing at an object is unreadable without the
plugin that wrote it, and yhub reports such a version as having no
content rather than as an error. Turning the toggle off stops the
writing and leaves the reading alone; it is the settings, not the
toggle, that a deployment whose bucket holds anything must keep. Half a
configuration is a startup error naming what is missing, as before.

The dev stack keeps the toggle off and creates its bucket versioned, so
flipping it on exercises what a deployment runs rather than a simpler
case. Its createbuckets job needed fixing to do so: the folded yaml
block joins its lines with a space, so the trailing backslashes reached
the shell as an escaped space glued to the next word and everything
past the first && silently did nothing - the media bucket never had
versioning enabled either.

Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>

CHANGELOG.md

@@ -89,6 +89,26 @@ and this project adheres to
   `COLLABORATION_SERVER_ORIGIN` now gates the http routes as well as the
   websocket
 
+- ⬆️(collaboration) upgrade yhub to 0.9.0 and delete superseded document blobs
+  for real. Its S3 persistence plugin now records the object version it wrote
+  and names that version when it deletes it. On a versioned bucket — which is
+  what a deployment runs — a delete that names no version deletes nothing: it
+  writes a delete marker and keeps every version underneath. Each compaction
+  supersedes the blobs of the one before, so what was kept was every version of
+  every document ever written, a document someone asked to erase included, still
+  readable by anyone who can list versions. Blobs are written to the bucket for
+  every branch of a document.
+
+  `YHUB_S3_PERSISTENCE` now governs only whether new blobs are *written* to the
+  bucket. The plugin itself is attached whenever the `YHUB_S3_*` settings name
+  one, on or off, so that the objects an earlier run wrote stay readable —
+  turning the toggle off used to strand them, since a row pointing at an object
+  is unreadable without the plugin and yhub reports such a version as having no
+  content rather than as an error. The settings, not the toggle, are what a
+  deployment whose bucket holds anything must keep. The dev stack keeps the
+  toggle off and now creates its bucket versioned, so flipping it on exercises
+  what a deployment runs rather than a simpler case
+
 ### Fixed
 
 - 🐛(frontend) stop reconnecting to the collaboration server when it has refused

UPGRADE.md

@@ -228,9 +228,10 @@ upgrade, in the order they are done, and end with the API changes.
   its own PostgreSQL database (`YHUB_S3_PERSISTENCE=true`, plus the
   `YHUB_S3_*` settings). It is off by default and nothing about this upgrade
   needs it. Read the "Document storage" section of `src/yhub-server/README.md`
-  before enabling it: a document persisted that way cannot be read back once
-  the setting is removed, and it is a third bucket, not the backend's
-  `AWS_S3_*` nor the legacy one the migration reads.
+  before enabling it: the `YHUB_S3_*` settings attach the bucket whether or not
+  the toggle is on and have to stay in place for as long as it holds anything,
+  and it is a third bucket, not the backend's `AWS_S3_*` nor the legacy one the
+  migration reads.
 
 - The endpoint `/api/v1.0/documents/{document_id}/content/`, added in 5.0.0, is
   removed, both its `GET` and its `PATCH`. The content of a document is now

compose.yml

@@ -49,11 +49,23 @@ services:
       minio:
         condition: service_healthy
         restart: true
+    # The yhub bucket is made here rather than left to the collaboration
+    # server, which creates the bucket it does not find with no versioning on
+    # it. Versioning is what a deployment runs, and it is the case where
+    # deleting an object needs the version named to delete anything at all
+    # (YHUB_S3_PERSISTENCE, src/yhub-server/README.md).
+    #
+    # The lines below end on `&&` and not on a backslash: yaml folds this block
+    # by joining its lines with a space, so a trailing `\` reaches the shell as
+    # an escaped space glued to the word after it — ` /usr/bin/mc`, which is no
+    # command, and everything past the first `&&` silently did nothing.
     entrypoint: >
       sh -c "
-      /usr/bin/mc alias set impress http://minio:9000 impress password && \
-      /usr/bin/mc mb impress/impress-media-storage && \
-      /usr/bin/mc version enable impress/impress-media-storage && \
+      /usr/bin/mc alias set impress http://minio:9000 impress password &&
+      /usr/bin/mc mb --ignore-existing impress/impress-media-storage &&
+      /usr/bin/mc version enable impress/impress-media-storage &&
+      /usr/bin/mc mb --ignore-existing impress/yhub-storage &&
+      /usr/bin/mc version enable impress/yhub-storage &&
       exit 0;"
 
   app-dev:
@@ -261,7 +273,12 @@ services:
         condition: service_healthy
       # soft migration reads the legacy document store at startup traffic —
       # starting before minio would cache 401s for the first accessed docs —
-      # and YHUB_S3_PERSISTENCE, when it is on, checks its own bucket at boot
+      # and YHUB_S3_PERSISTENCE checks its own bucket at boot. Waiting for the
+      # buckets to be *made*, not merely for the job to have started: the
+      # collaboration server creates the bucket it does not find, and the one
+      # it creates has no versioning
+      createbuckets:
+        condition: service_completed_successfully
       minio:
         condition: service_healthy
 

documentation/env.md

@@ -182,11 +182,11 @@ documents what each of them changes.
 | LEGACY_S3_BUCKET_NAME         | Name of the legacy media bucket                                                                                                                          | impress-media-storage  |
 | LEGACY_S3_REGION_NAME         | Region of that bucket, when its provider needs one                                                                                                       | us-east-1              |
 | LEGACY_S3_SIGNATURE_VERSION   | How the calls to that bucket are signed, s3v4 or v4                                                                                                      | s3v4                   |
-| YHUB_S3_PERSISTENCE           | Set to "true" to store the document blobs in a bucket instead of the yhub database. Read the "Document storage" section of `src/yhub-server/README.md` first: it cannot be turned back off | false |
-| YHUB_S3_ENDPOINT_URL          | Required by YHUB_S3_PERSISTENCE, endpoint of that bucket, without a path                                                                                 |                        |
-| YHUB_S3_ACCESS_KEY_ID         | Required by YHUB_S3_PERSISTENCE, read/write/delete access to that bucket (or YHUB_S3_ACCESS_KEY_ID_FILE)                                                 |                        |
-| YHUB_S3_SECRET_ACCESS_KEY     | Required by YHUB_S3_PERSISTENCE, secret of the key above (or YHUB_S3_SECRET_ACCESS_KEY_FILE)                                                             |                        |
-| YHUB_S3_BUCKET_NAME           | Required by YHUB_S3_PERSISTENCE, name of that bucket, created on startup when missing                                                                    |                        |
+| YHUB_S3_PERSISTENCE           | Set to "true" to write new document blobs to a bucket instead of the yhub database. The YHUB_S3_* settings below attach the bucket whether or not this is on, and must stay in place for as long as it holds anything — see the "Document storage" section of `src/yhub-server/README.md` | false |
+| YHUB_S3_ENDPOINT_URL          | Endpoint of the bucket the blobs live in, without a path. Required as a set with the three below                                                                                 |                        |
+| YHUB_S3_ACCESS_KEY_ID         | Read/write/delete access to that bucket, deleting object versions included (or YHUB_S3_ACCESS_KEY_ID_FILE)                                                 |                        |
+| YHUB_S3_SECRET_ACCESS_KEY     | Secret of the key above (or YHUB_S3_SECRET_ACCESS_KEY_FILE)                                                             |                        |
+| YHUB_S3_BUCKET_NAME           | Name of that bucket, created on startup when missing                                                                    |                        |
 | YHUB_S3_REGION_NAME           | Region of that bucket, when its provider needs one                                                                                                       |                        |
 
 ## impress-y-provider container

env.d/development/yhub

@@ -34,11 +34,15 @@ LEGACY_S3_ACCESS_KEY_ID=impress
 LEGACY_S3_SECRET_ACCESS_KEY=password
 
 # Document storage: where the blobs of a compaction are written. Off, they stay
-# in yhub's postgres, which is what this stack runs. Turning it on stores them
-# in object storage instead — here the same minio, in a bucket of its own that
-# the server creates on startup when it is missing. Read the "Document storage"
-# section of src/yhub-server/README.md first: a document persisted this way
-# cannot be read back with the plugin turned off again.
+# in yhub's postgres, which is what this stack runs. On, they go to object
+# storage instead — here the same minio, in a bucket of its own, created
+# versioned by compose so that flipping this to true exercises what a
+# deployment does rather than a simpler case.
+#
+# The settings below are read whether or not the toggle is on: they are what
+# attaches the S3 plugin, and the plugin is what can read back the objects a
+# previous run wrote. Only the writing follows the toggle. See the "Document
+# storage" section of src/yhub-server/README.md.
 YHUB_S3_PERSISTENCE=false
 YHUB_S3_ENDPOINT_URL=http://minio:9000
 YHUB_S3_ACCESS_KEY_ID=impress

src/helm/impress/README.md

@@ -384,7 +384,7 @@
 | `yhub.envVars.LEGACY_S3_REGION_NAME`               | Region of the legacy bucket, when its provider needs one                                                                                                |                           |
 | `yhub.envVars.LEGACY_S3_BUCKET_NAME`               | Name of the legacy Django media bucket (default: impress-media-storage)                                                                                 |                           |
 | `yhub.envVars.LEGACY_S3_SIGNATURE_VERSION`         | How the calls to the legacy bucket are signed, s3v4 or v4 (default: s3v4)                                                                               |                           |
-| `yhub.envVars.YHUB_S3_PERSISTENCE`                 | Set to "true" to store the document blobs in a bucket instead of the yhub database — read src/yhub-server/README.md first, it cannot be turned back off |                           |
+| `yhub.envVars.YHUB_S3_PERSISTENCE`                 | Set to "true" to write new document blobs to a bucket instead of the yhub database — the YHUB_S3_* settings attach the bucket whether or not this is on, read src/yhub-server/README.md first |                           |
 | `yhub.envVars.YHUB_S3_ENDPOINT_URL`                | Required by YHUB_S3_PERSISTENCE, endpoint of the bucket the blobs are stored in, without a path (e.g. https://s3.example.com)                           |                           |
 | `yhub.envVars.YHUB_S3_ACCESS_KEY_ID`               | Required by YHUB_S3_PERSISTENCE, read/write/delete access to that bucket (or YHUB_S3_ACCESS_KEY_ID_FILE)                                                |                           |
 | `yhub.envVars.YHUB_S3_SECRET_ACCESS_KEY`           | Required by YHUB_S3_PERSISTENCE, secret of the key above (or YHUB_S3_SECRET_ACCESS_KEY_FILE)                                                            |                           |

src/helm/impress/values.yaml

@@ -934,10 +934,11 @@ jwtKeys:
 ## - `LEGACY_S3_*`, turned on by `SOFT_MIGRATION`, is the legacy Django media
 ##   bucket it reads old documents *out of*. Not the backend's `AWS_S3_*`,
 ##   which names the same bucket for the backend's own use,
-## - `YHUB_S3_*`, turned on by `YHUB_S3_PERSISTENCE`, is a bucket of its own it
-##   stores the document blobs *into*, instead of its PostgreSQL database.
-##   Read `src/yhub-server/README.md` before enabling it: a document persisted
-##   this way cannot be read back once the setting is removed.
+## - `YHUB_S3_*` is a bucket of its own it stores the document blobs *into*,
+##   instead of its PostgreSQL database, when `YHUB_S3_PERSISTENCE` asks for
+##   it. Read `src/yhub-server/README.md` before enabling it: those settings
+##   attach the bucket whether or not the toggle is on, and dropping them from
+##   a deployment whose bucket holds anything makes those documents unreadable.
 yhub:
   ## @param yhub.enabled Enable the yhub collaboration server, its service and its init-db job
   enabled: true
@@ -1052,7 +1053,7 @@ yhub:
   ## @extra yhub.envVars.LEGACY_S3_REGION_NAME Region of the legacy bucket, when its provider needs one
   ## @extra yhub.envVars.LEGACY_S3_BUCKET_NAME Name of the legacy Django media bucket (default: impress-media-storage)
   ## @extra yhub.envVars.LEGACY_S3_SIGNATURE_VERSION How the calls to the legacy bucket are signed, s3v4 or v4 (default: s3v4)
-  ## @extra yhub.envVars.YHUB_S3_PERSISTENCE Set to "true" to store the document blobs in a bucket instead of the yhub database — read src/yhub-server/README.md first, it cannot be turned back off
+  ## @extra yhub.envVars.YHUB_S3_PERSISTENCE Set to "true" to write new document blobs to a bucket instead of the yhub database — the YHUB_S3_* settings attach the bucket whether or not this is on, read src/yhub-server/README.md first
   ## @extra yhub.envVars.YHUB_S3_ENDPOINT_URL Required by YHUB_S3_PERSISTENCE, endpoint of the bucket the blobs are stored in, without a path (e.g. https://s3.example.com)
   ## @extra yhub.envVars.YHUB_S3_ACCESS_KEY_ID Required by YHUB_S3_PERSISTENCE, read/write/delete access to that bucket (or YHUB_S3_ACCESS_KEY_ID_FILE)
   ## @extra yhub.envVars.YHUB_S3_SECRET_ACCESS_KEY Required by YHUB_S3_PERSISTENCE, secret of the key above (or YHUB_S3_SECRET_ACCESS_KEY_FILE)

src/yhub-server/README.md

@@ -290,27 +290,38 @@ content map and the content ids. By default they are `bytea` columns — the
 whole corpus lives on the database disk, which is the configuration Docs has
 been running and what this server does when nothing below is set.
 
-`YHUB_S3_PERSISTENCE=true` plugs yhub's own S3 persistence plugin
-(`S3PersistenceV1`, shipped with `@y/hub`) into the chain it consults before
-writing a blob and before reading one back. The blobs then go to a bucket and
-the row keeps a reference to them, `<column>_is_reference` saying which of the
-four it is: postgres holds the index of the documents, the bucket holds their
-bytes.
+Naming a bucket plugs yhub's own S3 persistence plugin (`S3PersistenceV1`,
+shipped with `@y/hub`) into the chain it consults before writing a blob and
+before reading one back. `YHUB_S3_PERSISTENCE=true` then sends the blobs to that
+bucket, the row keeping a reference to them and `<column>_is_reference` saying
+which of the four it is: postgres holds the index of the documents, the bucket
+holds their bytes.
+
+The two are deliberately separate. **The plugin is attached whenever the bucket
+is configured, on or off**, because reading is the half that must never be taken
+away: the objects an earlier run wrote are the only copy of those versions, and
+a row pointing at one is unreadable without the plugin that wrote it. Turning
+the toggle off stops the writing — new blobs go back to postgres — and leaves
+the reading alone. Keep the settings in place for as long as the bucket holds
+anything.
 
 | Variable | Required | What it is |
 | -------- | -------- | ---------- |
-| `YHUB_S3_PERSISTENCE` | — | `true` to store the blobs in a bucket (default: postgres) |
+| `YHUB_S3_PERSISTENCE` | — | `true` to write new blobs to the bucket (default: postgres) |
 | `YHUB_S3_ENDPOINT_URL` | yes | Endpoint of that bucket, without a path (e.g. `https://s3.example.com`) |
 | `YHUB_S3_ACCESS_KEY_ID` | yes | Key with read, write and delete on the bucket (or `…_FILE`) |
 | `YHUB_S3_SECRET_ACCESS_KEY` | yes | Secret of that key (or `…_FILE`) |
 | `YHUB_S3_BUCKET_NAME` | yes | Name of the bucket. No default: a typo would create one |
 | `YHUB_S3_REGION_NAME` | no | Region, when the provider needs one told rather than discovered |
 
-"Required" means required *when the plugin is on*: it is a startup error naming
-what is missing, rather than a client that ends up anonymous and only says so
-on the first compaction — which is a background task, so the failure would show
-up as documents quietly not being persisted. The bucket in use is logged next
-to the role (`"s3Bucket":"yhub-storage"`, `null` for postgres).
+"Required" means required *as a set*: name one of them and the rest are a
+startup error naming what is missing, rather than a client that ends up
+anonymous or against the wrong host and only says so on the first compaction —
+which is a background task, so the failure would show up as documents quietly
+not being persisted. Naming none of them, with the toggle off, is the one
+configuration with no plugin at all: postgres alone, and no object anywhere that
+would need reading back. The bucket is logged next to the role
+(`"s3Bucket":"yhub-storage","s3Writes":false` — a bucket that is only read).
 
 This is a **third** bucket, and it is deliberately configured apart from the
 other two: the backend's media bucket (`AWS_S3_*`, Django's own settings) and
@@ -320,27 +331,39 @@ the process it belongs to.
 
 A few things worth knowing before turning it on:
 
-- **It cannot be turned back off.** A row pointing at an object is unreadable
-  without the plugin that wrote it, and yhub reports such a version as having
-  no content rather than as an error — so a document compacted while the plugin
-  was on comes back *empty* once it is off, silently. Turning it on is safe in
-  the other direction: rows written before keep their bytes inline and are
-  served exactly as they were,
+- **the settings are what must not be dropped, not the toggle.** A row pointing
+  at an object is unreadable without the plugin, and yhub reports such a version
+  as having no content rather than as an error — so removing the `YHUB_S3_*`
+  settings from a deployment whose bucket holds anything makes those documents
+  come back *empty*, silently. `YHUB_S3_PERSISTENCE=false` is the safe way to
+  stop using the bucket, and it goes both ways: rows written while it was off
+  keep their bytes inline and are served exactly as they were,
 - **the bucket is created at startup** when it does not exist, so the
   credentials need `HeadBucket` and, the first time, `CreateBucket`. It is
-  checked on every boot, which is also what makes a wrong endpoint or a wrong
-  key fail loudly and immediately,
+  checked on every boot — including a boot with the toggle off, which is what
+  makes a wrong endpoint or a wrong key fail loudly and immediately rather than
+  on the first document that needs reading back,
 - **both halves need it.** The worker writes the blobs and the server reads
   them back, so a split deployment (`YHUB_ROLE`) configures the bucket on both
   — in the helm chart the worker inherits `yhub.envVars`, so there is nothing
   to repeat,
-- **only the `main` branch is offloaded.** The plugin declines everything else
-  and those blobs stay in postgres, which is yhub's behaviour, not a setting,
+- **every branch is offloaded.** The bucket is where the blobs of a document
+  belong whatever branch they were written on. This is the one thing the toggle
+  moves: `branches` is every branch when it is on and none when it is off, a
+  plugin that goes on retrieving and deleting what is in the bucket and adds
+  nothing to it,
 - **objects are deleted late.** When a version's row is dropped (pruning, a
   reset, a hard deletion), the object is removed about ten seconds later, so
   that readers holding the reference are not left with a 404. A delete that
   fails is logged and forgotten: the bucket may accumulate objects no row names
   anymore, and nothing collects them,
+- **on a versioned bucket, the version is what gets deleted.** A plain delete
+  there deletes nothing — it writes a delete marker over the object and keeps
+  every version underneath it, so the blobs of every compaction ever made would
+  stay, and so would a document someone asked to erase. The plugin records the
+  version id it wrote and the delete names it (`deleteVersions`, on), which
+  removes the bytes for real — on AWS that is `s3:DeleteObjectVersion`, which a
+  policy granting `s3:DeleteObject` alone does not cover,
 - the objects are Yjs blobs keyed by
   `id:ydoc:v1/{org}/{docid}/{branch}/{gc}/{clock}` (and `id:contentmap:v1/…`,
   `id:contentids:v1/…`) — one object per version and per column, not one file
@@ -349,9 +372,13 @@ A few things worth knowing before turning it on:
 
 In the dev stack the variables are in `env.d/development/yhub`, pointing at the
 same minio the rest of the stack uses with a bucket of its own
-(`yhub-storage`), and the toggle is off. Flipping it to `true` and restarting
-the service is enough to exercise the path — on a dev database, where losing
-the documents already compacted costs nothing.
+(`yhub-storage`), and the toggle is off — the plugin is attached and reads that
+bucket, the compactions go to postgres. Flipping `YHUB_S3_PERSISTENCE` to true
+and restarting the service is enough to exercise the writing path. The bucket is
+made by the `createbuckets` job of `compose.yml` rather than by this server, and
+made *versioned*, so what is exercised is what a deployment runs rather than a
+simpler case. Watch it with `mc ls --versions --recursive impress/yhub-storage`
+from an `mc` container on the stack's network.
 
 ## Container image
 

src/yhub-server/package-lock.json

@@ -7,7 +7,7 @@
       "name": "yhub-server",
       "dependencies": {
         "@aws-sdk/client-s3": "3.1110.0",
-        "@y/hub": "0.8.2",
+        "@y/hub": "0.9.0",
         "@y/y": "14.0.0-rc.24",
         "jose": "6.2.8"
       },
@@ -504,9 +504,9 @@
       "license": "ISC"
     },
     "node_modules/@y/hub": {
-      "version": "0.8.2",
-      "resolved": "https://registry.npmjs.org/@y/hub/-/hub-0.8.2.tgz",
-      "integrity": "sha512-JMwHMwmnUAp6N+qz2MKaIRp/eDGOYIqGTRnW+aNyFg4nypMnnsjPzx7rs6/uG4RR1IDEKiqr4A5Khta44Q2NPQ==",
+      "version": "0.9.0",
+      "resolved": "https://registry.npmjs.org/@y/hub/-/hub-0.9.0.tgz",
+      "integrity": "sha512-xGzLoPpJ1lrZo++ULu0IAs3bnLpjCNhsEgHcRiIf+XaTO5iu7GjwnbO6HGqI4LiP45Dn952kkh+WrmhK8qkfaQ==",
       "license": "AGPL-3.0 OR PROPRIETARY",
       "dependencies": {
         "@y-crdt/yn": "^0.1.4",

src/yhub-server/package.json

@@ -10,7 +10,7 @@
   },
   "dependencies": {
     "@aws-sdk/client-s3": "3.1110.0",
-    "@y/hub": "0.8.2",
+    "@y/hub": "0.9.0",
     "@y/y": "14.0.0-rc.24",
     "jose": "6.2.8"
   },

src/yhub-server/server.js

@@ -106,10 +106,14 @@ const TASK_CONCURRENCY = intEnv('YHUB_TASK_CONCURRENCY', 5, 1);
 // store. Off by default, which is postgres alone, the way Docs has been
 // running.
 //
-// Not a switch that can be flipped back: a row pointing at an object is
-// unreadable without the plugin that wrote it — yhub reports that version as
-// having no content rather than as an error — so turning it off after a
-// compaction strands what was stored while it was on. See README.md.
+// This decides where *new* blobs are written, and nothing else. The plugin
+// itself is loaded whenever the bucket below is configured, on or off, because
+// a row pointing at an object is unreadable without the plugin that wrote it —
+// yhub reports such a version as having no content rather than as an error, so
+// a deployment that has ever had this on and drops the plugin loses those
+// documents silently. Keep the YHUB_S3_* settings in place for as long as the
+// bucket holds anything; turning this off then stops the writing and leaves the
+// reading alone. See README.md.
 const S3_PERSISTENCE = process.env.YHUB_S3_PERSISTENCE === 'true';
 // Its own bucket, named apart from the backend's `AWS_S3_*` and from the legacy
 // document store's `LEGACY_S3_*` (migration.js): three buckets that may sit on
@@ -928,7 +932,13 @@ const workerEvents = {
 
 // The persistence plugins yhub consults, in order, before writing a blob to
 // postgres and before reading one back. An empty list keeps everything in the
-// database, which is the default.
+// database, which is what a deployment that names no bucket gets.
+//
+// Naming a bucket is enough to get the plugin, whether or not the toggle asks
+// for the writing: reading is the half that must never be taken away, since the
+// objects an earlier run wrote are the only copy of those versions. `branches`
+// is what the toggle actually moves — every branch, or none, which is a plugin
+// that retrieves and deletes what is in the bucket and adds nothing to it.
 //
 // Read here rather than in the call below so that an incomplete configuration
 // is a startup error naming what is missing: the client would otherwise be
@@ -936,18 +946,25 @@ const workerEvents = {
 // compaction, which is a background task — the failure would show up as
 // documents quietly not being persisted.
 const persistencePlugins = () => {
-  if (!S3_PERSISTENCE) return [];
-
-  const missing = [
+  const settings = [
     ['YHUB_S3_ENDPOINT_URL', YHUB_S3_ENDPOINT_URL],
     ['YHUB_S3_ACCESS_KEY_ID', YHUB_S3_ACCESS_KEY_ID],
     ['YHUB_S3_SECRET_ACCESS_KEY', YHUB_S3_SECRET_ACCESS_KEY],
     ['YHUB_S3_BUCKET_NAME', YHUB_S3_BUCKET_NAME],
-  ]
-    .filter(([, value]) => !value)
-    .map(([name]) => name);
+  ];
+  const missing = settings.filter(([, value]) => !value).map(([name]) => name);
+  // no bucket named at all, and the toggle does not ask for one: postgres
+  // alone, and no object anywhere that would need reading back
+  if (missing.length === settings.length && !S3_PERSISTENCE) return [];
   if (missing.length > 0) {
-    throw new Error(`YHUB_S3_PERSISTENCE=true requires ${missing.join(', ')}`);
+    // half a configuration is always a mistake, and the half that is set says
+    // which mistake: a bucket was meant to be reachable and is not
+    const named = missing.join(', ');
+    throw new Error(
+      S3_PERSISTENCE
+        ? `YHUB_S3_PERSISTENCE=true requires ${named}`
+        : `The YHUB_S3_* bucket is partly configured, missing ${named}`,
+    );
   }
 
   const url = new URL(YHUB_S3_ENDPOINT_URL);
@@ -976,6 +993,22 @@ const persistencePlugins = () => {
       // left out rather than passed empty: unset, the client discovers the
       // region of the bucket instead of validating an empty string
       ...(YHUB_S3_REGION_NAME ? { region: YHUB_S3_REGION_NAME } : {}),
+      // The branches whose blobs are written here: all of them, or none, which
+      // is how the plugin is kept for reading while the writing goes back to
+      // postgres. `store` declines a branch it is not given and yhub falls
+      // through to the database, while `retrieve` and `delete` go on answering
+      // for every object already in the bucket.
+      branches: S3_PERSISTENCE ? true : [],
+      // On a versioned bucket a plain delete deletes nothing: it writes a
+      // delete marker over the object and keeps every version underneath it.
+      // Each compaction supersedes the blobs of the one before, so that would
+      // be every version ever written kept forever — and a document a user
+      // asked to erase still readable by anyone who can list versions. The
+      // plugin records the version id it wrote at store time so that the
+      // delete can name it, which is what removes the bytes. Asked for here
+      // rather than left to the plugin's default: the delete is only as
+      // thorough as this line.
+      deleteVersions: true,
     }),
   ];
 };
@@ -1028,8 +1061,11 @@ logger.info(
     server: RUNS_SERVER,
     worker: RUNS_WORKER,
     taskConcurrency: RUNS_WORKER ? TASK_CONCURRENCY : null,
-    // where the compaction blobs go — null is yhub's own postgres
-    s3Bucket: S3_PERSISTENCE ? YHUB_S3_BUCKET_NAME : null,
+    // the bucket the plugin is attached to, null when there is no plugin at
+    // all, and whether the compaction blobs are written to it or to yhub's own
+    // postgres — a bucket with `s3Writes` false is one that is only read
+    s3Bucket: YHUB_S3_BUCKET_NAME ?? null,
+    s3Writes: S3_PERSISTENCE,
     taskDebounceMs: yhub.stream.taskDebounce,
     minMessageLifetimeMs: yhub.stream.minMessageLifetime,
   },