⬆️(collaboration) upgrade yhub to 0.9.0 and delete superseded blobs

The S3 persistence plugin records the version id of the object it wrote
and names that version when it deletes it. On a versioned bucket - what
a deployment runs - a delete that names no version deletes nothing: it
writes a delete marker and keeps every version underneath. Each
compaction supersedes the blobs of the one before, so what was kept was
every version of every document ever written, a document someone asked
to erase included, still readable by anyone who can list versions. On
AWS this needs s3:DeleteObjectVersion, which a policy granting
s3:DeleteObject alone does not cover.

Blobs are written to the bucket for every branch of a document.

YHUB_S3_PERSISTENCE now governs only whether new blobs are written
there. The plugin itself is attached whenever the YHUB_S3_* settings
name a bucket, on or off, because reading is the half that must never
be taken away: a row pointing at an object is unreadable without the
plugin that wrote it, and yhub reports such a version as having no
content rather than as an error. Turning the toggle off stops the
writing and leaves the reading alone; it is the settings, not the
toggle, that a deployment whose bucket holds anything must keep. Half a
configuration is a startup error naming what is missing, as before.

The dev stack keeps the toggle off and creates its bucket versioned, so
flipping it on exercises what a deployment runs rather than a simpler
case. Its createbuckets job needed fixing to do so: the folded yaml
block joins its lines with a space, so the trailing backslashes reached
the shell as an escaped space glued to the next word and everything
past the first && silently did nothing - the media bucket never had
versioning enabled either.

Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>

CHANGELOG.md

@@ -89,6 +89,26 @@ and this project adheres to
   `COLLABORATION_SERVER_ORIGIN` now gates the http routes as well as the
   websocket
 
+- ⬆️(collaboration) upgrade yhub to 0.9.0 and delete superseded document blobs
+  for real. Its S3 persistence plugin now records the object version it wrote
+  and names that version when it deletes it. On a versioned bucket — which is
+  what a deployment runs — a delete that names no version deletes nothing: it
+  writes a delete marker and keeps every version underneath. Each compaction
+  supersedes the blobs of the one before, so what was kept was every version of
+  every document ever written, a document someone asked to erase included, still
+  readable by anyone who can list versions. Blobs are written to the bucket for
+  every branch of a document.
+
+  `YHUB_S3_PERSISTENCE` now governs only whether new blobs are *written* to the
+  bucket. The plugin itself is attached whenever the `YHUB_S3_*` settings name
+  one, on or off, so that the objects an earlier run wrote stay readable —
+  turning the toggle off used to strand them, since a row pointing at an object
+  is unreadable without the plugin and yhub reports such a version as having no
+  content rather than as an error. The settings, not the toggle, are what a
+  deployment whose bucket holds anything must keep. The dev stack keeps the
+  toggle off and now creates its bucket versioned, so flipping it on exercises
+  what a deployment runs rather than a simpler case
+
 ### Fixed
 
 - 🐛(frontend) stop reconnecting to the collaboration server when it has refused

UPGRADE.md

@@ -228,9 +228,10 @@ upgrade, in the order they are done, and end with the API changes.
   its own PostgreSQL database (`YHUB_S3_PERSISTENCE=true`, plus the
   `YHUB_S3_*` settings). It is off by default and nothing about this upgrade
   needs it. Read the "Document storage" section of `src/yhub-server/README.md`
-  before enabling it: a document persisted that way cannot be read back once
-  the setting is removed, and it is a third bucket, not the backend's
-  `AWS_S3_*` nor the legacy one the migration reads.
+  before enabling it: the `YHUB_S3_*` settings attach the bucket whether or not
+  the toggle is on and have to stay in place for as long as it holds anything,
+  and it is a third bucket, not the backend's `AWS_S3_*` nor the legacy one the
+  migration reads.
 
 - The endpoint `/api/v1.0/documents/{document_id}/content/`, added in 5.0.0, is
   removed, both its `GET` and its `PATCH`. The content of a document is now

compose.yml

@@ -49,11 +49,23 @@ services:
       minio:
         condition: service_healthy
         restart: true
+    # The yhub bucket is made here rather than left to the collaboration
+    # server, which creates the bucket it does not find with no versioning on
+    # it. Versioning is what a deployment runs, and it is the case where
+    # deleting an object needs the version named to delete anything at all
+    # (YHUB_S3_PERSISTENCE, src/yhub-server/README.md).
+    #
+    # The lines below end on `&&` and not on a backslash: yaml folds this block
+    # by joining its lines with a space, so a trailing `\` reaches the shell as
+    # an escaped space glued to the word after it — ` /usr/bin/mc`, which is no
+    # command, and everything past the first `&&` silently did nothing.
     entrypoint: >
       sh -c "
-      /usr/bin/mc alias set impress http://minio:9000 impress password && \
-      /usr/bin/mc mb impress/impress-media-storage && \
-      /usr/bin/mc version enable impress/impress-media-storage && \
+      /usr/bin/mc alias set impress http://minio:9000 impress password &&
+      /usr/bin/mc mb --ignore-existing impress/impress-media-storage &&
+      /usr/bin/mc version enable impress/impress-media-storage &&
+      /usr/bin/mc mb --ignore-existing impress/yhub-storage &&
+      /usr/bin/mc version enable impress/yhub-storage &&
       exit 0;"
 
   app-dev:
@@ -261,7 +273,12 @@ services:
         condition: service_healthy
       # soft migration reads the legacy document store at startup traffic —
       # starting before minio would cache 401s for the first accessed docs —
-      # and YHUB_S3_PERSISTENCE, when it is on, checks its own bucket at boot
+      # and YHUB_S3_PERSISTENCE checks its own bucket at boot. Waiting for the
+      # buckets to be *made*, not merely for the job to have started: the
+      # collaboration server creates the bucket it does not find, and the one
+      # it creates has no versioning
+      createbuckets:
+        condition: service_completed_successfully
       minio:
         condition: service_healthy
 

documentation/env.md

@@ -193,11 +193,11 @@ documents what each of them changes.
 | LEGACY_S3_BUCKET_NAME         | Name of the legacy media bucket                                                                                                                          | impress-media-storage  |
 | LEGACY_S3_REGION_NAME         | Region of that bucket, when its provider needs one                                                                                                       | us-east-1              |
 | LEGACY_S3_SIGNATURE_VERSION   | How the calls to that bucket are signed, s3v4 or v4                                                                                                      | s3v4                   |
-| YHUB_S3_PERSISTENCE           | Set to "true" to store the document blobs in a bucket instead of the yhub database. Read the "Document storage" section of `src/yhub-server/README.md` first: it cannot be turned back off | false |
-| YHUB_S3_ENDPOINT_URL          | Required by YHUB_S3_PERSISTENCE, endpoint of that bucket, without a path                                                                                 |                        |
-| YHUB_S3_ACCESS_KEY_ID         | Required by YHUB_S3_PERSISTENCE, read/write/delete access to that bucket (or YHUB_S3_ACCESS_KEY_ID_FILE)                                                 |                        |
-| YHUB_S3_SECRET_ACCESS_KEY     | Required by YHUB_S3_PERSISTENCE, secret of the key above (or YHUB_S3_SECRET_ACCESS_KEY_FILE)                                                             |                        |
-| YHUB_S3_BUCKET_NAME           | Required by YHUB_S3_PERSISTENCE, name of that bucket, created on startup when missing                                                                    |                        |
+| YHUB_S3_PERSISTENCE           | Set to "true" to write new document blobs to a bucket instead of the yhub database. The YHUB_S3_* settings below attach the bucket whether or not this is on, and must stay in place for as long as it holds anything — see the "Document storage" section of `src/yhub-server/README.md` | false |
+| YHUB_S3_ENDPOINT_URL          | Endpoint of the bucket the blobs live in, without a path. Required as a set with the three below                                                                                 |                        |
+| YHUB_S3_ACCESS_KEY_ID         | Read/write/delete access to that bucket, deleting object versions included (or YHUB_S3_ACCESS_KEY_ID_FILE)                                                 |                        |
+| YHUB_S3_SECRET_ACCESS_KEY     | Secret of the key above (or YHUB_S3_SECRET_ACCESS_KEY_FILE)                                                             |                        |
+| YHUB_S3_BUCKET_NAME           | Name of that bucket, created on startup when missing                                                                    |                        |
 | YHUB_S3_REGION_NAME           | Region of that bucket, when its provider needs one                                                                                                       |                        |
 
 ## impress-y-provider container

env.d/development/yhub

@@ -34,11 +34,15 @@ LEGACY_S3_ACCESS_KEY_ID=impress
 LEGACY_S3_SECRET_ACCESS_KEY=password
 
 # Document storage: where the blobs of a compaction are written. Off, they stay
-# in yhub's postgres, which is what this stack runs. Turning it on stores them
-# in object storage instead — here the same minio, in a bucket of its own that
-# the server creates on startup when it is missing. Read the "Document storage"
-# section of src/yhub-server/README.md first: a document persisted this way
-# cannot be read back with the plugin turned off again.
+# in yhub's postgres, which is what this stack runs. On, they go to object
+# storage instead — here the same minio, in a bucket of its own, created
+# versioned by compose so that flipping this to true exercises what a
+# deployment does rather than a simpler case.
+#
+# The settings below are read whether or not the toggle is on: they are what
+# attaches the S3 plugin, and the plugin is what can read back the objects a
+# previous run wrote. Only the writing follows the toggle. See the "Document
+# storage" section of src/yhub-server/README.md.
 YHUB_S3_PERSISTENCE=false
 YHUB_S3_ENDPOINT_URL=http://minio:9000
 YHUB_S3_ACCESS_KEY_ID=impress

src/helm/impress/README.md

@@ -384,7 +384,7 @@
 | `yhub.envVars.LEGACY_S3_REGION_NAME`               | Region of the legacy bucket, when its provider needs one                                                                                                |                           |
 | `yhub.envVars.LEGACY_S3_BUCKET_NAME`               | Name of the legacy Django media bucket (default: impress-media-storage)                                                                                 |                           |
 | `yhub.envVars.LEGACY_S3_SIGNATURE_VERSION`         | How the calls to the legacy bucket are signed, s3v4 or v4 (default: s3v4)                                                                               |                           |
-| `yhub.envVars.YHUB_S3_PERSISTENCE`                 | Set to "true" to store the document blobs in a bucket instead of the yhub database — read src/yhub-server/README.md first, it cannot be turned back off |                           |
+| `yhub.envVars.YHUB_S3_PERSISTENCE`                 | Set to "true" to write new document blobs to a bucket instead of the yhub database — the YHUB_S3_* settings attach the bucket whether or not this is on, read src/yhub-server/README.md first |                           |
 | `yhub.envVars.YHUB_S3_ENDPOINT_URL`                | Required by YHUB_S3_PERSISTENCE, endpoint of the bucket the blobs are stored in, without a path (e.g. https://s3.example.com)                           |                           |
 | `yhub.envVars.YHUB_S3_ACCESS_KEY_ID`               | Required by YHUB_S3_PERSISTENCE, read/write/delete access to that bucket (or YHUB_S3_ACCESS_KEY_ID_FILE)                                                |                           |
 | `yhub.envVars.YHUB_S3_SECRET_ACCESS_KEY`           | Required by YHUB_S3_PERSISTENCE, secret of the key above (or YHUB_S3_SECRET_ACCESS_KEY_FILE)                                                            |                           |

src/helm/impress/values.yaml

@@ -934,10 +934,11 @@ jwtKeys:
 ## - `LEGACY_S3_*`, turned on by `SOFT_MIGRATION`, is the legacy Django media
 ##   bucket it reads old documents *out of*. Not the backend's `AWS_S3_*`,
 ##   which names the same bucket for the backend's own use,
-## - `YHUB_S3_*`, turned on by `YHUB_S3_PERSISTENCE`, is a bucket of its own it
-##   stores the document blobs *into*, instead of its PostgreSQL database.
-##   Read `src/yhub-server/README.md` before enabling it: a document persisted
-##   this way cannot be read back once the setting is removed.
+## - `YHUB_S3_*` is a bucket of its own it stores the document blobs *into*,
+##   instead of its PostgreSQL database, when `YHUB_S3_PERSISTENCE` asks for
+##   it. Read `src/yhub-server/README.md` before enabling it: those settings
+##   attach the bucket whether or not the toggle is on, and dropping them from
+##   a deployment whose bucket holds anything makes those documents unreadable.
 yhub:
   ## @param yhub.enabled Enable the yhub collaboration server, its service and its init-db job
   enabled: true
@@ -1052,7 +1053,7 @@ yhub:
   ## @extra yhub.envVars.LEGACY_S3_REGION_NAME Region of the legacy bucket, when its provider needs one
   ## @extra yhub.envVars.LEGACY_S3_BUCKET_NAME Name of the legacy Django media bucket (default: impress-media-storage)
   ## @extra yhub.envVars.LEGACY_S3_SIGNATURE_VERSION How the calls to the legacy bucket are signed, s3v4 or v4 (default: s3v4)
-  ## @extra yhub.envVars.YHUB_S3_PERSISTENCE Set to "true" to store the document blobs in a bucket instead of the yhub database — read src/yhub-server/README.md first, it cannot be turned back off
+  ## @extra yhub.envVars.YHUB_S3_PERSISTENCE Set to "true" to write new document blobs to a bucket instead of the yhub database — the YHUB_S3_* settings attach the bucket whether or not this is on, read src/yhub-server/README.md first
   ## @extra yhub.envVars.YHUB_S3_ENDPOINT_URL Required by YHUB_S3_PERSISTENCE, endpoint of the bucket the blobs are stored in, without a path (e.g. https://s3.example.com)
   ## @extra yhub.envVars.YHUB_S3_ACCESS_KEY_ID Required by YHUB_S3_PERSISTENCE, read/write/delete access to that bucket (or YHUB_S3_ACCESS_KEY_ID_FILE)
   ## @extra yhub.envVars.YHUB_S3_SECRET_ACCESS_KEY Required by YHUB_S3_PERSISTENCE, secret of the key above (or YHUB_S3_SECRET_ACCESS_KEY_FILE)

src/yhub-server/README.md

@@ -290,27 +290,38 @@ content map and the content ids. By default they are `bytea` columns — the
 whole corpus lives on the database disk, which is the configuration Docs has
 been running and what this server does when nothing below is set.
 
-`YHUB_S3_PERSISTENCE=true` plugs yhub's own S3 persistence plugin
-(`S3PersistenceV1`, shipped with `@y/hub`) into the chain it consults before
-writing a blob and before reading one back. The blobs then go to a bucket and
-the row keeps a reference to them, `<column>_is_reference` saying which of the
-four it is: postgres holds the index of the documents, the bucket holds their
-bytes.
+Naming a bucket plugs yhub's own S3 persistence plugin (`S3PersistenceV1`,
+shipped with `@y/hub`) into the chain it consults before writing a blob and
+before reading one back. `YHUB_S3_PERSISTENCE=true` then sends the blobs to that
+bucket, the row keeping a reference to them and `<column>_is_reference` saying
+which of the four it is: postgres holds the index of the documents, the bucket
+holds their bytes.
+
+The two are deliberately separate. **The plugin is attached whenever the bucket
+is configured, on or off**, because reading is the half that must never be taken
+away: the objects an earlier run wrote are the only copy of those versions, and
+a row pointing at one is unreadable without the plugin that wrote it. Turning
+the toggle off stops the writing — new blobs go back to postgres — and leaves
+the reading alone. Keep the settings in place for as long as the bucket holds
+anything.
 
 | Variable | Required | What it is |
 | -------- | -------- | ---------- |
-| `YHUB_S3_PERSISTENCE` | — | `true` to store the blobs in a bucket (default: postgres) |
+| `YHUB_S3_PERSISTENCE` | — | `true` to write new blobs to the bucket (default: postgres) |
 | `YHUB_S3_ENDPOINT_URL` | yes | Endpoint of that bucket, without a path (e.g. `https://s3.example.com`) |
 | `YHUB_S3_ACCESS_KEY_ID` | yes | Key with read, write and delete on the bucket (or `…_FILE`) |
 | `YHUB_S3_SECRET_ACCESS_KEY` | yes | Secret of that key (or `…_FILE`) |
 | `YHUB_S3_BUCKET_NAME` | yes | Name of the bucket. No default: a typo would create one |
 | `YHUB_S3_REGION_NAME` | no | Region, when the provider needs one told rather than discovered |
 
-"Required" means required *when the plugin is on*: it is a startup error naming
-what is missing, rather than a client that ends up anonymous and only says so
-on the first compaction — which is a background task, so the failure would show
-up as documents quietly not being persisted. The bucket in use is logged next
-to the role (`"s3Bucket":"yhub-storage"`, `null` for postgres).
+"Required" means required *as a set*: name one of them and the rest are a
+startup error naming what is missing, rather than a client that ends up
+anonymous or against the wrong host and only says so on the first compaction —
+which is a background task, so the failure would show up as documents quietly
+not being persisted. Naming none of them, with the toggle off, is the one
+configuration with no plugin at all: postgres alone, and no object anywhere that
+would need reading back. The bucket is logged next to the role
+(`"s3Bucket":"yhub-storage","s3Writes":false` — a bucket that is only read).
 
 This is a **third** bucket, and it is deliberately configured apart from the
 other two: the backend's media bucket (`AWS_S3_*`, Django's own settings) and
@@ -320,27 +331,39 @@ the process it belongs to.
 
 A few things worth knowing before turning it on:
 
-- **It cannot be turned back off.** A row pointing at an object is unreadable
-  without the plugin that wrote it, and yhub reports such a version as having
-  no content rather than as an error — so a document compacted while the plugin
-  was on comes back *empty* once it is off, silently. Turning it on is safe in
-  the other direction: rows written before keep their bytes inline and are
-  served exactly as they were,
+- **the settings are what must not be dropped, not the toggle.** A row pointing
+  at an object is unreadable without the plugin, and yhub reports such a version
+  as having no content rather than as an error — so removing the `YHUB_S3_*`
+  settings from a deployment whose bucket holds anything makes those documents
+  come back *empty*, silently. `YHUB_S3_PERSISTENCE=false` is the safe way to
+  stop using the bucket, and it goes both ways: rows written while it was off
+  keep their bytes inline and are served exactly as they were,
 - **the bucket is created at startup** when it does not exist, so the
   credentials need `HeadBucket` and, the first time, `CreateBucket`. It is
-  checked on every boot, which is also what makes a wrong endpoint or a wrong
-  key fail loudly and immediately,
+  checked on every boot — including a boot with the toggle off, which is what
+  makes a wrong endpoint or a wrong key fail loudly and immediately rather than
+  on the first document that needs reading back,
 - **both halves need it.** The worker writes the blobs and the server reads
   them back, so a split deployment (`YHUB_ROLE`) configures the bucket on both
   — in the helm chart the worker inherits `yhub.envVars`, so there is nothing
   to repeat,
-- **only the `main` branch is offloaded.** The plugin declines everything else
-  and those blobs stay in postgres, which is yhub's behaviour, not a setting,
+- **every branch is offloaded.** The bucket is where the blobs of a document
+  belong whatever branch they were written on. This is the one thing the toggle
+  moves: `branches` is every branch when it is on and none when it is off, a
+  plugin that goes on retrieving and deleting what is in the bucket and adds
+  nothing to it,
 - **objects are deleted late.** When a version's row is dropped (pruning, a
   reset, a hard deletion), the object is removed about ten seconds later, so
   that readers holding the reference are not left with a 404. A delete that
   fails is logged and forgotten: the bucket may accumulate objects no row names
   anymore, and nothing collects them,
+- **on a versioned bucket, the version is what gets deleted.** A plain delete
+  there deletes nothing — it writes a delete marker over the object and keeps
+  every version underneath it, so the blobs of every compaction ever made would
+  stay, and so would a document someone asked to erase. The plugin records the
+  version id it wrote and the delete names it (`deleteVersions`, on), which
+  removes the bytes for real — on AWS that is `s3:DeleteObjectVersion`, which a
+  policy granting `s3:DeleteObject` alone does not cover,
 - the objects are Yjs blobs keyed by
   `id:ydoc:v1/{org}/{docid}/{branch}/{gc}/{clock}` (and `id:contentmap:v1/…`,
   `id:contentids:v1/…`) — one object per version and per column, not one file
@@ -349,9 +372,13 @@ A few things worth knowing before turning it on:
 
 In the dev stack the variables are in `env.d/development/yhub`, pointing at the
 same minio the rest of the stack uses with a bucket of its own
-(`yhub-storage`), and the toggle is off. Flipping it to `true` and restarting
-the service is enough to exercise the path — on a dev database, where losing
-the documents already compacted costs nothing.
+(`yhub-storage`), and the toggle is off — the plugin is attached and reads that
+bucket, the compactions go to postgres. Flipping `YHUB_S3_PERSISTENCE` to true
+and restarting the service is enough to exercise the writing path. The bucket is
+made by the `createbuckets` job of `compose.yml` rather than by this server, and
+made *versioned*, so what is exercised is what a deployment runs rather than a
+simpler case. Watch it with `mc ls --versions --recursive impress/yhub-storage`
+from an `mc` container on the stack's network.
 
 ## Container image
 

src/yhub-server/package-lock.json

@@ -7,7 +7,7 @@
       "name": "yhub-server",
       "dependencies": {
         "@aws-sdk/client-s3": "3.1110.0",
-        "@y/hub": "0.8.2",
+        "@y/hub": "0.9.0",
         "@y/y": "14.0.0-rc.24",
         "jose": "6.2.8"
       },
@@ -504,9 +504,9 @@
       "license": "ISC"
     },
     "node_modules/@y/hub": {
-      "version": "0.8.2",
-      "resolved": "https://registry.npmjs.org/@y/hub/-/hub-0.8.2.tgz",
-      "integrity": "sha512-JMwHMwmnUAp6N+qz2MKaIRp/eDGOYIqGTRnW+aNyFg4nypMnnsjPzx7rs6/uG4RR1IDEKiqr4A5Khta44Q2NPQ==",
+      "version": "0.9.0",
+      "resolved": "https://registry.npmjs.org/@y/hub/-/hub-0.9.0.tgz",
+      "integrity": "sha512-xGzLoPpJ1lrZo++ULu0IAs3bnLpjCNhsEgHcRiIf+XaTO5iu7GjwnbO6HGqI4LiP45Dn952kkh+WrmhK8qkfaQ==",
       "license": "AGPL-3.0 OR PROPRIETARY",
       "dependencies": {
         "@y-crdt/yn": "^0.1.4",

src/yhub-server/package.json

@@ -10,7 +10,7 @@
   },
   "dependencies": {
     "@aws-sdk/client-s3": "3.1110.0",
-    "@y/hub": "0.8.2",
+    "@y/hub": "0.9.0",
     "@y/y": "14.0.0-rc.24",
     "jose": "6.2.8"
   },

src/yhub-server/server.js

@@ -106,10 +106,14 @@ const TASK_CONCURRENCY = intEnv('YHUB_TASK_CONCURRENCY', 5, 1);
 // store. Off by default, which is postgres alone, the way Docs has been
 // running.
 //
-// Not a switch that can be flipped back: a row pointing at an object is
-// unreadable without the plugin that wrote it — yhub reports that version as
-// having no content rather than as an error — so turning it off after a
-// compaction strands what was stored while it was on. See README.md.
+// This decides where *new* blobs are written, and nothing else. The plugin
+// itself is loaded whenever the bucket below is configured, on or off, because
+// a row pointing at an object is unreadable without the plugin that wrote it —
+// yhub reports such a version as having no content rather than as an error, so
+// a deployment that has ever had this on and drops the plugin loses those
+// documents silently. Keep the YHUB_S3_* settings in place for as long as the
+// bucket holds anything; turning this off then stops the writing and leaves the
+// reading alone. See README.md.
 const S3_PERSISTENCE = process.env.YHUB_S3_PERSISTENCE === 'true';
 // Its own bucket, named apart from the backend's `AWS_S3_*` and from the legacy
 // document store's `LEGACY_S3_*` (migration.js): three buckets that may sit on
@@ -928,7 +932,13 @@ const workerEvents = {
 
 // The persistence plugins yhub consults, in order, before writing a blob to
 // postgres and before reading one back. An empty list keeps everything in the
-// database, which is the default.
+// database, which is what a deployment that names no bucket gets.
+//
+// Naming a bucket is enough to get the plugin, whether or not the toggle asks
+// for the writing: reading is the half that must never be taken away, since the
+// objects an earlier run wrote are the only copy of those versions. `branches`
+// is what the toggle actually moves — every branch, or none, which is a plugin
+// that retrieves and deletes what is in the bucket and adds nothing to it.
 //
 // Read here rather than in the call below so that an incomplete configuration
 // is a startup error naming what is missing: the client would otherwise be
@@ -936,18 +946,25 @@ const workerEvents = {
 // compaction, which is a background task — the failure would show up as
 // documents quietly not being persisted.
 const persistencePlugins = () => {
-  if (!S3_PERSISTENCE) return [];
-
-  const missing = [
+  const settings = [
     ['YHUB_S3_ENDPOINT_URL', YHUB_S3_ENDPOINT_URL],
     ['YHUB_S3_ACCESS_KEY_ID', YHUB_S3_ACCESS_KEY_ID],
     ['YHUB_S3_SECRET_ACCESS_KEY', YHUB_S3_SECRET_ACCESS_KEY],
     ['YHUB_S3_BUCKET_NAME', YHUB_S3_BUCKET_NAME],
-  ]
-    .filter(([, value]) => !value)
-    .map(([name]) => name);
+  ];
+  const missing = settings.filter(([, value]) => !value).map(([name]) => name);
+  // no bucket named at all, and the toggle does not ask for one: postgres
+  // alone, and no object anywhere that would need reading back
+  if (missing.length === settings.length && !S3_PERSISTENCE) return [];
   if (missing.length > 0) {
-    throw new Error(`YHUB_S3_PERSISTENCE=true requires ${missing.join(', ')}`);
+    // half a configuration is always a mistake, and the half that is set says
+    // which mistake: a bucket was meant to be reachable and is not
+    const named = missing.join(', ');
+    throw new Error(
+      S3_PERSISTENCE
+        ? `YHUB_S3_PERSISTENCE=true requires ${named}`
+        : `The YHUB_S3_* bucket is partly configured, missing ${named}`,
+    );
   }
 
   const url = new URL(YHUB_S3_ENDPOINT_URL);
@@ -976,6 +993,22 @@ const persistencePlugins = () => {
       // left out rather than passed empty: unset, the client discovers the
       // region of the bucket instead of validating an empty string
       ...(YHUB_S3_REGION_NAME ? { region: YHUB_S3_REGION_NAME } : {}),
+      // The branches whose blobs are written here: all of them, or none, which
+      // is how the plugin is kept for reading while the writing goes back to
+      // postgres. `store` declines a branch it is not given and yhub falls
+      // through to the database, while `retrieve` and `delete` go on answering
+      // for every object already in the bucket.
+      branches: S3_PERSISTENCE ? true : [],
+      // On a versioned bucket a plain delete deletes nothing: it writes a
+      // delete marker over the object and keeps every version underneath it.
+      // Each compaction supersedes the blobs of the one before, so that would
+      // be every version ever written kept forever — and a document a user
+      // asked to erase still readable by anyone who can list versions. The
+      // plugin records the version id it wrote at store time so that the
+      // delete can name it, which is what removes the bytes. Asked for here
+      // rather than left to the plugin's default: the delete is only as
+      // thorough as this line.
+      deleteVersions: true,
     }),
   ];
 };
@@ -1028,8 +1061,11 @@ logger.info(
     server: RUNS_SERVER,
     worker: RUNS_WORKER,
     taskConcurrency: RUNS_WORKER ? TASK_CONCURRENCY : null,
-    // where the compaction blobs go — null is yhub's own postgres
-    s3Bucket: S3_PERSISTENCE ? YHUB_S3_BUCKET_NAME : null,
+    // the bucket the plugin is attached to, null when there is no plugin at
+    // all, and whether the compaction blobs are written to it or to yhub's own
+    // postgres — a bucket with `s3Writes` false is one that is only read
+    s3Bucket: YHUB_S3_BUCKET_NAME ?? null,
+    s3Writes: S3_PERSISTENCE,
     taskDebounceMs: yhub.stream.taskDebounce,
     minMessageLifetimeMs: yhub.stream.minMessageLifetime,
   },