✅(collaboration) test the legacy migrations against a real yhub

Cover both paths off the legacy Django store end to end: the lazy seed on
first access, and the migrate endpoint replaying every S3 version. The tests
need no database — the admin JWT short-circuits document authorization, so a
fixture is an S3 object on a random uuid — and read the timeline through
yhub 0.5.0's `Accept: application/json`, which spares python a lib0 decoder.
CI grows a valkey service and starts a collaboration server alongside the
backend test job; the tests skip themselves when nothing answers on the new
COLLABORATION_API_URL setting, so `make test` without the dev stack still
passes.

Writing them turned up three things worth fixing in the server.

Backend reads now seed too. getAccessType short-circuited on the admin token
before reaching the legacy store, so a server-side read of an unmigrated
document answered with an empty one, and a create-ydoc against it would have
written a second lineage beside the content the first user access was about
to seed in.

Seeding no longer decides access; the backend's answer alone does. A legacy
object that cannot be migrated — it does not decode, or it exceeds the size
we load — opens as a new document instead of denying, since no retry can fix
it and refusing would leave the document unopenable by anyone. The cause is
logged once per attempt with the bucket, key and stack, and every later access
logs that it admitted a caller without migrating.

That made the failure classifier dangerous, so it is inverted. It was an
allowlist of retryable errors — eight socket errnos — which left every way S3
can refuse (AccessDenied on a rotated key, NoSuchBucket, a region redirect)
counting as "this object is unusable". Denying, that was survivable; opening
empty, one misscoped credential would fork every document touched during the
window. Now only a failure raised while interpreting bytes we already hold is
permanent, marked at the throw site, and everything else answers a retryable
503. Guessing wrong that way costs a retry; the other way costs the document.

The admin seed is also fenced to the org and to main, like the user path
above it. The legacy store is branchless — {docid}/file is main — and the
bookkeeping is per document, so seeding ?branch=draft would have written
main's content into an orphan room and left the real one permanently empty.

Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>

src/yhub-server/README.md

@@ -59,7 +59,9 @@ bucket, as UTF-8 text that is the base64 encoding of a raw Yjs update, at key
 `{document-uuid}/file`. With `SOFT_MIGRATION=true`, this server migrates those
 documents into yhub lazily, on first access:
 
-1. After a user's document authorization succeeds, the auth plugin checks
+1. After a caller's document authorization succeeds — a user's, or the
+   backend's own admin JWT, so a server-side read never sees an *empty*
+   document where legacy content exists — the auth plugin checks
    whether yhub already has content for the room — the migrated set written by
    the full migration (below), then a bare postgres `SELECT` (persisted rows),
    then the valkey stream (uncompacted `ydoc:update:v1` messages), then the
@@ -94,16 +96,35 @@ Guarantees and failure behavior:
 - **Missing S3 object is not an error** — that is the brand-new-document case
   (Django writes no object until the first content save); the room simply
   starts empty.
-- **Everything else fails closed**, and says whether it is worth retrying
-  (yhub 0.5.0 error semantics): an oversized or corrupt legacy object will fail
-  the same way forever, so the connection is denied `403`; network errors,
-  timeouts and momentary seed backpressure answer `503`, which clients retry
-  with backoff. Either way a `soft-migration` error is logged. A cached failure
-  verdict prevents retry storms from hammering S3 — permanent failures
-  (corrupt/oversized objects) for 5 minutes, transient ones (network errors,
-  timeouts) for 15 seconds, and per-replica seed backpressure (more than 20
-  concurrent seeds) is denied without caching so the client's next retry goes
-  through.
+- **Seeding never decides access** — the backend's answer does. What a failure
+  changes is only what the room contains, and the two kinds are treated
+  differently (yhub 0.5.0 error semantics):
+  - **The legacy object cannot be migrated** — it does not decode, or it
+    exceeds the size we will load. Retrying cannot change that, and nobody can
+    repair the object from the outside, so refusing would make the document
+    permanently unopenable. It opens as a *new* document instead. The cause is
+    logged once per attempt (`seed.failed`, with the bucket, key and stack) and
+    every subsequent access logs a `seed.skipped` warning, because the caller
+    is now editing beside legacy content that stayed behind in S3.
+  - **Everything else** — network error, timeout, seed backpressure, and every
+    way S3 can refuse (`AccessDenied` on a rotated key, `NoSuchBucket` on a
+    misconfigured name, a region redirect). The same request later may well
+    succeed, so it answers `503` and clients retry with backoff.
+
+  The split is deliberately asymmetric: only a failure raised while
+  *interpreting bytes we already hold* counts as permanent, and it is marked as
+  such at the throw site. Everything else is retryable by default. An allowlist
+  of retryable errors would have to enumerate every way the store can say no,
+  and each case it missed would be read as "this document has no content" and
+  open the room empty over content that is alive in S3 — one misscoped
+  credential would fork the corpus. Guessing wrong this way costs a retry;
+  guessing wrong the other way costs the document.
+
+  A cached failure verdict prevents retry storms from hammering S3 — permanent
+  failures (corrupt/oversized objects) for 5 minutes, transient ones (network
+  errors, timeouts) for 15 seconds, and per-replica seed backpressure (more
+  than 20 concurrent seeds) is not cached at all, so the client's next retry
+  goes through.
 - **Seeding is idempotent**: the legacy S3 snapshots are frozen (the
   frontend no longer PATCHes content snapshots to Django) and share one Yjs
   lineage with everything in yhub, so duplicate or concurrent seeds merge as

src/yhub-server/migration.js

@@ -91,7 +91,9 @@ const s3 = SOFT_MIGRATION
       });
     })()
   : null;
-const migrationLog = logger.child({ module: 'soft-migration' });
+// exported so the auth path can report, under the same module name, that it
+// admitted a caller to a document it could not migrate
+export const migrationLog = logger.child({ module: 'soft-migration' });
 
 // Both keys are derived from the prefix yhub itself resolved, so they cannot
 // drift from the room keys, and both sit outside its scanned `:room:*` pattern.
@@ -118,10 +120,10 @@ const fetchLegacyDoc = async (docid, versionId = null) => {
   // stream once reading, so a stalled transfer cannot hold the ws upgrade
   const timeout = new Promise((_, reject) => {
     const timer = setTimeout(() => {
+      // unmarked, so it counts as retryable: a slow S3 may recover
       const err = new Error(
         `s3 fetch timed out after ${S3_FETCH_TIMEOUT_MS}ms`,
       );
-      err.transient = true; // a slow S3 may recover — cache the failure briefly
       stream?.destroy(err);
       reject(err);
     }, S3_FETCH_TIMEOUT_MS);
@@ -159,11 +161,11 @@ const fetchLegacyDoc = async (docid, versionId = null) => {
         stream.on('data', (chunk) => {
           received += chunk.byteLength;
           if (received > MAX_LEGACY_B64_BYTES) {
-            stream.destroy(
-              new Error(
-                `legacy object exceeds the ${MAX_LEGACY_B64_BYTES}B cap`,
-              ),
+            const err = new Error(
+              `legacy object exceeds the ${MAX_LEGACY_B64_BYTES}B cap`,
             );
+            err.permanent = true; // the object will be this big next time too
+            stream.destroy(err);
             return;
           }
           chunks.push(chunk);
@@ -175,9 +177,11 @@ const fetchLegacyDoc = async (docid, versionId = null) => {
     ]);
     const decoded = Buffer.from(body.toString('utf8'), 'base64');
     if (decoded.byteLength > MAX_LEGACY_BYTES) {
-      throw new Error(
+      const err = new Error(
         `decoded legacy update (${decoded.byteLength}B) exceeds the ${MAX_LEGACY_BYTES}B cap`,
       );
+      err.permanent = true; // the object will be this big next time too
+      throw err;
     }
     // compute-task schema requires an exact Uint8Array (lib0 compares the
     // constructor) — re-view the Buffer without copying
@@ -205,7 +209,6 @@ const listLegacyVersions = async (docid) => {
       const err = new Error(
         `s3 version listing timed out after ${S3_LIST_TIMEOUT_MS}ms`,
       );
-      err.transient = true;
       stream.destroy(err);
     }, S3_LIST_TIMEOUT_MS);
     stream.on('data', (obj) => {
@@ -265,26 +268,17 @@ const VERDICT_TTL_MS = { exists: 600000, empty: 60000, failed: 300000 };
 // transient failures (network blips, timeouts, S3 restarting) are cached just
 // long enough to blunt a retry storm without turning a hiccup into a lockout
 const TRANSIENT_TTL_MS = 15000;
-const TRANSIENT_CODES = new Set([
-  'ECONNREFUSED',
-  'ECONNRESET',
-  'ETIMEDOUT',
-  'EHOSTUNREACH',
-  'ENETUNREACH',
-  'ENOTFOUND',
-  'EAI_AGAIN',
-  'EPIPE',
-]);
-// Will this failure plausibly resolve on its own? Callers use it twice: to pick
-// the verdict TTL below, and — in server.js — to decide whether a denied
-// connection reports a permanent 403 or a retryable 503. `noCache` is the
-// per-replica seed backpressure, transient by construction (a slot frees up
-// within seconds).
-export const isTransientFailure = (err) =>
-  err?.transient === true ||
-  err?.noCache === true ||
-  TRANSIENT_CODES.has(err?.code) ||
-  TRANSIENT_CODES.has(err?.cause?.code);
+// Is this legacy object beyond saving, as opposed to merely out of reach right
+// now? Only a failure raised while *interpreting* bytes we already hold
+// qualifies: the object does not decode, or it is larger than we will load.
+// Those are marked at the throw site, and nothing else counts — an allowlist
+// of retryable errors would have to enumerate every way S3 can say no
+// (AccessDenied on a rotated key, NoSuchBucket on a misconfigured name, a
+// region redirect), and each one it missed would be read as "this document has
+// no content" and open the room empty over content that is alive in S3.
+// Guessing wrong in this direction costs a retry; guessing wrong in the other
+// costs the document.
+export const isPermanentFailure = (err) => err?.permanent === true;
 const VERDICT_CACHE_MAX = 50000;
 const verdicts = new Map(); // docid -> { verdict, error, expires }
 const rememberVerdict = (
@@ -358,6 +352,16 @@ const migrate = async (yhub, room) => {
         );
         return 'empty';
       }
+      // Decode before writing anything: a legacy object that is not a valid
+      // Yjs update fails here, on this thread, and is the one failure we know
+      // no retry can fix — so it is marked as such.
+      let contentids;
+      try {
+        contentids = Y.createContentIdsFromUpdate(update);
+      } catch (err) {
+        err.permanent = true;
+        throw err;
+      }
       await yhub.stream.addMessage(room, {
         type: 'ydoc:update:v1',
         // Deliberately no insertAt/deleteAt. A lazy seed is not an editing
@@ -368,12 +372,9 @@ const migrate = async (yhub, room) => {
         // whichever the row order happened to put last. Content seeded this way
         // carries an author but no timestamp, so it produces no activity entry
         // until fullMigrate supplies the history.
-        //
-        // Reading the ids also validates the update: a corrupt legacy object
-        // throws here, on this thread, before anything reaches the stream.
         contentmap: Y.encodeContentMap(
           Y.createContentMapFromContentIds(
-            Y.createContentIdsFromUpdate(update),
+            contentids,
             [
               Y.createContentAttribute('insert', 'system'),
               Y.createContentAttribute('insert:migration', 's3'),
@@ -428,22 +429,31 @@ export const maybeMigrate = async (yhub, room) => {
       .then(
         (verdict) => rememberVerdict(room.docid, verdict),
         (err) => {
-          // logged here (once per attempt) rather than per denied connection:
-          // cached failures deny without new logs until the verdict expires
+          // The one place the *cause* is recorded, once per attempt rather
+          // than per access: a cached verdict re-raises this error without
+          // logging again until it expires.
+          const permanent = isPermanentFailure(err);
           migrationLog.error(
-            { event: 'seed.failed', err, docid: room.docid },
-            'soft migration failed; denying access',
+            {
+              event: 'seed.failed',
+              err,
+              docid: room.docid,
+              permanent,
+              bucket: AWS_STORAGE_BUCKET_NAME,
+              key: `${room.docid}/file`,
+            },
+            permanent
+              ? 'soft migration is not possible for this legacy object'
+              : 'soft migration failed; the caller is asked to retry',
           );
           if (err?.noCache !== true) {
-            // transient failures get a short TTL so a hiccup cannot lock a
-            // doc out for the full poison-object window
+            // a retryable failure is remembered only briefly, so a hiccup
+            // cannot lock a document out for the full poison-object window
             rememberVerdict(
               room.docid,
               'failed',
               err,
-              isTransientFailure(err)
-                ? TRANSIENT_TTL_MS
-                : VERDICT_TTL_MS.failed,
+              permanent ? VERDICT_TTL_MS.failed : TRANSIENT_TTL_MS,
             );
           }
           throw err;

src/yhub-server/server.js

@@ -13,8 +13,9 @@ import { secret } from './env.js';
 import {
   SOFT_MIGRATION,
   fullMigrate,
-  isTransientFailure,
+  isPermanentFailure,
   maybeMigrate,
+  migrationLog,
 } from './migration.js';
 
 const PORT = Number(process.env.PORT || 3002);
@@ -73,6 +74,39 @@ const backendFetch = async (path, { cookie, origin }) => {
   return res.json();
 };
 
+// First access to a room yhub does not know: seed it from the legacy Django S3
+// store before admitting the caller. Awaited inside the upgrade handler, so the
+// post-upgrade initial sync (which merges postgres and the stream from clock 0)
+// is guaranteed to include the seed.
+//
+// Seeding never decides whether the caller may read the document — that is the
+// backend's answer alone. There are two ways this ends other than a seed:
+//
+//   the legacy object cannot be migrated (it does not decode, or it is bigger
+//     than we will load) — retrying will not change that, so the room opens as
+//     a new document. Refusing instead would lock a document nobody can repair
+//     from the outside. Logged per access, because the caller is now editing
+//     alongside legacy content that stayed behind in S3.
+//   the legacy store could not be reached (timeout, network, backpressure) —
+//     the same request later may well succeed, so it answers 503 rather than
+//     silently starting an empty document on top of content that exists.
+const seedFromLegacyStore = async (room) => {
+  try {
+    // `yhub` is declared at the bottom of this file — safe: auth callbacks only
+    // fire once the server is up, i.e. after that assignment
+    await maybeMigrate(yhub, room);
+  } catch (err) {
+    if (!isPermanentFailure(err)) {
+      throw apiError(503, 'Legacy document store is unavailable');
+    }
+    // why it failed was logged once, at the attempt, inside maybeMigrate
+    migrationLog.warn(
+      { event: 'seed.skipped', docid: room.docid, err: err?.message },
+      'admitting caller to a document that could not be migrated; it opens as new',
+    );
+  }
+};
+
 const auth = createAuthPlugin({
   // uws req is only valid synchronously — read headers AND query before first await.
   async readAuthInfo(req) {
@@ -150,10 +184,35 @@ const auth = createAuthPlugin({
     }
   },
   async getAccessType(authInfo, { org, docid, branch }, purpose) {
-    if (authInfo.admin === true) return 'rw'; // Django's admin token: full access
+    if (authInfo.admin === true) {
+      // Django's admin token: full access. It still goes through the legacy
+      // seed, on the same terms as a user (default purpose only, so a
+      // `migrate` call is not seeded out from under fullMigrate). Without it a
+      // backend read of an unmigrated document would answer with an *empty*
+      // doc, and a create-ydoc against one would write a second lineage next
+      // to the legacy content the first user access is about to seed in.
+      // Access itself is never in question here — the token already granted it.
+      // The same org/branch fence the user path applies below. The admin token
+      // is the only identity that can name an arbitrary org or branch, and the
+      // legacy store is branchless — `{docid}/file` *is* main — so seeding any
+      // other room would write main's content into an orphan room, and the
+      // per-docid verdict cache would then report that docid as done and leave
+      // the real room empty.
+      if (
+        SOFT_MIGRATION &&
+        purpose == null &&
+        org === ORG &&
+        branch === 'main' &&
+        UUID4.test(docid)
+      ) {
+        await seedFromLegacyStore({ org, docid, branch });
+      }
+      return 'rw';
+    }
     // Regular users only get access for the default purpose — custom-endpoint
-    // purposes (reset-connections) are backend-internal. Loose != on purpose:
-    // ws upgrades and rechecks pass undefined, built-in rest endpoints null.
+    // purposes (reset-connections, migrate) are backend-internal. Loose != on
+    // purpose: ws upgrades and rechecks pass undefined, built-in rest
+    // endpoints null.
     if (
       org !== ORG ||
       branch !== 'main' ||
@@ -177,29 +236,10 @@ const auth = createAuthPlugin({
     if (!doc.abilities?.retrieve) {
       return null;
     }
+    // the backend has already decided the caller may read this document; the
+    // seed only decides what is in it
     if (SOFT_MIGRATION) {
-      // First access to a room yhub does not know: seed it from the legacy
-      // Django S3 store before admitting the connection. Awaited inside the
-      // upgrade handler, so the post-upgrade initial sync (which merges
-      // postgres and the stream from clock 0) is guaranteed to include the
-      // seed. Runs only for authorized readers. A missing S3 object is the
-      // brand-new-document case and allows an empty room; a real S3/compute
-      // failure denies access, either way already logged (once per attempt)
-      // inside maybeMigrate.
-      try {
-        // `yhub` is declared at the bottom of this file — safe: auth callbacks
-        // only fire once the server is up, i.e. after that assignment
-        await maybeMigrate(yhub, { org, docid, branch });
-      } catch (err) {
-        // A corrupt or oversized legacy object will fail the same way forever,
-        // so that denial is permanent (403). An S3 timeout, a network blip or
-        // momentary seed backpressure will not — 503 tells the caller to come
-        // back rather than to treat the document as unreadable.
-        if (isTransientFailure(err)) {
-          throw apiError(503, 'Legacy document store is unavailable');
-        }
-        return null;
-      }
+      await seedFromLegacyStore({ org, docid, branch });
     }
     return doc.abilities.update ? 'rw' : 'r';
   },