diff --git a/CHANGELOG.md b/CHANGELOG.md index c80783939..478b05a38 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,10 +11,26 @@ and this project adheres to - ✨(collaboration) soft-migrate legacy S3 documents into yhub on first access (`SOFT_MIGRATION=true`): when yhub does not know a document yet, its legacy snapshot (`{id}/file`, base64 Yjs update) is fetched from the Django S3 - media bucket and seeded server-side (attributed to `system`) before the - connection is admitted. A missing S3 object means a brand-new document and - yields an empty room; any real S3/compute failure fails closed (opaque 401, - the client retries with backoff). Enabled in the dev stack via compose.yml + media bucket and seeded server-side (attributed to `system`, with no + timestamp — a lazy seed is not an editing event, and stamping one would + collide with the real per-version times the migrate endpoint writes) before + the connection is admitted. A missing S3 object means a brand-new document and + yields an empty room; any real S3/compute failure fails closed, as a + permanent `403` for a corrupt or oversized object and a retryable `503` for + timeouts and network errors. Enabled in the dev stack via compose.yml +- ✨(collaboration) add a migrate endpoint on yhub: + `POST /collaboration/migrate/v1/docs/{id}` replays a document's **full** + legacy version history from the versioned S3 media bucket into a + `gc: false` Yjs document, crediting each S3 version with its own S3 + timestamp — so `activity?group=false` reports the same timeline as the + backend's `/documents/{id}/versions/`, instead of the single + migration-time change the + lazy soft migration leaves behind. Purely additive: the result is stored as + one new row at clock `0`, so nothing existing is deleted and the next + compaction merges it like any other row. Idempotent by construction (the + clock-`0` insert is `ON CONFLICT DO NOTHING`, and migrated ids are recorded + in a valkey set), lock-free, admin JWT only, and the intended way to backfill + the corpus before `SOFT_MIGRATION` is turned off - ✨(collaboration) add a create-ydoc endpoint on yhub: `POST /collaboration/create-ydoc/v1/docs/{id}` seeds a document's initial Yjs state from a raw binary update posted as `application/octet-stream`, @@ -31,11 +47,16 @@ and this project adheres to an admin token Django issued for another service (e.g. the `y-converter` one) cannot be replayed here; not yet triggered by the backend on permission changes (follow-up) -- ⬆️(collaboration) upgrade yhub to 0.4.0 and serve all its routes under the +- ⬆️(collaboration) upgrade yhub to 0.5.0 and serve all its routes under the `/collaboration/` prefix (`server.apiPrefix`): the websocket moves to `/collaboration/ws/v1/docs`. All `/collaboration/` routes are meant to be - publicly exposed except `reset-connections`, which stays backend-internal - (admin JWT only) + publicly exposed except `reset-connections` and `migrate`, which stay + backend-internal (admin JWT only). Following 0.5.0's error semantics + (`4xx` permanent, `5xx`/`429` retryable), the auth plugin now reports a + temporarily unreachable Django backend, JWKS endpoint or legacy S3 store as + `503` instead of denying access like a permission failure, so clients retry + instead of giving up. The built-in endpoints can also answer JSON on + `Accept: application/json` - ✨(backend) add a service generating cached RS256 JWT tokens - ✨(backend) publish the JWT public key on a JWKS endpoint - 🔧(dev) generate the JWT signing key when bootstrapping the dev stack diff --git a/src/yhub-server/Dockerfile b/src/yhub-server/Dockerfile index 31394e2d9..b7ef20248 100644 --- a/src/yhub-server/Dockerfile +++ b/src/yhub-server/Dockerfile @@ -6,7 +6,8 @@ WORKDIR /app COPY package.json package-lock.json ./ RUN npm ci --omit=dev -COPY server.js ./ +# server.js, migration.js, env.js — glob so a new module cannot be forgotten +COPY *.js ./ EXPOSE 3002 diff --git a/src/yhub-server/README.md b/src/yhub-server/README.md index d6e04670f..a980f437e 100644 --- a/src/yhub-server/README.md +++ b/src/yhub-server/README.md @@ -4,7 +4,14 @@ This directory contains the La Suite Docs-specific configuration for [yhub](https://www.npmjs.com/package/@y/hub) (`@y/hub`), the collaboration server that synchronizes Yjs documents between editors in real time. -It is not a fork of yhub — it is a thin wrapper (`server.js`) that: +It is not a fork of yhub — it is a thin wrapper: + +- `server.js` — configuration, the auth plugin, and the custom REST endpoints, +- `migration.js` — everything that reads the legacy Django/S3 document store + (both migrations described below), +- `env.js` — the `*_FILE` secret indirection shared by the two. + +`server.js`: - starts a yhub instance (websocket sync on port 3002, backed by Redis/Valkey and PostgreSQL), @@ -29,6 +36,9 @@ It is not a fork of yhub — it is a thin wrapper (`server.js`) that: server-side. Strict create: 409 when the document already has content. Guarded by standard document write access (the admin JWT, or a user session with update ability), +- exposes `POST /collaboration/migrate/v1/{org}/{docid}`, which replays a + document's **full** legacy version history out of the S3 media bucket (see + "Full migration" below) — admin JWT only, like `reset-connections`, - mirrors the environment conventions used elsewhere in this repository (`*_FILE` secret indirection, `COLLABORATION_SERVER_ORIGIN` allowlist, …). @@ -36,8 +46,8 @@ Public exposure: route the whole `/collaboration/` prefix to this server — the websocket and the built-in document APIs (`ydoc`, `rollback`, `prune`, `changeset`, `activity`) are all guarded by the same cookie-based document authorization and are meant to be reachable by browsers. The one exception -is `/collaboration/reset-connections/`, which is backend-internal and should -not be routed through the public ingress. +is `/collaboration/reset-connections/` and `/collaboration/migrate/`, which are +backend-internal and should not be routed through the public ingress. The `Dockerfile` builds the container image used by the `yhub` service in `compose.yml`. @@ -50,11 +60,11 @@ bucket, as UTF-8 text that is the base64 encoding of a raw Yjs update, at key documents into yhub lazily, on first access: 1. After a user's document authorization succeeds, the auth plugin checks - whether yhub already has content for the room — a bare postgres `SELECT` - (persisted rows), then the valkey stream (uncompacted `ydoc:update:v1` - messages), then the `SELECT` again to close the compaction race. Verdicts - are cached in-process (existing docs 10 min, empty docs 60 s, failures - 5 min). + whether yhub already has content for the room — the migrated set written by + the full migration (below), then a bare postgres `SELECT` (persisted rows), + then the valkey stream (uncompacted `ydoc:update:v1` messages), then the + `SELECT` again to close the compaction race. Verdicts are cached in-process + (existing docs 10 min, empty docs 60 s, failures 5 min). 2. If the room is unknown, the legacy object is fetched from S3 (10 s timeout, 10 MiB decoded cap — the same limit as `create-ydoc`), decoded, diffed through yhub's compute pool and appended to the room's stream — @@ -63,8 +73,20 @@ documents into yhub lazily, on first access: initial sync always includes the seeded content. First access to an unmigrated document is therefore slower by one S3 round-trip plus one compute pass. + + **A seed carries no timestamp.** Its contentmap has `insert`/`delete` and + `migration=s3` but deliberately no `insertAt`/`deleteAt`: a lazy seed is not + an editing event, and the only honest timestamps for legacy content are the + S3 version times that the full migration writes. Stamping the seed too would + put a second `insertAt` on the same ids — persisted contentmaps are merged, + not de-duplicated — and `activity` would report whichever the (unordered) + row scan happened to put last. The practical consequence: seeded content + produces **no `activity` entry** and is skipped by `from`/`to`-filtered + `changeset`/`rollback`/`prune` queries until the full migration supplies the + real history. Unfiltered queries, `by=system` and + `withCustomAttributions=migration:s3` still match it. 3. Concurrent first-connections are collapsed: an in-process in-flight map, a - per-room valkey lock (`{prefix}:softmigrate:*`, 30 s TTL), and a cap of 4 + per-room valkey lock (`{prefix}:softmigrate:*`, 30 s TTL), and a cap of 20 concurrent seeds per replica (excess connections fail fast and retry). Guarantees and failure behavior: @@ -72,12 +94,14 @@ Guarantees and failure behavior: - **Missing S3 object is not an error** — that is the brand-new-document case (Django writes no object until the first content save); the room simply starts empty. -- **Everything else fails closed**: network/auth errors, timeouts, oversized - objects and corrupt updates deny the connection (an opaque 401 the client - retries with backoff) and log a `soft-migration` error. A cached failure +- **Everything else fails closed**, and says whether it is worth retrying + (yhub 0.5.0 error semantics): an oversized or corrupt legacy object will fail + the same way forever, so the connection is denied `403`; network errors, + timeouts and momentary seed backpressure answer `503`, which clients retry + with backoff. Either way a `soft-migration` error is logged. A cached failure verdict prevents retry storms from hammering S3 — permanent failures (corrupt/oversized objects) for 5 minutes, transient ones (network errors, - timeouts) for 15 seconds, and per-replica seed backpressure (more than 4 + timeouts) for 15 seconds, and per-replica seed backpressure (more than 20 concurrent seeds) is denied without caching so the client's next retry goes through. - **Seeding is idempotent**: the legacy S3 snapshots are frozen (the @@ -116,13 +140,95 @@ Operational notes: "exists" and suppress the re-seed. - Lazy migration never finishes on its own: documents that are never opened stay in S3 forever, and they are only reachable through this flag now that - the frontend's client-side seeding is gone. A batch backfill (Django - posting each document's **exact** S3 bytes to `create-ydoc` with the admin - JWT, treating 409 as success) is the intended completion path and composes - safely with concurrent first-accesses — as long as content is never - re-converted: independently generated updates for the same document would - duplicate its content on merge, while re-posting the stored bytes is a - no-op. Only after the backfill may `SOFT_MIGRATION` be turned off. + the frontend's client-side seeding is gone. Running `migrate` (below) over + the corpus is the intended completion path. Only after that backfill may + `SOFT_MIGRATION` be turned off. + +## Full migration (`POST /collaboration/migrate/v1/{org}/{docid}`) + +The media bucket is versioned, so `{docid}/file` keeps every snapshot Django +ever wrote — that is the version history the backend exposes at +`/documents/{id}/versions/`. The lazy seed above replays only the newest one, so +a soft-migrated document lands in yhub as a single `system` change stamped with +the migration time and its past is gone. + +`migrate` replays the whole history instead. It lists the object's versions and +applies them, oldest first, to a single `Y.Doc({ gc: false })`; after each one +it credits the ids that version introduced (and the ones it deleted) with +**that version's own S3 timestamp**. `GET +/collaboration/activity/v1/{org}/{docid}?group=false` then reports one entry per +S3 version, at the same timestamps the backend's version listing reports as +`last_modified` — which is what lines the two up. (Pass `group=false`: the +default grouping merges changes by the same author less than a second apart, +which would fold versions saved in quick succession into one entry.) +`gc: false` is what preserves content that later versions deleted — most of +what makes a history worth keeping. + +Since yhub 0.5.0 the built-in endpoints also speak JSON on request, so a +non-JavaScript caller can read that timeline without a lib0 decoder: send +`Accept: application/json` and `activity`/`changeset` answer +`application/json` (binary fields base64-encoded) instead of +`application/x-lib0any`. + +The result lands as **one new row in `yhub_ydoc_v1` at clock `0`**, written +through `yhub.persistence.store`. Nothing is deleted and nothing goes on the +redis stream: the migration is purely additive. Clock `0` is what makes that +safe — + +- `store` is `ON CONFLICT (org, docid, branch, t) DO NOTHING`, so a repeated or + concurrent call is a database no-op; +- `retrieveDoc` derives the room's `lastClock` from the *newest* row, so a `0` + row can never hide stream messages a live editor is writing; +- the history genuinely is the oldest thing in the room. + +The next compact task merges that row into the room's normal state and deletes +it, like any other row — yhub needs no special case for it. + +Called with the admin JWT (`aud: "yhub"`), doc-scoped, `branch=main` only (the +legacy store is branchless). Running it over the whole corpus — any 2xx means +done — is the backfill that finishes the migration. + +Guarantees: + +- **Idempotent, twice over.** The docid is recorded in the valkey set + `{prefix}:migrated:v1` and skipped on later calls; and even without that, the + `t = 0` insert is a no-op. There is no lock: concurrent calls for the same + document all succeed and the database keeps one row. +- **Never destructive.** No existing row, stream message or attribution is + removed, so a document's own yhub history — edits made after it was seeded — + survives untouched alongside the imported one. +- **Nothing usable, nothing touched, nothing remembered.** A document with no + legacy object (`versions: 0`) or with no readable version (`applied: 0`) is + left exactly as it is and is *not* added to the set, so a later run can still + pick it up. Both answer `200 {"migrated": false}`, so a backfill driver can + treat every 2xx as done. +- **A corrupt version is skipped, not fatal** (counted as `skipped`, logged with + its version id). Snapshots are decoded before they are applied, so a bad one + can neither corrupt the accumulating document nor kill a compute worker. + Later versions are full snapshots, so their content still arrives — only that + one timeline entry is lost. +- **More than 500 versions**: only the newest 500 are replayed and the rest fold + into the first replayed version, reported as `dropped`. + +Response (200): `{ migrated, versions, applied, skipped, dropped, bytes, +durationMs }`. + +Caveats: + +- **`?force=true` re-runs a document that is already in the set.** Only safe + while its clock-0 row is still there. Once compaction has folded that row + away, a forced re-run inserts a second contentmap for ids that already carry + one, both `insertAt` values survive the merge, and the activity timestamp for + that content becomes whichever the unordered row scan puts last. To genuinely + redo a document, wipe its yhub state first (rows, stream key, set member). +- **The replay runs on the server's main thread.** yhub's compute pool only + accepts its own fixed task types, so a very long history briefly blocks the + event loop; that is what the 500-version cap bounds. +- **`activity` and `changeset` responses are cached for ~5s** (yhub's + `redis.cacheTtl`). A call made right after a migration can still answer with + the pre-migration timeline; it resolves itself. +- Requires `SOFT_MIGRATION=true` (that is what configures the S3 client); + otherwise it answers `503`. ## ⚠️ License warning (AGPL) @@ -131,10 +237,10 @@ This directory depends on `@y/hub`, which is licensed under the the rest of this repository (MIT), the code in this directory is loaded into the same process as AGPL-licensed code. As a consequence: -- **Any modification to the code in this directory (in particular - `server.js`) must be released under an AGPL-compatible license** if you run - or distribute the resulting server, including making it available to users - over a network (AGPL section 13). +- **Any modification to the code in this directory (in particular `server.js` + and `migration.js`) must be released under an AGPL-compatible license** if + you run or distribute the resulting server, including making it available to + users over a network (AGPL section 13). - See the [LICENSE](./LICENSE) file in this directory for details. **The rest of La Suite Docs is not affected.** The Django backend and the diff --git a/src/yhub-server/env.js b/src/yhub-server/env.js new file mode 100644 index 000000000..a962bc9a7 --- /dev/null +++ b/src/yhub-server/env.js @@ -0,0 +1,10 @@ +import { readFileSync } from 'node:fs'; + +// Read a config value that may be supplied either directly (`NAME`) or as a +// path to a file holding it (`NAME_FILE`) — the secret-file convention used +// across this repository, mirroring y-provider's env.ts. Shared by server.js +// and migration.js. +export const secret = (name, dflt) => + process.env[`${name}_FILE`] + ? readFileSync(process.env[`${name}_FILE`], 'utf8').trim() + : process.env[name] || dflt; diff --git a/src/yhub-server/migration.js b/src/yhub-server/migration.js new file mode 100644 index 000000000..5e42fee89 --- /dev/null +++ b/src/yhub-server/migration.js @@ -0,0 +1,591 @@ +// Migration off the legacy Django document store (see README.md). +// +// Documents were historically stored by the Django backend in the S3 media +// bucket, as UTF-8 text that is the base64 encoding of a raw Yjs update, at key +// `{document-uuid}/file`. The bucket is versioned, so every snapshot Django +// ever wrote for a document survives as an object version — that is the +// version history the backend exposes at `/documents/{id}/versions/`. +// +// Two paths bring that content into yhub, and they compose: +// +// maybeMigrate — the lazy seed. On first access to a room yhub does not know, +// fetch the *newest* version and seed the room with it before admitting the +// connection. Attributed to `system`, with no timestamp (see below). +// +// fullMigrate — the backfill. Replay *every* version into one gc:false +// document and store the result as a single row at clock 0, so yhub's +// activity API reports the same timeline as the S3 version listing. +// +// Everything here takes the `yhub` instance explicitly rather than closing over +// it: the endpoint handlers already receive one as `req.yhub`, and the auth +// plugin has the module-level instance by the time it first runs. + +import { randomUUID } from 'node:crypto'; + +import { logger } from '@y/hub'; +import * as Y from '@y/y'; +import { Client as S3Client } from 'minio'; + +import { secret } from './env.js'; + +export const SOFT_MIGRATION = process.env.SOFT_MIGRATION === 'true'; +const AWS_S3_ENDPOINT_URL = process.env.AWS_S3_ENDPOINT_URL; +const AWS_S3_ACCESS_KEY_ID = secret('AWS_S3_ACCESS_KEY_ID'); +const AWS_S3_SECRET_ACCESS_KEY = secret('AWS_S3_SECRET_ACCESS_KEY'); +const AWS_S3_REGION_NAME = process.env.AWS_S3_REGION_NAME; +// Django's default bucket name (impress settings.py) — prod overrides it +const AWS_STORAGE_BUCKET_NAME = + process.env.AWS_STORAGE_BUCKET_NAME || 'impress-media-storage'; +// the same limit create-ydoc applies to a posted update in server.js: one +// legacy snapshot handed to a compute worker, or written to the stream as a +// single message +const MAX_LEGACY_BYTES = 10 * 1024 * 1024; +// base64 inflates 3 bytes to 4 — cap the streamed read at the encoded size of +// MAX_LEGACY_BYTES plus padding slack +const MAX_LEGACY_B64_BYTES = Math.ceil(MAX_LEGACY_BYTES / 3) * 4 + 1024; +const S3_FETCH_TIMEOUT_MS = 10000; +const MIGRATE_LOCK_TTL_MS = 30000; +const MAX_CONCURRENT_SEEDS = 20; +// The full migration replays *every* S3 version of a document, so its budget is +// per-document rather than per-connection — no client is waiting on it. +const S3_LIST_TIMEOUT_MS = 30000; +// A document with more versions than this is migrated from its newest +// MAX_MIGRATE_VERSIONS only: everything older folds into the first replayed +// version, which keeps the run bounded instead of failing it outright. The +// response reports how many were dropped. The replay runs on the main thread, +// so the cap also bounds how long the event loop is blocked. +const MAX_MIGRATE_VERSIONS = 500; +// an empty Yjs update, what patchYdoc diffs the first snapshot against +const EMPTY_YDOC = Y.encodeStateAsUpdate(new Y.Doc()); + +if ( + SOFT_MIGRATION && + (!AWS_S3_ENDPOINT_URL || !AWS_S3_ACCESS_KEY_ID || !AWS_S3_SECRET_ACCESS_KEY) +) { + // fail at boot instead of as an opaque 401 storm on first connect + throw new Error( + 'SOFT_MIGRATION=true requires AWS_S3_ENDPOINT_URL, AWS_S3_ACCESS_KEY_ID and AWS_S3_SECRET_ACCESS_KEY', + ); +} +const s3 = SOFT_MIGRATION + ? (() => { + const url = new URL(AWS_S3_ENDPOINT_URL); + if (url.pathname !== '/' && url.pathname !== '') { + // boto3 accepts path-prefixed endpoints but the minio client cannot + // address a base path — dropping it silently would probe the wrong + // keys and "migrate" every doc as empty + throw new Error('AWS_S3_ENDPOINT_URL must not contain a path'); + } + return new S3Client({ + endPoint: url.hostname, + port: + url.port !== '' + ? Number(url.port) + : url.protocol === 'https:' + ? 443 + : 80, + useSSL: url.protocol === 'https:', + accessKey: AWS_S3_ACCESS_KEY_ID, + secretKey: AWS_S3_SECRET_ACCESS_KEY, + ...(AWS_S3_REGION_NAME ? { region: AWS_S3_REGION_NAME } : {}), + }); + })() + : null; +const migrationLog = logger.child({ module: 'soft-migration' }); + +// Both keys are derived from the prefix yhub itself resolved, so they cannot +// drift from the room keys, and both sit outside its scanned `:room:*` pattern. +// +// One seeder per room: +const migrateLockKey = (yhub, room) => + `${yhub.stream.prefix}:softmigrate:${room.org}:${room.docid}:${room.branch}`; +// Documents whose version history has been replayed into postgres. Membership +// is permanent: a second replay of the same versions would attribute the same +// content twice (see fullMigrate). +const migratedSetKey = (yhub) => `${yhub.stream.prefix}:migrated:v1`; + +// Legacy Django document store: object `{docid}/file`, body = UTF-8 text that +// is the base64 encoding of a raw Yjs update. With `versionId`, reads that +// specific object version instead of the current one. Returns null when the +// object (or version) does not exist — a document that never had content +// saved, e.g. brand new. Throws on any other failure (network, auth, timeout, +// oversize); corrupt base64 decodes leniently to garbage that the callers +// reject. +const fetchLegacyDoc = async (docid, versionId = null) => { + let stream = null; + let cancelTimeout = () => {}; + // minio 8 takes no AbortSignal — race a timer that also destroys the body + // stream once reading, so a stalled transfer cannot hold the ws upgrade + const timeout = new Promise((_, reject) => { + const timer = setTimeout(() => { + const err = new Error( + `s3 fetch timed out after ${S3_FETCH_TIMEOUT_MS}ms`, + ); + err.transient = true; // a slow S3 may recover — cache the failure briefly + stream?.destroy(err); + reject(err); + }, S3_FETCH_TIMEOUT_MS); + cancelTimeout = () => clearTimeout(timer); + }); + try { + let objPromise; + try { + objPromise = s3.getObject( + AWS_STORAGE_BUCKET_NAME, + `${docid}/file`, + // minio stringifies the whole opts object into the query — pass + // undefined, not {}, so the unversioned read stays byte-identical + versionId != null ? { versionId } : undefined, + ); + stream = await Promise.race([objPromise, timeout]); + } catch (err) { + // NoSuchVersion: the version vanished between listing and reading + if (err?.code === 'NoSuchKey' || err?.code === 'NoSuchVersion') { + return null; + } + // if the timeout won the race, getObject may still resolve later — + // destroy the late-arriving response stream, otherwise its never-read + // socket leaks (minio 8 sets no request timeout and cannot abort) + objPromise?.then( + (s) => s.destroy(err), + () => {}, + ); + throw err; + } + const body = await Promise.race([ + new Promise((resolve, reject) => { + const chunks = []; + let received = 0; + stream.on('data', (chunk) => { + received += chunk.byteLength; + if (received > MAX_LEGACY_B64_BYTES) { + stream.destroy( + new Error( + `legacy object exceeds the ${MAX_LEGACY_B64_BYTES}B cap`, + ), + ); + return; + } + chunks.push(chunk); + }); + stream.on('error', reject); + stream.on('end', () => resolve(Buffer.concat(chunks))); + }), + timeout, + ]); + const decoded = Buffer.from(body.toString('utf8'), 'base64'); + if (decoded.byteLength > MAX_LEGACY_BYTES) { + throw new Error( + `decoded legacy update (${decoded.byteLength}B) exceeds the ${MAX_LEGACY_BYTES}B cap`, + ); + } + // compute-task schema requires an exact Uint8Array (lib0 compares the + // constructor) — re-view the Buffer without copying + return new Uint8Array( + decoded.buffer, + decoded.byteOffset, + decoded.byteLength, + ); + } finally { + cancelTimeout(); + } +}; + +// Every version of the legacy object, oldest first. Delete markers are skipped +// (they record a deletion and carry no body), and so are keys that merely share +// the prefix — S3 has no exact-key version listing. +const listLegacyVersions = async (docid) => { + const key = `${docid}/file`; + const found = await new Promise((resolve, reject) => { + const versions = []; + const stream = s3.listObjects(AWS_STORAGE_BUCKET_NAME, key, true, { + IncludeVersion: true, + }); + const timer = setTimeout(() => { + const err = new Error( + `s3 version listing timed out after ${S3_LIST_TIMEOUT_MS}ms`, + ); + err.transient = true; + stream.destroy(err); + }, S3_LIST_TIMEOUT_MS); + stream.on('data', (obj) => { + if (obj.name === key && obj.isDeleteMarker !== true && obj.versionId) { + versions.push({ + versionId: String(obj.versionId), + // the moment S3 accepted the write: what the backend's version + // listing reports as `last_modified`, and what we attribute to + timestamp: obj.lastModified?.getTime() ?? 0, + }); + } + }); + stream.on('error', (err) => { + clearTimeout(timer); + reject(err); + }); + stream.on('end', () => { + clearTimeout(timer); + resolve(versions); + }); + }); + // S3 lists a key's versions newest first; reverse to replay them in write + // order. The sort is a stable safeguard across paginated listings — equal + // timestamps keep S3's own ordering. + found.reverse(); + found.sort((a, b) => a.timestamp - b.timestamp); + const dropped = Math.max(0, found.length - MAX_MIGRATE_VERSIONS); + return { versions: found.slice(dropped), dropped }; +}; + +// Quick existence check: does yhub already have content for this room? +// Sequenced cheapest-first: a persisted postgres row (bare SELECT, no blob +// columns; rows are never deleted, so a hit is always safe) — then the valkey +// stream (only ydoc:update:v1 counts: awareness and auth-check messages share +// the stream but carry no content) — then the SELECT again, which closes the +// store-before-trim compaction race (and the worst case of a miss is only a +// redundant, idempotent re-seed). +const ydocExists = async (yhub, room) => { + if ((await yhub.persistence.retrieveDoc(room, {})).lastClock !== '0') { + return true; + } + const streams = await yhub.stream.getMessages([{ room, clock: '0' }]); + if ((streams[0]?.messages ?? []).some((m) => m.type === 'ydoc:update:v1')) { + return true; + } + return (await yhub.persistence.retrieveDoc(room, {})).lastClock !== '0'; +}; + +// Per-docid migration verdicts, in-memory (per replica). 'exists' is monotone +// in normal operation — its TTL only bounds staleness after an operator +// manually wipes a room's yhub state (restart yhub after a wipe to drop the +// cache immediately). 'empty' (no S3 object) keeps never-edited docs and +// rechecks off S3; 'failed' breaks the retry-refetch storm a permanently +// corrupt object would otherwise sustain (y-websocket retries denied upgrades +// forever). +const VERDICT_TTL_MS = { exists: 600000, empty: 60000, failed: 300000 }; +// transient failures (network blips, timeouts, S3 restarting) are cached just +// long enough to blunt a retry storm without turning a hiccup into a lockout +const TRANSIENT_TTL_MS = 15000; +const TRANSIENT_CODES = new Set([ + 'ECONNREFUSED', + 'ECONNRESET', + 'ETIMEDOUT', + 'EHOSTUNREACH', + 'ENETUNREACH', + 'ENOTFOUND', + 'EAI_AGAIN', + 'EPIPE', +]); +// Will this failure plausibly resolve on its own? Callers use it twice: to pick +// the verdict TTL below, and — in server.js — to decide whether a denied +// connection reports a permanent 403 or a retryable 503. `noCache` is the +// per-replica seed backpressure, transient by construction (a slot frees up +// within seconds). +export const isTransientFailure = (err) => + err?.transient === true || + err?.noCache === true || + TRANSIENT_CODES.has(err?.code) || + TRANSIENT_CODES.has(err?.cause?.code); +const VERDICT_CACHE_MAX = 50000; +const verdicts = new Map(); // docid -> { verdict, error, expires } +const rememberVerdict = ( + docid, + verdict, + error = null, + ttl = VERDICT_TTL_MS[verdict], +) => { + // delete-then-set keeps Map insertion order ≈ recency, so the FIFO eviction + // drops the stalest entry — and re-setting an existing docid never evicts + // an unrelated one + if (!verdicts.delete(docid) && verdicts.size >= VERDICT_CACHE_MAX) { + verdicts.delete(verdicts.keys().next().value); + } + verdicts.set(docid, { + verdict, + error, + expires: Date.now() + ttl, + }); +}; +const inflightMigrations = new Map(); // docid -> Promise +let activeSeeds = 0; + +const migrate = async (yhub, room) => { + // A fully migrated room holds a single row at clock 0, which leaves + // `lastClock` at '0' — so ydocExists cannot see it and would seed on top of a + // complete history. Harmless (the seed's attributions are excluded as already + // known) but a pointless S3 round-trip per document during a backfill. + if (await yhub.stream.redis.sIsMember(migratedSetKey(yhub), room.docid)) { + return 'exists'; + } + if (await ydocExists(yhub, room)) return 'exists'; + // collapse cross-replica herds: one seeder per room, the rest wait and + // re-probe + const lockKey = migrateLockKey(yhub, room); + const lockToken = randomUUID(); + const redis = yhub.stream.redis; + const acquired = await redis.set(lockKey, lockToken, { + condition: 'NX', + expiration: { type: 'PX', value: MIGRATE_LOCK_TTL_MS }, + }); + try { + if (acquired == null) { + // another connection or replica is seeding — wait for its lock, then + // re-probe. If the doc is still absent (the holder crashed or its S3 + // fetch failed), fall through and seed ourselves: duplicate seeds use + // byte-identical updates from one lineage and merge as CRDT no-ops. + const deadline = Date.now() + MIGRATE_LOCK_TTL_MS + 5000; + while (Date.now() < deadline && (await redis.exists(lockKey)) === 1) { + await new Promise((resolve) => setTimeout(resolve, 300)); + } + if (await ydocExists(yhub, room)) return 'exists'; + } + if (activeSeeds >= MAX_CONCURRENT_SEEDS) { + // fail fast under a herd of distinct cold docs — the client's retry + // backoff spreads the load. Probes above stay uncapped. noCache: + // momentary per-replica backpressure must deny once, not be cached as + // a failure — a slot frees up within seconds + const err = new Error('too many concurrent soft migrations'); + err.noCache = true; + throw err; + } + activeSeeds++; + try { + const start = Date.now(); + const update = await fetchLegacyDoc(room.docid); + if (update == null) { + migrationLog.info( + { event: 'seed.empty', docid: room.docid }, + 'no legacy s3 object; room starts empty', + ); + return 'empty'; + } + await yhub.stream.addMessage(room, { + type: 'ydoc:update:v1', + // Deliberately no insertAt/deleteAt. A lazy seed is not an editing + // event: stamping it would put a second, meaningless timestamp on + // content that the full migration attributes to its real S3 version + // time — and persisted contentmaps are merged, not de-duplicated, so + // both would survive on the same ids and the activity API would report + // whichever the row order happened to put last. Content seeded this way + // carries an author but no timestamp, so it produces no activity entry + // until fullMigrate supplies the history. + // + // Reading the ids also validates the update: a corrupt legacy object + // throws here, on this thread, before anything reaches the stream. + contentmap: Y.encodeContentMap( + Y.createContentMapFromContentIds( + Y.createContentIdsFromUpdate(update), + [ + Y.createContentAttribute('insert', 'system'), + Y.createContentAttribute('insert:migration', 's3'), + ], + [ + Y.createContentAttribute('delete', 'system'), + Y.createContentAttribute('delete:migration', 's3'), + ], + ), + ), + update, + }); + migrationLog.info( + { + event: 'seed.ok', + docid: room.docid, + bytes: update.byteLength, + durationMs: Date.now() - start, + }, + 'seeded legacy doc from s3', + ); + return 'exists'; + } finally { + activeSeeds--; + } + } finally { + if (acquired != null) { + // compare-and-delete: if this seed outlived the lock TTL, another + // seeder holds a fresh lock — a bare DEL would release it under them + redis + .eval( + "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) end", + { keys: [lockKey], arguments: [lockToken] }, + ) + .catch(() => {}); + } + } +}; + +// Resolves when the room is usable (already known, freshly seeded, or +// legitimately empty); rejects to deny access. Idempotent and safe to +// re-enter — it also runs on rechecks and default-purpose REST calls. +export const maybeMigrate = async (yhub, room) => { + const cached = verdicts.get(room.docid); + if (cached != null && cached.expires > Date.now()) { + if (cached.verdict === 'failed') throw cached.error; + return; + } + let migration = inflightMigrations.get(room.docid); + if (migration == null) { + migration = migrate(yhub, room) + .then( + (verdict) => rememberVerdict(room.docid, verdict), + (err) => { + // logged here (once per attempt) rather than per denied connection: + // cached failures deny without new logs until the verdict expires + migrationLog.error( + { event: 'seed.failed', err, docid: room.docid }, + 'soft migration failed; denying access', + ); + if (err?.noCache !== true) { + // transient failures get a short TTL so a hiccup cannot lock a + // doc out for the full poison-object window + rememberVerdict( + room.docid, + 'failed', + err, + isTransientFailure(err) + ? TRANSIENT_TTL_MS + : VERDICT_TTL_MS.failed, + ); + } + throw err; + }, + ) + .finally(() => inflightMigrations.delete(room.docid)); + inflightMigrations.set(room.docid, migration); + } + return migration; +}; + +// Add the legacy version history to the room. Returns a `status` the endpoint +// maps to a response: +// 'already' — already replayed for this docid; the room is left untouched +// 'empty' — no legacy object in S3; the room is left untouched +// 'nothing' — versions exist but none is readable; the room is left untouched +// 'ok' — history stored +// +// Additive, never destructive: the versions are replayed into one gc:false +// document, and the result lands as a *single new row* at clock 0 — nothing +// existing is deleted and nothing goes on the stream. Clock 0 is what makes +// that safe. `store` is ON CONFLICT DO NOTHING on (org, docid, branch, t), so a +// concurrent or repeated call is a database no-op; `retrieveDoc` derives +// `lastClock` from the *newest* row, so a 0 row can never hide the stream +// messages a live editor is writing; and the history is genuinely the oldest +// thing in the room. The next compact task merges the row into the room's +// normal state and drops it — yhub needs no special case for any of this. +export const fullMigrate = async (yhub, room, { force = false } = {}) => { + const start = Date.now(); + const redis = yhub.stream.redis; + // Membership is the guard against attributing the same content twice: once + // compaction has folded the clock-0 row into a normal one and deleted it, + // a second replay would insert a second contentmap for ids that already + // carry one, and the two timestamps would both survive the merge. + if (!force && (await redis.sIsMember(migratedSetKey(yhub), room.docid))) { + return { status: 'already' }; + } + const { versions, dropped } = await listLegacyVersions(room.docid); + if (versions.length === 0) { + return { status: 'empty', versions: 0, durationMs: Date.now() - start }; + } + // Replay every snapshot into one gc:false document. gc matters: a collected + // doc would lose the content later versions deleted, which is most of what + // makes a history worth having. + const ydoc = new Y.Doc({ gc: false }); + // ids already attributed, so each version is credited only with what it added + let seen = Y.createContentIds(); + const contentmaps = []; + let bytes = 0; + let skipped = 0; + try { + for (const version of versions) { + const update = await fetchLegacyDoc(room.docid, version.versionId); + if (update == null) continue; // deleted between listing and read + bytes += update.byteLength; + try { + // Decode before applying. applyUpdate throwing part-way through would + // leave the accumulating doc in an undefined state, and this is the + // same lazy structural scan it would fail on. + Y.createContentIdsFromUpdate(update); + } catch (err) { + // Skip an unreadable snapshot rather than failing the document: every + // later version is a full snapshot, so its content still arrives — only + // this timeline entry is lost, and a corrupt version has nothing else + // to give. + skipped++; + migrationLog.warn( + { + event: 'full.version-skipped', + err, + docid: room.docid, + versionId: version.versionId, + }, + 'legacy s3 version is not a valid yjs update; skipping it', + ); + continue; + } + Y.applyUpdate(ydoc, update); + // `true`: inserts include content the doc has already deleted, so this is + // the full structural snapshot rather than what is currently visible + const all = Y.createContentIdsFromDoc(ydoc, true); + const fresh = Y.excludeContentIds(all, seen); + seen = all; + if ( + fresh.inserts.clients.size === 0 && + fresh.deletes.clients.size === 0 + ) { + continue; // this snapshot added nothing new + } + // Legacy snapshots carry no author, so every version is attributed to the + // 'system' identity (as the lazy seed is). The timestamp is what makes an + // entry identifiable: it is the version's S3 `LastModified`, so activity + // entries line up with the backend's version listing by time. + const attrs = (verb) => [ + Y.createContentAttribute(verb, 'system'), + Y.createContentAttribute(`${verb}At`, version.timestamp), + ]; + contentmaps.push( + Y.createContentMapFromContentIds( + fresh, + attrs('insert'), + attrs('delete'), + ), + ); + } + if (contentmaps.length === 0) { + // every version was unreadable or contentless — nothing to store, and + // nothing to remember either, so a later run can still pick it up + return { + status: 'nothing', + versions: versions.length, + applied: 0, + skipped, + dropped, + bytes, + durationMs: Date.now() - start, + }; + } + const nongcDoc = Y.encodeStateAsUpdate(ydoc); + await yhub.persistence.store(room, { + lastClock: '0', + gcDoc: await yhub.computePool.mergeUpdates(true, [nongcDoc], { room }), + nongcDoc, + contentmap: Y.encodeContentMap(Y.mergeContentMaps(contentmaps)), + contentids: Y.encodeContentIds(seen), + }); + } finally { + ydoc.destroy(); + } + await redis.sAdd(migratedSetKey(yhub), room.docid); + const result = { + status: 'ok', + versions: versions.length, + applied: contentmaps.length, + skipped, + dropped, + bytes, + durationMs: Date.now() - start, + }; + migrationLog.info( + { event: 'full.ok', docid: room.docid, ...result }, + 'stored document history from s3 versions', + ); + return result; +}; diff --git a/src/yhub-server/package-lock.json b/src/yhub-server/package-lock.json index 814f8e8a0..b200dc431 100644 --- a/src/yhub-server/package-lock.json +++ b/src/yhub-server/package-lock.json @@ -6,7 +6,8 @@ "": { "name": "yhub-server", "dependencies": { - "@y/hub": "0.4.0", + "@y/hub": "0.5.0", + "@y/y": "14.0.0-rc.24", "jose": "6.2.8", "minio": "8.0.7" }, @@ -111,15 +112,15 @@ "license": "ISC" }, "node_modules/@y/hub": { - "version": "0.4.0", - "resolved": "https://registry.npmjs.org/@y/hub/-/hub-0.4.0.tgz", - "integrity": "sha512-LejJTSrBt86DI88pMO8rck4tQcHHR1SPLrZrOIfb5Ws/zNd601x5SXX0MOuTwKqoNOweT4esUX0Eh/2ER4sSsA==", + "version": "0.5.0", + "resolved": "https://registry.npmjs.org/@y/hub/-/hub-0.5.0.tgz", + "integrity": "sha512-hMpSC3R+TvV13qPRHsSy4D+vyoSvrJah7ZoGaYtwPnHh92Ce53ufEb678SVZMN2susjKNyHUNrjoFIB7d6DjWg==", "license": "AGPL-3.0 OR PROPRIETARY", "dependencies": { "@y-crdt/yn": "^0.1.4", "@y/protocols": "^1.0.6-rc.1", "@y/y": "^14.0.0-rc.24", - "lib0": "^1.0.0-rc.23", + "lib0": "^1.0.0-rc.25", "minio": "^8.0.6", "pino": "^10.3.1", "postgres": "^3.4.3", @@ -335,9 +336,9 @@ } }, "node_modules/lib0": { - "version": "1.0.0-rc.23", - "resolved": "https://registry.npmjs.org/lib0/-/lib0-1.0.0-rc.23.tgz", - "integrity": "sha512-JPomcbwgKoTIDoXP61DFZV+Yvkw8bCyQhr9QYxps0fmHzsliEw+mhbUP1/nyVmk6ugzapJw1hUpFEMFRA4sRIg==", + "version": "1.0.0-rc.25", + "resolved": "https://registry.npmjs.org/lib0/-/lib0-1.0.0-rc.25.tgz", + "integrity": "sha512-UVxr56D1kVTx8P2Om5cAWXg2Pawk7JiOkCio+GLwlHcMko3FRE5xklOKB+t+fZSLA4ZLcVTZUKEIXPd9cRhEJg==", "license": "MIT", "bin": { "0ecdsa-generate-keypair": "src/bin/0ecdsa-generate-keypair.js", diff --git a/src/yhub-server/package.json b/src/yhub-server/package.json index e0129e490..485e6e8fd 100644 --- a/src/yhub-server/package.json +++ b/src/yhub-server/package.json @@ -6,7 +6,8 @@ "start": "node server.js" }, "dependencies": { - "@y/hub": "0.4.0", + "@y/hub": "0.5.0", + "@y/y": "14.0.0-rc.24", "jose": "6.2.8", "minio": "8.0.7" }, diff --git a/src/yhub-server/server.js b/src/yhub-server/server.js index ac782202d..472da12f2 100644 --- a/src/yhub-server/server.js +++ b/src/yhub-server/server.js @@ -1,15 +1,21 @@ -import { createHash, randomUUID } from 'node:crypto'; -import { readFileSync } from 'node:fs'; +import { createHash } from 'node:crypto'; -import { createApiEndpoint, createAuthPlugin, createYHub, logger } from '@y/hub'; +import { + apiError, + createApiEndpoint, + createAuthPlugin, + createYHub, +} from '@y/hub'; import { createRemoteJWKSet, jwtVerify } from 'jose'; -import { Client as S3Client } from 'minio'; -// mirror y-provider's env.ts secret-file support -const secret = (name, dflt) => - process.env[`${name}_FILE`] - ? readFileSync(process.env[`${name}_FILE`], 'utf8').trim() - : process.env[name] || dflt; +import { secret } from './env.js'; +// legacy Django/S3 document store — see migration.js and README.md +import { + SOFT_MIGRATION, + fullMigrate, + isTransientFailure, + maybeMigrate, +} from './migration.js'; const PORT = Number(process.env.PORT || 3002); const REDIS = process.env.REDIS; @@ -44,61 +50,6 @@ const EMPTY_YDOC = new Uint8Array([0, 0]); // websocket path. const MAX_CREATE_BYTES = 10 * 1024 * 1024; -// Soft migration (see README.md): legacy documents live in Django's S3 media -// bucket as UTF-8 base64 of a raw Yjs update at key `{docid}/file`. With -// SOFT_MIGRATION=true, the first access to a room yhub does not know yet -// fetches that object, seeds the room with it (attributed to 'system'), and -// only then admits the connection. -const SOFT_MIGRATION = process.env.SOFT_MIGRATION === 'true'; -const AWS_S3_ENDPOINT_URL = process.env.AWS_S3_ENDPOINT_URL; -const AWS_S3_ACCESS_KEY_ID = secret('AWS_S3_ACCESS_KEY_ID'); -const AWS_S3_SECRET_ACCESS_KEY = secret('AWS_S3_SECRET_ACCESS_KEY'); -const AWS_S3_REGION_NAME = process.env.AWS_S3_REGION_NAME; -// Django's default bucket name (impress settings.py) — prod overrides it -const AWS_STORAGE_BUCKET_NAME = - process.env.AWS_STORAGE_BUCKET_NAME || 'impress-media-storage'; -// base64 inflates 3 bytes to 4 — cap the streamed read at the encoded size of -// MAX_CREATE_BYTES (the same effective limit as create-ydoc) plus padding slack -const MAX_LEGACY_B64_BYTES = Math.ceil(MAX_CREATE_BYTES / 3) * 4 + 1024; -const S3_FETCH_TIMEOUT_MS = 10000; -const MIGRATE_LOCK_TTL_MS = 30000; -const MAX_CONCURRENT_SEEDS = 4; - -if ( - SOFT_MIGRATION && - (!AWS_S3_ENDPOINT_URL || !AWS_S3_ACCESS_KEY_ID || !AWS_S3_SECRET_ACCESS_KEY) -) { - // fail at boot instead of as an opaque 401 storm on first connect - throw new Error( - 'SOFT_MIGRATION=true requires AWS_S3_ENDPOINT_URL, AWS_S3_ACCESS_KEY_ID and AWS_S3_SECRET_ACCESS_KEY', - ); -} -const s3 = SOFT_MIGRATION - ? (() => { - const url = new URL(AWS_S3_ENDPOINT_URL); - if (url.pathname !== '/' && url.pathname !== '') { - // boto3 accepts path-prefixed endpoints but the minio client cannot - // address a base path — dropping it silently would probe the wrong - // keys and "migrate" every doc as empty - throw new Error('AWS_S3_ENDPOINT_URL must not contain a path'); - } - return new S3Client({ - endPoint: url.hostname, - port: - url.port !== '' - ? Number(url.port) - : url.protocol === 'https:' - ? 443 - : 80, - useSSL: url.protocol === 'https:', - accessKey: AWS_S3_ACCESS_KEY_ID, - secretKey: AWS_S3_SECRET_ACCESS_KEY, - ...(AWS_S3_REGION_NAME ? { region: AWS_S3_REGION_NAME } : {}), - }); - })() - : null; -const migrationLog = logger.child({ module: 'soft-migration' }); - // Public keys verifying the RS256 admin tokens Django issues (JWTService). // Lazily fetched on first use; jose caches the keys and refetches on unknown // "kid", so Django can rotate the signing key without a yhub restart. @@ -122,273 +73,6 @@ const backendFetch = async (path, { cookie, origin }) => { return res.json(); }; -// Legacy Django document store: object `{docid}/file`, body = UTF-8 text that -// is the base64 encoding of a raw Yjs update. Returns null when the object -// does not exist — a document that never had content saved, e.g. brand new. -// Throws on any other failure (network, auth, timeout, oversize); corrupt -// base64 decodes leniently to garbage that patchYdoc later rejects. -const fetchLegacyDoc = async (docid) => { - let stream = null; - let cancelTimeout = () => {}; - // minio 8 takes no AbortSignal — race a timer that also destroys the body - // stream once reading, so a stalled transfer cannot hold the ws upgrade - const timeout = new Promise((_, reject) => { - const timer = setTimeout(() => { - const err = new Error( - `s3 fetch timed out after ${S3_FETCH_TIMEOUT_MS}ms`, - ); - err.transient = true; // a slow S3 may recover — cache the failure briefly - stream?.destroy(err); - reject(err); - }, S3_FETCH_TIMEOUT_MS); - cancelTimeout = () => clearTimeout(timer); - }); - try { - let objPromise; - try { - objPromise = s3.getObject(AWS_STORAGE_BUCKET_NAME, `${docid}/file`); - stream = await Promise.race([objPromise, timeout]); - } catch (err) { - if (err?.code === 'NoSuchKey') return null; - // if the timeout won the race, getObject may still resolve later — - // destroy the late-arriving response stream, otherwise its never-read - // socket leaks (minio 8 sets no request timeout and cannot abort) - objPromise?.then((s) => s.destroy(err), () => {}); - throw err; - } - const body = await Promise.race([ - new Promise((resolve, reject) => { - const chunks = []; - let received = 0; - stream.on('data', (chunk) => { - received += chunk.byteLength; - if (received > MAX_LEGACY_B64_BYTES) { - stream.destroy( - new Error(`legacy object exceeds the ${MAX_LEGACY_B64_BYTES}B cap`), - ); - return; - } - chunks.push(chunk); - }); - stream.on('error', reject); - stream.on('end', () => resolve(Buffer.concat(chunks))); - }), - timeout, - ]); - const decoded = Buffer.from(body.toString('utf8'), 'base64'); - if (decoded.byteLength > MAX_CREATE_BYTES) { - throw new Error( - `decoded legacy update (${decoded.byteLength}B) exceeds the ${MAX_CREATE_BYTES}B cap`, - ); - } - // compute-task schema requires an exact Uint8Array (lib0 compares the - // constructor) — re-view the Buffer without copying - return new Uint8Array( - decoded.buffer, - decoded.byteOffset, - decoded.byteLength, - ); - } finally { - cancelTimeout(); - } -}; - -// Quick existence check: does yhub already have content for this room? -// Sequenced cheapest-first: a persisted postgres row (bare SELECT, no blob -// columns; rows are never deleted, so a hit is always safe) — then the valkey -// stream (only ydoc:update:v1 counts: awareness and auth-check messages share -// the stream but carry no content) — then the SELECT again, which closes the -// store-before-trim compaction race (and the worst case of a miss is only a -// redundant, idempotent re-seed). -const ydocExists = async (room) => { - if ((await yhub.persistence.retrieveDoc(room, {})).lastClock !== '0') { - return true; - } - const streams = await yhub.stream.getMessages([{ room, clock: '0' }]); - if ((streams[0]?.messages ?? []).some((m) => m.type === 'ydoc:update:v1')) { - return true; - } - return (await yhub.persistence.retrieveDoc(room, {})).lastClock !== '0'; -}; - -// Per-docid migration verdicts, in-memory (per replica). 'exists' is monotone -// in normal operation — its TTL only bounds staleness after an operator -// manually wipes a room's yhub state (restart yhub after a wipe to drop the -// cache immediately). 'empty' (no S3 object) keeps never-edited docs and -// rechecks off S3; 'failed' breaks the retry-refetch storm a permanently -// corrupt object would otherwise sustain (y-websocket retries denied upgrades -// forever). -const VERDICT_TTL_MS = { exists: 600000, empty: 60000, failed: 300000 }; -// transient failures (network blips, timeouts, S3 restarting) are cached just -// long enough to blunt a retry storm without turning a hiccup into a lockout -const TRANSIENT_TTL_MS = 15000; -const TRANSIENT_CODES = new Set([ - 'ECONNREFUSED', - 'ECONNRESET', - 'ETIMEDOUT', - 'EHOSTUNREACH', - 'ENETUNREACH', - 'ENOTFOUND', - 'EAI_AGAIN', - 'EPIPE', -]); -const isTransient = (err) => - err?.transient === true || - TRANSIENT_CODES.has(err?.code) || - TRANSIENT_CODES.has(err?.cause?.code); -const VERDICT_CACHE_MAX = 50000; -const verdicts = new Map(); // docid -> { verdict, error, expires } -const rememberVerdict = (docid, verdict, error = null, ttl = VERDICT_TTL_MS[verdict]) => { - // delete-then-set keeps Map insertion order ≈ recency, so the FIFO eviction - // drops the stalest entry — and re-setting an existing docid never evicts - // an unrelated one - if (!verdicts.delete(docid) && verdicts.size >= VERDICT_CACHE_MAX) { - verdicts.delete(verdicts.keys().next().value); - } - verdicts.set(docid, { - verdict, - error, - expires: Date.now() + ttl, - }); -}; -const inflightMigrations = new Map(); // docid -> Promise -let activeSeeds = 0; - -const migrate = async (room) => { - if (await ydocExists(room)) return 'exists'; - // collapse cross-replica herds: one seeder per room, the rest wait and - // re-probe. The key sits outside yhub's scanned `:room:*` patterns. - const lockKey = `${REDIS_PREFIX}:softmigrate:${room.org}:${room.docid}:${room.branch}`; - const lockToken = randomUUID(); - const redis = yhub.stream.redis; - const acquired = await redis.set(lockKey, lockToken, { - condition: 'NX', - expiration: { type: 'PX', value: MIGRATE_LOCK_TTL_MS }, - }); - try { - if (acquired == null) { - // another connection or replica is seeding — wait for its lock, then - // re-probe. If the doc is still absent (the holder crashed or its S3 - // fetch failed), fall through and seed ourselves: duplicate seeds use - // byte-identical updates from one lineage and merge as CRDT no-ops. - const deadline = Date.now() + MIGRATE_LOCK_TTL_MS + 5000; - while (Date.now() < deadline && (await redis.exists(lockKey)) === 1) { - await new Promise((resolve) => setTimeout(resolve, 300)); - } - if (await ydocExists(room)) return 'exists'; - } - if (activeSeeds >= MAX_CONCURRENT_SEEDS) { - // fail fast under a herd of distinct cold docs — the client's retry - // backoff spreads the load. Probes above stay uncapped. noCache: - // momentary per-replica backpressure must deny once, not be cached as - // a failure — a slot frees up within seconds - const err = new Error('too many concurrent soft migrations'); - err.noCache = true; - throw err; - } - activeSeeds++; - try { - const start = Date.now(); - const update = await fetchLegacyDoc(room.docid); - if (update == null) { - migrationLog.info( - { event: 'seed.empty', docid: room.docid }, - 'no legacy s3 object; room starts empty', - ); - return 'empty'; - } - // legacy content has no per-user history — attribute it to 'system' - // (the admin-JWT identity), marked so audits can tell migrated content - // apart from other system writes - const result = await yhub.computePool.patchYdoc( - { - update, - currentDoc: EMPTY_YDOC, - userid: 'system', - customAttributions: [{ k: 'migration', v: 's3' }], - }, - { room }, - ); - if (result == null) { - // structurally valid but no effective content — nothing to seed - migrationLog.info( - { event: 'seed.empty', docid: room.docid }, - 'legacy s3 object has no effective content; room starts empty', - ); - return 'empty'; - } - await yhub.stream.addMessage(room, { - type: 'ydoc:update:v1', - contentmap: result.contentmap, - update: result.update, - }); - migrationLog.info( - { - event: 'seed.ok', - docid: room.docid, - bytes: update.byteLength, - durationMs: Date.now() - start, - }, - 'seeded legacy doc from s3', - ); - return 'exists'; - } finally { - activeSeeds--; - } - } finally { - if (acquired != null) { - // compare-and-delete: if this seed outlived the lock TTL, another - // seeder holds a fresh lock — a bare DEL would release it under them - redis - .eval( - "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) end", - { keys: [lockKey], arguments: [lockToken] }, - ) - .catch(() => {}); - } - } -}; - -// Resolves when the room is usable (already known, freshly seeded, or -// legitimately empty); rejects to deny access. Idempotent and safe to -// re-enter — it also runs on rechecks and default-purpose REST calls. -const maybeMigrate = async (room) => { - const cached = verdicts.get(room.docid); - if (cached != null && cached.expires > Date.now()) { - if (cached.verdict === 'failed') throw cached.error; - return; - } - let migration = inflightMigrations.get(room.docid); - if (migration == null) { - migration = migrate(room) - .then( - (verdict) => rememberVerdict(room.docid, verdict), - (err) => { - // logged here (once per attempt) rather than per denied connection: - // cached failures deny without new logs until the verdict expires - migrationLog.error( - { event: 'seed.failed', err, docid: room.docid }, - 'soft migration failed; denying access', - ); - if (err?.noCache !== true) { - // transient failures get a short TTL so a hiccup cannot lock a - // doc out for the full poison-object window - rememberVerdict( - room.docid, - 'failed', - err, - isTransient(err) ? TRANSIENT_TTL_MS : VERDICT_TTL_MS.failed, - ); - } - throw err; - }, - ) - .finally(() => inflightMigrations.delete(room.docid)); - inflightMigrations.set(room.docid, migration); - } - return migration; -}; - const auth = createAuthPlugin({ // uws req is only valid synchronously — read headers AND query before first await. async readAuthInfo(req) { @@ -420,9 +104,20 @@ const auth = createAuthPlugin({ return payload.admin === true ? { userid: 'system', admin: true } : null; - } catch { - // bad signature / expired / wrong (or missing) audience / JWKS - // unreachable — fail closed + } catch (err) { + // jose tags every token-validation failure with an `ERR_J…` code (bad + // signature, expired, wrong or missing audience) — those are permanent, + // fail closed. A JWKS fetch that times out or never connects has no + // such code (or ERR_JWKS_TIMEOUT): the token may be perfectly valid and + // we simply cannot check it, so report it as retryable instead of + // accusing the caller of forging it. + if ( + err?.code === 'ERR_JWKS_TIMEOUT' || + typeof err?.code !== 'string' || + !err.code.startsWith('ERR_J') + ) { + throw apiError(503, 'Token verification keys are unavailable'); + } return null; } } @@ -437,11 +132,14 @@ const auth = createAuthPlugin({ return { userid: String(user.id), cookie, origin }; // MUST be string (yhub server.js:667) } catch (err) { // Only a genuine "not signed in" falls back to the anonymous identity. - // On backend failure (5xx/network) fail closed: a signed-in editor - // authorized under an anon userid would be invisible to the targeted - // reset-connections recheck (users: []) for the connection's - // whole lifetime. - if (err?.status !== 401 && err?.status !== 403) return null; + // On backend failure (5xx/network) still refuse to admit the connection — + // a signed-in editor authorized under an anon userid would be invisible + // to the targeted reset-connections recheck (users: []) for the + // connection's whole lifetime — but report it as retryable rather than as + // an authentication failure the client should give up on. + if (err?.status !== 401 && err?.status !== 403) { + throw apiError(503, 'Authentication backend is unavailable'); + } // anonymous (public docs): stable per-session id — random ids would mint a new // permanent attribution identity per reconnect const anon = createHash('sha256') @@ -467,8 +165,14 @@ const auth = createAuthPlugin({ let doc; try { doc = await backendFetch(`/api/v1.0/documents/${docid}/`, authInfo); - } catch { - return null; + } catch (err) { + // the backend answered "no": a real, permanent denial (403 Forbidden) + if (err?.status === 401 || err?.status === 403 || err?.status === 404) { + return null; + } + // it did not answer at all — say so, so the caller retries instead of + // reading a 5xx or a network blip as a permission decision + throw apiError(503, 'Document authorization backend is unavailable'); } if (!doc.abilities?.retrieve) { return null; @@ -480,12 +184,21 @@ const auth = createAuthPlugin({ // postgres and the stream from clock 0) is guaranteed to include the // seed. Runs only for authorized readers. A missing S3 object is the // brand-new-document case and allows an empty room; a real S3/compute - // failure denies access — an opaque 401 the client retries with - // backoff. + // failure denies access, either way already logged (once per attempt) + // inside maybeMigrate. try { - await maybeMigrate({ org, docid, branch }); - } catch { - return null; // already logged (once per attempt) in maybeMigrate + // `yhub` is declared at the bottom of this file — safe: auth callbacks + // only fire once the server is up, i.e. after that assignment + await maybeMigrate(yhub, { org, docid, branch }); + } catch (err) { + // A corrupt or oversized legacy object will fail the same way forever, + // so that denial is permanent (403). An S3 timeout, a network blip or + // momentary seed backpressure will not — 503 tells the caller to come + // back rather than to treat the document as unreadable. + if (isTransientFailure(err)) { + throw apiError(503, 'Legacy document store is unavailable'); + } + return null; } } return doc.abilities.update ? 'rw' : 'r'; @@ -527,6 +240,66 @@ const api = [ }, }, }), + // POST /collaboration/migrate/v1/{org}/{docid} — replay a document's full + // legacy version history from the S3 media bucket into yhub (see README.md). + // Backend-internal, like reset-connections: gated to the admin token via the + // 'migrate' access purpose, since it writes history and reads the legacy + // store. + createApiEndpoint('migrate', { + accessPurpose: 'migrate', + post: { + handler: async (req) => { + if (req.org !== ORG) { + return jsonResponse(400, { error: 'Unknown org' }); + } + if (!UUID4.test(req.docid)) { + return jsonResponse(400, { error: 'Room name is invalid' }); + } + if (req.branch !== 'main') { + // the legacy store is branchless: `{docid}/file` is the main branch + return jsonResponse(400, { error: 'Unknown branch' }); + } + if (!SOFT_MIGRATION) { + // the flag is what configures the S3 client (migration.js) + return jsonResponse(503, { error: 'Legacy store is not configured' }); + } + // ?force=true replays a document that is already in the migrated set. + // Only safe while its clock-0 row is still there: once compaction has + // folded that row away, a second replay attributes the same content a + // second time and the activity timestamps become ambiguous. + const { status, ...stats } = await fullMigrate(req.yhub, req.room, { + force: req.query.force === 'true', + }); + if (status === 'already') { + return jsonResponse(200, { + message: 'Already migrated', + migrated: false, + }); + } + if (status === 'empty') { + // brand-new documents never had a legacy object; nothing to replay + // and nothing wrong — a backfill driver treats this as done + return jsonResponse(200, { + message: 'No legacy document in s3', + migrated: false, + ...stats, + }); + } + if (status === 'nothing') { + return jsonResponse(200, { + message: 'No usable content in the legacy versions', + migrated: false, + ...stats, + }); + } + return jsonResponse(200, { + message: 'Migration completed', + migrated: true, + ...stats, + }); + }, + }, + }), // POST /collaboration/create-ydoc/v1/{org}/{docid} — create a document's // initial Yjs state from a RAW binary update (`Y.encodeStateAsUpdate` / // pycrdt `get_update()` output) posted as application/octet-stream. Unlike @@ -561,8 +334,9 @@ const api = [ body.byteLength, ); if (update.byteLength > MAX_CREATE_BYTES) { - // 413 is missing from yhub's status-line map, so the reason phrase - // is empty ("HTTP/1.1 413 ") — legal, and callers switch on the code + // 413 is missing from yhub's status-line map (503 was added in + // 0.5.0, 413 was not), so the reason phrase is empty + // ("HTTP/1.1 413 ") — legal, and callers switch on the code return jsonResponse(413, { error: 'Update too large' }); } // <= 3 bytes is yhub's "no effective content" convention (an empty @@ -630,8 +404,8 @@ const api = [ }), ]; -// the instance is referenced by the soft-migration helpers above — safe: auth -// callbacks only fire once the server is up, i.e. after this assignment +// referenced by getAccessType above — safe: auth callbacks only fire once the +// server is up, i.e. after this assignment const yhub = await createYHub({ redis: { url: REDIS, @@ -647,6 +421,6 @@ const yhub = await createYHub({ server: { port: PORT, auth, api, apiPrefix: 'collaboration' }, worker: { taskConcurrency: 5 }, // TODO(yhub): worker.events.docUpdate could push snapshots to Django and replace the - // client useSaveDoc PATCH flow — blocked upstream: the payload is a DocTable without - // room/org/docid (yhub src/index.js:90); needs an upstream change first. + // client useSaveDoc PATCH flow. No longer blocked upstream — yhub 0.5.0 adds `room` + // to the event payload, which was the missing piece. });