WEBDIENSTE/lasuite-docs
1
0
Fork 0
mirror of https://github.com/suitenumerique/docs.git synced 2026-08-25 00:42:28 +02:00
Code Issues Packages Projects Releases Wiki Activity

✨(collaboration) replay legacy s3 version history into yhub

Browse Source 
Add POST /collaboration/migrate/v1/docs/{id}, which replays every S3 version
of a document's legacy `{id}/file` object into one gc:false Yjs document and
stores it as a single row at clock 0, crediting each version with its own S3
timestamp. Nothing existing is deleted and nothing goes on the stream, so the
next compaction merges that row like any other. The clock-0 insert is ON
CONFLICT DO NOTHING and migrated ids are kept in a valkey set, so the endpoint
is idempotent without a lock. The activity api then reports the same timeline
as the backend's /documents/{id}/versions/, instead of the single
migration-time change the lazy soft migration leaves behind.

That lazy seed now writes no insertAt/deleteAt. Persisted contentmaps are
merged rather than de-duplicated, so a seed timestamp would survive next to
the real per-version one on the same ids and the activity api would report
whichever the unordered row scan put last. A seed is not an editing event and
has no honest time to report.

Upgrade yhub to 0.5.0, where error codes encode retry semantics (4xx
permanent, 5xx and 429 retryable) and auth plugins may throw apiError(503). A
temporarily unreachable Django backend, JWKS endpoint or legacy S3 store is
now reported as 503 rather than denied like a permission failure, so clients
retry instead of giving up.

The legacy-store code moves out of server.js into migration.js, with the
shared *_FILE secret helper in env.js.

Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
This commit is contained in:
Kevin Jahns
2026-08-13 12:08:28 +02:00
committed by Manuel Raynaud
parent 00ac755283
commit 4c60fc5c65
8 changed files with 898 additions and 393 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+28 -7
View File
@@ -11,10 +11,26 @@ and this project adheres to
- ✨(collaboration) soft-migrate legacy S3 documents into yhub on first access
(`SOFT_MIGRATION=true`): when yhub does not know a document yet, its legacy
snapshot (`{id}/file`, base64 Yjs update) is fetched from the Django S3
media bucket and seeded server-side (attributed to `system`) before the
connection is admitted. A missing S3 object means a brand-new document and
yields an empty room; any real S3/compute failure fails closed (opaque 401,
the client retries with backoff). Enabled in the dev stack via compose.yml
media bucket and seeded server-side (attributed to `system`, with no
timestamp — a lazy seed is not an editing event, and stamping one would
collide with the real per-version times the migrate endpoint writes) before
the connection is admitted. A missing S3 object means a brand-new document and
yields an empty room; any real S3/compute failure fails closed, as a
permanent `403` for a corrupt or oversized object and a retryable `503` for
timeouts and network errors. Enabled in the dev stack via compose.yml
- ✨(collaboration) add a migrate endpoint on yhub:
`POST /collaboration/migrate/v1/docs/{id}` replays a document's **full**
legacy version history from the versioned S3 media bucket into a
`gc: false` Yjs document, crediting each S3 version with its own S3
timestamp — so `activity?group=false` reports the same timeline as the
backend's `/documents/{id}/versions/`, instead of the single
migration-time change the
lazy soft migration leaves behind. Purely additive: the result is stored as
one new row at clock `0`, so nothing existing is deleted and the next
compaction merges it like any other row. Idempotent by construction (the
clock-`0` insert is `ON CONFLICT DO NOTHING`, and migrated ids are recorded
in a valkey set), lock-free, admin JWT only, and the intended way to backfill
the corpus before `SOFT_MIGRATION` is turned off
- ✨(collaboration) add a create-ydoc endpoint on yhub:
`POST /collaboration/create-ydoc/v1/docs/{id}` seeds a document's initial
Yjs state from a raw binary update posted as `application/octet-stream`,
@@ -31,11 +47,16 @@ and this project adheres to
an admin token Django issued for another service (e.g. the `y-converter`
one) cannot be replayed here; not yet triggered by the backend on
permission changes (follow-up)
- ⬆️(collaboration) upgrade yhub to 0.4.0 and serve all its routes under the
- ⬆️(collaboration) upgrade yhub to 0.5.0 and serve all its routes under the
`/collaboration/` prefix (`server.apiPrefix`): the websocket moves to
`/collaboration/ws/v1/docs`. All `/collaboration/` routes are meant to be
publicly exposed except `reset-connections`, which stays backend-internal
(admin JWT only)
publicly exposed except `reset-connections` and `migrate`, which stay
backend-internal (admin JWT only). Following 0.5.0's error semantics
(`4xx` permanent, `5xx`/`429` retryable), the auth plugin now reports a
temporarily unreachable Django backend, JWKS endpoint or legacy S3 store as
`503` instead of denying access like a permission failure, so clients retry
instead of giving up. The built-in endpoints can also answer JSON on
`Accept: application/json`
- ✨(backend) add a service generating cached RS256 JWT tokens
- ✨(backend) publish the JWT public key on a JWKS endpoint
- 🔧(dev) generate the JWT signing key when bootstrapping the dev stack
src/yhub-server/Dockerfile
+2 -1
View File
@@ -6,7 +6,8 @@ WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY server.js ./
# server.js, migration.js, env.js — glob so a new module cannot be forgotten
COPY *.js ./
EXPOSE 3002
src/yhub-server/README.md
+130 -24
View File
@@ -4,7 +4,14 @@ This directory contains the La Suite Docs-specific configuration for
[yhub](https://www.npmjs.com/package/@y/hub) (`@y/hub`), the collaboration
server that synchronizes Yjs documents between editors in real time.
It is not a fork of yhub — it is a thin wrapper (`server.js`) that:
It is not a fork of yhub — it is a thin wrapper:
- `server.js` — configuration, the auth plugin, and the custom REST endpoints,
- `migration.js` — everything that reads the legacy Django/S3 document store
(both migrations described below),
- `env.js` — the `*_FILE` secret indirection shared by the two.
`server.js`:
- starts a yhub instance (websocket sync on port 3002, backed by Redis/Valkey
and PostgreSQL),
@@ -29,6 +36,9 @@ It is not a fork of yhub — it is a thin wrapper (`server.js`) that:
server-side. Strict create: 409 when the document already has content.
Guarded by standard document write access (the admin JWT, or a user
session with update ability),
- exposes `POST /collaboration/migrate/v1/{org}/{docid}`, which replays a
document's **full** legacy version history out of the S3 media bucket (see
"Full migration" below) — admin JWT only, like `reset-connections`,
- mirrors the environment conventions used elsewhere in this repository
(`*_FILE` secret indirection, `COLLABORATION_SERVER_ORIGIN` allowlist, …).
@@ -36,8 +46,8 @@ Public exposure: route the whole `/collaboration/` prefix to this server —
the websocket and the built-in document APIs (`ydoc`, `rollback`, `prune`,
`changeset`, `activity`) are all guarded by the same cookie-based document
authorization and are meant to be reachable by browsers. The one exception
is `/collaboration/reset-connections/`, which is backend-internal and should
not be routed through the public ingress.
is `/collaboration/reset-connections/` and `/collaboration/migrate/`, which are
backend-internal and should not be routed through the public ingress.
The `Dockerfile` builds the container image used by the `yhub` service in
`compose.yml`.
@@ -50,11 +60,11 @@ bucket, as UTF-8 text that is the base64 encoding of a raw Yjs update, at key
documents into yhub lazily, on first access:
1. After a user's document authorization succeeds, the auth plugin checks
whether yhub already has content for the room — a bare postgres `SELECT`
(persisted rows), then the valkey stream (uncompacted `ydoc:update:v1`
messages), then the `SELECT` again to close the compaction race. Verdicts
are cached in-process (existing docs 10 min, empty docs 60 s, failures
5 min).
whether yhub already has content for the room — the migrated set written by
the full migration (below), then a bare postgres `SELECT` (persisted rows),
then the valkey stream (uncompacted `ydoc:update:v1` messages), then the
`SELECT` again to close the compaction race. Verdicts are cached in-process
(existing docs 10 min, empty docs 60 s, failures 5 min).
2. If the room is unknown, the legacy object is fetched from S3 (10 s
timeout, 10 MiB decoded cap — the same limit as `create-ydoc`), decoded,
diffed through yhub's compute pool and appended to the room's stream —
@@ -63,8 +73,20 @@ documents into yhub lazily, on first access:
initial sync always includes the seeded content. First access to an
unmigrated document is therefore slower by one S3 round-trip plus one
compute pass.
**A seed carries no timestamp.** Its contentmap has `insert`/`delete` and
`migration=s3` but deliberately no `insertAt`/`deleteAt`: a lazy seed is not
an editing event, and the only honest timestamps for legacy content are the
S3 version times that the full migration writes. Stamping the seed too would
put a second `insertAt` on the same ids — persisted contentmaps are merged,
not de-duplicated — and `activity` would report whichever the (unordered)
row scan happened to put last. The practical consequence: seeded content
produces **no `activity` entry** and is skipped by `from`/`to`-filtered
`changeset`/`rollback`/`prune` queries until the full migration supplies the
real history. Unfiltered queries, `by=system` and
`withCustomAttributions=migration:s3` still match it.
3. Concurrent first-connections are collapsed: an in-process in-flight map, a
per-room valkey lock (`{prefix}:softmigrate:*`, 30 s TTL), and a cap of 4
per-room valkey lock (`{prefix}:softmigrate:*`, 30 s TTL), and a cap of 20
concurrent seeds per replica (excess connections fail fast and retry).
Guarantees and failure behavior:
@@ -72,12 +94,14 @@ Guarantees and failure behavior:
- **Missing S3 object is not an error** — that is the brand-new-document case
(Django writes no object until the first content save); the room simply
starts empty.
- **Everything else fails closed**: network/auth errors, timeouts, oversized
objects and corrupt updates deny the connection (an opaque 401 the client
retries with backoff) and log a `soft-migration` error. A cached failure
- **Everything else fails closed**, and says whether it is worth retrying
(yhub 0.5.0 error semantics): an oversized or corrupt legacy object will fail
the same way forever, so the connection is denied `403`; network errors,
timeouts and momentary seed backpressure answer `503`, which clients retry
with backoff. Either way a `soft-migration` error is logged. A cached failure
verdict prevents retry storms from hammering S3 — permanent failures
(corrupt/oversized objects) for 5 minutes, transient ones (network errors,
timeouts) for 15 seconds, and per-replica seed backpressure (more than 4
timeouts) for 15 seconds, and per-replica seed backpressure (more than 20
concurrent seeds) is denied without caching so the client's next retry goes
through.
- **Seeding is idempotent**: the legacy S3 snapshots are frozen (the
@@ -116,13 +140,95 @@ Operational notes:
"exists" and suppress the re-seed.
- Lazy migration never finishes on its own: documents that are never opened
stay in S3 forever, and they are only reachable through this flag now that
the frontend's client-side seeding is gone. A batch backfill (Django
posting each document's **exact** S3 bytes to `create-ydoc` with the admin
JWT, treating 409 as success) is the intended completion path and composes
safely with concurrent first-accesses — as long as content is never
re-converted: independently generated updates for the same document would
duplicate its content on merge, while re-posting the stored bytes is a
no-op. Only after the backfill may `SOFT_MIGRATION` be turned off.
the frontend's client-side seeding is gone. Running `migrate` (below) over
the corpus is the intended completion path. Only after that backfill may
`SOFT_MIGRATION` be turned off.
## Full migration (`POST /collaboration/migrate/v1/{org}/{docid}`)
The media bucket is versioned, so `{docid}/file` keeps every snapshot Django
ever wrote — that is the version history the backend exposes at
`/documents/{id}/versions/`. The lazy seed above replays only the newest one, so
a soft-migrated document lands in yhub as a single `system` change stamped with
the migration time and its past is gone.
`migrate` replays the whole history instead. It lists the object's versions and
applies them, oldest first, to a single `Y.Doc({ gc: false })`; after each one
it credits the ids that version introduced (and the ones it deleted) with
**that version's own S3 timestamp**. `GET
/collaboration/activity/v1/{org}/{docid}?group=false` then reports one entry per
S3 version, at the same timestamps the backend's version listing reports as
`last_modified` — which is what lines the two up. (Pass `group=false`: the
default grouping merges changes by the same author less than a second apart,
which would fold versions saved in quick succession into one entry.)
`gc: false` is what preserves content that later versions deleted — most of
what makes a history worth keeping.
Since yhub 0.5.0 the built-in endpoints also speak JSON on request, so a
non-JavaScript caller can read that timeline without a lib0 decoder: send
`Accept: application/json` and `activity`/`changeset` answer
`application/json` (binary fields base64-encoded) instead of
`application/x-lib0any`.
The result lands as **one new row in `yhub_ydoc_v1` at clock `0`**, written
through `yhub.persistence.store`. Nothing is deleted and nothing goes on the
redis stream: the migration is purely additive. Clock `0` is what makes that
safe —
- `store` is `ON CONFLICT (org, docid, branch, t) DO NOTHING`, so a repeated or
concurrent call is a database no-op;
- `retrieveDoc` derives the room's `lastClock` from the *newest* row, so a `0`
row can never hide stream messages a live editor is writing;
- the history genuinely is the oldest thing in the room.
The next compact task merges that row into the room's normal state and deletes
it, like any other row — yhub needs no special case for it.
Called with the admin JWT (`aud: "yhub"`), doc-scoped, `branch=main` only (the
legacy store is branchless). Running it over the whole corpus — any 2xx means
done — is the backfill that finishes the migration.
Guarantees:
- **Idempotent, twice over.** The docid is recorded in the valkey set
`{prefix}:migrated:v1` and skipped on later calls; and even without that, the
`t = 0` insert is a no-op. There is no lock: concurrent calls for the same
document all succeed and the database keeps one row.
- **Never destructive.** No existing row, stream message or attribution is
removed, so a document's own yhub history — edits made after it was seeded —
survives untouched alongside the imported one.
- **Nothing usable, nothing touched, nothing remembered.** A document with no
legacy object (`versions: 0`) or with no readable version (`applied: 0`) is
left exactly as it is and is *not* added to the set, so a later run can still
pick it up. Both answer `200 {"migrated": false}`, so a backfill driver can
treat every 2xx as done.
- **A corrupt version is skipped, not fatal** (counted as `skipped`, logged with
its version id). Snapshots are decoded before they are applied, so a bad one
can neither corrupt the accumulating document nor kill a compute worker.
Later versions are full snapshots, so their content still arrives — only that
one timeline entry is lost.
- **More than 500 versions**: only the newest 500 are replayed and the rest fold
into the first replayed version, reported as `dropped`.
Response (200): `{ migrated, versions, applied, skipped, dropped, bytes,
durationMs }`.
Caveats:
- **`?force=true` re-runs a document that is already in the set.** Only safe
while its clock-0 row is still there. Once compaction has folded that row
away, a forced re-run inserts a second contentmap for ids that already carry
one, both `insertAt` values survive the merge, and the activity timestamp for
that content becomes whichever the unordered row scan puts last. To genuinely
redo a document, wipe its yhub state first (rows, stream key, set member).
- **The replay runs on the server's main thread.** yhub's compute pool only
accepts its own fixed task types, so a very long history briefly blocks the
event loop; that is what the 500-version cap bounds.
- **`activity` and `changeset` responses are cached for ~5s** (yhub's
`redis.cacheTtl`). A call made right after a migration can still answer with
the pre-migration timeline; it resolves itself.
- Requires `SOFT_MIGRATION=true` (that is what configures the S3 client);
otherwise it answers `503`.
## ⚠️ License warning (AGPL)
@@ -131,10 +237,10 @@ This directory depends on `@y/hub`, which is licensed under the
the rest of this repository (MIT), the code in this directory is loaded into
the same process as AGPL-licensed code. As a consequence:
- **Any modification to the code in this directory (in particular
`server.js`) must be released under an AGPL-compatible license** if you run
or distribute the resulting server, including making it available to users
over a network (AGPL section 13).
- **Any modification to the code in this directory (in particular `server.js`
and `migration.js`) must be released under an AGPL-compatible license** if
you run or distribute the resulting server, including making it available to
users over a network (AGPL section 13).
- See the [LICENSE](./LICENSE) file in this directory for details.
**The rest of La Suite Docs is not affected.** The Django backend and the
src/yhub-server/env.js
+10
View File
@@ -0,0 +1,10 @@
import { readFileSync } from 'node:fs';
// Read a config value that may be supplied either directly (`NAME`) or as a
// path to a file holding it (`NAME_FILE`) — the secret-file convention used
// across this repository, mirroring y-provider's env.ts. Shared by server.js
// and migration.js.
export const secret = (name, dflt) =>
process.env[`${name}_FILE`]
? readFileSync(process.env[`${name}_FILE`], 'utf8').trim()
: process.env[name] || dflt;
src/yhub-server/migration.js
+591
View File
@@ -0,0 +1,591 @@
// Migration off the legacy Django document store (see README.md).
//
// Documents were historically stored by the Django backend in the S3 media
// bucket, as UTF-8 text that is the base64 encoding of a raw Yjs update, at key
// `{document-uuid}/file`. The bucket is versioned, so every snapshot Django
// ever wrote for a document survives as an object version — that is the
// version history the backend exposes at `/documents/{id}/versions/`.
//
// Two paths bring that content into yhub, and they compose:
//
// maybeMigrate — the lazy seed. On first access to a room yhub does not know,
// fetch the *newest* version and seed the room with it before admitting the
// connection. Attributed to `system`, with no timestamp (see below).
//
// fullMigrate — the backfill. Replay *every* version into one gc:false
// document and store the result as a single row at clock 0, so yhub's
// activity API reports the same timeline as the S3 version listing.
//
// Everything here takes the `yhub` instance explicitly rather than closing over
// it: the endpoint handlers already receive one as `req.yhub`, and the auth
// plugin has the module-level instance by the time it first runs.
import { randomUUID } from 'node:crypto';
import { logger } from '@y/hub';
import * as Y from '@y/y';
import { Client as S3Client } from 'minio';
import { secret } from './env.js';
export const SOFT_MIGRATION = process.env.SOFT_MIGRATION === 'true';
const AWS_S3_ENDPOINT_URL = process.env.AWS_S3_ENDPOINT_URL;
const AWS_S3_ACCESS_KEY_ID = secret('AWS_S3_ACCESS_KEY_ID');
const AWS_S3_SECRET_ACCESS_KEY = secret('AWS_S3_SECRET_ACCESS_KEY');
const AWS_S3_REGION_NAME = process.env.AWS_S3_REGION_NAME;
// Django's default bucket name (impress settings.py) — prod overrides it
const AWS_STORAGE_BUCKET_NAME =
process.env.AWS_STORAGE_BUCKET_NAME || 'impress-media-storage';
// the same limit create-ydoc applies to a posted update in server.js: one
// legacy snapshot handed to a compute worker, or written to the stream as a
// single message
const MAX_LEGACY_BYTES = 10 * 1024 * 1024;
// base64 inflates 3 bytes to 4 — cap the streamed read at the encoded size of
// MAX_LEGACY_BYTES plus padding slack
const MAX_LEGACY_B64_BYTES = Math.ceil(MAX_LEGACY_BYTES / 3) * 4 + 1024;
const S3_FETCH_TIMEOUT_MS = 10000;
const MIGRATE_LOCK_TTL_MS = 30000;
const MAX_CONCURRENT_SEEDS = 20;
// The full migration replays *every* S3 version of a document, so its budget is
// per-document rather than per-connection — no client is waiting on it.
const S3_LIST_TIMEOUT_MS = 30000;
// A document with more versions than this is migrated from its newest
// MAX_MIGRATE_VERSIONS only: everything older folds into the first replayed
// version, which keeps the run bounded instead of failing it outright. The
// response reports how many were dropped. The replay runs on the main thread,
// so the cap also bounds how long the event loop is blocked.
const MAX_MIGRATE_VERSIONS = 500;
// an empty Yjs update, what patchYdoc diffs the first snapshot against
const EMPTY_YDOC = Y.encodeStateAsUpdate(new Y.Doc());
if (
SOFT_MIGRATION &&
(!AWS_S3_ENDPOINT_URL || !AWS_S3_ACCESS_KEY_ID || !AWS_S3_SECRET_ACCESS_KEY)
) {
// fail at boot instead of as an opaque 401 storm on first connect
throw new Error(
'SOFT_MIGRATION=true requires AWS_S3_ENDPOINT_URL, AWS_S3_ACCESS_KEY_ID and AWS_S3_SECRET_ACCESS_KEY',
);
}
const s3 = SOFT_MIGRATION
? (() => {
const url = new URL(AWS_S3_ENDPOINT_URL);
if (url.pathname !== '/' && url.pathname !== '') {
// boto3 accepts path-prefixed endpoints but the minio client cannot
// address a base path — dropping it silently would probe the wrong
// keys and "migrate" every doc as empty
throw new Error('AWS_S3_ENDPOINT_URL must not contain a path');
}
return new S3Client({
endPoint: url.hostname,
port:
url.port !== ''
? Number(url.port)
: url.protocol === 'https:'
? 443
: 80,
useSSL: url.protocol === 'https:',
accessKey: AWS_S3_ACCESS_KEY_ID,
secretKey: AWS_S3_SECRET_ACCESS_KEY,
...(AWS_S3_REGION_NAME ? { region: AWS_S3_REGION_NAME } : {}),
});
})()
: null;
const migrationLog = logger.child({ module: 'soft-migration' });
// Both keys are derived from the prefix yhub itself resolved, so they cannot
// drift from the room keys, and both sit outside its scanned `:room:*` pattern.
//
// One seeder per room:
const migrateLockKey = (yhub, room) =>
`${yhub.stream.prefix}:softmigrate:${room.org}:${room.docid}:${room.branch}`;
// Documents whose version history has been replayed into postgres. Membership
// is permanent: a second replay of the same versions would attribute the same
// content twice (see fullMigrate).
const migratedSetKey = (yhub) => `${yhub.stream.prefix}:migrated:v1`;
// Legacy Django document store: object `{docid}/file`, body = UTF-8 text that
// is the base64 encoding of a raw Yjs update. With `versionId`, reads that
// specific object version instead of the current one. Returns null when the
// object (or version) does not exist — a document that never had content
// saved, e.g. brand new. Throws on any other failure (network, auth, timeout,
// oversize); corrupt base64 decodes leniently to garbage that the callers
// reject.
const fetchLegacyDoc = async (docid, versionId = null) => {
let stream = null;
let cancelTimeout = () => {};
// minio 8 takes no AbortSignal — race a timer that also destroys the body
// stream once reading, so a stalled transfer cannot hold the ws upgrade
const timeout = new Promise((_, reject) => {
const timer = setTimeout(() => {
const err = new Error(
`s3 fetch timed out after ${S3_FETCH_TIMEOUT_MS}ms`,
);
err.transient = true; // a slow S3 may recover — cache the failure briefly
stream?.destroy(err);
reject(err);
}, S3_FETCH_TIMEOUT_MS);
cancelTimeout = () => clearTimeout(timer);
});
try {
let objPromise;
try {
objPromise = s3.getObject(
AWS_STORAGE_BUCKET_NAME,
`${docid}/file`,
// minio stringifies the whole opts object into the query — pass
// undefined, not {}, so the unversioned read stays byte-identical
versionId != null ? { versionId } : undefined,
);
stream = await Promise.race([objPromise, timeout]);
} catch (err) {
// NoSuchVersion: the version vanished between listing and reading
if (err?.code === 'NoSuchKey' || err?.code === 'NoSuchVersion') {
return null;
}
// if the timeout won the race, getObject may still resolve later —
// destroy the late-arriving response stream, otherwise its never-read
// socket leaks (minio 8 sets no request timeout and cannot abort)
objPromise?.then(
(s) => s.destroy(err),
() => {},
);
throw err;
}
const body = await Promise.race([
new Promise((resolve, reject) => {
const chunks = [];
let received = 0;
stream.on('data', (chunk) => {
received += chunk.byteLength;
if (received > MAX_LEGACY_B64_BYTES) {
stream.destroy(
new Error(
`legacy object exceeds the ${MAX_LEGACY_B64_BYTES}B cap`,
),
);
return;
}
chunks.push(chunk);
});
stream.on('error', reject);
stream.on('end', () => resolve(Buffer.concat(chunks)));
}),
timeout,
]);
const decoded = Buffer.from(body.toString('utf8'), 'base64');
if (decoded.byteLength > MAX_LEGACY_BYTES) {
throw new Error(
`decoded legacy update (${decoded.byteLength}B) exceeds the ${MAX_LEGACY_BYTES}B cap`,
);
}
// compute-task schema requires an exact Uint8Array (lib0 compares the
// constructor) — re-view the Buffer without copying
return new Uint8Array(
decoded.buffer,
decoded.byteOffset,
decoded.byteLength,
);
} finally {
cancelTimeout();
}
};
// Every version of the legacy object, oldest first. Delete markers are skipped
// (they record a deletion and carry no body), and so are keys that merely share
// the prefix — S3 has no exact-key version listing.
const listLegacyVersions = async (docid) => {
const key = `${docid}/file`;
const found = await new Promise((resolve, reject) => {
const versions = [];
const stream = s3.listObjects(AWS_STORAGE_BUCKET_NAME, key, true, {
IncludeVersion: true,
});
const timer = setTimeout(() => {
const err = new Error(
`s3 version listing timed out after ${S3_LIST_TIMEOUT_MS}ms`,
);
err.transient = true;
stream.destroy(err);
}, S3_LIST_TIMEOUT_MS);
stream.on('data', (obj) => {
if (obj.name === key && obj.isDeleteMarker !== true && obj.versionId) {
versions.push({
versionId: String(obj.versionId),
// the moment S3 accepted the write: what the backend's version
// listing reports as `last_modified`, and what we attribute to
timestamp: obj.lastModified?.getTime() ?? 0,
});
}
});
stream.on('error', (err) => {
clearTimeout(timer);
reject(err);
});
stream.on('end', () => {
clearTimeout(timer);
resolve(versions);
});
});
// S3 lists a key's versions newest first; reverse to replay them in write
// order. The sort is a stable safeguard across paginated listings — equal
// timestamps keep S3's own ordering.
found.reverse();
found.sort((a, b) => a.timestamp - b.timestamp);
const dropped = Math.max(0, found.length - MAX_MIGRATE_VERSIONS);
return { versions: found.slice(dropped), dropped };
};
// Quick existence check: does yhub already have content for this room?
// Sequenced cheapest-first: a persisted postgres row (bare SELECT, no blob
// columns; rows are never deleted, so a hit is always safe) — then the valkey
// stream (only ydoc:update:v1 counts: awareness and auth-check messages share
// the stream but carry no content) — then the SELECT again, which closes the
// store-before-trim compaction race (and the worst case of a miss is only a
// redundant, idempotent re-seed).
const ydocExists = async (yhub, room) => {
if ((await yhub.persistence.retrieveDoc(room, {})).lastClock !== '0') {
return true;
}
const streams = await yhub.stream.getMessages([{ room, clock: '0' }]);
if ((streams[0]?.messages ?? []).some((m) => m.type === 'ydoc:update:v1')) {
return true;
}
return (await yhub.persistence.retrieveDoc(room, {})).lastClock !== '0';
};
// Per-docid migration verdicts, in-memory (per replica). 'exists' is monotone
// in normal operation — its TTL only bounds staleness after an operator
// manually wipes a room's yhub state (restart yhub after a wipe to drop the
// cache immediately). 'empty' (no S3 object) keeps never-edited docs and
// rechecks off S3; 'failed' breaks the retry-refetch storm a permanently
// corrupt object would otherwise sustain (y-websocket retries denied upgrades
// forever).
const VERDICT_TTL_MS = { exists: 600000, empty: 60000, failed: 300000 };
// transient failures (network blips, timeouts, S3 restarting) are cached just
// long enough to blunt a retry storm without turning a hiccup into a lockout
const TRANSIENT_TTL_MS = 15000;
const TRANSIENT_CODES = new Set([
'ECONNREFUSED',
'ECONNRESET',
'ETIMEDOUT',
'EHOSTUNREACH',
'ENETUNREACH',
'ENOTFOUND',
'EAI_AGAIN',
'EPIPE',
]);
// Will this failure plausibly resolve on its own? Callers use it twice: to pick
// the verdict TTL below, and — in server.js — to decide whether a denied
// connection reports a permanent 403 or a retryable 503. `noCache` is the
// per-replica seed backpressure, transient by construction (a slot frees up
// within seconds).
export const isTransientFailure = (err) =>
err?.transient === true ||
err?.noCache === true ||
TRANSIENT_CODES.has(err?.code) ||
TRANSIENT_CODES.has(err?.cause?.code);
const VERDICT_CACHE_MAX = 50000;
const verdicts = new Map(); // docid -> { verdict, error, expires }
const rememberVerdict = (
docid,
verdict,
error = null,
ttl = VERDICT_TTL_MS[verdict],
) => {
// delete-then-set keeps Map insertion order ≈ recency, so the FIFO eviction
// drops the stalest entry — and re-setting an existing docid never evicts
// an unrelated one
if (!verdicts.delete(docid) && verdicts.size >= VERDICT_CACHE_MAX) {
verdicts.delete(verdicts.keys().next().value);
}
verdicts.set(docid, {
verdict,
error,
expires: Date.now() + ttl,
});
};
const inflightMigrations = new Map(); // docid -> Promise<void>
let activeSeeds = 0;
const migrate = async (yhub, room) => {
// A fully migrated room holds a single row at clock 0, which leaves
// `lastClock` at '0' — so ydocExists cannot see it and would seed on top of a
// complete history. Harmless (the seed's attributions are excluded as already
// known) but a pointless S3 round-trip per document during a backfill.
if (await yhub.stream.redis.sIsMember(migratedSetKey(yhub), room.docid)) {
return 'exists';
}
if (await ydocExists(yhub, room)) return 'exists';
// collapse cross-replica herds: one seeder per room, the rest wait and
// re-probe
const lockKey = migrateLockKey(yhub, room);
const lockToken = randomUUID();
const redis = yhub.stream.redis;
const acquired = await redis.set(lockKey, lockToken, {
condition: 'NX',
expiration: { type: 'PX', value: MIGRATE_LOCK_TTL_MS },
});
try {
if (acquired == null) {
// another connection or replica is seeding — wait for its lock, then
// re-probe. If the doc is still absent (the holder crashed or its S3
// fetch failed), fall through and seed ourselves: duplicate seeds use
// byte-identical updates from one lineage and merge as CRDT no-ops.
const deadline = Date.now() + MIGRATE_LOCK_TTL_MS + 5000;
while (Date.now() < deadline && (await redis.exists(lockKey)) === 1) {
await new Promise((resolve) => setTimeout(resolve, 300));
}
if (await ydocExists(yhub, room)) return 'exists';
}
if (activeSeeds >= MAX_CONCURRENT_SEEDS) {
// fail fast under a herd of distinct cold docs — the client's retry
// backoff spreads the load. Probes above stay uncapped. noCache:
// momentary per-replica backpressure must deny once, not be cached as
// a failure — a slot frees up within seconds
const err = new Error('too many concurrent soft migrations');
err.noCache = true;
throw err;
}
activeSeeds++;
try {
const start = Date.now();
const update = await fetchLegacyDoc(room.docid);
if (update == null) {
migrationLog.info(
{ event: 'seed.empty', docid: room.docid },
'no legacy s3 object; room starts empty',
);
return 'empty';
}
await yhub.stream.addMessage(room, {
type: 'ydoc:update:v1',
// Deliberately no insertAt/deleteAt. A lazy seed is not an editing
// event: stamping it would put a second, meaningless timestamp on
// content that the full migration attributes to its real S3 version
// time — and persisted contentmaps are merged, not de-duplicated, so
// both would survive on the same ids and the activity API would report
// whichever the row order happened to put last. Content seeded this way
// carries an author but no timestamp, so it produces no activity entry
// until fullMigrate supplies the history.
//
// Reading the ids also validates the update: a corrupt legacy object
// throws here, on this thread, before anything reaches the stream.
contentmap: Y.encodeContentMap(
Y.createContentMapFromContentIds(
Y.createContentIdsFromUpdate(update),
[
Y.createContentAttribute('insert', 'system'),
Y.createContentAttribute('insert:migration', 's3'),
],
[
Y.createContentAttribute('delete', 'system'),
Y.createContentAttribute('delete:migration', 's3'),
],
),
),
update,
});
migrationLog.info(
{
event: 'seed.ok',
docid: room.docid,
bytes: update.byteLength,
durationMs: Date.now() - start,
},
'seeded legacy doc from s3',
);
return 'exists';
} finally {
activeSeeds--;
}
} finally {
if (acquired != null) {
// compare-and-delete: if this seed outlived the lock TTL, another
// seeder holds a fresh lock — a bare DEL would release it under them
redis
.eval(
"if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) end",
{ keys: [lockKey], arguments: [lockToken] },
)
.catch(() => {});
}
}
};
// Resolves when the room is usable (already known, freshly seeded, or
// legitimately empty); rejects to deny access. Idempotent and safe to
// re-enter — it also runs on rechecks and default-purpose REST calls.
export const maybeMigrate = async (yhub, room) => {
const cached = verdicts.get(room.docid);
if (cached != null && cached.expires > Date.now()) {
if (cached.verdict === 'failed') throw cached.error;
return;
}
let migration = inflightMigrations.get(room.docid);
if (migration == null) {
migration = migrate(yhub, room)
.then(
(verdict) => rememberVerdict(room.docid, verdict),
(err) => {
// logged here (once per attempt) rather than per denied connection:
// cached failures deny without new logs until the verdict expires
migrationLog.error(
{ event: 'seed.failed', err, docid: room.docid },
'soft migration failed; denying access',
);
if (err?.noCache !== true) {
// transient failures get a short TTL so a hiccup cannot lock a
// doc out for the full poison-object window
rememberVerdict(
room.docid,
'failed',
err,
isTransientFailure(err)
? TRANSIENT_TTL_MS
: VERDICT_TTL_MS.failed,
);
}
throw err;
},
)
.finally(() => inflightMigrations.delete(room.docid));
inflightMigrations.set(room.docid, migration);
}
return migration;
};
// Add the legacy version history to the room. Returns a `status` the endpoint
// maps to a response:
// 'already' — already replayed for this docid; the room is left untouched
// 'empty' — no legacy object in S3; the room is left untouched
// 'nothing' — versions exist but none is readable; the room is left untouched
// 'ok' — history stored
//
// Additive, never destructive: the versions are replayed into one gc:false
// document, and the result lands as a *single new row* at clock 0 — nothing
// existing is deleted and nothing goes on the stream. Clock 0 is what makes
// that safe. `store` is ON CONFLICT DO NOTHING on (org, docid, branch, t), so a
// concurrent or repeated call is a database no-op; `retrieveDoc` derives
// `lastClock` from the *newest* row, so a 0 row can never hide the stream
// messages a live editor is writing; and the history is genuinely the oldest
// thing in the room. The next compact task merges the row into the room's
// normal state and drops it — yhub needs no special case for any of this.
export const fullMigrate = async (yhub, room, { force = false } = {}) => {
const start = Date.now();
const redis = yhub.stream.redis;
// Membership is the guard against attributing the same content twice: once
// compaction has folded the clock-0 row into a normal one and deleted it,
// a second replay would insert a second contentmap for ids that already
// carry one, and the two timestamps would both survive the merge.
if (!force && (await redis.sIsMember(migratedSetKey(yhub), room.docid))) {
return { status: 'already' };
}
const { versions, dropped } = await listLegacyVersions(room.docid);
if (versions.length === 0) {
return { status: 'empty', versions: 0, durationMs: Date.now() - start };
}
// Replay every snapshot into one gc:false document. gc matters: a collected
// doc would lose the content later versions deleted, which is most of what
// makes a history worth having.
const ydoc = new Y.Doc({ gc: false });
// ids already attributed, so each version is credited only with what it added
let seen = Y.createContentIds();
const contentmaps = [];
let bytes = 0;
let skipped = 0;
try {
for (const version of versions) {
const update = await fetchLegacyDoc(room.docid, version.versionId);
if (update == null) continue; // deleted between listing and read
bytes += update.byteLength;
try {
// Decode before applying. applyUpdate throwing part-way through would
// leave the accumulating doc in an undefined state, and this is the
// same lazy structural scan it would fail on.
Y.createContentIdsFromUpdate(update);
} catch (err) {
// Skip an unreadable snapshot rather than failing the document: every
// later version is a full snapshot, so its content still arrives — only
// this timeline entry is lost, and a corrupt version has nothing else
// to give.
skipped++;
migrationLog.warn(
{
event: 'full.version-skipped',
err,
docid: room.docid,
versionId: version.versionId,
},
'legacy s3 version is not a valid yjs update; skipping it',
);
continue;
}
Y.applyUpdate(ydoc, update);
// `true`: inserts include content the doc has already deleted, so this is
// the full structural snapshot rather than what is currently visible
const all = Y.createContentIdsFromDoc(ydoc, true);
const fresh = Y.excludeContentIds(all, seen);
seen = all;
if (
fresh.inserts.clients.size === 0 &&
fresh.deletes.clients.size === 0
) {
continue; // this snapshot added nothing new
}
// Legacy snapshots carry no author, so every version is attributed to the
// 'system' identity (as the lazy seed is). The timestamp is what makes an
// entry identifiable: it is the version's S3 `LastModified`, so activity
// entries line up with the backend's version listing by time.
const attrs = (verb) => [
Y.createContentAttribute(verb, 'system'),
Y.createContentAttribute(`${verb}At`, version.timestamp),
];
contentmaps.push(
Y.createContentMapFromContentIds(
fresh,
attrs('insert'),
attrs('delete'),
),
);
}
if (contentmaps.length === 0) {
// every version was unreadable or contentless — nothing to store, and
// nothing to remember either, so a later run can still pick it up
return {
status: 'nothing',
versions: versions.length,
applied: 0,
skipped,
dropped,
bytes,
durationMs: Date.now() - start,
};
}
const nongcDoc = Y.encodeStateAsUpdate(ydoc);
await yhub.persistence.store(room, {
lastClock: '0',
gcDoc: await yhub.computePool.mergeUpdates(true, [nongcDoc], { room }),
nongcDoc,
contentmap: Y.encodeContentMap(Y.mergeContentMaps(contentmaps)),
contentids: Y.encodeContentIds(seen),
});
} finally {
ydoc.destroy();
}
await redis.sAdd(migratedSetKey(yhub), room.docid);
const result = {
status: 'ok',
versions: versions.length,
applied: contentmaps.length,
skipped,
dropped,
bytes,
durationMs: Date.now() - start,
};
migrationLog.info(
{ event: 'full.ok', docid: room.docid, ...result },
'stored document history from s3 versions',
);
return result;
};
src/yhub-server/package-lock.json
Generated
+9 -8
View File
@@ -6,7 +6,8 @@
"": {
"name": "yhub-server",
"dependencies": {
"@y/hub": "0.4.0",
"@y/hub": "0.5.0",
"@y/y": "14.0.0-rc.24",
"jose": "6.2.8",
"minio": "8.0.7"
},
@@ -111,15 +112,15 @@
"license": "ISC"
},
"node_modules/@y/hub": {
"version": "0.4.0",
"resolved": "https://registry.npmjs.org/@y/hub/-/hub-0.4.0.tgz",
"integrity": "sha512-LejJTSrBt86DI88pMO8rck4tQcHHR1SPLrZrOIfb5Ws/zNd601x5SXX0MOuTwKqoNOweT4esUX0Eh/2ER4sSsA==",
"version": "0.5.0",
"resolved": "https://registry.npmjs.org/@y/hub/-/hub-0.5.0.tgz",
"integrity": "sha512-hMpSC3R+TvV13qPRHsSy4D+vyoSvrJah7ZoGaYtwPnHh92Ce53ufEb678SVZMN2susjKNyHUNrjoFIB7d6DjWg==",
"license": "AGPL-3.0 OR PROPRIETARY",
"dependencies": {
"@y-crdt/yn": "^0.1.4",
"@y/protocols": "^1.0.6-rc.1",
"@y/y": "^14.0.0-rc.24",
"lib0": "^1.0.0-rc.23",
"lib0": "^1.0.0-rc.25",
"minio": "^8.0.6",
"pino": "^10.3.1",
"postgres": "^3.4.3",
@@ -335,9 +336,9 @@
}
},
"node_modules/lib0": {
"version": "1.0.0-rc.23",
"resolved": "https://registry.npmjs.org/lib0/-/lib0-1.0.0-rc.23.tgz",
"integrity": "sha512-JPomcbwgKoTIDoXP61DFZV+Yvkw8bCyQhr9QYxps0fmHzsliEw+mhbUP1/nyVmk6ugzapJw1hUpFEMFRA4sRIg==",
"version": "1.0.0-rc.25",
"resolved": "https://registry.npmjs.org/lib0/-/lib0-1.0.0-rc.25.tgz",
"integrity": "sha512-UVxr56D1kVTx8P2Om5cAWXg2Pawk7JiOkCio+GLwlHcMko3FRE5xklOKB+t+fZSLA4ZLcVTZUKEIXPd9cRhEJg==",
"license": "MIT",
"bin": {
"0ecdsa-generate-keypair": "src/bin/0ecdsa-generate-keypair.js",
src/yhub-server/package.json
+2 -1
View File
@@ -6,7 +6,8 @@
"start": "node server.js"
},
"dependencies": {
"@y/hub": "0.4.0",
"@y/hub": "0.5.0",
"@y/y": "14.0.0-rc.24",
"jose": "6.2.8",
"minio": "8.0.7"
},
src/yhub-server/server.js
+126 -352
View File
@@ -1,15 +1,21 @@
import { createHash, randomUUID } from 'node:crypto';
import { readFileSync } from 'node:fs';
import { createHash } from 'node:crypto';
import { createApiEndpoint, createAuthPlugin, createYHub, logger } from '@y/hub';
import {
apiError,
createApiEndpoint,
createAuthPlugin,
createYHub,
} from '@y/hub';
import { createRemoteJWKSet, jwtVerify } from 'jose';
import { Client as S3Client } from 'minio';
// mirror y-provider's env.ts secret-file support
const secret = (name, dflt) =>
process.env[`${name}_FILE`]
? readFileSync(process.env[`${name}_FILE`], 'utf8').trim()
: process.env[name] || dflt;
import { secret } from './env.js';
// legacy Django/S3 document store — see migration.js and README.md
import {
SOFT_MIGRATION,
fullMigrate,
isTransientFailure,
maybeMigrate,
} from './migration.js';
const PORT = Number(process.env.PORT || 3002);
const REDIS = process.env.REDIS;
@@ -44,61 +50,6 @@ const EMPTY_YDOC = new Uint8Array([0, 0]);
// websocket path.
const MAX_CREATE_BYTES = 10 * 1024 * 1024;
// Soft migration (see README.md): legacy documents live in Django's S3 media
// bucket as UTF-8 base64 of a raw Yjs update at key `{docid}/file`. With
// SOFT_MIGRATION=true, the first access to a room yhub does not know yet
// fetches that object, seeds the room with it (attributed to 'system'), and
// only then admits the connection.
const SOFT_MIGRATION = process.env.SOFT_MIGRATION === 'true';
const AWS_S3_ENDPOINT_URL = process.env.AWS_S3_ENDPOINT_URL;
const AWS_S3_ACCESS_KEY_ID = secret('AWS_S3_ACCESS_KEY_ID');
const AWS_S3_SECRET_ACCESS_KEY = secret('AWS_S3_SECRET_ACCESS_KEY');
const AWS_S3_REGION_NAME = process.env.AWS_S3_REGION_NAME;
// Django's default bucket name (impress settings.py) — prod overrides it
const AWS_STORAGE_BUCKET_NAME =
process.env.AWS_STORAGE_BUCKET_NAME || 'impress-media-storage';
// base64 inflates 3 bytes to 4 — cap the streamed read at the encoded size of
// MAX_CREATE_BYTES (the same effective limit as create-ydoc) plus padding slack
const MAX_LEGACY_B64_BYTES = Math.ceil(MAX_CREATE_BYTES / 3) * 4 + 1024;
const S3_FETCH_TIMEOUT_MS = 10000;
const MIGRATE_LOCK_TTL_MS = 30000;
const MAX_CONCURRENT_SEEDS = 4;
if (
SOFT_MIGRATION &&
(!AWS_S3_ENDPOINT_URL || !AWS_S3_ACCESS_KEY_ID || !AWS_S3_SECRET_ACCESS_KEY)
) {
// fail at boot instead of as an opaque 401 storm on first connect
throw new Error(
'SOFT_MIGRATION=true requires AWS_S3_ENDPOINT_URL, AWS_S3_ACCESS_KEY_ID and AWS_S3_SECRET_ACCESS_KEY',
);
}
const s3 = SOFT_MIGRATION
? (() => {
const url = new URL(AWS_S3_ENDPOINT_URL);
if (url.pathname !== '/' && url.pathname !== '') {
// boto3 accepts path-prefixed endpoints but the minio client cannot
// address a base path — dropping it silently would probe the wrong
// keys and "migrate" every doc as empty
throw new Error('AWS_S3_ENDPOINT_URL must not contain a path');
}
return new S3Client({
endPoint: url.hostname,
port:
url.port !== ''
? Number(url.port)
: url.protocol === 'https:'
? 443
: 80,
useSSL: url.protocol === 'https:',
accessKey: AWS_S3_ACCESS_KEY_ID,
secretKey: AWS_S3_SECRET_ACCESS_KEY,
...(AWS_S3_REGION_NAME ? { region: AWS_S3_REGION_NAME } : {}),
});
})()
: null;
const migrationLog = logger.child({ module: 'soft-migration' });
// Public keys verifying the RS256 admin tokens Django issues (JWTService).
// Lazily fetched on first use; jose caches the keys and refetches on unknown
// "kid", so Django can rotate the signing key without a yhub restart.
@@ -122,273 +73,6 @@ const backendFetch = async (path, { cookie, origin }) => {
return res.json();
};
// Legacy Django document store: object `{docid}/file`, body = UTF-8 text that
// is the base64 encoding of a raw Yjs update. Returns null when the object
// does not exist — a document that never had content saved, e.g. brand new.
// Throws on any other failure (network, auth, timeout, oversize); corrupt
// base64 decodes leniently to garbage that patchYdoc later rejects.
const fetchLegacyDoc = async (docid) => {
let stream = null;
let cancelTimeout = () => {};
// minio 8 takes no AbortSignal — race a timer that also destroys the body
// stream once reading, so a stalled transfer cannot hold the ws upgrade
const timeout = new Promise((_, reject) => {
const timer = setTimeout(() => {
const err = new Error(
`s3 fetch timed out after ${S3_FETCH_TIMEOUT_MS}ms`,
);
err.transient = true; // a slow S3 may recover — cache the failure briefly
stream?.destroy(err);
reject(err);
}, S3_FETCH_TIMEOUT_MS);
cancelTimeout = () => clearTimeout(timer);
});
try {
let objPromise;
try {
objPromise = s3.getObject(AWS_STORAGE_BUCKET_NAME, `${docid}/file`);
stream = await Promise.race([objPromise, timeout]);
} catch (err) {
if (err?.code === 'NoSuchKey') return null;
// if the timeout won the race, getObject may still resolve later —
// destroy the late-arriving response stream, otherwise its never-read
// socket leaks (minio 8 sets no request timeout and cannot abort)
objPromise?.then((s) => s.destroy(err), () => {});
throw err;
}
const body = await Promise.race([
new Promise((resolve, reject) => {
const chunks = [];
let received = 0;
stream.on('data', (chunk) => {
received += chunk.byteLength;
if (received > MAX_LEGACY_B64_BYTES) {
stream.destroy(
new Error(`legacy object exceeds the ${MAX_LEGACY_B64_BYTES}B cap`),
);
return;
}
chunks.push(chunk);
});
stream.on('error', reject);
stream.on('end', () => resolve(Buffer.concat(chunks)));
}),
timeout,
]);
const decoded = Buffer.from(body.toString('utf8'), 'base64');
if (decoded.byteLength > MAX_CREATE_BYTES) {
throw new Error(
`decoded legacy update (${decoded.byteLength}B) exceeds the ${MAX_CREATE_BYTES}B cap`,
);
}
// compute-task schema requires an exact Uint8Array (lib0 compares the
// constructor) — re-view the Buffer without copying
return new Uint8Array(
decoded.buffer,
decoded.byteOffset,
decoded.byteLength,
);
} finally {
cancelTimeout();
}
};
// Quick existence check: does yhub already have content for this room?
// Sequenced cheapest-first: a persisted postgres row (bare SELECT, no blob
// columns; rows are never deleted, so a hit is always safe) — then the valkey
// stream (only ydoc:update:v1 counts: awareness and auth-check messages share
// the stream but carry no content) — then the SELECT again, which closes the
// store-before-trim compaction race (and the worst case of a miss is only a
// redundant, idempotent re-seed).
const ydocExists = async (room) => {
if ((await yhub.persistence.retrieveDoc(room, {})).lastClock !== '0') {
return true;
}
const streams = await yhub.stream.getMessages([{ room, clock: '0' }]);
if ((streams[0]?.messages ?? []).some((m) => m.type === 'ydoc:update:v1')) {
return true;
}
return (await yhub.persistence.retrieveDoc(room, {})).lastClock !== '0';
};
// Per-docid migration verdicts, in-memory (per replica). 'exists' is monotone
// in normal operation — its TTL only bounds staleness after an operator
// manually wipes a room's yhub state (restart yhub after a wipe to drop the
// cache immediately). 'empty' (no S3 object) keeps never-edited docs and
// rechecks off S3; 'failed' breaks the retry-refetch storm a permanently
// corrupt object would otherwise sustain (y-websocket retries denied upgrades
// forever).
const VERDICT_TTL_MS = { exists: 600000, empty: 60000, failed: 300000 };
// transient failures (network blips, timeouts, S3 restarting) are cached just
// long enough to blunt a retry storm without turning a hiccup into a lockout
const TRANSIENT_TTL_MS = 15000;
const TRANSIENT_CODES = new Set([
'ECONNREFUSED',
'ECONNRESET',
'ETIMEDOUT',
'EHOSTUNREACH',
'ENETUNREACH',
'ENOTFOUND',
'EAI_AGAIN',
'EPIPE',
]);
const isTransient = (err) =>
err?.transient === true ||
TRANSIENT_CODES.has(err?.code) ||
TRANSIENT_CODES.has(err?.cause?.code);
const VERDICT_CACHE_MAX = 50000;
const verdicts = new Map(); // docid -> { verdict, error, expires }
const rememberVerdict = (docid, verdict, error = null, ttl = VERDICT_TTL_MS[verdict]) => {
// delete-then-set keeps Map insertion order ≈ recency, so the FIFO eviction
// drops the stalest entry — and re-setting an existing docid never evicts
// an unrelated one
if (!verdicts.delete(docid) && verdicts.size >= VERDICT_CACHE_MAX) {
verdicts.delete(verdicts.keys().next().value);
}
verdicts.set(docid, {
verdict,
error,
expires: Date.now() + ttl,
});
};
const inflightMigrations = new Map(); // docid -> Promise<void>
let activeSeeds = 0;
const migrate = async (room) => {
if (await ydocExists(room)) return 'exists';
// collapse cross-replica herds: one seeder per room, the rest wait and
// re-probe. The key sits outside yhub's scanned `:room:*` patterns.
const lockKey = `${REDIS_PREFIX}:softmigrate:${room.org}:${room.docid}:${room.branch}`;
const lockToken = randomUUID();
const redis = yhub.stream.redis;
const acquired = await redis.set(lockKey, lockToken, {
condition: 'NX',
expiration: { type: 'PX', value: MIGRATE_LOCK_TTL_MS },
});
try {
if (acquired == null) {
// another connection or replica is seeding — wait for its lock, then
// re-probe. If the doc is still absent (the holder crashed or its S3
// fetch failed), fall through and seed ourselves: duplicate seeds use
// byte-identical updates from one lineage and merge as CRDT no-ops.
const deadline = Date.now() + MIGRATE_LOCK_TTL_MS + 5000;
while (Date.now() < deadline && (await redis.exists(lockKey)) === 1) {
await new Promise((resolve) => setTimeout(resolve, 300));
}
if (await ydocExists(room)) return 'exists';
}
if (activeSeeds >= MAX_CONCURRENT_SEEDS) {
// fail fast under a herd of distinct cold docs — the client's retry
// backoff spreads the load. Probes above stay uncapped. noCache:
// momentary per-replica backpressure must deny once, not be cached as
// a failure — a slot frees up within seconds
const err = new Error('too many concurrent soft migrations');
err.noCache = true;
throw err;
}
activeSeeds++;
try {
const start = Date.now();
const update = await fetchLegacyDoc(room.docid);
if (update == null) {
migrationLog.info(
{ event: 'seed.empty', docid: room.docid },
'no legacy s3 object; room starts empty',
);
return 'empty';
}
// legacy content has no per-user history — attribute it to 'system'
// (the admin-JWT identity), marked so audits can tell migrated content
// apart from other system writes
const result = await yhub.computePool.patchYdoc(
{
update,
currentDoc: EMPTY_YDOC,
userid: 'system',
customAttributions: [{ k: 'migration', v: 's3' }],
},
{ room },
);
if (result == null) {
// structurally valid but no effective content — nothing to seed
migrationLog.info(
{ event: 'seed.empty', docid: room.docid },
'legacy s3 object has no effective content; room starts empty',
);
return 'empty';
}
await yhub.stream.addMessage(room, {
type: 'ydoc:update:v1',
contentmap: result.contentmap,
update: result.update,
});
migrationLog.info(
{
event: 'seed.ok',
docid: room.docid,
bytes: update.byteLength,
durationMs: Date.now() - start,
},
'seeded legacy doc from s3',
);
return 'exists';
} finally {
activeSeeds--;
}
} finally {
if (acquired != null) {
// compare-and-delete: if this seed outlived the lock TTL, another
// seeder holds a fresh lock — a bare DEL would release it under them
redis
.eval(
"if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) end",
{ keys: [lockKey], arguments: [lockToken] },
)
.catch(() => {});
}
}
};
// Resolves when the room is usable (already known, freshly seeded, or
// legitimately empty); rejects to deny access. Idempotent and safe to
// re-enter — it also runs on rechecks and default-purpose REST calls.
const maybeMigrate = async (room) => {
const cached = verdicts.get(room.docid);
if (cached != null && cached.expires > Date.now()) {
if (cached.verdict === 'failed') throw cached.error;
return;
}
let migration = inflightMigrations.get(room.docid);
if (migration == null) {
migration = migrate(room)
.then(
(verdict) => rememberVerdict(room.docid, verdict),
(err) => {
// logged here (once per attempt) rather than per denied connection:
// cached failures deny without new logs until the verdict expires
migrationLog.error(
{ event: 'seed.failed', err, docid: room.docid },
'soft migration failed; denying access',
);
if (err?.noCache !== true) {
// transient failures get a short TTL so a hiccup cannot lock a
// doc out for the full poison-object window
rememberVerdict(
room.docid,
'failed',
err,
isTransient(err) ? TRANSIENT_TTL_MS : VERDICT_TTL_MS.failed,
);
}
throw err;
},
)
.finally(() => inflightMigrations.delete(room.docid));
inflightMigrations.set(room.docid, migration);
}
return migration;
};
const auth = createAuthPlugin({
// uws req is only valid synchronously — read headers AND query before first await.
async readAuthInfo(req) {
@@ -420,9 +104,20 @@ const auth = createAuthPlugin({
return payload.admin === true
? { userid: 'system', admin: true }
: null;
} catch {
// bad signature / expired / wrong (or missing) audience / JWKS
// unreachable — fail closed
} catch (err) {
// jose tags every token-validation failure with an `ERR_J…` code (bad
// signature, expired, wrong or missing audience) — those are permanent,
// fail closed. A JWKS fetch that times out or never connects has no
// such code (or ERR_JWKS_TIMEOUT): the token may be perfectly valid and
// we simply cannot check it, so report it as retryable instead of
// accusing the caller of forging it.
if (
err?.code === 'ERR_JWKS_TIMEOUT' ||
typeof err?.code !== 'string' ||
!err.code.startsWith('ERR_J')
) {
throw apiError(503, 'Token verification keys are unavailable');
}
return null;
}
}
@@ -437,11 +132,14 @@ const auth = createAuthPlugin({
return { userid: String(user.id), cookie, origin }; // MUST be string (yhub server.js:667)
} catch (err) {
// Only a genuine "not signed in" falls back to the anonymous identity.
// On backend failure (5xx/network) fail closed: a signed-in editor
// authorized under an anon userid would be invisible to the targeted
// reset-connections recheck (users: [<uuid>]) for the connection's
// whole lifetime.
if (err?.status !== 401 && err?.status !== 403) return null;
// On backend failure (5xx/network) still refuse to admit the connection —
// a signed-in editor authorized under an anon userid would be invisible
// to the targeted reset-connections recheck (users: [<uuid>]) for the
// connection's whole lifetime — but report it as retryable rather than as
// an authentication failure the client should give up on.
if (err?.status !== 401 && err?.status !== 403) {
throw apiError(503, 'Authentication backend is unavailable');
}
// anonymous (public docs): stable per-session id — random ids would mint a new
// permanent attribution identity per reconnect
const anon = createHash('sha256')
@@ -467,8 +165,14 @@ const auth = createAuthPlugin({
let doc;
try {
doc = await backendFetch(`/api/v1.0/documents/${docid}/`, authInfo);
} catch {
return null;
} catch (err) {
// the backend answered "no": a real, permanent denial (403 Forbidden)
if (err?.status === 401 || err?.status === 403 || err?.status === 404) {
return null;
}
// it did not answer at all — say so, so the caller retries instead of
// reading a 5xx or a network blip as a permission decision
throw apiError(503, 'Document authorization backend is unavailable');
}
if (!doc.abilities?.retrieve) {
return null;
@@ -480,12 +184,21 @@ const auth = createAuthPlugin({
// postgres and the stream from clock 0) is guaranteed to include the
// seed. Runs only for authorized readers. A missing S3 object is the
// brand-new-document case and allows an empty room; a real S3/compute
// failure denies access — an opaque 401 the client retries with
// backoff.
// failure denies access, either way already logged (once per attempt)
// inside maybeMigrate.
try {
await maybeMigrate({ org, docid, branch });
} catch {
return null; // already logged (once per attempt) in maybeMigrate
// `yhub` is declared at the bottom of this file — safe: auth callbacks
// only fire once the server is up, i.e. after that assignment
await maybeMigrate(yhub, { org, docid, branch });
} catch (err) {
// A corrupt or oversized legacy object will fail the same way forever,
// so that denial is permanent (403). An S3 timeout, a network blip or
// momentary seed backpressure will not — 503 tells the caller to come
// back rather than to treat the document as unreadable.
if (isTransientFailure(err)) {
throw apiError(503, 'Legacy document store is unavailable');
}
return null;
}
}
return doc.abilities.update ? 'rw' : 'r';
@@ -527,6 +240,66 @@ const api = [
},
},
}),
// POST /collaboration/migrate/v1/{org}/{docid} — replay a document's full
// legacy version history from the S3 media bucket into yhub (see README.md).
// Backend-internal, like reset-connections: gated to the admin token via the
// 'migrate' access purpose, since it writes history and reads the legacy
// store.
createApiEndpoint('migrate', {
accessPurpose: 'migrate',
post: {
handler: async (req) => {
if (req.org !== ORG) {
return jsonResponse(400, { error: 'Unknown org' });
}
if (!UUID4.test(req.docid)) {
return jsonResponse(400, { error: 'Room name is invalid' });
}
if (req.branch !== 'main') {
// the legacy store is branchless: `{docid}/file` is the main branch
return jsonResponse(400, { error: 'Unknown branch' });
}
if (!SOFT_MIGRATION) {
// the flag is what configures the S3 client (migration.js)
return jsonResponse(503, { error: 'Legacy store is not configured' });
}
// ?force=true replays a document that is already in the migrated set.
// Only safe while its clock-0 row is still there: once compaction has
// folded that row away, a second replay attributes the same content a
// second time and the activity timestamps become ambiguous.
const { status, ...stats } = await fullMigrate(req.yhub, req.room, {
force: req.query.force === 'true',
});
if (status === 'already') {
return jsonResponse(200, {
message: 'Already migrated',
migrated: false,
});
}
if (status === 'empty') {
// brand-new documents never had a legacy object; nothing to replay
// and nothing wrong — a backfill driver treats this as done
return jsonResponse(200, {
message: 'No legacy document in s3',
migrated: false,
...stats,
});
}
if (status === 'nothing') {
return jsonResponse(200, {
message: 'No usable content in the legacy versions',
migrated: false,
...stats,
});
}
return jsonResponse(200, {
message: 'Migration completed',
migrated: true,
...stats,
});
},
},
}),
// POST /collaboration/create-ydoc/v1/{org}/{docid} — create a document's
// initial Yjs state from a RAW binary update (`Y.encodeStateAsUpdate` /
// pycrdt `get_update()` output) posted as application/octet-stream. Unlike
@@ -561,8 +334,9 @@ const api = [
body.byteLength,
);
if (update.byteLength > MAX_CREATE_BYTES) {
// 413 is missing from yhub's status-line map, so the reason phrase
// is empty ("HTTP/1.1 413 ") — legal, and callers switch on the code
// 413 is missing from yhub's status-line map (503 was added in
// 0.5.0, 413 was not), so the reason phrase is empty
// ("HTTP/1.1 413 ") — legal, and callers switch on the code
return jsonResponse(413, { error: 'Update too large' });
}
// <= 3 bytes is yhub's "no effective content" convention (an empty
@@ -630,8 +404,8 @@ const api = [
}),
];
// the instance is referenced by the soft-migration helpers above — safe: auth
// callbacks only fire once the server is up, i.e. after this assignment
// referenced by getAccessType above — safe: auth callbacks only fire once the
// server is up, i.e. after this assignment
const yhub = await createYHub({
redis: {
url: REDIS,
@@ -647,6 +421,6 @@ const yhub = await createYHub({
server: { port: PORT, auth, api, apiPrefix: 'collaboration' },
worker: { taskConcurrency: 5 },
// TODO(yhub): worker.events.docUpdate could push snapshots to Django and replace the
// client useSaveDoc PATCH flow — blocked upstream: the payload is a DocTable without
// room/org/docid (yhub src/index.js:90); needs an upstream change first.
// client useSaveDoc PATCH flow. No longer blocked upstream — yhub 0.5.0 adds `room`
// to the event payload, which was the missing piece.
});