diff --git a/CHANGELOG.md b/CHANGELOG.md index 3757e6f96..3a3cb9ca1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -77,7 +77,7 @@ and this project adheres to collide with the real per-version times the migrate endpoint writes) before the connection is admitted. A missing S3 object means a brand-new document and yields an empty room. Seeding never decides access: a legacy object that - cannot be migrated (undecodable or oversized) opens as a new document, logged + cannot be migrated (it does not decode) opens as a new document, logged per access, since no retry could fix it and refusing would make the document permanently unopenable. Every other failure — an unreachable store, but also any refusal from S3 such as `AccessDenied` on a rotated key or a wrong bucket @@ -88,10 +88,14 @@ and this project adheres to stack through `env.d/development/yhub`, the collaboration server's own environment file. The bucket it reads is configured under `LEGACY_S3_*` (`_ENDPOINT_URL`, `_ACCESS_KEY_ID`, `_SECRET_ACCESS_KEY`, `_REGION_NAME`, - `_BUCKET_NAME`), a set of its own and not the backend's `AWS_S3_*`: this is - the bucket the collaboration server migrates *out of*, while the one it will - persist *into* when the yhub S3 persistence plugin is enabled is a separate - bucket that may well sit on another provider with credentials of its own + `_BUCKET_NAME`, `_SIGNATURE_VERSION`), a set of its own and not the backend's + `AWS_S3_*`: this is the bucket the collaboration server migrates *out of*, + while the one it will persist *into* when the yhub S3 persistence plugin is + enabled is a separate bucket that may well sit on another provider with + credentials of its own. It is read with the AWS SDK for JavaScript v3, whose + signature version is configurable (`s3v4` by default, as in Django) because a + provider expecting another one answers 403, which reads exactly like wrong + credentials - ✨(collaboration) add a migrate endpoint on yhub: `POST /collaboration/migrate/v1/docs/{id}` replays a document's **full** legacy version history from the versioned S3 media bucket into a diff --git a/src/helm/impress/README.md b/src/helm/impress/README.md index d20b05479..219eadd90 100644 --- a/src/helm/impress/README.md +++ b/src/helm/impress/README.md @@ -383,6 +383,7 @@ | `yhub.envVars.LEGACY_S3_SECRET_ACCESS_KEY` | Required by SOFT_MIGRATION, secret of the key above (or LEGACY_S3_SECRET_ACCESS_KEY_FILE) | | | `yhub.envVars.LEGACY_S3_REGION_NAME` | Region of the legacy bucket, when its provider needs one | | | `yhub.envVars.LEGACY_S3_BUCKET_NAME` | Name of the legacy Django media bucket (default: impress-media-storage) | | +| `yhub.envVars.LEGACY_S3_SIGNATURE_VERSION` | How the calls to the legacy bucket are signed, s3v4 or v4 (default: s3v4) | | | `yhub.envVars.BY_VALUE` | Example environment variable by setting value directly | | | `yhub.envVars.FROM_CONFIGMAP.configMapKeyRef.name` | Name of a ConfigMap when configuring env vars from a ConfigMap | | | `yhub.envVars.FROM_CONFIGMAP.configMapKeyRef.key` | Key within a ConfigMap when configuring env vars from a ConfigMap | | diff --git a/src/helm/impress/values.yaml b/src/helm/impress/values.yaml index 906f849c0..a4a8b0829 100644 --- a/src/helm/impress/values.yaml +++ b/src/helm/impress/values.yaml @@ -1029,6 +1029,7 @@ yhub: ## @extra yhub.envVars.LEGACY_S3_SECRET_ACCESS_KEY Required by SOFT_MIGRATION, secret of the key above (or LEGACY_S3_SECRET_ACCESS_KEY_FILE) ## @extra yhub.envVars.LEGACY_S3_REGION_NAME Region of the legacy bucket, when its provider needs one ## @extra yhub.envVars.LEGACY_S3_BUCKET_NAME Name of the legacy Django media bucket (default: impress-media-storage) + ## @extra yhub.envVars.LEGACY_S3_SIGNATURE_VERSION How the calls to the legacy bucket are signed, s3v4 or v4 (default: s3v4) ## @extra yhub.envVars.BY_VALUE Example environment variable by setting value directly ## @extra yhub.envVars.FROM_CONFIGMAP.configMapKeyRef.name Name of a ConfigMap when configuring env vars from a ConfigMap ## @extra yhub.envVars.FROM_CONFIGMAP.configMapKeyRef.key Key within a ConfigMap when configuring env vars from a ConfigMap diff --git a/src/yhub-server/README.md b/src/yhub-server/README.md index cad57347d..d3500a79c 100644 --- a/src/yhub-server/README.md +++ b/src/yhub-server/README.md @@ -296,8 +296,8 @@ documents into yhub lazily, on first access: then the valkey stream (uncompacted `ydoc:update:v1` messages), then the `SELECT` again to close the compaction race. Verdicts are cached in-process (existing docs 10 min, empty docs 60 s, failures 5 min). -2. If the room is unknown, the legacy object is fetched from S3 (10 s - timeout, 10 MiB decoded cap — the same limit as `create-ydoc`), decoded, +2. If the room is unknown, the legacy object is fetched from S3 whole, + whatever its size (10 s timeout for the request and its body), decoded, diffed through yhub's compute pool and appended to the room's stream — attributed to the `system` identity with a `migration=s3` custom attribution. This completes before the websocket upgrade resolves, so the @@ -350,7 +350,7 @@ Guarantees and failure behavior: guessing wrong the other way costs the document. A cached failure verdict prevents retry storms from hammering S3 — permanent - failures (corrupt/oversized objects) for 5 minutes, transient ones (network + failures (objects that do not decode) for 5 minutes, transient ones (network errors, timeouts) for 15 seconds, and per-replica seed backpressure (more than 20 concurrent seeds) is not cached at all, so the client's next retry goes through. @@ -369,12 +369,29 @@ Guarantees and failure behavior: Configuration: `LEGACY_S3_ENDPOINT_URL`, `LEGACY_S3_ACCESS_KEY_ID`, `LEGACY_S3_SECRET_ACCESS_KEY` (both with `*_FILE` indirection), optional -`LEGACY_S3_REGION_NAME`, and `LEGACY_S3_BUCKET_NAME` (defaults to Django's dev -default `impress-media-storage`; production uses a different bucket name and -must set it explicitly). The server refuses to boot when the flag is set -without endpoint and credentials. In development they come, like everything -else this server reads, from `env.d/development/yhub` (and `yhub.local`, which -is not committed — `make create-env-local-files` creates it). +`LEGACY_S3_REGION_NAME` (`us-east-1` when unset, which every S3-compatible +provider answers to), `LEGACY_S3_SIGNATURE_VERSION` (see below), and +`LEGACY_S3_BUCKET_NAME` (defaults to Django's dev default +`impress-media-storage`; production uses a different bucket name and must set +it explicitly). The server refuses to boot when the flag is set without +endpoint and credentials. In development they come, like everything else this +server reads, from `env.d/development/yhub` (and `yhub.local`, which is not +committed — `make create-env-local-files` creates it). + +The bucket is read with the **AWS SDK for JavaScript v3** +(`@aws-sdk/client-s3`), the same library family boto3 is to Django, so the +provider quirks the backend already deals with apply here too. Two settings +follow from that: + +- `LEGACY_S3_SIGNATURE_VERSION` — the counterpart of Django's + `AWS_S3_SIGNATURE_VERSION`, since a provider expecting the other signature + answers `403`, which reads exactly like wrong credentials. It defaults to + `s3v4` and accepts `s3v4` or `v4`. **SigV2 (boto3's `s3`) is not available**: + the AWS SDK v3 dropped it, so asking for it fails at boot instead of signing + the other way and being bounced, +- addressing style is chosen from the endpoint: path style (`{host}/{bucket}`) + everywhere but `amazonaws.com`, which prefers virtual-host style. Self-hosted + providers have no per-bucket DNS record, so path style is what they need. The prefix is deliberate: these name **the bucket this server migrates out of**, which is the backend's media bucket and not the one yhub will persist diff --git a/src/yhub-server/migration.js b/src/yhub-server/migration.js index 0235b9f75..ea248e638 100644 --- a/src/yhub-server/migration.js +++ b/src/yhub-server/migration.js @@ -22,9 +22,13 @@ import { randomUUID } from 'node:crypto'; +import { + GetObjectCommand, + ListObjectVersionsCommand, + S3Client, +} from '@aws-sdk/client-s3'; import { logger } from '@y/hub'; import * as Y from '@y/y'; -import { Client as S3Client } from 'minio'; import { secret } from './env.js'; @@ -42,13 +46,18 @@ const LEGACY_S3_REGION_NAME = process.env.LEGACY_S3_REGION_NAME; // Django's default bucket name (impress settings.py) — prod overrides it const LEGACY_S3_BUCKET_NAME = process.env.LEGACY_S3_BUCKET_NAME || 'impress-media-storage'; -// the same limit create-ydoc applies to a posted update in server.js: one -// legacy snapshot handed to a compute worker, or written to the stream as a -// single message -const MAX_LEGACY_BYTES = 10 * 1024 * 1024; -// base64 inflates 3 bytes to 4 — cap the streamed read at the encoded size of -// MAX_LEGACY_BYTES plus padding slack -const MAX_LEGACY_B64_BYTES = Math.ceil(MAX_LEGACY_BYTES / 3) * 4 + 1024; +// How the requests are signed, the counterpart of Django's +// AWS_S3_SIGNATURE_VERSION: a provider that expects the other one answers 403, +// which reads exactly like wrong credentials, so it is worth being explicit +// about. Only SigV4 is offered — see SIGNATURE_VERSIONS below. +const LEGACY_S3_SIGNATURE_VERSION = + process.env.LEGACY_S3_SIGNATURE_VERSION || 's3v4'; +// What that variable accepts, mapped to what it means for the client. The AWS +// SDK v3 signs with SigV4 and dropped SigV2 altogether, so the spellings of +// SigV4 are the whole set: a value asking for SigV2 (`s3`, boto3's other +// choice) is refused at boot rather than silently signed the other way and +// bounced by the provider as a credentials error. +const SIGNATURE_VERSIONS = { s3v4: 'sigv4', v4: 'sigv4' }; const S3_FETCH_TIMEOUT_MS = 10000; const MIGRATE_LOCK_TTL_MS = 30000; const MAX_CONCURRENT_SEEDS = 20; @@ -79,23 +88,36 @@ const s3 = SOFT_MIGRATION ? (() => { const url = new URL(LEGACY_S3_ENDPOINT_URL); if (url.pathname !== '/' && url.pathname !== '') { - // boto3 accepts path-prefixed endpoints but the minio client cannot - // address a base path — dropping it silently would probe the wrong - // keys and "migrate" every doc as empty + // boto3 accepts path-prefixed endpoints but an S3 endpoint cannot + // carry a base path — dropping it silently would probe the wrong keys + // and "migrate" every doc as empty throw new Error('LEGACY_S3_ENDPOINT_URL must not contain a path'); } + const signature = + SIGNATURE_VERSIONS[LEGACY_S3_SIGNATURE_VERSION.toLowerCase()]; + if (signature == null) { + throw new Error( + `LEGACY_S3_SIGNATURE_VERSION must be one of ${Object.keys( + SIGNATURE_VERSIONS, + ).join(', ')} (got "${LEGACY_S3_SIGNATURE_VERSION}")`, + ); + } return new S3Client({ - endPoint: url.hostname, - port: - url.port !== '' - ? Number(url.port) - : url.protocol === 'https:' - ? 443 - : 80, - useSSL: url.protocol === 'https:', - accessKey: LEGACY_S3_ACCESS_KEY_ID, - secretKey: LEGACY_S3_SECRET_ACCESS_KEY, - ...(LEGACY_S3_REGION_NAME ? { region: LEGACY_S3_REGION_NAME } : {}), + endpoint: url.origin, + // required by the sdk even where the provider ignores it; us-east-1 is + // what every S3-compatible implementation answers to by default + region: LEGACY_S3_REGION_NAME || 'us-east-1', + credentials: { + accessKeyId: LEGACY_S3_ACCESS_KEY_ID, + secretAccessKey: LEGACY_S3_SECRET_ACCESS_KEY, + }, + // `sigv4` today, and the client is built from the setting rather than + // from the default so that the value is what decides + authSchemePreference: [`aws.auth#${signature}`], + // Virtual-host style addresses a bucket as `{bucket}.{host}`, which + // needs a DNS record self-hosted providers do not have. AWS is the one + // endpoint that prefers it — and the one deprecating path style. + forcePathStyle: !/(^|\.)amazonaws\.com$/i.test(url.hostname), }); })() : null; @@ -114,93 +136,60 @@ const migrateLockKey = (yhub, room) => // content twice (see fullMigrate). const migratedSetKey = (yhub) => `${yhub.stream.prefix}:migrated:v1`; +// An aborted request surfaces as whatever the sdk or the body stream raises +// when the socket goes away ("aborted", TimeoutError, …). Say what actually +// happened instead, and leave it unmarked so it stays retryable — a slow S3 +// may well recover. +const asTimeout = (err, signal, what, ms) => + signal.aborted ? new Error(`${what} timed out after ${ms}ms`) : err; + // Legacy Django document store: object `{docid}/file`, body = UTF-8 text that // is the base64 encoding of a raw Yjs update. With `versionId`, reads that // specific object version instead of the current one. Returns null when the // object (or version) does not exist — a document that never had content -// saved, e.g. brand new. Throws on any other failure (network, auth, timeout, -// oversize); corrupt base64 decodes leniently to garbage that the callers -// reject. +// saved, e.g. brand new. Throws on any other failure (network, auth, timeout); +// corrupt base64 decodes leniently to garbage that the callers reject. const fetchLegacyDoc = async (docid, versionId = null) => { - let stream = null; - let cancelTimeout = () => {}; - // minio 8 takes no AbortSignal — race a timer that also destroys the body - // stream once reading, so a stalled transfer cannot hold the ws upgrade - const timeout = new Promise((_, reject) => { - const timer = setTimeout(() => { - // unmarked, so it counts as retryable: a slow S3 may recover - const err = new Error( - `s3 fetch timed out after ${S3_FETCH_TIMEOUT_MS}ms`, - ); - stream?.destroy(err); - reject(err); - }, S3_FETCH_TIMEOUT_MS); - cancelTimeout = () => clearTimeout(timer); - }); + // One budget for the whole read, headers and body alike: the sdk aborts the + // request when it fires and the body stream dies with it, so a stalled + // transfer cannot hold the ws upgrade open. + const abortSignal = AbortSignal.timeout(S3_FETCH_TIMEOUT_MS); + let body; try { - let objPromise; - try { - objPromise = s3.getObject( - LEGACY_S3_BUCKET_NAME, - `${docid}/file`, - // minio stringifies the whole opts object into the query — pass - // undefined, not {}, so the unversioned read stays byte-identical - versionId != null ? { versionId } : undefined, - ); - stream = await Promise.race([objPromise, timeout]); - } catch (err) { - // NoSuchVersion: the version vanished between listing and reading - if (err?.code === 'NoSuchKey' || err?.code === 'NoSuchVersion') { - return null; - } - // if the timeout won the race, getObject may still resolve later — - // destroy the late-arriving response stream, otherwise its never-read - // socket leaks (minio 8 sets no request timeout and cannot abort) - objPromise?.then( - (s) => s.destroy(err), - () => {}, - ); - throw err; - } - const body = await Promise.race([ - new Promise((resolve, reject) => { - const chunks = []; - let received = 0; - stream.on('data', (chunk) => { - received += chunk.byteLength; - if (received > MAX_LEGACY_B64_BYTES) { - const err = new Error( - `legacy object exceeds the ${MAX_LEGACY_B64_BYTES}B cap`, - ); - err.permanent = true; // the object will be this big next time too - stream.destroy(err); - return; - } - chunks.push(chunk); - }); - stream.on('error', reject); - stream.on('end', () => resolve(Buffer.concat(chunks))); + ({ Body: body } = await s3.send( + new GetObjectCommand({ + Bucket: LEGACY_S3_BUCKET_NAME, + Key: `${docid}/file`, + ...(versionId != null ? { VersionId: versionId } : {}), }), - timeout, - ]); - const decoded = Buffer.from(body.toString('utf8'), 'base64'); - if (decoded.byteLength > MAX_LEGACY_BYTES) { - const err = new Error( - `decoded legacy update (${decoded.byteLength}B) exceeds the ${MAX_LEGACY_BYTES}B cap`, - ); - err.permanent = true; // the object will be this big next time too - throw err; + { abortSignal }, + )); + } catch (err) { + // NoSuchVersion: the version vanished between listing and reading. + // NotFound is the bare 404 some S3-compatible providers answer with + // instead; a missing *bucket* has a name of its own and is not caught + // here — that one is a misconfiguration, not an absent document. + if ( + err?.name === 'NoSuchKey' || + err?.name === 'NoSuchVersion' || + err?.name === 'NotFound' + ) { + return null; } - // compute-task schema requires an exact Uint8Array (lib0 compares the - // constructor) — re-view the Buffer without copying - return new Uint8Array( - decoded.buffer, - decoded.byteOffset, - decoded.byteLength, - ); - } finally { - cancelTimeout(); + throw asTimeout(err, abortSignal, 's3 fetch', S3_FETCH_TIMEOUT_MS); } + let encoded; + try { + // the object whole, whatever its size: it is one document's content, and + // refusing to read it is refusing to migrate that document at all + encoded = await body.transformToString('utf8'); + } catch (err) { + throw asTimeout(err, abortSignal, 's3 fetch', S3_FETCH_TIMEOUT_MS); + } + const decoded = Buffer.from(encoded, 'base64'); + // compute-task schema requires an exact Uint8Array (lib0 compares the + // constructor) — re-view the Buffer without copying + return new Uint8Array(decoded.buffer, decoded.byteOffset, decoded.byteLength); }; // Every version of the legacy object, oldest first. Delete markers are skipped @@ -208,36 +197,58 @@ const fetchLegacyDoc = async (docid, versionId = null) => { // the prefix — S3 has no exact-key version listing. const listLegacyVersions = async (docid) => { const key = `${docid}/file`; - const found = await new Promise((resolve, reject) => { - const versions = []; - const stream = s3.listObjects(LEGACY_S3_BUCKET_NAME, key, true, { - IncludeVersion: true, - }); - const timer = setTimeout(() => { - const err = new Error( - `s3 version listing timed out after ${S3_LIST_TIMEOUT_MS}ms`, + // one budget for the whole listing, however many pages it takes + const abortSignal = AbortSignal.timeout(S3_LIST_TIMEOUT_MS); + const found = []; + try { + let keyMarker; + let versionIdMarker; + let truncated = true; + while (truncated) { + const page = await s3.send( + new ListObjectVersionsCommand({ + Bucket: LEGACY_S3_BUCKET_NAME, + Prefix: key, + KeyMarker: keyMarker, + VersionIdMarker: versionIdMarker, + }), + { abortSignal }, ); - stream.destroy(err); - }, S3_LIST_TIMEOUT_MS); - stream.on('data', (obj) => { - if (obj.name === key && obj.isDeleteMarker !== true && obj.versionId) { - versions.push({ - versionId: String(obj.versionId), - // the moment S3 accepted the write: what the backend's version - // listing reports as `last_modified`, and what we attribute to - timestamp: obj.lastModified?.getTime() ?? 0, - }); + // delete markers record a deletion and carry no body; they come in a + // list of their own here, so reading `Versions` skips them by itself + for (const version of page.Versions ?? []) { + if (version.Key === key && version.VersionId) { + found.push({ + versionId: String(version.VersionId), + // the moment S3 accepted the write: what the backend's version + // listing reports as `last_modified`, and what we attribute to + timestamp: version.LastModified?.getTime() ?? 0, + }); + } } - }); - stream.on('error', (err) => { - clearTimeout(timer); - reject(err); - }); - stream.on('end', () => { - clearTimeout(timer); - resolve(versions); - }); - }); + truncated = page.IsTruncated === true; + keyMarker = page.NextKeyMarker; + versionIdMarker = page.NextVersionIdMarker; + } + } catch (err) { + const failure = asTimeout( + err, + abortSignal, + 's3 version listing', + S3_LIST_TIMEOUT_MS, + ); + migrationLog.error( + { + event: 'list_version.failed', + err: failure, + docid, + bucket: LEGACY_S3_BUCKET_NAME, + key, + }, + 'impossible to list object version', + ); + throw failure; + } // S3 lists a key's versions newest first; reverse to replay them in write // order. The sort is a stable safeguard across paginated listings — equal // timestamps keep S3's own ordering. diff --git a/src/yhub-server/package-lock.json b/src/yhub-server/package-lock.json index d12a212b5..267f09f21 100644 --- a/src/yhub-server/package-lock.json +++ b/src/yhub-server/package-lock.json @@ -6,10 +6,10 @@ "": { "name": "yhub-server", "dependencies": { + "@aws-sdk/client-s3": "3.1110.0", "@y/hub": "0.6.0", "@y/y": "14.0.0-rc.24", - "jose": "6.2.8", - "minio": "8.0.7" + "jose": "6.2.8" }, "devDependencies": { "nodemon": "3.1.14" @@ -18,6 +18,314 @@ "node": ">=22" } }, + "node_modules/@aws-sdk/checksums": { + "version": "3.1000.27", + "resolved": "https://registry.npmjs.org/@aws-sdk/checksums/-/checksums-3.1000.27.tgz", + "integrity": "sha512-insWOqKKNUrbN/dohEG7BJ0U5GkyqhjbMb/NHNaLUtq+7my2M8C4EnZZZoxMmXRqCC+P9dEr+KyJA2JGGzoKLg==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.7", + "@aws-sdk/types": "^3.974.3", + "@smithy/core": "^3.31.1", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/client-s3": { + "version": "3.1110.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/client-s3/-/client-s3-3.1110.0.tgz", + "integrity": "sha512-40xbEcWjdaYKlZ4/NvndIJ3LotAEQAvHVQ7Z4NVy4Z4xGRN7xXJlHI9bMh/4aMJQ++6h5W5sv+wqjfk0rEKOBg==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/checksums": "^3.1000.27", + "@aws-sdk/core": "^3.977.7", + "@aws-sdk/credential-provider-node": "^3.972.79", + "@aws-sdk/middleware-sdk-s3": "^3.972.73", + "@aws-sdk/signature-v4-multi-region": "^3.996.44", + "@aws-sdk/types": "^3.974.3", + "@smithy/core": "^3.31.1", + "@smithy/fetch-http-handler": "^5.6.13", + "@smithy/node-http-handler": "^4.9.13", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/core": { + "version": "3.977.7", + "resolved": "https://registry.npmjs.org/@aws-sdk/core/-/core-3.977.7.tgz", + "integrity": "sha512-I88Iov89NVmjSmJLKSv7Cn9M2J+a2942OkA8nZCbz+sl4ZeY4zEOcoLOrbt1GRfQ8zEQKnjAJdXixA3J/p1fDQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "^3.974.3", + "@aws-sdk/xml-builder": "^3.972.38", + "@aws/lambda-invoke-store": "^0.3.0", + "@smithy/core": "^3.31.1", + "@smithy/signature-v4": "^5.6.12", + "@smithy/types": "^4.16.1", + "bowser": "^2.11.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-env": { + "version": "3.972.68", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-env/-/credential-provider-env-3.972.68.tgz", + "integrity": "sha512-2a20A/IdNOwUvaDq91iqqS7BA0XlNMfW3iLGZGZLJv0EbUqhSxB0PIx4rQQqssvWj1uXImb3/UCCdHz/+1dOiA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.7", + "@aws-sdk/types": "^3.974.3", + "@smithy/core": "^3.31.1", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-http": { + "version": "3.972.70", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-http/-/credential-provider-http-3.972.70.tgz", + "integrity": "sha512-0yRem2Fs52r/Nn6UAqIlpjexfaYj8ziEozOe9tamtAVT/5bzFLKx8O2r7MaRqgS3hGKHIa1Jij9nKHSsNnb04A==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.7", + "@aws-sdk/types": "^3.974.3", + "@smithy/core": "^3.31.1", + "@smithy/fetch-http-handler": "^5.6.13", + "@smithy/node-http-handler": "^4.9.13", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-ini": { + "version": "3.973.13", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-ini/-/credential-provider-ini-3.973.13.tgz", + "integrity": "sha512-2M39DE02XpYYaSWYk/4AsImXYUU/1L2xmTMLUpMMWq7DfLv191/vCRy3baKtdr45AkJQyVgSjmuVOLm15SwrRQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.7", + "@aws-sdk/credential-provider-env": "^3.972.68", + "@aws-sdk/credential-provider-http": "^3.972.70", + "@aws-sdk/credential-provider-login": "^3.972.75", + "@aws-sdk/credential-provider-process": "^3.972.68", + "@aws-sdk/credential-provider-sso": "^3.973.12", + "@aws-sdk/credential-provider-web-identity": "^3.972.74", + "@aws-sdk/nested-clients": "^3.997.42", + "@aws-sdk/types": "^3.974.3", + "@smithy/core": "^3.31.1", + "@smithy/credential-provider-imds": "^4.4.16", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-login": { + "version": "3.972.75", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-login/-/credential-provider-login-3.972.75.tgz", + "integrity": "sha512-jaTESuJlQsoUZ44f/i2puyPt8VlF/dMMJ9HM3cStYtk7eKX4N9UWi83OLixUkoOJH3BwWlPLCq9YIK9nfWhVBg==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.7", + "@aws-sdk/nested-clients": "^3.997.42", + "@aws-sdk/types": "^3.974.3", + "@smithy/core": "^3.31.1", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-node": { + "version": "3.972.79", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-node/-/credential-provider-node-3.972.79.tgz", + "integrity": "sha512-RIw5dof1EHkWubrZzPC941CDtnFG1iAXsxbFgLkhdYZXHc4icU13c/uxSMI0J5eUx9bxa7LjfpdjfClBB1QsDA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/credential-provider-env": "^3.972.68", + "@aws-sdk/credential-provider-http": "^3.972.70", + "@aws-sdk/credential-provider-ini": "^3.973.13", + "@aws-sdk/credential-provider-process": "^3.972.68", + "@aws-sdk/credential-provider-sso": "^3.973.12", + "@aws-sdk/credential-provider-web-identity": "^3.972.74", + "@aws-sdk/types": "^3.974.3", + "@smithy/core": "^3.31.1", + "@smithy/credential-provider-imds": "^4.4.16", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-process": { + "version": "3.972.68", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-process/-/credential-provider-process-3.972.68.tgz", + "integrity": "sha512-nLP3Pda2MQTFJ25hKBMmUuB9Uv+bTZQNlufbeCwklP549Vwnkd8bRLJoCKp5k6xjmdyptrPrOfGOhN0mKuca8A==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.7", + "@aws-sdk/types": "^3.974.3", + "@smithy/core": "^3.31.1", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-sso": { + "version": "3.973.12", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-sso/-/credential-provider-sso-3.973.12.tgz", + "integrity": "sha512-EmgyyHn+f9WCcelp3L/vci+LGbX8GigWaVphRArjVo5Pktkr9YnLy/mQ6VDkDyBD72dtfRNTgHmD2ts4rTDXKQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.7", + "@aws-sdk/nested-clients": "^3.997.42", + "@aws-sdk/token-providers": "3.1108.0", + "@aws-sdk/types": "^3.974.3", + "@smithy/core": "^3.31.1", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-web-identity": { + "version": "3.972.74", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-web-identity/-/credential-provider-web-identity-3.972.74.tgz", + "integrity": "sha512-0YfczxGXF3RjGj8z7QG/Ho2HnLGKDHfPSHiTs47UU1U/+mmwISDN+rvGKt2zh+3FX8NdT4xd95LGBGyhQw2dgQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.7", + "@aws-sdk/nested-clients": "^3.997.42", + "@aws-sdk/types": "^3.974.3", + "@smithy/core": "^3.31.1", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/middleware-sdk-s3": { + "version": "3.972.73", + "resolved": "https://registry.npmjs.org/@aws-sdk/middleware-sdk-s3/-/middleware-sdk-s3-3.972.73.tgz", + "integrity": "sha512-oy7sRA5HvHcAvkcKX6F8RI240jcOf3c8y/Gqjs9qemIibdKQqGBIi0uwa+47ZRYqGLpdEO28TQU4G73yUzo06Q==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.7", + "@aws-sdk/signature-v4-multi-region": "^3.996.44", + "@aws-sdk/types": "^3.974.3", + "@smithy/core": "^3.31.1", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/nested-clients": { + "version": "3.997.42", + "resolved": "https://registry.npmjs.org/@aws-sdk/nested-clients/-/nested-clients-3.997.42.tgz", + "integrity": "sha512-XWRyon2MTHXD/zMoo0Mbge6Vwf+iE0qQaM/RyGO6NfZ9WukCFiQL27nQVZjYy2JwSIg+iXZxKOX95OBXqlSM4w==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.7", + "@aws-sdk/signature-v4-multi-region": "^3.996.44", + "@aws-sdk/types": "^3.974.3", + "@smithy/core": "^3.31.1", + "@smithy/fetch-http-handler": "^5.6.13", + "@smithy/node-http-handler": "^4.9.13", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/signature-v4-multi-region": { + "version": "3.996.44", + "resolved": "https://registry.npmjs.org/@aws-sdk/signature-v4-multi-region/-/signature-v4-multi-region-3.996.44.tgz", + "integrity": "sha512-ZSfQ35Qn4MhSY+A0Whyr+KBx+wJKZUyBsOrjB2pSHOafRzbFe47T8XcXM8hZqUAC69qnqIy0C9ArxTuud0CC2w==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "^3.974.3", + "@smithy/signature-v4": "^5.6.12", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/token-providers": { + "version": "3.1108.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/token-providers/-/token-providers-3.1108.0.tgz", + "integrity": "sha512-rI80zxDxGJ6904eC/YbjkdjY6JdaZvQ01kOmrMvw7cFQGIHo27fhnIVbMSVDS4T6foQImjxYSRoOu/uSJscXDw==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.7", + "@aws-sdk/nested-clients": "^3.997.42", + "@aws-sdk/types": "^3.974.3", + "@smithy/core": "^3.31.1", + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/types": { + "version": "3.974.3", + "resolved": "https://registry.npmjs.org/@aws-sdk/types/-/types-3.974.3.tgz", + "integrity": "sha512-ECAqfpNsef+7MO8qtR0h9KcFIBAygaE7Cm6UOiQl+ft+uVap+1G7bNEjs4mdJE2OnA4m6k7i8peH8uGIAsOMGw==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/xml-builder": { + "version": "3.972.38", + "resolved": "https://registry.npmjs.org/@aws-sdk/xml-builder/-/xml-builder-3.972.38.tgz", + "integrity": "sha512-grf7mzfVxBS5AlsuTvBN7uDpzqohFww9fRPCO+EBSUdvtsYMcPSKdz54h/7XiscqNcUM1Ae1MF7JLHmiYYuzbQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.16.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws/lambda-invoke-store": { + "version": "0.3.0", + "resolved": "https://registry.npmjs.org/@aws/lambda-invoke-store/-/lambda-invoke-store-0.3.0.tgz", + "integrity": "sha512-sl4Bm6yiMNYrZKkqqDFWN0UfnWhlS8ivKxrYl+6t0gCLrqr8y3B2IqZZbFRkfaVVp7C/baApyh71P+LeE1A2sQ==", + "license": "Apache-2.0", + "engines": { + "node": ">=18.0.0" + } + }, "node_modules/@nodable/entities": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/@nodable/entities/-/entities-3.0.0.tgz", @@ -108,6 +416,87 @@ "@redis/client": "^5.12.1" } }, + "node_modules/@smithy/core": { + "version": "3.33.0", + "resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.33.0.tgz", + "integrity": "sha512-uKbkxgqLyepQDZoq8aRSdUqD1ID//rOqG96ixBhp++O7vBtmwYM6fwldGhr9HJP0iYrdc7GP/AlgzPWEZIrNRg==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.17.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/credential-provider-imds": { + "version": "4.5.0", + "resolved": "https://registry.npmjs.org/@smithy/credential-provider-imds/-/credential-provider-imds-4.5.0.tgz", + "integrity": "sha512-2jsPi+7Zv2hSzD9IXR9D7DTqSn7mv4XalzRm+bESh53jiaUS3NKEUbpQFTJP0HhQy9qzZvluxQ3yS24zdRrqsA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.32.0", + "@smithy/types": "^4.17.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/fetch-http-handler": { + "version": "5.7.0", + "resolved": "https://registry.npmjs.org/@smithy/fetch-http-handler/-/fetch-http-handler-5.7.0.tgz", + "integrity": "sha512-W/exA8T0LEzCQtJ02w4IzaEQPIspgarqZprb7W8FwnYiDowgCrjl2fTQ6FvuSSUnJORuepBF81abmBJwqh+0XQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.32.0", + "@smithy/types": "^4.17.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/node-http-handler": { + "version": "4.11.0", + "resolved": "https://registry.npmjs.org/@smithy/node-http-handler/-/node-http-handler-4.11.0.tgz", + "integrity": "sha512-ssHIZsadPUA3lGdnoByxfnjtb9xPYQLvdfJRLKIwxOoa6tO1suG4sLFSsgd7D/CsvYd8QbBIuKTImuJha5l6aQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.33.0", + "@smithy/types": "^4.17.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/signature-v4": { + "version": "5.7.0", + "resolved": "https://registry.npmjs.org/@smithy/signature-v4/-/signature-v4-5.7.0.tgz", + "integrity": "sha512-hCynhm22wMJ8wTF9crcwu8mxggtUrSLLJgDcGUvYFBqpofxycYJCGKOMYg4xtPPFtgNiDJSYmhsWLTrcU/g59Q==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.32.0", + "@smithy/types": "^4.17.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/types": { + "version": "4.17.0", + "resolved": "https://registry.npmjs.org/@smithy/types/-/types-4.17.0.tgz", + "integrity": "sha512-Aw4joiM0ZdErpo39lCj8phT2lxoiKZV+KZzBxnnQhWVtU2Is/WffQSL04uUWRcXUse9Ln8vXZK6V/FwqRVnQpg==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, "node_modules/@y-crdt/yn": { "version": "0.1.4", "resolved": "https://registry.npmjs.org/@y-crdt/yn/-/yn-0.1.4.tgz", @@ -252,6 +641,12 @@ "readable-stream": "^3.4.0" } }, + "node_modules/bowser": { + "version": "2.14.1", + "resolved": "https://registry.npmjs.org/bowser/-/bowser-2.14.1.tgz", + "integrity": "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg==", + "license": "MIT" + }, "node_modules/brace-expansion": { "version": "5.0.9", "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.9.tgz", @@ -1067,6 +1462,12 @@ "nodetouch": "bin/nodetouch.js" } }, + "node_modules/tslib": { + "version": "2.8.1", + "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", + "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", + "license": "0BSD" + }, "node_modules/undefsafe": { "version": "2.0.5", "resolved": "https://registry.npmjs.org/undefsafe/-/undefsafe-2.0.5.tgz", diff --git a/src/yhub-server/package.json b/src/yhub-server/package.json index a4d33c681..368469697 100644 --- a/src/yhub-server/package.json +++ b/src/yhub-server/package.json @@ -8,10 +8,10 @@ "init-db": "node node_modules/@y/hub/bin/init-db.js" }, "dependencies": { + "@aws-sdk/client-s3": "3.1110.0", "@y/hub": "0.6.0", "@y/y": "14.0.0-rc.24", - "jose": "6.2.8", - "minio": "8.0.7" + "jose": "6.2.8" }, "devDependencies": { "nodemon": "3.1.14" diff --git a/src/yhub-server/server.js b/src/yhub-server/server.js index 16697c14d..2d4c10e18 100644 --- a/src/yhub-server/server.js +++ b/src/yhub-server/server.js @@ -16,7 +16,6 @@ import { jwtVerify, SignJWT, } from 'jose'; -import { Client as S3Client } from 'minio'; import { secret } from './env.js'; // legacy Django/S3 document store — see migration.js and README.md @@ -237,11 +236,11 @@ const backendFetch = async (path, { cookie, origin }) => { // Seeding never decides whether the caller may read the document — that is the // backend's answer alone. There are two ways this ends other than a seed: // -// the legacy object cannot be migrated (it does not decode, or it is bigger -// than we will load) — retrying will not change that, so the room opens as -// a new document. Refusing instead would lock a document nobody can repair -// from the outside. Logged per access, because the caller is now editing -// alongside legacy content that stayed behind in S3. +// the legacy object cannot be migrated (it does not decode) — retrying will +// not change that, so the room opens as a new document. Refusing instead +// would lock a document nobody can repair from the outside. Logged per +// access, because the caller is now editing alongside legacy content that +// stayed behind in S3. // the legacy store could not be reached (timeout, network, backpressure) — // the same request later may well succeed, so it answers 503 rather than // silently starting an empty document on top of content that exists.