mirror of
https://github.com/suitenumerique/docs.git
synced 2026-08-17 21:25:43 +02:00
📝(upgrade) complete the upgrade guide for yhub
The upgrade between version 5 and next version 6 is not simple. Lot of breaking changes are made and must be documented to ease the upgrade for the community.
This commit is contained in:
Manuel Raynaud
+210
-16
@@ -16,21 +16,160 @@ the following command inside your docker container:
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
- The endpoint `/api/v1.0/documents/{document_id}/content/`, added in 5.0.0, is
|
||||
removed, both its `GET` and its `PATCH`. The content of a document is now
|
||||
saved and served by the collaboration server, the editor exchanging it over
|
||||
the websocket, so nothing reads or writes it through the API anymore. If you
|
||||
integrate with Docs, stop calling this endpoint: the `content_patch` and
|
||||
`content_retrieve` abilities disappear from the document payload along with
|
||||
it. `/api/v1.0/documents/{document_id}/formatted-content/` is not affected.
|
||||
The `CONTENT_METADATA_CACHE_TIMEOUT` setting only tuned the cache of the
|
||||
removed `GET` and is no longer read, you can drop it from your configuration.
|
||||
- The JWKS of the resource server moved from `/api/{version}/jwks` to
|
||||
`/external_api/{version}/jwks`, alongside the rest of the resource server
|
||||
endpoints. `/api/{version}/jwks` now publishes the public key validating the
|
||||
tokens Docs issues to call external services. If you enabled the resource
|
||||
server (`OIDC_RESOURCE_SERVER_ENABLED`), update the JWKS URI declared to your
|
||||
OIDC provider accordingly.
|
||||
⚠️ This release replaces the collaboration server. The content of a document
|
||||
does not live in the object storage anymore, it lives in that server, and the
|
||||
`y-provider` that used to serve the websocket does not serve it. There is a new
|
||||
service to deploy, a database to create for it, and the existing documents have
|
||||
to be handed over to it: an instance that upgrades without doing so opens every
|
||||
one of its documents **empty**. The entries below start with the steps of that
|
||||
upgrade, in the order they are done, and end with the API changes.
|
||||
|
||||
- ⚠️ **A new service to deploy: the collaboration server**
|
||||
(`lasuite/impress-yhub`, listening on `3002`), which replaces the
|
||||
`y-provider` on everything under `/collaboration/`. It keeps the live state
|
||||
of the documents in Redis/Valkey and persists them to a PostgreSQL database
|
||||
of its own, so it needs both and there is nothing to default them to:
|
||||
|
||||
```yaml
|
||||
REDIS: redis://{redis-host}:6379/0
|
||||
POSTGRES: postgres://{user}:{password}@{postgres-host}:5432/yhub
|
||||
```
|
||||
|
||||
It also needs `COLLABORATION_BACKEND_BASE_URL`, the backend it asks about
|
||||
users and document access rights, and `COLLABORATION_SERVER_ORIGIN`, the
|
||||
origins allowed to open a websocket — the two the `y-provider` already had.
|
||||
Give it `Y_PROVIDER_API_KEY` as well, with the same value as the backend's:
|
||||
it is the header that exempts the collaboration server from the API
|
||||
throttling, and it calls the backend once per connection.
|
||||
|
||||
`REDIS_PREFIX` (default `yhub`) namespaces its keys when the Redis instance
|
||||
is shared with something else, and `YHUB_ORG` (default `docs`) names the
|
||||
organization the documents live under — the backend and the server must
|
||||
agree on it, a room of any other organization is refused. In the helm chart
|
||||
everything is under the `yhub` values key, enabled by default, and
|
||||
`yhub.worker.enabled` splits it into a server deployment and a worker
|
||||
deployment that scale on their own. `src/yhub-server/README.md` documents the
|
||||
rest of what it reads.
|
||||
|
||||
- ⚠️ **Its schema is not created when it starts.** The server never runs DDL:
|
||||
run the script it ships, `npm run init-db` (`node
|
||||
node_modules/@y/hub/bin/init-db.js` in the image), once before starting it
|
||||
and again after every upgrade that adds a table. It creates the database when
|
||||
it is missing, and it is idempotent, so re-running it is always safe. The
|
||||
helm chart runs it as a job (`yhub.initDb`, on by default), next to the
|
||||
backend migrate job and retrying while the PostgreSQL server does not answer
|
||||
— nothing in the chart creates that server. Until it has run, every document
|
||||
read fails with a `relation "..." does not exist` error.
|
||||
|
||||
- ⚠️ **The existing documents have to be migrated into it.** Until now the
|
||||
content of a document was a file in the media bucket, at key
|
||||
`{document-id}/file`; the collaboration server starts out knowing none of
|
||||
them, and the frontend no longer seeds content of its own. Two steps, in this
|
||||
order:
|
||||
|
||||
1. Turn `SOFT_MIGRATION=true` on the collaboration server **before letting
|
||||
anyone in**, and point it at the media bucket:
|
||||
`LEGACY_S3_ENDPOINT_URL` (no path), `LEGACY_S3_ACCESS_KEY_ID`,
|
||||
`LEGACY_S3_SECRET_ACCESS_KEY` (both with a `_FILE` variant),
|
||||
`LEGACY_S3_BUCKET_NAME` (defaults to `impress-media-storage`, the
|
||||
development name — production has to set it), and optionally
|
||||
`LEGACY_S3_REGION_NAME` and `LEGACY_S3_SIGNATURE_VERSION` (`s3v4` by
|
||||
default, `v4` for the providers wanting the other one; SigV2 is not
|
||||
available). Read-only, bucket-scoped credentials are enough, and on AWS
|
||||
they need `s3:ListBucket` beside `s3:GetObject` — without it a
|
||||
brand-new document reads as `403` instead of `404` and fails to open. A
|
||||
document is then seeded from its legacy snapshot the first time someone
|
||||
opens it. These are **not** the backend's `AWS_S3_*` settings: nothing
|
||||
here reads them, so a pod carrying both migrates out of the bucket named
|
||||
here and no other.
|
||||
2. Then backfill the corpus, which the lazy seeding never finishes on its own
|
||||
— a document nobody opens stays in S3 forever. `python manage.py
|
||||
migrate_documents` hands every document to the collaboration server, which
|
||||
replays its **full** S3 version history rather than its last snapshot, so
|
||||
`/documents/{id}/versions/` and the history the editor shows agree. It is
|
||||
bounded (`--concurrency`, `--rate`, `--limit`, `--created-before`),
|
||||
resumable and safe to re-run: what became of every document is recorded in
|
||||
the new `impress_document_migration` table, a document the server refused
|
||||
is left for a later `--retry-failed` run, and `--document-id` hands over a
|
||||
single one. `--dry-run` counts what a run would do.
|
||||
|
||||
`SOFT_MIGRATION` may only be turned off once that backfill has covered the
|
||||
corpus. Turning it off earlier loses nothing — what is migrated stays
|
||||
migrated — but an unmigrated document then opens as an *empty* room over
|
||||
content that is alive in S3.
|
||||
|
||||
Keep the media bucket, its objects and its versioning either way:
|
||||
`/documents/{id}/versions/` still serves the version history from there, and
|
||||
the full migration replays it.
|
||||
|
||||
- ⚠️ **The websocket url changed**, and so does what `/collaboration/` is
|
||||
routed to. The room is appended by the client, and the last segment of the
|
||||
base url is `YHUB_ORG`:
|
||||
|
||||
```yaml
|
||||
COLLABORATION_WS_URL: wss://{yourdocsdomain.tld}/collaboration/ws/v1/docs
|
||||
```
|
||||
|
||||
Both `/collaboration/` ingresses now point at the collaboration server. In
|
||||
the chart, `ingressCollaborationApi.path` (a single path) becomes
|
||||
`ingressCollaborationApi.paths` (a list, one ingress rule each), defaulting
|
||||
to `/collaboration/ydoc/` and `/collaboration/jwks/`. What is not listed
|
||||
stays in-cluster, which is how `create-ydoc`, `reset-connections`, `migrate`,
|
||||
`restore-ydoc` and `reset-ydoc` are kept unreachable: they are
|
||||
backend-internal, and publishing them would put document deletion and the
|
||||
legacy migration one request away from the internet. If you route
|
||||
`/collaboration/` by hand, publish the websocket, the browser-facing document
|
||||
routes (`ydoc`, `rollback`, `prune`, `changeset`, `activity`) and `jwks`, and
|
||||
keep those five in-cluster.
|
||||
|
||||
The `nginx.ingress.kubernetes.io/upstream-hash-by: $arg_room` annotation is
|
||||
dropped from the websocket ingress, and should be dropped from yours: the
|
||||
replicas exchange updates through Redis, so a room no longer needs a sticky
|
||||
upstream, and the new urls carry no `room` query argument — hashing on it
|
||||
would pin every connection to a single pod.
|
||||
|
||||
- ⚠️ **The backend has to reach the collaboration server.**
|
||||
`YHUB_API_BASE_URL` is now required — creating a document, duplicating one,
|
||||
`formatted-content`, the search indexation and the deletions all go through
|
||||
it, and it is also where the backend reads the JWKS verifying the calls it
|
||||
receives:
|
||||
|
||||
```yaml
|
||||
YHUB_API_BASE_URL: http://{yhub-service}:443
|
||||
```
|
||||
|
||||
Prefer the internal service url, the routes the backend calls are not meant
|
||||
to be reachable from the outside. `YHUB_API_TIMEOUT` (30 seconds) and
|
||||
`YHUB_MIGRATION_TIMEOUT` (600 seconds, the replay of one document's whole
|
||||
history) bound those calls.
|
||||
|
||||
`COLLABORATION_API_URL` is not read by the application anymore — it
|
||||
configured the safeguard removed below — and only the integration test suite
|
||||
still looks at it.
|
||||
|
||||
- ⚠️ **The backend needs an RSA private key of its own**: `JWT_PRIVATE_KEY`, or
|
||||
a file `JWT_PRIVATE_KEY_FILE` points at, which is easier since a PEM does not
|
||||
fit well in an environment variable:
|
||||
|
||||
```bash
|
||||
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out private.pem
|
||||
```
|
||||
|
||||
It signs the short-lived tokens the backend presents to the collaboration
|
||||
server and to the conversion service, which verify them against the public
|
||||
half it publishes on `/api/v1.0/jwks` — so no secret is shared with either,
|
||||
and each token carries an `aud` claim naming the service it was issued for,
|
||||
so one cannot be replayed against the other. Without the key the backend
|
||||
cannot call either service at all. `JWT_TOKEN_LIFETIME` (3600 seconds) is
|
||||
both the `exp` horizon and how long an issued token is cached.
|
||||
|
||||
The `y-provider` verifies the same way and only needs to reach the backend
|
||||
for it: `COLLABORATION_BACKEND_BASE_URL`, from which it derives
|
||||
`{base}/api/v1.0/jwks`, or `JWKS_URL` when that url is not the right one from
|
||||
where it runs. In a development environment, `make generate-secret-keys`
|
||||
creates the key in `data/jwt/`; on a cluster, `jwtKeys.enabled` makes the
|
||||
chart generate both keys in a secret the services mount read-only.
|
||||
|
||||
- ⚠️ The collaboration server now calls the backend on its own, to declare that
|
||||
a document was edited, and signs those calls: **it needs an RSA private key
|
||||
of its own**, which it had not before. Generate one and give it to the
|
||||
@@ -54,7 +193,62 @@ the following command inside your docker container:
|
||||
drift out of date. In a development environment,
|
||||
`make generate-secret-keys` creates the key in `data/jwt/`.
|
||||
|
||||
### [5.0.0] - 2026-04-30
|
||||
- ⚠️ **`COLLABORATION_SERVER_SECRET` is gone**, on both sides: remove it from
|
||||
the backend and from the collaboration server, which authenticate each other
|
||||
with the signed tokens above. The safeguard it served — "while someone is
|
||||
connected to the websocket, the users who are not are read-only" — is gone
|
||||
with it, and the question it answered no longer arises: the content is saved
|
||||
through the websocket only, so an editor that cannot open one saves nothing
|
||||
instead of overwriting what the others wrote. Consequently
|
||||
`COLLABORATION_WS_NOT_CONNECTED_READ_ONLY` (and its misspelled
|
||||
`COLLABORATION_WS_NOT_CONNECTED_READY_ONLY` alias) and
|
||||
`NO_WEBSOCKET_CACHE_TIMEOUT` are no longer read, the write-only `websocket`
|
||||
field disappears from `PATCH /api/v1.0/documents/{document_id}/`, and
|
||||
`/api/v1.0/documents/{document_id}/can-edit/` is removed along with the
|
||||
`can_edit` ability in the document payload. The `get-connections` API of the
|
||||
`y-provider` is dropped for good with them, its only consumer was that
|
||||
mechanism.
|
||||
|
||||
- ⚠️ **The `y-provider` is the conversion service and nothing else** — its
|
||||
published image no longer serves `/collaboration/ws/`. In the chart,
|
||||
`yProvider.converter`, its deployment and its service are dropped: the
|
||||
`yProvider` release *is* the converter, so a deployment that had
|
||||
`yProvider.converter.enabled: true` loses the `-converter` suffix on the url
|
||||
the backend calls, and `yProvider.converter.*` values are now ignored, their
|
||||
`yProvider.*` counterparts taking over:
|
||||
|
||||
```yaml
|
||||
# before
|
||||
Y_PROVIDER_API_BASE_URL: http://impress-docs-y-provider-converter:443/api/
|
||||
# now
|
||||
Y_PROVIDER_API_BASE_URL: http://impress-docs-y-provider:443/api/
|
||||
```
|
||||
|
||||
- The collaboration server can store the document blobs in a bucket instead of
|
||||
its own PostgreSQL database (`YHUB_S3_PERSISTENCE=true`, plus the
|
||||
`YHUB_S3_*` settings). It is off by default and nothing about this upgrade
|
||||
needs it. Read the "Document storage" section of `src/yhub-server/README.md`
|
||||
before enabling it: a document persisted that way cannot be read back once
|
||||
the setting is removed, and it is a third bucket, not the backend's
|
||||
`AWS_S3_*` nor the legacy one the migration reads.
|
||||
|
||||
- The endpoint `/api/v1.0/documents/{document_id}/content/`, added in 5.0.0, is
|
||||
removed, both its `GET` and its `PATCH`. The content of a document is now
|
||||
saved and served by the collaboration server, the editor exchanging it over
|
||||
the websocket, so nothing reads or writes it through the API anymore. If you
|
||||
integrate with Docs, stop calling this endpoint: the `content_patch` and
|
||||
`content_retrieve` abilities disappear from the document payload along with
|
||||
it. `/api/v1.0/documents/{document_id}/formatted-content/` is not affected.
|
||||
The `CONTENT_METADATA_CACHE_TIMEOUT` setting only tuned the cache of the
|
||||
removed `GET` and is no longer read, you can drop it from your configuration.
|
||||
- The JWKS of the resource server moved from `/api/{version}/jwks` to
|
||||
`/external_api/{version}/jwks`, alongside the rest of the resource server
|
||||
endpoints. `/api/{version}/jwks` now publishes the public key validating the
|
||||
tokens Docs issues to call external services. If you enabled the resource
|
||||
server (`OIDC_RESOURCE_SERVER_ENABLED`), update the JWKS URI declared to your
|
||||
OIDC provider accordingly.
|
||||
|
||||
## [5.0.0] - 2026-04-30
|
||||
|
||||
We made several changes around document content management leading to several breaking changes in the API.
|
||||
|
||||
|
||||
@@ -908,7 +908,11 @@ jwtKeys:
|
||||
## live state of a document in redis/valkey and persists it to its own
|
||||
## PostgreSQL database, so it needs both — set `yhub.envVars.REDIS` and
|
||||
## `yhub.envVars.POSTGRES`, there is nothing sensible to default them to.
|
||||
## Disabling it sends the /collaboration/ ingresses back to the y-provider.
|
||||
## Disabling it drops its deployment, its service and its init-db job, and
|
||||
## nothing takes that role over: the two /collaboration/ ingresses still name
|
||||
## its service, and the y-provider does not serve the websocket anymore. Turn
|
||||
## it off only to run the collaboration server outside of this release, and
|
||||
## turn `ingressCollaborationWS` and `ingressCollaborationApi` off with it.
|
||||
##
|
||||
## Two buckets can be added to that list, each with a prefix of its own so that
|
||||
## none of them is ambiguous — they may sit on different providers, with
|
||||
|
||||
Reference in New Issue
Block a user