WEBDIENSTE/lasuite-docs
1
0
Fork 0
mirror of https://github.com/suitenumerique/docs.git synced 2026-08-17 21:25:43 +02:00
Code Issues Packages Projects Releases Wiki Activity

📝(installation) upgrade documentation with yhub instruction

Browse Source 
All installation guides are updated to add the instructions about how to
deploy docs with yhub.
This commit is contained in:
Manuel Raynaud
2026-08-17 17:57:03 +02:00
parent de5089c0b2
commit dcf139d039
18 changed files with 490 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+13
View File
@@ -21,6 +21,19 @@ and this project adheres to
### Added
- 📝(installation) deploy the collaboration server in the compose install: the
example stack gains the `yhub` service, the valkey it persists through and an
`env.d/yhub` of its own, the proxy routes `/collaboration/` to it instead of
the y-provider (and publishes only what browsers call, the backend-internal
routes staying inside the network), and the backend gains the signing key and
the url it reaches it on. The guide walks through the two keys to generate,
the `npm run init-db` that creates the schema before the first start and
after every upgrade, and the migration of a corpus stored in the object
storage before the collaboration server existed. The kubernetes guide gains
the same reading of what is deployed and turns `jwtKeys` on in its example
values, the Scalingo one states that its buildpack does not start the
collaboration server yet, and `documentation/env.md` documents the variables
of the two node services, which it had none of
- ✨(collaboration) let the collaboration server keep the document blobs in a
bucket instead of its own PostgreSQL database, through yhub's S3 persistence
plugin: `YHUB_S3_PERSISTENCE=true`, plus `YHUB_S3_ENDPOINT_URL`,
docker/files/production/etc/nginx/conf.d/default.conf.template
+11 -5
View File
@@ -55,7 +55,10 @@ server {
try_files $uri @proxy_to_docs_backend;
}
# Proxy auth for collaboration server
# Collaboration server. Only the routes below are published: it also serves
# create-ydoc, reset-connections, migrate, restore-ydoc and reset-ydoc,
# which the backend calls in-cluster and which must not be reachable from
# the outside.
location /collaboration/ws/ {
# Ensure WebSocket upgrade
proxy_http_version 1.1;
@@ -63,7 +66,7 @@ server {
proxy_set_header Connection "Upgrade";
# Collaboration server
proxy_pass http://${YPROVIDER_HOST}:4444;
proxy_pass http://${YHUB_HOST}:3002;
# Set appropriate timeout for WebSocket
proxy_read_timeout 86400;
@@ -75,9 +78,12 @@ server {
proxy_set_header Host $host;
}
location /collaboration/api/ {
# Collaboration server
proxy_pass http://${YPROVIDER_HOST}:4444;
# Document routes, guarded by the same document authorization as the
# websocket, and the public keys the collaboration server signs with.
location ~ ^/collaboration/(ydoc|rollback|prune|changeset|activity|jwks)/ {
proxy_pass http://${YHUB_HOST}:3002;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header Origin $http_origin;
proxy_set_header Host $host;
}
documentation/env.md
+52 -1
View File
@@ -143,9 +143,60 @@ These are the environment variables you can set for the `impress-backend` contai
| USER_RECONCILIATION_FORM_URL | URL of a third-party form for user reconciliation requests | |
| YHUB_API_BASE_URL | Base url of the yhub collaboration server REST API | |
| YHUB_API_TIMEOUT | Timeout (in seconds) of the requests to the yhub API | 30 |
| YHUB_MIGRATION_TIMEOUT | Timeout (in seconds) of the call replaying the legacy history of one document, which reads every one of its S3 versions | 600 |
| YHUB_ORG | yhub organization the documents live in. Must match the YHUB_ORG of the yhub server | docs |
| Y_PROVIDER_API_BASE_URL | Y Provider url | |
| Y_PROVIDER_API_KEY | Y provider API key | |
| Y_PROVIDER_API_KEY | Key exempting the calls of the collaboration server from the API throttling, sent as X-Y-Provider-Key. Set the same value on the yhub container | |
## impress-yhub container
These are the environment variables you can set for the `impress-yhub`
container, the collaboration server. It reads none of the backend's settings:
what it shares with the backend is repeated here by value. `src/yhub-server/README.md`
documents what each of them changes.
| Option | Description | default |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------- |
| PORT | Port the server listens on | 3002 |
| REDIS | **Required.** Redis/Valkey url holding the live state of the documents. Not a cache: it holds what no worker has persisted yet | |
| POSTGRES | **Required.** Url of the yhub database. Created by `npm run init-db`, never by the server | |
| REDIS_PREFIX | Namespace of the redis keys, when the instance is shared | yhub |
| COLLABORATION_BACKEND_BASE_URL | Base url of the Docs backend, which answers who a user is and what they may do with a document | http://app-dev:8000 |
| COLLABORATION_SERVER_ORIGIN | Comma separated list of the origins allowed to open a websocket | http://localhost:3000 |
| Y_PROVIDER_API_KEY | Sent as X-Y-Provider-Key on the calls made to the backend, exempting them from the API throttling. The same value as the backend's | yprovider-api-key |
| YHUB_ORG | Organization the documents live under. Must match the YHUB_ORG of the backend | docs |
| YHUB_JWT_PRIVATE_KEY | PEM encoded RSA private key signing the calls made to the backend (RS256). Can be read from a file with YHUB_JWT_PRIVATE_KEY_FILE | |
| YHUB_ROLE | `all` runs the websockets and the worker in one process, `server` only the websockets and the routes, `worker` only the persistence | all |
| YHUB_TASK_CONCURRENCY | Tasks one worker process claims at once | 5 |
| YHUB_TASK_DEBOUNCE_MS | How long an update waits on the redis stream before a worker persists it | 10000 |
| YHUB_MIN_MESSAGE_LIFETIME_MS | How long persisted updates stay replayable from redis rather than read back from postgres | 60000 |
| SOFT_MIGRATION | Set to "true" to seed a room from the legacy Django/S3 document store the first time it is opened | false |
| LEGACY_S3_ENDPOINT_URL | Required by SOFT_MIGRATION, endpoint of the legacy media bucket, without a path | |
| LEGACY_S3_ACCESS_KEY_ID | Required by SOFT_MIGRATION, read access to that bucket (or LEGACY_S3_ACCESS_KEY_ID_FILE) | |
| LEGACY_S3_SECRET_ACCESS_KEY | Required by SOFT_MIGRATION, secret of the key above (or LEGACY_S3_SECRET_ACCESS_KEY_FILE) | |
| LEGACY_S3_BUCKET_NAME | Name of the legacy media bucket | impress-media-storage |
| LEGACY_S3_REGION_NAME | Region of that bucket, when its provider needs one | us-east-1 |
| LEGACY_S3_SIGNATURE_VERSION | How the calls to that bucket are signed, s3v4 or v4 | s3v4 |
| YHUB_S3_PERSISTENCE | Set to "true" to store the document blobs in a bucket instead of the yhub database. Read the "Document storage" section of `src/yhub-server/README.md` first: it cannot be turned back off | false |
| YHUB_S3_ENDPOINT_URL | Required by YHUB_S3_PERSISTENCE, endpoint of that bucket, without a path | |
| YHUB_S3_ACCESS_KEY_ID | Required by YHUB_S3_PERSISTENCE, read/write/delete access to that bucket (or YHUB_S3_ACCESS_KEY_ID_FILE) | |
| YHUB_S3_SECRET_ACCESS_KEY | Required by YHUB_S3_PERSISTENCE, secret of the key above (or YHUB_S3_SECRET_ACCESS_KEY_FILE) | |
| YHUB_S3_BUCKET_NAME | Required by YHUB_S3_PERSISTENCE, name of that bucket, created on startup when missing | |
| YHUB_S3_REGION_NAME | Region of that bucket, when its provider needs one | |
## impress-y-provider container
These are the environment variables you can set for the `impress-y-provider`
container, the conversion service. It no longer serves the collaboration.
| Option | Description | default |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
| PORT | Port the service listens on | 4444 |
| COLLABORATION_BACKEND_BASE_URL | Base url of the Docs backend. The JWKS verifying the tokens it is called with is fetched from `{this}/api/v1.0/jwks`, so it has to reach it | http://app-dev:8000 |
| COLLABORATION_SERVER_ORIGIN | Comma separated list of the allowed origins | http://localhost:3000 |
| COLLABORATION_LOGGING | Set to "true" to log the requests | false |
| CONVERSION_FILE_MAX_SIZE | Maximum size, in bytes, of a file submitted for conversion | 20971520 |
| SENTRY_DSN | Sentry DSN, unset disables it | |
## impress-frontend image
documentation/examples/compose/compose.yaml
+52
View File
@@ -17,6 +17,22 @@ services:
redis:
image: redis:8
# The valkey of the collaboration server, apart from the redis above: this one
# holds the updates no worker has persisted to postgres yet, so it is a store
# and not a cache. Hence the append-only file, and a policy that only evicts
# the keys carrying an expiry.
yhub-valkey:
image: valkey/valkey:8-alpine
command: ["valkey-server", "--maxmemory-policy", "volatile-lru",
"--appendonly", "yes", "--appendfsync", "everysec"]
healthcheck:
test: ["CMD", "valkey-cli", "ping"]
interval: 1s
timeout: 2s
retries: 60
volumes:
- ./data/yhub-valkey:/data
backend:
image: lasuite/impress-backend:latest
user: ${DOCKER_USER:-1000}
@@ -28,6 +44,9 @@ services:
- env.d/backend
- env.d/yprovider
- env.d/postgresql
volumes:
# signs the calls made to the collaboration server and to the converter
- ./keys/private.pem:/keys/private.pem:ro
healthcheck:
test: ["CMD", "python", "manage.py", "check"]
interval: 15s
@@ -41,6 +60,37 @@ services:
redis:
condition: service_started
# The collaboration server: it serves everything under /collaboration/, the
# websocket included, and holds the content of the documents. Its schema is
# not created when it starts — run `docker compose run --rm yhub npm run
# init-db` once before the first start, and again after every upgrade.
yhub:
image: lasuite/impress-yhub:latest
user: ${DOCKER_USER:-1000}
restart: always
env_file:
- env.d/common
- env.d/yhub
volumes:
# signs the calls made to the backend, which reads the public half from
# the JWKS this server publishes
- ./keys/yhub-private.pem:/keys/yhub-private.pem:ro
healthcheck:
test: ["CMD", "node", "-e", "fetch('http://localhost:3002/collaboration/ready/v1').then(r=>process.exit(r.ok?0:1),()=>process.exit(1))"]
interval: 15s
timeout: 5s
retries: 20
start_period: 10s
depends_on:
postgresql:
condition: service_healthy
restart: true
yhub-valkey:
condition: service_healthy
# The conversion service. It used to serve the collaboration as well, it does
# not anymore: the backend calls it on Y_PROVIDER_API_BASE_URL, and nothing is
# routed to it by the proxy.
y-provider:
image: lasuite/impress-y-provider:latest
user: ${DOCKER_USER:-1000}
@@ -66,6 +116,8 @@ services:
depends_on:
backend:
condition: service_healthy
yhub:
condition: service_healthy
# Uncomment if using our nginx proxy example
# networks:
documentation/examples/compose/keycloak/README.md
+1 -1
View File
@@ -9,7 +9,7 @@
```bash
mkdir keycloak
curl -o keycloak/compose.yaml https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/docs/examples/compose/keycloak/compose.yaml
curl -o keycloak/compose.yaml https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/documentation/examples/compose/keycloak/compose.yaml
curl -o keycloak/env.d/kc_postgresql https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/env.d/production.dist/kc_postgresql
curl -o keycloak/env.d/keycloak https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/env.d/production.dist/keycloak
```
documentation/examples/compose/minio/README.md
+1 -1
View File
@@ -9,7 +9,7 @@
```bash
mkdir minio
curl -o minio/compose.yaml https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/docs/examples/compose/minio/compose.yaml
curl -o minio/compose.yaml https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/documentation/examples/compose/minio/compose.yaml
```
### Step 2:. Update compose file with your own values
documentation/examples/compose/nginx-proxy/README.md
+1 -1
View File
@@ -13,7 +13,7 @@ Acme-companion is a lightweight companion container for nginx-proxy. It handles
```bash
mkdir nginx-proxy
curl -o nginx-proxy/compose.yaml https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/docs/examples/compose/nginx-proxy/compose.yaml
curl -o nginx-proxy/compose.yaml https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/documentation/examples/compose/nginx-proxy/compose.yaml
```
### Step 2: Edit `DEFAULT_EMAIL` in the compose file.
documentation/examples/helm/impress.values.yaml
+7
View File
@@ -157,6 +157,13 @@ yhub:
COLLABORATION_BACKEND_BASE_URL: https://docs.127.0.0.1.nip.io
COLLABORATION_SERVER_ORIGIN: https://docs.127.0.0.1.nip.io
# The backend and the collaboration server authenticate each other with signed
# tokens, each with a key of its own, verified against the JWKS the other
# publishes. This generates the two keys on the cluster, once, in a secret both
# mount read-only — nothing to put in this file.
jwtKeys:
enabled: true
ingress:
enabled: true
host: docs.127.0.0.1.nip.io
documentation/format_conversion.md
+1 -1
View File
@@ -24,7 +24,7 @@ On the `y-provider` side, point it at the Django backend so it can fetch the JWK
COLLABORATION_BACKEND_BASE_URL: http://{django-service}:8000
```
The JWKS url defaults to `{COLLABORATION_BACKEND_BASE_URL}/api/v1.0/jwks`; override it with `JWKS_URL` if Django is not reachable at that base url from the y-provider service.
The JWKS is fetched from `{COLLABORATION_BACKEND_BASE_URL}/api/v1.0/jwks`, so that url has to be reachable from the y-provider service.
### One service, not two anymore
documentation/installation/README.md
+10 -1
View File
@@ -2,6 +2,15 @@
If you want to install Docs you've come to the right place.
Here are a bunch of resources to help you install the project.
Whichever method you pick, Docs is made of four services: the **frontend**, the
Django **backend**, the **collaboration server** (`yhub`), which holds the
content of the documents and syncs the editors over the websocket, and the
**conversion service** (`y-provider`), which converts documents between formats.
The collaboration server needs a PostgreSQL database and a Redis/Valkey instance
of its own, beside the ones the backend uses. See [the collaboration
documentation](../collaboration.md) for how the services find and authenticate
each other.
## Kubernetes
We (Docs maintainers) are only using the Kubernetes deployment method in production. We can only provide advanced support for this method.
Please follow the instructions laid out [here](/documentation/installation/kubernetes.md).
@@ -13,7 +22,7 @@ Please follow the instructions [here](/documentation/installation/compose.md).
⚠️ Please keep in mind that we do not use it ourselves in production. Let us know in the issues if you run into troubles, we'll try to help.
## Scalingo
You can deploy Docs on [Scalingo](https://scalingo.com/) using a custom buildpack. This method handles both frontend and backend builds, serving them through Nginx with the collaboration server (y-provider).
You can deploy Docs on [Scalingo](https://scalingo.com/) using a custom buildpack. This method handles both frontend and backend builds, serving them through Nginx with the conversion service (y-provider). ⚠️ The buildpack does not start the collaboration server, which has to be run separately.
Please follow the instructions [here](/documentation/installation/scalingo.md).
## Other ways to install Docs
documentation/installation/compose.md
+130 -8
View File
@@ -8,8 +8,28 @@ We provide a sample configuration for running Docs using Docker Compose. Please
- A domain name and DNS configured to your server.
- An Identity Provider that supports OpenID Connect protocol - we provide [an example to deploy Keycloak](../examples/compose/keycloak/README.md).
- An Object Storage that implements S3 API - we provide [an example to deploy Minio](../examples/compose/minio/README.md).
- A Postgresql database - we provide [an example in the compose file](../examples/compose/compose.yaml).
- A Postgresql database - we provide [an example in the compose file](../examples/compose/compose.yaml). Two databases are needed on it, one for the backend and one for the collaboration server.
- A Redis database - we provide [an example in the compose file](../examples/compose/compose.yaml).
- A Valkey (or Redis) instance for the collaboration server, separate from the one above - we provide [an example in the compose file](../examples/compose/compose.yaml).
## The services
Docs is made of four services, all of them in the example compose file beside
their stores:
| Service | What it does |
| ------- | ------------ |
| `frontend` | Serves the editor, and is the nginx proxy routing everything else |
| `backend` | The Django application: documents, users, accesses, search |
| `yhub` | The collaboration server. It holds the **content** of the documents, syncs the editors over the websocket, and the backend reads and writes documents through it |
| `y-provider` | The conversion service (markdown, html, pdf, docx). It served the collaboration in the previous releases, it does not anymore |
| `postgresql`, `redis`, `yhub-valkey` | The stores |
The content of a document is not in the object storage: it is in the
collaboration server, in its own PostgreSQL database. The object storage keeps
the attachments and the version history. This matters when you upgrade an
instance that ran before the collaboration server existed — see [the last
section of this page](#upgrading-from-a-release-without-the-collaboration-server).
## Software Requirements
@@ -32,10 +52,11 @@ For older versions of Docker Engine that do not include Docker Compose:
```bash
mkdir -p docs/env.d
cd docs
curl -o compose.yaml https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/docs/examples/compose/compose.yaml
curl -o compose.yaml https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/documentation/examples/compose/compose.yaml
curl -o env.d/common https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/env.d/production.dist/common
curl -o env.d/backend https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/env.d/production.dist/backend
curl -o env.d/yprovider https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/env.d/production.dist/yprovider
curl -o env.d/yhub https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/env.d/production.dist/yhub
curl -o env.d/postgresql https://raw.githubusercontent.com/suitenumerique/docs/refs/heads/main/env.d/production.dist/postgresql
```
@@ -83,17 +104,65 @@ If you are using the example provided, you need to generate a secure key for `DB
If you are using an external service or not using our default values, you should update the variables in `env.d/postgresql`
The collaboration server keeps its own database on the same server, `yhub`,
whose connection string is `POSTGRES` in `env.d/yhub` — set the same password
there. The `init-db` step below creates that database when the user is allowed
to; if yours is not, create an empty `yhub` database beforehand and grant it on
that one.
### Redis
Docs uses Redis for caching. While an external Redis can be used, our example provides a deployment method.
If you are using an external service, you need to set `REDIS_URL` environment variable in `env.d/backend`.
The collaboration server has a Valkey of its own, `yhub-valkey`, configured
with `REDIS` in `env.d/yhub`. Give it an instance apart rather than the one
above: it is not a cache, it holds the updates that no worker has written to
PostgreSQL yet, so it has to be durable and must never evict a key it was not
told to expire. Our example configures it accordingly (append-only file,
`volatile-lru`).
### Collaboration server
The collaboration server (`yhub`) synchronizes the editors over the websocket
and holds the content of the documents. It authenticates with the backend, and
the backend with it, using RS256 keys — each service signs with its own key and
verifies the other against the JWKS it publishes, so no secret is shared.
Generate the two keys next to your compose file:
```bash
mkdir -p keys
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out keys/private.pem
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out keys/yhub-private.pem
chmod 600 keys/*.pem
# readable by the uid the containers run as (DOCKER_USER, 1000 by default)
sudo chown 1000:1000 keys/*.pem
```
`keys/private.pem` is the backend's (`JWT_PRIVATE_KEY_FILE` in `env.d/backend`)
and `keys/yhub-private.pem` is the collaboration server's
(`YHUB_JWT_PRIVATE_KEY_FILE` in `env.d/yhub`). Never give the same key to both,
and treat them as secrets: either one signs calls the other trusts.
Then set in `env.d/yhub`:
- `POSTGRES` and `REDIS`, the two stores above,
- `Y_PROVIDER_API_KEY`, **the same value** as the one you generate in
`env.d/yprovider` below. It is the header exempting the collaboration server
from the API throttling of the backend, which it calls once per connection.
`COLLABORATION_SERVER_ORIGIN` lists the origins a browser may open a websocket
from, and `COLLABORATION_BACKEND_BASE_URL` is the backend it asks who a user is
and what they may do — both default to your `DOCS_HOST`.
### Y Provider
The Y provider service enables collaboration through websockets.
The Y provider service converts documents between formats (markdown, html, pdf,
docx). It no longer serves the collaboration.
Generates a secure key for `Y_PROVIDER_API_KEY` in ``env.d/yprovider``.
Generates a secure key for `Y_PROVIDER_API_KEY` in ``env.d/yprovider``, and
repeat it in `env.d/yhub`.
### Docs
@@ -185,7 +254,22 @@ You will need to uncomment the environment and network sections in compose file
# external: true
```
## Step 4: Start Docs
## Step 4: Create the schema of the collaboration server
The collaboration server never runs DDL itself, so its schema has to be created
before it starts:
```bash
docker compose run --rm yhub npm run init-db
```
It creates the `yhub` database when it is missing, and every table the version
you are installing needs. It is idempotent, so re-running it is always safe —
and it has to be re-run after every upgrade, see below. Until it has run, the
collaboration server answers every read with a `relation "..." does not exist`
error and stays unhealthy.
## Step 5: Start Docs
You are ready to start your Docs application !
@@ -195,7 +279,7 @@ docker compose up -d
> [!NOTE]
> Version of the images are set to latest, you should pin it to the desired version to avoid unwanted upgrades when pulling latest image.
## Step 5: Run the database migration and create Django admin user
## Step 6: Run the database migration and create Django admin user
```bash
docker compose run --rm backend python manage.py migrate
@@ -228,8 +312,46 @@ docker compose pull
docker compose restart
```
### Step 4: Run the database migration
Your database schema may need to be updated, run:
### Step 4: Run the database migrations
Your database schemas may need to be updated. The backend's:
```bash
docker compose run --rm backend python manage.py migrate
```
and the collaboration server's, which is the same command as at install time:
```bash
docker compose run --rm yhub npm run init-db
```
## Upgrading from a release without the collaboration server
Documents created before the collaboration server existed have their content in
the object storage, one object per document at key `{document-id}/file`, and the
collaboration server starts out knowing none of them. It has to be handed the
corpus, otherwise those documents open **empty**.
1. Before letting anyone in, uncomment the `SOFT_MIGRATION` block of
`env.d/yhub` and point it at your media bucket (`LEGACY_S3_ENDPOINT_URL`,
`LEGACY_S3_ACCESS_KEY_ID`, `LEGACY_S3_SECRET_ACCESS_KEY`,
`LEGACY_S3_BUCKET_NAME`). Read-only credentials scoped to that bucket are
enough — this service terminates untrusted traffic, do not give it the
backend's read-write keys. A document is then migrated from its legacy
object the first time someone opens it.
2. Then migrate the whole corpus, which that lazy migration never finishes on
its own — a document nobody opens stays in the bucket forever:
```bash
docker compose run --rm backend python manage.py migrate_documents
```
It hands every document to the collaboration server, which replays its full
version history. The run is resumable and safe to repeat: what became of
every document is recorded, and `--retry-failed` picks up the ones that
failed. `--dry-run` counts what it would do, and `--concurrency`, `--rate`
and `--limit` bound it.
Only once that run has covered the corpus may `SOFT_MIGRATION` be commented out
again. And keep the media bucket, its objects and its versioning either way:
the version history of a document is still served from there.
The full procedure, including what changes in the environment variables of an
existing instance, is in the [Upgrade document](../../UPGRADE.md).
documentation/installation/kubernetes.md
+85 -7
View File
@@ -6,10 +6,30 @@ This document is a step-by-step guide that describes how to install Docs on a k8
- k8s cluster with an nginx-ingress controller
- an OIDC provider (if you don't have one, we provide an example)
- a PostgreSQL server (if you don't have one, we provide an example)
- a PostgreSQL server (if you don't have one, we provide an example). Two databases are created on it, one for the backend and one for the collaboration server
- a Redis server (if you don't have one, we provide an example)
- a S3 bucket (if you don't have one, we provide an example)
## What gets deployed
The chart deploys four services:
| Deployment | What it does |
| ---------- | ------------ |
| `frontend` | Serves the editor |
| `backend` (and `celery-worker`) | The Django application: documents, users, accesses, search |
| `yhub` | The collaboration server. It holds the **content** of the documents, syncs the editors over the websocket, and the backend reads and writes documents through it |
| `y-provider` | The conversion service (markdown, html, pdf, docx). It served the collaboration in the previous releases, it does not anymore |
plus two jobs that run before them: the Django `migrate` job, and the `yhub`
`init-db` job creating the schema of the collaboration server — it never runs
DDL itself. Both wait for the PostgreSQL server, which nothing in this chart
creates.
The content of a document is not in the S3 bucket: it is in the collaboration
server, in a PostgreSQL database of its own. The bucket keeps the attachments
and the version history.
### Test cluster
If you do not have a test cluster, you can install everything on a local Kind cluster. In this case, the simplest way is to use our script **bin/start-kind.sh**.
@@ -118,7 +138,7 @@ You can install it on your cluster to deploy keycloak, minio, postgresql and red
Docs uses OIDC, so if you already have an OIDC provider, obtain the necessary information to use it. In the next step, we will see how to configure Django (and thus Docs) to use it. If you do not have a provider, we will show you how to deploy a local Keycloak instance (this is not a production deployment, just a demo).
```
$ helm install --repo https://suitenumerique.github.io/helm-dev-backend -f docs/examples/helm/keycloak.values.yaml keycloak dev-backend
$ helm install --repo https://suitenumerique.github.io/helm-dev-backend -f documentation/examples/helm/keycloak.values.yaml keycloak dev-backend
$ #wait until
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
@@ -140,14 +160,14 @@ OIDC_RP_SIGN_ALGO: RS256
OIDC_RP_SCOPES: "openid email"
```
You can find these values in **examples/helm/keycloak.values.yaml**
You can find these values in **documentation/examples/helm/keycloak.values.yaml**
### Find redis server connection values
Docs needs a redis so we start by deploying one:
```
$ helm install --repo https://suitenumerique.github.io/helm-dev-backend -f docs/examples/helm/redis.values.yaml redis dev-backend
$ helm install --repo https://suitenumerique.github.io/helm-dev-backend -f documentation/examples/helm/redis.values.yaml redis dev-backend
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
keycloak-dev-backend-keycloak-0 1/1 Running 0 113s
@@ -162,12 +182,25 @@ REDIS_URL: redis://user:pass@redis-dev-backend-redis:6379/1
DJANGO_CELERY_BROKER_URL: redis://user:pass@redis-dev-backend-redis:6379/1
```
The collaboration server needs one too, under `yhub.envVars.REDIS`. This example
puts it in another database of the same server:
```yaml
REDIS: redis://user:pass@redis-dev-backend-redis:6379/2
REDIS_PREFIX: yhub
```
> [!NOTE]
> In production, give it an instance of its own. It is not a cache: it holds the
> updates that no worker has written to PostgreSQL yet, so it has to be durable
> and must never evict a key it was not told to expire.
### Find postgresql connection values
Docs uses a postgresql database as backend, so if you have a provider, obtain the necessary information to use it. If you don't, you can install a postgresql testing environment as follow:
```
$ helm install --repo https://suitenumerique.github.io/helm-dev-backend -f docs/examples/helm/postgresql.values.yaml postgresql dev-backend
$ helm install --repo https://suitenumerique.github.io/helm-dev-backend -f documentation/examples/helm/postgresql.values.yaml postgresql dev-backend
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
keycloak-dev-backend-keycloak-0 1/1 Running 0 3m42s
@@ -196,12 +229,24 @@ DB_PASSWORD:
DB_PORT: 5432
```
The collaboration server keeps its own database on that same server, configured
as a single url under `yhub.envVars.POSTGRES`:
```yaml
POSTGRES: postgres://dinum:pass@postgresql-dev-backend-postgres:5432/yhub
```
Being one url, the credentials are in it — put the whole url in a secret and
reference it with `secretKeyRef` if you would rather not have it in your values
file. The `init-db` job creates that database when the user is allowed to;
otherwise create an empty `yhub` database beforehand and grant it on that one.
### Find s3 bucket connection values
Docs uses an s3 bucket to store documents, so if you have a provider obtain the necessary information to use it. If you don't, you can install a local minio testing environment as follow:
```
$ helm install --repo https://suitenumerique.github.io/helm-dev-backend -f docs/examples/helm/minio.values.yaml minio dev-backend
$ helm install --repo https://suitenumerique.github.io/helm-dev-backend -f documentation/examples/helm/minio.values.yaml minio dev-backend
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
keycloak-dev-backend-keycloak-0 1/1 Running 0 6m12s
@@ -212,6 +257,28 @@ redis-dev-backend-redis-68c9f66786-4dgxj 1/1 Running 0 4m21s
```
### Signing keys of the services
The backend and the collaboration server call each other, and sign those calls:
each has an RSA key of its own and verifies the other against the JWKS it
publishes, so neither holds a copy of the other's key and no shared secret is
involved. The chart generates both for you:
```yaml
jwtKeys:
enabled: true
```
A job creates them once, with `openssl`, in a secret that the backend and the
collaboration server mount read-only — no key is written in a values file or
templated into a manifest. It leaves an existing secret alone, so it is safe on
every sync, and rolling the keys is deleting the secret and letting the next run
create it again. If you already hold your keys in a secret,
`jwtKeys.existingSecret` points at it instead and the job is not created at all.
This is in **documentation/examples/helm/impress.values.yaml**, and the
[collaboration documentation](../collaboration.md) covers it in more detail.
## Deployment
Now you are ready to deploy Docs without AI. AI requires more dependencies (OpenAI API). To deploy Docs you need to provide all previous information to the helm chart.
@@ -219,7 +286,7 @@ Now you are ready to deploy Docs without AI. AI requires more dependencies (Open
```
$ helm repo add impress https://suitenumerique.github.io/docs/
$ helm repo update
$ helm install impress impress/docs -f docs/examples/helm/impress.values.yaml
$ helm install impress impress/docs -f documentation/examples/helm/impress.values.yaml
$ kubectl get po
NAME READY STATUS RESTARTS AGE
impress-docs-backend-8494fb797d-8k8wt 1/1 Running 0 6m45s
@@ -234,6 +301,17 @@ postgresql-dev-backend-postgres-0 1/1 Running 0 20m
redis-dev-backend-redis-68c9f66786-4dgxj 1/1 Running 0 22m
```
The jobs are not in that list anymore: the `migrate` one, the `jwt-keys` one and
the `yhub` `init-db` one ran and were removed, 30 seconds after they finished.
If the collaboration server never becomes ready, that is where to look first —
raise `yhub.jobs.ttlSecondsAfterFinished` to keep the job around long enough to
read it:
```
$ kubectl get jobs
$ kubectl logs job/impress-docs-yhub-init-db
```
## Test your deployment
In order to test your deployment you have to log into your instance. If you exclusively use our examples you can do:
documentation/installation/scalingo.md
+55 -5
View File
@@ -4,7 +4,22 @@ This guide explains how to deploy Docs on [Scalingo](https://scalingo.com/) usin
## Overview
Scalingo is a Platform-as-a-Service (PaaS) that simplifies application deployment. This setup uses a custom buildpack to handle both the frontend (Next.js static export) and backend (Django) builds, serving them through Nginx. The collaboration server (y-provider) runs alongside the Django backend.
Scalingo is a Platform-as-a-Service (PaaS) that simplifies application deployment. This setup uses a custom buildpack to handle both the frontend (Next.js static export) and backend (Django) builds, serving them through Nginx. The conversion service (y-provider) runs alongside the Django backend.
> [!WARNING]
> The buildpack does not start the collaboration server yet. Since the content
> of a document moved there, a Scalingo app deployed on its own **cannot save
> documents**: the editor opens, and nothing is persisted. The y-provider the
> buildpack starts is the conversion service only — it stopped serving the
> collaboration websocket in this release.
>
> Until the buildpack starts it, run the collaboration server
> (`lasuite/impress-yhub`) elsewhere — a container platform, a VM, another
> provider — and point this app at it with `YHUB_API_BASE_URL` and
> `COLLABORATION_WS_URL`, as described in [Collaboration
> server](#collaboration-server) below. The
> [compose](compose.md) and [kubernetes](kubernetes.md) guides deploy it as
> part of the stack.
## Prerequisites
@@ -14,6 +29,7 @@ Scalingo is a Platform-as-a-Service (PaaS) that simplifies application deploymen
- A Redis Scalingo addon (for caching and sessions)
- An external Identity Provider that supports OpenID Connect protocol
- An external Object Storage that implements S3 API
- Somewhere to run the collaboration server, with a PostgreSQL database and a Redis/Valkey instance of its own (see the warning above)
## Step 1: Create Your App
@@ -92,6 +108,35 @@ scalingo env-set AWS_STORAGE_BUCKET_NAME="docs-media"
scalingo env-set AWS_S3_REGION_NAME="eu-west-1"
```
### Collaboration server
The collaboration server holds the content of the documents. The backend reads
and writes them through its REST API, and the editors sync with it over the
websocket, so the app needs to know where it is — the first url is reached by
the backend, the second by the browser:
```bash
scalingo env-set YHUB_API_BASE_URL="https://yhub.yourdomain.com"
scalingo env-set COLLABORATION_WS_URL="wss://yhub.yourdomain.com/collaboration/ws/v1/docs"
```
They authenticate each other with signed tokens rather than a shared secret:
each signs with an RSA key of its own and verifies the other against the JWKS it
publishes. Generate the backend one and set it:
```bash
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out private.pem
scalingo env-set JWT_PRIVATE_KEY="$(cat private.pem)"
```
The collaboration server needs a key of its own (`YHUB_JWT_PRIVATE_KEY`), the
backend it calls (`COLLABORATION_BACKEND_BASE_URL`), the origins a browser may
open a websocket from (`COLLABORATION_SERVER_ORIGIN`), and its two stores
(`POSTGRES`, `REDIS`). Its schema is created by the script it ships,
`npm run init-db`, which has to run once before it starts and again after every
upgrade. Its variables are listed in [env.md](../env.md), and
`src/yhub-server/README.md` documents what each of them changes.
### Email Configuration (Optional)
For email notifications see [https://doc.scalingo.com/platform/app/sending-emails](https://doc.scalingo.com/platform/app/sending-emails):
@@ -118,7 +163,7 @@ The buildpack will automatically:
2. Build the backend (Django)
3. Run the post-compile script (cleanup unused files to reduce slug size)
4. Run the post-frontend script (move assets, inject theme, prepare for deployment)
5. Start uvicorn, the y-provider collaboration server, and Nginx
5. Start uvicorn, the y-provider conversion service, and Nginx
6. Run Django migrations
## Step 5: Create superuser
@@ -205,7 +250,8 @@ scalingo logs --tail
3. **Static files not served**: Ensure the buildpack post-frontend script ran successfully
4. **OIDC errors**: Verify your OIDC provider configuration and callback URLs
5. **Theme not updating**: Clear Redis cache with `scalingo run python -c "from django.core.cache import cache; cache.clear()"`
6. **Collaboration not working**: Verify the y-provider server is running and the WebSocket URL is configured
6. **Collaboration not working, or documents opening empty**: the collaboration server is a separate deployment. Verify it is running, that `COLLABORATION_WS_URL` and `YHUB_API_BASE_URL` point at it, that `JWT_PRIVATE_KEY` is set on this app and `YHUB_JWT_PRIVATE_KEY` on that one, and that its `npm run init-db` has been run
7. **Conversion not working** (export, markdown import): verify the y-provider process is running and `Y_PROVIDER_API_BASE_URL` points at it
### Useful Commands
@@ -246,14 +292,18 @@ The `bin/buildpack_start.sh` script starts three processes:
- **Nginx** serves static files and proxies requests to the backend
- **uvicorn** runs the Django ASGI application on port 8000
- **y-provider** runs the collaboration WebSocket server on port 4444
- **y-provider** runs the conversion service on port 4444
Nginx routes:
- `/api/` and `/admin/` → Django backend (port 8000)
- `/collaboration/api/` and `/collaboration/ws/` → y-provider (port 4444)
- `/media/` → S3 object storage (with auth proxy)
- `/` → Static frontend files
The collaboration server is not one of these processes. `/collaboration/` is
served by the separate deployment `COLLABORATION_WS_URL` points at, and the
conversion service is called by the backend directly on
`Y_PROVIDER_API_BASE_URL` — nothing is routed to it.
## Additional Resources
- [Scalingo Documentation](https://doc.scalingo.com/)
documentation/release.md
+6
View File
@@ -28,6 +28,12 @@ Whenever we are cooking a new release (e.g. `4.18.1`) we should follow a standar
repository: lasuite/impress-y-provider
pullPolicy: Always
tag: "v4.18.1"
yhub:
image:
repository: lasuite/impress-yhub
pullPolicy: Always
tag: "v4.18.1"
```
The new images don't exist _yet_: they will be created automatically later in the process.
env.d/production.dist/backend
+12
View File
@@ -13,6 +13,18 @@ LOGGING_LEVEL_LOGGERS_APP=INFO
# Python
PYTHONPATH=/app
# JWT
# Signs the short-lived tokens the backend presents to the collaboration server
# and to the conversion service, which verify them against the public half
# published on /api/v1.0/jwks. A PEM does not fit in an environment variable,
# hence the _FILE variant.
JWT_PRIVATE_KEY_FILE=/keys/private.pem
# Collaboration server
# Where the content of the documents lives. Reached in-cluster, the routes the
# backend calls are not meant to be reachable from the outside.
YHUB_API_BASE_URL=http://${YHUB_HOST}:3002
# Mail
DJANGO_EMAIL_HOST=<smtp host>
DJANGO_EMAIL_HOST_USER=<smtp user>
env.d/production.dist/common
+2 -2
View File
@@ -4,7 +4,7 @@ S3_HOST=storage.domain.tld
BACKEND_HOST=backend
FRONTEND_HOST=frontend
YPROVIDER_HOST=y-provider
YHUB_HOST=yhub
BUCKET_NAME=docs-media-storage
REALM_NAME=docs
# TODO(yhub): route is /ws/docs once prod ingress is migrated
#COLLABORATION_WS_URL=wss://${DOCS_HOST}/collaboration/ws/
#COLLABORATION_WS_URL=wss://${DOCS_HOST}/collaboration/ws/v1/docs
env.d/production.dist/yhub
+44
View File
@@ -0,0 +1,44 @@
# Collaboration server (yhub)
#
# It holds the content of the documents: the editors sync with it over the
# websocket, and the backend reads and writes documents through its REST API.
# Everything it reads is here, it loads none of the backend's own environment.
# Stores. The live state of a document is in valkey until a worker persists it
# to postgres, so this valkey is not a cache: losing it loses what has not been
# persisted yet. Give it its own instance, not the one Django caches in.
PORT=3002
REDIS=redis://yhub-valkey:6379
REDIS_PREFIX=yhub
# Its own database, next to the Django one on the same server in this example.
# The schema is created by `docker compose run --rm yhub npm run init-db`,
# which also creates the database when the user is allowed to.
POSTGRES=postgres://docs:<the DB_PASSWORD of env.d/postgresql>@postgresql:5432/yhub
# Backend. It answers who a user is and what they may do with a document, and
# publishes the JWKS the admin tokens it signs are verified against. The origin
# list is what a browser may open a websocket from.
COLLABORATION_BACKEND_BASE_URL=https://${DOCS_HOST}
COLLABORATION_SERVER_ORIGIN=https://${DOCS_HOST}
# Sent as X-Y-Provider-Key on the calls made to the backend, which is how they
# are exempt from the API throttling — one call per connection would hit it.
# The same value as in env.d/yprovider, where the backend reads the one it
# compares it to.
Y_PROVIDER_API_KEY=<the same random key as in env.d/yprovider>
# Signs the calls made to the backend, which reads the public half from the
# JWKS this server publishes. Its own key, never the backend's.
YHUB_JWT_PRIVATE_KEY_FILE=/keys/yhub-private.pem
# Soft migration: upgrading an instance whose documents were stored in the S3
# media bucket, seed a room from that bucket the first time it is opened. A new
# instance has nothing to migrate and leaves this off. Read the "Upgrading from
# a release before the collaboration server" section of
# documentation/installation/compose.md before turning it on.
#SOFT_MIGRATION=true
#LEGACY_S3_ENDPOINT_URL=https://${S3_HOST}
#LEGACY_S3_ACCESS_KEY_ID=<s3 access key, read-only is enough>
#LEGACY_S3_SECRET_ACCESS_KEY=<s3 secret key>
#LEGACY_S3_BUCKET_NAME=${BUCKET_NAME}
#LEGACY_S3_REGION_NAME=<region, if your provider needs one>
#LEGACY_S3_SIGNATURE_VERSION=s3v4
env.d/production.dist/yprovider
+7 -1
View File
@@ -1,5 +1,11 @@
# Conversion service (y-provider), and the backend side of it: this file is
# loaded by both. It serves the format conversion and nothing else — the
# collaboration moved to the yhub service, see env.d/yhub.
Y_PROVIDER_API_BASE_URL=http://${YPROVIDER_HOST}:4444/api/
# Exempts the calls the collaboration server makes to the backend from the API
# throttling. Repeat the value in env.d/yhub, which is what sends it.
Y_PROVIDER_API_KEY=<generate a random key>
COLLABORATION_SERVER_ORIGIN=https://${DOCS_HOST}
# The backend it fetches the JWKS from, to verify the tokens it is called with.
COLLABORATION_BACKEND_BASE_URL=https://${DOCS_HOST}
COLLABORATION_LOGGING=true
COLLABORATION_LOGGING=true