🔧(thub) allow to configure task concurrency

We want to configure the number of concurrency tasks a work run. For
this a new environment variable YHUB_TASK_CONCURRENCY is added

CHANGELOG.md

@@ -163,7 +163,9 @@ and this project adheres to
   In the helm chart, `yhub.worker.enabled` turns the single deployment into
   two, sets the variable on each, and gives the worker no service and no probes
   since it binds nothing; everything not named under `yhub.worker` is the
-  server's
+  server's. `YHUB_TASK_CONCURRENCY` (default 5, unchanged) sets how many tasks
+  one worker process claims at once — the other half of the throughput knob the
+  replica count is, since redis hands each task to a single worker
 - ✨(collaboration) serve two probes on yhub: `GET /collaboration/ping/v1`
   answers `pong` without touching a store — being answered is what a liveness
   check should conclude, and restarting a server over a store it cannot reach

src/helm/env.d/dev/values.impress.yaml.gotmpl

@@ -219,6 +219,7 @@ yhub:
     COLLABORATION_SERVER_ORIGIN: https://docs.127.0.0.1.nip.io
     NODE_EXTRA_CA_CERTS: /cert/cacert.pem
     # YHUB_JWT_PRIVATE_KEY_FILE comes from the jwtKeys job below
+    LOG_LEVEL: debug
 
   # Extra volume mounts to manage our local custom CA and avoid to set ssl_verify: false
   extraVolumeMounts:

src/helm/env.d/feature/values.impress.yaml.gotmpl

@@ -180,6 +180,8 @@ yhub:
     COLLABORATION_BACKEND_BASE_URL: https://{{ .Values.feature }}-docs.{{ .Values.domain }}
     COLLABORATION_SERVER_ORIGIN: https://{{ .Values.feature }}-docs.{{ .Values.domain }}
     NODE_OPTIONS: "--max-old-space-size=1024"
+    UWS_HTTP_MAX_HEADERS_SIZE: 32768
+    LOG_LEVEL: debug
 
 docSpec:
   enabled: true

src/helm/impress/README.md

@@ -374,6 +374,7 @@
 | `yhub.envVars.COLLABORATION_BACKEND_BASE_URL`      | Base url of the Docs backend, which yhub asks about users and document access rights                          |                           |
 | `yhub.envVars.COLLABORATION_SERVER_ORIGIN`         | Comma separated list of the origins allowed to open a websocket                                               |                           |
 | `yhub.envVars.YHUB_JWT_PRIVATE_KEY_FILE`           | Path to the RSA private key (PEM) yhub signs its calls to the backend with, mounted from a secret             |                           |
+| `yhub.envVars.YHUB_TASK_CONCURRENCY`               | Tasks one worker process claims at once, times the replicas running a worker (default: 5)                     |                           |
 | `yhub.envVars.SOFT_MIGRATION`                      | Set to "true" to seed rooms from the legacy Django/S3 document store on first access                          |                           |
 | `yhub.envVars.BY_VALUE`                            | Example environment variable by setting value directly                                                        |                           |
 | `yhub.envVars.FROM_CONFIGMAP.configMapKeyRef.name` | Name of a ConfigMap when configuring env vars from a ConfigMap                                                |                           |

src/helm/impress/templates/_helpers.tpl

@@ -215,12 +215,18 @@ Requires top level scope
 
 {{/*
 yhub worker env vars - combines common yhub.envVars with yhub.worker.envVars
+
+Merged rather than appended: a variable the worker sets differently from the
+server (YHUB_TASK_CONCURRENCY, typically) is meant to replace it, and emitting
+both would leave the value to kubernetes' last-one-wins rule and show the
+variable twice in the pod. deepCopy because merge writes into its first
+argument, which is a live values map.
 */}}
 {{- define "impress.yhub.worker.env" -}}
 {{- $topLevelScope := index . 0 -}}
 {{- $workerScope := index . 1 -}}
-{{- include "impress.env.transformDict" $workerScope.envVars -}}
-{{- include "impress.env.transformDict" (($workerScope.worker | default dict).envVars | default dict) -}}
+{{- $workerEnvVars := ($workerScope.worker | default dict).envVars | default dict -}}
+{{- include "impress.env.transformDict" (merge (deepCopy $workerEnvVars) $workerScope.envVars) -}}
 {{- end }}
 
 {{/*

src/helm/impress/values.yaml

@@ -955,6 +955,11 @@ yhub:
   ## The worker binds nothing: no service, no ingress, and no probes to give it
   ## (its liveness is its process). Everything not named here is the server's:
   ## same image, same envVars, same secrets, same volumes.
+  ##
+  ## How much each pod chews through is `worker.envVars.YHUB_TASK_CONCURRENCY`
+  ## (default 5, and `yhub.envVars` when the two halves share a process), the
+  ## other half of the throughput knob `worker.replicas` is: redis hands each
+  ## task to a single worker, so the two multiply.
   worker:
     enabled: false
     replicas: 1
@@ -1008,6 +1013,7 @@ yhub:
   ## @extra yhub.envVars.COLLABORATION_BACKEND_BASE_URL Base url of the Docs backend, which yhub asks about users and document access rights
   ## @extra yhub.envVars.COLLABORATION_SERVER_ORIGIN Comma separated list of the origins allowed to open a websocket
   ## @extra yhub.envVars.YHUB_JWT_PRIVATE_KEY_FILE Path to the RSA private key (PEM) yhub signs its calls to the backend with, mounted from a secret
+  ## @extra yhub.envVars.YHUB_TASK_CONCURRENCY Tasks one worker process claims at once, times the replicas running a worker (default: 5)
   ## @extra yhub.envVars.SOFT_MIGRATION Set to "true" to seed rooms from the legacy Django/S3 document store on first access
   ## @extra yhub.envVars.BY_VALUE Example environment variable by setting value directly
   ## @extra yhub.envVars.FROM_CONFIGMAP.configMapKeyRef.name Name of a ConfigMap when configuring env vars from a ConfigMap

src/yhub-server/README.md

@@ -119,6 +119,14 @@ Redis consumer groups hand each task to exactly one worker, so the number of
 workers is a throughput knob and nothing else: no leader, no partitioning, no
 coordination between them.
 
+`YHUB_TASK_CONCURRENCY` (default `5`) is the other half of that knob: how many
+tasks one process claims at once. What actually runs in parallel is that number
+times the processes running a worker, so the two are interchangeable up to the
+point where a pod runs out of memory — each task holds the document it merges.
+A value that is not a positive integer is refused at startup, like an unknown
+role: `Number()` would otherwise read a typo as `NaN` and leave the worker
+claiming nothing.
+
 ## Container image
 
 The `Dockerfile` has two final stages, like the other services of this

src/yhub-server/server.js

@@ -57,6 +57,18 @@ if (!['all', 'server', 'worker'].includes(ROLE)) {
 }
 const RUNS_SERVER = ROLE !== 'worker';
 const RUNS_WORKER = ROLE !== 'server';
+// How many tasks one worker process claims at once. Redis hands each task to a
+// single worker, so what a deployment actually runs in parallel is this times
+// the number of worker processes — the two knobs are interchangeable up to the
+// point where a pod runs out of memory, each task holding the document it
+// merges. Refused rather than guessed when it is not a positive integer:
+// `Number()` would otherwise turn a typo into NaN and yhub into an idle worker.
+const TASK_CONCURRENCY = Number(process.env.YHUB_TASK_CONCURRENCY || 5);
+if (!Number.isInteger(TASK_CONCURRENCY) || TASK_CONCURRENCY < 1) {
+  throw new Error(
+    `YHUB_TASK_CONCURRENCY must be a positive integer (got "${process.env.YHUB_TASK_CONCURRENCY}")`,
+  );
+}
 // Segment every route is mounted under (`server.apiPrefix` below), matching the
 // URL scheme Docs already routes to the collaboration server. Hardcoded like
 // the audiences: the backend builds its urls with the same prefix.
@@ -845,10 +857,17 @@ const yhub = await createYHub({
   server: RUNS_SERVER
     ? { port: PORT, auth, api, apiPrefix: API_PREFIX }
     : null,
-  worker: RUNS_WORKER ? { taskConcurrency: 5, events: workerEvents } : null,
+  worker: RUNS_WORKER
+    ? { taskConcurrency: TASK_CONCURRENCY, events: workerEvents }
+    : null,
 });
 
 logger.info(
-  { role: ROLE, server: RUNS_SERVER, worker: RUNS_WORKER },
+  {
+    role: ROLE,
+    server: RUNS_SERVER,
+    worker: RUNS_WORKER,
+    taskConcurrency: RUNS_WORKER ? TASK_CONCURRENCY : null,
+  },
   'yhub role',
 );