✨(loadtest) add k6 scenarios for the backend application

To stress test the backend application we will use the k6 project from
grafana. 2 scenarios are implemented and documented in the README. How
to use this project is also documented in the README

.gitignore

@@ -15,6 +15,7 @@ downloads/
 eggs/
 .eggs/
 lib/
+!src/loadtest/k6/lib/
 lib64/
 parts/
 sdist/

CHANGELOG.md

@@ -15,6 +15,8 @@ and this project adheres to
 - ✨(backend) expose the attachment max size in the config endpoint #2577
 - ✨(frontend) warn before uploading an attachment over the size limit #2577
 - ✨(loadtest) add a websocket load generator for the collaboration server
+- ✨(loadtest) add k6 scenarios for the page-open sequence and the heavy
+  endpoints
 - ✨(backend) measure the calls to yhub and to the converters, the database
   pool and the celery queue
 - ✨(backend) add a `LoadTest` configuration and its `loadtest` application,

src/loadtest/k6/README.md

@@ -0,0 +1,96 @@
+# k6 — HTTP load scenarios for the backend
+
+The HTTP half of `documentation/stress-test-plan.md`: what a browser asks the
+Django backend when a user opens a document, and the endpoints that call the
+collaboration server or walk a subtree inside one request. The websocket half is
+`../swarm`.
+
+Plain [k6](https://grafana.com/docs/k6/) scripts, no build step, no
+dependencies: the `grafana/k6` image runs them as they are.
+
+## Who the virtual users are
+
+The scripts log nobody in. They read the manifest that the backend's
+`create_load_test_sessions` command writes (`src/backend/loadtest`, only with
+the `LoadTest` configuration): the cookie name, and per logged-in user a session
+key and the documents that user may edit or read. **The manifest holds live
+sessions and is a secret**: hand it to k6 and to nothing else, and revoke the
+sessions when the campaign is over.
+
+Each VU keeps the session of its rank. With more VUs than sessions a user is
+logged in several times, and the API throttles per user (80 requests a minute
+on the document endpoints): ask for at least as many sessions as VUs.
+
+Unsafe requests carry Django's double-submit CSRF pair (a `csrftoken` cookie and
+the same value in `X-CSRFToken`) and the `Origin` of the application, which has
+to be one of the backend's `CSRF_TRUSTED_ORIGINS`.
+
+## Running
+
+```bash
+docker run --rm -v ./src/loadtest/k6:/k6:ro -v ./manifest.json:/manifest.json:ro \
+    -e MANIFEST=/manifest.json -e BASE_URL=https://docs.example.com \
+    -e RATE=50 -e DURATION=10m -w /k6 grafana/k6 run scenarios/page-open.js
+```
+
+| Variable | Default | What it is |
+| -------- | ------- | ---------- |
+| `MANIFEST` | `/manifest.json` | The manifest file |
+| `BASE_URL` | `http://localhost:8071` | Where the API is, without `/api/v1.0` |
+| `ORIGIN` | `BASE_URL` | The frontend's origin, for CSRF. `http://localhost:3000` in the dev stack |
+| `MEDIA_BASE_URL` | `https://docs.example.com` | The host of the media urls `media-auth` is asked about |
+
+### `scenarios/page-open.js` — the HTTP baseline (plan, scenario 2)
+
+The sequence the frontend runs when a document is opened: `config/`,
+`users/me/`, `documents/{id}/`, `documents/{id}/tree/`, `documents/?page=1`,
+and the `media-auth` subrequest nginx makes when the page loads an attachment.
+A rate-based scenario: `RATE` page opens per second (5 to 6 requests each),
+ramped up over `RAMP` (default `1m`), held for `DURATION` (default `5m`),
+ramped down. `VUS` (default `4 × RATE`, at least 10) bounds the concurrency
+when the API slows down: past it, iterations are dropped and
+`dropped_iterations` says so.
+
+Run it against the current release and against this branch, on the same data,
+with the same options: what differs is what the new architecture costs on the
+HTTP side.
+
+The `media-auth` call names a file the bucket does not hold: the access check
+— the costly part, and what brought production down on 2026-08-18 — runs, and
+the answer is a 403 once it has passed. That status is expected and not counted
+as a failure.
+
+### `scenarios/heavy.js` — the synchronous endpoints (plan, scenario 9)
+
+`VUS` (default 5) users, for `DURATION` (default `5m`), each looping on:
+`duplicate` of one of their documents with its descendants (two collaboration
+server round-trips per node, inside a transaction), `formatted-content` of the
+copy, then delete, restore and delete of the copy (each a Celery task walking
+the subtree), and the creation of a document from a markdown file (converted,
+then handed to the collaboration server) which is deleted too.
+
+The users' own documents are only read. What an iteration creates it deletes,
+but a deleted document stays in the trash for the retention period: the
+database grows by one subtree per iteration for as long as that lasts.
+
+## Reading the results
+
+Every request is tagged with a `name` free of identifiers
+(`documents/{id}/`, not the url), so the per-endpoint percentiles k6 prints are
+one series per endpoint. Thresholds are set in each script; k6 exits non-zero
+when one is crossed.
+
+To see the run next to the server-side metrics, send k6's metrics to the
+Prometheus of the campaign:
+
+```bash
+-e K6_PROMETHEUS_RW_SERVER_URL=http://prometheus:9090/api/v1/write \
+grafana/k6 run -o experimental-prometheus-rw scenarios/page-open.js
+```
+
+## Where to run it
+
+Like the swarm: in the cluster, on nodes that do not host the application,
+through the ingress. A k6 process drives a few hundred requests a second
+comfortably; past that, run several with distinct slices of the manifest's
+`sessions`.

src/loadtest/k6/lib/api.js

@@ -0,0 +1,77 @@
+/**
+ * Calls to the Docs API as a logged-in browser would make them: the session
+ * cookie, and on unsafe methods Django's double-submit CSRF token (a cookie and
+ * the same value in `X-CSRFToken`) with the `Origin` of the application.
+ *
+ * Every request is tagged with a `name` free of identifiers (`documents/{id}/`
+ * rather than the url): k6 keeps one time series per distinct tag value, and
+ * one per document would be one per document.
+ */
+import http from 'k6/http';
+import { check } from 'k6';
+
+import { COOKIE_NAME } from './session.js';
+
+export const BASE_URL = (__ENV.BASE_URL || 'http://localhost:8071').replace(/\/+$/, '');
+export const API = `${BASE_URL}/api/v1.0`;
+// The `Origin` a browser on the application sends, which Django checks a POST
+// against (CSRF_TRUSTED_ORIGINS): the frontend's origin, which is BASE_URL in a
+// deployment where /api is routed on the application's host, and something else
+// in the dev stack (`http://localhost:3000`).
+export const ORIGIN = __ENV.ORIGIN || BASE_URL;
+// Django accepts any 32 alphanumeric characters as long as the cookie and the header agree
+const CSRF_TOKEN = 'k6loadtestk6loadtestk6loadtest00';
+
+const headers = (session, unsafe) => ({
+  Accept: 'application/json',
+  'Content-Type': 'application/json',
+  Cookie: unsafe
+    ? `${COOKIE_NAME}=${session.session_key}; csrftoken=${CSRF_TOKEN}`
+    : `${COOKIE_NAME}=${session.session_key}`,
+  ...(unsafe ? { 'X-CSRFToken': CSRF_TOKEN, Origin: ORIGIN } : {}),
+});
+
+const params = (session, name, unsafe, extra = {}) => ({
+  headers: { ...headers(session, unsafe), ...(extra.headers || {}) },
+  tags: { name, ...(extra.tags || {}) },
+  // the cookie is in the header: k6's own jar must not add what the server sets
+  jar: http.cookieJar(),
+  timeout: extra.timeout || '60s',
+  // what `http_req_failed` counts: an answer the scenario expects is not a
+  // failure. The key is only set when asked for — set to `undefined` it turns
+  // k6's default classification off
+  ...(extra.expected ? { responseCallback: http.expectedStatuses(...extra.expected) } : {}),
+});
+
+export const get = (session, path, name, extra) =>
+  http.get(`${API}/${path}`, params(session, name, false, extra));
+
+export const post = (session, path, body, name, extra) =>
+  http.post(`${API}/${path}`, body == null ? null : JSON.stringify(body), params(session, name, true, extra));
+
+export const del = (session, path, name, extra) =>
+  http.del(`${API}/${path}`, null, params(session, name, true, extra));
+
+/**
+ * Check the status, under the request's name. An unexpected answer is logged
+ * with the start of its body: the summary only says how many failed, not why.
+ */
+export const expect = (response, name, ...statuses) => {
+  const ok = check(response, { [`${name} → ${statuses.join('|')}`]: (r) => statuses.includes(r.status) }, { name });
+  if (!ok) {
+    console.warn(`${name}: ${response.status} ${String(response.body || '').slice(0, 200)}`);
+  }
+  return ok;
+};
+
+/** A multipart POST (a file upload): the same cookie and CSRF pair, no JSON content type. */
+export const postMultipart = (session, path, body, name, extra = {}) => {
+  const { 'Content-Type': _json, ...rest } = headers(session, true);
+  return http.post(`${API}/${path}`, body, {
+    headers: rest,
+    tags: { name },
+    jar: http.cookieJar(),
+    timeout: extra.timeout || '120s',
+    ...(extra.expected ? { responseCallback: http.expectedStatuses(...extra.expected) } : {}),
+  });
+};

src/loadtest/k6/lib/session.js

@@ -0,0 +1,41 @@
+/**
+ * The users of a run: the manifest `create_load_test_sessions` writes
+ * (src/backend/loadtest), one session per virtual user.
+ *
+ * k6 reads the file once, in the init context, and every VU keeps the session
+ * of its rank: with more VUs than sessions a user is logged in several times,
+ * and the API throttles per user (80/min on documents) — ask for at least as
+ * many sessions as VUs.
+ */
+import { SharedArray } from 'k6/data';
+
+const manifest = JSON.parse(open(__ENV.MANIFEST || '/manifest.json'));
+if (!manifest.cookie_name || !manifest.sessions || manifest.sessions.length === 0) {
+  throw new Error('The manifest holds no session');
+}
+if (manifest.expires_at && Date.parse(manifest.expires_at) < Date.now()) {
+  throw new Error(`The sessions of the manifest expired at ${manifest.expires_at}`);
+}
+
+// shared between the VUs rather than copied into each of them
+const sessions = new SharedArray('sessions', () => manifest.sessions);
+const publicDocuments = new SharedArray('public documents', () => manifest.public_documents || []);
+
+export const COOKIE_NAME = manifest.cookie_name;
+export const sessionCount = () => sessions.length;
+
+/** The session of this VU. */
+export const mySession = () => sessions[(__VU - 1) % sessions.length];
+
+/** A document of the VU's user, `editable` or any it may read; a public one when it has none. */
+export const myDocument = (session, editable = false) => {
+  const own = editable
+    ? session.editable_documents
+    : session.readonly_documents.concat(session.editable_documents);
+  const pool = own.length > 0 ? own : publicDocuments;
+  if (pool.length === 0) return null;
+  return pool[__ITER % pool.length];
+};
+
+export const publicDocument = () =>
+  publicDocuments.length > 0 ? publicDocuments[__ITER % publicDocuments.length] : null;

src/loadtest/k6/scenarios/heavy.js

@@ -0,0 +1,90 @@
+/**
+ * The endpoints that call the collaboration server or walk a subtree inside one
+ * request (scenario 9 of the plan): `duplicate` with descendants — two yhub
+ * round-trips per node, inside a transaction — `formatted-content`, document
+ * creation from a file, and the delete/restore cascades that land on the single
+ * Celery queue.
+ *
+ * Every iteration works on documents it creates itself and deletes at the end:
+ * a duplicate of one of the user's own documents, then the duplicate is
+ * deleted, restored and deleted again. The user's documents are only read. The
+ * deleted copies stay in the trash for the retention period, so the database
+ * grows by one subtree per iteration for as long as that lasts.
+ *
+ *   k6 run --env MANIFEST=manifest.json --env BASE_URL=https://docs.example.com \
+ *          --env VUS=5 --env DURATION=5m scenarios/heavy.js
+ */
+import { group, sleep } from 'k6';
+import http from 'k6/http';
+
+import { del, expect, get, post, postMultipart } from '../lib/api.js';
+import { myDocument, mySession } from '../lib/session.js';
+
+const VUS = Number(__ENV.VUS || 5);
+const DURATION = __ENV.DURATION || '5m';
+const MARKDOWN = '# k6\n\nA document created by the load test, and deleted by it.\n';
+
+export const options = {
+  scenarios: {
+    heavy: {
+      executor: 'constant-vus',
+      vus: VUS,
+      duration: DURATION,
+    },
+  },
+  thresholds: {
+    http_req_failed: ['rate<0.05'],
+    'http_req_duration{name:documents/{id}/duplicate/}': ['p(95)<10000'],
+    'http_req_duration{name:documents/{id}/formatted-content/}': ['p(95)<5000'],
+    'http_req_duration{name:documents/ (create from file)}': ['p(95)<10000'],
+    checks: ['rate>0.95'],
+  },
+  summaryTrendStats: ['avg', 'min', 'med', 'p(90)', 'p(95)', 'p(99)', 'max'],
+};
+
+export default function () {
+  const session = mySession();
+  const doc = myDocument(session, true);
+  if (!doc) return;
+
+  group('duplicate with descendants', () => {
+    const res = post(
+      session,
+      `documents/${doc}/duplicate/`,
+      { with_accesses: false, with_descendants: true },
+      'documents/{id}/duplicate/',
+    );
+    if (!expect(res, 'documents/{id}/duplicate/', 201)) return;
+    const copy = res.json('id');
+
+    group('formatted content', () => {
+      expect(
+        get(session, `documents/${copy}/formatted-content/?content_format=markdown`, 'documents/{id}/formatted-content/'),
+        'documents/{id}/formatted-content/',
+        200,
+      );
+    });
+
+    group('delete, restore, delete', () => {
+      // each delete and each restore walks the subtree from a Celery task
+      expect(del(session, `documents/${copy}/`, 'documents/{id}/ (delete)'), 'documents/{id}/ (delete)', 204);
+      expect(post(session, `documents/${copy}/restore/`, null, 'documents/{id}/restore/'), 'documents/{id}/restore/', 200);
+      expect(del(session, `documents/${copy}/`, 'documents/{id}/ (delete)'), 'documents/{id}/ (delete)', 204);
+    });
+  });
+
+  group('create from file', () => {
+    // the file is converted, then handed to yhub, before the response
+    const res = postMultipart(
+      session,
+      'documents/',
+      { file: http.file(MARKDOWN, 'k6.md', 'text/markdown'), title: 'k6 import' },
+      'documents/ (create from file)',
+    );
+    if (expect(res, 'documents/ (create from file)', 201)) {
+      expect(del(session, `documents/${res.json('id')}/`, 'documents/{id}/ (delete)'), 'documents/{id}/ (delete)', 204);
+    }
+  });
+
+  sleep(1);
+}

src/loadtest/k6/scenarios/page-open.js

@@ -0,0 +1,87 @@
+/**
+ * What the browser does when a user opens a document, in the order the
+ * frontend does it — `config`, `users/me`, the document, its tree, the list of
+ * the user's documents — plus the `media-auth` subrequest nginx makes when the
+ * page loads an attachment. It is the HTTP baseline of the plan
+ * (`documentation/stress-test-plan.md`, scenario 2): run it against the
+ * current release and against this branch, on the same data, with the same
+ * options.
+ *
+ *   k6 run --env MANIFEST=manifest.json --env BASE_URL=https://docs.example.com \
+ *          --env RATE=50 --env DURATION=10m scenarios/page-open.js
+ *
+ * RATE is in page opens per second, ramped up over RAMP (default 1m), held for
+ * DURATION, then ramped down over RAMP. Each open is 5 to 6 requests.
+ */
+import { group, sleep } from 'k6';
+
+import { expect, get } from '../lib/api.js';
+import { myDocument, mySession, sessionCount } from '../lib/session.js';
+
+const RATE = Number(__ENV.RATE || 10);
+const RAMP = __ENV.RAMP || '1m';
+const DURATION = __ENV.DURATION || '5m';
+// one VU serves about one open per second when the API answers in well under
+// that; more are preallocated so that a slow API does not starve the rate
+const VUS = Number(__ENV.VUS || Math.max(10, RATE * 4));
+
+export const options = {
+  scenarios: {
+    page_open: {
+      executor: 'ramping-arrival-rate',
+      startRate: 1,
+      timeUnit: '1s',
+      preAllocatedVUs: VUS,
+      maxVUs: VUS,
+      stages: [
+        { target: RATE, duration: RAMP },
+        { target: RATE, duration: DURATION },
+        { target: 0, duration: RAMP },
+      ],
+    },
+  },
+  thresholds: {
+    http_req_failed: ['rate<0.01'],
+    'http_req_duration{name:documents/{id}/}': ['p(95)<1000'],
+    'http_req_duration{name:documents/{id}/tree/}': ['p(95)<1000'],
+    'http_req_duration{name:documents/}': ['p(95)<1000'],
+    'http_req_duration{name:documents/media-auth/}': ['p(95)<500'],
+    checks: ['rate>0.99'],
+  },
+  summaryTrendStats: ['avg', 'min', 'med', 'p(90)', 'p(95)', 'p(99)', 'max'],
+};
+
+export function setup() {
+  if (VUS > sessionCount()) {
+    console.warn(`${VUS} VUs for ${sessionCount()} sessions: some users are logged in several times, and the API throttles per user`);
+  }
+}
+
+export default function () {
+  const session = mySession();
+  const doc = myDocument(session);
+  if (!doc) return;
+
+  group('open', () => {
+    expect(get(session, 'config/', 'config/'), 'config/', 200);
+    expect(get(session, 'users/me/', 'users/me/'), 'users/me/', 200);
+    expect(get(session, `documents/${doc}/`, 'documents/{id}/'), 'documents/{id}/', 200);
+    expect(get(session, `documents/${doc}/tree/`, 'documents/{id}/tree/'), 'documents/{id}/tree/', 200);
+    expect(get(session, 'documents/?page=1&ordering=-updated_at', 'documents/'), 'documents/', 200);
+    // the auth subrequest nginx makes for `/media/{doc}/attachments/{file}`. The
+    // file need not exist for the access check — the costly part, and what
+    // brought production down on 2026-08-18 — to run: a file the bucket does
+    // not hold answers 403 once that check has passed
+    const original = `${__ENV.MEDIA_BASE_URL || 'https://docs.example.com'}/media/${doc}/attachments/00000000-0000-4000-8000-000000000000.png`;
+    expect(
+      get(session, 'documents/media-auth/', 'documents/media-auth/', {
+        headers: { 'X-Original-URL': original },
+        expected: [200, 403],
+      }),
+      'documents/media-auth/',
+      200, 403,
+    );
+  });
+  // a real user does not open the next document at once
+  sleep(Math.random() * 2);
+}