diff --git a/.gitignore b/.gitignore index 631e0bf09..33e82438a 100644 --- a/.gitignore +++ b/.gitignore @@ -15,6 +15,7 @@ downloads/ eggs/ .eggs/ lib/ +!src/loadtest/k6/lib/ lib64/ parts/ sdist/ diff --git a/CHANGELOG.md b/CHANGELOG.md index bbfb7345c..a4562879a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,8 @@ and this project adheres to - ✨(backend) expose the attachment max size in the config endpoint #2577 - ✨(frontend) warn before uploading an attachment over the size limit #2577 - ✨(loadtest) add a websocket load generator for the collaboration server +- ✨(loadtest) add k6 scenarios for the page-open sequence and the heavy + endpoints - ✨(backend) measure the calls to yhub and to the converters, the database pool and the celery queue - ✨(backend) add a `LoadTest` configuration and its `loadtest` application, diff --git a/src/loadtest/k6/README.md b/src/loadtest/k6/README.md new file mode 100644 index 000000000..10f45eeee --- /dev/null +++ b/src/loadtest/k6/README.md @@ -0,0 +1,96 @@ +# k6 — HTTP load scenarios for the backend + +The HTTP half of `documentation/stress-test-plan.md`: what a browser asks the +Django backend when a user opens a document, and the endpoints that call the +collaboration server or walk a subtree inside one request. The websocket half is +`../swarm`. + +Plain [k6](https://grafana.com/docs/k6/) scripts, no build step, no +dependencies: the `grafana/k6` image runs them as they are. + +## Who the virtual users are + +The scripts log nobody in. They read the manifest that the backend's +`create_load_test_sessions` command writes (`src/backend/loadtest`, only with +the `LoadTest` configuration): the cookie name, and per logged-in user a session +key and the documents that user may edit or read. **The manifest holds live +sessions and is a secret**: hand it to k6 and to nothing else, and revoke the +sessions when the campaign is over. + +Each VU keeps the session of its rank. With more VUs than sessions a user is +logged in several times, and the API throttles per user (80 requests a minute +on the document endpoints): ask for at least as many sessions as VUs. + +Unsafe requests carry Django's double-submit CSRF pair (a `csrftoken` cookie and +the same value in `X-CSRFToken`) and the `Origin` of the application, which has +to be one of the backend's `CSRF_TRUSTED_ORIGINS`. + +## Running + +```bash +docker run --rm -v ./src/loadtest/k6:/k6:ro -v ./manifest.json:/manifest.json:ro \ + -e MANIFEST=/manifest.json -e BASE_URL=https://docs.example.com \ + -e RATE=50 -e DURATION=10m -w /k6 grafana/k6 run scenarios/page-open.js +``` + +| Variable | Default | What it is | +| -------- | ------- | ---------- | +| `MANIFEST` | `/manifest.json` | The manifest file | +| `BASE_URL` | `http://localhost:8071` | Where the API is, without `/api/v1.0` | +| `ORIGIN` | `BASE_URL` | The frontend's origin, for CSRF. `http://localhost:3000` in the dev stack | +| `MEDIA_BASE_URL` | `https://docs.example.com` | The host of the media urls `media-auth` is asked about | + +### `scenarios/page-open.js` — the HTTP baseline (plan, scenario 2) + +The sequence the frontend runs when a document is opened: `config/`, +`users/me/`, `documents/{id}/`, `documents/{id}/tree/`, `documents/?page=1`, +and the `media-auth` subrequest nginx makes when the page loads an attachment. +A rate-based scenario: `RATE` page opens per second (5 to 6 requests each), +ramped up over `RAMP` (default `1m`), held for `DURATION` (default `5m`), +ramped down. `VUS` (default `4 × RATE`, at least 10) bounds the concurrency +when the API slows down: past it, iterations are dropped and +`dropped_iterations` says so. + +Run it against the current release and against this branch, on the same data, +with the same options: what differs is what the new architecture costs on the +HTTP side. + +The `media-auth` call names a file the bucket does not hold: the access check +— the costly part, and what brought production down on 2026-08-18 — runs, and +the answer is a 403 once it has passed. That status is expected and not counted +as a failure. + +### `scenarios/heavy.js` — the synchronous endpoints (plan, scenario 9) + +`VUS` (default 5) users, for `DURATION` (default `5m`), each looping on: +`duplicate` of one of their documents with its descendants (two collaboration +server round-trips per node, inside a transaction), `formatted-content` of the +copy, then delete, restore and delete of the copy (each a Celery task walking +the subtree), and the creation of a document from a markdown file (converted, +then handed to the collaboration server) which is deleted too. + +The users' own documents are only read. What an iteration creates it deletes, +but a deleted document stays in the trash for the retention period: the +database grows by one subtree per iteration for as long as that lasts. + +## Reading the results + +Every request is tagged with a `name` free of identifiers +(`documents/{id}/`, not the url), so the per-endpoint percentiles k6 prints are +one series per endpoint. Thresholds are set in each script; k6 exits non-zero +when one is crossed. + +To see the run next to the server-side metrics, send k6's metrics to the +Prometheus of the campaign: + +```bash +-e K6_PROMETHEUS_RW_SERVER_URL=http://prometheus:9090/api/v1/write \ +grafana/k6 run -o experimental-prometheus-rw scenarios/page-open.js +``` + +## Where to run it + +Like the swarm: in the cluster, on nodes that do not host the application, +through the ingress. A k6 process drives a few hundred requests a second +comfortably; past that, run several with distinct slices of the manifest's +`sessions`. diff --git a/src/loadtest/k6/lib/api.js b/src/loadtest/k6/lib/api.js new file mode 100644 index 000000000..fb2d97deb --- /dev/null +++ b/src/loadtest/k6/lib/api.js @@ -0,0 +1,77 @@ +/** + * Calls to the Docs API as a logged-in browser would make them: the session + * cookie, and on unsafe methods Django's double-submit CSRF token (a cookie and + * the same value in `X-CSRFToken`) with the `Origin` of the application. + * + * Every request is tagged with a `name` free of identifiers (`documents/{id}/` + * rather than the url): k6 keeps one time series per distinct tag value, and + * one per document would be one per document. + */ +import http from 'k6/http'; +import { check } from 'k6'; + +import { COOKIE_NAME } from './session.js'; + +export const BASE_URL = (__ENV.BASE_URL || 'http://localhost:8071').replace(/\/+$/, ''); +export const API = `${BASE_URL}/api/v1.0`; +// The `Origin` a browser on the application sends, which Django checks a POST +// against (CSRF_TRUSTED_ORIGINS): the frontend's origin, which is BASE_URL in a +// deployment where /api is routed on the application's host, and something else +// in the dev stack (`http://localhost:3000`). +export const ORIGIN = __ENV.ORIGIN || BASE_URL; +// Django accepts any 32 alphanumeric characters as long as the cookie and the header agree +const CSRF_TOKEN = 'k6loadtestk6loadtestk6loadtest00'; + +const headers = (session, unsafe) => ({ + Accept: 'application/json', + 'Content-Type': 'application/json', + Cookie: unsafe + ? `${COOKIE_NAME}=${session.session_key}; csrftoken=${CSRF_TOKEN}` + : `${COOKIE_NAME}=${session.session_key}`, + ...(unsafe ? { 'X-CSRFToken': CSRF_TOKEN, Origin: ORIGIN } : {}), +}); + +const params = (session, name, unsafe, extra = {}) => ({ + headers: { ...headers(session, unsafe), ...(extra.headers || {}) }, + tags: { name, ...(extra.tags || {}) }, + // the cookie is in the header: k6's own jar must not add what the server sets + jar: http.cookieJar(), + timeout: extra.timeout || '60s', + // what `http_req_failed` counts: an answer the scenario expects is not a + // failure. The key is only set when asked for — set to `undefined` it turns + // k6's default classification off + ...(extra.expected ? { responseCallback: http.expectedStatuses(...extra.expected) } : {}), +}); + +export const get = (session, path, name, extra) => + http.get(`${API}/${path}`, params(session, name, false, extra)); + +export const post = (session, path, body, name, extra) => + http.post(`${API}/${path}`, body == null ? null : JSON.stringify(body), params(session, name, true, extra)); + +export const del = (session, path, name, extra) => + http.del(`${API}/${path}`, null, params(session, name, true, extra)); + +/** + * Check the status, under the request's name. An unexpected answer is logged + * with the start of its body: the summary only says how many failed, not why. + */ +export const expect = (response, name, ...statuses) => { + const ok = check(response, { [`${name} → ${statuses.join('|')}`]: (r) => statuses.includes(r.status) }, { name }); + if (!ok) { + console.warn(`${name}: ${response.status} ${String(response.body || '').slice(0, 200)}`); + } + return ok; +}; + +/** A multipart POST (a file upload): the same cookie and CSRF pair, no JSON content type. */ +export const postMultipart = (session, path, body, name, extra = {}) => { + const { 'Content-Type': _json, ...rest } = headers(session, true); + return http.post(`${API}/${path}`, body, { + headers: rest, + tags: { name }, + jar: http.cookieJar(), + timeout: extra.timeout || '120s', + ...(extra.expected ? { responseCallback: http.expectedStatuses(...extra.expected) } : {}), + }); +}; diff --git a/src/loadtest/k6/lib/session.js b/src/loadtest/k6/lib/session.js new file mode 100644 index 000000000..539309cdf --- /dev/null +++ b/src/loadtest/k6/lib/session.js @@ -0,0 +1,41 @@ +/** + * The users of a run: the manifest `create_load_test_sessions` writes + * (src/backend/loadtest), one session per virtual user. + * + * k6 reads the file once, in the init context, and every VU keeps the session + * of its rank: with more VUs than sessions a user is logged in several times, + * and the API throttles per user (80/min on documents) — ask for at least as + * many sessions as VUs. + */ +import { SharedArray } from 'k6/data'; + +const manifest = JSON.parse(open(__ENV.MANIFEST || '/manifest.json')); +if (!manifest.cookie_name || !manifest.sessions || manifest.sessions.length === 0) { + throw new Error('The manifest holds no session'); +} +if (manifest.expires_at && Date.parse(manifest.expires_at) < Date.now()) { + throw new Error(`The sessions of the manifest expired at ${manifest.expires_at}`); +} + +// shared between the VUs rather than copied into each of them +const sessions = new SharedArray('sessions', () => manifest.sessions); +const publicDocuments = new SharedArray('public documents', () => manifest.public_documents || []); + +export const COOKIE_NAME = manifest.cookie_name; +export const sessionCount = () => sessions.length; + +/** The session of this VU. */ +export const mySession = () => sessions[(__VU - 1) % sessions.length]; + +/** A document of the VU's user, `editable` or any it may read; a public one when it has none. */ +export const myDocument = (session, editable = false) => { + const own = editable + ? session.editable_documents + : session.readonly_documents.concat(session.editable_documents); + const pool = own.length > 0 ? own : publicDocuments; + if (pool.length === 0) return null; + return pool[__ITER % pool.length]; +}; + +export const publicDocument = () => + publicDocuments.length > 0 ? publicDocuments[__ITER % publicDocuments.length] : null; diff --git a/src/loadtest/k6/scenarios/heavy.js b/src/loadtest/k6/scenarios/heavy.js new file mode 100644 index 000000000..36dba56cb --- /dev/null +++ b/src/loadtest/k6/scenarios/heavy.js @@ -0,0 +1,90 @@ +/** + * The endpoints that call the collaboration server or walk a subtree inside one + * request (scenario 9 of the plan): `duplicate` with descendants — two yhub + * round-trips per node, inside a transaction — `formatted-content`, document + * creation from a file, and the delete/restore cascades that land on the single + * Celery queue. + * + * Every iteration works on documents it creates itself and deletes at the end: + * a duplicate of one of the user's own documents, then the duplicate is + * deleted, restored and deleted again. The user's documents are only read. The + * deleted copies stay in the trash for the retention period, so the database + * grows by one subtree per iteration for as long as that lasts. + * + * k6 run --env MANIFEST=manifest.json --env BASE_URL=https://docs.example.com \ + * --env VUS=5 --env DURATION=5m scenarios/heavy.js + */ +import { group, sleep } from 'k6'; +import http from 'k6/http'; + +import { del, expect, get, post, postMultipart } from '../lib/api.js'; +import { myDocument, mySession } from '../lib/session.js'; + +const VUS = Number(__ENV.VUS || 5); +const DURATION = __ENV.DURATION || '5m'; +const MARKDOWN = '# k6\n\nA document created by the load test, and deleted by it.\n'; + +export const options = { + scenarios: { + heavy: { + executor: 'constant-vus', + vus: VUS, + duration: DURATION, + }, + }, + thresholds: { + http_req_failed: ['rate<0.05'], + 'http_req_duration{name:documents/{id}/duplicate/}': ['p(95)<10000'], + 'http_req_duration{name:documents/{id}/formatted-content/}': ['p(95)<5000'], + 'http_req_duration{name:documents/ (create from file)}': ['p(95)<10000'], + checks: ['rate>0.95'], + }, + summaryTrendStats: ['avg', 'min', 'med', 'p(90)', 'p(95)', 'p(99)', 'max'], +}; + +export default function () { + const session = mySession(); + const doc = myDocument(session, true); + if (!doc) return; + + group('duplicate with descendants', () => { + const res = post( + session, + `documents/${doc}/duplicate/`, + { with_accesses: false, with_descendants: true }, + 'documents/{id}/duplicate/', + ); + if (!expect(res, 'documents/{id}/duplicate/', 201)) return; + const copy = res.json('id'); + + group('formatted content', () => { + expect( + get(session, `documents/${copy}/formatted-content/?content_format=markdown`, 'documents/{id}/formatted-content/'), + 'documents/{id}/formatted-content/', + 200, + ); + }); + + group('delete, restore, delete', () => { + // each delete and each restore walks the subtree from a Celery task + expect(del(session, `documents/${copy}/`, 'documents/{id}/ (delete)'), 'documents/{id}/ (delete)', 204); + expect(post(session, `documents/${copy}/restore/`, null, 'documents/{id}/restore/'), 'documents/{id}/restore/', 200); + expect(del(session, `documents/${copy}/`, 'documents/{id}/ (delete)'), 'documents/{id}/ (delete)', 204); + }); + }); + + group('create from file', () => { + // the file is converted, then handed to yhub, before the response + const res = postMultipart( + session, + 'documents/', + { file: http.file(MARKDOWN, 'k6.md', 'text/markdown'), title: 'k6 import' }, + 'documents/ (create from file)', + ); + if (expect(res, 'documents/ (create from file)', 201)) { + expect(del(session, `documents/${res.json('id')}/`, 'documents/{id}/ (delete)'), 'documents/{id}/ (delete)', 204); + } + }); + + sleep(1); +} diff --git a/src/loadtest/k6/scenarios/page-open.js b/src/loadtest/k6/scenarios/page-open.js new file mode 100644 index 000000000..ccd8c1f9b --- /dev/null +++ b/src/loadtest/k6/scenarios/page-open.js @@ -0,0 +1,87 @@ +/** + * What the browser does when a user opens a document, in the order the + * frontend does it — `config`, `users/me`, the document, its tree, the list of + * the user's documents — plus the `media-auth` subrequest nginx makes when the + * page loads an attachment. It is the HTTP baseline of the plan + * (`documentation/stress-test-plan.md`, scenario 2): run it against the + * current release and against this branch, on the same data, with the same + * options. + * + * k6 run --env MANIFEST=manifest.json --env BASE_URL=https://docs.example.com \ + * --env RATE=50 --env DURATION=10m scenarios/page-open.js + * + * RATE is in page opens per second, ramped up over RAMP (default 1m), held for + * DURATION, then ramped down over RAMP. Each open is 5 to 6 requests. + */ +import { group, sleep } from 'k6'; + +import { expect, get } from '../lib/api.js'; +import { myDocument, mySession, sessionCount } from '../lib/session.js'; + +const RATE = Number(__ENV.RATE || 10); +const RAMP = __ENV.RAMP || '1m'; +const DURATION = __ENV.DURATION || '5m'; +// one VU serves about one open per second when the API answers in well under +// that; more are preallocated so that a slow API does not starve the rate +const VUS = Number(__ENV.VUS || Math.max(10, RATE * 4)); + +export const options = { + scenarios: { + page_open: { + executor: 'ramping-arrival-rate', + startRate: 1, + timeUnit: '1s', + preAllocatedVUs: VUS, + maxVUs: VUS, + stages: [ + { target: RATE, duration: RAMP }, + { target: RATE, duration: DURATION }, + { target: 0, duration: RAMP }, + ], + }, + }, + thresholds: { + http_req_failed: ['rate<0.01'], + 'http_req_duration{name:documents/{id}/}': ['p(95)<1000'], + 'http_req_duration{name:documents/{id}/tree/}': ['p(95)<1000'], + 'http_req_duration{name:documents/}': ['p(95)<1000'], + 'http_req_duration{name:documents/media-auth/}': ['p(95)<500'], + checks: ['rate>0.99'], + }, + summaryTrendStats: ['avg', 'min', 'med', 'p(90)', 'p(95)', 'p(99)', 'max'], +}; + +export function setup() { + if (VUS > sessionCount()) { + console.warn(`${VUS} VUs for ${sessionCount()} sessions: some users are logged in several times, and the API throttles per user`); + } +} + +export default function () { + const session = mySession(); + const doc = myDocument(session); + if (!doc) return; + + group('open', () => { + expect(get(session, 'config/', 'config/'), 'config/', 200); + expect(get(session, 'users/me/', 'users/me/'), 'users/me/', 200); + expect(get(session, `documents/${doc}/`, 'documents/{id}/'), 'documents/{id}/', 200); + expect(get(session, `documents/${doc}/tree/`, 'documents/{id}/tree/'), 'documents/{id}/tree/', 200); + expect(get(session, 'documents/?page=1&ordering=-updated_at', 'documents/'), 'documents/', 200); + // the auth subrequest nginx makes for `/media/{doc}/attachments/{file}`. The + // file need not exist for the access check — the costly part, and what + // brought production down on 2026-08-18 — to run: a file the bucket does + // not hold answers 403 once that check has passed + const original = `${__ENV.MEDIA_BASE_URL || 'https://docs.example.com'}/media/${doc}/attachments/00000000-0000-4000-8000-000000000000.png`; + expect( + get(session, 'documents/media-auth/', 'documents/media-auth/', { + headers: { 'X-Original-URL': original }, + expected: [200, 403], + }), + 'documents/media-auth/', + 200, 403, + ); + }); + // a real user does not open the next document at once + sleep(Math.random() * 2); +}