mirror of
https://github.com/suitenumerique/docs.git
synced 2026-09-23 18:15:10 +02:00
✨(loadtest) add browser canaries measuring what a user feels under load
After implementing the websocket and api stress tests, this last step is using playwright to simulate a real user using docs. A real chromium will be used for this. The idea is to see what a user feels when Docs is under load.
This commit is contained in:
Manuel Raynaud
@@ -17,6 +17,7 @@ and this project adheres to
|
||||
- ✨(loadtest) add a websocket load generator for the collaboration server
|
||||
- ✨(loadtest) add k6 scenarios for the page-open sequence and the heavy
|
||||
endpoints
|
||||
- ✨(loadtest) add browser canaries measuring what a user feels under load
|
||||
- ✨(backend) measure the calls to yhub and to the converters, the database
|
||||
pool and the celery queue
|
||||
- ✨(backend) add a `LoadTest` configuration and its `loadtest` application,
|
||||
|
||||
@@ -0,0 +1,463 @@
|
||||
# Stress test plan — yhub architecture (next major release)
|
||||
|
||||
Status: plan, written 2026-09-18 from a code read of branch `yhub`. Nothing in
|
||||
here has been run yet. File and line references are valid for that branch at
|
||||
that date; re-check them before relying on them.
|
||||
|
||||
## Goal
|
||||
|
||||
Determine whether the new architecture (Django backend + `yhub-server` built on
|
||||
`@y/hub`, Valkey streams, Postgres persistence) scales, and find where it stops
|
||||
scaling, before the release reaches production.
|
||||
|
||||
Environment: preprod, loaded with an anonymized copy of the production database
|
||||
on which the full migration has been run, so that the volumetry is realistic.
|
||||
|
||||
Deliverable: a capacity table (sockets per yhub pod, connects per second per
|
||||
backend pod, edits per second per yhub worker, migrated documents per hour)
|
||||
that tells how to size production and where autoscaling makes sense.
|
||||
|
||||
## Architecture facts that shape the tests
|
||||
|
||||
Collaboration path:
|
||||
|
||||
- The browser uses plain `y-websocket` `WebsocketProvider`
|
||||
(`src/frontend/apps/impress/src/features/docs/doc-management/stores/useProviderStore.tsx:154-160`)
|
||||
with `disableBc: true`, `maxBackoffTime: 30000`, `resyncInterval: 20000`.
|
||||
Wire protocol is y-protocols sync + awareness.
|
||||
- WS URL: `wss://{host}/collaboration/ws/v1/docs/{docid}`. The doc id must be a
|
||||
lowercase uuid4, the org must be `docs`.
|
||||
- HTTP fallback on the same rooms (`@y/yhub-http-fallback`), about one poll
|
||||
every 10 s, only while the socket is down.
|
||||
- Authentication of the WS upgrade is the Django session cookie, forwarded by
|
||||
yhub to the backend. There is no `collaboration-auth` ingress subrequest
|
||||
anymore.
|
||||
- Every WS connect, reconnect and fallback request triggers 2 to 3 Django
|
||||
calls from yhub: `GET /users/me/` (`src/yhub-server/src/server.ts:190`),
|
||||
`GET /documents/{id}/` (`:252`), and `GET /documents/{id}/accesses/me/` when
|
||||
the user can list versions (`:279`).
|
||||
- `backendFetch` in yhub has no timeout and no retry
|
||||
(`src/yhub-server/src/backend.ts:116-139`). A failure gives a 503, which
|
||||
`y-websocket` retries forever.
|
||||
- yhub pods are stateless, cross-pod fan-out goes through Valkey, there are no
|
||||
sticky sessions (`upstream-hash-by` was removed on purpose).
|
||||
- A worker drains the Valkey stream into Postgres. Throughput is
|
||||
`YHUB_TASK_CONCURRENCY` (default 5) times worker replicas, debounce
|
||||
`YHUB_TASK_DEBOUNCE_MS` 10000.
|
||||
- Each compaction fires `POST /documents/{id}/content-updated/` on Django
|
||||
(5 s timeout, not awaited). Django updates the row and triggers the search
|
||||
indexer, which calls `get_ydoc` back on yhub. This is a feedback loop.
|
||||
|
||||
Backend:
|
||||
|
||||
- The Django to yhub client opens a new connection per call, with no
|
||||
`requests.Session`, no retry, 30 s timeout
|
||||
(`src/backend/core/services/yhub_services.py:196-229`).
|
||||
- Synchronous yhub calls inside HTTP requests: document creation from a file,
|
||||
`duplicate` (2 round-trips per node, inside `transaction.atomic`),
|
||||
`formatted-content` (yhub + converter), `create-for-owner`, first login with
|
||||
the onboarding sandbox document.
|
||||
- Asynchronous (Celery): delete, restore and access changes walk the whole
|
||||
subtree with one HTTP call per node. Single default queue, 1 worker replica.
|
||||
- No `CONN_MAX_AGE`. The psycopg pool exists but is off by default
|
||||
(`DB_PSYCOPG_POOL_ENABLED`).
|
||||
- uvicorn, `WEB_CONCURRENCY=4`, `--limit-max-requests=20000`.
|
||||
- Sessions are cache-backed (`SESSION_ENGINE = cache`).
|
||||
- DRF throttles: 80/min per user on documents, 50/min on document accesses.
|
||||
yhub is exempt through `X-Y-Provider-Key`.
|
||||
|
||||
Migration:
|
||||
|
||||
- One cheap Django migration (`0034_documentmigration`).
|
||||
- `migrate_documents`: thread pool, `--concurrency` default 2, `--rate`,
|
||||
`--limit`, resumable and idempotent.
|
||||
- `SOFT_MIGRATION=true`: unmigrated documents are seeded on first open, inside
|
||||
the WS upgrade handler. Each replica runs at most `MAX_CONCURRENT_SEEDS=20`
|
||||
seeds at once and fails fast beyond that: the caller gets a 503 and relies
|
||||
on the client's retry backoff, it is not queued. A 30 s Redis lock
|
||||
serialises seeds of the same document across replicas
|
||||
(`src/yhub-server/src/migration.ts:63-64`, `:419-429`).
|
||||
- `generate_volumetry` writes no document content. Only `create_demo` and
|
||||
`migrate_documents` put content in yhub.
|
||||
|
||||
Deployment (`src/helm`):
|
||||
|
||||
- No HPA anywhere, `resources: {}` on every impress container, static replicas
|
||||
(backend 3, yhub 3, yhub worker disabled by default, Celery 1).
|
||||
- When this plan was written there was no metrics endpoint in Django or yhub
|
||||
(done since, see sections 0.1 and 0.2).
|
||||
- Valkey is not deployed from this repository on real clusters: another team
|
||||
runs it, through the valkey operator (Sentinel mode), one instance for the
|
||||
backend (`valkey-docs`: cache, sessions, Celery) and one for yhub
|
||||
(`valkey-yhub`). In this repository, dev and feature only run two standalone
|
||||
instances from the `valkey/valkey` chart
|
||||
(`src/helm/env.d/{env}/values.valkey.yaml.gotmpl`), with
|
||||
`maxmemory-policy volatile-lru`. Stream entries without a TTL cannot be
|
||||
evicted under that policy, so the memory limit of `valkey-yhub` matters.
|
||||
- The only nginx auth subrequest left is `media-auth`.
|
||||
|
||||
Frontend:
|
||||
|
||||
- On connection loss, each client waits up to 3 s of jitter, then refetches
|
||||
`GET /documents/{id}/`.
|
||||
- The service worker `SyncManager` replays queued mutations with no backoff
|
||||
and no jitter
|
||||
(`src/frontend/apps/impress/src/features/service-worker/SyncManager.ts:17-45`).
|
||||
|
||||
## 0. Prerequisites
|
||||
|
||||
Without metrics a test only shows that something broke, not where. Real
|
||||
clusters are synced by Argo CD and there is no kubectl access, so dashboards
|
||||
are the only way to observe a run. Everything below ships through git and
|
||||
helm values.
|
||||
|
||||
First question to settle with the ops team: what scrapes metrics in preprod
|
||||
(Prometheus operator with ServiceMonitor and PodMonitor, or annotation-based
|
||||
scraping) and where dashboards live. The chart work depends on the answer.
|
||||
|
||||
### 0.1 yhub-server
|
||||
|
||||
**Implemented** (2026-09-18): metrics and Sentry. See the "Metrics" and "Error
|
||||
reporting" sections of `src/yhub-server/README.md`.
|
||||
|
||||
- `PROMETHEUS_METRICS_ENABLED=true` and `PROMETHEUS_API_KEY` (required) start
|
||||
a listener on its own port (`9464`) in every role, the worker included. It
|
||||
answers one path to a bearer token, like the backend's `/metrics`.
|
||||
- Chart: `yhub.metrics.enabled`. `serviceMonitor.enabled` or
|
||||
`podMonitor.enabled` then scrapes every pod from inside the cluster (one
|
||||
monitor per component, `backend.metrics.enabled` for the backend); the
|
||||
dedicated `ingressMetrics` publishes `/metrics/yhub` and
|
||||
`/metrics/yhub-worker` next to the backend's `/metrics` for a Prometheus
|
||||
outside of it.
|
||||
- `@y/hub` 0.9.0 owns the uWebSockets server and has no hook on socket open,
|
||||
close or message. What is measured is what our code and its public events
|
||||
allow:
|
||||
|
||||
| Metric | What to watch during a run |
|
||||
|---|---|
|
||||
| `yhub_ws_connections`, `yhub_rooms` | sockets and open documents per replica (read off `stream.subs`, an internal of yhub) |
|
||||
| `yhub_auth_duration_seconds{phase,endpoint,result}` | cost of admitting a caller: upgrades, rechecks, fallback polls. `result="unavailable"` is a 503 sent to the client |
|
||||
| `yhub_backend_request_duration_seconds{route,status}`, `yhub_backend_requests_inflight{route}` | the 2 to 3 Django calls per connect. Inflight is what piles up when Django slows down, since `backendFetch` has no timeout |
|
||||
| `yhub_worker_pending_tasks` | compaction backlog for the whole deployment: use `max`, not `sum` |
|
||||
| `yhub_worker_task_duration_seconds`, `yhub_worker_tasks_inflight` | worker saturation against `YHUB_TASK_CONCURRENCY` |
|
||||
| `yhub_doc_updates_total` | rate of `content-updated` notifications sent to Django |
|
||||
| `yhub_seed_duration_seconds`, `yhub_seeds_inflight`, `yhub_seed_rejected_total` | soft migration under load, and the opens refused at 20 concurrent seeds |
|
||||
| `nodejs_eventloop_lag_seconds` | the first signal of a saturated replica: one thread serves all its sockets |
|
||||
|
||||
- Not visible from inside yhub: messages and bytes per socket, close codes,
|
||||
Postgres pool state. Read them from the Valkey exporter, ingress-nginx and
|
||||
`pg_stat_activity` on the `yhub` database.
|
||||
- Logs: `@y/hub` logs every socket connect and close at info level. Check the
|
||||
log pipeline can take it at 10k sockets.
|
||||
- Sentry: `error` and `fatal` log lines are reported. The refusal of a seed at
|
||||
20 concurrent ones is logged at error level (`seed.failed`), so the soft
|
||||
migration scenario will send one Sentry event per refused open. Lower that
|
||||
log line to `warn`, or expect the volume.
|
||||
|
||||
Decision still to take before the first storm scenario: `backendFetch` has no
|
||||
timeout. Either leave it as is for the first run, to measure the real
|
||||
behaviour, or add `AbortSignal.timeout` first.
|
||||
|
||||
### 0.2 Django backend
|
||||
|
||||
1. Request and SQL metrics: **implemented** (2026-09-18), see
|
||||
`documentation/metrics.md`. `django-prometheus` is opt-in
|
||||
(`PROMETHEUS_METRICS_ENABLED`), labels are view name, method and status.
|
||||
- Served on `/metrics` (not under `/api/`), behind a bearer token
|
||||
(`PROMETHEUS_API_KEY`, required) checked by the first middleware. The
|
||||
chart has a dedicated `ingressMetrics`, to be filtered by address.
|
||||
- The uvicorn workers share their numbers through a local directory
|
||||
(multiprocess mode, required with `WEB_CONCURRENCY=4`), defaulted so that
|
||||
nothing has to be configured.
|
||||
- Scraping through the ingress reaches a different pod each time. Every
|
||||
sample has a `hostname` label so the replicas stay apart, but each series
|
||||
is only sampled one scrape out of N. For the load test, with many backend
|
||||
pods, prefer scraping each pod from inside the cluster if preprod allows
|
||||
it; otherwise use a short scrape interval and 5 min rate windows.
|
||||
- Known limit: workers recycled by `--limit-max-requests=20000` leave
|
||||
their files behind, so the directory grows until the pod is replaced.
|
||||
Watch `scrape_duration_seconds` during the soak scenario.
|
||||
2. Custom metrics on the known pressure points: **implemented**
|
||||
(2026-09-21), see `documentation/metrics.md`.
|
||||
- `docs_outgoing_request_duration_seconds{service,operation,method,status}`
|
||||
and `docs_outgoing_requests_inflight`: the calls to yhub
|
||||
(`YHubService.request`) and to the two converters, timeouts told apart
|
||||
from errors;
|
||||
- `docs_db_pool_*`: state and exact counters of the psycopg pool when
|
||||
`DB_PSYCOPG_POOL_ENABLED` is on. `requests_queued_total` and
|
||||
`requests_wait_seconds_total` are the signal that was missing in the
|
||||
2026-08-18 and 2026-09-07 outages;
|
||||
- `docs_celery_queue_length`: backlog of the single default queue the
|
||||
subtree cascades land on (use `max`, not `sum`);
|
||||
- not done: Celery task duration by task name. The workers serve no
|
||||
endpoint, so it needs either a pushgateway or the multiprocess directory
|
||||
shared with a small exporter.
|
||||
3. Query-level detail: silk is already wired. Enable it in preprod with
|
||||
`SILKY_INTERCEPT_PERCENT` at 1 to 2 % and run `purge_silk_profiles`
|
||||
between runs. Set `SENTRY_TRACES_SAMPLE_RATE` (default 0.0,
|
||||
`settings.py:532`) to about 0.01 if Sentry is available in preprod, for
|
||||
traces across Django, Celery and outgoing HTTP calls.
|
||||
4. Request logs: keep `LOGGING_LEVEL_REQUEST_SUMMARY` on, the dockerflow
|
||||
summary carries the path, status and duration of every request and is the
|
||||
fallback when a metric is missing.
|
||||
5. Configuration parity with production: `DB_PSYCOPG_POOL_*`, pooler,
|
||||
`WEB_CONCURRENCY`, cache and session Valkey settings, Celery concurrency,
|
||||
throttle rates. Disable outgoing emails, PostHog, AI and webhooks.
|
||||
|
||||
### 0.3 Synthetic sessions
|
||||
|
||||
**Implemented** (2026-09-21) as a dedicated Django application,
|
||||
`src/backend/loadtest/`, that only the `LoadTest` configuration installs.
|
||||
|
||||
Why it is needed: authentication is session-only
|
||||
(`DEFAULT_AUTHENTICATION_CLASSES` holds only `SessionAuthentication`) and the
|
||||
Keycloak realm has 4 static users, so load clients cannot log in through OIDC at
|
||||
scale. Sessions are cache-backed, `MIDDLEWARE` contains no OIDC token-refresh
|
||||
middleware, and yhub forwards the same cookie to `/users/me/`, so a session
|
||||
written straight to the store covers HTTP, WS and the HTTP fallback.
|
||||
|
||||
How it is kept out of production:
|
||||
|
||||
- `LOAD_TEST_TOOLS_ENABLED` is `False` in `Base`, pinned to `False` again in
|
||||
`Production` (which `Feature`, `Staging`, `PreProduction` and `Demo` inherit),
|
||||
and is not read from the environment: no variable can turn it on.
|
||||
- `class LoadTest(Production)` is the only configuration that sets it to `True`
|
||||
and adds `loadtest` to `INSTALLED_APPS`. Everywhere else the commands do not
|
||||
exist (`Unknown command`).
|
||||
- The application refuses to load when the setting is off, so adding it to
|
||||
another configuration stops the process from starting.
|
||||
- The commands check the setting again, and refuse the `Production`
|
||||
configuration by name.
|
||||
|
||||
Usage, in preprod: deploy the backend with `DJANGO_CONFIGURATION=LoadTest`,
|
||||
then run the commands through the chart's generic backend job (Argo CD
|
||||
`PostSync` hook, `backend.job.command`):
|
||||
|
||||
```bash
|
||||
python manage.py create_load_test_sessions 5000 --heaviest 50 \
|
||||
--storage-name campaign-1.json --ttl-hours 8
|
||||
python manage.py revoke_load_test_sessions --storage-name campaign-1.json
|
||||
```
|
||||
|
||||
- Users: active, not staff, not superuser, with at least one access to a live
|
||||
document. `--heaviest N` takes the N users holding the most accesses, the
|
||||
rest is a random draw. Staff are excluded because a minted session of theirs
|
||||
would open the admin.
|
||||
- Manifest (JSON): `cookie_name`, `expires_at`, `public_documents`, and per
|
||||
session `user_id`, `session_key`, `editable_documents`,
|
||||
`readonly_documents` (most recently updated first, `--documents-per-user`
|
||||
each). This is what the swarm and k6 read to open documents they are allowed
|
||||
to open.
|
||||
- The manifest holds live cookies and is a secret. It goes to a `0600` file
|
||||
(`--output`) or to a private object under `loadtest/` in the default storage
|
||||
(`--storage-name`), never to stdout or logs. The media route only serves
|
||||
`{document id}/attachments/…` keys, so that prefix cannot be fetched through
|
||||
`/media/`. The load generator needs read access to that object.
|
||||
- Revocation reads an index kept next to the sessions, so it works without the
|
||||
manifest and across several runs. Sessions live 12 h by default, 7 days at
|
||||
most.
|
||||
|
||||
Rules for the load clients:
|
||||
|
||||
- One distinct user per virtual client, otherwise the per-user DRF throttles
|
||||
(80/min on documents) distort results. Keep the throttles on, they are part
|
||||
of production behaviour.
|
||||
- Keep the cookie jar. `ForceSessionMiddleware` creates a 12 h session in
|
||||
Valkey for every request that carries none.
|
||||
- Team-based accesses are not listed in the manifest, only direct ones.
|
||||
- `documents/search/` refreshes the OIDC token when
|
||||
`OIDC_STORE_REFRESH_TOKEN` is on: minted sessions hold no token, so leave
|
||||
that endpoint out of the scenarios or keep the setting off in preprod.
|
||||
|
||||
### 0.4 Chart and infrastructure
|
||||
|
||||
- Done in the chart: the metrics of the backend, of yhub and of its worker
|
||||
are served behind a bearer token, and `ingressMetrics` publishes them on a
|
||||
dedicated host (`/metrics`, `/metrics/yhub`, `/metrics/yhub-worker`), to be
|
||||
filtered by address. Through an ingress each scrape reaches one replica at
|
||||
random: if the preprod Prometheus runs inside the cluster, scraping each pod
|
||||
directly (ServiceMonitor and PodMonitor carrying the token) gives better data
|
||||
with many replicas, and would be a chart addition.
|
||||
- Valkey, to ask the team that runs it (it is not managed from this
|
||||
repository): the exporter enabled on `valkey-docs` and `valkey-yhub`, their
|
||||
metrics in the same Prometheus, and the memory limit and `maxmemory-policy` of
|
||||
each instance in preprod and in production.
|
||||
- Postgres: pghero or `pg_stat_statements`, connection counts per database
|
||||
and per application name, replication and failover state.
|
||||
- ingress-nginx metrics: request rate, latency, active connections, per
|
||||
ingress.
|
||||
- Pod CPU, memory, restarts and throttling for every component, load
|
||||
generators included.
|
||||
- Preprod matches production on replicas, resource requests and limits,
|
||||
Postgres pooler, Valkey memory and eviction policy for `valkey-docs` and
|
||||
`valkey-yhub`, yhub worker replicas and `YHUB_TASK_CONCURRENCY`. If preprod
|
||||
differs from production, results are only relative.
|
||||
|
||||
### 0.5 Dashboards
|
||||
|
||||
One board per question, built before the first run:
|
||||
|
||||
- users: `canary_page_open_seconds`, `canary_editor_ready_seconds`,
|
||||
`canary_propagation_seconds`, `canary_failures_total{step}` (from
|
||||
`src/loadtest/canary/`);
|
||||
- yhub: sockets and rooms per pod, event-loop lag, auth duration, backend
|
||||
call duration and inflight, pending tasks, task duration, seeds;
|
||||
- Django: latency and rate per view, yhub client latency, pool waiting,
|
||||
Celery queue length;
|
||||
- stores: Postgres connections and top queries, Valkey memory, commands and
|
||||
evictions.
|
||||
|
||||
## 1. Tooling
|
||||
|
||||
- HTTP: k6 (scenarios, thresholds, Prometheus output). Replay the real
|
||||
page-open sequence: `config`, `users/me`, `documents/{id}`, tree, list,
|
||||
plus `media-auth`.
|
||||
- Collaboration: a Node swarm using the real client stack (`yjs`,
|
||||
`y-websocket`, `ws` with the cookie header, same options as the frontend).
|
||||
One process holds a few thousand sockets. Run it as a Job that goes through
|
||||
the ingress, on nodes that do not host the application.
|
||||
- Latency: writers stamp a timestamp into the doc, readers measure the
|
||||
propagation delay.
|
||||
- Convergence: compare state vectors across clients at the end.
|
||||
- Durability: compare against the admin `ydoc` endpoint.
|
||||
- k6 cannot reasonably speak the Yjs protocol, which is why this is a
|
||||
separate tool.
|
||||
- Playwright canaries: 3 to 5 browsers reusing the `auth.setup.ts` storage
|
||||
state and the helpers in `src/frontend/apps/e2e` (`createDoc`,
|
||||
`writeInEditor`). They measure the user-perceived time to open a document
|
||||
and start typing while the swarm applies load.
|
||||
- Workload model from production, not guesses:
|
||||
- `profile_volumetry` for the data shape;
|
||||
- access logs or PostHog for peak concurrent users, connections per
|
||||
document (long tail: mostly 1, a few at 50+), edit rate, reader/writer
|
||||
ratio.
|
||||
- Run at 1x, 2x and 5x, then ramp until the knee.
|
||||
|
||||
Location of the tooling: `src/loadtest/` (`swarm/` exists, see its README).
|
||||
|
||||
Note from the dev-stack check of the swarm: `create_load_test_sessions` writes
|
||||
the sessions where the `LoadTest` configuration keeps them. In preprod the
|
||||
server runs that same configuration; a server running another one reads its
|
||||
sessions elsewhere (the dev stack's `Development` uses Redis db 2, `LoadTest`
|
||||
db 0) and the cookies are worth nothing to it. yhub then admits the clients
|
||||
anonymously on public documents, which looks like success: check the `userid`
|
||||
in yhub's logs once before a campaign.
|
||||
|
||||
## 2. Scenarios, in order of value
|
||||
|
||||
1. **Migration.** Time the `migrate_documents` backfill at full volumetry:
|
||||
documents per second, growth of the yhub Postgres database, total
|
||||
duration, best `--concurrency`. Then test the `SOFT_MIGRATION` path under
|
||||
load: many users opening unmigrated documents at once. Each replica
|
||||
refuses seeds beyond 20 at once with a 503, so measure how many opens are
|
||||
refused and how long clients take to get in.
|
||||
2. **HTTP baseline against the current release.** Same dataset, same k6
|
||||
script. Catches regressions unrelated to yhub.
|
||||
3. **Connect ramp.** Find the connects-per-second ceiling per backend pod.
|
||||
`documents/{id}/` is not cheap: ancestor annotation, LinkTrace `exists()`
|
||||
and a possible INSERT.
|
||||
4. **Idle steady state.** Ramp to 10k or more open sockets. Measure memory
|
||||
per connection, the 20 s resync floor, Valkey pub/sub traffic.
|
||||
5. **Hot document.** 50 to 200 editors plus readers on one document.
|
||||
Awareness fan-out grows with the square of the number of clients.
|
||||
6. **Wide editing.** Thousands of documents with 1 to 3 typists each.
|
||||
Stresses stream to worker to Postgres, and the `content-updated` to
|
||||
indexer to `get_ydoc` feedback loop. Watch stream length and worker lag.
|
||||
7. **Reconnect storms.** Restart yhub (Argo-driven rollout), call
|
||||
`reset-connections`, trigger a Valkey failover and a Patroni failover (the
|
||||
Valkey one has to be run by the team that operates it).
|
||||
Clients return within 30 s of backoff plus 3 s of jitter, each costing 2
|
||||
to 3 Django calls and a document refetch. With no timeout in
|
||||
`backendFetch`, a slow Django means piled-up upgrades. This is the failure
|
||||
shape of the 2026-08-18, 2026-09-07 and 2026-09-14 outages.
|
||||
8. **WS blocked, forcing the HTTP fallback.** Every poll re-runs the Django
|
||||
permission calls: N clients generate roughly 0.3 N requests per second on
|
||||
the backend. Probably the worst amplification in the system.
|
||||
9. **Heavy synchronous endpoints.** Duplicate with descendants (holds a
|
||||
Postgres connection in a transaction during N yhub round-trips),
|
||||
`formatted-content`, import, and delete, restore and access changes on deep
|
||||
trees (one HTTP call per node on a single Celery queue with 1 worker).
|
||||
10. **Large and long-history documents.** Open the 100 largest documents
|
||||
after migration and measure time to first sync.
|
||||
11. **Soak.** 4 to 8 h at 1x. Watch memory leaks, Valkey memory growth,
|
||||
Postgres bloat in the yhub database, uvicorn worker recycling.
|
||||
12. **Offline replay burst (browser only).** A fleet coming back online
|
||||
replays queued mutations at once. Lower priority.
|
||||
|
||||
## 3. Method
|
||||
|
||||
- Define SLOs first. Example targets: document open p95 under 2 s, edit
|
||||
propagation p95 under 500 ms, errors under 0.1 %, zero divergence, zero
|
||||
lost updates.
|
||||
- One scenario at a time, stepped ramps with plateaus.
|
||||
- At each knee, record which resource saturated (CPU, Postgres connections,
|
||||
Valkey memory, Node event loop), fix it, rerun.
|
||||
- Keep each run's parameters, dashboards snapshot and results together so
|
||||
runs are comparable.
|
||||
|
||||
## 4. Hypotheses to verify
|
||||
|
||||
Suspected weak points from the code read, none confirmed by measurement:
|
||||
|
||||
- 2 to 3 uncached Django calls per WS connect and per fallback poll.
|
||||
- No timeout or retry in yhub `backendFetch`.
|
||||
- Django to yhub client without connection reuse, 30 s timeout, called inside
|
||||
`transaction.atomic` in `duplicate`.
|
||||
- `content-updated` to indexer to `get_ydoc` feedback loop under wide editing.
|
||||
- Single Celery queue with 1 replica for subtree cascades.
|
||||
- Limit of 20 concurrent seeds per replica during soft migration, refused
|
||||
with a 503 beyond it, inside the upgrade handler.
|
||||
- `migrate_documents` default concurrency of 2 making the backfill very long.
|
||||
- Valkey memory: non-evictable stream entries against a small limit.
|
||||
- No database connection persistence or pooling by default.
|
||||
- No HPA and no resource requests, so scheduling and scaling are untested.
|
||||
- `SyncManager` replay without backoff or jitter.
|
||||
- JWKS fetch inside request authentication on cache miss (10 s timeout).
|
||||
|
||||
## 4b. Findings so far
|
||||
|
||||
Found while checking the tooling against the dev stack, before any campaign:
|
||||
|
||||
- **Concurrent `duplicate` calls collide on treebeard paths** (2026-09-22).
|
||||
Three users duplicating their own document at once, in `heavy.js`, get a
|
||||
500 (`IntegrityError: duplicate key value violates unique constraint
|
||||
"impress_document_path_key"`) or a 400 (`Document with this Path already
|
||||
exists`) on about a request out of seven. Root path allocation races between
|
||||
requests. To fix before the campaign, or scenario 9 measures the bug rather
|
||||
than the endpoint.
|
||||
|
||||
## 5. Work to do in this repository
|
||||
|
||||
1. Done: metrics in the backend (`/metrics`) and in `src/yhub-server`, Sentry
|
||||
in yhub, and the dedicated `ingressMetrics` in the chart (sections 0.1 and
|
||||
0.2).
|
||||
2. Done: the `loadtest` application and the `LoadTest` configuration, minting
|
||||
sessions for synthetic clients (section 0.3).
|
||||
3. Load generators reading the manifest of section 0.3, under `src/loadtest/`
|
||||
(not `src/backend/loadtest/`, which is the Django application):
|
||||
- Done (2026-09-22): the Node Yjs swarm, `src/loadtest/swarm/` (see its
|
||||
README). Modes `idle`, `hot`, `wide`, a ramp, a reconnect storm,
|
||||
propagation latency, convergence check, `/metrics`, a JSON report, a
|
||||
Dockerfile. Checked against the dev stack with sessions minted by
|
||||
`create_load_test_sessions`.
|
||||
- Done (2026-09-22): the k6 scripts, `src/loadtest/k6/` (see its README):
|
||||
`page-open.js`, the HTTP baseline (scenario 2) as a rate-based ramp of the
|
||||
frontend's document-open sequence plus `media-auth`; `heavy.js`, the
|
||||
synchronous endpoints (scenario 9): duplicate with descendants,
|
||||
`formatted-content`, delete/restore cascades, creation from a file. Both
|
||||
checked against the dev stack.
|
||||
- Done (2026-09-22): the canaries, `src/loadtest/canary/` (see its README).
|
||||
Real Chromium browsers in writer/reader pairs, through the real frontend
|
||||
and editor: page open, editor ready (provider synced), keystroke to the
|
||||
other screen, failures by step, on `/metrics` and in a JSON report.
|
||||
Checked against the dev stack: open p50 0.5 s, editor ready 1.3 s,
|
||||
propagation 70 ms with nothing else running.
|
||||
- Not done: the durability check against the admin `ydoc` endpoint (it
|
||||
needs an admin JWT the swarm should not hold), and the HTTP fallback
|
||||
scenario (`@y/yhub-http-fallback` is not driven by the swarm).
|
||||
4. To do: preprod helm values — `DJANGO_CONFIGURATION=LoadTest`, metrics
|
||||
enabled on both services, resources and replicas matching production.
|
||||
Outside of this repository: the Valkey exporter and sizing, with the team
|
||||
that runs Valkey.
|
||||
5. Done, from section 0.2: the custom backend metrics (outgoing calls,
|
||||
psycopg pool, Celery queue length). Left: Celery task duration.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Built from the repository root, like every other image here:
|
||||
# docker build -f src/loadtest/canary/Dockerfile -t docs-canary .
|
||||
# The base image carries the browsers Playwright needs, at the same version as
|
||||
# the `playwright` package in package.json: bump both together.
|
||||
FROM mcr.microsoft.com/playwright:v1.62.1-noble AS base
|
||||
WORKDIR /app
|
||||
COPY ./src/loadtest/canary/package.json ./src/loadtest/canary/yarn.lock ./
|
||||
|
||||
FROM base AS builder
|
||||
RUN yarn install --frozen-lockfile
|
||||
COPY ./src/loadtest/canary/tsconfig.json ./src/loadtest/canary/tsconfig.build.json ./
|
||||
COPY ./src/loadtest/canary/src ./src
|
||||
RUN yarn build
|
||||
|
||||
FROM base AS canary
|
||||
RUN yarn install --frozen-lockfile --production
|
||||
COPY --from=builder /app/dist ./dist
|
||||
USER pwuser
|
||||
ENTRYPOINT ["node", "dist/index.js"]
|
||||
@@ -0,0 +1,96 @@
|
||||
# Canary — a few real browsers, measuring what a user would feel
|
||||
|
||||
While the swarm and k6 apply load, the canary keeps a handful of real Chromium
|
||||
browsers opening and editing documents through the real frontend, the real
|
||||
editor and the real collaboration path, and times what a user would notice:
|
||||
|
||||
| Metric | What it is |
|
||||
| ------ | ---------- |
|
||||
| `canary_page_open_seconds` | From the navigation to the document page being visible |
|
||||
| `canary_editor_ready_seconds` | From the navigation to the editor accepting input, i.e. the collaboration provider synced |
|
||||
| `canary_propagation_seconds` | From a keystroke in one browser to the text showing in another browser on the same document |
|
||||
| `canary_iterations_total{result}`, `canary_failures_total{step}` | Iterations, and the step the failed ones died at (`page-open`, `editor-ready`, `focus`, `propagation`) |
|
||||
| `canary_console_errors_total` | Errors the pages logged |
|
||||
|
||||
It is the "users" board of the plan (`documentation/stress-test-plan.md`,
|
||||
section 0.5): the swarm and k6 tell what the servers do, the canary tells what
|
||||
it feels like. It is not a load generator — a few pairs of browsers, nothing
|
||||
more — and it is the collaboration half's counterpart of the e2e tests, built
|
||||
with the Playwright library rather than its test runner so that it can run for
|
||||
hours and export metrics.
|
||||
|
||||
Each pair is a writer and a reader on one document. The reader keeps the
|
||||
document open for the whole run, like a colleague who has the page open. Every
|
||||
`--interval` seconds the writer opens the document in a new tab, waits for the
|
||||
editor, types a token at the end of the document, and the reader waits for it
|
||||
to show; the token is then erased. The document is left as it was found, but
|
||||
the edits are in its history: **only run this against anonymised data**.
|
||||
|
||||
## Who the browsers are
|
||||
|
||||
By default the manifest that the backend's `create_load_test_sessions` command
|
||||
writes (`src/backend/loadtest`, only with the `LoadTest` configuration): each
|
||||
pair takes a session and opens one of that user's editable documents (or
|
||||
`--doc`). The session cookie and a CSRF cookie are set on the browser context,
|
||||
which is what a login would have left. **The manifest holds live sessions and
|
||||
is a secret.**
|
||||
|
||||
`--storage-state file.json --doc <id>` runs with a Playwright storage state
|
||||
instead — a real OIDC session, saved the way `src/frontend/apps/e2e`'s
|
||||
`auth.setup.ts` does.
|
||||
|
||||
A user who has never opened Docs is shown a tour on the first document; the
|
||||
canary skips it, as that user would.
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
cd src/loadtest/canary
|
||||
yarn install --frozen-lockfile && npx playwright install --with-deps chromium && yarn build
|
||||
node dist/index.js --manifest manifest.json --url https://docs.example.com \
|
||||
--pairs 3 --duration 3600 --interval 5
|
||||
```
|
||||
|
||||
| Option | Default | What it does |
|
||||
| ------ | ------- | ------------ |
|
||||
| `--manifest` | — | The manifest file. Required, unless `--storage-state` |
|
||||
| `--url` | — | `https://host` of the frontend. Required |
|
||||
| `--pairs` | `2` | Pairs of browsers (each a writer and a reader) |
|
||||
| `--duration` | `300` | Seconds to run |
|
||||
| `--interval` | `5` | Seconds a pair waits between two iterations |
|
||||
| `--doc` | one of the user's | The document every pair opens |
|
||||
| `--timeout` | `30` | Seconds before an open, a focus or a propagation is given up on |
|
||||
| `--headed` | off | Show the browsers |
|
||||
| `--storage-state` | — | Playwright storage state, in place of the manifest's cookies |
|
||||
| `--screenshots` | — | Directory where a screenshot of each failed iteration goes |
|
||||
| `--metrics-port` | `9466` | Where `/metrics` is served, `0` for nowhere; `--metrics-token` requires a bearer token |
|
||||
| `--report` | `-` | Where the JSON report goes, `-` for stdout |
|
||||
|
||||
Progress goes to stderr, one line per iteration. The exit code is `0` when at
|
||||
least one iteration ran and none failed.
|
||||
|
||||
Keep it on the dev stack's `http://localhost:3000` for a local check: the
|
||||
frontend served by the compose `frontend-development` container, or `yarn dev`
|
||||
in `src/frontend/apps/impress`.
|
||||
|
||||
## Container
|
||||
|
||||
```bash
|
||||
docker build -f src/loadtest/canary/Dockerfile -t docs-canary .
|
||||
docker run --rm -v ./manifest.json:/manifest.json:ro docs-canary \
|
||||
--manifest /manifest.json --url https://docs.example.com --pairs 3
|
||||
```
|
||||
|
||||
Run it where the swarm runs: in the cluster, on nodes that do not host the
|
||||
application, through the ingress. A browser costs a few hundred MB: size the
|
||||
pod for `2 × --pairs` of them.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
yarn typecheck && yarn lint && yarn test
|
||||
```
|
||||
|
||||
The tests drive a real Chromium against a page served in-process that behaves
|
||||
like the editor: a skeleton, then a `contenteditable`, behind a tour to skip,
|
||||
mirroring what is typed to the other tab through the server.
|
||||
@@ -0,0 +1,216 @@
|
||||
// The measurement loop, against a page of its own served in-process: an
|
||||
// "editor" that shows a skeleton, then a contenteditable, behind a tour that
|
||||
// has to be skipped, and mirrors what is typed into a second tab through a
|
||||
// BroadcastChannel — enough to check what the canary times, and that it skips
|
||||
// the tour and reports the steps that fail.
|
||||
import { createServer } from 'node:http';
|
||||
import type { Server } from 'node:http';
|
||||
import type { AddressInfo } from 'node:net';
|
||||
|
||||
import { chromium } from 'playwright';
|
||||
import type { Browser } from 'playwright';
|
||||
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
|
||||
|
||||
import { Pair, documentFor } from '../src/canary.js';
|
||||
import { parseConfig } from '../src/config.js';
|
||||
import type { Config } from '../src/config.js';
|
||||
import { registry } from '../src/metrics.js';
|
||||
import { Samples } from '../src/stats.js';
|
||||
|
||||
// The two tabs are separate browser contexts, so the text travels through the
|
||||
// server: typed text is posted, and the other tab polls for it.
|
||||
const PAGE = (
|
||||
readyAfterMs: number,
|
||||
withTour: boolean,
|
||||
) => `<!doctype html><html><body>
|
||||
<h1>Doc</h1>
|
||||
<div class="--docs--editor-container"><div id="skeleton">loading</div></div>
|
||||
${withTour ? '<div id="tour" style="position:fixed;inset:0;background:rgba(0,0,0,.5)"><button>Skip</button></div>' : ''}
|
||||
<script>
|
||||
const container = document.querySelector('.--docs--editor-container');
|
||||
setTimeout(() => {
|
||||
container.innerHTML = '<div class="ProseMirror" contenteditable="true"><p>existing text</p></div>';
|
||||
const editor = container.firstChild;
|
||||
editor.addEventListener('input', () => fetch('/text' + location.pathname + '?set=' + encodeURIComponent(editor.textContent)));
|
||||
setInterval(async () => {
|
||||
const text = await (await fetch('/text' + location.pathname)).text();
|
||||
if (text && !editor.textContent.includes(text)) editor.textContent = text;
|
||||
}, 50);
|
||||
}, ${readyAfterMs});
|
||||
const tour = document.getElementById('tour');
|
||||
if (tour) tour.querySelector('button').addEventListener('click', () => tour.remove());
|
||||
</script></body></html>`;
|
||||
|
||||
describe('Pair', () => {
|
||||
let browser: Browser;
|
||||
let server: Server;
|
||||
let url: string;
|
||||
let readyAfterMs = 300;
|
||||
let withTour = true;
|
||||
|
||||
beforeAll(async () => {
|
||||
browser = await chromium.launch();
|
||||
const texts = new Map<string, string>();
|
||||
server = createServer((req, res) => {
|
||||
const url = new URL(req.url ?? '/', 'http://x');
|
||||
if (url.pathname.startsWith('/text/')) {
|
||||
const set = url.searchParams.get('set');
|
||||
if (set !== null) texts.set(url.pathname, set);
|
||||
return res
|
||||
.writeHead(200, { 'content-type': 'text/plain' })
|
||||
.end(texts.get(url.pathname) ?? '');
|
||||
}
|
||||
if (!url.pathname.startsWith('/docs/')) return res.writeHead(404).end();
|
||||
res
|
||||
.writeHead(200, { 'content-type': 'text/html' })
|
||||
.end(PAGE(readyAfterMs, withTour));
|
||||
});
|
||||
await new Promise<void>((resolve) =>
|
||||
server.listen(0, '127.0.0.1', resolve),
|
||||
);
|
||||
url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
|
||||
});
|
||||
afterAll(async () => {
|
||||
await browser.close();
|
||||
await new Promise((resolve) => server.close(resolve));
|
||||
});
|
||||
|
||||
const config = (...extra: string[]): Config =>
|
||||
parseConfig([
|
||||
'--url',
|
||||
url,
|
||||
'--storage-state',
|
||||
'',
|
||||
'--manifest',
|
||||
'm',
|
||||
'--doc',
|
||||
'd1',
|
||||
'--timeout',
|
||||
'5',
|
||||
'--metrics-port',
|
||||
'0',
|
||||
...extra,
|
||||
]);
|
||||
const samples = () => ({
|
||||
pageOpen: new Samples(),
|
||||
editorReady: new Samples(),
|
||||
propagation: new Samples(),
|
||||
});
|
||||
|
||||
it('times the open, the editor and the propagation, skipping the tour', async () => {
|
||||
const s = samples();
|
||||
const pair = new Pair({
|
||||
id: 0,
|
||||
config: config(),
|
||||
session: {
|
||||
user_id: 'u',
|
||||
session_key: 'k',
|
||||
editable_documents: [],
|
||||
readonly_documents: [],
|
||||
},
|
||||
cookieName: 'docs_sessionid',
|
||||
doc: 'd1',
|
||||
samples: s,
|
||||
log: () => {},
|
||||
});
|
||||
await pair.open(browser);
|
||||
const result = await pair.iterate();
|
||||
await pair.close();
|
||||
|
||||
expect(result.failedStep).toBeUndefined();
|
||||
expect(result.pageOpen).toBeGreaterThan(0);
|
||||
expect(result.editorReady).toBeGreaterThanOrEqual(0.3);
|
||||
expect(result.propagation).toBeGreaterThan(0);
|
||||
expect(result.propagation).toBeLessThan(3);
|
||||
expect(s.propagation.count).toBe(1);
|
||||
expect(await registry.metrics()).toMatch(
|
||||
/canary_iterations_total\{result="ok"\} 1/,
|
||||
);
|
||||
});
|
||||
|
||||
it('sends the session and the csrf cookies', async () => {
|
||||
let cookie = '';
|
||||
const seen = createServer((req, res) => {
|
||||
cookie = req.headers.cookie ?? '';
|
||||
res.writeHead(200, { 'content-type': 'text/html' }).end(PAGE(10, false));
|
||||
});
|
||||
await new Promise<void>((resolve) => seen.listen(0, '127.0.0.1', resolve));
|
||||
const seenUrl = `http://127.0.0.1:${(seen.address() as AddressInfo).port}`;
|
||||
const pair = new Pair({
|
||||
id: 1,
|
||||
config: parseConfig([
|
||||
'--url',
|
||||
seenUrl,
|
||||
'--manifest',
|
||||
'm',
|
||||
'--doc',
|
||||
'd',
|
||||
'--timeout',
|
||||
'5',
|
||||
'--metrics-port',
|
||||
'0',
|
||||
]),
|
||||
session: {
|
||||
user_id: 'u',
|
||||
session_key: 'the-key',
|
||||
editable_documents: [],
|
||||
readonly_documents: [],
|
||||
},
|
||||
cookieName: 'docs_sessionid',
|
||||
doc: 'd',
|
||||
samples: samples(),
|
||||
log: () => {},
|
||||
});
|
||||
await pair.open(browser);
|
||||
await pair.close();
|
||||
await new Promise((resolve) => seen.close(resolve));
|
||||
expect(cookie).toContain('docs_sessionid=the-key');
|
||||
expect(cookie).toMatch(/csrftoken=[A-Za-z0-9]{32}/);
|
||||
});
|
||||
|
||||
it('reports the step that failed', async () => {
|
||||
readyAfterMs = 60000;
|
||||
withTour = false;
|
||||
const pair = new Pair({
|
||||
id: 2,
|
||||
config: config('--timeout', '2'),
|
||||
session: null,
|
||||
cookieName: '',
|
||||
doc: 'd2',
|
||||
samples: samples(),
|
||||
log: () => {},
|
||||
});
|
||||
await expect(pair.open(browser)).rejects.toThrow();
|
||||
readyAfterMs = 300;
|
||||
await pair.close();
|
||||
});
|
||||
|
||||
it('picks the document of the pair', () => {
|
||||
const manifest = {
|
||||
cookie_name: 'c',
|
||||
public_documents: ['p'],
|
||||
sessions: [],
|
||||
};
|
||||
const session = {
|
||||
user_id: 'u',
|
||||
session_key: 'k',
|
||||
editable_documents: ['e1', 'e2'],
|
||||
readonly_documents: [],
|
||||
};
|
||||
expect(documentFor(config(), session, manifest, 0)).toBe('d1');
|
||||
const noDoc = parseConfig([
|
||||
'--url',
|
||||
url,
|
||||
'--manifest',
|
||||
'm',
|
||||
'--metrics-port',
|
||||
'0',
|
||||
]);
|
||||
expect(documentFor(noDoc, session, manifest, 0)).toBe('e1');
|
||||
expect(documentFor(noDoc, session, manifest, 1)).toBe('e2');
|
||||
expect(
|
||||
documentFor(noDoc, { ...session, editable_documents: [] }, manifest, 0),
|
||||
).toBe('p');
|
||||
expect(documentFor(noDoc, null, manifest, 0)).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,82 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { parseConfig } from '../src/config.js';
|
||||
|
||||
const base = ['--manifest', 'm.json', '--url', 'https://docs.example.com/'];
|
||||
|
||||
describe('parseConfig', () => {
|
||||
it('has defaults', () => {
|
||||
const config = parseConfig(base);
|
||||
expect(config.url).toBe('https://docs.example.com');
|
||||
expect(config.pairs).toBe(2);
|
||||
expect(config.duration).toBe(300);
|
||||
expect(config.interval).toBe(5);
|
||||
expect(config.timeout).toBe(30);
|
||||
expect(config.headless).toBe(true);
|
||||
expect(config.metricsPort).toBe(9466);
|
||||
});
|
||||
|
||||
it('reads every option', () => {
|
||||
const config = parseConfig([
|
||||
...base,
|
||||
'--pairs',
|
||||
'4',
|
||||
'--duration',
|
||||
'60',
|
||||
'--interval',
|
||||
'0',
|
||||
'--doc',
|
||||
'd',
|
||||
'--timeout',
|
||||
'10',
|
||||
'--headed',
|
||||
'--metrics-port',
|
||||
'0',
|
||||
'--metrics-token',
|
||||
't',
|
||||
'--report',
|
||||
'r.json',
|
||||
'--screenshots',
|
||||
'shots',
|
||||
]);
|
||||
expect(config).toMatchObject({
|
||||
pairs: 4,
|
||||
duration: 60,
|
||||
interval: 0,
|
||||
doc: 'd',
|
||||
timeout: 10,
|
||||
headless: false,
|
||||
metricsPort: 0,
|
||||
metricsToken: 't',
|
||||
report: 'r.json',
|
||||
screenshots: 'shots',
|
||||
});
|
||||
});
|
||||
|
||||
it('accepts a storage state in place of the manifest, with a document', () => {
|
||||
const config = parseConfig([
|
||||
'--url',
|
||||
'https://x',
|
||||
'--storage-state',
|
||||
's.json',
|
||||
'--doc',
|
||||
'd',
|
||||
]);
|
||||
expect(config.manifest).toBe('');
|
||||
expect(config.storageState).toBe('s.json');
|
||||
});
|
||||
|
||||
it.each([
|
||||
[['--url', 'https://x'], '--manifest is required'],
|
||||
[
|
||||
['--url', 'https://x', '--storage-state', 's'],
|
||||
'--storage-state needs --doc',
|
||||
],
|
||||
[['--manifest', 'm'], '--url is required'],
|
||||
[['--manifest', 'm', '--url', 'wss://x'], 'must be http:// or https://'],
|
||||
[[...base, '--pairs', '0'], '--pairs must be a number'],
|
||||
[[...base, '--unknown'], 'Unknown option'],
|
||||
])('refuses %j', (argv, message) => {
|
||||
expect(() => parseConfig(argv)).toThrow(message);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,62 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { parseManifest } from '../src/manifest.js';
|
||||
|
||||
const valid = {
|
||||
cookie_name: 'docs_sessionid',
|
||||
public_documents: ['p1'],
|
||||
sessions: [
|
||||
{
|
||||
user_id: 'u1',
|
||||
session_key: 'k1',
|
||||
editable_documents: ['e1'],
|
||||
readonly_documents: ['r1'],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
describe('parseManifest', () => {
|
||||
it('reads what the backend writes', () => {
|
||||
const manifest = parseManifest(JSON.stringify(valid));
|
||||
expect(manifest.cookie_name).toBe('docs_sessionid');
|
||||
expect(manifest.sessions[0]).toEqual(valid.sessions[0]);
|
||||
expect(manifest.public_documents).toEqual(['p1']);
|
||||
});
|
||||
|
||||
it('fills what is missing rather than failing on it', () => {
|
||||
const manifest = parseManifest(
|
||||
JSON.stringify({ cookie_name: 'c', sessions: [{ session_key: 'k' }] }),
|
||||
);
|
||||
expect(manifest.sessions[0]).toEqual({
|
||||
user_id: '0',
|
||||
session_key: 'k',
|
||||
editable_documents: [],
|
||||
readonly_documents: [],
|
||||
});
|
||||
expect(manifest.public_documents).toEqual([]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
['not json', 'not JSON'],
|
||||
['null', 'not an object'],
|
||||
['[]', 'no cookie_name'],
|
||||
['{"sessions":[{"session_key":"k"}]}', 'no cookie_name'],
|
||||
['{"cookie_name":"c","sessions":[]}', 'holds no session'],
|
||||
['{"cookie_name":"c","sessions":[{"user_id":"u"}]}', 'no session_key'],
|
||||
])('refuses %s', (text, message) => {
|
||||
expect(() => parseManifest(text)).toThrow(message);
|
||||
});
|
||||
|
||||
it('refuses sessions that have expired', () => {
|
||||
expect(() =>
|
||||
parseManifest(
|
||||
JSON.stringify({ ...valid, expires_at: '2000-01-01T00:00:00+00:00' }),
|
||||
),
|
||||
).toThrow('expired');
|
||||
expect(() =>
|
||||
parseManifest(
|
||||
JSON.stringify({ ...valid, expires_at: '2999-01-01T00:00:00+00:00' }),
|
||||
),
|
||||
).not.toThrow();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,23 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { Samples } from '../src/stats.js';
|
||||
|
||||
describe('Samples', () => {
|
||||
it('summarises nothing as null', () => {
|
||||
expect(new Samples().summary()).toBeNull();
|
||||
});
|
||||
|
||||
it('computes the percentiles', () => {
|
||||
const samples = new Samples();
|
||||
for (let i = 1; i <= 100; i++) samples.add(i);
|
||||
expect(samples.summary()).toEqual({
|
||||
count: 100,
|
||||
min: 1,
|
||||
p50: 50,
|
||||
p95: 95,
|
||||
p99: 99,
|
||||
max: 100,
|
||||
mean: 50.5,
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,24 @@
|
||||
import js from '@eslint/js';
|
||||
import prettier from 'eslint-config-prettier/flat';
|
||||
import tseslint from 'typescript-eslint';
|
||||
|
||||
export default tseslint.config(
|
||||
{
|
||||
ignores: ['dist/**', 'node_modules/**'],
|
||||
},
|
||||
js.configs.recommended,
|
||||
...tseslint.configs.recommended,
|
||||
{
|
||||
files: ['**/*.ts'],
|
||||
rules: {
|
||||
'@typescript-eslint/no-explicit-any': 'error',
|
||||
'@typescript-eslint/no-non-null-assertion': 'error',
|
||||
'@typescript-eslint/consistent-type-imports': 'error',
|
||||
'@typescript-eslint/no-unused-vars': [
|
||||
'error',
|
||||
{ varsIgnorePattern: '^_', argsIgnorePattern: '^_' },
|
||||
],
|
||||
},
|
||||
},
|
||||
prettier,
|
||||
);
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "docs-loadtest-canary",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"description": "A few real browsers opening and editing documents in a loop, measuring what a user would feel while the load generators run",
|
||||
"scripts": {
|
||||
"build": "tsc -p tsconfig.build.json",
|
||||
"typecheck": "tsc -p tsconfig.json --noEmit",
|
||||
"start": "node dist/index.js",
|
||||
"lint": "eslint",
|
||||
"test": "vitest run"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=22"
|
||||
},
|
||||
"packageManager": "yarn@1.22.22",
|
||||
"dependencies": {
|
||||
"playwright": "1.62.1",
|
||||
"prom-client": "15.1.3"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@eslint/js": "10.0.1",
|
||||
"@types/node": "24.13.3",
|
||||
"eslint": "10.9.0",
|
||||
"eslint-config-prettier": "10.1.8",
|
||||
"typescript": "6.0.3",
|
||||
"typescript-eslint": "8.69.0",
|
||||
"vitest": "3.2.7"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,250 @@
|
||||
/**
|
||||
* One pair of browsers on one document: a writer and a reader. In a loop, the
|
||||
* writer opens the document, waits for the editor, types a unique token; the
|
||||
* reader, already on the document, waits for that token to show. What is
|
||||
* timed is what a user would feel — from the navigation to the editor being
|
||||
* usable, and from a keystroke to its arrival on another screen — through the
|
||||
* real frontend, the real editor and the real collaboration path.
|
||||
*/
|
||||
import { mkdirSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
|
||||
import type { Browser, BrowserContext, Page } from 'playwright';
|
||||
|
||||
import type { Config } from './config.js';
|
||||
import type { Manifest, Session } from './manifest.js';
|
||||
import * as metrics from './metrics.js';
|
||||
import type { Samples } from './stats.js';
|
||||
|
||||
const TITLE = '[aria-label="Document title"], input[name="title"], h1';
|
||||
const EDITOR = '.--docs--editor-container .ProseMirror';
|
||||
// where the e2e helpers click to type at the end of the document
|
||||
const TRAILING_BLOCK = '.bn-trailing-block.ProseMirror-widget';
|
||||
// 32 alphanumeric characters, what Django accepts as a CSRF secret
|
||||
const CSRF_TOKEN = 'canaryloadtestcanaryloadtest0000';
|
||||
|
||||
// A user who never opened Docs is shown a tour on the first document. It sits
|
||||
// over the editor: skipped when it shows, as that user would.
|
||||
const dismissOnboarding = async (page: Page): Promise<void> => {
|
||||
const skip = page.getByRole('button', { name: 'Skip' });
|
||||
if (await skip.isVisible().catch(() => false)) {
|
||||
await skip.click().catch(() => {});
|
||||
await skip.waitFor({ state: 'hidden', timeout: 5000 }).catch(() => {});
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Put the caret at the end of the document, the way the e2e helpers do. A click
|
||||
* the tour intercepts is retried once the tour is skipped, until `timeoutMs`.
|
||||
*/
|
||||
const focusEnd = async (page: Page, timeoutMs: number): Promise<void> => {
|
||||
const editor = page.locator(EDITOR).first();
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
for (;;) {
|
||||
await dismissOnboarding(page);
|
||||
try {
|
||||
const trailing = editor.locator(TRAILING_BLOCK);
|
||||
if ((await trailing.count()) > 0) await trailing.click({ timeout: 3000 });
|
||||
else await editor.click({ timeout: 3000 });
|
||||
break;
|
||||
} catch (err) {
|
||||
if (Date.now() > deadline) throw err;
|
||||
}
|
||||
}
|
||||
await page.keyboard.press('Control+End');
|
||||
};
|
||||
|
||||
export interface PairOptions {
|
||||
id: number;
|
||||
config: Config;
|
||||
session: Session | null;
|
||||
cookieName: string;
|
||||
doc: string;
|
||||
samples: { pageOpen: Samples; editorReady: Samples; propagation: Samples };
|
||||
log: (line: string) => void;
|
||||
}
|
||||
|
||||
export interface IterationResult {
|
||||
pageOpen?: number;
|
||||
editorReady?: number;
|
||||
propagation?: number;
|
||||
failedStep?: string;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
|
||||
|
||||
/** The document every browser of the run opens for a pair: `--doc`, else the user's. */
|
||||
export const documentFor = (
|
||||
config: Config,
|
||||
session: Session | null,
|
||||
manifest: Manifest | null,
|
||||
pairId: number,
|
||||
): string | null => {
|
||||
if (config.doc) return config.doc;
|
||||
if (!session) return null;
|
||||
const own = session.editable_documents;
|
||||
const pool = own.length > 0 ? own : (manifest?.public_documents ?? []);
|
||||
return pool.length > 0 ? pool[pairId % pool.length] : null;
|
||||
};
|
||||
|
||||
export class Pair {
|
||||
readonly options: PairOptions;
|
||||
private writer: BrowserContext | null = null;
|
||||
private reader: BrowserContext | null = null;
|
||||
private readerPage: Page | null = null;
|
||||
private seq = 0;
|
||||
|
||||
constructor(options: PairOptions) {
|
||||
this.options = options;
|
||||
}
|
||||
|
||||
private async context(
|
||||
browser: Browser,
|
||||
role: string,
|
||||
): Promise<BrowserContext> {
|
||||
const { config, session, cookieName } = this.options;
|
||||
const context = await browser.newContext({
|
||||
...(config.storageState ? { storageState: config.storageState } : {}),
|
||||
locale: 'en-US',
|
||||
viewport: { width: 1280, height: 800 },
|
||||
ignoreHTTPSErrors: true,
|
||||
});
|
||||
if (session) {
|
||||
const url = new URL(config.url);
|
||||
// what a login would have left: the session, on the host of the
|
||||
// application, which is the host of its API as well (a cookie ignores
|
||||
// the port, so the dev stack's :3000 and :8071 share it)
|
||||
const cookie = {
|
||||
domain: url.hostname,
|
||||
path: '/',
|
||||
secure: url.protocol === 'https:',
|
||||
sameSite: 'Lax' as const,
|
||||
};
|
||||
await context.addCookies([
|
||||
{
|
||||
...cookie,
|
||||
name: cookieName,
|
||||
value: session.session_key,
|
||||
httpOnly: true,
|
||||
},
|
||||
// Django's double-submit CSRF token, which the frontend reads to sign
|
||||
// its POSTs (the title, "onboarding done", ...). A login leaves one
|
||||
// behind; any value does as long as the cookie and the header agree.
|
||||
{ ...cookie, name: 'csrftoken', value: CSRF_TOKEN, httpOnly: false },
|
||||
]);
|
||||
}
|
||||
context.on('page', (page) => {
|
||||
page.on('console', (message) => {
|
||||
if (message.type() === 'error') metrics.consoleErrors.inc();
|
||||
});
|
||||
page.on('pageerror', () => metrics.consoleErrors.inc());
|
||||
});
|
||||
context.setDefaultTimeout(config.timeout * 1000);
|
||||
void role;
|
||||
return context;
|
||||
}
|
||||
|
||||
async open(browser: Browser): Promise<void> {
|
||||
this.writer = await this.context(browser, 'writer');
|
||||
this.reader = await this.context(browser, 'reader');
|
||||
// the reader stays on the document for the whole run, like a colleague
|
||||
// who has the page open: it only measures propagation
|
||||
this.readerPage = await this.reader.newPage();
|
||||
await this.readerPage.goto(
|
||||
`${this.options.config.url}/docs/${this.options.doc}/`,
|
||||
);
|
||||
await this.readerPage.locator(EDITOR).first().waitFor({ state: 'visible' });
|
||||
await dismissOnboarding(this.readerPage);
|
||||
}
|
||||
|
||||
/** One iteration: open, wait for the editor, type, watch the reader. */
|
||||
async iterate(): Promise<IterationResult> {
|
||||
const { config, doc } = this.options;
|
||||
if (!this.writer || !this.readerPage) throw new Error('pair not opened');
|
||||
const result: IterationResult = {};
|
||||
const page = await this.writer.newPage();
|
||||
try {
|
||||
const started = Date.now();
|
||||
await page.goto(`${config.url}/docs/${doc}/`, { waitUntil: 'commit' });
|
||||
await page
|
||||
.locator(TITLE)
|
||||
.first()
|
||||
.waitFor({ state: 'visible' })
|
||||
.catch((err) => {
|
||||
result.failedStep = 'page-open';
|
||||
throw err;
|
||||
});
|
||||
result.pageOpen = (Date.now() - started) / 1000;
|
||||
metrics.pageOpen.observe(result.pageOpen);
|
||||
this.options.samples.pageOpen.add(result.pageOpen);
|
||||
|
||||
const editor = page.locator(`${EDITOR}[contenteditable="true"]`).first();
|
||||
await editor.waitFor({ state: 'visible' }).catch((err) => {
|
||||
result.failedStep = 'editor-ready';
|
||||
throw err;
|
||||
});
|
||||
result.editorReady = (Date.now() - started) / 1000;
|
||||
metrics.editorReady.observe(result.editorReady);
|
||||
this.options.samples.editorReady.add(result.editorReady);
|
||||
|
||||
// type at the end of the document, where nobody is
|
||||
this.seq += 1;
|
||||
const token = `canary-${this.options.id}-${process.pid}-${this.seq}`;
|
||||
await dismissOnboarding(page);
|
||||
await focusEnd(page, config.timeout * 1000).catch((err) => {
|
||||
result.failedStep = 'focus';
|
||||
throw err;
|
||||
});
|
||||
const typed = Date.now();
|
||||
await page.keyboard.type(token);
|
||||
await this.readerPage
|
||||
.locator(EDITOR)
|
||||
.first()
|
||||
.getByText(token)
|
||||
.first()
|
||||
.waitFor({ state: 'visible' })
|
||||
.catch((err) => {
|
||||
result.failedStep = 'propagation';
|
||||
throw err;
|
||||
});
|
||||
result.propagation = (Date.now() - typed) / 1000;
|
||||
metrics.propagation.observe(result.propagation);
|
||||
this.options.samples.propagation.add(result.propagation);
|
||||
|
||||
// leave the document as it was found
|
||||
for (let i = 0; i < token.length; i++)
|
||||
await page.keyboard.press('Backspace');
|
||||
await sleep(200);
|
||||
metrics.iterations.inc({ result: 'ok' });
|
||||
} catch (err) {
|
||||
result.failedStep = result.failedStep ?? 'unknown';
|
||||
result.error =
|
||||
err instanceof Error ? err.message.split('\n')[0] : String(err);
|
||||
metrics.iterations.inc({ result: 'failed' });
|
||||
metrics.failures.inc({ step: result.failedStep });
|
||||
if (config.screenshots) {
|
||||
mkdirSync(config.screenshots, { recursive: true });
|
||||
await page
|
||||
.screenshot({
|
||||
path: join(
|
||||
config.screenshots,
|
||||
`pair${this.options.id}-${Date.now()}-${result.failedStep}.png`,
|
||||
),
|
||||
})
|
||||
.catch(() => {});
|
||||
}
|
||||
} finally {
|
||||
await page.close().catch(() => {});
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
await this.reader?.close().catch(() => {});
|
||||
await this.writer?.close().catch(() => {});
|
||||
this.reader = null;
|
||||
this.writer = null;
|
||||
this.readerPage = null;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
/**
|
||||
* Everything the canary is told, from the command line, validated once here.
|
||||
*
|
||||
* canary --manifest m.json --url https://docs.example.com --pairs 3 --duration 600
|
||||
*/
|
||||
import { parseArgs } from 'node:util';
|
||||
|
||||
export interface Config {
|
||||
manifest: string;
|
||||
/** The frontend, `https://host`. */
|
||||
url: string;
|
||||
/** Pairs of browsers (a writer and a reader on one document). */
|
||||
pairs: number;
|
||||
/** Seconds to run. */
|
||||
duration: number;
|
||||
/** Seconds a pair waits between two iterations. */
|
||||
interval: number;
|
||||
/** The document every pair opens; else each pair opens one of its user's. */
|
||||
doc?: string;
|
||||
/** Seconds before an open or a propagation is given up on. */
|
||||
timeout: number;
|
||||
headless: boolean;
|
||||
/** Playwright's `storageState` file, in place of the manifest's cookies. */
|
||||
storageState?: string;
|
||||
metricsPort: number;
|
||||
metricsToken: string;
|
||||
report: string;
|
||||
/** Where screenshots of failed iterations go, empty for none. */
|
||||
screenshots: string;
|
||||
}
|
||||
|
||||
const number = (
|
||||
name: string,
|
||||
raw: string | undefined,
|
||||
dflt: number,
|
||||
min: number,
|
||||
max = Infinity,
|
||||
): number => {
|
||||
if (raw === undefined || raw === '') return dflt;
|
||||
const value = Number(raw);
|
||||
if (!Number.isFinite(value) || value < min || value > max) {
|
||||
throw new Error(
|
||||
`--${name} must be a number between ${min} and ${max} (got "${raw}")`,
|
||||
);
|
||||
}
|
||||
return value;
|
||||
};
|
||||
|
||||
export const parseConfig = (argv: string[]): Config => {
|
||||
const { values } = parseArgs({
|
||||
args: argv,
|
||||
options: {
|
||||
manifest: { type: 'string' },
|
||||
url: { type: 'string' },
|
||||
pairs: { type: 'string' },
|
||||
duration: { type: 'string' },
|
||||
interval: { type: 'string' },
|
||||
doc: { type: 'string' },
|
||||
timeout: { type: 'string' },
|
||||
headed: { type: 'boolean', default: false },
|
||||
'storage-state': { type: 'string' },
|
||||
'metrics-port': { type: 'string' },
|
||||
'metrics-token': { type: 'string', default: '' },
|
||||
report: { type: 'string', default: '-' },
|
||||
screenshots: { type: 'string', default: '' },
|
||||
},
|
||||
strict: true,
|
||||
});
|
||||
if (!values.manifest && !values['storage-state']) {
|
||||
throw new Error('--manifest is required (or --storage-state with --doc)');
|
||||
}
|
||||
if (!values.url)
|
||||
throw new Error('--url is required (https://host of the frontend)');
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(values.url);
|
||||
} catch {
|
||||
throw new Error(`--url is not a url (got "${values.url}")`);
|
||||
}
|
||||
if (url.protocol !== 'http:' && url.protocol !== 'https:') {
|
||||
throw new Error(`--url must be http:// or https:// (got "${values.url}")`);
|
||||
}
|
||||
if (values['storage-state'] && !values.doc) {
|
||||
throw new Error(
|
||||
'--storage-state needs --doc: without the manifest there is no list of documents',
|
||||
);
|
||||
}
|
||||
return {
|
||||
manifest: values.manifest ?? '',
|
||||
url: values.url.replace(/\/+$/, ''),
|
||||
pairs: number('pairs', values.pairs, 2, 1, 50),
|
||||
duration: number('duration', values.duration, 300, 1),
|
||||
interval: number('interval', values.interval, 5, 0),
|
||||
doc: values.doc,
|
||||
timeout: number('timeout', values.timeout, 30, 1),
|
||||
headless: !values.headed,
|
||||
storageState: values['storage-state'],
|
||||
metricsPort: number('metrics-port', values['metrics-port'], 9466, 0, 65535),
|
||||
metricsToken: values['metrics-token'] ?? '',
|
||||
report: values.report ?? '-',
|
||||
screenshots: values.screenshots ?? '',
|
||||
};
|
||||
};
|
||||
@@ -0,0 +1,147 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* The command line: a few pairs of real browsers, looping on the documents of
|
||||
* the manifest for `--duration` seconds. Progress on stderr, the report on
|
||||
* stdout (or `--report`), the metrics on `--metrics-port`.
|
||||
*/
|
||||
import { writeFileSync } from 'node:fs';
|
||||
|
||||
import { chromium } from 'playwright';
|
||||
|
||||
import { Pair, documentFor } from './canary.js';
|
||||
import type { IterationResult } from './canary.js';
|
||||
import { parseConfig } from './config.js';
|
||||
import { readManifest } from './manifest.js';
|
||||
import type { Manifest } from './manifest.js';
|
||||
import { pairsRunning, startMetricsServer } from './metrics.js';
|
||||
import { Samples } from './stats.js';
|
||||
|
||||
const log = (line: string) =>
|
||||
process.stderr.write(`${new Date().toISOString()} ${line}\n`);
|
||||
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
|
||||
|
||||
const main = async (): Promise<number> => {
|
||||
const config = parseConfig(process.argv.slice(2));
|
||||
const manifest: Manifest | null = config.manifest
|
||||
? readManifest(config.manifest)
|
||||
: null;
|
||||
if (config.metricsPort > 0) {
|
||||
await startMetricsServer(config.metricsPort, config.metricsToken);
|
||||
log(`metrics on :${config.metricsPort}/metrics`);
|
||||
}
|
||||
const samples = {
|
||||
pageOpen: new Samples(),
|
||||
editorReady: new Samples(),
|
||||
propagation: new Samples(),
|
||||
};
|
||||
const browser = await chromium.launch({ headless: config.headless });
|
||||
let stopping = false;
|
||||
const stop = () => {
|
||||
log('stopping');
|
||||
stopping = true;
|
||||
};
|
||||
process.on('SIGINT', stop);
|
||||
process.on('SIGTERM', stop);
|
||||
|
||||
const pairs: Pair[] = [];
|
||||
const warnings: string[] = [];
|
||||
for (let id = 0; id < config.pairs; id++) {
|
||||
const session = manifest
|
||||
? manifest.sessions[id % manifest.sessions.length]
|
||||
: null;
|
||||
const doc = documentFor(config, session, manifest, id);
|
||||
if (!doc) {
|
||||
warnings.push(`pair ${id}: no document to open, skipped`);
|
||||
continue;
|
||||
}
|
||||
pairs.push(
|
||||
new Pair({
|
||||
id,
|
||||
config,
|
||||
session,
|
||||
cookieName: manifest?.cookie_name ?? '',
|
||||
doc,
|
||||
samples,
|
||||
log,
|
||||
}),
|
||||
);
|
||||
}
|
||||
if (manifest && config.pairs > manifest.sessions.length) {
|
||||
warnings.push(
|
||||
`${config.pairs} pairs for ${manifest.sessions.length} sessions: some users are logged in several times`,
|
||||
);
|
||||
}
|
||||
|
||||
const startedAt = new Date();
|
||||
const until = Date.now() + config.duration * 1000;
|
||||
const results: IterationResult[] = [];
|
||||
const runPair = async (pair: Pair) => {
|
||||
try {
|
||||
await pair.open(browser);
|
||||
} catch (err) {
|
||||
const message =
|
||||
err instanceof Error ? err.message.split('\n')[0] : String(err);
|
||||
warnings.push(
|
||||
`pair ${pair.options.id}: could not open the document for the reader: ${message}`,
|
||||
);
|
||||
log(`pair ${pair.options.id}: ${message}`);
|
||||
return;
|
||||
}
|
||||
pairsRunning.inc();
|
||||
try {
|
||||
while (!stopping && Date.now() < until) {
|
||||
const result = await pair.iterate();
|
||||
results.push(result);
|
||||
log(
|
||||
`pair ${pair.options.id}: ` +
|
||||
(result.failedStep
|
||||
? `FAILED at ${result.failedStep}: ${result.error}`
|
||||
: `open ${result.pageOpen?.toFixed(2)}s, ready ${result.editorReady?.toFixed(2)}s, propagation ${result.propagation?.toFixed(2)}s`),
|
||||
);
|
||||
await sleep(config.interval * 1000);
|
||||
}
|
||||
} finally {
|
||||
pairsRunning.dec();
|
||||
await pair.close();
|
||||
}
|
||||
};
|
||||
log(
|
||||
`starting: ${pairs.length} pair(s) on ${new Set(pairs.map((p) => p.options.doc)).size} document(s), for ${config.duration}s`,
|
||||
);
|
||||
await Promise.all(pairs.map(runPair));
|
||||
await browser.close();
|
||||
|
||||
const { metricsToken: _token, ...publicConfig } = config;
|
||||
const failed = results.filter((r) => r.failedStep);
|
||||
const report = {
|
||||
config: publicConfig,
|
||||
startedAt: startedAt.toISOString(),
|
||||
endedAt: new Date().toISOString(),
|
||||
pairs: pairs.length,
|
||||
iterations: results.length,
|
||||
failed: failed.length,
|
||||
failedSteps: failed.reduce<Record<string, number>>((acc, r) => {
|
||||
acc[r.failedStep ?? 'unknown'] =
|
||||
(acc[r.failedStep ?? 'unknown'] ?? 0) + 1;
|
||||
return acc;
|
||||
}, {}),
|
||||
pageOpen: samples.pageOpen.summary(),
|
||||
editorReady: samples.editorReady.summary(),
|
||||
propagation: samples.propagation.summary(),
|
||||
warnings,
|
||||
};
|
||||
const text = JSON.stringify(report, null, 2);
|
||||
if (config.report === '-') process.stdout.write(`${text}\n`);
|
||||
else writeFileSync(config.report, text);
|
||||
for (const warning of warnings) log(`warning: ${warning}`);
|
||||
log(`done: ${results.length} iterations, ${failed.length} failed`);
|
||||
return results.length > 0 && failed.length === 0 ? 0 : 1;
|
||||
};
|
||||
|
||||
main().then(
|
||||
(code) => process.exit(code),
|
||||
(err) => {
|
||||
process.stderr.write(`${err instanceof Error ? err.message : err}\n`);
|
||||
process.exit(2);
|
||||
},
|
||||
);
|
||||
@@ -0,0 +1,80 @@
|
||||
/**
|
||||
* The manifest `create_load_test_sessions` writes (src/backend/loadtest): which
|
||||
* cookie to send, and which documents each of the logged-in users may open.
|
||||
*/
|
||||
import { readFileSync } from 'node:fs';
|
||||
|
||||
export interface Session {
|
||||
user_id: string;
|
||||
session_key: string;
|
||||
editable_documents: string[];
|
||||
readonly_documents: string[];
|
||||
}
|
||||
|
||||
export interface Manifest {
|
||||
cookie_name: string;
|
||||
expires_at?: string;
|
||||
public_documents: string[];
|
||||
sessions: Session[];
|
||||
}
|
||||
|
||||
const isStringArray = (value: unknown): value is string[] =>
|
||||
Array.isArray(value) && value.every((item) => typeof item === 'string');
|
||||
|
||||
export const parseManifest = (text: string): Manifest => {
|
||||
let data: unknown;
|
||||
try {
|
||||
data = JSON.parse(text);
|
||||
} catch (err) {
|
||||
throw new Error(
|
||||
`The manifest is not JSON: ${err instanceof Error ? err.message : err}`,
|
||||
{ cause: err },
|
||||
);
|
||||
}
|
||||
if (typeof data !== 'object' || data === null) {
|
||||
throw new Error('The manifest is not an object');
|
||||
}
|
||||
const manifest = data as Record<string, unknown>;
|
||||
if (typeof manifest.cookie_name !== 'string' || !manifest.cookie_name) {
|
||||
throw new Error('The manifest has no cookie_name');
|
||||
}
|
||||
if (!Array.isArray(manifest.sessions) || manifest.sessions.length === 0) {
|
||||
throw new Error('The manifest holds no session');
|
||||
}
|
||||
const sessions = manifest.sessions.map((session, index): Session => {
|
||||
const s = session as Record<string, unknown>;
|
||||
if (typeof s.session_key !== 'string' || !s.session_key) {
|
||||
throw new Error(`Session #${index} has no session_key`);
|
||||
}
|
||||
return {
|
||||
user_id: typeof s.user_id === 'string' ? s.user_id : String(index),
|
||||
session_key: s.session_key,
|
||||
editable_documents: isStringArray(s.editable_documents)
|
||||
? s.editable_documents
|
||||
: [],
|
||||
readonly_documents: isStringArray(s.readonly_documents)
|
||||
? s.readonly_documents
|
||||
: [],
|
||||
};
|
||||
});
|
||||
if (manifest.expires_at !== undefined) {
|
||||
const expires = Date.parse(String(manifest.expires_at));
|
||||
if (!Number.isNaN(expires) && expires < Date.now()) {
|
||||
throw new Error(
|
||||
`The sessions of the manifest expired at ${manifest.expires_at}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return {
|
||||
cookie_name: manifest.cookie_name,
|
||||
expires_at:
|
||||
typeof manifest.expires_at === 'string' ? manifest.expires_at : undefined,
|
||||
public_documents: isStringArray(manifest.public_documents)
|
||||
? manifest.public_documents
|
||||
: [],
|
||||
sessions,
|
||||
};
|
||||
};
|
||||
|
||||
export const readManifest = (path: string): Manifest =>
|
||||
parseManifest(readFileSync(path, 'utf8'));
|
||||
@@ -0,0 +1,104 @@
|
||||
/**
|
||||
* What the canary measures, served on /metrics for the Prometheus of the
|
||||
* campaign and summarised in the final report: the numbers a user would feel.
|
||||
*/
|
||||
import { timingSafeEqual } from 'node:crypto';
|
||||
import type { IncomingMessage, Server, ServerResponse } from 'node:http';
|
||||
import { createServer } from 'node:http';
|
||||
|
||||
import {
|
||||
Counter,
|
||||
Gauge,
|
||||
Histogram,
|
||||
Registry,
|
||||
collectDefaultMetrics,
|
||||
} from 'prom-client';
|
||||
|
||||
export const registry = new Registry();
|
||||
|
||||
const SECONDS = [
|
||||
0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 3, 5, 7.5, 10, 15, 20, 30, 60,
|
||||
];
|
||||
|
||||
export const pageOpen = new Histogram({
|
||||
name: 'canary_page_open_seconds',
|
||||
help: 'From the navigation to the editor being visible',
|
||||
buckets: SECONDS,
|
||||
registers: [registry],
|
||||
});
|
||||
export const editorReady = new Histogram({
|
||||
name: 'canary_editor_ready_seconds',
|
||||
help: 'From the navigation to the editor accepting input (the collaboration provider synced)',
|
||||
buckets: SECONDS,
|
||||
registers: [registry],
|
||||
});
|
||||
export const propagation = new Histogram({
|
||||
name: 'canary_propagation_seconds',
|
||||
help: 'From a keystroke in one browser to the text showing in another browser on the same document',
|
||||
buckets: SECONDS,
|
||||
registers: [registry],
|
||||
});
|
||||
export const iterations = new Counter({
|
||||
name: 'canary_iterations_total',
|
||||
help: 'Iterations, by result',
|
||||
labelNames: ['result'] as const,
|
||||
registers: [registry],
|
||||
});
|
||||
export const failures = new Counter({
|
||||
name: 'canary_failures_total',
|
||||
help: 'What failed, by step',
|
||||
labelNames: ['step'] as const,
|
||||
registers: [registry],
|
||||
});
|
||||
export const consoleErrors = new Counter({
|
||||
name: 'canary_console_errors_total',
|
||||
help: 'Errors the pages logged to their console',
|
||||
registers: [registry],
|
||||
});
|
||||
export const pairsRunning = new Gauge({
|
||||
name: 'canary_pairs',
|
||||
help: 'Browser pairs running',
|
||||
registers: [registry],
|
||||
});
|
||||
|
||||
const authorized = (req: IncomingMessage, token: string): boolean => {
|
||||
if (!token) return true;
|
||||
const presented = Buffer.from(req.headers.authorization ?? '');
|
||||
const expected = Buffer.from(`Bearer ${token}`);
|
||||
return (
|
||||
presented.length === expected.length && timingSafeEqual(presented, expected)
|
||||
);
|
||||
};
|
||||
|
||||
export const handleMetricsRequest = async (
|
||||
req: IncomingMessage,
|
||||
res: ServerResponse,
|
||||
token: string,
|
||||
): Promise<void> => {
|
||||
if ((req.url ?? '').split('?')[0] !== '/metrics') {
|
||||
res.writeHead(404).end();
|
||||
return;
|
||||
}
|
||||
if (!authorized(req, token)) {
|
||||
res.writeHead(401, { 'www-authenticate': 'Bearer' }).end();
|
||||
return;
|
||||
}
|
||||
res
|
||||
.writeHead(200, { 'content-type': registry.contentType })
|
||||
.end(await registry.metrics());
|
||||
};
|
||||
|
||||
export const startMetricsServer = (
|
||||
port: number,
|
||||
token: string,
|
||||
): Promise<Server> => {
|
||||
collectDefaultMetrics({ register: registry });
|
||||
const server = createServer((req, res) => {
|
||||
void handleMetricsRequest(req, res, token);
|
||||
});
|
||||
server.unref();
|
||||
return new Promise((resolve, reject) => {
|
||||
server.once('error', reject);
|
||||
server.listen(port, () => resolve(server));
|
||||
});
|
||||
};
|
||||
@@ -0,0 +1,44 @@
|
||||
/** Percentiles of what was measured, for the final report. */
|
||||
|
||||
export interface Summary {
|
||||
count: number;
|
||||
min: number;
|
||||
p50: number;
|
||||
p95: number;
|
||||
p99: number;
|
||||
max: number;
|
||||
mean: number;
|
||||
}
|
||||
|
||||
export class Samples {
|
||||
private values: number[] = [];
|
||||
|
||||
add(value: number): void {
|
||||
this.values.push(value);
|
||||
}
|
||||
|
||||
get count(): number {
|
||||
return this.values.length;
|
||||
}
|
||||
|
||||
summary(): Summary | null {
|
||||
if (this.values.length === 0) return null;
|
||||
const sorted = [...this.values].sort((a, b) => a - b);
|
||||
const at = (q: number): number =>
|
||||
sorted[
|
||||
Math.min(
|
||||
sorted.length - 1,
|
||||
Math.max(0, Math.ceil(q * sorted.length) - 1),
|
||||
)
|
||||
];
|
||||
return {
|
||||
count: sorted.length,
|
||||
min: sorted[0],
|
||||
p50: at(0.5),
|
||||
p95: at(0.95),
|
||||
p99: at(0.99),
|
||||
max: sorted[sorted.length - 1],
|
||||
mean: sorted.reduce((sum, value) => sum + value, 0) / sorted.length,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"extends": "./tsconfig.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "./src",
|
||||
"noEmit": false
|
||||
},
|
||||
"include": ["src/**/*.ts"],
|
||||
"exclude": ["node_modules", "dist", "__tests__"]
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "es2022",
|
||||
"lib": ["es2023"],
|
||||
// the tool runs the compiled JS straight through Node's ESM loader, so
|
||||
// resolution has to be Node's — import specifiers keep their `.js`
|
||||
// extension, as they already did in the JS sources.
|
||||
"module": "nodenext",
|
||||
"moduleResolution": "nodenext",
|
||||
"types": ["node"],
|
||||
"strict": true,
|
||||
"noImplicitOverride": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"resolveJsonModule": true,
|
||||
"isolatedModules": true,
|
||||
"verbatimModuleSyntax": true,
|
||||
"noEmit": true,
|
||||
"outDir": "./dist",
|
||||
"sourceMap": true
|
||||
},
|
||||
"include": ["src/**/*.ts", "__tests__/**/*.ts"],
|
||||
"exclude": ["node_modules", "dist"]
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
import { defineConfig } from 'vitest/config';
|
||||
|
||||
// `__tests__/*.spec.ts`: unit tests of the pure parts, and one integration test
|
||||
// running a swarm against an in-process y-websocket server (`_server.ts`).
|
||||
export default defineConfig({
|
||||
test: {
|
||||
include: ['__tests__/**/*.spec.ts'],
|
||||
testTimeout: 30000,
|
||||
clearMocks: true,
|
||||
},
|
||||
});
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user