✨(backend) profile api using django-silk

We have performance issue with the media_auth endpoint. We need to
monitor and profile this endpoint to understand what is happening. We
decided to install and configure django-silk. By default django-silk is
not enabled and not accessible, once enabled migration must be run and
the /silk/ endpoint will be accessible only using a super user.
To have relecant profiles and monitor we must have a database comparable
to a production one because we are facing a scaling issue. Unless a
million documents probably nothing is happening, for this we added
several management commands to help us. A first one to anonymize a
database than can be reused without any risk to leak PII data, can be
called using python manage.py anonymize_database with some mandatory
options. An other command to generate a database profile, this command
should be run on a production database, this profile can then be used by
the generate_volumetry to populate a development database with same
volumetry than describe in the profile.

documentation/profiling.md

@@ -0,0 +1,144 @@
+# Profiling the backend with django-silk
+
+The backend ships an **opt-in** profiler, [django-silk](https://github.com/jazzband/django-silk),
+to answer questions like *"where does `media-auth` spend its time — Postgres, S3,
+or Python?"* under a realistic data volume. It records, per HTTP request:
+
+- the **SQL** it ran, each query with its own duration and the Python stack that
+  issued it (so a slow `readable_per_se` `EXISTS` points straight at the line in
+  `viewsets.py`);
+- a **cProfile** call graph of the request, downloadable as a binary `.prof`;
+- silk's own overhead, so you can subtract it.
+
+It is **disabled by default and absent from every environment that does not opt
+in, production included** — when `SILK_ENABLED` is unset, `silk` is not in
+`INSTALLED_APPS`, its middleware is not installed, and `/silk/` is not routed.
+
+> ⚠️ **Never enable silk against production with real users.** It persists
+> request metadata to the database. Enable it only on an isolated environment —
+> local dev, or a throwaway staging database populated with synthetic volumetry
+> (see [Reproducing production volumetry](#reproducing-production-volumetry)).
+> Request and response **bodies are never stored** (`SILKY_MAX_*_BODY_SIZE = 0`),
+> so document content, titles and emails cannot leak into the silk tables.
+
+## Enabling it
+
+Set `SILK_ENABLED=1` in the backend environment and run silk's migrations once:
+
+```bash
+# in the backend container / environment
+export SILK_ENABLED=1
+python manage.py migrate silk
+```
+
+Then restart the backend so the middleware is picked up. The profiling UI is
+served at **`/silk/`**, gated behind an authenticated **staff** session
+(`SILKY_AUTHENTICATION` / `SILKY_AUTHORISATION`). Create one if needed:
+
+```bash
+python manage.py createsuperuser
+```
+
+All knobs are environment variables (defaults in `documentation/env.md`):
+
+| Variable | Purpose | Default |
+| --- | --- | --- |
+| `SILK_ENABLED` | Master switch | `False` |
+| `SILK_PYTHON_PROFILER` | cProfile each request | `True` |
+| `SILK_PYTHON_PROFILER_BINARY` | Also emit a downloadable `.prof` | `True` |
+| `SILK_INTERCEPT_PERCENT` | Sample only N % of requests | `100` |
+| `SILK_MAX_RECORDED_REQUESTS` | Ring-buffer cap on stored requests | `10000` |
+
+Under load (a thundering-herd reproduction), lower `SILK_INTERCEPT_PERCENT`
+(e.g. `5`) so silk records a representative sample without becoming the
+bottleneck itself, and keep `SILK_MAX_RECORDED_REQUESTS` bounded so a long run
+cannot fill the disk.
+
+## Analysing a request in the UI
+
+1. Reproduce the traffic (open a document, or run a load test — see below).
+2. Open `/silk/`, sort **Requests** by *Time* or *Overall time*.
+3. Click the slow request:
+   - **SQL** tab — every query with its duration and the *join / N+1* summary.
+     Click a query to see its `EXPLAIN`-free timing and the **stack trace** back
+     into our code.
+   - **Profile** tab — the cProfile breakdown (cumulative/total time per
+     function), the fastest way to see the Python↔DB↔S3 split.
+
+## Analysing the binary `.prof` offline
+
+With `SILK_PYTHON_PROFILER_BINARY=True`, each request's cProfile is also written
+as a binary `.prof`, **through the Django storage framework** rather than the
+local disk. It is stored via the `SILKY_STORAGE` entry of the `STORAGES` setting,
+which points at the S3 backend under the `silk/` prefix — so profiling works on
+read-only / ephemeral Kubernetes pods where there is no writable filesystem.
+
+Download the `.prof` from the request's **Profile** tab in the UI (silk serves it
+back out of object storage), then open it with any standard tool:
+
+```bash
+# interactive flame-ish call graph in the browser
+pip install snakeviz && snakeviz <request>.prof
+
+# or the stdlib, sorted by cumulative time
+python -m pstats <request>.prof <<< "sort cumulative
+stats 30"
+```
+
+To point the binaries at a different bucket/backend, override the `SILKY_STORAGE`
+entry in `STORAGES` (any Django storage backend works).
+
+## Cleaning up stored profiles
+
+Silk's ring-buffer (`SILK_MAX_RECORDED_REQUESTS`) prunes the request rows in the
+database, but it does **not** delete the corresponding `.prof` objects, so they
+accumulate in storage. Reclaim the space with:
+
+```bash
+python manage.py purge_silk_profiles --dry-run    # preview orphans
+python manage.py purge_silk_profiles              # delete orphaned profiles
+python manage.py purge_silk_profiles --all        # wipe every profile
+python manage.py purge_silk_profiles --older-than 7   # only >7 days old
+```
+
+By default it deletes only **orphans** (profiles no longer referenced by any
+`silk.Request` row), which is always safe. `--all` is required — and is the only
+mode available — when silk is disabled, since there are then no rows to compare
+against. Everything goes through the `SILKY_STORAGE` backend, so it works on
+filesystem-less pods.
+
+## Profiling a code block directly (no HTTP)
+
+To profile a specific path without going through nginx/DRF — handy for the
+`media-auth` authorization logic, which normally runs as an nginx subrequest —
+wrap it with silk's context manager or decorator; it appears under **Profiling**
+in the UI:
+
+```python
+from silk.profiling.profiler import silk_profile
+
+with silk_profile(name="media_auth authorization"):
+    ...  # the code under test
+```
+
+## Reproducing production volumetry
+
+Silk shows *where* time goes; it only tells the truth if the database has a
+production-shaped volume. The incident class we chase (`media-auth` under a large
+readable set) does not reproduce on an empty staging DB. Rebuild the shape from a
+value-free profile — **without copying any production data** — then profile
+against it:
+
+```bash
+# 1. On a read replica: capture the shape (counts + distributions only, no PII)
+python manage.py profile_volumetry --output volumetry/prod.json
+
+# 2. On the throwaway profiling DB: synthesize a same-shaped dataset
+python manage.py generate_volumetry --profile volumetry/prod.json --scale 1.0
+
+# 3. Enable silk, drive traffic (or the media-auth load tool), read /silk/
+```
+
+Alternatively, anonymize a real production dump in an isolated database with
+`anonymize_database` and profile against that. See those commands' `--help` for
+details.