After removing the ForceSessionMiddleware we can detect without calling
the /users/me endpoint if a user is anonymous or not. We now check the
presence of the cookie session to avoid the http call and directly
return an anonymous user.
This middleware was introduced in 651f2d1d to create a session for every
user — connected to the collaboration websocket or not — so the backend
could arbitrate who may save a document (websocket payload flag and save
lock). The whole safeguard was removed in 68f5f5cdf when the content and
its permissions moved to yhub, leaving the middleware orphaned: nothing
reads an anonymous session anymore (the OIDC login flow saves its own
session, CSRF is cookie-based, throttling is not session-keyed).
It only kept a cost: a Redis write plus a Set-Cookie on every anonymous
response, which had already required two bugfix commits to exempt the
probes. A regression test pins the probe invariant going forward.
Where websockets are blocked, a tab whose access was revoked or whose
session expired took a 403 on every PATCH, forever, since only the
websocket's 4401 close made the tab fetch the document again. Treat a
401, 403 or 404 on the fallback the same way: stop both transports and
fetch the document again.
An editor on the http fallback publishes its presence every 10 seconds,
a request on its own, and the library then never polls less often than
every 15 seconds. On a deployment where many browsers cannot open the
websocket, this is most of the traffic and each request is authorized
against the backend.
Add COLLABORATION_HTTP_FALLBACK_AWARENESS_ENABLED (default True, so
nothing changes unless it is set), served by the config endpoint. Off,
an editor sends a request only when it has changes and
COLLABORATION_HTTP_FALLBACK_SYNC_INTERVAL applies in full. The cost is
no remote cursors on the fallback.
Signed-off-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
Each poll of a browser on the http fallback is authorized against the
backend, and the 10 second interval was hardcoded. Add
COLLABORATION_HTTP_FALLBACK_SYNC_INTERVAL (milliseconds, default 10000,
so nothing changes unless it is set), served by the config endpoint, so
operators can lower that load without a new release.
A doc never opened on this device has an empty local copy, and the
socket opening is not the content arriving either. Rendering on either
showed the empty-doc placeholder for a moment, which reads as a doc whose
content was deleted.
Signed-off-by: Dor Zairi <dorzairidev@gmail.com>
With zh-tw active, selecting English did nothing: its bundle is empty, so
i18next reports `resolvedLanguage` as en while `language` is zh-tw, and
the hook skipped the change. Compare with `language` instead, and add
unit tests on the language pickers.
A throttling bypass exists on the DocumentThrottle class allowing YHub
server to fetch django and not be blocked by the throttling. YHub also
fetch an endpont on the DocumentAccess viewset and this bypass is not
existing. We must add it. We factorized the bypass in a mixin and this
mixin is now used by both the DocumentThrottle and
DocumentAccessThrottle classes.
Added
- 👷(ci) lint, typecheck, build and test the collaboration server (yhub) #2757
- 👷(ci) check the load-test tooling: swarm, canary, k6 and dashboards #2757
- 👷(ci) cancel the superseded runs of a pull request #2757
- ✨(backend) add mention endpoint with cooldown-limited email
notification #2447
- 🚩(setting) add feature flag on Duplicate with Children #2721
- 💄(frontend) redesign 404 error standalone page #2696
- 💄(frontend) redesign 403 access denied page #2720
- ✨(frontend) duplicate with subdocuments #2584
- ✨(frontend) add markdown download option #2608
- ✨(frontend) turn pasted doc links into interlinks #2713
- ✨(helm) allow disallowing search engine indexing per instance #2694
- ✨(backend) expose the attachment max size in the config endpoint #2577
- ✨(frontend) warn before uploading an attachment over the size limit #2577
- ✨(loadtest) add a websocket load generator for the collaboration
server #2556
- ✨(loadtest) add k6 scenarios for the page-open sequence and the heavy
endpoints #2556
- ✨(loadtest) add browser canaries measuring what a user feels under
load #2556
- ✨(loadtest) add the grafana dashboards of the load-test campaign, valkey
included #2556
- 📝(documentation) add the load-testing guide #2556
- 🔧(helm) run grafana with those dashboards in the dev cluster, in place of
the prometheus console #2556
- ✨(backend) measure the calls to yhub and to the converters, the database
pool and the celery queue #2556
- ✨(backend) add a `LoadTest` configuration and its `loadtest` application,
minting sessions for load tests #2556
- ✨(collaboration) add opt-in prometheus metrics to yhub, server and worker,
protected by a bearer token #2556
- ✨(collaboration) report yhub errors to sentry, configured through
`SENTRY_*` #2556
- ✨(backend) add opt-in prometheus metrics on `/metrics`, protected by a
bearer token #2556
- ✨(helm) add a dedicated ingress for the prometheus metrics of the backend
and of yhub #2556
- ✨(helm) add a ServiceMonitor and a PodMonitor per component whose metrics
are enabled, and `backend.metrics.enabled` #2556
- 🔧(helm) scrape the metrics of the dev cluster with a trimmed
kube-prometheus-stack and the ServiceMonitors of the chart #2556
- 🔧(helm) serve the example console of django-prometheus on the dev
Prometheus #2556
- ✨(backend) add a service generating cached RS256 JWT tokens #2556
- ✨(backend) publish the JWT public key on a JWKS endpoint #2556
- 🔧(dev) generate the JWT signing key when bootstrapping the dev stack #2556
- ✨(collaboration) add an admin reset-connections endpoint on yhub #2556
- ✨(collaboration) add a create-ydoc endpoint on yhub #2556
- ✨(collaboration) soft-migrate legacy S3 documents into yhub #2556
- ✨(collaboration) replay legacy s3 version history into yhub #2556
- ✨(backend) add a service to call the yhub REST API #2556
- ✨(backend) call YHubService to seed initial document content #2556
- ✨(collaboration) add a get-ydoc endpoint on yhub #2556
- ✨(backend) duplicate a document through the collaboration server #2556
- ✨(backend) serve `documents/{id}/formatted-content/` from yhub #2556
- ✨(collaboration) notify the backend when the worker persists new
content #2556
- ✨(collaboration) let a user read the document's editing history #2556
- ✨(frontend) fall back to http polling when the websocket cannot be
opened. #2556
- 🔧(collaboration) make the version-history granularity configurable through
`COLLABORATION_VERSION_GRANULARITY_MS` #2556
- ✨(frontend) keep a local copy of documents, so they open and stay editable
offline #2556
Changed
- 🛂(backend) let users allowed to comment list each other's accesses #2447
- ♻️(collaboration) migrate the collaboration server from hocuspocus to
yhub #2556
- 💥(y-provider) y-provider becomes converter-only #2556
- 💥(backend) move the resource server JWKS from `/api/{version}/jwks` to
`/external_api/{version}/jwks` #2556
- 🔧(collaboration) adapt docker stack for development purpose #2556
- 🔧(helm) run a valkey for the backend and one for yhub in dev and
feature #2556
- 💄(frontend) redesign 401 error standalone page #2716
Fixed
- 🐛(documentation) fix minio port and keycloak realm in the k8s guide #2751
- 🐛(collaboration) bound the calls yhub makes to the backend with a timeout,
`YHUB_BACKEND_REQUEST_TIMEOUT_MS`, 5s by default #2753
- 🐛(backend) compensate document duplication failures #2755
- 🐛(frontend) open search results in a new tab with ctrl/cmd+click #2719
- 🐛(docker) pull minio images from pgsty #2739
- 🐛(backend) retry the duplicate of a document on a tree path collision #2726
- 🐛(frontend) keep caption and alignment when replacing an image #2730
- 🐛(backend) reset the collaboration connections on every access change #2725
- ⚡️(backend) coalesce and pace the resets of collaboration connections #2725
- 🐛(frontend) clear callout background on Backspace #2052
- 🐛(export) keep image aspect ratio in PDF columns #2670
- 🐛(frontend) fix redirect after deleting a document #2706
- 🐛(frontend) keep documents draggable with a mouse when zoomed #2727
- 📝(docs) fix markdown typo in installation README #2766
- 🐛(i18n) export locale correctly #2750
- 🐛(frontend) keep login and logout out of the service worker cache #2771
- 🐛(frontend) reduce PostHog volume from web vitals and opt_in spam #2701
- 🐛(frontend) stop reconnecting to the websocket based on the status
code #2556
- 🐛(frontend) stop the service worker from caching the collaboration server's
rest api #2556
Removed
- 🔥(backend) remove the unused `CollaborationService` #2556
- 💥(backend) remove the `documents/{id}/can-edit/` endpoint #2556
- 💥(backend) remove the `documents/{id}/content/` endpoint #2556
- 🔥(backend) remove the document version endpoints. #2556
- 🔥(backend) remove `Document.content` #2556
Security
- 🔒️(collaboration) stop read-only users from sharing their cursor #2556
the task `sync_service_deletions_in_cascade` is no more called when a
document is soft deleted. We rename it in
sync_service_restorations_in_cascade and still call it when a document
is restored. No ydoc should be tombstone in yhub anymore but we want to
keep it as a safe guard, it is still possible to tombstone a ydoc
programmatically but not used in the application for now.
When a document is deleted, we must reset the connection in the
collaboration server otherwise all users remain connected and can still
edit it if they are at least editor.
One feature we have is to still be able to read a soft deleted document
in the application. Calling the yhub service to delete the document
inside yhub make it unreachable, so it does not load anymore in the
editor. Yhub map the permission on what Docs returns in its abilities.
A soft deleted document will be in read only.
If two requests at the exact same time update a document and
link-configuration, a race condition can happen and one action can
overwrite the other one. If a document link_reach is changed from public
to restricted, this change can be overwritten and the owner does not
have this information. To fix this issue we set the updated_fields for
every save.
A version is identified by the moment it ends, which is also all the
changeset and the rollback are given. Two entries of the timeline can end
at the same moment without merging: an imported `system` save next to an
edit, or concurrent edits sorted by start. They produced two versions
with the same id, which both looked selected and shared a React key.
Entries ending together are the same document, so they are now folded
into one version, keeping every author and the earliest start.
Signed-off-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
Files under /_next/static/ carry a content hash in their name but were
served without Cache-Control, so browsers fell back to heuristic freshness
and revalidated each of the ~45 files of a page with a conditional request.
Serve them with a one-year immutable lifetime.
HTML pages, service-worker.js and public files keep a stable name: send
Cache-Control: no-cache so the browser revalidates them and always gets
the chunk list and service worker of the current release. The headers are
repeated in every location, as nginx drops the inherited add_header
directives as soon as a location defines its own.
Closes#2770
Signed-off-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
For big documents, calling yhub for each sub documents can be a big
amount of request at the same time. We decided to group reset by batch
in the path order and call again the reset celery task with a delay to
continue.
Fix#2656
Reset to the collaboration server when an access is changed was
partially made. There is still action in the application that does not
cover this like when a document is moved or soft deleted.
Instead of doing the reset in the viewset and searching for every action
it should be made, we are making it now in the model to miss new or
modified path acting on an access. For detecting when a bulk
update/delete is made on the access model, we use existing
signal.
The login and logout navigations (authenticate/, callback/, logout/,
logout-callback/) reach the service worker on the same origin, and the
catch-all api route cached their redirects. When the network step
failed, the worker answered authenticate/ with a cached redirect that
carried an OIDC state already used, so the login failed: a 400 on the
callback before django-lasuite 0.0.30.
Give these urls their own NetworkOnly route, registered before the
catch-all one, so they never read or write the cache. ApiPlugin stays
on it, so the offline mutations are still replayed before a logout
ends the session.
Fixes#2477
Signed-off-by: Roee Sivan <roee.sivan@post.runi.ac.il>
A test was getting flaky, and was blocking the CI pipeline.
This change improves the handling of sub-pages in
the e2e tests to make them more reliable.
Some services, like celery-dev, attach to the `default` network
explicitly, next to `lasuite`. Docker compose creates that implicit
network on its own. Podman compose does not when the top-level
`networks` section is declared without it, so `compose up` failed on
the unknown `default` network.
Declare `default: {}` so that the network exists with both tools.
The language picker marked an option as selected when it matched the
current language, falling back to a match on the language alone. With the
current language zh-tw, the zh-cn option matched too, so several regions
of a language were selected at once, and the legacy picker could show the
label of the wrong one. Now that zh-cn and zh-tw are separate frontend
translations, this shows up for the Chinese locales, and it would for any
language with several regions (en-us/en-gb, pt-pt/pt-br).
Add getClosestLocale, which returns the one available locale closest to
the current language (the exact match first, then the first region of the
language), and use it in both pickers to select the option and to display
the current label.
Pull the translations from Crowdin with the new locale-based export, so
that the frontend translations reflect the new taxonomy: zh-CN and zh-TW
are now two distinct entries (zh-cn and zh-tw) in translations.json
instead of a single "zh" one, and the backend .po files are refreshed.
Crowdin exported the frontend translations to a folder named after
%two_letters_code%, so every regional variant of a language landed in the
same folder: zh-CN and zh-TW both gave "zh", and the last one downloaded
overwrote the other. Mapping the Chinese locales apart would have fixed
only that pair, and each new variant (pt-BR, es-MX, en-GB...) would have
needed its own entry.
Use %locale% instead, which is unique per locale, like the backend already
does with %locale_with_underscore%. format-deploy keeps the keys of
translations.json as they are today (fr, de...) and only keeps the region
(zh-cn, zh-tw) when several locales of the same language are translated.
The language detection and the language picker already match a locale
exactly before falling back to its language, so they need no change.
The export pattern is stored by Crowdin with the source file: the sources
must be uploaded again before the next download, and the zh-CN data that
was polluted with Traditional Chinese still needs to be re-uploaded.
Fixes#2749
The antivirus loader rebuilt the image block with an empty caption and
left alignment, so both were lost after a replace.
Signed-off-by: Ovgodd <c.gromoff@gmail.com>
Add missing jobs to run the tests in swarm and canary projects. For k6
the project is inspected using k6 docker image and for the grafana
dashboard we check if the generated dashboard equals the one existing.
The Playwright base image in the Dockerfile has been
updated from v1.62.1 to v1.63.0, in order to follow the
version of the `playwright` package in package.json.
The export feature is not working correctly with
the newer version, so we are downgrading to 0.54.0.
We need to refactorize the code to be compatible
with typst in order to work correctly.
