Added
- 👷(ci) lint, typecheck, build and test the collaboration server (yhub) #2757
- 👷(ci) check the load-test tooling: swarm, canary, k6 and dashboards #2757
- 👷(ci) cancel the superseded runs of a pull request #2757
- ✨(backend) add mention endpoint with cooldown-limited email
notification #2447
- 🚩(setting) add feature flag on Duplicate with Children #2721
- 💄(frontend) redesign 404 error standalone page #2696
- 💄(frontend) redesign 403 access denied page #2720
- ✨(frontend) duplicate with subdocuments #2584
- ✨(frontend) add markdown download option #2608
- ✨(frontend) turn pasted doc links into interlinks #2713
- ✨(helm) allow disallowing search engine indexing per instance #2694
- ✨(backend) expose the attachment max size in the config endpoint #2577
- ✨(frontend) warn before uploading an attachment over the size limit #2577
- ✨(loadtest) add a websocket load generator for the collaboration
server #2556
- ✨(loadtest) add k6 scenarios for the page-open sequence and the heavy
endpoints #2556
- ✨(loadtest) add browser canaries measuring what a user feels under
load #2556
- ✨(loadtest) add the grafana dashboards of the load-test campaign, valkey
included #2556
- 📝(documentation) add the load-testing guide #2556
- 🔧(helm) run grafana with those dashboards in the dev cluster, in place of
the prometheus console #2556
- ✨(backend) measure the calls to yhub and to the converters, the database
pool and the celery queue #2556
- ✨(backend) add a `LoadTest` configuration and its `loadtest` application,
minting sessions for load tests #2556
- ✨(collaboration) add opt-in prometheus metrics to yhub, server and worker,
protected by a bearer token #2556
- ✨(collaboration) report yhub errors to sentry, configured through
`SENTRY_*` #2556
- ✨(backend) add opt-in prometheus metrics on `/metrics`, protected by a
bearer token #2556
- ✨(helm) add a dedicated ingress for the prometheus metrics of the backend
and of yhub #2556
- ✨(helm) add a ServiceMonitor and a PodMonitor per component whose metrics
are enabled, and `backend.metrics.enabled` #2556
- 🔧(helm) scrape the metrics of the dev cluster with a trimmed
kube-prometheus-stack and the ServiceMonitors of the chart #2556
- 🔧(helm) serve the example console of django-prometheus on the dev
Prometheus #2556
- ✨(backend) add a service generating cached RS256 JWT tokens #2556
- ✨(backend) publish the JWT public key on a JWKS endpoint #2556
- 🔧(dev) generate the JWT signing key when bootstrapping the dev stack #2556
- ✨(collaboration) add an admin reset-connections endpoint on yhub #2556
- ✨(collaboration) add a create-ydoc endpoint on yhub #2556
- ✨(collaboration) soft-migrate legacy S3 documents into yhub #2556
- ✨(collaboration) replay legacy s3 version history into yhub #2556
- ✨(backend) add a service to call the yhub REST API #2556
- ✨(backend) call YHubService to seed initial document content #2556
- ✨(collaboration) add a get-ydoc endpoint on yhub #2556
- ✨(backend) duplicate a document through the collaboration server #2556
- ✨(backend) serve `documents/{id}/formatted-content/` from yhub #2556
- ✨(collaboration) notify the backend when the worker persists new
content #2556
- ✨(collaboration) let a user read the document's editing history #2556
- ✨(frontend) fall back to http polling when the websocket cannot be
opened. #2556
- 🔧(collaboration) make the version-history granularity configurable through
`COLLABORATION_VERSION_GRANULARITY_MS` #2556
- ✨(frontend) keep a local copy of documents, so they open and stay editable
offline #2556
Changed
- 🛂(backend) let users allowed to comment list each other's accesses #2447
- ♻️(collaboration) migrate the collaboration server from hocuspocus to
yhub #2556
- 💥(y-provider) y-provider becomes converter-only #2556
- 💥(backend) move the resource server JWKS from `/api/{version}/jwks` to
`/external_api/{version}/jwks` #2556
- 🔧(collaboration) adapt docker stack for development purpose #2556
- 🔧(helm) run a valkey for the backend and one for yhub in dev and
feature #2556
- 💄(frontend) redesign 401 error standalone page #2716
Fixed
- 🐛(documentation) fix minio port and keycloak realm in the k8s guide #2751
- 🐛(collaboration) bound the calls yhub makes to the backend with a timeout,
`YHUB_BACKEND_REQUEST_TIMEOUT_MS`, 5s by default #2753
- 🐛(backend) compensate document duplication failures #2755
- 🐛(frontend) open search results in a new tab with ctrl/cmd+click #2719
- 🐛(docker) pull minio images from pgsty #2739
- 🐛(backend) retry the duplicate of a document on a tree path collision #2726
- 🐛(frontend) keep caption and alignment when replacing an image #2730
- 🐛(backend) reset the collaboration connections on every access change #2725
- ⚡️(backend) coalesce and pace the resets of collaboration connections #2725
- 🐛(frontend) clear callout background on Backspace #2052
- 🐛(export) keep image aspect ratio in PDF columns #2670
- 🐛(frontend) fix redirect after deleting a document #2706
- 🐛(frontend) keep documents draggable with a mouse when zoomed #2727
- 📝(docs) fix markdown typo in installation README #2766
- 🐛(i18n) export locale correctly #2750
- 🐛(frontend) keep login and logout out of the service worker cache #2771
- 🐛(frontend) reduce PostHog volume from web vitals and opt_in spam #2701
- 🐛(frontend) stop reconnecting to the websocket based on the status
code #2556
- 🐛(frontend) stop the service worker from caching the collaboration server's
rest api #2556
Removed
- 🔥(backend) remove the unused `CollaborationService` #2556
- 💥(backend) remove the `documents/{id}/can-edit/` endpoint #2556
- 💥(backend) remove the `documents/{id}/content/` endpoint #2556
- 🔥(backend) remove the document version endpoints. #2556
- 🔥(backend) remove `Document.content` #2556
Security
- 🔒️(collaboration) stop read-only users from sharing their cursor #2556
the task `sync_service_deletions_in_cascade` is no more called when a
document is soft deleted. We rename it in
sync_service_restorations_in_cascade and still call it when a document
is restored. No ydoc should be tombstone in yhub anymore but we want to
keep it as a safe guard, it is still possible to tombstone a ydoc
programmatically but not used in the application for now.
When a document is deleted, we must reset the connection in the
collaboration server otherwise all users remain connected and can still
edit it if they are at least editor.
One feature we have is to still be able to read a soft deleted document
in the application. Calling the yhub service to delete the document
inside yhub make it unreachable, so it does not load anymore in the
editor. Yhub map the permission on what Docs returns in its abilities.
A soft deleted document will be in read only.
If two requests at the exact same time update a document and
link-configuration, a race condition can happen and one action can
overwrite the other one. If a document link_reach is changed from public
to restricted, this change can be overwritten and the owner does not
have this information. To fix this issue we set the updated_fields for
every save.
A version is identified by the moment it ends, which is also all the
changeset and the rollback are given. Two entries of the timeline can end
at the same moment without merging: an imported `system` save next to an
edit, or concurrent edits sorted by start. They produced two versions
with the same id, which both looked selected and shared a React key.
Entries ending together are the same document, so they are now folded
into one version, keeping every author and the earliest start.
Signed-off-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
Files under /_next/static/ carry a content hash in their name but were
served without Cache-Control, so browsers fell back to heuristic freshness
and revalidated each of the ~45 files of a page with a conditional request.
Serve them with a one-year immutable lifetime.
HTML pages, service-worker.js and public files keep a stable name: send
Cache-Control: no-cache so the browser revalidates them and always gets
the chunk list and service worker of the current release. The headers are
repeated in every location, as nginx drops the inherited add_header
directives as soon as a location defines its own.
Closes#2770
Signed-off-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
For big documents, calling yhub for each sub documents can be a big
amount of request at the same time. We decided to group reset by batch
in the path order and call again the reset celery task with a delay to
continue.
Fix#2656
Reset to the collaboration server when an access is changed was
partially made. There is still action in the application that does not
cover this like when a document is moved or soft deleted.
Instead of doing the reset in the viewset and searching for every action
it should be made, we are making it now in the model to miss new or
modified path acting on an access. For detecting when a bulk
update/delete is made on the access model, we use existing
signal.
The login and logout navigations (authenticate/, callback/, logout/,
logout-callback/) reach the service worker on the same origin, and the
catch-all api route cached their redirects. When the network step
failed, the worker answered authenticate/ with a cached redirect that
carried an OIDC state already used, so the login failed: a 400 on the
callback before django-lasuite 0.0.30.
Give these urls their own NetworkOnly route, registered before the
catch-all one, so they never read or write the cache. ApiPlugin stays
on it, so the offline mutations are still replayed before a logout
ends the session.
Fixes#2477
Signed-off-by: Roee Sivan <roee.sivan@post.runi.ac.il>
A test was getting flaky, and was blocking the CI pipeline.
This change improves the handling of sub-pages in
the e2e tests to make them more reliable.
Some services, like celery-dev, attach to the `default` network
explicitly, next to `lasuite`. Docker compose creates that implicit
network on its own. Podman compose does not when the top-level
`networks` section is declared without it, so `compose up` failed on
the unknown `default` network.
Declare `default: {}` so that the network exists with both tools.
The language picker marked an option as selected when it matched the
current language, falling back to a match on the language alone. With the
current language zh-tw, the zh-cn option matched too, so several regions
of a language were selected at once, and the legacy picker could show the
label of the wrong one. Now that zh-cn and zh-tw are separate frontend
translations, this shows up for the Chinese locales, and it would for any
language with several regions (en-us/en-gb, pt-pt/pt-br).
Add getClosestLocale, which returns the one available locale closest to
the current language (the exact match first, then the first region of the
language), and use it in both pickers to select the option and to display
the current label.
Pull the translations from Crowdin with the new locale-based export, so
that the frontend translations reflect the new taxonomy: zh-CN and zh-TW
are now two distinct entries (zh-cn and zh-tw) in translations.json
instead of a single "zh" one, and the backend .po files are refreshed.
Crowdin exported the frontend translations to a folder named after
%two_letters_code%, so every regional variant of a language landed in the
same folder: zh-CN and zh-TW both gave "zh", and the last one downloaded
overwrote the other. Mapping the Chinese locales apart would have fixed
only that pair, and each new variant (pt-BR, es-MX, en-GB...) would have
needed its own entry.
Use %locale% instead, which is unique per locale, like the backend already
does with %locale_with_underscore%. format-deploy keeps the keys of
translations.json as they are today (fr, de...) and only keeps the region
(zh-cn, zh-tw) when several locales of the same language are translated.
The language detection and the language picker already match a locale
exactly before falling back to its language, so they need no change.
The export pattern is stored by Crowdin with the source file: the sources
must be uploaded again before the next download, and the zh-CN data that
was polluted with Traditional Chinese still needs to be re-uploaded.
Fixes#2749
The antivirus loader rebuilt the image block with an empty caption and
left alignment, so both were lost after a replace.
Signed-off-by: Ovgodd <c.gromoff@gmail.com>
Add missing jobs to run the tests in swarm and canary projects. For k6
the project is inspected using k6 docker image and for the grafana
dashboard we check if the generated dashboard equals the one existing.
The Playwright base image in the Dockerfile has been
updated from v1.62.1 to v1.63.0, in order to follow the
version of the `playwright` package in package.json.
The export feature is not working correctly with
the newer version, so we are downgrading to 0.54.0.
We need to refactorize the code to be compatible
with typst in order to work correctly.
Duplicating a document tree made two calls to the collaboration server
per node, all inside the database transaction: it held locks for the
duration of every call, and a failure midway left the documents already
created on the collaboration server behind, as orphans nothing can
reach anymore.
The duplication now runs in two phases. The structure is first
replicated in a transaction that makes no call to the collaboration
server, then each duplicate is seeded with the content of its source.
A failure while seeding deletes what was seeded - the deletion of the
collaboration server is idempotent and never refused - and removes the
replicated documents, so neither side keeps an orphan.
Signed-off-by: Marrrrrrrrry <34876935+Marrrrrrrrry@users.noreply.github.com>
`backendFetch` had no timeout: a backend that hangs hangs every
websocket
upgrade and fallback poll with it, piling up under load. The three
auth-path
calls it serves (`users/me`, `documents/{id}`, `accesses/me`) waited
forever,
so a slow backend took the whole collaboration service down with it.
Signed-off-by: Marrrrrrrrry <34876935+Marrrrrrrrry@users.noreply.github.com>
The guide showed the impress realm and client, while keycloak.values.yaml
creates the docs realm and impress.values.yaml uses the docs client.
Copying the endpoints from the guide led to a 404.
Signed-off-by: Lewis Bertin <lewis@tactiqual.tech>
Since helm-dev-backend 0.0.13 (silo image) the server always listens
on 9000. The example still set api.port to 80, so the service pointed
to a closed port, the bucket job waited forever and helm install timed
out. Dropping it keeps the chart default, which impress.values.yaml
and the dev values already expect.
Signed-off-by: Lewis Bertin <lewis@tactiqual.tech>
Users mentioning themselves got an email about something they just
did, which is noise. Keep recording the mention, skip the email.
Signed-off-by: BOUKERFA Mohamed El Amine <boukerfa.ma@gmail.com>
The cache key claimed before sending outlived a failed notification,
so the next mention in the same context was silently dropped until the
key expired although nobody had been notified. Delete it unless the
email went out and `notified_at` was saved.
Signed-off-by: BOUKERFA Mohamed El Amine <boukerfa.ma@gmail.com>
Gate the ability to mention to users with explicit
access so only collaborators can mention collaborators.
Signed-off-by: BOUKERFA Mohamed El Amine <boukerfa.ma@gmail.com>
