The left panel home button was a plain button calling
router.push, so it could not be opened in a new
tab with Ctrl/Cmd+click or the context menu.
Render it with ButtonLink so it is a real link to /,
keeping client-side navigation on a normal click.
Signed-off-by: Jul11us <233702296+Jul11us@users.noreply.github.com>
Bind the thread to the document named in the URL path: a comment must
never be stored in a thread that belongs to another document, even when
the caller is allowed to comment on that other document.
Drag-and-drop to move a document was disabled below a viewport
width breakpoint.
Browser zoom shrinks window.innerWidth like a narrower window
does, so zooming in on a desktop with a mouse silently disabled
moving documents, in the docs grid and in the doc tree.
A fix was made on the doc grid, but not in the tree,
we updated useResponsiveStore to tell us if the window
has coarse pointer or not (hasCoarsePointer).
Closes#2599
Last version of @y/hub contains a fix we need, it fixes the websocket
reconnection loop for anonymous user on a read only document sending a
keepalive every 20 seconds. In both version 0.10.0 and 0.10.1 breaking
changes are introduced, one is impacting us by removing the
computePool.patchYdoc function used by the create-ydoc endpoint declated
in our yhub server. To fix it, we are doing what the version 0.9.0 was
doing and we keep the same behavior.
A new environment variable YHUB_AUTH_TIMEOUT_MS is added to configure
the server.authTimeout parameters in the createYhub options
The loader used Card, adding an unnamed region per upload, and neither
the analysis start nor the antivirus warning was announced. Use Box
again, announce both through announce(), hide the decorative warning
icon and keep upload error toasts 10s so the cause can be read.
We updated the UI to provide a clearer indication
when an unexpected upload error occurs.
Before the error was displayed on the top
of the editor, making it less noticeable to the user,
now we display the error directly within the
upload block, making it more visible to the user
and more user-friendly.
BlockNote XL packages moved from AGPL-3.0 to GPL-3.0 (or a
proprietary license) between v0.30 and v0.40.
We renamed the occurences of AGPL to GPL in the codebase.
Signed-off-by: Roee Sivan <roee.sivan@post.runi.ac.il>
DocSpec now returns 415 when the file type is rejected, 422 when it
can't process the file content, and 500 when it crashes, instead of
one generic failure. Propagate those statuses through the API and show
a matching toast message per case on the frontend.
Closes#2552
Added:
- ✨(backend) add a setting for the http fallback polling interval
- ✨(backend) add a setting to turn off awareness on the http fallback
- ✨(backend) add a setting to send json on the http fallback
- ⚡️(yhub) cache authz verdicts to cut backend calls on connections
Changed:
- ⚡️(yhub) skip the `users/me` call for callers without a session
cookie
- 🔧(helm) allow to set the period and failure threshold of probes
- ♿️(frontend) fix the accessibility of the document tree
Fixed:
- 🐛(frontend) stop the http fallback when the collaboration server
refuses it
- 🐛(backend) bypass throttling on access/me action from yhub
- 🐛(frontend) switch language when the current one is untranslated
- 🐛(frontend) keep the loading skeleton until an empty doc has synced
Removed:
- 🔥(backend) remove `ForceSessionMiddleware`, an orphan of the removed
non-websocket save safeguard
Signed-off-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
## Purpose
A connection to the collaboration server — websocket or http fallback —
can chain up to three calls to the backend: users/me, documents/{id} and
accesses/me. On the fallback poll path this repeats every few seconds per
client, and the calls bypass the backend throttling (X-Y-Provider-Key),
so a busy instance hammers the backend unthrottled.
Add an opt-in authz cache in yhub, three layers with different
invalidation physics:
- identity: users/me -> userid, in-process per replica, keyed on the
SHA-256 of the Cookie header. Nothing can invalidate it and nothing
needs to: a dead session fails closed at the abilities fetch. Kills
the users/me call on every poll and REST request.
- abilities: the documents/{id} verdict, in the shared redis per
(document, cookie hash). The one payload a stale entry would
over-grant, so reset-connections retires every verdict of the
document — a single generation write — BEFORE triggering the recheck;
the recheck then re-runs authorize against post-invalidation state
and the revocation stays immediate. The TTL is only the backstop for
a lost kick. Anonymous callers share one entry per document: what
they may do is the link configuration, and nothing about them
distinguishes one from another.
- access date: the accesses/me created_at bounding the history a user
may read. Append-mostly by design, retired by the same per-document
kick as the abilities — a stale earlier date would widen the readable
history, hence the longest TTL of the three.
All three TTLs default to 0: caching off, byte-identical behavior.
Backend refusals (401/403/404) are cached like any verdict, so a client
that lost access costs one call per TTL instead of one per poll. An
unanswered backend is never cached — an error is not a permission
decision. Keys carry only sha256 hashes, never session ids.
Warm-state connections and polls make zero backend calls. Watch
yhub_authz_cache_lookups_total{layer,result} against
yhub_backend_request_duration_seconds{route} before raising the TTLs.
The dev-backend chart 0.0.10 started the media storage with a hardcoded
`minio server` shell command and pulled quay.io/minio/mc to create the
bucket. Since the switch to pgsty/silo, the first one failed with
"minio: command not found" and the second one with a 401 from quay.io, so
the feature deployments (ArgoCD) did not come up.
Chart 0.0.15 defaults to pgsty/silo and pgsty/mc, and starts the container
with `args` instead of a shell command, so the name of the executable no
longer matters.
Some firewalls refuse the application/x-lib0any content type the http
fallback gets from the collaboration server, and the octet-stream it
sends, so the browsers behind them cannot sync when their websocket is
blocked too.
Add COLLABORATION_HTTP_FALLBACK_JSON_ENABLED (default False), served by
the config endpoint. On, the fallback sends and receives application/json
with the binary parts in base64, which the collaboration server already
supports. The cost is a document about a third larger on each transfer.
After removing the ForceSessionMiddleware we can detect without calling
the /users/me endpoint if a user is anonymous or not. We now check the
presence of the cookie session to avoid the http call and directly
return an anonymous user.
This middleware was introduced in 651f2d1d to create a session for every
user — connected to the collaboration websocket or not — so the backend
could arbitrate who may save a document (websocket payload flag and save
lock). The whole safeguard was removed in 68f5f5cdf when the content and
its permissions moved to yhub, leaving the middleware orphaned: nothing
reads an anonymous session anymore (the OIDC login flow saves its own
session, CSRF is cookie-based, throttling is not session-keyed).
It only kept a cost: a Redis write plus a Set-Cookie on every anonymous
response, which had already required two bugfix commits to exempt the
probes. A regression test pins the probe invariant going forward.
Where websockets are blocked, a tab whose access was revoked or whose
session expired took a 403 on every PATCH, forever, since only the
websocket's 4401 close made the tab fetch the document again. Treat a
401, 403 or 404 on the fallback the same way: stop both transports and
fetch the document again.
An editor on the http fallback publishes its presence every 10 seconds,
a request on its own, and the library then never polls less often than
every 15 seconds. On a deployment where many browsers cannot open the
websocket, this is most of the traffic and each request is authorized
against the backend.
Add COLLABORATION_HTTP_FALLBACK_AWARENESS_ENABLED (default True, so
nothing changes unless it is set), served by the config endpoint. Off,
an editor sends a request only when it has changes and
COLLABORATION_HTTP_FALLBACK_SYNC_INTERVAL applies in full. The cost is
no remote cursors on the fallback.
Signed-off-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
Each poll of a browser on the http fallback is authorized against the
backend, and the 10 second interval was hardcoded. Add
COLLABORATION_HTTP_FALLBACK_SYNC_INTERVAL (milliseconds, default 10000,
so nothing changes unless it is set), served by the config endpoint, so
operators can lower that load without a new release.
A doc never opened on this device has an empty local copy, and the
socket opening is not the content arriving either. Rendering on either
showed the empty-doc placeholder for a moment, which reads as a doc whose
content was deleted.
Signed-off-by: Dor Zairi <dorzairidev@gmail.com>
With zh-tw active, selecting English did nothing: its bundle is empty, so
i18next reports `resolvedLanguage` as en while `language` is zh-tw, and
the hook skipped the change. Compare with `language` instead, and add
unit tests on the language pickers.
A throttling bypass exists on the DocumentThrottle class allowing YHub
server to fetch django and not be blocked by the throttling. YHub also
fetch an endpont on the DocumentAccess viewset and this bypass is not
existing. We must add it. We factorized the bypass in a mixin and this
mixin is now used by both the DocumentThrottle and
DocumentAccessThrottle classes.
Added
- 👷(ci) lint, typecheck, build and test the collaboration server (yhub) #2757
- 👷(ci) check the load-test tooling: swarm, canary, k6 and dashboards #2757
- 👷(ci) cancel the superseded runs of a pull request #2757
- ✨(backend) add mention endpoint with cooldown-limited email
notification #2447
- 🚩(setting) add feature flag on Duplicate with Children #2721
- 💄(frontend) redesign 404 error standalone page #2696
- 💄(frontend) redesign 403 access denied page #2720
- ✨(frontend) duplicate with subdocuments #2584
- ✨(frontend) add markdown download option #2608
- ✨(frontend) turn pasted doc links into interlinks #2713
- ✨(helm) allow disallowing search engine indexing per instance #2694
- ✨(backend) expose the attachment max size in the config endpoint #2577
- ✨(frontend) warn before uploading an attachment over the size limit #2577
- ✨(loadtest) add a websocket load generator for the collaboration
server #2556
- ✨(loadtest) add k6 scenarios for the page-open sequence and the heavy
endpoints #2556
- ✨(loadtest) add browser canaries measuring what a user feels under
load #2556
- ✨(loadtest) add the grafana dashboards of the load-test campaign, valkey
included #2556
- 📝(documentation) add the load-testing guide #2556
- 🔧(helm) run grafana with those dashboards in the dev cluster, in place of
the prometheus console #2556
- ✨(backend) measure the calls to yhub and to the converters, the database
pool and the celery queue #2556
- ✨(backend) add a `LoadTest` configuration and its `loadtest` application,
minting sessions for load tests #2556
- ✨(collaboration) add opt-in prometheus metrics to yhub, server and worker,
protected by a bearer token #2556
- ✨(collaboration) report yhub errors to sentry, configured through
`SENTRY_*` #2556
- ✨(backend) add opt-in prometheus metrics on `/metrics`, protected by a
bearer token #2556
- ✨(helm) add a dedicated ingress for the prometheus metrics of the backend
and of yhub #2556
- ✨(helm) add a ServiceMonitor and a PodMonitor per component whose metrics
are enabled, and `backend.metrics.enabled` #2556
- 🔧(helm) scrape the metrics of the dev cluster with a trimmed
kube-prometheus-stack and the ServiceMonitors of the chart #2556
- 🔧(helm) serve the example console of django-prometheus on the dev
Prometheus #2556
- ✨(backend) add a service generating cached RS256 JWT tokens #2556
- ✨(backend) publish the JWT public key on a JWKS endpoint #2556
- 🔧(dev) generate the JWT signing key when bootstrapping the dev stack #2556
- ✨(collaboration) add an admin reset-connections endpoint on yhub #2556
- ✨(collaboration) add a create-ydoc endpoint on yhub #2556
- ✨(collaboration) soft-migrate legacy S3 documents into yhub #2556
- ✨(collaboration) replay legacy s3 version history into yhub #2556
- ✨(backend) add a service to call the yhub REST API #2556
- ✨(backend) call YHubService to seed initial document content #2556
- ✨(collaboration) add a get-ydoc endpoint on yhub #2556
- ✨(backend) duplicate a document through the collaboration server #2556
- ✨(backend) serve `documents/{id}/formatted-content/` from yhub #2556
- ✨(collaboration) notify the backend when the worker persists new
content #2556
- ✨(collaboration) let a user read the document's editing history #2556
- ✨(frontend) fall back to http polling when the websocket cannot be
opened. #2556
- 🔧(collaboration) make the version-history granularity configurable through
`COLLABORATION_VERSION_GRANULARITY_MS` #2556
- ✨(frontend) keep a local copy of documents, so they open and stay editable
offline #2556
Changed
- 🛂(backend) let users allowed to comment list each other's accesses #2447
- ♻️(collaboration) migrate the collaboration server from hocuspocus to
yhub #2556
- 💥(y-provider) y-provider becomes converter-only #2556
- 💥(backend) move the resource server JWKS from `/api/{version}/jwks` to
`/external_api/{version}/jwks` #2556
- 🔧(collaboration) adapt docker stack for development purpose #2556
- 🔧(helm) run a valkey for the backend and one for yhub in dev and
feature #2556
- 💄(frontend) redesign 401 error standalone page #2716
Fixed
- 🐛(documentation) fix minio port and keycloak realm in the k8s guide #2751
- 🐛(collaboration) bound the calls yhub makes to the backend with a timeout,
`YHUB_BACKEND_REQUEST_TIMEOUT_MS`, 5s by default #2753
- 🐛(backend) compensate document duplication failures #2755
- 🐛(frontend) open search results in a new tab with ctrl/cmd+click #2719
- 🐛(docker) pull minio images from pgsty #2739
- 🐛(backend) retry the duplicate of a document on a tree path collision #2726
- 🐛(frontend) keep caption and alignment when replacing an image #2730
- 🐛(backend) reset the collaboration connections on every access change #2725
- ⚡️(backend) coalesce and pace the resets of collaboration connections #2725
- 🐛(frontend) clear callout background on Backspace #2052
- 🐛(export) keep image aspect ratio in PDF columns #2670
- 🐛(frontend) fix redirect after deleting a document #2706
- 🐛(frontend) keep documents draggable with a mouse when zoomed #2727
- 📝(docs) fix markdown typo in installation README #2766
- 🐛(i18n) export locale correctly #2750
- 🐛(frontend) keep login and logout out of the service worker cache #2771
- 🐛(frontend) reduce PostHog volume from web vitals and opt_in spam #2701
- 🐛(frontend) stop reconnecting to the websocket based on the status
code #2556
- 🐛(frontend) stop the service worker from caching the collaboration server's
rest api #2556
Removed
- 🔥(backend) remove the unused `CollaborationService` #2556
- 💥(backend) remove the `documents/{id}/can-edit/` endpoint #2556
- 💥(backend) remove the `documents/{id}/content/` endpoint #2556
- 🔥(backend) remove the document version endpoints. #2556
- 🔥(backend) remove `Document.content` #2556
Security
- 🔒️(collaboration) stop read-only users from sharing their cursor #2556
the task `sync_service_deletions_in_cascade` is no more called when a
document is soft deleted. We rename it in
sync_service_restorations_in_cascade and still call it when a document
is restored. No ydoc should be tombstone in yhub anymore but we want to
keep it as a safe guard, it is still possible to tombstone a ydoc
programmatically but not used in the application for now.
When a document is deleted, we must reset the connection in the
collaboration server otherwise all users remain connected and can still
edit it if they are at least editor.
One feature we have is to still be able to read a soft deleted document
in the application. Calling the yhub service to delete the document
inside yhub make it unreachable, so it does not load anymore in the
editor. Yhub map the permission on what Docs returns in its abilities.
A soft deleted document will be in read only.
If two requests at the exact same time update a document and
link-configuration, a race condition can happen and one action can
overwrite the other one. If a document link_reach is changed from public
to restricted, this change can be overwritten and the owner does not
have this information. To fix this issue we set the updated_fields for
every save.
A version is identified by the moment it ends, which is also all the
changeset and the rollback are given. Two entries of the timeline can end
at the same moment without merging: an imported `system` save next to an
edit, or concurrent edits sorted by start. They produced two versions
with the same id, which both looked selected and shared a React key.
Entries ending together are the same document, so they are now folded
into one version, keeping every author and the earliest start.
Signed-off-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
Files under /_next/static/ carry a content hash in their name but were
served without Cache-Control, so browsers fell back to heuristic freshness
and revalidated each of the ~45 files of a page with a conditional request.
Serve them with a one-year immutable lifetime.
HTML pages, service-worker.js and public files keep a stable name: send
Cache-Control: no-cache so the browser revalidates them and always gets
the chunk list and service worker of the current release. The headers are
repeated in every location, as nginx drops the inherited add_header
directives as soon as a location defines its own.
Closes#2770
Signed-off-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
For big documents, calling yhub for each sub documents can be a big
amount of request at the same time. We decided to group reset by batch
in the path order and call again the reset celery task with a delay to
continue.
Fix#2656
Reset to the collaboration server when an access is changed was
partially made. There is still action in the application that does not
cover this like when a document is moved or soft deleted.
Instead of doing the reset in the viewset and searching for every action
it should be made, we are making it now in the model to miss new or
modified path acting on an access. For detecting when a bulk
update/delete is made on the access model, we use existing
signal.
The login and logout navigations (authenticate/, callback/, logout/,
logout-callback/) reach the service worker on the same origin, and the
catch-all api route cached their redirects. When the network step
failed, the worker answered authenticate/ with a cached redirect that
carried an OIDC state already used, so the login failed: a 400 on the
callback before django-lasuite 0.0.30.
Give these urls their own NetworkOnly route, registered before the
catch-all one, so they never read or write the cache. ApiPlugin stays
on it, so the offline mutations are still replayed before a logout
ends the session.
Fixes#2477
Signed-off-by: Roee Sivan <roee.sivan@post.runi.ac.il>
