Added
- 👷(ci) lint, typecheck, build and test the collaboration server (yhub) #2757
- 👷(ci) check the load-test tooling: swarm, canary, k6 and dashboards #2757
- 👷(ci) cancel the superseded runs of a pull request #2757
- ✨(backend) add mention endpoint with cooldown-limited email
notification #2447
- 🚩(setting) add feature flag on Duplicate with Children #2721
- 💄(frontend) redesign 404 error standalone page #2696
- 💄(frontend) redesign 403 access denied page #2720
- ✨(frontend) duplicate with subdocuments #2584
- ✨(frontend) add markdown download option #2608
- ✨(frontend) turn pasted doc links into interlinks #2713
- ✨(helm) allow disallowing search engine indexing per instance #2694
- ✨(backend) expose the attachment max size in the config endpoint #2577
- ✨(frontend) warn before uploading an attachment over the size limit #2577
- ✨(loadtest) add a websocket load generator for the collaboration
server #2556
- ✨(loadtest) add k6 scenarios for the page-open sequence and the heavy
endpoints #2556
- ✨(loadtest) add browser canaries measuring what a user feels under
load #2556
- ✨(loadtest) add the grafana dashboards of the load-test campaign, valkey
included #2556
- 📝(documentation) add the load-testing guide #2556
- 🔧(helm) run grafana with those dashboards in the dev cluster, in place of
the prometheus console #2556
- ✨(backend) measure the calls to yhub and to the converters, the database
pool and the celery queue #2556
- ✨(backend) add a `LoadTest` configuration and its `loadtest` application,
minting sessions for load tests #2556
- ✨(collaboration) add opt-in prometheus metrics to yhub, server and worker,
protected by a bearer token #2556
- ✨(collaboration) report yhub errors to sentry, configured through
`SENTRY_*` #2556
- ✨(backend) add opt-in prometheus metrics on `/metrics`, protected by a
bearer token #2556
- ✨(helm) add a dedicated ingress for the prometheus metrics of the backend
and of yhub #2556
- ✨(helm) add a ServiceMonitor and a PodMonitor per component whose metrics
are enabled, and `backend.metrics.enabled` #2556
- 🔧(helm) scrape the metrics of the dev cluster with a trimmed
kube-prometheus-stack and the ServiceMonitors of the chart #2556
- 🔧(helm) serve the example console of django-prometheus on the dev
Prometheus #2556
- ✨(backend) add a service generating cached RS256 JWT tokens #2556
- ✨(backend) publish the JWT public key on a JWKS endpoint #2556
- 🔧(dev) generate the JWT signing key when bootstrapping the dev stack #2556
- ✨(collaboration) add an admin reset-connections endpoint on yhub #2556
- ✨(collaboration) add a create-ydoc endpoint on yhub #2556
- ✨(collaboration) soft-migrate legacy S3 documents into yhub #2556
- ✨(collaboration) replay legacy s3 version history into yhub #2556
- ✨(backend) add a service to call the yhub REST API #2556
- ✨(backend) call YHubService to seed initial document content #2556
- ✨(collaboration) add a get-ydoc endpoint on yhub #2556
- ✨(backend) duplicate a document through the collaboration server #2556
- ✨(backend) serve `documents/{id}/formatted-content/` from yhub #2556
- ✨(collaboration) notify the backend when the worker persists new
content #2556
- ✨(collaboration) let a user read the document's editing history #2556
- ✨(frontend) fall back to http polling when the websocket cannot be
opened. #2556
- 🔧(collaboration) make the version-history granularity configurable through
`COLLABORATION_VERSION_GRANULARITY_MS` #2556
- ✨(frontend) keep a local copy of documents, so they open and stay editable
offline #2556
Changed
- 🛂(backend) let users allowed to comment list each other's accesses #2447
- ♻️(collaboration) migrate the collaboration server from hocuspocus to
yhub #2556
- 💥(y-provider) y-provider becomes converter-only #2556
- 💥(backend) move the resource server JWKS from `/api/{version}/jwks` to
`/external_api/{version}/jwks` #2556
- 🔧(collaboration) adapt docker stack for development purpose #2556
- 🔧(helm) run a valkey for the backend and one for yhub in dev and
feature #2556
- 💄(frontend) redesign 401 error standalone page #2716
Fixed
- 🐛(documentation) fix minio port and keycloak realm in the k8s guide #2751
- 🐛(collaboration) bound the calls yhub makes to the backend with a timeout,
`YHUB_BACKEND_REQUEST_TIMEOUT_MS`, 5s by default #2753
- 🐛(backend) compensate document duplication failures #2755
- 🐛(frontend) open search results in a new tab with ctrl/cmd+click #2719
- 🐛(docker) pull minio images from pgsty #2739
- 🐛(backend) retry the duplicate of a document on a tree path collision #2726
- 🐛(frontend) keep caption and alignment when replacing an image #2730
- 🐛(backend) reset the collaboration connections on every access change #2725
- ⚡️(backend) coalesce and pace the resets of collaboration connections #2725
- 🐛(frontend) clear callout background on Backspace #2052
- 🐛(export) keep image aspect ratio in PDF columns #2670
- 🐛(frontend) fix redirect after deleting a document #2706
- 🐛(frontend) keep documents draggable with a mouse when zoomed #2727
- 📝(docs) fix markdown typo in installation README #2766
- 🐛(i18n) export locale correctly #2750
- 🐛(frontend) keep login and logout out of the service worker cache #2771
- 🐛(frontend) reduce PostHog volume from web vitals and opt_in spam #2701
- 🐛(frontend) stop reconnecting to the websocket based on the status
code #2556
- 🐛(frontend) stop the service worker from caching the collaboration server's
rest api #2556
Removed
- 🔥(backend) remove the unused `CollaborationService` #2556
- 💥(backend) remove the `documents/{id}/can-edit/` endpoint #2556
- 💥(backend) remove the `documents/{id}/content/` endpoint #2556
- 🔥(backend) remove the document version endpoints. #2556
- 🔥(backend) remove `Document.content` #2556
Security
- 🔒️(collaboration) stop read-only users from sharing their cursor #2556
A version is identified by the moment it ends, which is also all the
changeset and the rollback are given. Two entries of the timeline can end
at the same moment without merging: an imported `system` save next to an
edit, or concurrent edits sorted by start. They produced two versions
with the same id, which both looked selected and shared a React key.
Entries ending together are the same document, so they are now folded
into one version, keeping every author and the earliest start.
Signed-off-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
Files under /_next/static/ carry a content hash in their name but were
served without Cache-Control, so browsers fell back to heuristic freshness
and revalidated each of the ~45 files of a page with a conditional request.
Serve them with a one-year immutable lifetime.
HTML pages, service-worker.js and public files keep a stable name: send
Cache-Control: no-cache so the browser revalidates them and always gets
the chunk list and service worker of the current release. The headers are
repeated in every location, as nginx drops the inherited add_header
directives as soon as a location defines its own.
Closes#2770
Signed-off-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
The login and logout navigations (authenticate/, callback/, logout/,
logout-callback/) reach the service worker on the same origin, and the
catch-all api route cached their redirects. When the network step
failed, the worker answered authenticate/ with a cached redirect that
carried an OIDC state already used, so the login failed: a 400 on the
callback before django-lasuite 0.0.30.
Give these urls their own NetworkOnly route, registered before the
catch-all one, so they never read or write the cache. ApiPlugin stays
on it, so the offline mutations are still replayed before a logout
ends the session.
Fixes#2477
Signed-off-by: Roee Sivan <roee.sivan@post.runi.ac.il>
A test was getting flaky, and was blocking the CI pipeline.
This change improves the handling of sub-pages in
the e2e tests to make them more reliable.
The language picker marked an option as selected when it matched the
current language, falling back to a match on the language alone. With the
current language zh-tw, the zh-cn option matched too, so several regions
of a language were selected at once, and the legacy picker could show the
label of the wrong one. Now that zh-cn and zh-tw are separate frontend
translations, this shows up for the Chinese locales, and it would for any
language with several regions (en-us/en-gb, pt-pt/pt-br).
Add getClosestLocale, which returns the one available locale closest to
the current language (the exact match first, then the first region of the
language), and use it in both pickers to select the option and to display
the current label.
Pull the translations from Crowdin with the new locale-based export, so
that the frontend translations reflect the new taxonomy: zh-CN and zh-TW
are now two distinct entries (zh-cn and zh-tw) in translations.json
instead of a single "zh" one, and the backend .po files are refreshed.
The antivirus loader rebuilt the image block with an empty caption and
left alignment, so both were lost after a replace.
Signed-off-by: Ovgodd <c.gromoff@gmail.com>
The export feature is not working correctly with
the newer version, so we are downgrading to 0.54.0.
We need to refactorize the code to be compatible
with typst in order to work correctly.
We can now paste docs url that will become interlinking.
So we don't want to pass titles through the inline content itself
anymore, to not leak any titles. The problem is that inline content
Inline content mappers are not async, so titles must be
resolved upfront.
We adapted the export interlinking to get titles from
a pre-resolved map instead of relying on inline content.
Pasting a bare link to another doc on the same domain now inserts an
interlink instead of a plain hyperlink, so it benefits from the same
title-sync and navigation behaviour as links added through the `/` or
`@` menu. Pasting over a selection, inside a code block, or a link the
current user can't resolve will display the doc id instead.
We want to introduce a feature flag that allows
duplicating documents along with their children when
enabled. We will be able to control not only
frontend side but also backend behavior
regarding document with children duplication.
We want to have a fine grained control over the
Duplicate with Children feature.
By adding the feature flag for Duplicate with Children,
we can enable or disable this feature for specific
users or groups without deploying new code.
This allows us to test the feature in a controlled
environment and gather feedback before a full rollout.
Search results were cmdk items navigating with router.push on select,
so the browser had no link to open elsewhere. Each result is now a real
link to the document: a plain click still goes through the quick search
onSelect, while ctrl, cmd, shift or middle click let the browser open it
in a new tab or window and keep the search modal open.
Signed-off-by: Armand <schneideretcollier.innovation@gmail.com>
The offline support couldn't work with the existing
implementation anymore, because there is no request
to get or save data anymore, everything is handled
with web sockets.
In order to support offline functionality, we
leveraged y-indexeddb to store and synchronize
local changes, ensuring that the application remains
functional even when offline.
Version grouping used a hard-coded 60s window, and its
timestamps are minted server-side.
We now expose COLLABORATION_VERSION_GRANULARITY_MS
through /config, read it in useDocActivity.
By doing so, we can control the granularity of
version history with settings, it can be adjusted as
needed. It will help us to test different version
history granularities in our e2e tests.
Previously we added to retrieve document response a new property
user_access_since that contains the date from when the user started to
have access to the document. The way t was made added an other
annotation to the Document queryset making the sql query more and more
complex. We decided to lighten the queryset and expose the access the
user has on the document instead and read the history from the
created_at property.
The check was dropped during the yhub migration, on the grounds that
`reset_connections` was a no-op. It is not one any more: the backend calls
yhub's reset-connections route, which re-runs the authorization of each
connection and closes the sockets whose access is no longer the one they
were opened with, with the code 4401.
The old check cannot come back as it was. It had the author of the document
change its visibility and waited for the author's own socket to close, which
is what the collaboration server used to do - close every connection to the
document, whatever it was that changed. yhub leaves alone the connections
that are unaffected, and an owner's access is the same before and after, so
the author's socket is now deliberately kept.
The client to watch is therefore another one: the anonymous reader holding
the public link, while the author switches that link from reading to
editing. Their socket goes down and the client comes back on its own.
That the new connection is live is asserted by what flows through it rather
than by the socket being open - the author writes after the change and the
text reaches the reader, with nothing reloaded in between.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The export regression tests injected their fixture by intercepting
`GET /documents/{id}/content/`, and the 401 routing test intercepted the
`PATCH .../content/` a document used to send on the way out. Both endpoints
went with the yhub migration, so the interceptions matched nothing: the
exported document was whatever the two images added to an empty page, and
the 401 page was never reached.
The fixture now goes to the collaboration server, which is the only thing
that holds document content. It is seeded before the editor mounts: its
first block carries the id BlockNote gives the empty paragraph it writes
into any document it opens empty, so seeding an already open document would
leave a stray paragraph in front of the fixture and shift every page. The
reference PDFs are unchanged and still match, page count, text and pixels.
The 401 test reloads rather than navigating to another document, nothing
being sent on the way out any more.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
A document's imported past is attributed to `system`, and until now it went
through the same grouping as ordinary editing. That is wrong twice over.
It loses history. Before the migration the editor saved the whole document on
an interval of exactly 60s, each save becoming one version of the legacy file,
and the migration replays them at their original timestamps — so an imported
history arrives as a chain of entries spaced almost exactly the width of the
grouping window. Whether two of them survived as two came down to whether the
round trip that wrote them ran a few milliseconds fast or slow: measured over
simulated sessions, about a third of the chain collapsed. A document whose
saves happened closer together lost far more — four versions two seconds apart
became one.
And it is the wrong question to ask of them. Grouping summarises someone's
editing into the moments that are worth listing; there was no editing session
here, only a record of saves that already happened, and the only honest thing
to do with that is to show it one for one.
`system` is now excluded from grouping on both sides. The collaboration server
takes the exclusion as a parameter, which is what the yhub bump in this commit
is for. The panel applies it again and more strictly: the server declines to
merge an excluded entry into the one before it, having already refused on the
author comparison it makes, while the panel merges across authors and so must
refuse in both directions — an edit made moments after a document was migrated
must not absorb the imported entry beside it, or be absorbed by it.
Live editing is unchanged: several edits by one person seconds apart are still
one version.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The version history has been dead since the migration. It listed S3 object
versions of the legacy `{pk}/file` key, and nothing writes that key any more,
so every document's list has been frozen at its migration date; restoring one
was a stub that closed the modal and did nothing, while still promising that
the document would be replaced.
It now reads the collaboration server, which is what keeps the history: the
list comes from `activity`, a selected version is previewed from `changeset`
as the document stood at that moment, and restoring one is a `rollback`.
A version is a minute of editing — changes less than a minute apart become
one, and none spans more than a minute. The collaboration server groups only
changes by the same author, so the browser merges what is left across authors:
a version is a moment in the document, not a moment in one person's editing.
Both are needed, and both use the same rule.
This grants `history.rollback` to editors, which is the first time a browser
may change the past rather than read it, and publishes the rollback route.
A reader is refused it twice over — the collaboration server treats it as a
dead grant without document write access, and the endpoint is withheld as well.
Mutations refuse where reads clamp, so a rollback reaching further back than
the history a user was granted is rejected rather than trimmed: nobody can
undo work that predates their own access, and a rollback with no bound at all
is refused outright. `prune`, which erases, stays granted to nobody. Restoring
is not destructive: it appends a change that undoes another, so what it
replaced stays in the history and can be restored again.
The backend's version endpoints are untouched and now have no caller. They are
marked deprecated with the condition for removing them, since until a document
has been replayed by `migrate_documents` they hold the only record of what it
looked like before it moved.
Also fixes the e2e helper that waited for the removed content endpoint, so it
never returned, and the three version tests that hung behind it.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
yhub 0.8.0 retires the 'r' | 'rw' | null access vocabulary. The auth
plugin now answers a typed permission object stating, facet by facet,
what a subject may do with a document, and yhub enforces every facet
itself - on the websocket and on the REST routes alike. Three rules we
wanted but could not express under the old vocabulary become one-line
facets. Our whole access policy now lives in
src/yhub-server/permissions.js, apart from the server so that it can be
read and tested without standing up redis and postgres.
Read-only users no longer share their cursor #2544. A read-only
connection could still propagate awareness updates to everyone else in
the document, even though its document updates were already dropped.
Presence is now a permission of its own, separate from the right to
edit: a reader receives it and never publishes it. The collaboration
server enforces that rather than trusting the editor to stay quiet, so a
modified or stale client changes nothing. The frontend has to know it
too - the http fallback provider has no receive-only mode, so a reader's
provider is built with no awareness instance at all, or its first PATCH
would take a 403 and close it for good.
The browser is granted only the two routes it uses, the websocket and
ydoc for the http fallback. Everything else - history, rollback, prune,
and every backend-internal endpoint - is refused to it, as is any
endpoint a future release adds, because the grant names no wildcard.
create-ydoc in particular was reachable by any signed-in editor and is
now the backend's alone.
Anonymous visitors are given the userid "anonymous" rather than no
identity at all, which is what lets them keep editing public documents:
yhub refuses the upgrade of a caller that holds the write but cannot be
attributed. Their edits share one author.
Room is DocRef throughout, which is a rename of object keys and not only
of types: the worker event payload and the stream message lookup both
carry it, and both fail silently rather than loudly when missed.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
Document content stopped going through Django when the collaboration
server took it over, and these two are what the move left behind. Both
are unreachable rather than merely unused, so they are removed instead
of being carried forward.
useUpdateDoc sent websocket: true whenever the provider was synced, to
unlock a cache lock the backend used to hold while another user was
connected. That lock is gone - there is no websocket field on the
serializer and nothing reads one - so DRF has been silently dropping the
key. Removing it also drops the store's only isSynced reader.
The service worker still wrote an entry to a doc-content table keyed on
documents/{id}/content/, an endpoint that no longer exists. Nothing ever
read it back: handlerDidErrorRead only ever asks for doc-list and
doc-item. The table is dropped on the next upgrade so it does not linger
in browsers that already have it.
This also clears the five standing typescript errors in the repository,
which were all stale doc-content literals in the tests for the code
being removed.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
Some networks refuse a websocket upgrade - corporate proxies, captive
portals - and a browser is told nothing more than "the connection
closed", so those users could not edit at all. The editor now runs a
second transport next to the socket, polling the collaboration server's
REST api on the same room, with the same session cookie and the same
authorization, and only while the socket is down. Local changes go out
about a second after the last keystroke and remote ones arrive within
ten seconds, so editing works with visibly more latency rather than not
at all. The socket keeps being retried underneath, so a client that fell
back during an outage returns to it on its own, and nothing is lost in
either direction - both transports publish from the same document.
This makes /collaboration/ydoc/ a route browsers call, so
COLLABORATION_SERVER_ORIGIN is now handed to yhub as its cors
configuration and gates the http routes as well as the websocket.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The yhub server returns custom status code when the websocket is not
accessible, like 4401 when an access is removed and 4404 when a document
is deleted. the websocket client now use these custom status code to
stop reconnection forever.
We remove the code related to the content GET and
PATCH endpoints, as they are no longer used in the
codebase. The yhub server will handle the content
management directly, providing the content and
managing the updates. This change simplifies the code
and reduces the complexity of the frontend application.
We will need to reimplement the saving mechanism
in the service worker when we are offline.
Let's wait that the service is fully developed
on the yhub side before we implement this feature.
We will not block anymore the users not connected
to the collaboration server from editing the document,
we will have an HTTP fallback instead, so we can
remove the "can-edit" mechanism and the related code.
Add POST /collaboration/reset-connections/v1/{org}/{docid} (optional
X-User-Id header) to yhub-server. It distributes yhub recheckAuth: every
server re-runs the access check per matching connection and closes only
those whose access actually changed (close code 4401), so unaffected
clients see no reconnect churn. The endpoint authenticates with the
RS256 admin JWT issued by JWTService, verified against the backend JWKS
(new jose dependency); the admin token acts as the "system" user and is
the only principal granted the reset-connections access purpose. The
backend does not trigger it on permission changes yet - that wiring
comes separately, now that CollaborationService is gone.
yhub is upgraded to 0.4.0 and serves every route under the
/collaboration/ prefix (server.apiPrefix): the websocket moves to
/collaboration/ws/v1/docs, and the built-in document apis are meant to
be publicly exposed alongside it, with reset-connections as the one
backend-internal exception.
Also harden websocket auth: fail closed when the backend errors (only a
genuine 401/403 falls back to the anonymous identity, so a signed-in
editor can never hide from a targeted recheck under an anon userid) and
tolerate small clock skew when verifying the cached admin token.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
Reset the background when Backspace converts a callout to a paragraph.
Preserve content, alignment, regular text deletion, and undo behavior.
Fixes#2052
Signed-off-by: Manixhor <manigururam@hotmail.com>
Export Markdown as a ZIP containing the document and same-origin media.
Normalize filenames and restore the export UI after asynchronous failures.
Cover media packaging, modal behavior, filenames, and the browser flow.
Signed-off-by: Gokul Krishna <gokul.06krishna@gmail.com>
Ctrl+Alt+P (Cmd+Option+P on macOS) opens the current document in
presentation mode, saving two clicks through the document menu.
Signed-off-by: Anveet Pal <anveetpal12@gmail.com>
We can now choose to duplicate documents along with
their subdocuments. A confirmation modal will now
appear to let the user choose whether to include
subdocuments in the duplication, only if the document
has subdocuments.
A toast notification will inform the user and
give the possibility to undo the duplication if
needed.
Update lightly the ui to match the new design mockups,
to fit with what was proposed for the duplicate subdocs
feature.
It updates:
- modal closing position
- checkbox style
- toast style
- create a ButtonLink component, a wrapper above
the ui-kit component to fit the Next.js routing system.
We fixed the redirect behavior after deleting a
document to ensure the user stays on the correct page
after deleting a document.
We added a test to avoid regression in the redirect
behavior.
Manuel Raynaud