The offline support couldn't work with the existing
implementation anymore, because there is no request
to get or save data anymore, everything is handled
with web sockets.
In order to support offline functionality, we
leveraged y-indexeddb to store and synchronize
local changes, ensuring that the application remains
functional even when offline.
Version grouping used a hard-coded 60s window, and its
timestamps are minted server-side.
We now expose COLLABORATION_VERSION_GRANULARITY_MS
through /config, read it in useDocActivity.
By doing so, we can control the granularity of
version history with settings, it can be adjusted as
needed. It will help us to test different version
history granularities in our e2e tests.
Every other package in this repository is installed
with yarn; yhub-server was the only one on npm,
with its own package-lock.json.
To ensure consistency it now uses yarn like the rest
of the project.
package-lock.json is replaced by yarn.lock, a packageManager
field is added, and the Dockerfile, CI workflow, Makefile,
helm init-db job and the documentation move from
`npm ci` / `npm run init-db` to
`yarn install --frozen-lockfile` / `yarn init-db`.
The S3 persistence plugin records the version id of the object it wrote
and names that version when it deletes it. On a versioned bucket - what
a deployment runs - a delete that names no version deletes nothing: it
writes a delete marker and keeps every version underneath. Each
compaction supersedes the blobs of the one before, so what was kept was
every version of every document ever written, a document someone asked
to erase included, still readable by anyone who can list versions. On
AWS this needs s3:DeleteObjectVersion, which a policy granting
s3:DeleteObject alone does not cover.
Blobs are written to the bucket for every branch of a document.
YHUB_S3_PERSISTENCE now governs only whether new blobs are written
there. The plugin itself is attached whenever the YHUB_S3_* settings
name a bucket, on or off, because reading is the half that must never
be taken away: a row pointing at an object is unreadable without the
plugin that wrote it, and yhub reports such a version as having no
content rather than as an error. Turning the toggle off stops the
writing and leaves the reading alone; it is the settings, not the
toggle, that a deployment whose bucket holds anything must keep. Half a
configuration is a startup error naming what is missing, as before.
The dev stack keeps the toggle off and creates its bucket versioned, so
flipping it on exercises what a deployment runs rather than a simpler
case. Its createbuckets job needed fixing to do so: the folded yaml
block joins its lines with a space, so the trailing backslashes reached
the shell as an escaped space glued to the next word and everything
past the first && silently did nothing - the media bucket never had
versioning enabled either.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
A document's imported past is attributed to `system`, and until now it went
through the same grouping as ordinary editing. That is wrong twice over.
It loses history. Before the migration the editor saved the whole document on
an interval of exactly 60s, each save becoming one version of the legacy file,
and the migration replays them at their original timestamps — so an imported
history arrives as a chain of entries spaced almost exactly the width of the
grouping window. Whether two of them survived as two came down to whether the
round trip that wrote them ran a few milliseconds fast or slow: measured over
simulated sessions, about a third of the chain collapsed. A document whose
saves happened closer together lost far more — four versions two seconds apart
became one.
And it is the wrong question to ask of them. Grouping summarises someone's
editing into the moments that are worth listing; there was no editing session
here, only a record of saves that already happened, and the only honest thing
to do with that is to show it one for one.
`system` is now excluded from grouping on both sides. The collaboration server
takes the exclusion as a parameter, which is what the yhub bump in this commit
is for. The panel applies it again and more strictly: the server declines to
merge an excluded entry into the one before it, having already refused on the
author comparison it makes, while the panel merges across authors and so must
refuse in both directions — an edit made moments after a document was migrated
must not absorb the imported entry beside it, or be absorbed by it.
Live editing is unchanged: several edits by one person seconds apart are still
one version.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The version history has been dead since the migration. It listed S3 object
versions of the legacy `{pk}/file` key, and nothing writes that key any more,
so every document's list has been frozen at its migration date; restoring one
was a stub that closed the modal and did nothing, while still promising that
the document would be replaced.
It now reads the collaboration server, which is what keeps the history: the
list comes from `activity`, a selected version is previewed from `changeset`
as the document stood at that moment, and restoring one is a `rollback`.
A version is a minute of editing — changes less than a minute apart become
one, and none spans more than a minute. The collaboration server groups only
changes by the same author, so the browser merges what is left across authors:
a version is a moment in the document, not a moment in one person's editing.
Both are needed, and both use the same rule.
This grants `history.rollback` to editors, which is the first time a browser
may change the past rather than read it, and publishes the rollback route.
A reader is refused it twice over — the collaboration server treats it as a
dead grant without document write access, and the endpoint is withheld as well.
Mutations refuse where reads clamp, so a rollback reaching further back than
the history a user was granted is rejected rather than trimmed: nobody can
undo work that predates their own access, and a rollback with no bound at all
is refused outright. `prune`, which erases, stays granted to nobody. Restoring
is not destructive: it appends a change that undoes another, so what it
replaced stays in the history and can be restored again.
The backend's version endpoints are untouched and now have no caller. They are
marked deprecated with the condition for removing them, since until a document
has been replayed by `migrate_documents` they hold the only record of what it
looked like before it moved.
Also fixes the e2e helper that waited for the removed content endpoint, so it
never returned, and the three version tests that hung behind it.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The collaboration server's activity and changeset routes are opened to
the browser, so a document's editing history can be read from where it
actually lives now. What a user may see of it is bounded to the moment
they were given access to the document: joining a document that has been
written for a year does not hand them the year.
That rule is not new. It is the one the version endpoints have always
applied - "only those created after the user got access to the document"
- and the date is the same one: the earliest access the user holds on
the document or on any of its ancestors, so sharing a folder shares its
subtree from that moment. It was computed twice in the backend,
differently, and exposed nowhere. It is now a single annotation,
user_access_since, that the version endpoints and the collaboration
server both read, the latter through the document detail response it
already fetches to authorize a connection.
The bound is applied server-side and silently: a client asks for
whatever range it likes and receives only its own share, so there is no
bound for it to get wrong and none it can widen. It is a stored date
rather than a wall-clock-relative one, which is what keeps it stable
across a websocket re-check, and it is never zero - the one value that
would also unlock a full-history connection.
A reader who reaches a document through its link alone holds no access
and so has no date to bound a history with. They get none, which is why
the backend has always refused them their versions. rollback and prune
stay refused to everyone: restoring a version is a separate decision.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
yhub 0.8.0 retires the 'r' | 'rw' | null access vocabulary. The auth
plugin now answers a typed permission object stating, facet by facet,
what a subject may do with a document, and yhub enforces every facet
itself - on the websocket and on the REST routes alike. Three rules we
wanted but could not express under the old vocabulary become one-line
facets. Our whole access policy now lives in
src/yhub-server/permissions.js, apart from the server so that it can be
read and tested without standing up redis and postgres.
Read-only users no longer share their cursor #2544. A read-only
connection could still propagate awareness updates to everyone else in
the document, even though its document updates were already dropped.
Presence is now a permission of its own, separate from the right to
edit: a reader receives it and never publishes it. The collaboration
server enforces that rather than trusting the editor to stay quiet, so a
modified or stale client changes nothing. The frontend has to know it
too - the http fallback provider has no receive-only mode, so a reader's
provider is built with no awareness instance at all, or its first PATCH
would take a 403 and close it for good.
The browser is granted only the two routes it uses, the websocket and
ydoc for the http fallback. Everything else - history, rollback, prune,
and every backend-internal endpoint - is refused to it, as is any
endpoint a future release adds, because the grant names no wildcard.
create-ydoc in particular was reachable by any signed-in editor and is
now the backend's alone.
Anonymous visitors are given the userid "anonymous" rather than no
identity at all, which is what lets them keep editing public documents:
yhub refuses the upgrade of a caller that holds the write but cannot be
attributed. Their edits share one author.
Room is DocRef throughout, which is a rename of object keys and not only
of types: the worker event payload and the stream message lookup both
carry it, and both fail silently rather than loudly when missed.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
Some networks refuse a websocket upgrade - corporate proxies, captive
portals - and a browser is told nothing more than "the connection
closed", so those users could not edit at all. The editor now runs a
second transport next to the socket, polling the collaboration server's
REST api on the same room, with the same session cookie and the same
authorization, and only while the socket is down. Local changes go out
about a second after the last keystroke and remote ones arrive within
ten seconds, so editing works with visibly more latency rather than not
at all. The socket keeps being retried underneath, so a client that fell
back during an outage returns to it on its own, and nothing is lost in
either direction - both transports publish from the same document.
This makes /collaboration/ydoc/ a route browsers call, so
COLLABORATION_SERVER_ORIGIN is now handed to yhub as its cors
configuration and gates the http routes as well as the websocket.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The new infra we have must be configured in the helm chart. This commit
all the missing templates to deploy yhub, it also automate the creation
of the private keys needed by all services.
The backend application will have to call the yhub REST API for some
operations. We want to use a dedicated service to do that. This first
commit introduces the shape of this service, it only does the
configuration for now, calling actions will be implemented later.
Y_PROVIDER_API_KEY is no longer used in the codebase,
so we can remove it from the helm chart and the documentation.
We adapt the documentation to use the new JWT conversion
mechanism instead.
The CollaborationService was doing nothing since we started the
migration to yhub, all the code using it is now removed. Also the
`can-edit` endpoint and all the safeguard mechanism relying on the
presence of other users connected to the websocket will not be used
anymore, it will be possible to replace all of this with yhub, so all
this code is also removed.
The yhub service will need our public key in order to validate the jwt
token we will used. We choose to expose a jwks endpoint as it is a
standard wat to do this.
We want to configure other options on the redis cache. By default there
is no timeout on the connection to socket and no timeout for read/write
operations. We set default values in all caches used in production. The
settings IGNORE_EXCEPTIONS differ between the default and the session
cache. Activating it behaves like a missed cache. Enabling it for the
session should lead to unwanted side effects, by returning falsy on the
session creation, a retry mechanism of 10000 attempts is made in the
SessionStore.create method, the request can stay in this loop for a long
time.
Release 3.17.2 of DRF now takes care of DATA_UPLOAD_MAX_MEMORY_SIZE
and is checked when the body request is parsed. Before that, DRF wasn't
using it at all and we were only looking for custom settings linked to
the media and conversion file upload. We must now also configure this
setting.
whitenoise middleware is failing a lot with a cancelled exception from
asyncio. Using whitenoise is not needed in our case, we are just serving
an API with django and DRF. We decided to completely remove it.
We have performance issue with the media_auth endpoint. We need to
monitor and profile this endpoint to understand what is happening. We
decided to install and configure django-silk. By default django-silk is
not enabled and not accessible, once enabled migration must be run and
the /silk/ endpoint will be accessible only using a super user.
To have relecant profiles and monitor we must have a database comparable
to a production one because we are facing a scaling issue. Unless a
million documents probably nothing is happening, for this we added
several management commands to help us. A first one to anonymize a
database than can be reused without any risk to leak PII data, can be
called using python manage.py anonymize_database with some mandatory
options. An other command to generate a database profile, this command
should be run on a production database, this profile can then be used by
the generate_volumetry to populate a development database with same
volumetry than describe in the profile.
We have renamed the Chinese locale from `cn_CN`
to `zh_CN` to align with standard locale naming
conventions.
Additionally, we have added support for two new
locales: Esperanto (`eo_PL`) and
Traditional Chinese (`zh_TW`).
We write a documentation explaining how to enable the safe guard
collaboration feature to prevent a user not connected to the websocket
to erase the data coming from users connected to the websocket.
The settings COLLABORATION_WS_NOT_CONNECTED_READY_ONLY contains a typo.
We don't want to see READY_ONLY but READ_ONLY. This commit fix the typo
and change everywhere in the codebase the settings name.
The S3 usage made by docs can be complex. We want to write a
documentation to explain the auth_request nginx module usage and list
known issues with the configuration.
Anthony LC