We want to have a fine grained control over the
Duplicate with Children feature.
By adding the feature flag for Duplicate with Children,
we can enable or disable this feature for specific
users or groups without deploying new code.
This allows us to test the feature in a controlled
environment and gather feedback before a full rollout.
The quay.io/minio/minio and quay.io/minio/mc images now answer 401, and
the minio images are gone from Docker Hub as well: the dev stack, the
backend CI job and the helm dev-backend could not pull them anymore.
[pgsty/silo](https://hub.docker.com/r/pgsty/silo) and
[pgsty/mc](https://hub.docker.com/r/pgsty/mc) are community builds and drop
in replacement of minio. Only the executable name is changing from minio
to silo.
Search results were cmdk items navigating with router.push on select,
so the browser had no link to open elsewhere. Each result is now a real
link to the document: a plain click still goes through the quick search
onSelect, while ctrl, cmd, shift or middle click let the browser open it
in a new tab or window and keep the search modal open.
Signed-off-by: Armand <schneideretcollier.innovation@gmail.com>
After implementing the websocket and api stress tests, this last step is
using playwright to simulate a real user using docs. A real chromium
will be used for this. The idea is to see what a user feels when Docs is
under load.
When metrics is enabled, we want to create a ServiceMonitor od
PodMonitor in order to scrap the metrics from django and yhub. We have
to add an exception on the redirect to ssl for the metrics endpoint,
like for the probes endpoint, the traffic is internal.
To stress test the backend application we will use the k6 project from
grafana. 2 scenarios are implemented and documented in the README. How
to use this project is also documented in the README
standlone package simulating traffic on the yhub server. It reuses the
same stack used bu the frontend application to connect to the websocket.
How to use it is explained in its README file.
We added custom metrics when the monitoring is enabled, we will measure
the cost of calling outside services like yhub and the converters, the
database pool and celery queue.
We don't want to manage a valkey operator and cluster inside the Docs
repo, this will be managed outside. For dev purpose we only need a
standalone instance of valkey.
To test the application on real condition we want to authenticate users.
The authentication is made using OIDC and we don't want to depend on
this during the tests. To bypass it, we will create real user sessions,
export thei identifiers and then use it later with the tool firing the
requests. The session are linked to real users, everything is made in a
dedicated django application and this application is explicitely loaded
in a dedicated environment and can't be used in the Production
environment.
Same we did before with django-prometheus, we want metrics for yhub, the
is no integration available like django-prometheus so everything is
craft in the repository. The documentation is also available in
documentation/metrics.md
We want to export metrics to prometheus. We install for this
django-prometheus. By default the feature is disabled and must be
explicitly enabled. A complete documentation is available in
documentation/metrics.md
We need to use valkey instead of an old redis version. We decided to use
the operator https://github.com/chideat/valkey-operator/ and configure
it in both feature and dev environments.
The offline support couldn't work with the existing
implementation anymore, because there is no request
to get or save data anymore, everything is handled
with web sockets.
In order to support offline functionality, we
leveraged y-indexeddb to store and synchronize
local changes, ensuring that the application remains
functional even when offline.
Version grouping used a hard-coded 60s window, and its
timestamps are minted server-side.
We now expose COLLABORATION_VERSION_GRANULARITY_MS
through /config, read it in useDocActivity.
By doing so, we can control the granularity of
version history with settings, it can be adjusted as
needed. It will help us to test different version
history granularities in our e2e tests.
Every other package in this repository is installed
with yarn; yhub-server was the only one on npm,
with its own package-lock.json.
To ensure consistency it now uses yarn like the rest
of the project.
package-lock.json is replaced by yarn.lock, a packageManager
field is added, and the Dockerfile, CI workflow, Makefile,
helm init-db job and the documentation move from
`npm ci` / `npm run init-db` to
`yarn install --frozen-lockfile` / `yarn init-db`.
The content is not managed anymore in the Document object nor the django
application. The last piece using it was the versioning API and it has
been deleted previously. There is no more reason to keep code related of
the content in the backend application. The only remaining code is the
`file_key` property in the model. It is kept as a safe guard use by the
clean_document management command. If the file is kept on the bucket,
then the document will be reseed with it the first time the document
will be reopen.
Previously we added to retrieve document response a new property
user_access_since that contains the date from when the user started to
have access to the document. The way t was made added an other
annotation to the Document queryset making the sql query more and more
complex. We decided to lighten the queryset and expose the access the
user has on the document instead and read the history from the
created_at property.
yhub 0.8.0 retires the 'r' | 'rw' | null access vocabulary. The auth
plugin now answers a typed permission object stating, facet by facet,
what a subject may do with a document, and yhub enforces every facet
itself - on the websocket and on the REST routes alike. Three rules we
wanted but could not express under the old vocabulary become one-line
facets. Our whole access policy now lives in
src/yhub-server/permissions.js, apart from the server so that it can be
read and tested without standing up redis and postgres.
Read-only users no longer share their cursor #2544. A read-only
connection could still propagate awareness updates to everyone else in
the document, even though its document updates were already dropped.
Presence is now a permission of its own, separate from the right to
edit: a reader receives it and never publishes it. The collaboration
server enforces that rather than trusting the editor to stay quiet, so a
modified or stale client changes nothing. The frontend has to know it
too - the http fallback provider has no receive-only mode, so a reader's
provider is built with no awareness instance at all, or its first PATCH
would take a 403 and close it for good.
The browser is granted only the two routes it uses, the websocket and
ydoc for the http fallback. Everything else - history, rollback, prune,
and every backend-internal endpoint - is refused to it, as is any
endpoint a future release adds, because the grant names no wildcard.
create-ydoc in particular was reachable by any signed-in editor and is
now the backend's alone.
Anonymous visitors are given the userid "anonymous" rather than no
identity at all, which is what lets them keep editing public documents:
yhub refuses the upgrade of a caller that holds the write but cannot be
attributed. Their edits share one author.
Room is DocRef throughout, which is a rename of object keys and not only
of types: the worker event payload and the stream message lookup both
carry it, and both fail silently rather than loudly when missed.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The yhub server returns custom status code when the websocket is not
accessible, like 4401 when an access is removed and 4404 when a document
is deleted. the websocket client now use these custom status code to
stop reconnection forever.
notify the backend when the worker persists new content for
a document, so the lists ordered by `updated_at` follow the edits made on the
collaboration server. The backend serves it on
`POST /api/v1.0/documents/{id}/content-updated/`, authenticated with a short
lived RS256 JWT the collaboration server signs (`aud: "docs-backend"`) and
the backend verifies against the JWKS the collaboration server publishes on
`/collaboration/jwks/v1` — the mirror of the admin token the backend signs to
call it, so no long lived secret is shared and either side can roll its key
on its own
The formatted-content endpoint was using the `document.content` to fetch
the ydoc from s3, we want to move from this usage to using yhub to
retrieve the content, so yhub is becoming our source of thruth.
Both its PATCH and its GET: the content of a document is saved and
served by the collaboration server. The `content_patch` and
`content_retrieve` abilities go with it.
`GET /collaboration/get-ydoc/v1/docs/{id}` answers the current Yjs state of a
document as a raw binary update, the read counterpart of create-ydoc, and
204 when the document has no content yet
When a new Docs is created and a file is sent, as before we convert it
first and we need to use the raw content to seed it by calling the
create-ydoc api in the YHub service.
When an access change or is deleted or a link configuration changes, we
call the yhub server to reset connections and remove them if needed. The
YHubService is used for this.
The backend application will have to call the yhub REST API for some
operations. We want to use a dedicated service to do that. This first
commit introduces the shape of this service, it only does the
configuration for now, calling actions will be implemented later.
The yhub image was build only for a production usage. In development we
want to have a hot reload when a file is modified. For this the
Dockerfile is modified, the nodemon package install in dev environment
and used to watch modification against the source files.
yhub verified Django's RS256 admin JWT without checking "aud", so the
y-converter token Django hands to the converter process was replayable
here — and admin: true short-circuits getAccessType to "rw" on every
document, plus the backend-internal reset-connections purpose and the
X-User-Id attribution override. Require aud: "yhub", as y-provider
already does for its own audience. Nothing in the backend calls yhub's
admin endpoints yet, so no caller is affected.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
With SOFT_MIGRATION=true, the first access to a document yhub does not
know yet fetches the legacy snapshot from Django's S3 media bucket
({id}/file, UTF-8 base64 of a raw Yjs update), seeds the room through
the compute pool - attributed to "system" with a migration=s3 custom
attribution - and only then admits the connection, so the initial sync
always includes the seed. Now that the frontend no longer bootstraps
rooms client-side (content GET/PATCH removal), this is the only path
that brings legacy content into yhub; keep the flag on until a batch
backfill has migrated the full corpus.
A missing S3 object is the brand-new-document case and yields an empty
room; every real failure fails closed (opaque 401, y-websocket retries
with backoff). Existence is probed postgres-first (bare SELECT, then
the valkey stream, then the SELECT again to close the compaction
race). Guard rails: a per-docid verdict cache (poison objects cannot
sustain an S3 retry storm, transient errors expire in 15s, per-replica
seed backpressure denies once without caching), in-flight dedup, a
token-owned cross-replica valkey lock released by compare-and-delete,
a 10s S3 fetch timeout that also destroys a late-arriving response
stream, and the same 10MiB decoded cap as create-ydoc. Concurrent
seeds stay correct regardless: the frozen snapshots share one Yjs
lineage, so duplicates merge as CRDT no-ops.
Also reject non-lowercase docids (Django serializes UUIDs lowercase; a
case variant would open a parallel room and miss its S3 object) and
refuse to boot when AWS_S3_ENDPOINT_URL carries a path the minio
client cannot address. On AWS the read-only credentials must include
s3:ListBucket so a missing object surfaces as NoSuchKey rather than
AccessDenied - see the README for the full guarantees and ops notes.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
Python cannot call yhub's built-in PATCH ydoc api because its body must
be lib0-any encoded - a lib0-specific binary framing with no
implementation outside javascript. The new endpoint
POST /collaboration/create-ydoc/v1/{org}/{docid} accepts the raw binary
Yjs update (pycrdt get_update() / Y.encodeStateAsUpdate output) as
application/octet-stream, so Django can seed a document's initial state
with a plain requests.post(url, data=raw_bytes) - needed by the
server-side creation flows (file import, create-for-owner, duplication,
template instantiation) whose yhub rooms currently stay empty until the
first browser connects.
Strict create semantics: 409 when the room already has content
(checked via getDoc, covering persisted state and uncompacted stream
messages; yhub has no atomic create, concurrent creates merge via CRDT
and never corrupt). The initial content is attributed to the optional
X-User-Id header, else to the caller's identity. Access uses the
default purpose, i.e. standard document write access like the built-in
ydoc routes: the admin JWT, or a user session with update ability.
Malformed updates map to 400 (the compute worker rejects them and the
pool replaces the thread), empty updates to 400, bodies over 10MiB to
413.
Gotcha worth noting: req.bytes() resolves to a Node Buffer, but yhub's
compute-task schema validates with lib0's exact-constructor Uint8Array
check, so the body is re-viewed as a plain Uint8Array before it is
handed to the compute pool.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
Add POST /collaboration/reset-connections/v1/{org}/{docid} (optional
X-User-Id header) to yhub-server. It distributes yhub recheckAuth: every
server re-runs the access check per matching connection and closes only
those whose access actually changed (close code 4401), so unaffected
clients see no reconnect churn. The endpoint authenticates with the
RS256 admin JWT issued by JWTService, verified against the backend JWKS
(new jose dependency); the admin token acts as the "system" user and is
the only principal granted the reset-connections access purpose. The
backend does not trigger it on permission changes yet - that wiring
comes separately, now that CollaborationService is gone.
yhub is upgraded to 0.4.0 and serves every route under the
/collaboration/ prefix (server.apiPrefix): the websocket moves to
/collaboration/ws/v1/docs, and the built-in document apis are meant to
be publicly exposed alongside it, with reset-connections as the one
backend-internal exception.
Also harden websocket auth: fail closed when the backend errors (only a
genuine 401/403 falls back to the anonymous identity, so a signed-in
editor can never hide from a targeted recheck under an anon userid) and
tolerate small clock skew when verifying the cached admin token.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The CollaborationService was doing nothing since we started the
migration to yhub, all the code using it is now removed. Also the
`can-edit` endpoint and all the safeguard mechanism relying on the
presence of other users connected to the websocket will not be used
anymore, it will be possible to replace all of this with yhub, so all
this code is also removed.
The yhub service will need our public key in order to validate the jwt
token we will used. We choose to expose a jwks endpoint as it is a
standard wat to do this.
Reset the background when Backspace converts a callout to a paragraph.
Preserve content, alignment, regular text deletion, and undo behavior.
Fixes#2052
Signed-off-by: Manixhor <manigururam@hotmail.com>
Export Markdown as a ZIP containing the document and same-origin media.
Normalize filenames and restore the export UI after asynchronous failures.
Cover media packaging, modal behavior, filenames, and the browser flow.
Signed-off-by: Gokul Krishna <gokul.06krishna@gmail.com>
We can now choose to duplicate documents along with
their subdocuments. A confirmation modal will now
appear to let the user choose whether to include
subdocuments in the duplication, only if the document
has subdocuments.
A toast notification will inform the user and
give the possibility to undo the duplication if
needed.
