The yhub database schema have new update and will probably be modified
in the future. We don't want to maintain this sql schema in the Docs
repo, we want to reuse what is directly made in the yhub project. For
this we reuse the existing bin/init-db.js script
Cover both paths off the legacy Django store end to end: the lazy seed on
first access, and the migrate endpoint replaying every S3 version. The tests
need no database — the admin JWT short-circuits document authorization, so a
fixture is an S3 object on a random uuid — and read the timeline through
yhub 0.5.0's `Accept: application/json`, which spares python a lib0 decoder.
CI grows a valkey service and starts a collaboration server alongside the
backend test job; the tests skip themselves when nothing answers on the new
COLLABORATION_API_URL setting, so `make test` without the dev stack still
passes.
Writing them turned up three things worth fixing in the server.
Backend reads now seed too. getAccessType short-circuited on the admin token
before reaching the legacy store, so a server-side read of an unmigrated
document answered with an empty one, and a create-ydoc against it would have
written a second lineage beside the content the first user access was about
to seed in.
Seeding no longer decides access; the backend's answer alone does. A legacy
object that cannot be migrated — it does not decode, or it exceeds the size
we load — opens as a new document instead of denying, since no retry can fix
it and refusing would leave the document unopenable by anyone. The cause is
logged once per attempt with the bucket, key and stack, and every later access
logs that it admitted a caller without migrating.
That made the failure classifier dangerous, so it is inverted. It was an
allowlist of retryable errors — eight socket errnos — which left every way S3
can refuse (AccessDenied on a rotated key, NoSuchBucket, a region redirect)
counting as "this object is unusable". Denying, that was survivable; opening
empty, one misscoped credential would fork every document touched during the
window. Now only a failure raised while interpreting bytes we already hold is
permanent, marked at the throw site, and everything else answers a retryable
503. Guessing wrong that way costs a retry; the other way costs the document.
The admin seed is also fenced to the org and to main, like the user path
above it. The legacy store is branchless — {docid}/file is main — and the
bookkeeping is per document, so seeding ?branch=draft would have written
main's content into an orphan room and left the real one permanently empty.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
Add POST /collaboration/migrate/v1/docs/{id}, which replays every S3 version
of a document's legacy `{id}/file` object into one gc:false Yjs document and
stores it as a single row at clock 0, crediting each version with its own S3
timestamp. Nothing existing is deleted and nothing goes on the stream, so the
next compaction merges that row like any other. The clock-0 insert is ON
CONFLICT DO NOTHING and migrated ids are kept in a valkey set, so the endpoint
is idempotent without a lock. The activity api then reports the same timeline
as the backend's /documents/{id}/versions/, instead of the single
migration-time change the lazy soft migration leaves behind.
That lazy seed now writes no insertAt/deleteAt. Persisted contentmaps are
merged rather than de-duplicated, so a seed timestamp would survive next to
the real per-version one on the same ids and the activity api would report
whichever the unordered row scan put last. A seed is not an editing event and
has no honest time to report.
Upgrade yhub to 0.5.0, where error codes encode retry semantics (4xx
permanent, 5xx and 429 retryable) and auth plugins may throw apiError(503). A
temporarily unreachable Django backend, JWKS endpoint or legacy S3 store is
now reported as 503 rather than denied like a permission failure, so clients
retry instead of giving up.
The legacy-store code moves out of server.js into migration.js, with the
shared *_FILE secret helper in env.js.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
yhub verified Django's RS256 admin JWT without checking "aud", so the
y-converter token Django hands to the converter process was replayable
here — and admin: true short-circuits getAccessType to "rw" on every
document, plus the backend-internal reset-connections purpose and the
X-User-Id attribution override. Require aud: "yhub", as y-provider
already does for its own audience. Nothing in the backend calls yhub's
admin endpoints yet, so no caller is affected.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
With SOFT_MIGRATION=true, the first access to a document yhub does not
know yet fetches the legacy snapshot from Django's S3 media bucket
({id}/file, UTF-8 base64 of a raw Yjs update), seeds the room through
the compute pool - attributed to "system" with a migration=s3 custom
attribution - and only then admits the connection, so the initial sync
always includes the seed. Now that the frontend no longer bootstraps
rooms client-side (content GET/PATCH removal), this is the only path
that brings legacy content into yhub; keep the flag on until a batch
backfill has migrated the full corpus.
A missing S3 object is the brand-new-document case and yields an empty
room; every real failure fails closed (opaque 401, y-websocket retries
with backoff). Existence is probed postgres-first (bare SELECT, then
the valkey stream, then the SELECT again to close the compaction
race). Guard rails: a per-docid verdict cache (poison objects cannot
sustain an S3 retry storm, transient errors expire in 15s, per-replica
seed backpressure denies once without caching), in-flight dedup, a
token-owned cross-replica valkey lock released by compare-and-delete,
a 10s S3 fetch timeout that also destroys a late-arriving response
stream, and the same 10MiB decoded cap as create-ydoc. Concurrent
seeds stay correct regardless: the frozen snapshots share one Yjs
lineage, so duplicates merge as CRDT no-ops.
Also reject non-lowercase docids (Django serializes UUIDs lowercase; a
case variant would open a parallel room and miss its S3 object) and
refuse to boot when AWS_S3_ENDPOINT_URL carries a path the minio
client cannot address. On AWS the read-only credentials must include
s3:ListBucket so a missing object surfaces as NoSuchKey rather than
AccessDenied - see the README for the full guarantees and ops notes.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The Playwright MCP browser session writes snapshots and console logs
into .playwright-mcp/ at the repository root while driving the app
during development — keep them out of version control.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
We remove the code related to the content GET and
PATCH endpoints, as they are no longer used in the
codebase. The yhub server will handle the content
management directly, providing the content and
managing the updates. This change simplifies the code
and reduces the complexity of the frontend application.
We will need to reimplement the saving mechanism
in the service worker when we are offline.
Let's wait that the service is fully developed
on the yhub side before we implement this feature.
We will not block anymore the users not connected
to the collaboration server from editing the document,
we will have an HTTP fallback instead, so we can
remove the "can-edit" mechanism and the related code.
Address the findings of an adversarial review of the new endpoint:
- Only the backend admin token may attribute content to another user via
the X-User-Id header. The endpoint uses the default access purpose, so
any editor with update ability can call it — honoring the header for
them would let an editor forge the attribution history of the first
revision (the websocket path likewise stamps the server-side
identity). Regular callers now always author as themselves; verified:
an editor session posting X-User-Id gets its own userid stamped.
- Reject non-main ?branch= requests (400). Cookie users are main-only
via getAccessType, but the admin token bypasses it and could seed an
orphan (org, docid, branch) room no user-facing path reads — while
dodging the branch-scoped 409 existence check.
- Correct the concurrent-create comment: two racing creates merge as
independently generated updates (fresh clientIDs), so the seeded
content appears twice — user-visible duplication, not merely a
doubly-attributed revision. Still accepted (Django creates each doc
once and a duplicated seed is user-fixable), but the tradeoff is now
stated accurately.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
Python cannot call yhub's built-in PATCH ydoc api because its body must
be lib0-any encoded - a lib0-specific binary framing with no
implementation outside javascript. The new endpoint
POST /collaboration/create-ydoc/v1/{org}/{docid} accepts the raw binary
Yjs update (pycrdt get_update() / Y.encodeStateAsUpdate output) as
application/octet-stream, so Django can seed a document's initial state
with a plain requests.post(url, data=raw_bytes) - needed by the
server-side creation flows (file import, create-for-owner, duplication,
template instantiation) whose yhub rooms currently stay empty until the
first browser connects.
Strict create semantics: 409 when the room already has content
(checked via getDoc, covering persisted state and uncompacted stream
messages; yhub has no atomic create, concurrent creates merge via CRDT
and never corrupt). The initial content is attributed to the optional
X-User-Id header, else to the caller's identity. Access uses the
default purpose, i.e. standard document write access like the built-in
ydoc routes: the admin JWT, or a user session with update ability.
Malformed updates map to 400 (the compute worker rejects them and the
pool replaces the thread), empty updates to 400, bodies over 10MiB to
413.
Gotcha worth noting: req.bytes() resolves to a Node Buffer, but yhub's
compute-task schema validates with lib0's exact-constructor Uint8Array
check, so the body is re-viewed as a plain Uint8Array before it is
handed to the compute pool.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
Y_PROVIDER_API_KEY is no longer used in the codebase,
so we can remove it from the helm chart and the documentation.
We adapt the documentation to use the new JWT conversion
mechanism instead.
The /api/convert route no longer accepts the Y_PROVIDER_API_KEY shared
secret. It now verifies the admin JWT signed by Django against the
JWKS published on its /api/v1.0/jwks endpoint.
Add POST /collaboration/reset-connections/v1/{org}/{docid} (optional
X-User-Id header) to yhub-server. It distributes yhub recheckAuth: every
server re-runs the access check per matching connection and closes only
those whose access actually changed (close code 4401), so unaffected
clients see no reconnect churn. The endpoint authenticates with the
RS256 admin JWT issued by JWTService, verified against the backend JWKS
(new jose dependency); the admin token acts as the "system" user and is
the only principal granted the reset-connections access purpose. The
backend does not trigger it on permission changes yet - that wiring
comes separately, now that CollaborationService is gone.
yhub is upgraded to 0.4.0 and serves every route under the
/collaboration/ prefix (server.apiPrefix): the websocket moves to
/collaboration/ws/v1/docs, and the built-in document apis are meant to
be publicly exposed alongside it, with reset-connections as the one
backend-internal exception.
Also harden websocket auth: fail closed when the backend errors (only a
genuine 401/403 falls back to the anonymous identity, so a signed-in
editor can never hide from a targeted recheck under an anon userid) and
tolerate small clock skew when verifying the cached admin token.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The CollaborationService was doing nothing since we started the
migration to yhub, all the code using it is now removed. Also the
`can-edit` endpoint and all the safeguard mechanism relying on the
presence of other users connected to the websocket will not be used
anymore, it will be possible to replace all of this with yhub, so all
this code is also removed.
For now the only token we will need is ont with the admin claim set to
True. To not repeat the creation of this token again and again, we
created a dedicated method to issue this token in the JWTService class.
The yhub service will need our public key in order to validate the jwt
token we will used. We choose to expose a jwks endpoint as it is a
standard wat to do this.
Render the slides off-screen as A4 landscape pages and print them via
the browser, one slide per page with the watermark. Add a "Download PDF"
action to the floating bar.
Closes#2446
Release 3.17.2 of DRF now takes care of DATA_UPLOAD_MAX_MEMORY_SIZE
and is checked when the body request is parsed. Before that, DRF wasn't
using it at all and we were only looking for custom settings linked to
the media and conversion file upload. We must now also configure this
setting.
Leave was shown in the document menu for anonymous visitors.
Hide it unless the current user can actually leave the doc.
Signed-off-by: Mora <razvi.morariu.mihai@gmail.com>
Added:
- ✨(frontend) Add "Copy link to block" feature
- ✨(frontend) add word count to doc header toolbox
- ✨(frontend) add find and replace feature to the editor
Changed:
- ♿️(frontend) use anchor links for interlinking sub-documents
- ✨(frontend) reset side panel state between documents
- ♿️(frontend) announce search loading state for screen readers
- ♻️(frontend) change favorite to star
- 🚚(frontend) add doc move to doc options
- ♻️(frontend) unified menu
- ♿(frontend) hide decorative emojis in document titles from SR
- ♻️(frontend) save the doc with a keepalive request when
leaving the page
Fixed:
- 🐛(frontend) fix clipped formatting toolbar in new comment
composer
- 🐛(backend) fix duplicating a document that has no content
- 📄(frontend) allowed partially export when MIT
- 🐛(backend) manage async support for Docs custom middleware
Removed:
- 🔥(backend) remove whitenoise package
Implementation of code block throws a errors when a `language` prop
is not supported by BlockNote, which crashes the whole editor.
In our case, the language is supported but flagged as an alias
(e.g. `js` instead of `javascript`),
but Blocknote does not resolve aliases and throws an error.
This file wraps the code block spec to normalize the `language`
prop before BlockNote renders it,
so a single legacy code block does not crash the whole editor.
Blocknote changed the way the side menu is positioned.
Because we are modifying the height of the heading blocks
we need to adjust the offset of the side menu to match
the new height of the heading blocks.
To do so we have to override the default offset of
the side menu with a custom offset that takes into
account the new height of the heading blocks.
Last version of Blocknote added support for math
and diagram blocks. This commit updates the
dependencies to include the necessary toolbar
items to create math and diagram blocks in
the editor.
We bumped Blocknote to 0.54.0, which introduced
some breaking changes. This commit adapts our
codebase to the new API and ensures compatibility
with the latest version of Blocknote.
Since we fix the error in the backend application coming from middleware
not managing async request lifecycle, the e2e tests were failing. This
is because the browser cancel the request when made in the beforeunload
event. To fix it, we set the keepalive property on the fetch method to
True to not abort the request when the page is unloaded.
whitenoise middleware is failing a lot with a cancelled exception from
asyncio. Using whitenoise is not needed in our case, we are just serving
an API with django and DRF. We decided to completely remove it.
Docs have 2 custom middlewares, both are only managing sync
requests. With Python 3.13 we didn't have any errors, but
since we upgraded to Python 3.14, we have a CancelledError
exception. We decided to use the MiddlewareMixin from Django
that is sync and async capable and will be responsible for
executing both middleware in the good mode.
We need to invalidate the react-query cache during
some actions that modify the tree, invalidating
the react-query cache was not enough, we also needed
to reset the tree state to null, so that the tree with
react-arborist would be reloaded with the new data.
We now binded the reloadTree function with the
react-query cache invalidation, so that when we
invalidate the cache, automatically the tree state
is reset to null and the tree is reloaded with the
new data. No need to call reloadTree manually anymore.
We had different and duplicated accessibility behaviors
between the tree root and the subpages.
This PR unifies the behaviors and improves
accessibility of the doc tree.
We can now:
- directly focus on the tree then navigate
through the tree with the arrows keyboard
- When using F2, we focus on the actions, click
on escape to go back to the tree and continue navigating
- When using F2, and we arrives at the last action,
another F2 will go back to the start of the actions
- When using F2, you can then use the arrows to navigate
through the actions
Depend the part of the app, we had different implementations
of the toolbox menu. This commit unifies the implementation
and uses the same component for both the docs grid, the doc tree
and the doc header.
It will make it easier to maintain and add new
features to the toolbox menu.
Document.content reads from object storage and returns None when nothing
was ever written there. That None, raised "content should be a string.",
so the duplicate endpoint answered a 500. Default to an empty string instead.
Signed-off-by: BOUKERFA Mohamed El Amine <boukerfa.ma@gmail.com>
In order to harmonize the doc options, we are adding
the "Move to my docs" and "Move into a doc" options
to the doc options menu.
This allows users to move documents directly from
the options menu, providing a more streamlined
experience.
We have added a new feature to the editor that allows
users to find and replace text within their documents.
This feature enhances the editing experience by
providing a convenient way to search for specific
words or phrases and replace them with new content.
