Move the mention email off the request/response cycle into a Celery task
so the SMTP round-trip no longer delays the API response.
Signed-off-by: BOUKERFA Mohamed El Amine <boukerfa.ma@gmail.com>
Allow users with at least the commenter role on a document to mention
other collaborators in the document body or in a comment thread via
POST /documents/{id}/mention/. The mentioned user must already have
access to the document.
The mention record is always created, and the mentioned user is
notified by email with a link to the anchor, unless a notification
was already sent in the same context (document body or specific
thread) within a configurable cooldown period (15 minutes by default).
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
Return all accesses on the document and its ancestors to any
user with access, keeping the limited user details serializer
and adding the user id to it, so that collaborators allowed
to comment can identify and mention each other.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
We can now paste docs url that will become interlinking.
So we don't want to pass titles through the inline content itself
anymore, to not leak any titles. The problem is that inline content
Inline content mappers are not async, so titles must be
resolved upfront.
We adapted the export interlinking to get titles from
a pre-resolved map instead of relying on inline content.
Pasting a bare link to another doc on the same domain now inserts an
interlink instead of a plain hyperlink, so it benefits from the same
title-sync and navigation behaviour as links added through the `/` or
`@` menu. Pasting over a selection, inside a code block, or a link the
current user can't resolve will display the doc id instead.
We want to introduce a feature flag that allows
duplicating documents along with their children when
enabled. We will be able to control not only
frontend side but also backend behavior
regarding document with children duplication.
We want to have a fine grained control over the
Duplicate with Children feature.
By adding the feature flag for Duplicate with Children,
we can enable or disable this feature for specific
users or groups without deploying new code.
This allows us to test the feature in a controlled
environment and gather feedback before a full rollout.
The quay.io/minio/minio and quay.io/minio/mc images now answer 401, and
the minio images are gone from Docker Hub as well: the dev stack, the
backend CI job and the helm dev-backend could not pull them anymore.
[pgsty/silo](https://hub.docker.com/r/pgsty/silo) and
[pgsty/mc](https://hub.docker.com/r/pgsty/mc) are community builds and drop
in replacement of minio. Only the executable name is changing from minio
to silo.
Search results were cmdk items navigating with router.push on select,
so the browser had no link to open elsewhere. Each result is now a real
link to the document: a plain click still goes through the quick search
onSelect, while ctrl, cmd, shift or middle click let the browser open it
in a new tab or window and keep the search modal open.
Signed-off-by: Armand <schneideretcollier.innovation@gmail.com>
After implementing the websocket and api stress tests, this last step is
using playwright to simulate a real user using docs. A real chromium
will be used for this. The idea is to see what a user feels when Docs is
under load.
When metrics is enabled, we want to create a ServiceMonitor od
PodMonitor in order to scrap the metrics from django and yhub. We have
to add an exception on the redirect to ssl for the metrics endpoint,
like for the probes endpoint, the traffic is internal.
To stress test the backend application we will use the k6 project from
grafana. 2 scenarios are implemented and documented in the README. How
to use this project is also documented in the README
standlone package simulating traffic on the yhub server. It reuses the
same stack used bu the frontend application to connect to the websocket.
How to use it is explained in its README file.
We added custom metrics when the monitoring is enabled, we will measure
the cost of calling outside services like yhub and the converters, the
database pool and celery queue.
We don't want to manage a valkey operator and cluster inside the Docs
repo, this will be managed outside. For dev purpose we only need a
standalone instance of valkey.
To test the application on real condition we want to authenticate users.
The authentication is made using OIDC and we don't want to depend on
this during the tests. To bypass it, we will create real user sessions,
export thei identifiers and then use it later with the tool firing the
requests. The session are linked to real users, everything is made in a
dedicated django application and this application is explicitely loaded
in a dedicated environment and can't be used in the Production
environment.
Same we did before with django-prometheus, we want metrics for yhub, the
is no integration available like django-prometheus so everything is
craft in the repository. The documentation is also available in
documentation/metrics.md
We want to export metrics to prometheus. We install for this
django-prometheus. By default the feature is disabled and must be
explicitly enabled. A complete documentation is available in
documentation/metrics.md
We need to use valkey instead of an old redis version. We decided to use
the operator https://github.com/chideat/valkey-operator/ and configure
it in both feature and dev environments.
we have to import @y/y dependency, we use it in the migration part of
the application and @y/hub is also using it. This can lead to a
situation where 2 versions of @y/y are installed, this is not something
we want.
The offline support couldn't work with the existing
implementation anymore, because there is no request
to get or save data anymore, everything is handled
with web sockets.
In order to support offline functionality, we
leveraged y-indexeddb to store and synchronize
local changes, ensuring that the application remains
functional even when offline.
Version grouping used a hard-coded 60s window, and its
timestamps are minted server-side.
We now expose COLLABORATION_VERSION_GRANULARITY_MS
through /config, read it in useDocActivity.
By doing so, we can control the granularity of
version history with settings, it can be adjusted as
needed. It will help us to test different version
history granularities in our e2e tests.
Divide server.ts into 3 focused modules:
- config.ts: every environment variable the server reads, parsed and
validated at import time,
- backend.ts: the calls made to the Docs Django backend, the token
signing and the JWKS both ends verify each other with,
- api.ts: the custom REST endpoints mounted under /collaboration/.
server.ts keeps only what wires these together: the auth plugin and its
legacy-store seed (which closes over the yhub instance), the persistence
plugins, the worker events and the createYHub call.
The AppAuthInfo identity shape moves to yhub.ts so server.ts and api.ts
can share it without importing from each other.
Tests follow the split: config.spec.ts and backend.spec.ts are new,
server.spec.ts narrows to the composition root's boot contract. No
behaviour change.
Convert the yhub-server source and tests from plain JS/mjs to
TypeScript: move sources under src/, add tsconfig, tsconfig.build,
eslint flat config and nodemon config, and update the Dockerfile,
compose and CI workflow for the build step.
Every other package in this repository is installed
with yarn; yhub-server was the only one on npm,
with its own package-lock.json.
To ensure consistency it now uses yarn like the rest
of the project.
package-lock.json is replaced by yarn.lock, a packageManager
field is added, and the Dockerfile, CI workflow, Makefile,
helm init-db job and the documentation move from
`npm ci` / `npm run init-db` to
`yarn install --frozen-lockfile` / `yarn init-db`.
We use vitest in most of our applications,
to ensure consistency and maintainability across our
codebase we are now using vitest for yhub-server as well.
We added more tests on yhub-server.
The tests cover migration, permissions, and server
functionality. We also added a helper file for test
utilities.
The content is not managed anymore in the Document object nor the django
application. The last piece using it was the versioning API and it has
been deleted previously. There is no more reason to keep code related of
the content in the backend application. The only remaining code is the
`file_key` property in the model. It is kept as a safe guard use by the
clean_document management command. If the file is kept on the bucket,
then the document will be reseed with it the first time the document
will be reopen.
Previously we added to retrieve document response a new property
user_access_since that contains the date from when the user started to
have access to the document. The way t was made added an other
annotation to the Document queryset making the sql query more and more
complex. We decided to lighten the queryset and expose the access the
user has on the document instead and read the history from the
created_at property.
The yhub database ran under its own yhub role. Every other database in
this stack uses the backend's dev credentials, dinum/pass, which is also
what CI already uses for this database; only the database name stays
apart, since it is a separate database on a separate server.
The url the collaboration server connects with moves into the same file,
beside the credentials it repeats, written once rather than twice — the
yhub service reads the file as well. A yhub-postgres.local override is
created like for the other env files, and the healthcheck asks about
POSTGRES_DB and POSTGRES_USER like the other two do instead of naming the
role.
POSTGRES_USER only takes effect on a fresh initdb, so an existing stack
needs its yhub-pgdata volume dropped and `make migrate-yhub` re-run; the
volume holds local document state only.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The S3 persistence plugin records the version id of the object it wrote
and names that version when it deletes it. On a versioned bucket - what
a deployment runs - a delete that names no version deletes nothing: it
writes a delete marker and keeps every version underneath. Each
compaction supersedes the blobs of the one before, so what was kept was
every version of every document ever written, a document someone asked
to erase included, still readable by anyone who can list versions. On
AWS this needs s3:DeleteObjectVersion, which a policy granting
s3:DeleteObject alone does not cover.
Blobs are written to the bucket for every branch of a document.
YHUB_S3_PERSISTENCE now governs only whether new blobs are written
there. The plugin itself is attached whenever the YHUB_S3_* settings
name a bucket, on or off, because reading is the half that must never
be taken away: a row pointing at an object is unreadable without the
plugin that wrote it, and yhub reports such a version as having no
content rather than as an error. Turning the toggle off stops the
writing and leaves the reading alone; it is the settings, not the
toggle, that a deployment whose bucket holds anything must keep. Half a
configuration is a startup error naming what is missing, as before.
The dev stack keeps the toggle off and creates its bucket versioned, so
flipping it on exercises what a deployment runs rather than a simpler
case. Its createbuckets job needed fixing to do so: the folded yaml
block joins its lines with a space, so the trailing backslashes reached
the shell as an escaped space glued to the next word and everything
past the first && silently did nothing - the media bucket never had
versioning enabled either.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The check was dropped during the yhub migration, on the grounds that
`reset_connections` was a no-op. It is not one any more: the backend calls
yhub's reset-connections route, which re-runs the authorization of each
connection and closes the sockets whose access is no longer the one they
were opened with, with the code 4401.
The old check cannot come back as it was. It had the author of the document
change its visibility and waited for the author's own socket to close, which
is what the collaboration server used to do - close every connection to the
document, whatever it was that changed. yhub leaves alone the connections
that are unaffected, and an owner's access is the same before and after, so
the author's socket is now deliberately kept.
The client to watch is therefore another one: the anonymous reader holding
the public link, while the author switches that link from reading to
editing. Their socket goes down and the client comes back on its own.
That the new connection is live is asserted by what flows through it rather
than by the socket being open - the author writes after the change and the
text reaches the reader, with nothing reloaded in between.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The export regression tests injected their fixture by intercepting
`GET /documents/{id}/content/`, and the 401 routing test intercepted the
`PATCH .../content/` a document used to send on the way out. Both endpoints
went with the yhub migration, so the interceptions matched nothing: the
exported document was whatever the two images added to an empty page, and
the 401 page was never reached.
The fixture now goes to the collaboration server, which is the only thing
that holds document content. It is seeded before the editor mounts: its
first block carries the id BlockNote gives the empty paragraph it writes
into any document it opens empty, so seeding an already open document would
leave a stray paragraph in front of the fixture and shift every page. The
reference PDFs are unchanged and still match, page count, text and pixels.
The 401 test reloads rather than navigating to another document, nothing
being sent on the way out any more.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
A document's imported past is attributed to `system`, and until now it went
through the same grouping as ordinary editing. That is wrong twice over.
It loses history. Before the migration the editor saved the whole document on
an interval of exactly 60s, each save becoming one version of the legacy file,
and the migration replays them at their original timestamps — so an imported
history arrives as a chain of entries spaced almost exactly the width of the
grouping window. Whether two of them survived as two came down to whether the
round trip that wrote them ran a few milliseconds fast or slow: measured over
simulated sessions, about a third of the chain collapsed. A document whose
saves happened closer together lost far more — four versions two seconds apart
became one.
And it is the wrong question to ask of them. Grouping summarises someone's
editing into the moments that are worth listing; there was no editing session
here, only a record of saves that already happened, and the only honest thing
to do with that is to show it one for one.
`system` is now excluded from grouping on both sides. The collaboration server
takes the exclusion as a parameter, which is what the yhub bump in this commit
is for. The panel applies it again and more strictly: the server declines to
merge an excluded entry into the one before it, having already refused on the
author comparison it makes, while the panel merges across authors and so must
refuse in both directions — an edit made moments after a document was migrated
must not absorb the imported entry beside it, or be absorbed by it.
Live editing is unchanged: several edits by one person seconds apart are still
one version.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The version history has been dead since the migration. It listed S3 object
versions of the legacy `{pk}/file` key, and nothing writes that key any more,
so every document's list has been frozen at its migration date; restoring one
was a stub that closed the modal and did nothing, while still promising that
the document would be replaced.
It now reads the collaboration server, which is what keeps the history: the
list comes from `activity`, a selected version is previewed from `changeset`
as the document stood at that moment, and restoring one is a `rollback`.
A version is a minute of editing — changes less than a minute apart become
one, and none spans more than a minute. The collaboration server groups only
changes by the same author, so the browser merges what is left across authors:
a version is a moment in the document, not a moment in one person's editing.
Both are needed, and both use the same rule.
This grants `history.rollback` to editors, which is the first time a browser
may change the past rather than read it, and publishes the rollback route.
A reader is refused it twice over — the collaboration server treats it as a
dead grant without document write access, and the endpoint is withheld as well.
Mutations refuse where reads clamp, so a rollback reaching further back than
the history a user was granted is rejected rather than trimmed: nobody can
undo work that predates their own access, and a rollback with no bound at all
is refused outright. `prune`, which erases, stays granted to nobody. Restoring
is not destructive: it appends a change that undoes another, so what it
replaced stays in the history and can be restored again.
The backend's version endpoints are untouched and now have no caller. They are
marked deprecated with the condition for removing them, since until a document
has been replayed by `migrate_documents` they hold the only record of what it
looked like before it moved.
Also fixes the e2e helper that waited for the removed content endpoint, so it
never returned, and the three version tests that hung behind it.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
