The offline support couldn't work with the existing
implementation anymore, because there is no request
to get or save data anymore, everything is handled
with web sockets.
In order to support offline functionality, we
leveraged y-indexeddb to store and synchronize
local changes, ensuring that the application remains
functional even when offline.
Version grouping used a hard-coded 60s window, and its
timestamps are minted server-side.
We now expose COLLABORATION_VERSION_GRANULARITY_MS
through /config, read it in useDocActivity.
By doing so, we can control the granularity of
version history with settings, it can be adjusted as
needed. It will help us to test different version
history granularities in our e2e tests.
The content is not managed anymore in the Document object nor the django
application. The last piece using it was the versioning API and it has
been deleted previously. There is no more reason to keep code related of
the content in the backend application. The only remaining code is the
`file_key` property in the model. It is kept as a safe guard use by the
clean_document management command. If the file is kept on the bucket,
then the document will be reseed with it the first time the document
will be reopen.
Previously we added to retrieve document response a new property
user_access_since that contains the date from when the user started to
have access to the document. The way t was made added an other
annotation to the Document queryset making the sql query more and more
complex. We decided to lighten the queryset and expose the access the
user has on the document instead and read the history from the
created_at property.
The version history has been dead since the migration. It listed S3 object
versions of the legacy `{pk}/file` key, and nothing writes that key any more,
so every document's list has been frozen at its migration date; restoring one
was a stub that closed the modal and did nothing, while still promising that
the document would be replaced.
It now reads the collaboration server, which is what keeps the history: the
list comes from `activity`, a selected version is previewed from `changeset`
as the document stood at that moment, and restoring one is a `rollback`.
A version is a minute of editing — changes less than a minute apart become
one, and none spans more than a minute. The collaboration server groups only
changes by the same author, so the browser merges what is left across authors:
a version is a moment in the document, not a moment in one person's editing.
Both are needed, and both use the same rule.
This grants `history.rollback` to editors, which is the first time a browser
may change the past rather than read it, and publishes the rollback route.
A reader is refused it twice over — the collaboration server treats it as a
dead grant without document write access, and the endpoint is withheld as well.
Mutations refuse where reads clamp, so a rollback reaching further back than
the history a user was granted is rejected rather than trimmed: nobody can
undo work that predates their own access, and a rollback with no bound at all
is refused outright. `prune`, which erases, stays granted to nobody. Restoring
is not destructive: it appends a change that undoes another, so what it
replaced stays in the history and can be restored again.
The backend's version endpoints are untouched and now have no caller. They are
marked deprecated with the condition for removing them, since until a document
has been replayed by `migrate_documents` they hold the only record of what it
looked like before it moved.
Also fixes the e2e helper that waited for the removed content endpoint, so it
never returned, and the three version tests that hung behind it.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The collaboration server's activity and changeset routes are opened to
the browser, so a document's editing history can be read from where it
actually lives now. What a user may see of it is bounded to the moment
they were given access to the document: joining a document that has been
written for a year does not hand them the year.
That rule is not new. It is the one the version endpoints have always
applied - "only those created after the user got access to the document"
- and the date is the same one: the earliest access the user holds on
the document or on any of its ancestors, so sharing a folder shares its
subtree from that moment. It was computed twice in the backend,
differently, and exposed nowhere. It is now a single annotation,
user_access_since, that the version endpoints and the collaboration
server both read, the latter through the document detail response it
already fetches to authorize a connection.
The bound is applied server-side and silently: a client asks for
whatever range it likes and receives only its own share, so there is no
bound for it to get wrong and none it can widen. It is a stored date
rather than a wall-clock-relative one, which is what keeps it stable
across a websocket re-check, and it is never zero - the one value that
would also unlock a full-history connection.
A reader who reaches a document through its link alone holds no access
and so has no date to bound a history with. They get none, which is why
the backend has always refused them their versions. rollback and prune
stay refused to everyone: restoring a version is a separate decision.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The management command migrating document to yhub didn't allow to target
a specific document. This can be usefull for debugging purpose but also
to replay the migration of a specific document.
The clean_document command makes a reset of a document deleting its
content and all the attachments linked to this subdocument and its
children. The hard delete api in yhub make the room, so the document id,
not usable at all and this is not not what we want. We added a new
custom api in yhub to manage this case, the document is hard deleted and
then the Tombstone to make the room reusable again.
yhub is the source of truth, when a user delete a document, it should
also be deleted in the yhub server. We call the yhub server in the
perform_destroy action but also the restore endpoint of yhub when a
document is restored.
command replaying the legacy content of
the documents into the collaboration server, one call to its migrate endpoint
per document. Resumable and safe to re-run: what became of every document is
recorded (`impress_document_migration`), a server that is unwell is retried
with a backoff and a document it refuses is left for a later run
(`--retry-failed`). Bounded by `--concurrency`, `--rate` and `--limit`, most
recently edited documents first
After removing most of the usage of S3 in the tests, these ones are
faster and make some flakyness more relevant. For example, in tests
related to the external api we have to reload the urls based on the
settings. We now have some race conditions where tests collapsed and
urls are not correctly reloaded.
Tehe DocumentFactory was always creating a content and this content was
saved on S3. This leads to the creation of huge amount of content in the
S3 storage but not necesseraly used in the tests. In order to keep the
refactor to remove the usage of content from document.content but from
Yhub service, this content is no more generated. It is kept for part of
the code not yet refactor like the versionning feature.
The version 0.5.0 can manage response format by using accept and
content-type headers. In python we can't use for now the lib0 decoder so
we have to use the json format. When the lib0 decoder will be available
in pycrdt we will use it. So we can now use directly the /ydoc api to
fetch a document content instead the custom api made for this.
Duplicate the onboarding sandbox document through the
collaboration server: its content is read from there and copied under the
identity of the user the sandbox is created for. A collaboration server that
cannot be reached skips the sandbox, as a missing template already did, and
never fails the signup
the search indexer reads it with `YHubService`, and the indexation
of an edited document is triggered by the `content-updated` call
the collaboration server makes — nothing else sees the content change
anymore.
It is queued as a celery task, throttled like the other updates, so
no indexation ever runs in the process serving the request.
A document whose content cannot be read is left out of the batch
rather than indexed empty, which would have erased it from the search
backend
notify the backend when the worker persists new content for
a document, so the lists ordered by `updated_at` follow the edits made on the
collaboration server. The backend serves it on
`POST /api/v1.0/documents/{id}/content-updated/`, authenticated with a short
lived RS256 JWT the collaboration server signs (`aud: "docs-backend"`) and
the backend verifies against the JWKS the collaboration server publishes on
`/collaboration/jwks/v1` — the mirror of the admin token the backend signs to
call it, so no long lived secret is shared and either side can roll its key
on its own
The formatted-content endpoint was using the `document.content` to fetch
the ydoc from s3, we want to move from this usage to using yhub to
retrieve the content, so yhub is becoming our source of thruth.
Both its PATCH and its GET: the content of a document is saved and
served by the collaboration server. The `content_patch` and
`content_retrieve` abilities go with it.
`GET /collaboration/get-ydoc/v1/docs/{id}` answers the current Yjs state of a
document as a raw binary update, the read counterpart of create-ydoc, and
204 when the document has no content yet
When a new Docs is created and a file is sent, as before we convert it
first and we need to use the raw content to seed it by calling the
create-ydoc api in the YHub service.
When an access change or is deleted or a link configuration changes, we
call the yhub server to reset connections and remove them if needed. The
YHubService is used for this.
To ease the use of the audience with the JWTService, we choose to create
an enum holding all the possible values and then use them in the Yhub
and Y-converter services.
The backend application will have to call the yhub REST API for some
operations. We want to use a dedicated service to do that. This first
commit introduces the shape of this service, it only does the
configuration for now, calling actions will be implemented later.
Cover both paths off the legacy Django store end to end: the lazy seed on
first access, and the migrate endpoint replaying every S3 version. The tests
need no database — the admin JWT short-circuits document authorization, so a
fixture is an S3 object on a random uuid — and read the timeline through
yhub 0.5.0's `Accept: application/json`, which spares python a lib0 decoder.
CI grows a valkey service and starts a collaboration server alongside the
backend test job; the tests skip themselves when nothing answers on the new
COLLABORATION_API_URL setting, so `make test` without the dev stack still
passes.
Writing them turned up three things worth fixing in the server.
Backend reads now seed too. getAccessType short-circuited on the admin token
before reaching the legacy store, so a server-side read of an unmigrated
document answered with an empty one, and a create-ydoc against it would have
written a second lineage beside the content the first user access was about
to seed in.
Seeding no longer decides access; the backend's answer alone does. A legacy
object that cannot be migrated — it does not decode, or it exceeds the size
we load — opens as a new document instead of denying, since no retry can fix
it and refusing would leave the document unopenable by anyone. The cause is
logged once per attempt with the bucket, key and stack, and every later access
logs that it admitted a caller without migrating.
That made the failure classifier dangerous, so it is inverted. It was an
allowlist of retryable errors — eight socket errnos — which left every way S3
can refuse (AccessDenied on a rotated key, NoSuchBucket, a region redirect)
counting as "this object is unusable". Denying, that was survivable; opening
empty, one misscoped credential would fork every document touched during the
window. Now only a failure raised while interpreting bytes we already hold is
permanent, marked at the throw site, and everything else answers a retryable
503. Guessing wrong that way costs a retry; the other way costs the document.
The admin seed is also fenced to the org and to main, like the user path
above it. The legacy store is branchless — {docid}/file is main — and the
bookkeeping is per document, so seeding ?branch=draft would have written
main's content into an orphan room and left the real one permanently empty.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The CollaborationService was doing nothing since we started the
migration to yhub, all the code using it is now removed. Also the
`can-edit` endpoint and all the safeguard mechanism relying on the
presence of other users connected to the websocket will not be used
anymore, it will be possible to replace all of this with yhub, so all
this code is also removed.
For now the only token we will need is ont with the admin claim set to
True. To not repeat the creation of this token again and again, we
created a dedicated method to issue this token in the JWTService class.
The yhub service will need our public key in order to validate the jwt
token we will used. We choose to expose a jwks endpoint as it is a
standard wat to do this.
The duplicate endpoint used to respond with only `{"id": ...}`. It now
returns the complete duplicated document representation, consistent
with the other document detail endpoints, so the frontend doesn't have
to make a follow-up request to get the new document's data.
This required setting `is_favorite` explicitly on the duplicated
document before serializing it: it is normally set by the
`annotate_is_favorite` queryset method, which the newly created
document never goes through. Being a read-only serializer field, it
was silently dropped from the response instead of raising an error. A
document can't be a favorite right after being created, so it is set
to `False` directly.
Dropping a file larger than the allowed size showed a bare "unknown
error" in the editor. The proxy in front of the API cuts the request
and answers a 413 with an HTML body, so errorCauses threw while
parsing it as JSON and no cause ever reached the error panel.
The size limit the backend already enforces is now exposed by the
config endpoint, and the editor checks the file against it before
sending anything, with the same toast wording the document import
uses. errorCauses no longer throws on a body it cannot parse, and a
413 without a usable cause falls back to an explicit message, which
covers the instances whose proxy limit is lower than the application
one.
The size formatting duplicated in the import hook moved to a shared
util.
Signed-off-by: risk-alt <aldu6974@gmail.com>
The nb_accesses cache TTL was very short, 30 seconds. That mean that the
user will hit the cache for a very short period and the cache is
probably not be hit. This is what we can see in the slow queries from
the pg_stat_statements table. The query to compute the nb_accesses is
executed a little bit less than the number of queries to list or
retrieve documents, meaning the cache is not used.
The filtering made in the DocumentViewset.get_queryset method is not
optimal and lead to a full scan of the Document table. The heavy part is
on the filtering on what the user can access between the accesses and
the link traces. To have better performance we make an union operation
of both document_id list and the filter the id on this list. Postgresql
will use the index on the id column.
Comparing path with LEFT(value, LENGTH(path)) makes a sequential scan on
all the Document table, the more this table grow, the more the query
using it will be slow. We dediced instead to lookup on the path
extracting all ancestors path for a given document and then make a path
IN statement to use the index existing on the path column.
The readiness probe should also not create a new session. A new session
will live in redis and increase the number of keys inside it for
nothing. The readiness path is isgnored in the ForceSessionMiddleware
The ForceSessionMiddleware force the session creation, we want to
ignore it when the request is the liveness probe. The liveness probe
must not check if redis is available, this is the readiness probe job
Docs have 2 custom middlewares, both are only managing sync
requests. With Python 3.13 we didn't have any errors, but
since we upgraded to Python 3.14, we have a CancelledError
exception. We decided to use the MiddlewareMixin from Django
that is sync and async capable and will be responsible for
executing both middleware in the good mode.
Document.content reads from object storage and returns None when nothing
was ever written there. That None, raised "content should be a string.",
so the duplicate endpoint answered a 500. Default to an empty string instead.
Signed-off-by: BOUKERFA Mohamed El Amine <boukerfa.ma@gmail.com>
Once the sql queries improved we have still a bottleneck on large
concurrent requests on this endpoint. We notive in the profiles generated
that lot of time was spent in creating a new s3 client instance on each
request. django_storage use a thread local cache for signed and unsigned
connection, but using uvicorn we have a new thread for each request, so
on each request a new s3 client is generated and it appears to be an
expensive operation. To fix this issue, we cache the client and share it
accross all the thread and requests.
On the media_auth endpoint the first bottleneck we have is with
postgresql. We are looking for too much data and no index is used on the
attachments colum. When the lookup filter on the attachement columns, a
full scan is made on all the document table looking for each element in
the array, this operation is really expensive. To fix this we created a
GIN index on the attachments column. Also the readable_per_se lookup was
selecting too much data combined with the filter_descendants function.
We remove the usage of the filter_descendants, we choose to first fetch
all the paths where the attachment is found, this operation is fast
thanks to the new index, split all the paths in candidate paths and then
filter readable_per_se queryset with these paths. All these
modifications make the endpoint faster.
We have performance issue with the media_auth endpoint. We need to
monitor and profile this endpoint to understand what is happening. We
decided to install and configure django-silk. By default django-silk is
not enabled and not accessible, once enabled migration must be run and
the /silk/ endpoint will be accessible only using a super user.
To have relecant profiles and monitor we must have a database comparable
to a production one because we are facing a scaling issue. Unless a
million documents probably nothing is happening, for this we added
several management commands to help us. A first one to anonymize a
database than can be reused without any risk to leak PII data, can be
called using python manage.py anonymize_database with some mandatory
options. An other command to generate a database profile, this command
should be run on a production database, this profile can then be used by
the generate_volumetry to populate a development database with same
volumetry than describe in the profile.
Object storage metadata keys are case insensitive per the S3
specification, but implementations don't agree on the case they give
back. When head_object returns a capitalized "Status", updating the
attachment status added a second, lowercase entry instead of replacing
it, and the copy request ended up carrying two x-amz-meta-status
headers. Ceph RadosGW loses one of them behind a proxy, which
invalidates the request signature.
The same assumption was made when reading the status back in media-auth
and media-check, where an attachment stored on such a backend stayed in
"processing" forever.
Metadata read from the storage is now normalized to lowercase keys
before being consumed or copied over.
Signed-off-by: risk-alt <aldu6974@gmail.com>
Since pydantic-ai has ben upgraded deprecation warning were present in
the logs. This commint change how the instrumentation is enabled to be
compatible with hoe pydantic-ai manage it. Also prompt system are
ignored when they come from the front, like we are doing and dropped. To
maintain the system prompt we use the instruction parameter instead.
We upgraded pydantic-ai-slim to the latest version and add mistral has
extra dependency. Doing this, the direct dependance to mistral can be
removed and will be managed with pidantic-ai-slim package. The import of
mistralai has changed, we have to updagre them from mistralai to
mistralai.client
