When docs is ran using an ASGI server, the django StreamingHttpResponse
expect to consume an async generator. If a sync generator is used, it
fully consumes it and then return all the content and we loose the
benefits of using a StreamingHttpResponse.
When all the tests are ran, there is a PytestWarning log present to warn
us that there is remaining database connection open and can not be
closed. It appears that the tests about concurrence are responsible of
leaving connection open. We need to manually close them in these tests.
We want to allow or not the usage of the /documents/all/ endpoint. It has
been created to be used for an other purpose than the js client. For
those who don't use it this new settings allow to disable it. By default
it is set to True to keep the same behavior.
When an AI action is called from ai_translate, ai_transform and
ai_proxy, an event doc_ai_action is captured with a method property
giving the information from where it is called.
The posthog information were only available using the POSTHOG_KEY
settings. We split it in POSTHOG_KEY and POSTHOG_HOST to use them with
the pyhton posthog sdk
We want to allow users to leave a document where they have an access or
they have visited creating a link_trace. All subdocuments should also be
leaved at the same time.
To know if the user can leave a doc we have to check when computing the
abilities if a record is existing in the LinkTrace table. This is a N+1
query situation. To avoid it, we added an annotation in the
DocumentQueryset like we already do to annotate the user role.
There is one edge case where the annotation is made to soon, it is when the
user is visiting a document for the first time, the `get_object` add the
annotation and in the permission, we compute the abilities. The `leave`
property is False because the entry in the LinkTrace table is not made,
when the serializer ask for the abilities again, it is still False. So
in the `retrieve` method in the viewset we force the
`user_has_link_trace` to the correct value.
The mask feature has never been implemented by the frontend application
and it is not what we want to do. We want to implement a feature to
leave a document.
In the model method User::_handle_onboarding_documents_access we do not
allow using documents with restricted link_reach to be added as
onboarding documents. To check the real link_reach of the document, we
must use instead the computed_link_reach to be sure that a sub document
can also be used and compute its correct link_reach.
In the search response, we want to display a breadcrumb for every
document returned. For this we added a "parents" property containing a list
documents, all are the parents of the current document (ordered by their
depth). With this we can easily create a breadcrumb.
The global search endpoint filtering by title in the database was not
searching for accessible sub documents. We change the how global search
is made: it is now returning accessible sub documents matching the
title query.
Admins/owners could overwrite another user's comment text and silently
take ownership of it.
Split write permissions: destroy stays broad (author + admin/owner) for
moderation.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
When a document is moved outside its current permission scope (root
document, cross-tree move, or promotion to root), its direct accesses
and pending invitations are now deleted server-side within the same
atomic transaction as the move itself. This ensures consistency: if
the move fails, deletions are rolled back.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
The children/ endpoint was missing file upload support that the root
documents endpoint already had. Added file-to-YJS conversion handling to
subdocument creation.
We observe some cases where the frontend and
backend versions can get out of sync, which can
cause issues.
To mitigate this, we want to implement a mechanism
that detects when the frontend and backend
versions are mismatched and triggers a
reload of the application to ensure they are in sync.
We have situation where the number of locks in the database can increase
dangerously creating deadlock situation. To remove this situation we
decided to change the strategy to manage document creation concurrency.
We decided to use a retry strategy, trying to create the document
multiple times while a usable path is found. To avoid having an
inifinite loop, we use a max_attempts counter configurable using the
setting TREEBEARD_PATH_COMPUTE_RETRY_MAX_ATTEMPTS
We need to split the core/utils.py in multiple submodule created in
core/utils/*.py. We need to do this to avoir circular import between
this module and the models module.
When a PATCH and a GET on the content endpoint are made at the same time
for different users a race condition can happen and the metadata
returned
by the S3 head_object can be outdated when the object is fetched leading
to an error raised because the Content-Length header does not match the
size of the response body. To avoid this, we no longer used head_object
followed bu get_object, we have to manage
everything in one call with the get_object. The get_object also accepts
as parameters an etag or last-modified header and will return a 304 if
the content has not changed, so we can use this to not return the entire
body if this one has not changed.
The return of the legacy AI translate prompt is sometimes surrounded
with a html code block. We improve the prompt to not return this
surrounding code block.
We also want to use the mistral sdk with the legacy AI feature when this
one is configured with the settings. In order to separate bot feature,
they now live in their own module.
We give the possibility, for the new ai feature, to choose between using
the OpenAI or Mistral sdk. For instances having access to the mistral
infrastructure, using it is mor appropriated than using the openai
compatible chat model.
One test about the search descendants test was flaky. It is because the
link_reach and link_role were used to test the
ancestors_link_(reach|role). The properties ancestors_link_reach and
ancestors_link_role should be used instead.
Catch RuntimeError raised by Document.restore() and translate it into a
DRF ValidationError so callers get a 400 instead of a 500, when trying
to restore a non-deleted document.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
When attempting to move a document to itself or to any of its
descendants, the server would crash with a 500 Internal Server
Error.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
When exporting a document to PDF, having whitespace before or after
the media URL causes the image to not be downloaded via the CORS proxy,
resulting in missing images in the exported PDF.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
Validate emojis in ReactionSerializer (previously accepted
any string), preventing multiple emojis or text uploads in
a single reaction
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
In deployment, Traefik is used, not nginx, as an ingress. Traefik
uses `X-Forwarded-Ur`i instead of `X-Original-Url`. This adds a setting
which lets users adapt Docs to their ingress proxy of choice
The settings name is MEDIA_AUTH_ORIGINAL_URL_HEADER
Signed-off-by: Erin Shepherd <erin.shepherd@e43.eu>
We want to give to the js client the ability to use some headers to
avoid fetching a content it already have. For this, the content endpoint
will return an ETag and Last-Modified headers corresponding to the file
content ETag and its last modification. For future fetch, the client can
use the If-None-Match or If-Modified-Since request headers, if one of
these headers are satisfied, the endpoint will return a 304 response. If
not it will still return a 200
The content was always loaded in the document reponse. We remove this
behavior in order to not make an http call to the s3 storage. To get the
document content it is needed now to use the new endpoint dedicated to
retrive the document content.
We want a dedicated endpoint to update a document content. Previously,
updating the content was made on the update action shared with all other
document's properties. When the title is updated, the response contains
the content, so a call to the s3 storage is made and we don't want this.
Isolating the content update will allow us in the next commit to remove
the content from the Document serializer.
The endpoint /api/v1.0/documents/{document_id}/content/ has been renamed
in /api/v1.0/documents/{document_id}/formatted-content/. formatted-content
seems more accurante and the content endpoint will be used for another
purpose more appropriated.
Summary
- Bump docspec Docker image from `2.6.3` to `3.0.0` and adapt
`DocSpecConverter` to the new API (raw body upload with explicit
`Content-Type`/`Accept` headers instead of multipart form)
Important
**The Docker image (`ghcr.io/docspecio/api:3.0.0`) must be updated
alongside the code changes.** The new request format is incompatible
with v2.x — deploying only the code without updating the image (or vice
versa) will break document conversion.
