When a document is moved outside its current permission scope (root
document, cross-tree move, or promotion to root), its direct accesses
and pending invitations are now deleted server-side within the same
atomic transaction as the move itself. This ensures consistency: if
the move fails, deletions are rolled back.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
The children/ endpoint was missing file upload support that the root
documents endpoint already had. Added file-to-YJS conversion handling to
subdocument creation.
We observe some cases where the frontend and
backend versions can get out of sync, which can
cause issues.
To mitigate this, we want to implement a mechanism
that detects when the frontend and backend
versions are mismatched and triggers a
reload of the application to ensure they are in sync.
We have situation where the number of locks in the database can increase
dangerously creating deadlock situation. To remove this situation we
decided to change the strategy to manage document creation concurrency.
We decided to use a retry strategy, trying to create the document
multiple times while a usable path is found. To avoid having an
inifinite loop, we use a max_attempts counter configurable using the
setting TREEBEARD_PATH_COMPUTE_RETRY_MAX_ATTEMPTS
We need to split the core/utils.py in multiple submodule created in
core/utils/*.py. We need to do this to avoir circular import between
this module and the models module.
When a PATCH and a GET on the content endpoint are made at the same time
for different users a race condition can happen and the metadata
returned
by the S3 head_object can be outdated when the object is fetched leading
to an error raised because the Content-Length header does not match the
size of the response body. To avoid this, we no longer used head_object
followed bu get_object, we have to manage
everything in one call with the get_object. The get_object also accepts
as parameters an etag or last-modified header and will return a 304 if
the content has not changed, so we can use this to not return the entire
body if this one has not changed.
The return of the legacy AI translate prompt is sometimes surrounded
with a html code block. We improve the prompt to not return this
surrounding code block.
We also want to use the mistral sdk with the legacy AI feature when this
one is configured with the settings. In order to separate bot feature,
they now live in their own module.
We give the possibility, for the new ai feature, to choose between using
the OpenAI or Mistral sdk. For instances having access to the mistral
infrastructure, using it is mor appropriated than using the openai
compatible chat model.
One test about the search descendants test was flaky. It is because the
link_reach and link_role were used to test the
ancestors_link_(reach|role). The properties ancestors_link_reach and
ancestors_link_role should be used instead.
Catch RuntimeError raised by Document.restore() and translate it into a
DRF ValidationError so callers get a 400 instead of a 500, when trying
to restore a non-deleted document.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
When attempting to move a document to itself or to any of its
descendants, the server would crash with a 500 Internal Server
Error.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
When exporting a document to PDF, having whitespace before or after
the media URL causes the image to not be downloaded via the CORS proxy,
resulting in missing images in the exported PDF.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
Validate emojis in ReactionSerializer (previously accepted
any string), preventing multiple emojis or text uploads in
a single reaction
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
In deployment, Traefik is used, not nginx, as an ingress. Traefik
uses `X-Forwarded-Ur`i instead of `X-Original-Url`. This adds a setting
which lets users adapt Docs to their ingress proxy of choice
The settings name is MEDIA_AUTH_ORIGINAL_URL_HEADER
Signed-off-by: Erin Shepherd <erin.shepherd@e43.eu>
We want to give to the js client the ability to use some headers to
avoid fetching a content it already have. For this, the content endpoint
will return an ETag and Last-Modified headers corresponding to the file
content ETag and its last modification. For future fetch, the client can
use the If-None-Match or If-Modified-Since request headers, if one of
these headers are satisfied, the endpoint will return a 304 response. If
not it will still return a 200
The content was always loaded in the document reponse. We remove this
behavior in order to not make an http call to the s3 storage. To get the
document content it is needed now to use the new endpoint dedicated to
retrive the document content.
We want a dedicated endpoint to update a document content. Previously,
updating the content was made on the update action shared with all other
document's properties. When the title is updated, the response contains
the content, so a call to the s3 storage is made and we don't want this.
Isolating the content update will allow us in the next commit to remove
the content from the Document serializer.
The endpoint /api/v1.0/documents/{document_id}/content/ has been renamed
in /api/v1.0/documents/{document_id}/formatted-content/. formatted-content
seems more accurante and the content endpoint will be used for another
purpose more appropriated.
Summary
- Bump docspec Docker image from `2.6.3` to `3.0.0` and adapt
`DocSpecConverter` to the new API (raw body upload with explicit
`Content-Type`/`Accept` headers instead of multipart form)
Important
**The Docker image (`ghcr.io/docspecio/api:3.0.0`) must be updated
alongside the code changes.** The new request format is incompatible
with v2.x — deploying only the code without updating the image (or vice
versa) will break document conversion.
The exception block was never being executed because URLValidator raises
django.core.exceptions.ValidationError, not
drf.exceptions.ValidationError, so the except block was dead code.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
We add the User Accesses before saving content so the user is sure to
have access to the the first version when creating a doc through
create_for_owner (fixes#2123)
We want to control the conversion of document at upload time. We want to
disable this feature using a settings. The new settings
CONVERSION_UPLOAD_ENABLED should be used to enable or not the conversion
at upload feature. If disabled and a file is uploaded, the reponse will
return a 400
Add test_api_document_favorite_list_with_deleted_child to verify favorite_list
endpoint does not include deleted sub documents
Signed-off-by: Paul Vernin <paul.vernin@gmail.com>
The frontend application is making PATCH request with an empty body.
This PATCH request is not making any change but an UPDATE sql query is
made, the `updated_at` field is the only one updated. When can skip this
save in the databse by returning the Document instance in the serializer
update method
No tests were made using the PATCH method to update a Document using the
API. The frontend appllication mostly use the patch method instead of
the PUT method.
The search endpoint was using the refresh_roken method decorator. This
decorator force having a valid refresh token stored in the session for
the entire viewset. The search endpoint still allow having the legacy
search behavior and for this we don't need to configure at all the OIDC
refrsh mechanism.
When a document is duplicated, it is duplicated at the direct right of
the duplicated document. Doing this force to move all the other
documents at the right, if it is duplicated at the root this can impact
a lot of documents, create lot of locks in the database. If the process
is stop for any reason then the paths can be in an inconsistent paths in
the Document table
In a past release we added a feature to create a sandbox document to a
newly created used. To create this sandbox document, we duplicate an
existing document and this duplicate is using the add_sibling method
with the "right" agument on this original document. Adding a sibling at
the right to a document involve moving right every root document created
after the original document, so the path of all this documents are
recalculated and changed. This can lead to the lost of some leaf in a
tree because to do this operation, multiple locks are created on the
database, creating lot of connection to the database and if the max
number connection to the database is reached or if the memory allocated
by the database is too hight, the database can close all connections
leading to inconsistent paths in the Document table.
## Purpose
For beta testing purposes we need to be able to activate Find hybrid
search to some users, Find full-text search to some others and leave
remaining users on basic DRF title search.
## Proposal
The solution proposed is based on [django-waffle
](https://waffle.readthedocs.io/en/stable/types/flag.html).
- [x] install waffle and activate the default app in settings.
- [x] implement `_get_search_type` in `DocumentViewset` to determine
which search type (title, hybrid or full-text) to use.
- [x] send the `search_type` in the search query.
## External contributions
Thank you for your contribution! 🎉
Please ensure the following items are checked before submitting your
pull request:
- [x] I have read and followed the [contributing
guidelines](https://github.com/suitenumerique/docs/blob/main/CONTRIBUTING.md)
- [x] I have read and agreed to the [Code of
Conduct](https://github.com/suitenumerique/docs/blob/main/CODE_OF_CONDUCT.md)
- [x] I have signed off my commits with `git commit --signoff` (DCO
compliance)
- [x] I have signed my commits with my SSH or GPG key (`git commit -S`)
- [x] My commit messages follow the required format: `<gitmoji>(type)
title description`
- [x] I have added a changelog entry under `## [Unreleased]` section (if
noticeable change)
- [x] I have added corresponding tests for new features or bug fixes (if
applicable)
---------
Signed-off-by: charles <charles.englebert@protonmail.com>
## Purpose
integrate Find to Docs
## Proposal
- [x] ✨ add a `useSeachDocs` hook in charged of calling the search
endpoint.
- [x] ✨ add a optional `path` param to the `search` route. This param
represents the parent document path in case of a sub-documents
(descendants) search.
- [x] ⚡️return Indexer results directly without DB calls to retrieve the
Document objects. All informations necessary for display are indexed in
Find. We can skip the DB calls and improve performance.
- [x] ♻️ refactor react `DocSearchContent` components.
`DocSearchContent` and `DocSearchSubContent` are now merged a unique
component handling all search scenarios and relying on the unique
`search` route.
- [x] 🔥remove pagination logic in the Indexer. Removing the DB calls
also removes the DRF queryset object which handles the pagination. Also
we consider pagination not to be necessary for search v1.
- [x] 🔥remove the `document/<document_id>/descendants` route. This route
is not used anymore. The logic of finding the descendants are moved to
the internal `_list_descendants` method. This method is based on the
parent `path` instead of the parent `id` which has some consequence
about the user access management. Relying on the path prevents the use
of the `self.get_object()` method which used to handle the user access
logic.
- [x] ✨handle fallback logic on DRF based title search in case of
non-configured, badly configured or failing at run time indexer.
- [x] ✨handle language extension in `title` field. Find returns titles
with a language extension (ex: `{ title.fr: "rapport d'activité" }`
instead of `{ "title": "rapport d'activité" }`.
- [x] 🔧 add a `common.test` file to allow running the tests without
docker
- [x] ♻️ rename `SearchIndexer` -> `FindDocumentIndexer`. This class has
to do with Find in particular and the convention is more coherent with
`BaseDocumentIndexer`
- [x] ♻️ rename `SEARCH_INDEXER_URL` -> `INDEXING_URL` and
`SEARCH_INDEXER_QUERY_URL` -> `SEARCH_URL`. I found the original names
very confusing.
- [x] 🔧 update the environment variables to activate the
FindDocumentIndexer.
- [x] ✨automate the generation of encryption key during bootstrap.
OIDC_STORE_REFRESH_TOKEN_KEY is a mandatory secret key. We can not push
it on Github and we want any contributor to be able to run the app by
only running the `make bootstrap`. We chose to generate and wright it
into the `common.local` during bootstrap.
## External contributions
Thank you for your contribution! 🎉
Please ensure the following items are checked before submitting your
pull request:
- [x] I have read and followed the [contributing
guidelines](https://github.com/suitenumerique/docs/blob/main/CONTRIBUTING.md)
- [x] I have read and agreed to the [Code of
Conduct](https://github.com/suitenumerique/docs/blob/main/CODE_OF_CONDUCT.md)
- [x] I have signed off my commits with `git commit --signoff` (DCO
compliance)
- [x] I have signed my commits with my SSH or GPG key (`git commit -S`)
- [x] My commit messages follow the required format: `<gitmoji>(type)
title description`
- [x] I have added a changelog entry under `## [Unreleased]` section (if
noticeable change)
- [x] I have added corresponding tests for new features or bug fixes (if
applicable)
---------
Signed-off-by: charles <charles.englebert@protonmail.com>
The psycopg pool config was enabled by default forcing its usage. Using
psycopg pool can be difficult, finding the good configuration take time.
By default its usage should be disable and the maintainer of the
instance should decide to enable it or not.
We change the strategy on how the new users have access to the
onboarding documents. We should remove all created accesses we don't
want to have anymore. There is no need to add them in the link_trace
table, they are already present in the favorites and user have already
access to it.
When a user is created and a sandbox document should be created, we can
have a race condition on the document creation leading to an error for
the user. To avoid this we have to manage this part in a transaction and
locking the document table
When a user is created, we created accesses to a list of onboarding
documents. Doing this have side effect on the proximity search feature.
Instead of creating access, we should create link_reach
Backend part of #1796.
This changes allows to display an onboarding modal the first time that
the get_me() API view is called.
I originally tried to check if `User.last_login` was `None`, but it is
updated as soon as the user is logged, so I chose to create a flag on
the model.
Privileged users of a document (owners and admins) should not be
allowed to create an access request on a document they already control.
Without aguardrail, they could inadvertently inflate the access
request queue with redundant entries.
More critically, if an owner submits an access request on their own
document, another admin could accept it and grant them a lower role
(e.g. reader), which would silently strip them of their ownership.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
The Blocknote AI feature is a bit flaky, we want
to be able to disable it if to much issues arise,
without having to do a new release.
We add a bunch of feature flags to be able to
disable the AI features if needed:
- add AI_FEATURE_BLOCKNOTE_ENABLED, to display
or not the feature powered by blocknote
- add AI_FEATURE_LEGACY_ENABLED, to display or not
the legacy AI features
When the tool applyDocumentOperations is used, we have to force the
usage of a system prompt in order to force the model to use it the right
without inventing different actions. The pydantic Agent class can use a
system prompt but this noe is ignoried when a UI adapter is used like
the VercelAiAdapter.
The frontend application is using Vercel AI SDK and it's data stream
protocol. We decided to use the pydantic AI library to use it's vercel
ai adapter. It will make the payload validation, use AsyncIterator and
deal with vercel specification.
