On the media_auth endpoint the first bottleneck we have is with
postgresql. We are looking for too much data and no index is used on the
attachments colum. When the lookup filter on the attachement columns, a
full scan is made on all the document table looking for each element in
the array, this operation is really expensive. To fix this we created a
GIN index on the attachments column. Also the readable_per_se lookup was
selecting too much data combined with the filter_descendants function.
We remove the usage of the filter_descendants, we choose to first fetch
all the paths where the attachment is found, this operation is fast
thanks to the new index, split all the paths in candidate paths and then
filter readable_per_se queryset with these paths. All these
modifications make the endpoint faster.
We have performance issue with the media_auth endpoint. We need to
monitor and profile this endpoint to understand what is happening. We
decided to install and configure django-silk. By default django-silk is
not enabled and not accessible, once enabled migration must be run and
the /silk/ endpoint will be accessible only using a super user.
To have relecant profiles and monitor we must have a database comparable
to a production one because we are facing a scaling issue. Unless a
million documents probably nothing is happening, for this we added
several management commands to help us. A first one to anonymize a
database than can be reused without any risk to leak PII data, can be
called using python manage.py anonymize_database with some mandatory
options. An other command to generate a database profile, this command
should be run on a production database, this profile can then be used by
the generate_volumetry to populate a development database with same
volumetry than describe in the profile.
Object storage metadata keys are case insensitive per the S3
specification, but implementations don't agree on the case they give
back. When head_object returns a capitalized "Status", updating the
attachment status added a second, lowercase entry instead of replacing
it, and the copy request ended up carrying two x-amz-meta-status
headers. Ceph RadosGW loses one of them behind a proxy, which
invalidates the request signature.
The same assumption was made when reading the status back in media-auth
and media-check, where an attachment stored on such a backend stayed in
"processing" forever.
Metadata read from the storage is now normalized to lowercase keys
before being consumed or copied over.
Signed-off-by: risk-alt <aldu6974@gmail.com>
Since pydantic-ai has ben upgraded deprecation warning were present in
the logs. This commint change how the instrumentation is enabled to be
compatible with hoe pydantic-ai manage it. Also prompt system are
ignored when they come from the front, like we are doing and dropped. To
maintain the system prompt we use the instruction parameter instead.
We upgraded pydantic-ai-slim to the latest version and add mistral has
extra dependency. Doing this, the direct dependance to mistral can be
removed and will be managed with pidantic-ai-slim package. The import of
mistralai has changed, we have to updagre them from mistralai to
mistralai.client
Add send_notification_email flag (defaults to True) on
the ServerCreateDocumentSerializer so that we can bypass
sending the email notification when creating a document
with the server to server API.
Signed-off-by: Florent Chehab <flo@nextmile.fr>
When an access is updated or deleted, or a link_configuration is
changed, the collaration_service is used to reset the connection in the
collaboration server. As accesses and link_configuration are inherited
in a Docs tree, if the user is connected to a child, the connection is
not reset. This commit fix this issue by calling the reset on every
children in the tree.
The search in a document tree was triggered by the usage of the document
path. The path is something guessable by incrementing it you can
discover public documents. We decided to change this to use the document
id which is not guessable and prevent discovering public documents.
Thanks to @maboukerfa for discovering it.
The settings COLLABORATION_WS_NOT_CONNECTED_READY_ONLY contains a typo.
We don't want to see READY_ONLY but READ_ONLY. This commit fix the typo
and change everywhere in the codebase the settings name.
We need a management command to reset a Document to an initial state and
deletes everything related to it. This command can be usefull to reset a
demo for example.
The authentication backend
mozilla_django_oidc.contrib.drf.OIDCAuthentication
is present in the default authentication classes for the REST_FRAMEWORK
settings. This backend should not be used by our application and can
lead to the usage of our main api with an access_token instead of the
cookie session.
We need to override the drf SessionAuthentication backend to implement
the authenticate_header method. Without this, a 403 status code is
returned, but it is not valid. It must a be 401
Updated the restore ability so that only directly deleted documents can
be restored. This prevents the restore action from being exposed for
child documents whose parent is deleted. This PR addresses #2127.
Owners could leave trashed documents via the leave endpoint, deleting
their `DocumentAccess` and all subtree access. When the last owner
leaves a document it creates an orphan as no owners remains in the doc.
Signed-off-by: BOUKERFA Mohamed El Amine <boukerfa.ma@gmail.com>
In the admin there were 2 ways to delete a user : by the actions select
box and the button on its change view. Both are leading to a failure. To
fix it, both way are removed and we implemented a custom action calling
the user.delete method with all the specific workflow we implemented.
Deleting a user is not possible when it has created docs because the
on_delete on the Document class id RESTRICT and we don't want to change
it. We decided to have a specific workflow for correctly delete a user.
The document where the user is the sole owner must be deleted, the other
only the owner access must be deleted. For the remaining Documents where
the user is the creator, we set it to `null`, then the user can be
delete, remaining relations are deleted in cascade.
In the sharing module, the user model was used instead of working with
the user id. Only the user id is used in this module, forcing in the
module calling this functions to load the user model for nothing and
doing a N+1 query.
Prevent users from adding more reactions once the per-message
limit has been exceeded. It Disables reaction buttons
when limit is reached
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
Implement a configurable limit (default: 15) on the number of distinct
emoji reactions per comment.
- Backend validation ensures the limit cannot be exceeded via API
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
Retrieving a thread serialized its nested comments, reactions and
reaction users without prefetching, issuing one query per comment for
its author and reactions and one query per reaction for its users.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
In the search endpoint, we were returning all the parents for a document
mathing the search. We want instead to return only the top parent the
user has access to.
In the content_retrieve action, if the document is not existing on the
object storage, we return an empty binary string. In the case the app is
ran as an ASGI application we still have a warning explaining it must
consume the iterator before sending it.
When docs is ran using an ASGI server, the django StreamingHttpResponse
expect to consume an async generator. If a sync generator is used, it
fully consumes it and then return all the content and we loose the
benefits of using a StreamingHttpResponse.
When all the tests are ran, there is a PytestWarning log present to warn
us that there is remaining database connection open and can not be
closed. It appears that the tests about concurrence are responsible of
leaving connection open. We need to manually close them in these tests.
In the tests there is a UnorderedObjectListWarning, we also find this same
warning in the logs in production. This warning appears because the
queryset used by the DocumentAskForAccess paginator is not ordered and
can lead to inconsistent result. Ordering the queryset fix this warning.
We want to allow or not the usage of the /documents/all/ endpoint. It has
been created to be used for an other purpose than the js client. For
those who don't use it this new settings allow to disable it. By default
it is set to True to keep the same behavior.
When an AI action is called from ai_translate, ai_transform and
ai_proxy, an event doc_ai_action is captured with a method property
giving the information from where it is called.
The posthog information were only available using the POSTHOG_KEY
settings. We split it in POSTHOG_KEY and POSTHOG_HOST to use them with
the pyhton posthog sdk
We want to allow users to leave a document where they have an access or
they have visited creating a link_trace. All subdocuments should also be
leaved at the same time.
To know if the user can leave a doc we have to check when computing the
abilities if a record is existing in the LinkTrace table. This is a N+1
query situation. To avoid it, we added an annotation in the
DocumentQueryset like we already do to annotate the user role.
There is one edge case where the annotation is made to soon, it is when the
user is visiting a document for the first time, the `get_object` add the
annotation and in the permission, we compute the abilities. The `leave`
property is False because the entry in the LinkTrace table is not made,
when the serializer ask for the abilities again, it is still False. So
in the `retrieve` method in the viewset we force the
`user_has_link_trace` to the correct value.
The mask feature has never been implemented by the frontend application
and it is not what we want to do. We want to implement a feature to
leave a document.
In the model method User::_handle_onboarding_documents_access we do not
allow using documents with restricted link_reach to be added as
onboarding documents. To check the real link_reach of the document, we
must use instead the computed_link_reach to be sure that a sub document
can also be used and compute its correct link_reach.
In the search response, we want to display a breadcrumb for every
document returned. For this we added a "parents" property containing a list
documents, all are the parents of the current document (ordered by their
depth). With this we can easily create a breadcrumb.
The global search endpoint filtering by title in the database was not
searching for accessible sub documents. We change the how global search
is made: it is now returning accessible sub documents matching the
title query.
Manuel Raynaud