The duplicate endpoint used to respond with only `{"id": ...}`. It now
returns the complete duplicated document representation, consistent
with the other document detail endpoints, so the frontend doesn't have
to make a follow-up request to get the new document's data.
This required setting `is_favorite` explicitly on the duplicated
document before serializing it: it is normally set by the
`annotate_is_favorite` queryset method, which the newly created
document never goes through. Being a read-only serializer field, it
was silently dropped from the response instead of raising an error. A
document can't be a favorite right after being created, so it is set
to `False` directly.
Dropping a file larger than the allowed size showed a bare "unknown
error" in the editor. The proxy in front of the API cuts the request
and answers a 413 with an HTML body, so errorCauses threw while
parsing it as JSON and no cause ever reached the error panel.
The size limit the backend already enforces is now exposed by the
config endpoint, and the editor checks the file against it before
sending anything, with the same toast wording the document import
uses. errorCauses no longer throws on a body it cannot parse, and a
413 without a usable cause falls back to an explicit message, which
covers the instances whose proxy limit is lower than the application
one.
The size formatting duplicated in the import hook moved to a shared
util.
Signed-off-by: risk-alt <aldu6974@gmail.com>
The nb_accesses cache TTL was very short, 30 seconds. That mean that the
user will hit the cache for a very short period and the cache is
probably not be hit. This is what we can see in the slow queries from
the pg_stat_statements table. The query to compute the nb_accesses is
executed a little bit less than the number of queries to list or
retrieve documents, meaning the cache is not used.
The filtering made in the DocumentViewset.get_queryset method is not
optimal and lead to a full scan of the Document table. The heavy part is
on the filtering on what the user can access between the accesses and
the link traces. To have better performance we make an union operation
of both document_id list and the filter the id on this list. Postgresql
will use the index on the id column.
Comparing path with LEFT(value, LENGTH(path)) makes a sequential scan on
all the Document table, the more this table grow, the more the query
using it will be slow. We dediced instead to lookup on the path
extracting all ancestors path for a given document and then make a path
IN statement to use the index existing on the path column.
Document.content reads from object storage and returns None when nothing
was ever written there. That None, raised "content should be a string.",
so the duplicate endpoint answered a 500. Default to an empty string instead.
Signed-off-by: BOUKERFA Mohamed El Amine <boukerfa.ma@gmail.com>
Once the sql queries improved we have still a bottleneck on large
concurrent requests on this endpoint. We notive in the profiles generated
that lot of time was spent in creating a new s3 client instance on each
request. django_storage use a thread local cache for signed and unsigned
connection, but using uvicorn we have a new thread for each request, so
on each request a new s3 client is generated and it appears to be an
expensive operation. To fix this issue, we cache the client and share it
accross all the thread and requests.
On the media_auth endpoint the first bottleneck we have is with
postgresql. We are looking for too much data and no index is used on the
attachments colum. When the lookup filter on the attachement columns, a
full scan is made on all the document table looking for each element in
the array, this operation is really expensive. To fix this we created a
GIN index on the attachments column. Also the readable_per_se lookup was
selecting too much data combined with the filter_descendants function.
We remove the usage of the filter_descendants, we choose to first fetch
all the paths where the attachment is found, this operation is fast
thanks to the new index, split all the paths in candidate paths and then
filter readable_per_se queryset with these paths. All these
modifications make the endpoint faster.
Object storage metadata keys are case insensitive per the S3
specification, but implementations don't agree on the case they give
back. When head_object returns a capitalized "Status", updating the
attachment status added a second, lowercase entry instead of replacing
it, and the copy request ended up carrying two x-amz-meta-status
headers. Ceph RadosGW loses one of them behind a proxy, which
invalidates the request signature.
The same assumption was made when reading the status back in media-auth
and media-check, where an attachment stored on such a backend stayed in
"processing" forever.
Metadata read from the storage is now normalized to lowercase keys
before being consumed or copied over.
Signed-off-by: risk-alt <aldu6974@gmail.com>
When an access is updated or deleted, or a link_configuration is
changed, the collaration_service is used to reset the connection in the
collaboration server. As accesses and link_configuration are inherited
in a Docs tree, if the user is connected to a child, the connection is
not reset. This commit fix this issue by calling the reset on every
children in the tree.
The search in a document tree was triggered by the usage of the document
path. The path is something guessable by incrementing it you can
discover public documents. We decided to change this to use the document
id which is not guessable and prevent discovering public documents.
Thanks to @maboukerfa for discovering it.
The settings COLLABORATION_WS_NOT_CONNECTED_READY_ONLY contains a typo.
We don't want to see READY_ONLY but READ_ONLY. This commit fix the typo
and change everywhere in the codebase the settings name.
In the sharing module, the user model was used instead of working with
the user id. Only the user id is used in this module, forcing in the
module calling this functions to load the user model for nothing and
doing a N+1 query.
Prevent users from adding more reactions once the per-message
limit has been exceeded. It Disables reaction buttons
when limit is reached
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
Implement a configurable limit (default: 15) on the number of distinct
emoji reactions per comment.
- Backend validation ensures the limit cannot be exceeded via API
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
Retrieving a thread serialized its nested comments, reactions and
reaction users without prefetching, issuing one query per comment for
its author and reactions and one query per reaction for its users.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
In the search endpoint, we were returning all the parents for a document
mathing the search. We want instead to return only the top parent the
user has access to.
In the content_retrieve action, if the document is not existing on the
object storage, we return an empty binary string. In the case the app is
ran as an ASGI application we still have a warning explaining it must
consume the iterator before sending it.
When docs is ran using an ASGI server, the django StreamingHttpResponse
expect to consume an async generator. If a sync generator is used, it
fully consumes it and then return all the content and we loose the
benefits of using a StreamingHttpResponse.
In the tests there is a UnorderedObjectListWarning, we also find this same
warning in the logs in production. This warning appears because the
queryset used by the DocumentAskForAccess paginator is not ordered and
can lead to inconsistent result. Ordering the queryset fix this warning.
We want to allow or not the usage of the /documents/all/ endpoint. It has
been created to be used for an other purpose than the js client. For
those who don't use it this new settings allow to disable it. By default
it is set to True to keep the same behavior.
When an AI action is called from ai_translate, ai_transform and
ai_proxy, an event doc_ai_action is captured with a method property
giving the information from where it is called.
The posthog information were only available using the POSTHOG_KEY
settings. We split it in POSTHOG_KEY and POSTHOG_HOST to use them with
the pyhton posthog sdk
We want to allow users to leave a document where they have an access or
they have visited creating a link_trace. All subdocuments should also be
leaved at the same time.
To know if the user can leave a doc we have to check when computing the
abilities if a record is existing in the LinkTrace table. This is a N+1
query situation. To avoid it, we added an annotation in the
DocumentQueryset like we already do to annotate the user role.
There is one edge case where the annotation is made to soon, it is when the
user is visiting a document for the first time, the `get_object` add the
annotation and in the permission, we compute the abilities. The `leave`
property is False because the entry in the LinkTrace table is not made,
when the serializer ask for the abilities again, it is still False. So
in the `retrieve` method in the viewset we force the
`user_has_link_trace` to the correct value.
The mask feature has never been implemented by the frontend application
and it is not what we want to do. We want to implement a feature to
leave a document.
In the search response, we want to display a breadcrumb for every
document returned. For this we added a "parents" property containing a list
documents, all are the parents of the current document (ordered by their
depth). With this we can easily create a breadcrumb.
The global search endpoint filtering by title in the database was not
searching for accessible sub documents. We change the how global search
is made: it is now returning accessible sub documents matching the
title query.
We need to include resolved threads in the queryset
of the ThreadViewSet, otherwise they won't be
included in the list of threads and we will not be
able to list resolved threads in the side panel.
Admins/owners could overwrite another user's comment text and silently
take ownership of it.
Split write permissions: destroy stays broad (author + admin/owner) for
moderation.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
When a document is moved outside its current permission scope (root
document, cross-tree move, or promotion to root), its direct accesses
and pending invitations are now deleted server-side within the same
atomic transaction as the move itself. This ensures consistency: if
the move fails, deletions are rolled back.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
The children/ endpoint was missing file upload support that the root
documents endpoint already had. Added file-to-YJS conversion handling to
subdocument creation.
We observe some cases where the frontend and
backend versions can get out of sync, which can
cause issues.
To mitigate this, we want to implement a mechanism
that detects when the frontend and backend
versions are mismatched and triggers a
reload of the application to ensure they are in sync.
We have situation where the number of locks in the database can increase
dangerously creating deadlock situation. To remove this situation we
decided to change the strategy to manage document creation concurrency.
We decided to use a retry strategy, trying to create the document
multiple times while a usable path is found. To avoid having an
inifinite loop, we use a max_attempts counter configurable using the
setting TREEBEARD_PATH_COMPUTE_RETRY_MAX_ATTEMPTS
Anthony LC