Since pydantic-ai has ben upgraded deprecation warning were present in
the logs. This commint change how the instrumentation is enabled to be
compatible with hoe pydantic-ai manage it. Also prompt system are
ignored when they come from the front, like we are doing and dropped. To
maintain the system prompt we use the instruction parameter instead.
We upgraded pydantic-ai-slim to the latest version and add mistral has
extra dependency. Doing this, the direct dependance to mistral can be
removed and will be managed with pidantic-ai-slim package. The import of
mistralai has changed, we have to updagre them from mistralai to
mistralai.client
When an access is updated or deleted, or a link_configuration is
changed, the collaration_service is used to reset the connection in the
collaboration server. As accesses and link_configuration are inherited
in a Docs tree, if the user is connected to a child, the connection is
not reset. This commit fix this issue by calling the reset on every
children in the tree.
In the settings, the logging has set the propagate property to False.
While writing tests it wasn't possible to assert log was made using the
caplog fixtures because this propagate is set to False. Changing it to
True allow to correctly tests that a log is made using the caplog
fixture.
The search endpoint does not accept anymore the usage of the path
parameter but it is using the document id directly. We have to reflect
this changes in the frontend application.
The search in a document tree was triggered by the usage of the document
path. The path is something guessable by incrementing it you can
discover public documents. We decided to change this to use the document
id which is not guessable and prevent discovering public documents.
Thanks to @maboukerfa for discovering it.
We write a documentation explaining how to enable the safe guard
collaboration feature to prevent a user not connected to the websocket
to erase the data coming from users connected to the websocket.
The settings COLLABORATION_WS_NOT_CONNECTED_READY_ONLY contains a typo.
We don't want to see READY_ONLY but READ_ONLY. This commit fix the typo
and change everywhere in the codebase the settings name.
The S3 usage made by docs can be complex. We want to write a
documentation to explain the auth_request nginx module usage and list
known issues with the configuration.
We need a management command to reset a Document to an initial state and
deletes everything related to it. This command can be usefull to reset a
demo for example.
The authentication backend
mozilla_django_oidc.contrib.drf.OIDCAuthentication
is present in the default authentication classes for the REST_FRAMEWORK
settings. This backend should not be used by our application and can
lead to the usage of our main api with an access_token instead of the
cookie session.
We need to override the drf SessionAuthentication backend to implement
the authenticate_header method. Without this, a 403 status code is
returned, but it is not valid. It must a be 401
In the yprovider deployment we now need to mount the cacert containing
the certificates generated by mkcert. Then, we have to declare the
environment variable NODE_EXTRA_CA_CERTS with the path where this cacert
is mounted. With this new config we don't have tls issue anymore.
Added
- ✨(backend) add limit on distinct reactions per comment #1978
- ✨(frontend) leave a document #2410
- ✨(frontend) add top parent on sub docs search #1952
- ✨(frontend) unauthenticated users can search #2407
- ✨(backend) specific user delete method to delete its relations #2437
Changed
- 👷(CI) remove test-e2e-other-browser job #2404
- ♿️(frontend) use heading element for pinned documents section title #2380
- ♿️(frontend) use anchor links for table of contents entries #2390
- ♿️(frontend) improve presenter mode screen reader and keyboard support #2383
- ♿️(frontend) link export modal name to its heading #2422
Fixed
- 🐛(frontend) overlap of block menu dropdown #2406
- ⚡️(backend) fix N+1 queries when serializing thread comments #2415
In the admin there were 2 ways to delete a user : by the actions select
box and the button on its change view. Both are leading to a failure. To
fix it, both way are removed and we implemented a custom action calling
the user.delete method with all the specific workflow we implemented.
Deleting a user is not possible when it has created docs because the
on_delete on the Document class id RESTRICT and we don't want to change
it. We decided to have a specific workflow for correctly delete a user.
The document where the user is the sole owner must be deleted, the other
only the owner access must be deleted. For the remaining Documents where
the user is the creator, we set it to `null`, then the user can be
delete, remaining relations are deleted in cascade.
In the sharing module, the user model was used instead of working with
the user id. Only the user id is used in this module, forcing in the
module calling this functions to load the user model for nothing and
doing a N+1 query.
In the search endpoint, we were returning all the parents for a document
mathing the search. We want instead to return only the top parent the
user has access to.
In the content_retrieve action, if the document is not existing on the
object storage, we return an empty binary string. In the case the app is
ran as an ASGI application we still have a warning explaining it must
consume the iterator before sending it.
The tests are using the local object storage configured in the docker
compose stack. Every time a test using the DocumentFactory is executed,
an object is created in the object storage and will stay here but never
used. The data/media directory is growing and time to time we have to
delete it in order to free some gigabytes. In this commit we add a
global conftest responsible to delete the object created during the test
execution. The space will be really free when the object storage
container will be shutdown.
When docs is ran using an ASGI server, the django StreamingHttpResponse
expect to consume an async generator. If a sync generator is used, it
fully consumes it and then return all the content and we loose the
benefits of using a StreamingHttpResponse.
When all the tests are ran, there is a PytestWarning log present to warn
us that there is remaining database connection open and can not be
closed. It appears that the tests about concurrence are responsible of
leaving connection open. We need to manually close them in these tests.
In the tests there is a UnorderedObjectListWarning, we also find this same
warning in the logs in production. This warning appears because the
queryset used by the DocumentAskForAccess paginator is not ordered and
can lead to inconsistent result. Ordering the queryset fix this warning.
We want to allow or not the usage of the /documents/all/ endpoint. It has
been created to be used for an other purpose than the js client. For
those who don't use it this new settings allow to disable it. By default
it is set to True to keep the same behavior.
When an AI action is called from ai_translate, ai_transform and
ai_proxy, an event doc_ai_action is captured with a method property
giving the information from where it is called.
The posthog information were only available using the POSTHOG_KEY
settings. We split it in POSTHOG_KEY and POSTHOG_HOST to use them with
the pyhton posthog sdk
We want to allow users to leave a document where they have an access or
they have visited creating a link_trace. All subdocuments should also be
leaved at the same time.
To know if the user can leave a doc we have to check when computing the
abilities if a record is existing in the LinkTrace table. This is a N+1
query situation. To avoid it, we added an annotation in the
DocumentQueryset like we already do to annotate the user role.
There is one edge case where the annotation is made to soon, it is when the
user is visiting a document for the first time, the `get_object` add the
annotation and in the permission, we compute the abilities. The `leave`
property is False because the entry in the LinkTrace table is not made,
when the serializer ask for the abilities again, it is still False. So
in the `retrieve` method in the viewset we force the
`user_has_link_trace` to the correct value.
The mask feature has never been implemented by the frontend application
and it is not what we want to do. We want to implement a feature to
leave a document.
We wanto to have in the compose stack a dedicated service for the
converter like we did in the helm chart. The change is also made with
the e2e compose file
We want to isolate the converter service built in the yprovider server in
order to not mix conversion and collaboration together. We decided to
create a dedicated service and deployment, by default they use the
values defined in yProvider values but all can be overridden in
yProvider.converter.
Also, by default these new service and deployment are not enabled, they
must be explictly enabled and configured. If not it works like now with
only one service and deployment for the yProvider server.
In the model method User::_handle_onboarding_documents_access we do not
allow using documents with restricted link_reach to be added as
onboarding documents. To check the real link_reach of the document, we
must use instead the computed_link_reach to be sure that a sub document
can also be used and compute its correct link_reach.
In the search response, we want to display a breadcrumb for every
document returned. For this we added a "parents" property containing a list
documents, all are the parents of the current document (ordered by their
depth). With this we can easily create a breadcrumb.
The global search endpoint filtering by title in the database was not
searching for accessible sub documents. We change the how global search
is made: it is now returning accessible sub documents matching the
title query.
The djang-lasuite package manages the settings
OIDC_OP_USER_ENDPOINT_FORMAT allowing to enforce the format use by the
user endpoint. By default it is set to `AUTO`.
The docker network lasuite is created in the makefile using the command
`docker network create lasuite-network`. When a network is created with
this command it can not be used by docker compose if this one is not set
as external. We must use `external: true` in the compose file to fix
this issue.`
The migration 00227 must be run with a postgres superuser, most af
managed postgresql database can not be run using this kind of user.
Ti fix this, we use postgresql unnacent function instead of accessing C
function.
In development environment we wanto tu use the application with a
postgres user not having superuser role. The migration 0027 needs
superuser role to be executed, so we want to be able to test migrations
and other scenarios with a normal user. In order to create this user,
the compose service must be deleted first and then recreated with
DB_USER and DB_PASSWORD environment variable values different with
POSTGRES_USER and POSTGRES_PASSWORD
Added
- ⚡️(frontend) add skeleton on content loading #2254
- ⚡️(frontend) close websocket connection when user change tab #2264
Changed
- 🏗️(core) migrate from pip to uv
Fixed
- 🩺(project) reload app if front and back unsync #2276
- 🐛(frontend) fix patch and comments #2273
- 🐛(frontend) interlinking are exported correctly in print mode #2269
- 💬(frontend) add missing link in onboarding description #2233
- 🐛(frontend) sanitize pasted and dropped content in document title #2210
- 🐛(frontend) Emoji menu doesn't display above comment box #2229
- 🐛(frontend) Block menu doesn't stay open on 1st line #2229
- 🐛(frontend) The "+" on the first line of a new doc doesn't work #2229
- 🐛(backend) manage race condition between GET and PATCH content #2271
- 🐛(backend) replace document creation table locks with retry strategy #2274
Security
- 🔒️(frontend) sanitize color during collaboration #2270
The compilemessages management command was also compiling messages for
all the libraries present in the .venv folder. We have to ignore it, on
arm64 this management command is taking too much time otherwise.
We already migrate from pip to uv to manage our dependencies. We can also
migrate the build backend from setuptool to uv_build.
In the pyproject file, the readme property has been removed, because
uv_build try to read it, but the readme is at the root of the project
and not copied into the Dockerfile instructions. This readme can be used
when the package is published on pypi but it is not the case for Docs.
We have situation where the number of locks in the database can increase
dangerously creating deadlock situation. To remove this situation we
decided to change the strategy to manage document creation concurrency.
We decided to use a retry strategy, trying to create the document
multiple times while a usable path is found. To avoid having an
inifinite loop, we use a max_attempts counter configurable using the
setting TREEBEARD_PATH_COMPUTE_RETRY_MAX_ATTEMPTS
We need to split the core/utils.py in multiple submodule created in
core/utils/*.py. We need to do this to avoir circular import between
this module and the models module.
When a PATCH and a GET on the content endpoint are made at the same time
for different users a race condition can happen and the metadata
returned
by the S3 head_object can be outdated when the object is fetched leading
to an error raised because the Content-Length header does not match the
size of the response body. To avoid this, we no longer used head_object
followed bu get_object, we have to manage
everything in one call with the get_object. The get_object also accepts
as parameters an etag or last-modified header and will return a 304 if
the content has not changed, so we can use this to not return the entire
body if this one has not changed.
Added
- ✨(backend) create a dedicated endpoint to update document content
- ⚡️(backend) stream s3 file content with a dedicated endpoint
- ✨(backend) allow to use new ai feature using mistral sdk
Changed
- ♻️(backend) rename documents content endpoint in `formatted-content` (BC)
- 🚸(frontend) show Crisp from the help menu #2222
- ♿️(frontend) structure correctly 5xx error alerts #2128
- ♿️(frontend) make doc search result labels uniquely identifiable #2212
- ⬆️(backend) upgrade docspec to v3.0.x and adapt converter API #2220
- ✨(backend) make forward auth request uri header configurable #2241
- ♿️(frontend) fix sidebar resize handle for screen readers #2122
Fixed
- 🚸(frontend) redirect on current url tab after 401 #2197
- 🐛(frontend) abort check media status unmount #2194
- ✨(backend) order pinned documents by last updated at #2028
- 🐛(frontend) fix app shallow reload #2231
- 🐛(frontend) fix interlinking modal clipping #2213
- 🛂(frontend) fix cannot manage member on small screen #2226
- 🐛(backend) load jwks url when OIDC_RS_PRIVATE_KEY_STR is set
- 🐛(backend) Prevent moving document to its own descendant or self #2208
- 🐛(backend) return 400 when restoring a non-deleted document #2225
The return of the legacy AI translate prompt is sometimes surrounded
with a html code block. We improve the prompt to not return this
surrounding code block.
We also want to use the mistral sdk with the legacy AI feature when this
one is configured with the settings. In order to separate bot feature,
they now live in their own module.
When developping using the tilt environment, we need to allow the domain
docs.127.0.0.1.nip.io in the next configuration.
Also the the customization is removed as it is not maintained.
We give the possibility, for the new ai feature, to choose between using
the OpenAI or Mistral sdk. For instances having access to the mistral
infrastructure, using it is mor appropriated than using the openai
compatible chat model.
One test about the search descendants test was flaky. It is because the
link_reach and link_role were used to test the
ancestors_link_(reach|role). The properties ancestors_link_reach and
ancestors_link_role should be used instead.
We used one drf extra action with both PATCH and GET https methods and
then split in two private methods and call them based on the http method
of the request. DRF allow to do this by using a mapping annotation
allowing us to have directly twi viewset actions used
django-rest-framework.org/api-guide/viewsets/#marking-extra-actions-for-routing
We want to give to the js client the ability to use some headers to
avoid fetching a content it already have. For this, the content endpoint
will return an ETag and Last-Modified headers corresponding to the file
content ETag and its last modification. For future fetch, the client can
use the If-None-Match or If-Modified-Since request headers, if one of
these headers are satisfied, the endpoint will return a 304 response. If
not it will still return a 200
The content was always loaded in the document reponse. We remove this
behavior in order to not make an http call to the s3 storage. To get the
document content it is needed now to use the new endpoint dedicated to
retrive the document content.
We want a dedicated endpoint to update a document content. Previously,
updating the content was made on the update action shared with all other
document's properties. When the title is updated, the response contains
the content, so a call to the s3 storage is made and we don't want this.
Isolating the content update will allow us in the next commit to remove
the content from the Document serializer.
The endpoint /api/v1.0/documents/{document_id}/content/ has been renamed
in /api/v1.0/documents/{document_id}/formatted-content/. formatted-content
seems more accurante and the content endpoint will be used for another
purpose more appropriated.
The django application is running in ASGI in production, to have the
same behavior we run the development container with uvicorn too with
options more appropriated for a development evironment.
When the resource server is enabled and the backend used is
JWTResourceServerBackend, then the API should expose a JWKS endpoint to
share the RSA public key to the OIDC provider. Everything is made in the
Django LaSuite library, but the URL is not included in the Docs URLs.
This commit adds it when the setting OIDC_RS_PRIVATE_KEY_STR is set.
For the create_for_owner action, all the db operation are made in the
serializer. But the lock of the table was acquired in the viewsets, lot
of operation are made between the lock is made and the insert in db. We
move the lock operation closer to the insert in the database. We wrap it
in a transaction to release the lock once the commit made.
We want to control the conversion of document at upload time. We want to
disable this feature using a settings. The new settings
CONVERSION_UPLOAD_ENABLED should be used to enable or not the conversion
at upload feature. If disabled and a file is uploaded, the reponse will
return a 400
Manuel Raynaud