We have performance issue with the media_auth endpoint. We need to
monitor and profile this endpoint to understand what is happening. We
decided to install and configure django-silk. By default django-silk is
not enabled and not accessible, once enabled migration must be run and
the /silk/ endpoint will be accessible only using a super user.
To have relecant profiles and monitor we must have a database comparable
to a production one because we are facing a scaling issue. Unless a
million documents probably nothing is happening, for this we added
several management commands to help us. A first one to anonymize a
database than can be reused without any risk to leak PII data, can be
called using python manage.py anonymize_database with some mandatory
options. An other command to generate a database profile, this command
should be run on a production database, this profile can then be used by
the generate_volumetry to populate a development database with same
volumetry than describe in the profile.
With Content Security Policies activated, swagger (and redoc)
auto-generated API documentation is no longer accessible even locally.
To restore this feature, we've excluded CSP for related URLs only in
the `Development` configuration.
In the settings, the logging has set the propagate property to False.
While writing tests it wasn't possible to assert log was made using the
caplog fixtures because this propagate is set to False. Changing it to
True allow to correctly tests that a log is made using the caplog
fixture.
Keycloak's hostname, its client's redirect/web
origins, and Django's CSRF trusted origins were
hardcoded to localhost, so the dev stack was
only reachable from that exact origin.
It is now possible to run the dev stack depending
what is configured in the env files.
The settings COLLABORATION_WS_NOT_CONNECTED_READY_ONLY contains a typo.
We don't want to see READY_ONLY but READ_ONLY. This commit fix the typo
and change everywhere in the codebase the settings name.
The authentication backend
mozilla_django_oidc.contrib.drf.OIDCAuthentication
is present in the default authentication classes for the REST_FRAMEWORK
settings. This backend should not be used by our application and can
lead to the usage of our main api with an access_token instead of the
cookie session.
We need to override the drf SessionAuthentication backend to implement
the authenticate_header method. Without this, a 403 status code is
returned, but it is not valid. It must a be 401
Implement a configurable limit (default: 15) on the number of distinct
emoji reactions per comment.
- Backend validation ensures the limit cannot be exceeded via API
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
We want to allow or not the usage of the /documents/all/ endpoint. It has
been created to be used for an other purpose than the js client. For
those who don't use it this new settings allow to disable it. By default
it is set to True to keep the same behavior.
The posthog information were only available using the POSTHOG_KEY
settings. We split it in POSTHOG_KEY and POSTHOG_HOST to use them with
the pyhton posthog sdk
The djang-lasuite package manages the settings
OIDC_OP_USER_ENDPOINT_FORMAT allowing to enforce the format use by the
user endpoint. By default it is set to `AUTO`.
We have situation where the number of locks in the database can increase
dangerously creating deadlock situation. To remove this situation we
decided to change the strategy to manage document creation concurrency.
We decided to use a retry strategy, trying to create the document
multiple times while a usable path is found. To avoid having an
inifinite loop, we use a max_attempts counter configurable using the
setting TREEBEARD_PATH_COMPUTE_RETRY_MAX_ATTEMPTS
We give the possibility, for the new ai feature, to choose between using
the OpenAI or Mistral sdk. For instances having access to the mistral
infrastructure, using it is mor appropriated than using the openai
compatible chat model.
In deployment, Traefik is used, not nginx, as an ingress. Traefik
uses `X-Forwarded-Ur`i instead of `X-Original-Url`. This adds a setting
which lets users adapt Docs to their ingress proxy of choice
The settings name is MEDIA_AUTH_ORIGINAL_URL_HEADER
Signed-off-by: Erin Shepherd <erin.shepherd@e43.eu>
We want to give to the js client the ability to use some headers to
avoid fetching a content it already have. For this, the content endpoint
will return an ETag and Last-Modified headers corresponding to the file
content ETag and its last modification. For future fetch, the client can
use the If-None-Match or If-Modified-Since request headers, if one of
these headers are satisfied, the endpoint will return a 304 response. If
not it will still return a 200
We want to control the conversion of document at upload time. We want to
disable this feature using a settings. The new settings
CONVERSION_UPLOAD_ENABLED should be used to enable or not the conversion
at upload feature. If disabled and a file is uploaded, the reponse will
return a 400
## Purpose
For beta testing purposes we need to be able to activate Find hybrid
search to some users, Find full-text search to some others and leave
remaining users on basic DRF title search.
## Proposal
The solution proposed is based on [django-waffle
](https://waffle.readthedocs.io/en/stable/types/flag.html).
- [x] install waffle and activate the default app in settings.
- [x] implement `_get_search_type` in `DocumentViewset` to determine
which search type (title, hybrid or full-text) to use.
- [x] send the `search_type` in the search query.
## External contributions
Thank you for your contribution! 🎉
Please ensure the following items are checked before submitting your
pull request:
- [x] I have read and followed the [contributing
guidelines](https://github.com/suitenumerique/docs/blob/main/CONTRIBUTING.md)
- [x] I have read and agreed to the [Code of
Conduct](https://github.com/suitenumerique/docs/blob/main/CODE_OF_CONDUCT.md)
- [x] I have signed off my commits with `git commit --signoff` (DCO
compliance)
- [x] I have signed my commits with my SSH or GPG key (`git commit -S`)
- [x] My commit messages follow the required format: `<gitmoji>(type)
title description`
- [x] I have added a changelog entry under `## [Unreleased]` section (if
noticeable change)
- [x] I have added corresponding tests for new features or bug fixes (if
applicable)
---------
Signed-off-by: charles <charles.englebert@protonmail.com>
## Purpose
integrate Find to Docs
## Proposal
- [x] ✨ add a `useSeachDocs` hook in charged of calling the search
endpoint.
- [x] ✨ add a optional `path` param to the `search` route. This param
represents the parent document path in case of a sub-documents
(descendants) search.
- [x] ⚡️return Indexer results directly without DB calls to retrieve the
Document objects. All informations necessary for display are indexed in
Find. We can skip the DB calls and improve performance.
- [x] ♻️ refactor react `DocSearchContent` components.
`DocSearchContent` and `DocSearchSubContent` are now merged a unique
component handling all search scenarios and relying on the unique
`search` route.
- [x] 🔥remove pagination logic in the Indexer. Removing the DB calls
also removes the DRF queryset object which handles the pagination. Also
we consider pagination not to be necessary for search v1.
- [x] 🔥remove the `document/<document_id>/descendants` route. This route
is not used anymore. The logic of finding the descendants are moved to
the internal `_list_descendants` method. This method is based on the
parent `path` instead of the parent `id` which has some consequence
about the user access management. Relying on the path prevents the use
of the `self.get_object()` method which used to handle the user access
logic.
- [x] ✨handle fallback logic on DRF based title search in case of
non-configured, badly configured or failing at run time indexer.
- [x] ✨handle language extension in `title` field. Find returns titles
with a language extension (ex: `{ title.fr: "rapport d'activité" }`
instead of `{ "title": "rapport d'activité" }`.
- [x] 🔧 add a `common.test` file to allow running the tests without
docker
- [x] ♻️ rename `SearchIndexer` -> `FindDocumentIndexer`. This class has
to do with Find in particular and the convention is more coherent with
`BaseDocumentIndexer`
- [x] ♻️ rename `SEARCH_INDEXER_URL` -> `INDEXING_URL` and
`SEARCH_INDEXER_QUERY_URL` -> `SEARCH_URL`. I found the original names
very confusing.
- [x] 🔧 update the environment variables to activate the
FindDocumentIndexer.
- [x] ✨automate the generation of encryption key during bootstrap.
OIDC_STORE_REFRESH_TOKEN_KEY is a mandatory secret key. We can not push
it on Github and we want any contributor to be able to run the app by
only running the `make bootstrap`. We chose to generate and wright it
into the `common.local` during bootstrap.
## External contributions
Thank you for your contribution! 🎉
Please ensure the following items are checked before submitting your
pull request:
- [x] I have read and followed the [contributing
guidelines](https://github.com/suitenumerique/docs/blob/main/CONTRIBUTING.md)
- [x] I have read and agreed to the [Code of
Conduct](https://github.com/suitenumerique/docs/blob/main/CODE_OF_CONDUCT.md)
- [x] I have signed off my commits with `git commit --signoff` (DCO
compliance)
- [x] I have signed my commits with my SSH or GPG key (`git commit -S`)
- [x] My commit messages follow the required format: `<gitmoji>(type)
title description`
- [x] I have added a changelog entry under `## [Unreleased]` section (if
noticeable change)
- [x] I have added corresponding tests for new features or bug fixes (if
applicable)
---------
Signed-off-by: charles <charles.englebert@protonmail.com>
The psycopg pool config was enabled by default forcing its usage. Using
psycopg pool can be difficult, finding the good configuration take time.
By default its usage should be disable and the maintainer of the
instance should decide to enable it or not.
We enable the pool option on the DB configuration. We want to allow the
configuration of the min and max sixe in a first time. They can be
configured using the settings DB_PSYCOPG_POOL_MIN_SIZE and
DB_PSYCOPG_POOL_MAX_SIZE. They have their default value to 4 and None.
The Blocknote AI feature is a bit flaky, we want
to be able to disable it if to much issues arise,
without having to do a new release.
We add a bunch of feature flags to be able to
disable the AI features if needed:
- add AI_FEATURE_BLOCKNOTE_ENABLED, to display
or not the feature powered by blocknote
- add AI_FEATURE_LEGACY_ENABLED, to display or not
the legacy AI features
The frontend application is using Vercel AI SDK and it's data stream
protocol. We decided to use the pydantic AI library to use it's vercel
ai adapter. It will make the payload validation, use AsyncIterator and
deal with vercel specification.
The settings CONVERSION_FILE_MAX_SIZE was not used
in the y-provider, which caused a 413 Payload
Too Large error when trying to convert a file larger
than 500kb.
This commit updates the y-provider to use the
CONVERSION_FILE_MAX_SIZE settings, allowing it to
handle larger files without throwing an error.
CONVERSION_FILE_MAX_SIZE should follow the same
value as the one defined in the backend settings,
which is 20mb by default.
Adds two methods to allow new users to start with some docs.
User._handle_onboarding_documents_access() gives READER access to
each document listed in settings.USER_ONBOARDING_DOCUMENTS.
User._duplicate_onboarding_sandbox_document() creates a local copy
of the sandbox document specified in
settings.USER_ONBOARDING_SANDBOX_DOCUMENT.
For now, the reconciliation requests are imported through CSV in the
Django admin, which sends confirmation email to both addresses. When
both are checked, the actual reconciliation is processed, and all
user-related content is updated.
## Purpose
Fix#1616 // Replaces #1708
For now, the reconciliation requests are imported through CSV in the
Django admin, which sends confirmation email to both addresses. When
both are checked, the actual reconciliation is processed, and all
user-related content is updated.
## Proposal
- [x] New `UserReconciliationCsvImport` model to manage the import of
reconciliation requests through a task
(`user_reconciliation_csv_import_job`)
- [x] New `UserReconciliation` model to store the user reconciliation
requests themselves (a row = a `active_user`/`inactive_user` pair)
- [x] On save, a confirmation email is sent to the users
- [x] A `process_reconciliation` admin action process the action on the
requested entries, if both emails have been checked.
- [x] Bulk update the `DocumentAccess` items, while managing the case
where both users have access to the document (keeping the higher role)
- [x] Bulk update the `LinkTrace` items, while managing the case where
both users have link traces to the document
- [x] Bulk update the `DocumentFavorite` items, while managing the case
where both users have put the document in their favorites
- [x] Bulk update the comment system items (`Thread`, `Comment` and
`Reaction` items)
- [x] Bulk update the `is_active` status on both users
- [x] New `USER_RECONCILIATION_FORM_URL` env variable for the "make a
new request" URL in an email.
- [x] Write unit tests
- [x] Remove the unused `email_user()` method on `User`, replaced with
`send_email()` similar to the one on the `Document` model
## Demo page reconciliation success
<img width="1149" height="746" alt="image"
src="https://github.com/user-attachments/assets/09ba2b38-7af3-41fa-a64f-ce3c4fd8548d"
/>
---------
Co-authored-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
## Purpose
Allows a user to find more easily the other users they search, with the
following order of priority:
- users they already share documents with (more recent first)
- users that share the same full email domain
- ~~users that share the same partial email domain (last two parts)~~
- ~~other users~~
Edit: We need to ilter out other users in order to not reveal email
addresses from members of other organisations. It's still possible to
invite them by email.
Solves #1521
## Proposal
- [x] Add a new function in `core/utils.py`:
`users_sharing_documents_with()`
- [x] Use it as a key to sort the results of a basic user search
- [x] Filter user results to avoid reveal of users (and email addresses)
of other orgs or that have not been interacted with.
- [x] User research through "full" email address (contains the '@') is
left unaffected.
---------
Co-authored-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
Add support for the `AWS_S3_SIGNATURE_VERSION` environment variable to
allow configuring S3 signature version for compatibility with
S3-compatible storage services like Linode Object Storage.
Fixes#1788
Signed-off-by: dtinth on MBP M1 <dtinth@spacet.me>
We want to split the cache config between the app cache and the session
cache. In the app cache, the default one, we allow to configure a
prefix. By default this prefix is a fixed string so the cache will be
never revoked because it is changing but it allow every instance to
implement its own strategy like prefixing the keyx cache with a
timestamp.
To not impact session, the session cache is splitted in the settings.
Not every project requires silent login.
This commit adds a new feature flag
FRONTEND_SILENT_LOGIN_ENABLED to enable or
disable silent login functionality.
Currently users already logged in to the SSO have to click on
the login button again to be connected.
This extra step should not be necessary.
This commit uses the "silent=true" parameter to the login
endpoint to avoid the extra step.
Most of Docs app is configured thanks to environment
variables, except the url in the email that
was from the django site table.
Now we can set it with DJANGO_EMAIL_URL_APP
environment variable to have a better consistency.
We keep the previous way to avoid breaking
changes.
Refactored converter services based on PR #1609 review comments:
- Renamed parameter to `data` across all convert methods for consistency
- Replaced recursive call with explicit sequential calls for readability
- Hardcoded CONVERSION_API_SECURE=True in Production class for security
- Removed unused YdocConverter import from viewsets.py
- Updated tests to match new error message wording
Signed-off-by: Stephan Meijer <me@stephanmeijer.com>
Added comprehensive tests covering DocSpec converter service,
converter orchestration, and document creation with file uploads.
Tests validate DOCX and Markdown conversion workflows, error
handling, service availability, and edge cases including empty
files and Unicode filenames.
Signed-off-by: Stephan Meijer <me@stephanmeijer.com>
We can now import documents in formats .docx and .md.
To do so we added a new container "docspec", which
uses the docspec service to convert
these formats to Blocknote format.
More here: #1567#1569.
The template feature is removed.
Migration created to drop related tables.
Files modified:
- viewsets
- serializers
- models
- admin
- factories
- urls
- tests
- demo data
We want to monitor AI actions. For this we choose to use langfuse. As
this usage is optional, we load langfuse sdk only if settings are
configured. Also, the openai client from langfuse is a dropin
replacement of openai client, so we only have to change how openai is
imported.
