The exception block was never being executed because URLValidator raises
django.core.exceptions.ValidationError, not
drf.exceptions.ValidationError, so the except block was dead code.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
The call to the background task is now wrapped in a on_commit to ensure
that it isn't called before the save is finished, in order to avoid race
condition issues.
We add the User Accesses before saving content so the user is sure to
have access to the the first version when creating a doc through
create_for_owner (fixes#2123)
For the create_for_owner action, all the db operation are made in the
serializer. But the lock of the table was acquired in the viewsets, lot
of operation are made between the lock is made and the insert in db. We
move the lock operation closer to the insert in the database. We wrap it
in a transaction to release the lock once the commit made.
We want to control the conversion of document at upload time. We want to
disable this feature using a settings. The new settings
CONVERSION_UPLOAD_ENABLED should be used to enable or not the conversion
at upload feature. If disabled and a file is uploaded, the reponse will
return a 400
We want to lock the table just before the insert we want to protect is
made. In the case of the perform_create action in the Document viewset,
an http call is made after the lock and can take a very long time,
blocking for nothing the table.
Add test_api_document_favorite_list_with_deleted_child to verify favorite_list
endpoint does not include deleted sub documents
Signed-off-by: Paul Vernin <paul.vernin@gmail.com>
The frontend application is making PATCH request with an empty body.
This PATCH request is not making any change but an UPDATE sql query is
made, the `updated_at` field is the only one updated. When can skip this
save in the databse by returning the Document instance in the serializer
update method
No tests were made using the PATCH method to update a Document using the
API. The frontend appllication mostly use the patch method instead of
the PUT method.
The search endpoint was using the refresh_roken method decorator. This
decorator force having a valid refresh token stored in the session for
the entire viewset. The search endpoint still allow having the legacy
search behavior and for this we don't need to configure at all the OIDC
refrsh mechanism.
When a document is duplicated, it is duplicated at the direct right of
the duplicated document. Doing this force to move all the other
documents at the right, if it is duplicated at the root this can impact
a lot of documents, create lot of locks in the database. If the process
is stop for any reason then the paths can be in an inconsistent paths in
the Document table
In a past release we added a feature to create a sandbox document to a
newly created used. To create this sandbox document, we duplicate an
existing document and this duplicate is using the add_sibling method
with the "right" agument on this original document. Adding a sibling at
the right to a document involve moving right every root document created
after the original document, so the path of all this documents are
recalculated and changed. This can lead to the lost of some leaf in a
tree because to do this operation, multiple locks are created on the
database, creating lot of connection to the database and if the max
number connection to the database is reached or if the memory allocated
by the database is too hight, the database can close all connections
leading to inconsistent paths in the Document table.
## Purpose
For beta testing purposes we need to be able to activate Find hybrid
search to some users, Find full-text search to some others and leave
remaining users on basic DRF title search.
## Proposal
The solution proposed is based on [django-waffle
](https://waffle.readthedocs.io/en/stable/types/flag.html).
- [x] install waffle and activate the default app in settings.
- [x] implement `_get_search_type` in `DocumentViewset` to determine
which search type (title, hybrid or full-text) to use.
- [x] send the `search_type` in the search query.
## External contributions
Thank you for your contribution! 🎉
Please ensure the following items are checked before submitting your
pull request:
- [x] I have read and followed the [contributing
guidelines](https://github.com/suitenumerique/docs/blob/main/CONTRIBUTING.md)
- [x] I have read and agreed to the [Code of
Conduct](https://github.com/suitenumerique/docs/blob/main/CODE_OF_CONDUCT.md)
- [x] I have signed off my commits with `git commit --signoff` (DCO
compliance)
- [x] I have signed my commits with my SSH or GPG key (`git commit -S`)
- [x] My commit messages follow the required format: `<gitmoji>(type)
title description`
- [x] I have added a changelog entry under `## [Unreleased]` section (if
noticeable change)
- [x] I have added corresponding tests for new features or bug fixes (if
applicable)
---------
Signed-off-by: charles <charles.englebert@protonmail.com>
## Purpose
integrate Find to Docs
## Proposal
- [x] ✨ add a `useSeachDocs` hook in charged of calling the search
endpoint.
- [x] ✨ add a optional `path` param to the `search` route. This param
represents the parent document path in case of a sub-documents
(descendants) search.
- [x] ⚡️return Indexer results directly without DB calls to retrieve the
Document objects. All informations necessary for display are indexed in
Find. We can skip the DB calls and improve performance.
- [x] ♻️ refactor react `DocSearchContent` components.
`DocSearchContent` and `DocSearchSubContent` are now merged a unique
component handling all search scenarios and relying on the unique
`search` route.
- [x] 🔥remove pagination logic in the Indexer. Removing the DB calls
also removes the DRF queryset object which handles the pagination. Also
we consider pagination not to be necessary for search v1.
- [x] 🔥remove the `document/<document_id>/descendants` route. This route
is not used anymore. The logic of finding the descendants are moved to
the internal `_list_descendants` method. This method is based on the
parent `path` instead of the parent `id` which has some consequence
about the user access management. Relying on the path prevents the use
of the `self.get_object()` method which used to handle the user access
logic.
- [x] ✨handle fallback logic on DRF based title search in case of
non-configured, badly configured or failing at run time indexer.
- [x] ✨handle language extension in `title` field. Find returns titles
with a language extension (ex: `{ title.fr: "rapport d'activité" }`
instead of `{ "title": "rapport d'activité" }`.
- [x] 🔧 add a `common.test` file to allow running the tests without
docker
- [x] ♻️ rename `SearchIndexer` -> `FindDocumentIndexer`. This class has
to do with Find in particular and the convention is more coherent with
`BaseDocumentIndexer`
- [x] ♻️ rename `SEARCH_INDEXER_URL` -> `INDEXING_URL` and
`SEARCH_INDEXER_QUERY_URL` -> `SEARCH_URL`. I found the original names
very confusing.
- [x] 🔧 update the environment variables to activate the
FindDocumentIndexer.
- [x] ✨automate the generation of encryption key during bootstrap.
OIDC_STORE_REFRESH_TOKEN_KEY is a mandatory secret key. We can not push
it on Github and we want any contributor to be able to run the app by
only running the `make bootstrap`. We chose to generate and wright it
into the `common.local` during bootstrap.
## External contributions
Thank you for your contribution! 🎉
Please ensure the following items are checked before submitting your
pull request:
- [x] I have read and followed the [contributing
guidelines](https://github.com/suitenumerique/docs/blob/main/CONTRIBUTING.md)
- [x] I have read and agreed to the [Code of
Conduct](https://github.com/suitenumerique/docs/blob/main/CODE_OF_CONDUCT.md)
- [x] I have signed off my commits with `git commit --signoff` (DCO
compliance)
- [x] I have signed my commits with my SSH or GPG key (`git commit -S`)
- [x] My commit messages follow the required format: `<gitmoji>(type)
title description`
- [x] I have added a changelog entry under `## [Unreleased]` section (if
noticeable change)
- [x] I have added corresponding tests for new features or bug fixes (if
applicable)
---------
Signed-off-by: charles <charles.englebert@protonmail.com>
The psycopg pool config was enabled by default forcing its usage. Using
psycopg pool can be difficult, finding the good configuration take time.
By default its usage should be disable and the maintainer of the
instance should decide to enable it or not.
Since we upgraded to django-treebeard version 5 we have anormal behavior
and a high error rate on the document.path property. We must downgrade
it and avoid future upgrade from renovate.
We change the strategy on how the new users have access to the
onboarding documents. We should remove all created accesses we don't
want to have anymore. There is no need to add them in the link_trace
table, they are already present in the favorites and user have already
access to it.
When a user is created and a sandbox document should be created, we can
have a race condition on the document creation leading to an error for
the user. To avoid this we have to manage this part in a transaction and
locking the document table
When a user is created, we created accesses to a list of onboarding
documents. Doing this have side effect on the proximity search feature.
Instead of creating access, we should create link_reach
Backend part of #1796.
This changes allows to display an onboarding modal the first time that
the get_me() API view is called.
I originally tried to check if `User.last_login` was `None`, but it is
updated as soon as the user is logged, so I chose to create a flag on
the model.
In version 5 of django-treebeard they change the usage of NumConv for
something easier. They also introduce a breaking change, we have to
change the usage of this class in the migration 15.
Privileged users of a document (owners and admins) should not be
allowed to create an access request on a document they already control.
Without aguardrail, they could inadvertently inflate the access
request queue with redundant entries.
More critically, if an owner submits an access request on their own
document, another admin could accept it and grant them a lower role
(e.g. reader), which would silently strip them of their ownership.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
This patch changes the Content-Disposition header set on S3 PutObject
requests when uploading attachments to be encoded using a Django helper,
properly encoding UTF-8 in filenames.
This fixes compatibility for uploading attachments with non ASCII file names
to Garage, since the HTTP library used by Garage validates that HTTP Headers
are ASCII only.
Signed-off-by: networkException <git@nwex.de>
The Blocknote AI feature is a bit flaky, we want
to be able to disable it if to much issues arise,
without having to do a new release.
We add a bunch of feature flags to be able to
disable the AI features if needed:
- add AI_FEATURE_BLOCKNOTE_ENABLED, to display
or not the feature powered by blocknote
- add AI_FEATURE_LEGACY_ENABLED, to display or not
the legacy AI features
When the tool applyDocumentOperations is used, we have to force the
usage of a system prompt in order to force the model to use it the right
without inventing different actions. The pydantic Agent class can use a
system prompt but this noe is ignoried when a UI adapter is used like
the VercelAiAdapter.
The frontend application is using Vercel AI SDK and it's data stream
protocol. We decided to use the pydantic AI library to use it's vercel
ai adapter. It will make the payload validation, use AsyncIterator and
deal with vercel specification.
Standard can vary depending on the AI service used.
To work with Albert API:
- a description field is required in the payload
for every tools call.
- if stream is set to false, stream_options must
be omitted from the payload.
- the response from Albert sometimes didn't respect
the format expected by Blocknote, so we added a
system prompt to enforce it.
The frontend can fetch the retrieve endpoint just for having the title.
Everytime, the content is added and to get the content, a request is
made to the s3 bucket. A query string `without_content` can be used, if
the value is `true` then the content is not added (and not also not
fetch).
Adds two methods to allow new users to start with some docs.
User._handle_onboarding_documents_access() gives READER access to
each document listed in settings.USER_ONBOARDING_DOCUMENTS.
User._duplicate_onboarding_sandbox_document() creates a local copy
of the sandbox document specified in
settings.USER_ONBOARDING_SANDBOX_DOCUMENT.
Adds a new with_descendants parameter to the doc duplication API.
The logic of the duplicate() method has been moved to a new
internal _duplicate_document() method to allow for recursion.
Adds unit tests for the new feature.
Add utm_source=docssharelink and utm_campaign={docId} query parameters
to shared document links (copy link and invitation emails).
Signed-off-by: Chaïb Martinez <chaibax@gmail.com>
Some identity providers might change the case, but in our products we
don't consider case variation to be consider as different email
addresses.
Next step would be to normalize the DB value of email to be lower-case.
For now, the reconciliation requests are imported through CSV in the
Django admin, which sends confirmation email to both addresses. When
both are checked, the actual reconciliation is processed, and all
user-related content is updated.
## Purpose
Fix#1616 // Replaces #1708
For now, the reconciliation requests are imported through CSV in the
Django admin, which sends confirmation email to both addresses. When
both are checked, the actual reconciliation is processed, and all
user-related content is updated.
## Proposal
- [x] New `UserReconciliationCsvImport` model to manage the import of
reconciliation requests through a task
(`user_reconciliation_csv_import_job`)
- [x] New `UserReconciliation` model to store the user reconciliation
requests themselves (a row = a `active_user`/`inactive_user` pair)
- [x] On save, a confirmation email is sent to the users
- [x] A `process_reconciliation` admin action process the action on the
requested entries, if both emails have been checked.
- [x] Bulk update the `DocumentAccess` items, while managing the case
where both users have access to the document (keeping the higher role)
- [x] Bulk update the `LinkTrace` items, while managing the case where
both users have link traces to the document
- [x] Bulk update the `DocumentFavorite` items, while managing the case
where both users have put the document in their favorites
- [x] Bulk update the comment system items (`Thread`, `Comment` and
`Reaction` items)
- [x] Bulk update the `is_active` status on both users
- [x] New `USER_RECONCILIATION_FORM_URL` env variable for the "make a
new request" URL in an email.
- [x] Write unit tests
- [x] Remove the unused `email_user()` method on `User`, replaced with
`send_email()` similar to the one on the `Document` model
## Demo page reconciliation success
<img width="1149" height="746" alt="image"
src="https://github.com/user-attachments/assets/09ba2b38-7af3-41fa-a64f-ce3c4fd8548d"
/>
---------
Co-authored-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
## Purpose
Allows a user to find more easily the other users they search, with the
following order of priority:
- users they already share documents with (more recent first)
- users that share the same full email domain
- ~~users that share the same partial email domain (last two parts)~~
- ~~other users~~
Edit: We need to ilter out other users in order to not reveal email
addresses from members of other organisations. It's still possible to
invite them by email.
Solves #1521
## Proposal
- [x] Add a new function in `core/utils.py`:
`users_sharing_documents_with()`
- [x] Use it as a key to sort the results of a basic user search
- [x] Filter user results to avoid reveal of users (and email addresses)
of other orgs or that have not been interacted with.
- [x] User research through "full" email address (contains the '@') is
left unaffected.
---------
Co-authored-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
Not every project requires silent login.
This commit adds a new feature flag
FRONTEND_SILENT_LOGIN_ENABLED to enable or
disable silent login functionality.
Most of Docs app is configured thanks to environment
variables, except the url in the email that
was from the django site table.
Now we can set it with DJANGO_EMAIL_URL_APP
environment variable to have a better consistency.
We keep the previous way to avoid breaking
changes.
The max size and allowed extensions for document
import are now fetched from the application
configuration.
This ensures consistency across the app and
allows for easier updates to these
settings in the future.
