We want to limit what type of file can be uploaded. For this we use an
allowed list of file extension. This allowed list can be disabled using
a setting and this list is configurable using a setting. When enabled,
file with no extension and hidden file are also rejected.
The link reach/role on an item is derived from its ancestors. Pre-computing
values for each item allows passing them in a mapping that avoids repeated
queries when computing each item's abilities.
We can calculate a child's number of accesses by adding the count for its
parent item and adding the count directly attached to it. This addition
can be done in the same list query via an annotation so we don't generate
N+1 queries (N equals 200 by default on the children view).
Some calc files have application/octet-stream mime type sometimes, like
the .numbers file for instance. So checking the mimetype does not make
that much sense.
The mimetype detection were sometimes not accurate and return a wrong
indication. For example pptx files were identified as an archive because
they are zip files. This version relies on the file extension is the
gussed mimtype is not relevant.
Guessing mimetype from file few bytes is called magic bytes
https://en.wikipedia.org/wiki/List_of_file_signatures
In the ItemViewset::perfom_update method, we trigger the rename_file
task when the title of the item has changed. On a PATCH request, the
title can be absent in the payload leading to renaming the file with a
None value. We have to chank first of the title is present in the
validated_data and also check first in the renamin task if the provided
value is not empty or None.
Fix # 477
Before those route were using the item permission, causing such
error: AttributeError: 'ItemAccess' object has no
attribute 'ancestors_deleted_at'.
Update to use the appropriate permission.
We want to allow instance to not setup email sending without causing
crashes. Also add the envvar to the test setup so that the existing
tests that were running without DJANGO_EMAIL_HOST to None still
work.
Add setting FEATURES_INDEXED_SEARCH that allows to disable the
search of indexed files while using the indexation tools.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
When indexer service is not configured, the search view should work
event with a disabled OIDC_STORE_ACCESS_TOKEN.
Disable token storage for the unit tests.
Add bin/fernetkey that generates a key for the OIDC_STORE_REFRESH_TOKEN_KEY
setting.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Use SEARCH_INDEXER_CONTENT_MAX_SIZE as limit (in bytes) for the file content.
Fix default configuration of OIDC_STORE_ACCESS_TOKEN
Signed-off-by: Fabre Florian <ffabre@hybird.org>
In Drive the search should find files in the trashbin for a limited amount
of time. So the soft-delete cannot disable a indexed entry
Be more strict on mimetype patterns.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Use nb_results instead of page/page_size argument for /search API.
Add --batch-size argument to the index command.
Fix an issue in SearchIndexer.has_text when item.mimetype is empty.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Keep ordering by score from Find API on search/ results when the
fulltext search is enabled.
Refactor pagination to work with a list instead of a queryset
Fix Changelog
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Set SEARCH_INDEXER_CLASS=None as default configuration for dev.
Add documentation for Find service setup.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Reduce the number of Find API calls by grouping all the latest changes
for indexation : send all the items updated or deleted since the
triggering of the task.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
When the file indexer is enabled (SEARCH_INDEXER_* settings are set) use it
in place of the title filtering.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Add a celery task that send an item changes to the Find API
A simple flag is set in cache for an amount of time that block any other task
creation and do the throttle.
The SEARCH_INDEXER_COUNTDOWN setting gives the number of seconds between tasks
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Add SearchIndexer service that handles indexation & search API calls to Find
Add SEARCH_INDEXER_* settings to configure it.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Add a new Django app 'demo' that contains the command 'create_demo'
Generate dummy users and files with existing factories.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Search in Drive relies on an external project like "La Suite Find".
We need to declare a common external network in order to connect to
the search app and index our documents.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
We have some instances that requires more open resource server features.
This change adds a setting that allows to cherry-pick
which api routes needs to be available or not.
- Updated LeftPanelMobile component to wrap Gaufre and UserProfile in a
new container for improved layout.
- Added styles for the new container to position it at the bottom of the
panel.
- Modified Gaufre component to conditionally render based on user
authentication status.
- Updated the @gouvfr-lasuite/ui-kit dependency to version 0.18.4 in
package.json and yarn.lock.
- Added yarn engine requirement to package.json for consistency across
environments.
- Updated ExplorerTree styles to use flexbox for better layout
management.
- Refactored Header component to integrate new UserProfile and Gaufre
components.
- Simplified LeftPanelMobile by removing unnecessary components and
adding UserProfile.
- Introduced Gaufre and UserProfile components for improved user
experience and language selection.
- Enhanced LanguagePicker functionality to support user-specific
language settings.
- Introduced a new gaufre theme with widget paths and API URLs.
- Updated the getComponents function to include gaufre theme components.
- Enhanced styles to support the new gaufre theme in cunningham-tokens.
- Added console logs for debugging theme configurations.
Added
- ✨(backend) add more info on the item detail in the admin
- ✨(backend) add an admin action to trigger new file analysis
- ✨(backend) add a command to update file_hash in malware_detection_info
Fixed
- 🐛(front) fix responsive item row
- 🐛(front) fix responsive tree
We have migrated to uv but pip is still installed in the python docker
image. We must upgrade it to avoid having an outdated version of it with
security issues.
We want to update existing item in suspicious upload_state and to the
malware_detection_info the file hash. This is an info we need to track
suspicious files.
Refactored the AppExplorerBreadcrumbs component to utilize a breadcrumb
query for improved navigation. Added mobile-specific breadcrumbs and
integrated workspace icons. Introduced new pages for public and shared
items, leveraging the useInfiniteItems hook for pagination. Updated
styles for better responsiveness in the explorer interface.
Introduced a new custom hook, useInfiniteItems, leveraging react-query
to handle infinite scrolling and pagination for items based on specified
filters. This hook simplifies the process of fetching and managing
paginated data in the frontend application.
We added an action in the item list allowing to trigger new malware
analysis from the beginning. This can be usefull if an analysis has
failed with a 500 or an unknow error.
We lack information on the item admin. Let's add some fields to
allow filtering, and also display facets. Also added ability
to update the upload_state in case we need to do some debugging.
Manuel Raynaud