In the get_file_content action we want to log the received max expected
file size when this one is lower than the file size the user wants to
edit. This will help us to correctly configure the WOPI host.
On the duplicate action we are using the decorator @transaction.atomic
to be sure that the duplicated item and its accesses are correctly
created. The problem with that is that the celery task doing the file
duplication on the object storage can be executed before the transaction
is commited leading to an error saying that the duplicated_item does not
exists.
Added
- ✨(backend) make invitation validity duration configurable via env var
- ✨(frontend) enhance upload toast with progress, errors and cancel support
- ✨(frontend) add ErrorIcon component and support numeric icon sizes
- ✨(frontend) make file upload abortable in driver layer
- ✨(frontend) files preview v2
- 🔧(project) add DJANGO_EMAIL_URL_APP environment variable
Fixed
- 🐛(frontend) add actions menu on mobile My Files page
- 🐛(frontend) show actual selection count in hard delete modal
- 🐛(frontend) Responsive broken with long filters in search #659
- 🐛(front) set size and variant on trash navigate modal #666
- 🐛(frontend) fix uploads continuing after parent folder deletion
- 🐛(frontend) fix SDK picker link reach promotion
- 🐛(backend) route share invitation link to file view for files
- 🐛(frontend) fix "+ New" menu in read-only folders and virtual tabs
- 🐛(frontend) range selection freezes when there are many items in the list
- 🐛(backend) fix openapi schema for item access endpoints
- 🐛(backend) load jwks url when OIDC_RS_PRIVATE_KEY_STR is set
When the resource server is enabled and the backend used is
JWTResourceServerBackend, then the API should expose a JWKS endpoint
to share the RSA public key to the OIDC provider. Everything is made
in the Django LaSuite library, but the URL is not included in the
Drive URLs. This commit adds it when the setting OIDC_RS_PRIVATE_KEY_STR
is set.
Added
- ✨(frontend) add PDF viewer with thumbnail sidebar, zoom and page navigation
- ✨(frontend) integrate PDF viewer into file preview modal
- 📝(doc) add local network setup documentation
- ✨(global) add custom columns feature with configurable grid columns
- 🔒️(frontend) prevent search engine indexing
- ✨(backend) allow ordering items by creator full name
- ✨(frontend) add item duplication with polling and visual feedback
- ⚡(ci) shard e2e tests and cache playwright browsers
- ⬆️(frontend) upgrade cunningham-react and ui-kit to 0.20.0
- ✨(frontend) improve custom columns with sortable config and i18n
Changed
- 🏷️(sdk) update Item interface by adding url_permalink
- 🔧(backend) allow extra CSRF origins via env variable
- 🔧(nginx) serve .mjs files with correct MIME type
Fixed
- 🐛(backend) fix hard delete of files created by other users
- 🐛(backend) handle race condition on concurrent LinkTrace creation
- 🐛(frontend) fix React SVG attributes in AddFolderButton
- 🔧(scalingo) compile translation files at deploy time
- 🐛(frontend) fix trash items not refreshing after hard delete
- 🐛(frontend) show modal when clicking files in trash
- 🐛(frontend) fix toasts appearing above modals
We added a dedicated Ordering filter dedicated to the ItemViewset. On
this viewset we want to add an extra ordering if this one is not present
in order to have consistent ordering if the front order by title for
example and two title are equal.
We are using the drf_standardized_errors handler to have "nice" errors
for the frontend application. We decided to override the SchemaField
from the django_pydantic_field library in order to make it compatible
with this error handler.
We want to store in the user model the column_preferences parameters.
These parameters are saved by the front application. In order to
validate it, we use pydantic BaseModel and store it in a json field
Super useful for validation when handling unstructured dictionaries.
Follow qbey's recommendation and align with the
suitenumerique/conversation project approach to improve schema
validation and data integrity.
A user with at leat editor role, connected can duplicate a file item
which has its upload_state ready. The duplicate_file task is used to do
the copy of the file on the storage.
We plan to add an action to duplicate an item. We want to use a celery
task to make the copy using the s3 client. The task is retried 10 times
if failing and then the item is deleted if the copy fail after 10
retriees.
In order to add in next commit a duplicate action in the ItemViewset, we
add a `duplicate` entry in the item abilities. A user can duplicate a
file if he can update it.
In the item model the properties numchild and numchild_folder must be
maintain by doing a +/- 1 operation in several actions. Maintaining is
error prone, hard to catch and can have lot of root cause we can't
identified. We choose to use annotation instead and to compute the value
when needed.
When a user with priviledged role on an item but inherited he can not
invite external person. We refacto the invitation viewset, serializer
and permission to check roles against all the tree.
We use the Docker Hub Workflow to build and push
our images to Docker Hub, but to check if we
have vulnerabilities in our images as well.
When we are just checking for vulnerabilities,
we don't need to do all the builing steps.
This commit optimizes the workflow by only doing the
necessary steps when we are just checking for
vulnerabilities, so during pull requests we skip the build
steps, and we do not activate QEMU and buildx
Added
- 👷(docker) add arm64 platform support for image builds
- ✨(global) add create file from template feature
- ✨(global) add FRONTEND_CSS_URL and FRONTEND_JS_URL settings
- ✨(backend) add a download action returning the media url
- ✨(frontend) add right click feature
- ✨(backend) allow customization of wopi parameters
- ✨(backend) expose DATA_UPLOAD_MAX_MEMORY_SIZE in the config endpoint
- ✨(frontend) stop upload if file higher than DATA_UPLOAD_MAX_MEMORY_SIZE
- ✨(backend) reject uploaded file higher than DATA_UPLOAD_MAX_MEMORY_SIZE
Changed
- ✨(backend) allow root item creation on the external API by default
- ♻️(backend) set item read only in the mirror item admin detail
Fixed
- ✨(frontend) sync backend user language to browser on load
- 🐛(backend) fix WOPI PutFile to check stored file size
- 🐛(frontend) fix 401 page infinite redirect loop after login
- 🐛(backend) fix OIDC redirect allowed hosts format in dev config
- 🐛(global) update ui when renaming file from wopi editor
- 🐛(frontend) fix clipboard copy-paste in WOPI editor iframe
To not have to compute the download url, we decided to add a new
property url_permalink, this property will have the download action url
set in it when available.
The media url is subject to change if the file is renamed. If someone is
using it as a permalink, it will lose the ability to download the file.
We created a new download action, always returning a redirection to the
media url.
For celery beat deployment, we have to use a directory allowing write
permission to allow the scheduler to write its configuration. We use the
--schedule option to the /tmp repo.
The text/x-tex mimetype is used by .tex files containing LaTeX document.
The application/x-tex is already allowed but we also have to allow this
mimetype otherwise some document with the .tex extension can't be
uploaded.
Add a management command processing mirror task in pending task. The
number of processed task is limited with the --batch-size option and has
a default value to 100. This management command should be run
periodically like a cron to enqueue the pending tasks.
We need a management command adding in the mirror_item_task table
existing items that needs to be mirror. It look for all the items of
type files, not present already in the mirror_item_task table and then
insert them by batch of 100 in the table.
We want to have the information in the admin about the MirrorItemTask
records, filter them by status. An action is added to force triggering
the task again for a given record.
We implement a custom retry mechanism for the mirror file celery task.
We catch all boto3 exceptions, retrying 10 times and after the task is
stopped changing its status in failed and also the reason why it failed.
In order to be more fault tolerant, to not lose data if the redis is
reset, etc. We decided to put the item to mirror in a new table and then
the task will manage the status in the database.
This is the first step to then add tasks in the admin to retry or check
why the task has failed and also implement a retry strategy.
The Dockerfile must also be modified, we need the libmagic library to
build the image at the collect stage because the magic module is
imported in the storage module and all tasks are imported when the
application starts.
when an upload is ended, we have to call the mirror_file task. The
upload is made in the client browser so we can't use the S3 storage
class dedicated to the mirroring.
The mirror_file task is reponsible to copy an object from a bucket to an
other one. The buckets can be in different location, in different cloud
providers so we have to download it to reupload it. The `copy` method on
the s3 client only work when the bucket are in the cloud provider infra,
like for AWS.
We want to save a file on an other s3 bucket. For this we created a
dedicated storage inheriting from the S3Storage. This storage only
dispatch a message in a celery task. The logic of mirroring will be
implemented in this task in order to do it in an async way.
We don't want to use the s3_client anymore but to rely on the django
storages API. This first step will allow us to create a dedicated
backend for mirroring file saving.
The user's short_name field length was set to 20. This is not enought
and we have some users who cannot register because of that. We changed
this length to a higher one, 100, like the full_name.
Microsoft has a generic format for files like .xls .doc .ppt and the
mimetype returned is application/x-ole-storage. We want to add it to
generic type list in order to use the extension instead.
Since the link_reach is inherited and not set on every items, we can't
filter anymore, in the ItemViewset.get_queryset method directly on the
link_reach column, we must have the computed_link_reach. We introduce a
bug on the item list methods because too much link_traces record were
return. We have to remove the link_traces related to a restricted items.
For the same reason we not create an access when a child is created, we
have to not set value to link_reach allowing null value. A root will
have the link_reach set to restricted when created and then every
children are set to null. when the link_configuration will be updated,
descendants will be sync when the new link_reach is higher than the
previous one. Also move scenarios are impacted. When moving to the root,
we force the link_reach to restricted and when moving in an existing
item, we sync it with parents.
Once all accesses filtered, we need to check it we have to change the
max_ancestors_role for accesses that are inherited. An inherited access is will
not have the same item_id than the current item used in this viewset. In that
case, we use the role of the current access and the item_id it is related to.
To compute the abilities, the current item mu be aware of ancestors
link_reach and link_role. On the recents and favorite_list view if we
want to avoid N+1 queries we have to compute it for each deepest items
present in the final queryset, compute the ancestors link_definition and
then add it in the serializer. We added a new parameter to the
`get_response_for_queryset` method allowing to do this job.
The type filtering on the search endpoint was adding a workspace type to
filter only on top level folders and the the type folder was filtering
on the folder in depth. This behavior is not relevant anymore and all
this part can be deleted.
We added the user main workspace on the /me endpoint. This is useless,
we want now unifiy the list view mixing all top level items available
for the user.
Previously on the /items list endpoint, the main_workspace was removed,
only folders were also present and it was possible to filter by
workspace type (shared or public). We decided to revert all these
changes.
The ItemAvvessViewset list endpoint is really complicated and half of
the algo is made to cache the role the current user have on the tree.
This code is now removed and replaced by a queryset annotation allowing
to fetch the user_roles and then determine in the model the max role the
user have on the tree. It reduces the number of sql query by 1 and ease
the code comprehension
We don't need the whole accesses list for all the users on the list
endpoint. We want to return the last access, meaning the access
on the deepest item in the items tree.
For a given access we are able to compute the max ancestors role. We
also need the item_id related to this max_ancestors_role. We compute it
both in the vewset and in the models like we already did for the
max_ancestors_role
When an access role is updated and the new role is equal to the access
max_ancestors_role, we want to resync accesses. For this, the current
access is deleted instead of updating it. We don't want to have 2
explicits accesses sharing the same role.
The set_role_to was computed to have only roles strictly higher than the
max_ancestors_role. We want this behavior only for inherited accesses.
When the role is explicit, we want to allow the lower role in
set_role_to equal to max_ancestors_role
We need to know, for a given if the related access is explicit or
inherited. This is usefull on the list endpoint. This endpoint returns
the entire list of accesses add on the entire tree, we need to know when
retrieving this list if each access is explicit on the item or
inherited.
We don't need the whole accesses list for all the users on the list
endpoint. We want to return the last two accesses, meaning the accesses
on the two items the deepest in the items tree.
We open a specific endpoint to update items link configuration
because it makes it more secure and simple to limit access rights
to administrators/owners whereas other item fields like title
can be edited by anonymous or authenticated users with
much less access rights.
when an explicit access is creating and an other explicit access exists
higher in the tree, then we must validate the role and ensure this new
role is strictly higher than previous explicit accesses.
The max_ancestors_role was only computes in the ItemAccessViewSet.list
method to avoid having to compute it multiple times in the list and then
in the serializer. The result is save in the model instance has a cache,
but if not present whe computing the abilities outside the list method,
then the expected abilities are wrong. The set_role_to can't be computed
without it. So we search for the max_ancestors_role in the class itself
if not set.
We need to manage inheritence on item accesses. We backport all the work
made on the docs project. To list accesses, we look for all accesses on
all the item ancestors and the item itself.
The itemAccessViewset.get_queryset was the one used by the, now removed,
ResourceAccessViewsetMixin. We don't need it's complexity anymore and it
can be refactor to be clearer. The list function is added in order to
complexify it later.
There is a delay between the time the signature is issued and the
time it is checked. Although this delay is minimal, if the signature
is issued at the end of a second, both timestamps can differ of 1s.
Some permissions were made in the BaseAccessSerializer. Permissions
should not occur in serializers. We refactor this, moving permission
check in permissions classes but also in the ItemAccessViewset. Also,
there is no need to keep the BaseAccessPermission
Manuel Raynaud