Cover both paths off the legacy Django store end to end: the lazy seed on
first access, and the migrate endpoint replaying every S3 version. The tests
need no database — the admin JWT short-circuits document authorization, so a
fixture is an S3 object on a random uuid — and read the timeline through
yhub 0.5.0's `Accept: application/json`, which spares python a lib0 decoder.
CI grows a valkey service and starts a collaboration server alongside the
backend test job; the tests skip themselves when nothing answers on the new
COLLABORATION_API_URL setting, so `make test` without the dev stack still
passes.
Writing them turned up three things worth fixing in the server.
Backend reads now seed too. getAccessType short-circuited on the admin token
before reaching the legacy store, so a server-side read of an unmigrated
document answered with an empty one, and a create-ydoc against it would have
written a second lineage beside the content the first user access was about
to seed in.
Seeding no longer decides access; the backend's answer alone does. A legacy
object that cannot be migrated — it does not decode, or it exceeds the size
we load — opens as a new document instead of denying, since no retry can fix
it and refusing would leave the document unopenable by anyone. The cause is
logged once per attempt with the bucket, key and stack, and every later access
logs that it admitted a caller without migrating.
That made the failure classifier dangerous, so it is inverted. It was an
allowlist of retryable errors — eight socket errnos — which left every way S3
can refuse (AccessDenied on a rotated key, NoSuchBucket, a region redirect)
counting as "this object is unusable". Denying, that was survivable; opening
empty, one misscoped credential would fork every document touched during the
window. Now only a failure raised while interpreting bytes we already hold is
permanent, marked at the throw site, and everything else answers a retryable
503. Guessing wrong that way costs a retry; the other way costs the document.
The admin seed is also fenced to the org and to main, like the user path
above it. The legacy store is branchless — {docid}/file is main — and the
bookkeeping is per document, so seeding ?branch=draft would have written
main's content into an orphan room and left the real one permanently empty.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The CollaborationService was doing nothing since we started the
migration to yhub, all the code using it is now removed. Also the
`can-edit` endpoint and all the safeguard mechanism relying on the
presence of other users connected to the websocket will not be used
anymore, it will be possible to replace all of this with yhub, so all
this code is also removed.
For now the only token we will need is ont with the admin claim set to
True. To not repeat the creation of this token again and again, we
created a dedicated method to issue this token in the JWTService class.
The yhub service will need our public key in order to validate the jwt
token we will used. We choose to expose a jwks endpoint as it is a
standard wat to do this.
Added:
- ♿️(frontend) restore skip to content link after header redesign
- 🌐(i18n) rename cn_CN to zh_CN, add eo_PL and zh_TW locales
- ✨(backend) conditional email notification in server to server api
- ✨(backend) profile api using django-silk
Changed:
- ♿️(frontend) use semantic `<dl>` structure in document info card
- ⚡️(frontend) replace onboarding assets with webm and webp
- 💄(frontend) use the same highlight color for cells and moves
- ⚡️(backend) optimize media_auth endpoint
- 🚸(frontend) print from document options menu
Fixed:
- 🐛(frontend) refresh pins after document deletion and restoration
- 🐛(frontend) redirect homepage to login when homepage feat
is disabled
- 🐛(backend) ignore CSPs for API docs in development
- 🐛(frontend) export images embedded with a relative url
- 🐛(y-provider) fix sentry init
- 🐛(backend) handle object storage metadata keys case-insensitively
- 🐛(keycloak) fix database env variables in the self-hosting example
- 🐛(helm) show the database error while jobs wait for it to be ready
A new language has been added to the Django system,
allowing for Polish translations and localization
support.
We need to initialize the Polish language files before
being able to download the translations from
Crowdin. This commit includes the initial setup for
the Polish language, including the necessary configuration
files and directory structure.
Once the sql queries improved we have still a bottleneck on large
concurrent requests on this endpoint. We notive in the profiles generated
that lot of time was spent in creating a new s3 client instance on each
request. django_storage use a thread local cache for signed and unsigned
connection, but using uvicorn we have a new thread for each request, so
on each request a new s3 client is generated and it appears to be an
expensive operation. To fix this issue, we cache the client and share it
accross all the thread and requests.
On the media_auth endpoint the first bottleneck we have is with
postgresql. We are looking for too much data and no index is used on the
attachments colum. When the lookup filter on the attachement columns, a
full scan is made on all the document table looking for each element in
the array, this operation is really expensive. To fix this we created a
GIN index on the attachments column. Also the readable_per_se lookup was
selecting too much data combined with the filter_descendants function.
We remove the usage of the filter_descendants, we choose to first fetch
all the paths where the attachment is found, this operation is fast
thanks to the new index, split all the paths in candidate paths and then
filter readable_per_se queryset with these paths. All these
modifications make the endpoint faster.
We have performance issue with the media_auth endpoint. We need to
monitor and profile this endpoint to understand what is happening. We
decided to install and configure django-silk. By default django-silk is
not enabled and not accessible, once enabled migration must be run and
the /silk/ endpoint will be accessible only using a super user.
To have relecant profiles and monitor we must have a database comparable
to a production one because we are facing a scaling issue. Unless a
million documents probably nothing is happening, for this we added
several management commands to help us. A first one to anonymize a
database than can be reused without any risk to leak PII data, can be
called using python manage.py anonymize_database with some mandatory
options. An other command to generate a database profile, this command
should be run on a production database, this profile can then be used by
the generate_volumetry to populate a development database with same
volumetry than describe in the profile.
Object storage metadata keys are case insensitive per the S3
specification, but implementations don't agree on the case they give
back. When head_object returns a capitalized "Status", updating the
attachment status added a second, lowercase entry instead of replacing
it, and the copy request ended up carrying two x-amz-meta-status
headers. Ceph RadosGW loses one of them behind a proxy, which
invalidates the request signature.
The same assumption was made when reading the status back in media-auth
and media-check, where an attachment stored on such a backend stayed in
"processing" forever.
Metadata read from the storage is now normalized to lowercase keys
before being consumed or copied over.
Signed-off-by: risk-alt <aldu6974@gmail.com>
With Content Security Policies activated, swagger (and redoc)
auto-generated API documentation is no longer accessible even locally.
To restore this feature, we've excluded CSP for related URLs only in
the `Development` configuration.
Since pydantic-ai has ben upgraded deprecation warning were present in
the logs. This commint change how the instrumentation is enabled to be
compatible with hoe pydantic-ai manage it. Also prompt system are
ignored when they come from the front, like we are doing and dropped. To
maintain the system prompt we use the instruction parameter instead.
We upgraded pydantic-ai-slim to the latest version and add mistral has
extra dependency. Doing this, the direct dependance to mistral can be
removed and will be managed with pidantic-ai-slim package. The import of
mistralai has changed, we have to updagre them from mistralai to
mistralai.client
Add send_notification_email flag (defaults to True) on
the ServerCreateDocumentSerializer so that we can bypass
sending the email notification when creating a document
with the server to server API.
Signed-off-by: Florent Chehab <flo@nextmile.fr>
We have renamed the Chinese locale from `cn_CN`
to `zh_CN` to align with standard locale naming
conventions.
Additionally, we have added support for two new
locales: Esperanto (`eo_PL`) and
Traditional Chinese (`zh_TW`).
Changed:
- ♻️(backend) reset collaboration connection in cascade for all children
Fixed:
- 💄(frontend) fix some UI/UX in the left panel
- 🐛(frontend) fix tree dnd firefox
- Collapse panel button is now:
- fully clickable,
- the title is hidden
- when the doc title is visible
- when small mobile the title is hidden
- Add button login when logout whatever the device
- Fix flickering logo
- New icon name to avoid cache
When an access is updated or deleted, or a link_configuration is
changed, the collaration_service is used to reset the connection in the
collaboration server. As accesses and link_configuration are inherited
in a Docs tree, if the user is connected to a child, the connection is
not reset. This commit fix this issue by calling the reset on every
children in the tree.
In the settings, the logging has set the propagate property to False.
While writing tests it wasn't possible to assert log was made using the
caplog fixtures because this propagate is set to False. Changing it to
True allow to correctly tests that a log is made using the caplog
fixture.
Added:
- ✨(y-provider) preserve callouts, PDFs, page breaks, interlinking
links and commented text on HTML/markdown export
- ✨(frontend) add a user menu
- ✨(frontend) new header and responsive harmonization
- ✨(backend) add management command to reset a Document
Changed:
- ♿️(frontend) hide mobile left panel from screen readers when collapsed
- ♿️(frontend) enable blocknote heading ids for toc anchors
- ♿️(frontend) focus export modal on format select
- ♿️(frontend) configurable legal submenu in HelpMenu, remove Crisp
- ♻️(frontend) new create button for docs
- ♿️(frontend) align search modal field label with placeholder
- 🚚(frontend) move Waffle to bottom left
- ♿️(frontend) remove redundant aria-label on table of contents links
- ♻️(core) fix typo in settings COLLABORATION_WS_NOT_CONNECTED_READY_ONLY
- ♻️(backend) scope document search by document id instead of path
Fixed:
- 🐛(backend) prevent owner from leaving a soft-deleted document
- 🐛(frontend) fix removed item in the tree
- 🐛(frontend) fix service worker causing reload on tab focus
- 🐛(backend) update restore ability for inherited deletion
- 🔧(dev) make the dev stack domain-agnostic
- 🐛(frontend) stop force index redirect when delete doc
- 🐛(frontend) fix CTA on move modal on mobile
Removed:
- 🔥(backend) remove unused default authentication backend
The search in a document tree was triggered by the usage of the document
path. The path is something guessable by incrementing it you can
discover public documents. We decided to change this to use the document
id which is not guessable and prevent discovering public documents.
Thanks to @maboukerfa for discovering it.
Following the recommendations of the design team,
we lightly improved some UI/UX elements to be
more consistent and user-friendly before the
next release.
It includes the following changes:
- add a background to the grid
- add a fadein to the header buttons
- reduce the generic size icon
- add a title to the doc item
Keycloak's hostname, its client's redirect/web
origins, and Django's CSRF trusted origins were
hardcoded to localhost, so the dev stack was
only reachable from that exact origin.
It is now possible to run the dev stack depending
what is configured in the env files.
The settings COLLABORATION_WS_NOT_CONNECTED_READY_ONLY contains a typo.
We don't want to see READY_ONLY but READ_ONLY. This commit fix the typo
and change everywhere in the codebase the settings name.
We replace the header by a floating header, to have more
space for the content and in fine to have more
harmony between the doc and the doc-grid, in mobile view
or not.
The layout have to be adapted, as well the legacy home
page.
We need a management command to reset a Document to an initial state and
deletes everything related to it. This command can be usefull to reset a
demo for example.
The authentication backend
mozilla_django_oidc.contrib.drf.OIDCAuthentication
is present in the default authentication classes for the REST_FRAMEWORK
settings. This backend should not be used by our application and can
lead to the usage of our main api with an access_token instead of the
cookie session.
We need to override the drf SessionAuthentication backend to implement
the authenticate_header method. Without this, a 403 status code is
returned, but it is not valid. It must a be 401
Updated the restore ability so that only directly deleted documents can
be restored. This prevents the restore action from being exposed for
child documents whose parent is deleted. This PR addresses #2127.
Owners could leave trashed documents via the leave endpoint, deleting
their `DocumentAccess` and all subtree access. When the last owner
leaves a document it creates an orphan as no owners remains in the doc.
Signed-off-by: BOUKERFA Mohamed El Amine <boukerfa.ma@gmail.com>
Added
- ✨(backend) add limit on distinct reactions per comment #1978
- ✨(frontend) leave a document #2410
- ✨(frontend) add top parent on sub docs search #1952
- ✨(frontend) unauthenticated users can search #2407
- ✨(backend) specific user delete method to delete its relations #2437
Changed
- 👷(CI) remove test-e2e-other-browser job #2404
- ♿️(frontend) use heading element for pinned documents section title #2380
- ♿️(frontend) use anchor links for table of contents entries #2390
- ♿️(frontend) improve presenter mode screen reader and keyboard support #2383
- ♿️(frontend) link export modal name to its heading #2422
Fixed
- 🐛(frontend) overlap of block menu dropdown #2406
- ⚡️(backend) fix N+1 queries when serializing thread comments #2415
In the admin there were 2 ways to delete a user : by the actions select
box and the button on its change view. Both are leading to a failure. To
fix it, both way are removed and we implemented a custom action calling
the user.delete method with all the specific workflow we implemented.
Deleting a user is not possible when it has created docs because the
on_delete on the Document class id RESTRICT and we don't want to change
it. We decided to have a specific workflow for correctly delete a user.
The document where the user is the sole owner must be deleted, the other
only the owner access must be deleted. For the remaining Documents where
the user is the creator, we set it to `null`, then the user can be
delete, remaining relations are deleted in cascade.
In the sharing module, the user model was used instead of working with
the user id. Only the user id is used in this module, forcing in the
module calling this functions to load the user model for nothing and
doing a N+1 query.
Prevent users from adding more reactions once the per-message
limit has been exceeded. It Disables reaction buttons
when limit is reached
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
