The yhub service will need our public key in order to validate the jwt
token we will used. We choose to expose a jwks endpoint as it is a
standard wat to do this.
The duplicate endpoint used to respond with only `{"id": ...}`. It now
returns the complete duplicated document representation, consistent
with the other document detail endpoints, so the frontend doesn't have
to make a follow-up request to get the new document's data.
This required setting `is_favorite` explicitly on the duplicated
document before serializing it: it is normally set by the
`annotate_is_favorite` queryset method, which the newly created
document never goes through. Being a read-only serializer field, it
was silently dropped from the response instead of raising an error. A
document can't be a favorite right after being created, so it is set
to `False` directly.
Dropping a file larger than the allowed size showed a bare "unknown
error" in the editor. The proxy in front of the API cuts the request
and answers a 413 with an HTML body, so errorCauses threw while
parsing it as JSON and no cause ever reached the error panel.
The size limit the backend already enforces is now exposed by the
config endpoint, and the editor checks the file against it before
sending anything, with the same toast wording the document import
uses. errorCauses no longer throws on a body it cannot parse, and a
413 without a usable cause falls back to an explicit message, which
covers the instances whose proxy limit is lower than the application
one.
The size formatting duplicated in the import hook moved to a shared
util.
Signed-off-by: risk-alt <aldu6974@gmail.com>
Added:
- 🔧(backend) fine tune redis cache options
- ✨(frontend) make the full last-update date available
- 💄(frontend) redesign email confirmation standalone page
Changed:
- ⬆️(backend) upgrade celery to version 5.6.3
- ⚡️(backend) stop using LEFT(value, LENGTH(path)) in sql queries
- 🚚(project) switch docspec image to ghcr.io/docspec/api
- 🚚(global) move favorite documents API endpoint
to `/documents/favorites/`
Fixed:
- 🐛(backend) skip session creation for the liveness probe
- 🐛(frontend) preserve page titles when adding an emoji
- 🐛(frontend) scroll to the linked block in read-only documents
- 🐛(frontend) hide the selection highlight on presenter images
- 🐛(y-provider) prevent process crash on malformed websocket frames
- 🐛(frontend) keep commented text sharp when printing to PDF
- 🐛(docker) pull minio images from quay.io
- ♿️(frontend) restore presenter focus trapping after share links
- 🐛(frontend) export any raster image supported by the browser to a PDF
Add a configuration knob for the trace sampling rate, so we can
enable tracing on middleware and cache spans when debugging slow
requests in production.
Sampling is set to 0 by default, so tracing stays fully off unless
explicitly enabled.
Copied from suitenumerique/meet#1690
We decided to move the DockerflowMiddleware higher in the middleware
list to prevent future access to the database or redis in other
middleware that can have an impact on the liveness probe.
The nb_accesses cache TTL was very short, 30 seconds. That mean that the
user will hit the cache for a very short period and the cache is
probably not be hit. This is what we can see in the slow queries from
the pg_stat_statements table. The query to compute the nb_accesses is
executed a little bit less than the number of queries to list or
retrieve documents, meaning the cache is not used.
The filtering made in the DocumentViewset.get_queryset method is not
optimal and lead to a full scan of the Document table. The heavy part is
on the filtering on what the user can access between the accesses and
the link traces. To have better performance we make an union operation
of both document_id list and the filter the id on this list. Postgresql
will use the index on the id column.
Comparing path with LEFT(value, LENGTH(path)) makes a sequential scan on
all the Document table, the more this table grow, the more the query
using it will be slow. We dediced instead to lookup on the path
extracting all ancestors path for a given document and then make a path
IN statement to use the index existing on the path column.
We want to configure other options on the redis cache. By default there
is no timeout on the connection to socket and no timeout for read/write
operations. We set default values in all caches used in production. The
settings IGNORE_EXCEPTIONS differ between the default and the session
cache. Activating it behaves like a missed cache. Enabling it for the
session should lead to unwanted side effects, by returning falsy on the
session creation, a retry mechanism of 10000 attempts is made in the
SessionStore.create method, the request can stay in this loop for a long
time.
The readiness probe should also not create a new session. A new session
will live in redis and increase the number of keys inside it for
nothing. The readiness path is isgnored in the ForceSessionMiddleware
The ForceSessionMiddleware force the session creation, we want to
ignore it when the request is the liveness probe. The liveness probe
must not check if redis is available, this is the readiness probe job
Added
- ✨(frontend) export presenter slides as PDF #2487
Fixed
- 🐛(frontend) hide Leave in the doc menu when not logged in #2626
- 🐛(backend) allow to configure settings DATA_UPLOAD_MAX_MEMORY_SIZE
We removed previously whitenoise because it was not working with asgi
application. By removing it we also removed the way to serve the static
files in the application. There is an existing fork of whitenoise,
servestatic, that manage async application and we can use it to serve
static files.
Release 3.17.2 of DRF now takes care of DATA_UPLOAD_MAX_MEMORY_SIZE
and is checked when the body request is parsed. Before that, DRF wasn't
using it at all and we were only looking for custom settings linked to
the media and conversion file upload. We must now also configure this
setting.
Added:
- ✨(frontend) Add "Copy link to block" feature
- ✨(frontend) add word count to doc header toolbox
- ✨(frontend) add find and replace feature to the editor
Changed:
- ♿️(frontend) use anchor links for interlinking sub-documents
- ✨(frontend) reset side panel state between documents
- ♿️(frontend) announce search loading state for screen readers
- ♻️(frontend) change favorite to star
- 🚚(frontend) add doc move to doc options
- ♻️(frontend) unified menu
- ♿(frontend) hide decorative emojis in document titles from SR
- ♻️(frontend) save the doc with a keepalive request when
leaving the page
Fixed:
- 🐛(frontend) fix clipped formatting toolbar in new comment
composer
- 🐛(backend) fix duplicating a document that has no content
- 📄(frontend) allowed partially export when MIT
- 🐛(backend) manage async support for Docs custom middleware
Removed:
- 🔥(backend) remove whitenoise package
whitenoise middleware is failing a lot with a cancelled exception from
asyncio. Using whitenoise is not needed in our case, we are just serving
an API with django and DRF. We decided to completely remove it.
Docs have 2 custom middlewares, both are only managing sync
requests. With Python 3.13 we didn't have any errors, but
since we upgraded to Python 3.14, we have a CancelledError
exception. We decided to use the MiddlewareMixin from Django
that is sync and async capable and will be responsible for
executing both middleware in the good mode.
Document.content reads from object storage and returns None when nothing
was ever written there. That None, raised "content should be a string.",
so the duplicate endpoint answered a 500. Default to an empty string instead.
Signed-off-by: BOUKERFA Mohamed El Amine <boukerfa.ma@gmail.com>
Added:
- ♿️(frontend) restore skip to content link after header redesign
- 🌐(i18n) rename cn_CN to zh_CN, add eo_PL and zh_TW locales
- ✨(backend) conditional email notification in server to server api
- ✨(backend) profile api using django-silk
Changed:
- ♿️(frontend) use semantic `<dl>` structure in document info card
- ⚡️(frontend) replace onboarding assets with webm and webp
- 💄(frontend) use the same highlight color for cells and moves
- ⚡️(backend) optimize media_auth endpoint
- 🚸(frontend) print from document options menu
Fixed:
- 🐛(frontend) refresh pins after document deletion and restoration
- 🐛(frontend) redirect homepage to login when homepage feat
is disabled
- 🐛(backend) ignore CSPs for API docs in development
- 🐛(frontend) export images embedded with a relative url
- 🐛(y-provider) fix sentry init
- 🐛(backend) handle object storage metadata keys case-insensitively
- 🐛(keycloak) fix database env variables in the self-hosting example
- 🐛(helm) show the database error while jobs wait for it to be ready
A new language has been added to the Django system,
allowing for Polish translations and localization
support.
We need to initialize the Polish language files before
being able to download the translations from
Crowdin. This commit includes the initial setup for
the Polish language, including the necessary configuration
files and directory structure.
Once the sql queries improved we have still a bottleneck on large
concurrent requests on this endpoint. We notive in the profiles generated
that lot of time was spent in creating a new s3 client instance on each
request. django_storage use a thread local cache for signed and unsigned
connection, but using uvicorn we have a new thread for each request, so
on each request a new s3 client is generated and it appears to be an
expensive operation. To fix this issue, we cache the client and share it
accross all the thread and requests.
On the media_auth endpoint the first bottleneck we have is with
postgresql. We are looking for too much data and no index is used on the
attachments colum. When the lookup filter on the attachement columns, a
full scan is made on all the document table looking for each element in
the array, this operation is really expensive. To fix this we created a
GIN index on the attachments column. Also the readable_per_se lookup was
selecting too much data combined with the filter_descendants function.
We remove the usage of the filter_descendants, we choose to first fetch
all the paths where the attachment is found, this operation is fast
thanks to the new index, split all the paths in candidate paths and then
filter readable_per_se queryset with these paths. All these
modifications make the endpoint faster.
We have performance issue with the media_auth endpoint. We need to
monitor and profile this endpoint to understand what is happening. We
decided to install and configure django-silk. By default django-silk is
not enabled and not accessible, once enabled migration must be run and
the /silk/ endpoint will be accessible only using a super user.
To have relecant profiles and monitor we must have a database comparable
to a production one because we are facing a scaling issue. Unless a
million documents probably nothing is happening, for this we added
several management commands to help us. A first one to anonymize a
database than can be reused without any risk to leak PII data, can be
called using python manage.py anonymize_database with some mandatory
options. An other command to generate a database profile, this command
should be run on a production database, this profile can then be used by
the generate_volumetry to populate a development database with same
volumetry than describe in the profile.
Object storage metadata keys are case insensitive per the S3
specification, but implementations don't agree on the case they give
back. When head_object returns a capitalized "Status", updating the
attachment status added a second, lowercase entry instead of replacing
it, and the copy request ended up carrying two x-amz-meta-status
headers. Ceph RadosGW loses one of them behind a proxy, which
invalidates the request signature.
The same assumption was made when reading the status back in media-auth
and media-check, where an attachment stored on such a backend stayed in
"processing" forever.
Metadata read from the storage is now normalized to lowercase keys
before being consumed or copied over.
Signed-off-by: risk-alt <aldu6974@gmail.com>
With Content Security Policies activated, swagger (and redoc)
auto-generated API documentation is no longer accessible even locally.
To restore this feature, we've excluded CSP for related URLs only in
the `Development` configuration.
Since pydantic-ai has ben upgraded deprecation warning were present in
the logs. This commint change how the instrumentation is enabled to be
compatible with hoe pydantic-ai manage it. Also prompt system are
ignored when they come from the front, like we are doing and dropped. To
maintain the system prompt we use the instruction parameter instead.
We upgraded pydantic-ai-slim to the latest version and add mistral has
extra dependency. Doing this, the direct dependance to mistral can be
removed and will be managed with pidantic-ai-slim package. The import of
mistralai has changed, we have to updagre them from mistralai to
mistralai.client
Add send_notification_email flag (defaults to True) on
the ServerCreateDocumentSerializer so that we can bypass
sending the email notification when creating a document
with the server to server API.
Signed-off-by: Florent Chehab <flo@nextmile.fr>
We have renamed the Chinese locale from `cn_CN`
to `zh_CN` to align with standard locale naming
conventions.
Additionally, we have added support for two new
locales: Esperanto (`eo_PL`) and
Traditional Chinese (`zh_TW`).
Changed:
- ♻️(backend) reset collaboration connection in cascade for all children
Fixed:
- 💄(frontend) fix some UI/UX in the left panel
- 🐛(frontend) fix tree dnd firefox
- Collapse panel button is now:
- fully clickable,
- the title is hidden
- when the doc title is visible
- when small mobile the title is hidden
- Add button login when logout whatever the device
- Fix flickering logo
- New icon name to avoid cache
When an access is updated or deleted, or a link_configuration is
changed, the collaration_service is used to reset the connection in the
collaboration server. As accesses and link_configuration are inherited
in a Docs tree, if the user is connected to a child, the connection is
not reset. This commit fix this issue by calling the reset on every
children in the tree.
In the settings, the logging has set the propagate property to False.
While writing tests it wasn't possible to assert log was made using the
caplog fixtures because this propagate is set to False. Changing it to
True allow to correctly tests that a log is made using the caplog
fixture.
Added:
- ✨(y-provider) preserve callouts, PDFs, page breaks, interlinking
links and commented text on HTML/markdown export
- ✨(frontend) add a user menu
- ✨(frontend) new header and responsive harmonization
- ✨(backend) add management command to reset a Document
Changed:
- ♿️(frontend) hide mobile left panel from screen readers when collapsed
- ♿️(frontend) enable blocknote heading ids for toc anchors
- ♿️(frontend) focus export modal on format select
- ♿️(frontend) configurable legal submenu in HelpMenu, remove Crisp
- ♻️(frontend) new create button for docs
- ♿️(frontend) align search modal field label with placeholder
- 🚚(frontend) move Waffle to bottom left
- ♿️(frontend) remove redundant aria-label on table of contents links
- ♻️(core) fix typo in settings COLLABORATION_WS_NOT_CONNECTED_READY_ONLY
- ♻️(backend) scope document search by document id instead of path
Fixed:
- 🐛(backend) prevent owner from leaving a soft-deleted document
- 🐛(frontend) fix removed item in the tree
- 🐛(frontend) fix service worker causing reload on tab focus
- 🐛(backend) update restore ability for inherited deletion
- 🔧(dev) make the dev stack domain-agnostic
- 🐛(frontend) stop force index redirect when delete doc
- 🐛(frontend) fix CTA on move modal on mobile
Removed:
- 🔥(backend) remove unused default authentication backend
Manuel Raynaud