When docs is ran using an ASGI server, the django StreamingHttpResponse
expect to consume an async generator. If a sync generator is used, it
fully consumes it and then return all the content and we loose the
benefits of using a StreamingHttpResponse.
The app expect threads to always have
comments, but in some cases threads can end up without
comments, which causes crashes. This PR adds a
check to delete orphaned threads without comments
to avoid this issue.
When all the tests are ran, there is a PytestWarning log present to warn
us that there is remaining database connection open and can not be
closed. It appears that the tests about concurrence are responsible of
leaving connection open. We need to manually close them in these tests.
Added:
- ✨(backend) support creating subdoc from file
- ✨(frontend) comment side panel
- ✨(buildpack) add PaaS deployment support, tested with Scalingo
- 🔧(backend) allow configuring settings
OIDC_OP_USER_ENDPOINT_FORMAT
- ⚡️(helm) create a dedicated svc and deployment for
yprovider converter
- ✨(backend) allow to leave a document
- ✨(frontend) add the presenter mode
- 📈(backend) create a utils to capture event with posthog
- 🔧(backend) new setting DOCUMENT_ALL_ENDPOINT_ENABLED
Changed:
- ♻️(backend) allow global search in sub documents
- ✨(backend) add a breadcrumb in the search response
- ♻️(frontend) move doc action buttons to fix toolbar
- ♿️(frontend) add aria-hidden to decorative avatar SVGs
in share modal
- 🏗️(frontend) move comments to its own folder feature
Fixed:
- 🐛(docs) run migration 0027 without superuser role
- 🐛(backend) prevent admins/owners from overwriting other
users comments
- 🐛(y-provider) return empty output when converting empty
Yjs document
- 🐛(backend) use computed_link_reach in
handle_onboarding_document
- 🐛(frontend) fix toolbar blocknote hidden
- 🐛(frontend) fix application crashes when using
GTranslate and zoom
- 🐛(frontend) fix emoji pdf not matching
- 🐛(backend) fix UnorderedObjectListWarning for
DocumentAskForAccess viewset
In the tests there is a UnorderedObjectListWarning, we also find this same
warning in the logs in production. This warning appears because the
queryset used by the DocumentAskForAccess paginator is not ordered and
can lead to inconsistent result. Ordering the queryset fix this warning.
We want to allow or not the usage of the /documents/all/ endpoint. It has
been created to be used for an other purpose than the js client. For
those who don't use it this new settings allow to disable it. By default
it is set to True to keep the same behavior.
Add a presenter overlay that turns the current document into a
slide deck. The editor's blocks are snapshot at open time and
split into slides on each divider; navigation is driven by
keyboard shortcuts and a floating bar with browser fullscreen
support. The overlay is wired to the doc header toolbox via a
new "Present" entry, lazy-loaded to keep the editor bundle lean.
The feature "doc-editor" start to be quiete big,
and the comments related code is quite a lot.
We move all the comments related code to a new
folder "doc-comments" to make the code more
organized and easier to maintain.
The header could hide the formatting bar of the
blocknote editor. This commit improve the z-index
of the header to ensure that the formatting bar
is always visible.
Error boundary page is used when an error occurs
in the application, this page can have states still
set, causing side effects depend the component that
throw the error. In this case, the left panel is
still loaded and try to load the tree, but the
provider is not set, causing a crash error inside
the error boundary page.
We don't need the left panel in the error boundary
page, so we can just not load it to avoid this issue.
The emoji algorithm to find the emoji was not totally
correct, it could choose a bad family of emojis.
The first part of the emoji "code" is
what should be used first to find the "family" of the
emoji, we can then try to find the exact match in
that specific family.
We want to allow users to leave a document where they have an access or
they have visited creating a link_trace. All subdocuments should also be
leaved at the same time.
To know if the user can leave a doc we have to check when computing the
abilities if a record is existing in the LinkTrace table. This is a N+1
query situation. To avoid it, we added an annotation in the
DocumentQueryset like we already do to annotate the user role.
There is one edge case where the annotation is made to soon, it is when the
user is visiting a document for the first time, the `get_object` add the
annotation and in the permission, we compute the abilities. The `leave`
property is False because the entry in the LinkTrace table is not made,
when the serializer ask for the abilities again, it is still False. So
in the `retrieve` method in the viewset we force the
`user_has_link_trace` to the correct value.
We are modifying the doc header, we are moving elements
to a floating bar. In this case, we are moving the
doc toolbox to the floating bar.
We adapted the toolbox with the updated actions.
We update the doc header to use all empty space.
We moved the emoji part from the toolbox directly
to the header.
Replace ContentTypes enum with a structured object mapping MIME types
to their file extensions, removing the manual switch-case in useImport
Signed-off-by: Stephan Meijer <me@stephanmeijer.com>
The convert handler returned 500 "No valid blocks were generated" whenever
the reader produced an empty blocks array, which is the normal state of a
freshly-created Yjs document. Treat empty input as a valid case and return
200 with an empty body for every supported output format.
Signed-off-by: Sylvain Zimmer <sylvain@sylvainzimmer.com>
We want to isolate the converter service built in the yprovider server in
order to not mix conversion and collaboration together. We decided to
create a dedicated service and deployment, by default they use the
values defined in yProvider values but all can be overridden in
yProvider.converter.
Also, by default these new service and deployment are not enabled, they
must be explictly enabled and configured. If not it works like now with
only one service and deployment for the yProvider server.
In the model method User::_handle_onboarding_documents_access we do not
allow using documents with restricted link_reach to be added as
onboarding documents. To check the real link_reach of the document, we
must use instead the computed_link_reach to be sure that a sub document
can also be used and compute its correct link_reach.
In the search response, we want to display a breadcrumb for every
document returned. For this we added a "parents" property containing a list
documents, all are the parents of the current document (ordered by their
depth). With this we can easily create a breadcrumb.
The global search endpoint filtering by title in the database was not
searching for accessible sub documents. We change the how global search
is made: it is now returning accessible sub documents matching the
title query.
Add comment side panel to the right panel.
We will be able to manage the threads of the
document and see the content of the comments in
a side panel.
The advantage of this approach is that we will
be able to:
- see the comments that have been removed because
of deleted text
- see the resolved comments
- see the unresolved comments
Admins/owners could overwrite another user's comment text and silently
take ownership of it.
Split write permissions: destroy stays broad (author + admin/owner) for
moderation.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
The djang-lasuite package manages the settings
OIDC_OP_USER_ENDPOINT_FORMAT allowing to enforce the format use by the
user endpoint. By default it is set to `AUTO`.
The migration 00227 must be run with a postgres superuser, most af
managed postgresql database can not be run using this kind of user.
Ti fix this, we use postgresql unnacent function instead of accessing C
function.
The children/ endpoint was missing file upload support that the root
documents endpoint already had. Added file-to-YJS conversion handling to
subdocument creation.
Added
- ⚡️(frontend) add skeleton on content loading #2254
- ⚡️(frontend) close websocket connection when user change tab #2264
Changed
- 🏗️(core) migrate from pip to uv
Fixed
- 🩺(project) reload app if front and back unsync #2276
- 🐛(frontend) fix patch and comments #2273
- 🐛(frontend) interlinking are exported correctly in print mode #2269
- 💬(frontend) add missing link in onboarding description #2233
- 🐛(frontend) sanitize pasted and dropped content in document title #2210
- 🐛(frontend) Emoji menu doesn't display above comment box #2229
- 🐛(frontend) Block menu doesn't stay open on 1st line #2229
- 🐛(frontend) The "+" on the first line of a new doc doesn't work #2229
- 🐛(backend) manage race condition between GET and PATCH content #2271
- 🐛(backend) replace document creation table locks with retry strategy #2274
Security
- 🔒️(frontend) sanitize color during collaboration #2270
We observe some cases where the frontend and
backend versions can get out of sync, which can
cause issues.
To mitigate this, we want to implement a mechanism
that detects when the frontend and backend
versions are mismatched and triggers a
reload of the application to ensure they are in sync.
We have situation where the number of locks in the database can increase
dangerously creating deadlock situation. To remove this situation we
decided to change the strategy to manage document creation concurrency.
We decided to use a retry strategy, trying to create the document
multiple times while a usable path is found. To avoid having an
inifinite loop, we use a max_attempts counter configurable using the
setting TREEBEARD_PATH_COMPUTE_RETRY_MAX_ATTEMPTS
When a PATCH and a GET on the content endpoint are made at the same time
for different users a race condition can happen and the metadata
returned
by the S3 head_object can be outdated when the object is fetched leading
to an error raised because the Content-Length header does not match the
size of the response body. To avoid this, we no longer used head_object
followed bu get_object, we have to manage
everything in one call with the get_object. The get_object also accepts
as parameters an etag or last-modified header and will return a 304 if
the content has not changed, so we can use this to not return the entire
body if this one has not changed.
When a user change to another tab, after a delay of "inactivity"
we disconnect the user from the collaboration server.
When the user come back we reconnect to the server
again. It will reduce the connection to the collaboration
server and reduce outburst during reconnection during
a ingress ngnix restart.
When we load the comments we have to notify the
subscribers of the DocsThreadStore. This generates
a Yjs transaction that is currently treated as a
user-initiated content change that will trigger
a patch request when the doc will try to save.
We now update the transaction origin when we notify
the subscribers so that we can reliably identify
and ignore those transactions in the useSaveDoc
hook.
To improve security we sanitize the color used
for collaboration presence to ensure it's a valid
hex color. If the color is not valid, we generate
a random color instead. This prevents potential
issues with invalid color values being used in the UI.
We added a missing link in the onboarding step
description to direct users to ready-made templates f
or common use cases. This enhancement aims to improve
the user experience by providing easy access to
resources that can help users get started quickly
and customize their workflow efficiently.
Content is longer to load than other parts of the
editor because of the connection with websocket
to the collaboration server. To improve the user
experience, we add a skeleton on the content part
of the editor while the others parts are displayed.
Added
- ✨(backend) create a dedicated endpoint to update document content
- ⚡️(backend) stream s3 file content with a dedicated endpoint
- ✨(backend) allow to use new ai feature using mistral sdk
Changed
- ♻️(backend) rename documents content endpoint in `formatted-content` (BC)
- 🚸(frontend) show Crisp from the help menu #2222
- ♿️(frontend) structure correctly 5xx error alerts #2128
- ♿️(frontend) make doc search result labels uniquely identifiable #2212
- ⬆️(backend) upgrade docspec to v3.0.x and adapt converter API #2220
- ✨(backend) make forward auth request uri header configurable #2241
- ♿️(frontend) fix sidebar resize handle for screen readers #2122
Fixed
- 🚸(frontend) redirect on current url tab after 401 #2197
- 🐛(frontend) abort check media status unmount #2194
- ✨(backend) order pinned documents by last updated at #2028
- 🐛(frontend) fix app shallow reload #2231
- 🐛(frontend) fix interlinking modal clipping #2213
- 🛂(frontend) fix cannot manage member on small screen #2226
- 🐛(backend) load jwks url when OIDC_RS_PRIVATE_KEY_STR is set
- 🐛(backend) Prevent moving document to its own descendant or self #2208
- 🐛(backend) return 400 when restoring a non-deleted document #2225
We give the possibility, for the new ai feature, to choose between using
the OpenAI or Mistral sdk. For instances having access to the mistral
infrastructure, using it is mor appropriated than using the openai
compatible chat model.
Catch RuntimeError raised by Document.restore() and translate it into a
DRF ValidationError so callers get a 400 instead of a 500, when trying
to restore a non-deleted document.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
When attempting to move a document to itself or to any of its
descendants, the server would crash with a 500 Internal Server
Error.
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
Validate emojis in ReactionSerializer (previously accepted
any string), preventing multiple emojis or text uploads in
a single reaction
Signed-off-by: Mohamed El Amine BOUKERFA <boukerfa.ma@gmail.com>
Anthony LC