The "may" level was verifying the peer cert and falling back to
cleartext on mismatch, which then bounced on STARTTLS-required
servers (e.g. Mandrill's SES-backed inbound returns 530 to MAIL
FROM in cleartext). Realigned on Postfix's documented behavior:
- "may": opportunistic TLS, no cert verification.
- "secure": mandatory TLS + CA chain + hostname check; defers if
STARTTLS isn't advertised or handshake fails.
- "encrypt" is dropped (replaced by "secure").
Also wires MTA_OUT_SMTP_TLS_SECURITY_LEVEL through both the direct
and relay paths — it had been declared but never read — and
collapses the four proxy_* kwargs + sender_hostname of
send_smtp_mail into a single SmtpProxy dataclass.
This allows to use S3-compatible object storage to offload blobs, making Postgres much lighter. We design for storing ~1B emails on a single instance.
We also take this opportunity to do model changes on blobs & attachments.
Migration 0027 is one-way, no going back after this one so check your backups.
Add `NEXT_PUBLIC_FEEDBACK_WIDGET_HOME_CHANNEL` env var to be able to
set a specific channel id to receive feedback from unauthenticated
users.
For configuration ease and backward compatibility, if
`NEXT_PUBLIC_FEEDBACK_WIDGET_HOME_CHANNEL` is not set, the home feedback
widget fallback to `NEXT_PUBLIC_FEEDBACK_WIDGET_CHANNEL`.
OpenSearch was returning 503/429 on delete_by_query under the load of
the periodic reindex. Each bulk_reindex_threads_task could fire up to
100 delete_by_query calls (one per chunk of 100 threads) to sweep
orphan messages, and bulk_delete_threads_task did one more to drop a
thread and all its children. delete_by_query holds a scroll context,
scans the index and refreshes per call — far heavier than the bulk
upserts running alongside it.
Tracking message deletes explicitly at signal time lets us replace
every hot-path delete_by_query with a bulk delete by _id:
- New search:pending_delete_messages set storing thread_id:message_id
pairs, fed by Message.post_delete (covers cascaded deletes too).
- New bulk_delete_messages_task issuing bulk DELETE actions with the
parent thread_id as routing.
- bulk_delete_threads_task rewritten to bulk DELETE thread parent docs
by _id; child message docs ride the new task via cascaded signals.
- _purge_orphan_docs and the per-chunk purge in reindex_bulk_threads
removed; reindex is now pure upsert.
Currently, when user has not its language set into local storage, we
retrieve
the default language through the navigator language. In some instance,
we would
like to enforce the default language. So we add a new env var
`NEXT_PUBLIC_FORCED_DEFAULT_LANGUAGE`, if this one is set to `true`, we
skip
the navigator.language and use the `NEXT_PUBLIC_DEFAULT_LANGUAGE` as
default
`bulk_data` method was missing some options (timeout, max_retries) so
sometimes when
the payload was heavy, the request can failed because the timeout was
too slow by default
(10s) and there is no `max_retries`. Furthermore, we build bulk payload
by chunking thread
but it did not check the payload bytes size, so in case of heavy
message, the payload could
be heavy. Now a max_bytes limit is set (50 Mib by default)
Opensearch index is updated each time a Thread, Message or MessageRecipient
is updated through signals. The current logic has performance issues has
n update of a resource will generate n celery task to update the same
resource... So this work aims to batch updates. Index is updated each
30s and resource ids is deduplicated to only update a resource once.
Furthermore, in an import context, the index will be updated only
when the import will be completed to prevent to spam the celery
worker with numerous indexation task.
ThreadEvent IM mentions previously lived only inside the event payload,
with no per-user tracking, so a user had no way to see or filter the
threads where they were mentioned. The new UserEvent model materializes
mentions as first-class records (one row per mentioned user per event),
reconciled by a post_save signal whenever a ThreadEvent is created or
edited.
ThreadEvent edits and deletes are now bounded by THREAD_EVENT_EDIT_DELAY
(1h default) so UserEvent records cannot drift out of sync with stale
audit data past the window.
Allow to configure a help center URL via the
NEXT_PUBLIC_HELP_CENTER_URL environment variable. The button
adapts based on what's configured:
- Help center only: opens external link directly
- Feedback widget only: opens widget directly
- Both: shows dropdown with both options
The button is now displayed in the header to save space in sidebar,
rather than in the left panel.
Images are embedded as base64 data URLs directly in the BlockNote content,
unlike the message composer which uses blob uploads + CID references.
This approach keeps templates and signatures self-contained without
requiring an attachment system.
Furthermore, a email-safe safe html exporter has been created to
serialize blocknote content into html.
A new backend setting MAX_TEMPLATE_IMAGE_SIZE (default 2 MiB) controls
the maximum allowed image size for these composers.
Add outbox conditional folder to display messages having message recipients with None, retry or failed state.
Inform visually the user is something is wrong with a message.
Allow user to update MessageRecipient from failed to cancelled to hide ui warning.
Updated the DJANGO_ADMIN_URL setting to ensure it does not end with a '/'.
It allows to keep simple redirect logic from the nginx proxy server and avoid
issues with django admin route paths.
Allow to users to display external images from their email
through a secure proxy endpoint to ensure security and privacy
and respect iframe csp policy.
Co-authored-by: =?UTF-8?q?Ri=C3=ABl=20Notermans?= <riel@mosa.cloud>
Add DRIVE_APP_NAME variable that gets exposed via the config endpoint then
update frontend translation to use this variable into it.
Co-Authored-By: jbpenrath <jb.penrath@gmail.com>
- Added multipart upload capabilities for EML and MBOX files to the
message imports bucket.
- Introduced new API endpoints for initiating uploads, completing
uploads, and aborting uploads.
- Updated serializers and viewsets to handle file uploads efficiently.
- Improved frontend components to display upload progress and handle
file uploads seamlessly.
- Adjusted backend services to process uploaded files asynchronously,
ensuring better performance and user experience.
- Updated documentation and tests to reflect the new functionality.
Rework totally the way to manage frontend translation keys. Currently we were
managing manually all those keys but now we want to use crowdin to manage our
languages so first we need to be able to extract frontend translation keys in
dedicated files.
Furthermore, we allow to setup frontend application languages and default
language through environment variables
(NEXT_PUBLIC_DEFAULT_LANGUAGE and NEXT_PUBLIC_LANGUAGES)
This PR adds a new build system for embeddable widgets and a first implementation of a "Feedback" popup widget.
It also refactors inbound message routes into channels, of which there are 2 for now: MTA (by default) and Widget. More to come!
This fixes delivery errors seen in the wild where:
* Some servers advertise STARTTLS but have broken certificates.
Because of this, we follow Postfix's recommentations to use
opportunistic TLS by default.
* As we are sending emails straight to other SMTPs in direct mode,
we need to have stricter RFC compliance for our raw mime DATA, namely
CRLF endings and maximum line length. This was transparently done
by Postfix in relay mode.
Some env vars have also been renamed for clarity.
This PR adds support for 2 MTA out modes : Direct-to-MX and SMTP-relay outbound delivery. Direct mode supports SOCKS5 proxies, and we bundle a new `src/socks-proxy` component to support it.
We also add an end-to-end self-check command plus scheduled health-check task with optional Prometheus metrics.
---------
Co-authored-by: Bastien Ogier <bastien.ogier@ext.anct.gouv.fr>
Co-authored-by: Stanislas Bruhiere <stanislas@bruhiere.fr>
When a message is received, if the auto labelling feature is enabled,
we try to add all relevant labels that are flagged for auto labelling to this
message.
Co-authored-by: jbpenrath <jb.penrath@gmail.com>
In the personal mailbox form, we now render extra fields according to the
`SCHEMA_CUSTOM_ATTRIBUTES_USER` got through config endpoint.
Those fields are validated through zod and the schema is generated through
`zod-from-json-schema` package. Then a custom component is in charge to
render the right component according to the schema property `type`.
We want to add ability to Messages to interop with our Drive application.
First, we want to be able to set the Drive base url through environment
variable then pass down Drive configuration to frontend through the config
api endpoint.
https://github.com/suitenumerique/drive
We are going to release a 0.1 version soon, along with our first production deployment. Starting from there, migrations and a consistent developer experience will be officially supported. To make that easier, this large patch cleans up several areas:
* Reset migrations one last time
* Update models for storage efficiency (move textchoices to integerchoices on high-volume tables)
* Use Blobs for mail mime data and draft bodies. Having them in a separate PG table is a first step, we will later start offloading them to object storage.
* Add default ZSTD compression to blobs
* Add per-domain DKIM Keys
* Add DNS check and provisioning, with a first Scaleway provider
* Fix Keycloak user provisioning
* Fix Attachment storage, they are now stored individually only at the drafting stage. Afterwards they are extracted from the main blob. This may be optimized later but at least we only store once. For JMAP compatibility, this requires using fake IDs in the blob API route.
* Add a management command and recurring task to retry unsent messages
* Improve the local developer experience with new ports and make commands
* Repackage MTA-in and MTA-out to be closer to Backend: Poetry, multi-step Dockerfile, move compose and makefile to the root
* Migrate to OpenSearch
* Improve overall documentation and add a self-hosting page
Contributes to #177 and #185
