Next.js has been concerned by several CVEs last time and it's clearly
overkill for our needs (static export) so in order to enlight our stack
we migrate to vite & tanstack-router
Co-authored-by: jbpenrath <jb.penrath@gmail.com>
This allows to use S3-compatible object storage to offload blobs, making Postgres much lighter. We design for storing ~1B emails on a single instance.
We also take this opportunity to do model changes on blobs & attachments.
Migration 0027 is one-way, no going back after this one so check your backups.
Add `NEXT_PUBLIC_FEEDBACK_WIDGET_HOME_CHANNEL` env var to be able to
set a specific channel id to receive feedback from unauthenticated
users.
For configuration ease and backward compatibility, if
`NEXT_PUBLIC_FEEDBACK_WIDGET_HOME_CHANNEL` is not set, the home feedback
widget fallback to `NEXT_PUBLIC_FEEDBACK_WIDGET_CHANNEL`.
Allow to configure a help center URL via the
NEXT_PUBLIC_HELP_CENTER_URL environment variable. The button
adapts based on what's configured:
- Help center only: opens external link directly
- Feedback widget only: opens widget directly
- Both: shows dropdown with both options
The button is now displayed in the header to save space in sidebar,
rather than in the left panel.
This large PR modernizes the backend of the app:
- Python Dependency Management: Poetry → uv
- Object Storage for local dev: MinIO (now unmaintained) → RustFS
- Makefile Target Standardization to align with other LaSuite repos
- Internationalization Removal on backend: we only care about i18n on the frontend
- Backend dependencies upgrade
This adds an intermediate Postgres-backed queue for inbound messages, that
allows us to run filters like spam processing before inserting messages in their
final storage (soon to be object storage). Also include misc. refactorings.
- Added multipart upload capabilities for EML and MBOX files to the
message imports bucket.
- Introduced new API endpoints for initiating uploads, completing
uploads, and aborting uploads.
- Updated serializers and viewsets to handle file uploads efficiently.
- Improved frontend components to display upload progress and handle
file uploads seamlessly.
- Adjusted backend services to process uploaded files asynchronously,
ensuring better performance and user experience.
- Updated documentation and tests to reflect the new functionality.
Rework totally the way to manage frontend translation keys. Currently we were
managing manually all those keys but now we want to use crowdin to manage our
languages so first we need to be able to extract frontend translation keys in
dedicated files.
Furthermore, we allow to setup frontend application languages and default
language through environment variables
(NEXT_PUBLIC_DEFAULT_LANGUAGE and NEXT_PUBLIC_LANGUAGES)
This PR adds a new build system for embeddable widgets and a first implementation of a "Feedback" popup widget.
It also refactors inbound message routes into channels, of which there are 2 for now: MTA (by default) and Widget. More to come!
This fixes delivery errors seen in the wild where:
* Some servers advertise STARTTLS but have broken certificates.
Because of this, we follow Postfix's recommentations to use
opportunistic TLS by default.
* As we are sending emails straight to other SMTPs in direct mode,
we need to have stricter RFC compliance for our raw mime DATA, namely
CRLF endings and maximum line length. This was transparently done
by Postfix in relay mode.
Some env vars have also been renamed for clarity.
This PR adds support for 2 MTA out modes : Direct-to-MX and SMTP-relay outbound delivery. Direct mode supports SOCKS5 proxies, and we bundle a new `src/socks-proxy` component to support it.
We also add an end-to-end self-check command plus scheduled health-check task with optional Prometheus metrics.
---------
Co-authored-by: Bastien Ogier <bastien.ogier@ext.anct.gouv.fr>
Co-authored-by: Stanislas Bruhiere <stanislas@bruhiere.fr>
When a message is received, if the auto labelling feature is enabled,
we try to add all relevant labels that are flagged for auto labelling to this
message.
Co-authored-by: jbpenrath <jb.penrath@gmail.com>
We want to add ability to Messages to interop with our Drive application.
First, we want to be able to set the Drive base url through environment
variable then pass down Drive configuration to frontend through the config
api endpoint.
https://github.com/suitenumerique/drive
We are going to release a 0.1 version soon, along with our first production deployment. Starting from there, migrations and a consistent developer experience will be officially supported. To make that easier, this large patch cleans up several areas:
* Reset migrations one last time
* Update models for storage efficiency (move textchoices to integerchoices on high-volume tables)
* Use Blobs for mail mime data and draft bodies. Having them in a separate PG table is a first step, we will later start offloading them to object storage.
* Add default ZSTD compression to blobs
* Add per-domain DKIM Keys
* Add DNS check and provisioning, with a first Scaleway provider
* Fix Keycloak user provisioning
* Fix Attachment storage, they are now stored individually only at the drafting stage. Afterwards they are extracted from the main blob. This may be optimized later but at least we only store once. For JMAP compatibility, this requires using fake IDs in the blob API route.
* Add a management command and recurring task to retry unsent messages
* Improve the local developer experience with new ports and make commands
* Repackage MTA-in and MTA-out to be closer to Backend: Poetry, multi-step Dockerfile, move compose and makefile to the root
* Migrate to OpenSearch
* Improve overall documentation and add a self-hosting page
Contributes to #177 and #185
This allows user creation, passwords reset and listing users from
Keycloak, plus a new MailDomainAccess model for admins.
* ✨(admin) add maildomainacess model, api route and backend tests
* ♻️(drf) simplify API code
* ✨(keycloack) add deployable keycloak to PaaS
* ✨(keycloak) add integration via API, upgrade to 26.x
* 🗑️(settings) remove invitation setting
* 🐛(tests) fix failing tests after rebase
* 🚨(all) fix lint
Before, sending was done synchronously on the api/send POST call. Now
we just build the MIME message and mark the message for sending, and
queue the actual SMTP (or internal) sending for the worker. We use
this opportunity to add fine-grained delivery statuses for each
recipient, something that traditional email systems are unable to do!
Expanding on that, this (too large) PR adds an initial Elasticsearch
index of threads. It also overhauls the build and CI system, using
Poetry instead of Pip.
You can now send emails to the inbound MTA, which will forward them
to the Django MDA. You can then read them (served through REST) from
the frontend and compose a reply! The frontend will POST the message
to Django, which will build an RFC5322 message and send it to the
outbound MTA, which will relay it to the local mailcatcher where you
can view your reply. Phew!
Initial documentation in the README but this is the core loop that
will be expanded upon, with diagrams and videos!
