This allows to use S3-compatible object storage to offload blobs, making Postgres much lighter. We design for storing ~1B emails on a single instance.
We also take this opportunity to do model changes on blobs & attachments.
Migration 0027 is one-way, no going back after this one so check your backups.
We recently remove backend translation but this script
still tries to create a empty django.pot in an folder that
does not existe anymore and compile .mo files
This large PR modernizes the backend of the app:
- Python Dependency Management: Poetry → uv
- Object Storage for local dev: MinIO (now unmaintained) → RustFS
- Makefile Target Standardization to align with other LaSuite repos
- Internationalization Removal on backend: we only care about i18n on the frontend
- Backend dependencies upgrade
Update all GitHub Actions to their latest major versions for improved
performance, security patches, and Node.js runtime compatibility.
Signed-off-by: Stephan Meijer <me@stephanmeijer.com>
* (ops) implement production docker images
Github workflows:
- moved docker-publish jobs into a separate file on push main only
Backend:
- added healthcheck on backend Dockerfile + minor fixes
- bumped django to 5.1.15 to mitigate major CVE
- moved /healthz endpoint to /__heartbeat__ + added db connection
check
Frontend:
- Added runtime-prod Dockerfile target + minor fixes on the Dockerfile
- bump next to 15.5.9 to mitigate major CVE
- moved nginx config to standard nginx container configuration
template, fixed `scalingo_postfrontend` accordingly
Keycloak:
- Added production Dockerfile
- Removed scalingo_pgdump script & cron.json
socks-proxy:
- added package upgrades run into runtime Dockerfile stage
Misc:
- removed scalingo_pgdump.sh & cron.json
- fixed compose and e2e nginx configs with /healthz replacement
* (fix) coderabbit recommandations
- Added multipart upload capabilities for EML and MBOX files to the
message imports bucket.
- Introduced new API endpoints for initiating uploads, completing
uploads, and aborting uploads.
- Updated serializers and viewsets to handle file uploads efficiently.
- Improved frontend components to display upload progress and handle
file uploads seamlessly.
- Adjusted backend services to process uploaded files asynchronously,
ensuring better performance and user experience.
- Updated documentation and tests to reflect the new functionality.
Enable commands to upload and download translation file to/from crowdin.
Setup ci workflow to automatically update translation files on each merge on
main branch. Also add a worklow that can be manually triggered which is in
charge to update application translation files then open a PR.
This PR adds support for 2 MTA out modes : Direct-to-MX and SMTP-relay outbound delivery. Direct mode supports SOCKS5 proxies, and we bundle a new `src/socks-proxy` component to support it.
We also add an end-to-end self-check command plus scheduled health-check task with optional Prometheus metrics.
---------
Co-authored-by: Bastien Ogier <bastien.ogier@ext.anct.gouv.fr>
Co-authored-by: Stanislas Bruhiere <stanislas@bruhiere.fr>
The OpenAPI Schema does not embedded abilities property.
We fix that and make two serializers for User model in order to get proper
openapi definition for User instance with abilities and without abilities.
* ✨(dns) add admin page for DNS records. Fixes#177
* 👷(dockerflow) add a flag to disable Dockerflow
* 💚(ghactions) fix check-api-update that was missing Orval
* 💚(ci) fix tests that depended on Keycloak
* ♻️(admin) refactor CopyableInput into a separate component
* 🎨(dns) improve code style
We are going to release a 0.1 version soon, along with our first production deployment. Starting from there, migrations and a consistent developer experience will be officially supported. To make that easier, this large patch cleans up several areas:
* Reset migrations one last time
* Update models for storage efficiency (move textchoices to integerchoices on high-volume tables)
* Use Blobs for mail mime data and draft bodies. Having them in a separate PG table is a first step, we will later start offloading them to object storage.
* Add default ZSTD compression to blobs
* Add per-domain DKIM Keys
* Add DNS check and provisioning, with a first Scaleway provider
* Fix Keycloak user provisioning
* Fix Attachment storage, they are now stored individually only at the drafting stage. Afterwards they are extracted from the main blob. This may be optimized later but at least we only store once. For JMAP compatibility, this requires using fake IDs in the blob API route.
* Add a management command and recurring task to retry unsent messages
* Improve the local developer experience with new ports and make commands
* Repackage MTA-in and MTA-out to be closer to Backend: Poetry, multi-step Dockerfile, move compose and makefile to the root
* Migrate to OpenSearch
* Improve overall documentation and add a self-hosting page
Contributes to #177 and #185
Before, sending was done synchronously on the api/send POST call. Now
we just build the MIME message and mark the message for sending, and
queue the actual SMTP (or internal) sending for the worker. We use
this opportunity to add fine-grained delivery statuses for each
recipient, something that traditional email systems are unable to do!
Expanding on that, this (too large) PR adds an initial Elasticsearch
index of threads. It also overhauls the build and CI system, using
Poetry instead of Pip.
Sylvain Zimmer