The pnpm-style layout bakes install-local .store/<hash> paths into the
native project files committed by cap sync, and the named node_modules
volume masked the host tree — breaking the IDE and the Gradle/Xcode
builds that resolve Capacitor plugins through relative paths into
src/frontend/node_modules. Back to the hoisted layout installed in the
bind mount, with the bootstrap check updated accordingly.
Use the new preview_text method of jmap_email to generate
clean snippet for thread (denormalized at update_stats)
and message (at serialization). Now when a mesage is folded, we
display this snippet. In the thread list we also display this snippet above
the subject.
The 0.3.0 parser refuses what 0.1.0 truncated and the composer raises
where it silently mangled, so the app has to take a position at each
seam: a ComposeError on send becomes a 400 (a property of the draft,
not a server fault), an unparseable inbound message is abandoned
outright instead of retried for 48h (deterministic failure — logged
at error level since abandoned rows are purged after 7 days), and a
stored message the stricter parser now refuses is flagged unreadable
to the UI rather than rendered blank. Attachment display names move
to a single service so serializer, blob download and draft builder
synthesize the same name for a nameless MIME part — the bug that
started this branch. The inbound retry sweep gains age-based backoff
so a dependency outage is not polled harder the longer it lasts. The
dev compose mounts the jmap-email working tree over the installed
wheel so local edits propagate without a rebuild.
Archive reconstruction (PST) composes with allow_smtputf8: an EAI
address is legal in an Exchange archive and the reconstructed .eml is
stored, never retransmitted, so refusing it would exclude the message
from the import. The unquote-message reply patterns bound every
whitespace quantifier that could cross newlines: under the m flag an
unbounded \s* backtracks once per line start, quadratic in the line
count of an attacker-supplied body.
Notably, we try to reduce disk usage by standardizind on common
base Docker images. We also improve node_modules by reducing
duplicate dependencies and install speed.
Ship the existing SPA as native iOS/Android apps without forking the
codebase: Capacitor wraps the web build, and every mobile-specific
behavior is gated behind isNativePlatform() so the web app is
untouched. The native shells route fetch/cookies through the native
HTTP layer (CapacitorHttp) — the WebView cookie jar is unreliable for
cross-origin sessions — which is why login runs in the system browser
(cross-app SSO via the shared IdP cookie) and finishes through the
backend session handoff, with the deep-link scheme pinned by
sso-invariants tests. Downloads/share go through the Filesystem/Share
plugins since WebView navigation would lose the session.
Postfix was already removed as a mta-out dependency, this is the second step so we have a pure python, more auditable path for incoming emails. We plan to keep postfix as a compatible option for a while but it won't be the default once this is battle tested.
Now that jmap-email 0.1.0 is available on pypi we install it from
this registry and remove all tweaks to install the deps from local
folder. We keep the volume override for backend services in order
to be able to work on jmap-email and test it with ease in local
development environment.
Opensearch index is updated each time a Thread, Message or MessageRecipient
is updated through signals. The current logic has performance issues has
n update of a resource will generate n celery task to update the same
resource... So this work aims to batch updates. Index is updated each
30s and resource ids is deduplicated to only update a resource once.
Furthermore, in an import context, the index will be updated only
when the import will be completed to prevent to spam the celery
worker with numerous indexation task.
This large PR modernizes the backend of the app:
- Python Dependency Management: Poetry → uv
- Object Storage for local dev: MinIO (now unmaintained) → RustFS
- Makefile Target Standardization to align with other LaSuite repos
- Internationalization Removal on backend: we only care about i18n on the frontend
- Backend dependencies upgrade
This avoids a case where a very large import would overflow the
"celery" queue with reindex tasks that would block the inbound
processing tasks. Now we have good priorization.
* (ops) implement production docker images
Github workflows:
- moved docker-publish jobs into a separate file on push main only
Backend:
- added healthcheck on backend Dockerfile + minor fixes
- bumped django to 5.1.15 to mitigate major CVE
- moved /healthz endpoint to /__heartbeat__ + added db connection
check
Frontend:
- Added runtime-prod Dockerfile target + minor fixes on the Dockerfile
- bump next to 15.5.9 to mitigate major CVE
- moved nginx config to standard nginx container configuration
template, fixed `scalingo_postfrontend` accordingly
Keycloak:
- Added production Dockerfile
- Removed scalingo_pgdump script & cron.json
socks-proxy:
- added package upgrades run into runtime Dockerfile stage
Misc:
- removed scalingo_pgdump.sh & cron.json
- fixed compose and e2e nginx configs with /healthz replacement
* (fix) coderabbit recommandations
This adds an intermediate Postgres-backed queue for inbound messages, that
allows us to run filters like spam processing before inserting messages in their
final storage (soon to be object storage). Also include misc. refactorings.
- Added multipart upload capabilities for EML and MBOX files to the
message imports bucket.
- Introduced new API endpoints for initiating uploads, completing
uploads, and aborting uploads.
- Updated serializers and viewsets to handle file uploads efficiently.
- Improved frontend components to display upload progress and handle
file uploads seamlessly.
- Adjusted backend services to process uploaded files asynchronously,
ensuring better performance and user experience.
- Updated documentation and tests to reflect the new functionality.
Enable commands to upload and download translation file to/from crowdin.
Setup ci workflow to automatically update translation files on each merge on
main branch. Also add a worklow that can be manually triggered which is in
charge to update application translation files then open a PR.
This PR adds a new build system for embeddable widgets and a first implementation of a "Feedback" popup widget.
It also refactors inbound message routes into channels, of which there are 2 for now: MTA (by default) and Widget. More to come!
This PR adds support for 2 MTA out modes : Direct-to-MX and SMTP-relay outbound delivery. Direct mode supports SOCKS5 proxies, and we bundle a new `src/socks-proxy` component to support it.
We also add an end-to-end self-check command plus scheduled health-check task with optional Prometheus metrics.
---------
Co-authored-by: Bastien Ogier <bastien.ogier@ext.anct.gouv.fr>
Co-authored-by: Stanislas Bruhiere <stanislas@bruhiere.fr>
* 🐛(docker) stop ALL services when running make stop
* ✨(docker) make sure backend is healthy before exiting docker compose up
* 🧑💻 (docker) use default user and not root for healthchecks
* ➖(backend) Remove dockerflow
* ✨(backend) add healthcheck route, and suppress healthcheck access logs
* 🚨(backend) ignore monkey patching
* 🚨(backend) coderabbitai + linter fixes
* 🚚(backend) remove version in healthz route
* 🔨(docker) use python http lib instead of curl and remove curl from dockerfile
We are going to release a 0.1 version soon, along with our first production deployment. Starting from there, migrations and a consistent developer experience will be officially supported. To make that easier, this large patch cleans up several areas:
* Reset migrations one last time
* Update models for storage efficiency (move textchoices to integerchoices on high-volume tables)
* Use Blobs for mail mime data and draft bodies. Having them in a separate PG table is a first step, we will later start offloading them to object storage.
* Add default ZSTD compression to blobs
* Add per-domain DKIM Keys
* Add DNS check and provisioning, with a first Scaleway provider
* Fix Keycloak user provisioning
* Fix Attachment storage, they are now stored individually only at the drafting stage. Afterwards they are extracted from the main blob. This may be optimized later but at least we only store once. For JMAP compatibility, this requires using fake IDs in the blob API route.
* Add a management command and recurring task to retry unsent messages
* Improve the local developer experience with new ports and make commands
* Repackage MTA-in and MTA-out to be closer to Backend: Poetry, multi-step Dockerfile, move compose and makefile to the root
* Migrate to OpenSearch
* Improve overall documentation and add a self-hosting page
Contributes to #177 and #185
This replaces the two plugins we had (check recipients + delivery)
to a single Milter plugin. The real change is that now the delivery
happens during the SMTP session, and can result in a temporary
failure to the client. This makes the MTA-in entirely stateless.
This allows user creation, passwords reset and listing users from
Keycloak, plus a new MailDomainAccess model for admins.
* ✨(admin) add maildomainacess model, api route and backend tests
* ♻️(drf) simplify API code
* ✨(keycloack) add deployable keycloak to PaaS
* ✨(keycloak) add integration via API, upgrade to 26.x
* 🗑️(settings) remove invitation setting
* 🐛(tests) fix failing tests after rebase
* 🚨(all) fix lint
Before, sending was done synchronously on the api/send POST call. Now
we just build the MIME message and mark the message for sending, and
queue the actual SMTP (or internal) sending for the worker. We use
this opportunity to add fine-grained delivery statuses for each
recipient, something that traditional email systems are unable to do!
Expanding on that, this (too large) PR adds an initial Elasticsearch
index of threads. It also overhauls the build and CI system, using
Poetry instead of Pip.
There was an extra \r\n added, that broke validation. We now test
the source of the received message on the mailcatcher, and validate
that DKIM header. Should be much more robust even if it adds a new
dependency for tests.
You can now send emails to the inbound MTA, which will forward them
to the Django MDA. You can then read them (served through REST) from
the frontend and compose a reply! The frontend will POST the message
to Django, which will build an RFC5322 message and send it to the
outbound MTA, which will relay it to the local mailcatcher where you
can view your reply. Phew!
Initial documentation in the README but this is the core loop that
will be expanded upon, with diagrams and videos!
Simplified Docker setup for frontend, use frontend-dev for running
the app itself, and frontend-tools for all npm usage. The local
node_modules directory should be used, without any copy in the container
jbpenrath