mirror of
https://github.com/suitenumerique/messages.git
synced 2026-09-11 04:17:47 +02:00
Sylvain ZimmerandGitHub
f1a89a5bdb
We are going to release a 0.1 version soon, along with our first production deployment. Starting from there, migrations and a consistent developer experience will be officially supported. To make that easier, this large patch cleans up several areas: * Reset migrations one last time * Update models for storage efficiency (move textchoices to integerchoices on high-volume tables) * Use Blobs for mail mime data and draft bodies. Having them in a separate PG table is a first step, we will later start offloading them to object storage. * Add default ZSTD compression to blobs * Add per-domain DKIM Keys * Add DNS check and provisioning, with a first Scaleway provider * Fix Keycloak user provisioning * Fix Attachment storage, they are now stored individually only at the drafting stage. Afterwards they are extracted from the main blob. This may be optimized later but at least we only store once. For JMAP compatibility, this requires using fake IDs in the blob API route. * Add a management command and recurring task to retry unsent messages * Improve the local developer experience with new ports and make commands * Repackage MTA-in and MTA-out to be closer to Backend: Poetry, multi-step Dockerfile, move compose and makefile to the root * Migrate to OpenSearch * Improve overall documentation and add a self-hosting page Contributes to #177 and #185
4.0 KiB
4.0 KiB
Architecture
System Architecture Overview
Core Components
Frontend App
- Next.js Application: React-based SPA with TypeScript
- Auto-generated API Client: Generated from OpenAPI schema using Orval
- Multi-panel Interface: Mailbox panel, thread list, and message view
- Real-time Updates: Using TanStack Query for efficient state management
Backend Services
- Django REST Framework: Main API service handling business logic, including email processing
- Celery Workers: Asynchronous task processing for heavy operations
- Search Service: OpenSearch integration for full-text search
Mail Transfer Layer
- MTA-In (Inbound): Postfix server with Python-based recipient validation
- MTA-Out (Outbound): Postfix server for email delivery and relay
- Mail Processing Agent: rspamd for spam filtering and mail processing
Data Storage
- PostgreSQL: Primary relational database for all structured data
- Redis: Caching layer and Celery message broker
- OpenSearch: Full-text search index for messages and threads
- S3-Compatible Storage: File and attachment storage (In progress)
Authentication & Authorization
- Keycloak: OIDC provider for user authentication
- Role-based Access: Multi-tenant access control via mailbox and thread permissions
Data Flow
Inbound Email Processing
- External email arrives at MTA-In via SMTP
- MTA-In validates recipients against Django backend
- MDA parses and stores messages in PostgreSQL
- Celery tasks index content in OpenSearch
- Users see new messages in real-time via frontend
Outbound Email Processing
- User composes message in frontend
- Frontend sends draft via REST API
- Backend validates and queues message
- Celery processes sending via MTA-Out
- MTA-Out delivers email externally or to MailCatcher (dev)
Search Operations
- User submits search query via frontend
- Backend directly queries OpenSearch for real-time results
- Results are ranked and filtered by permissions
- Frontend displays paginated results
Search Indexing
- New messages/threads are saved to PostgreSQL
- Backend queues indexing tasks to Celery
- Celery workers asynchronously index content in OpenSearch
- Heavy operations (bulk imports, reindexing) are handled via Celery
Key Features
Multi-tenancy
- Domain-based: Mail domains with administrative roles
- Mailbox-based: Individual mailbox access permissions
- Thread-based: Granular access control for conversations
Scalability
- Microservices Architecture: Independent scaling of components
- Async Processing: Non-blocking operations via Celery
- Caching Strategy: Redis for session and query caching
- Search Optimization: OpenSearch for fast full-text search
Development Experience
- OpenAPI-First: Auto-generated client from backend schema
- Docker Compose: Complete development environment
- Hot Reloading: Frontend and backend development servers
- Testing Tools: Comprehensive test suites and monitoring
Security Considerations
Authentication
- OIDC integration with Keycloak
- JWT token validation
- Session management via Redis
Authorization
- Role-based access control (RBAC)
- Resource-level permissions
- Multi-tenant isolation
Email Security
- DKIM signing for outbound messages
- SPF and DMARC policy enforcement
- Anti-spam filtering via rspamd
Data Protection
- Encrypted storage for sensitive data
- Secure file upload handling
- CORS and CSRF protection
Deployment Architecture
The system is designed for containerized deployment with:
- Docker containers for all services
- Environment-specific configurations (dev, staging, production)
- Horizontal scaling capability for backend and Celery workers
- Load balancing support via nginx reverse proxy
- Health checks and monitoring integration (In Progress)