📝(docs) add architecture and environment docs (#145)

* 🧹() update gitignore

* 📝(docs) add architecture overview

* 📝(docs) add env docs
This commit is contained in:
Niels Kersic
2025-06-13 23:19:24 +02:00
committed by GitHub
parent 33a3195b17
commit 147e6a165a
3 changed files with 498 additions and 0 deletions
+4
View File
@@ -79,5 +79,9 @@ db.sqlite3
*.iml
.devcontainer
# AI tooling
.claude
CLAUDE.md
# Various
.turbo
+220
View File
@@ -0,0 +1,220 @@
# Architecture
## System Architecture Overview
```mermaid
graph TB
%% External actors
ExternalSender[External Email Sender]
User[User/Browser]
%% Frontend
Frontend[Next.js Frontend<br/>React + TypeScript]
%% Backend services
subgraph "Core Services"
Backend[Django REST API<br/>Backend Service]
Celery[Celery Worker<br/>Async Task Processing]
%% Backend components
subgraph "Backend Components"
API[REST API Endpoints]
MDA[Mail Delivery Agent]
Search[Search Service]
Tasks[Async Tasks]
end
end
%% Mail Transfer Agents
subgraph "Mail Processing Layer"
MTAIn[MTA-In<br/>Postfix + Python]
MTAOut[MTA-Out<br/>Postfix]
MPA[Mail Processing Agent<br/>rspamd]
end
%% Data layer
subgraph "Data Layer"
PostgreSQL[(PostgreSQL<br/>Primary Database)]
Redis[(Redis<br/>Cache & Message Broker)]
Elasticsearch[(Elasticsearch<br/>Search Index)]
S3[(S3 Storage<br/>File Attachments)]
end
%% Authentication
subgraph "Authentication"
Keycloak[Keycloak<br/>OIDC Provider]
end
%% Development tools
subgraph "Development Support"
MailCatcher[MailCatcher<br/>Email Testing]
Flower[Flower<br/>Celery Monitoring]
ElasticUI[ElasticVue<br/>Search Monitoring]
end
%% External flows
ExternalSender -->|SMTP| MTAIn
User -->|HTTPS| Frontend
%% Frontend to backend
Frontend -->|REST API| Backend
Frontend -->|Authentication| Keycloak
%% Backend internal flows
Backend --> API
Backend --> MDA
Backend --> Search
Backend --> Tasks
%% Mail processing flows
MTAIn -->|Recipient Validation| Backend
MTAIn -->|Message Delivery| MDA
Backend -->|Send Email| MTAOut
MTAOut -->|External Relay| MailCatcher
MPA -->|Filter/Process| MTAIn
%% Data access
Backend --> PostgreSQL
Backend --> Redis
Backend -->|Direct Queries| Elasticsearch
Backend --> S3
Celery --> Redis
Celery --> PostgreSQL
%% Async processing
Backend -->|Queue Tasks| Celery
Celery -->|Async Indexing| Elasticsearch
%% Development monitoring
Celery -.-> Flower
Elasticsearch -.-> ElasticUI
MTAOut -.-> MailCatcher
%% Authentication flow
Backend -->|Verify Tokens| Keycloak
```
## Core Components
### Frontend Layer
- **Next.js Application**: React-based SPA with TypeScript
- **Auto-generated API Client**: Generated from OpenAPI schema using Orval
- **Multi-panel Interface**: Mailbox panel, thread list, and message view
- **Real-time Updates**: Using TanStack Query for efficient state management
### Backend Services
- **Django REST Framework**: Main API service handling business logic
- **Celery Workers**: Asynchronous task processing for heavy operations
- **Mail Delivery Agent (MDA)**: Email processing and parsing
- **Search Service**: Elasticsearch integration for full-text search
### Mail Transfer Layer
- **MTA-In (Inbound)**: Postfix server with Python-based recipient validation
- **MTA-Out (Outbound)**: Postfix server for email delivery and relay
- **Mail Processing Agent**: rspamd for spam filtering and mail processing
### Data Storage
- **PostgreSQL**: Primary relational database for all structured data
- **Redis**: Caching layer and Celery message broker
- **Elasticsearch**: Full-text search index for messages and threads
- **S3-Compatible Storage**: File and attachment storage
### Authentication & Authorization
- **Keycloak**: OIDC provider for user authentication
- **Role-based Access**: Multi-tenant access control via mailbox and thread permissions
## Data Flow
### Inbound Email Processing
1. External email arrives at **MTA-In** via SMTP
2. **MTA-In** validates recipients against Django backend
3. Valid emails are processed by **rspamd** for filtering
4. **MDA** parses and stores messages in PostgreSQL
5. **Celery** tasks index content in Elasticsearch
6. Users see new messages in real-time via frontend
### Outbound Email Processing
1. User composes message in frontend
2. Frontend sends draft via REST API
3. Backend validates and queues message
4. **Celery** processes sending via **MTA-Out**
5. **MTA-Out** delivers email externally or to MailCatcher (dev)
### Search Operations
1. User submits search query via frontend
2. Backend directly queries Elasticsearch for real-time results
3. Results are ranked and filtered by permissions
4. Frontend displays paginated results
### Search Indexing
1. New messages/threads are saved to PostgreSQL
2. Backend queues indexing tasks to Celery
3. Celery workers asynchronously index content in Elasticsearch
4. Heavy operations (bulk imports, reindexing) are handled via Celery
## Key Features
### Multi-tenancy
- **Domain-based**: Mail domains with administrative roles
- **Mailbox-based**: Individual mailbox access permissions
- **Thread-based**: Granular access control for conversations
### Scalability
- **Microservices Architecture**: Independent scaling of components
- **Async Processing**: Non-blocking operations via Celery
- **Caching Strategy**: Redis for session and query caching
- **Search Optimization**: Elasticsearch for fast full-text search
### Development Experience
- **OpenAPI-First**: Auto-generated client from backend schema
- **Docker Compose**: Complete development environment
- **Hot Reloading**: Frontend and backend development servers
- **Testing Tools**: Comprehensive test suites and monitoring
## Security Considerations
### Authentication
- OIDC integration with Keycloak
- JWT token validation
- Session management via Redis
### Authorization
- Role-based access control (RBAC)
- Resource-level permissions
- Multi-tenant isolation
### Email Security
- DKIM signing for outbound messages
- SPF and DMARC policy enforcement
- Anti-spam filtering via rspamd
### Data Protection
- Encrypted storage for sensitive data
- Secure file upload handling
- CORS and CSRF protection
## Deployment Architecture
The system is designed for containerized deployment with:
- **Docker containers** for all services
- **Environment-specific configurations** (dev, staging, production)
- **Horizontal scaling** capability for backend and Celery workers
- **Load balancing** support via nginx reverse proxy
- **Health checks** and monitoring integration
+274
View File
@@ -0,0 +1,274 @@
# Environment Variables
This document provides a comprehensive overview of all environment variables used in the Messages application. These variables are organized by service and functionality.
## Core Application Configuration
### Django Settings
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `DJANGO_CONFIGURATION` | `Development` | Django configuration class to use (Development, Production, Test, etc.) | Required |
| `DJANGO_SECRET_KEY` | None | Secret key for cryptographic signing | Required |
| `DJANGO_ALLOWED_HOSTS` | `[]` | List of allowed hostnames | Required |
| `DJANGO_SETTINGS_MODULE` | `messages.settings` | Django settings module | Required |
| `DJANGO_SUPERUSER_PASSWORD` | `admin` | Default superuser password for development | Dev |
| `DJANGO_DATA_DIR` | `/data` | Base directory for data storage | Optional |
### Database Configuration
#### PostgreSQL (Main Database)
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `DATABASE_URL` | None | Complete database URL (overrides individual DB_* vars) | Optional |
| `DB_ENGINE` | `django.db.backends.postgresql_psycopg2` | Database engine | Optional |
| `DB_HOST` | `localhost` | Database hostname | Optional |
| `DB_NAME` | `messages` | Database name | Optional |
| `DB_USER` | `dbuser` | Database username | Optional |
| `DB_PASSWORD` | `dbpass` | Database password | Optional |
| `DB_PORT` | `5432` | Database port | Optional |
#### PostgreSQL (Keycloak)
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `POSTGRES_DB` | `keycloak` | Keycloak database name | Dev |
| `POSTGRES_USER` | `user` | Keycloak database user | Dev |
| `POSTGRES_PASSWORD` | `pass` | Keycloak database password | Dev |
### Redis Configuration
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `REDIS_URL` | `redis://redis:6379` | Redis connection URL | Optional |
| `CELERY_BROKER_URL` | `redis://redis:6379` | Celery message broker URL | Optional |
| `CACHES_DEFAULT_TIMEOUT` | `30` | Default cache timeout in seconds | Optional |
### Elasticsearch Configuration
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `ELASTICSEARCH_URL` | `["http://elasticsearch:9200"]` | Elasticsearch hosts list | Optional |
| `ELASTICSEARCH_TIMEOUT` | `20` | Elasticsearch query timeout | Optional |
| `ELASTICSEARCH_INDEX_THREADS` | `True` | Enable thread indexing | Optional |
## Mail Processing Configuration
### MTA Settings
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `MTA_OUT_HOST` | None | Outbound SMTP server host | Required |
| `MTA_OUT_SMTP_USERNAME` | None | Outbound SMTP username | Optional |
| `MTA_OUT_SMTP_PASSWORD` | None | Outbound SMTP password | Optional |
| `MTA_OUT_SMTP_USE_TLS` | `True` | Use TLS for outbound SMTP | Optional |
| `MDA_API_SECRET` | `default-mda-api-secret` | Shared secret for MDA API | Required |
| `MDA_API_BASE_URL` | None | Base URL for MDA API | Dev |
### MTA-Out Specific
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `SMTP_RELAY_HOST` | None | SMTP relay server | Dev |
| `SMTP_USERNAME` | None | SMTP authentication username | Dev |
| `SMTP_PASSWORD` | None | SMTP authentication password | Dev |
### Email Domain Configuration
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `MESSAGES_TESTDOMAIN` | `localhost` | Test domain for development | Dev |
| `MESSAGES_TESTDOMAIN_MAPPING_BASEDOMAIN` | `gouv.fr` | Base domain mapping | Dev |
| `MESSAGES_ACCEPT_ALL_EMAILS` | `False` | Accept emails to any domain | Optional |
### DKIM Configuration
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `MESSAGES_DKIM_SELECTOR` | `default` | DKIM selector | Optional |
| `MESSAGES_DKIM_DOMAINS` | `[]` | List of domains for DKIM signing | Optional |
| `MESSAGES_DKIM_PRIVATE_KEY_B64` | None | Base64 encoded DKIM private key | Optional |
| `MESSAGES_DKIM_PRIVATE_KEY_FILE` | None | Path to DKIM private key file | Optional |
## Storage Configuration
### S3-Compatible Storage
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `AWS_S3_ENDPOINT_URL` | None | S3 endpoint URL | Optional |
| `AWS_S3_ACCESS_KEY_ID` | None | S3 access key | Optional |
| `AWS_S3_SECRET_ACCESS_KEY` | None | S3 secret key | Optional |
| `AWS_S3_REGION_NAME` | None | S3 region | Optional |
| `AWS_STORAGE_BUCKET_NAME` | `st-messages-media-storage` | S3 bucket name | Optional |
| `AWS_S3_UPLOAD_POLICY_EXPIRATION` | `86400` | Upload policy expiration (24h) | Optional |
| `MEDIA_BASE_URL` | None | Base URL for media files | Optional |
| `ITEM_FILE_MAX_SIZE` | `5368709120` | Max file size (5GB) | Optional |
### Static Files
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `STORAGES_STATICFILES_BACKEND` | `whitenoise.storage.CompressedManifestStaticFilesStorage` | Static files storage backend | Optional |
## Authentication & Authorization
### OIDC Configuration
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `OIDC_CREATE_USER` | `False` | Automatically create users from OIDC | Optional |
| `OIDC_RP_CLIENT_ID` | `st_messages` | OIDC client ID | Required |
| `OIDC_RP_CLIENT_SECRET` | None | OIDC client secret | Required |
| `OIDC_RP_SIGN_ALGO` | `RS256` | OIDC signing algorithm | Optional |
| `OIDC_RP_SCOPES` | `openid email` | OIDC scopes | Optional |
| `OIDC_OP_JWKS_ENDPOINT` | None | OIDC JWKS endpoint | Required |
| `OIDC_OP_AUTHORIZATION_ENDPOINT` | None | OIDC authorization endpoint | Required |
| `OIDC_OP_TOKEN_ENDPOINT` | None | OIDC token endpoint | Required |
| `OIDC_OP_USER_ENDPOINT` | None | OIDC user info endpoint | Required |
| `OIDC_OP_LOGOUT_ENDPOINT` | None | OIDC logout endpoint | Optional |
### OIDC Advanced Settings
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `OIDC_USE_NONCE` | `True` | Use nonce in OIDC flow | Optional |
| `OIDC_REDIRECT_REQUIRE_HTTPS` | `False` | Require HTTPS for redirects | Optional |
| `OIDC_REDIRECT_ALLOWED_HOSTS` | `[]` | Allowed redirect hosts | Optional |
| `OIDC_STORE_ID_TOKEN` | `True` | Store ID token | Optional |
| `OIDC_FALLBACK_TO_EMAIL_FOR_IDENTIFICATION` | `True` | Use email as fallback identifier | Optional |
| `OIDC_ALLOW_DUPLICATE_EMAILS` | `False` | Allow duplicate emails (⚠️ Security risk) | Optional |
| `OIDC_AUTH_REQUEST_EXTRA_PARAMS` | `{}` | Extra parameters for auth requests | Optional |
### Authentication URLs
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `LOGIN_REDIRECT_URL` | None | Post-login redirect URL | Optional |
| `LOGIN_REDIRECT_URL_FAILURE` | None | Login failure redirect URL | Optional |
| `LOGOUT_REDIRECT_URL` | None | Post-logout redirect URL | Optional |
| `ALLOW_LOGOUT_GET_METHOD` | `True` | Allow GET method for logout | Optional |
### User Mapping
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `USER_OIDC_ESSENTIAL_CLAIMS` | `[]` | Essential OIDC claims | Optional |
| `USER_OIDC_FIELDS_TO_FULLNAME` | `["first_name", "last_name"]` | Fields for full name | Optional |
| `USER_OIDC_FIELD_TO_SHORTNAME` | `first_name` | Field for short name | Optional |
## Security & CORS
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `CORS_ALLOW_ALL_ORIGINS` | `False` | Allow all CORS origins | Optional |
| `CORS_ALLOWED_ORIGINS` | `[]` | Specific allowed CORS origins | Optional |
| `CORS_ALLOWED_ORIGIN_REGEXES` | `[]` | Regex patterns for allowed origins | Optional |
| `CSRF_TRUSTED_ORIGINS` | `[]` | Trusted origins for CSRF | Optional |
| `SERVER_TO_SERVER_API_TOKENS` | `[]` | API tokens for server-to-server auth | Optional |
## Monitoring & Observability
### Sentry
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `SENTRY_DSN` | None | Sentry DSN for error tracking | Optional |
### PostHog
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `POSTHOG_KEY` | None | PostHog analytics key | Optional |
### Logging
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `LOGGING_LEVEL_LOGGERS_ROOT` | `INFO` | Root logger level | Optional |
| `LOGGING_LEVEL_LOGGERS_APP` | `INFO` | Application logger level | Optional |
| `LOGGING_LEVEL_HANDLERS_CONSOLE` | `INFO` | Console handler level | Optional |
## API Configuration
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `API_USERS_LIST_LIMIT` | `5` | Default limit for user list API | Optional |
| `API_USERS_LIST_THROTTLE_RATE_SUSTAINED` | `180/hour` | Sustained throttle rate | Optional |
| `API_USERS_LIST_THROTTLE_RATE_BURST` | `30/minute` | Burst throttle rate | Optional |
### OpenAPI Schema
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `SPECTACULAR_SETTINGS_ENABLE_DJANGO_DEPLOY_CHECK` | `False` | Enable deploy check in OpenAPI | Optional |
## Frontend Configuration
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `FRONTEND_THEME` | None | Frontend theme identifier | Optional |
| `NEXT_PUBLIC_API_ORIGIN` | None | Frontend API origin | Dev |
| `NEXT_PUBLIC_S3_DOMAIN_REPLACE` | None | S3 domain replacement for frontend | Dev |
## Development Tools
### Crowdin (Translations)
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `CROWDIN_PERSONAL_TOKEN` | None | Crowdin API token | Dev |
| `CROWDIN_PROJECT_ID` | None | Crowdin project ID | Dev |
| `CROWDIN_BASE_PATH` | `/app/src` | Base path for translations | Dev |
## Application Settings
### Business Logic
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `TRASHBIN_CUTOFF_DAYS` | `30` | Days before permanent deletion | Optional |
| `INVITATION_VALIDITY_DURATION` | `604800` | Invitation validity (7 days) | Optional |
### Internationalization
| Variable | Default | Description | Required |
|----------|---------|-------------|----------|
| `LANGUAGE_CODE` | `en-us` | Default language code | Optional |
## Legend
- **Required**: Must be set for the application to function
- **Dev**: Required for development/testing environments
- **Optional**: Has sensible defaults, can be customized
## Environment Files
The application uses environment files located in `env.d/development/` for different services:
- `backend.dist` - Main Django application settings
- `common.dist` - Shared settings across services
- `postgresql.dist` - PostgreSQL database configuration
- `kc_postgresql.dist` - Keycloak database configuration
- `mta-in.dist` - Inbound mail server settings
- `mta-out.dist` - Outbound mail server settings
- `crowdin.dist` - Translation service configuration
## Security Notes
⚠️ **Important Security Considerations:**
1. **Never commit actual secrets** - Use `.dist` files as templates
2. **OIDC_ALLOW_DUPLICATE_EMAILS** - Should remain `False` in production
3. **CORS_ALLOW_ALL_ORIGINS** - Should be `False` in production
4. **DJANGO_SECRET_KEY** - Must be unique and secret in production
5. **Database passwords** - Use strong, unique passwords
6. **API tokens** - Rotate regularly and keep secure
## Production Deployment
For production deployments, ensure:
1. All **Required** variables are properly configured
2. Secrets are managed through secure secret management systems
3. HTTPS is enforced for all external communications
4. Database connections use SSL/TLS
5. File storage uses appropriate access controls