mirror of
https://github.com/suitenumerique/messages.git
synced 2026-08-21 07:02:22 +02:00
112 lines
3.4 KiB
Markdown
112 lines
3.4 KiB
Markdown
# Background Task Worker
|
|
|
|
The application uses a background task worker to process asynchronous jobs like email processing, file imports, and search indexing.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Start with all queues and scheduler (default)
|
|
python worker.py
|
|
|
|
# Start with specific queues only
|
|
python worker.py --queues=inbound,outbound
|
|
|
|
# Exclude low-priority queues
|
|
python worker.py --exclude=reindex,imports
|
|
|
|
# Disable the scheduler (for secondary workers)
|
|
python worker.py --disable-scheduler
|
|
```
|
|
|
|
## Queues
|
|
|
|
Tasks are routed to specific queues based on their type. Queues are listed in priority order.
|
|
|
|
| Priority | Queue | Description |
|
|
|----------|-------|-------------|
|
|
| 1 (highest) | `management` | Admin/management tasks (migrations, cleanup) |
|
|
| 2 | `inbound` | Inbound email processing (time-sensitive) |
|
|
| 3 | `outbound` | Outbound email sending and retries |
|
|
| 4 | `default` | General tasks (fallback for unrouted tasks) |
|
|
| 5 | `imports` | File import processing (MBOX, EML, PST, IMAP) |
|
|
| 6 (lowest) | `reindex` | Search indexing |
|
|
|
|
### Queue Routing
|
|
|
|
Tasks are automatically routed to queues based on their module:
|
|
|
|
| Task Module | Queue |
|
|
|-------------|-------|
|
|
| `core.mda.inbound_tasks.*` | `inbound` |
|
|
| `core.mda.outbound_tasks.*` | `outbound` |
|
|
| `core.services.importer.mbox_tasks.*` | `imports` |
|
|
| `core.services.importer.eml_tasks.*` | `imports` |
|
|
| `core.services.importer.imap_tasks.*` | `imports` |
|
|
| `core.services.importer.pst_tasks.*` | `imports` |
|
|
| `core.services.search.tasks.*` | `reindex` |
|
|
| Everything else | `default` |
|
|
|
|
## CLI Options
|
|
|
|
| Option | Description |
|
|
|--------|-------------|
|
|
| `--queues`, `-Q` | Comma-separated list of queues to process |
|
|
| `--exclude`, `-X` | Comma-separated list of queues to exclude |
|
|
| `--concurrency`, `-c` | Number of worker processes (default: CPU count) |
|
|
| `--disable-scheduler` | Disable the task scheduler (enabled by default) |
|
|
| `--loglevel`, `-l` | Logging level (default: INFO) |
|
|
|
|
## Environment Variables
|
|
|
|
| Variable | Description |
|
|
|----------|-------------|
|
|
| `WORKER_CONCURRENCY` | Default concurrency if `--concurrency` not specified |
|
|
| `CELERY_CONCURRENCY` | Fallback for `WORKER_CONCURRENCY` (legacy) |
|
|
|
|
## Deployment
|
|
|
|
### Scalingo (Procfile)
|
|
|
|
```text
|
|
worker: python worker.py
|
|
```
|
|
|
|
### Docker Compose
|
|
|
|
The `worker-dev` service in `compose.yaml` runs the worker for local development:
|
|
|
|
```yaml
|
|
worker-dev:
|
|
command: ["python", "worker.py", "--loglevel=DEBUG"]
|
|
```
|
|
|
|
### Running Multiple Workers
|
|
|
|
For high-throughput deployments or strict queue isolation, run specialized workers for different queue groups:
|
|
|
|
```bash
|
|
# Worker 1: High-priority email processing (with scheduler)
|
|
python worker.py --queues=management,inbound,outbound
|
|
|
|
# Worker 2: Background tasks only (no scheduler)
|
|
python worker.py --queues=default,imports,reindex --disable-scheduler
|
|
```
|
|
|
|
This ensures that low-priority tasks (imports, reindex) never compete with email processing.
|
|
|
|
## Monitoring
|
|
|
|
The `worker-ui` service provides a web UI for monitoring tasks (Flower). Access it at `http://localhost:8903` in development.
|
|
|
|
Task events are enabled by default for monitoring tools.
|
|
|
|
## Scheduled Tasks
|
|
|
|
The scheduler is enabled by default. The following tasks are scheduled:
|
|
|
|
| Task | Schedule | Queue |
|
|
|------|----------|-------|
|
|
| Retry pending messages | Every 5 minutes | `outbound` |
|
|
| System selfcheck | Configurable interval | `outbound` |
|
|
| Process inbound queue | Every 5 minutes | `inbound` |
|