WEBDIENSTE/huly-selfhost
1
0
Fork 0
mirror of https://github.com/hcengineering/huly-selfhost.git synced 2026-08-17 21:25:55 +02:00
Code Issues Packages Projects Releases Wiki Activity

Add architecture diagrams (#233)

Browse Source 
* Add architecture diagrams

Signed-off-by: Artem Savchenko <armisav@gmail.com>

* Update diagram

Signed-off-by: Artem Savchenko <armisav@gmail.com>

* Rename

Signed-off-by: Artem Savchenko <armisav@gmail.com>

* Clean up

Signed-off-by: Artem Savchenko <armisav@gmail.com>

* Remove docker compose diagram

Signed-off-by: Artem Savchenko <armisav@gmail.com>

* Fix rekoni description

Signed-off-by: Artem Savchenko <armisav@gmail.com>

---------

Signed-off-by: Artem Savchenko <armisav@gmail.com>
This commit is contained in:
Artyom Savchenko
2025-12-03 13:13:10 +07:00
committed by GitHub
parent 5b1b98f437
commit 524830ca11
1 changed files with 346 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ARCHITECTURE_OVERVIEW.md
+346
View File
@@ -0,0 +1,346 @@
# Huly Self-Hosted Architecture Overview
## Service Overview
The Huly self-hosted deployment consists of 14 services working together. This section describes each service and its role in the platform.
### Application Services
| Service | Description |
|---------|-------------|
| **front** | Web application server that serves the Huly UI. Handles static assets, client-side routing, and coordinates with backend services for data and authentication. |
| **account** | Authentication and user management service. Handles user registration, login, JWT token generation/validation, and workspace membership. |
| **transactor** | Core transaction processing engine. Maintains WebSocket connections with clients for real-time updates, processes all data mutations, enforces business logic, and publishes events to the message queue. |
| **workspace** | Workspace lifecycle management. Handles workspace creation, initialization, upgrades, and configuration. Runs background jobs for workspace maintenance. |
| **collaborator** | Real-time document collaboration service using Y.js CRDT. Enables multiple users to edit documents simultaneously with automatic conflict resolution and presence awareness. |
| **fulltext** | Search indexing service. Consumes events from the message queue, extracts content from documents, and maintains the Elasticsearch search index for fast full-text search. |
| **rekoni** | Content intelligence service. Extracts text and structured data from binary documents (PDF, DOC, DOCX, RTF) to make them searchable. Used by fulltext service for indexing document attachments. |
| **stats** | Metrics collection service. Aggregates usage statistics and health metrics from all services for monitoring and debugging. |
### Infrastructure Services
| Service | Description |
|---------|-------------|
| **nginx** | Reverse proxy and SSL termination. Routes external requests to internal services, handles HTTPS certificates, and provides a single entry point for all client connections. |
| **cockroach** | CockroachDB - the primary database. Stores all application data including users, workspaces, documents, and metadata. Provides ACID transactions and horizontal scalability. |
| **elastic** | Elasticsearch search engine. Stores and indexes document content for fast full-text search queries. Managed by the fulltext service. |
| **minio** | S3-compatible object storage. Stores all binary files including attachments, images, and document blobs. Accessed directly by nginx for file downloads. |
| **redpanda** | Kafka-compatible event streaming platform. Provides reliable message delivery between services for asynchronous processing (e.g., search indexing after document changes). |
| **kvs (HulyKVS)** | Key-value store service. Provides fast key-value storage for application configuration, user preferences, and cached data. |
### Service Communication Patterns
- **Synchronous (HTTP/WebSocket)**: Client ↔ Nginx ↔ Services
- **Asynchronous (Events)**: Transactor → Redpanda → Fulltext
- **Direct Database**: Services → CockroachDB
- **File Storage**: Services → MinIO (via S3 API)
---
## 1. High-Level System Architecture
```mermaid
graph TB
subgraph "Client Layer"
Browser[Web Browser]
Desktop[Desktop App]
end
subgraph "Reverse Proxy"
Nginx[Nginx<br/>:80/:443]
end
subgraph "Frontend Layer"
Front[Frontend Server<br/>:8080]
end
subgraph "Core Backend Services"
Account[Account Service<br/>:3000<br/>Authentication & Users]
Workspace[Workspace Service<br/>Workspace Management]
Transactor[Transactor Service<br/>:3333<br/>Transaction Processing]
Collaborator[Collaborator Service<br/>:3078<br/>Real-time Collaboration]
end
subgraph "Data Services"
HulyKVS[HulyKVS<br/>:8094<br/>Key-Value Store]
Fulltext[Fulltext Service<br/>:4700<br/>Search Indexing]
Rekoni[Rekoni Service<br/>:4004<br/>AI/ML Recognition]
end
subgraph "Monitoring"
Stats[Stats Service<br/>:4900<br/>Metrics Collection]
end
subgraph "Primary Database"
CockroachDB[(CockroachDB<br/>:26257<br/>Distributed SQL)]
end
subgraph "Supporting Infrastructure"
Elasticsearch[(Elasticsearch<br/>:9200<br/>Search Engine)]
Minio[(MinIO<br/>:9000<br/>Object Storage)]
Redpanda[Redpanda<br/>:9092<br/>Event Streaming]
end
Browser --> Nginx
Desktop --> Nginx
Nginx --> Front
Nginx --> Account
Nginx --> Transactor
Nginx --> Collaborator
Account --> CockroachDB
Workspace --> CockroachDB
Transactor --> CockroachDB
Transactor --> Fulltext
Transactor --> Redpanda
HulyKVS --> CockroachDB
Fulltext --> Elasticsearch
Fulltext --> CockroachDB
Fulltext --> Rekoni
Fulltext --> Redpanda
Workspace --> Redpanda
Account --> Redpanda
Collaborator --> Minio
Front --> Minio
style Front fill:#4A90E2
style Account fill:#E24A4A
style Transactor fill:#E24A4A
style CockroachDB fill:#7ED321
style Redpanda fill:#F5A623
style Nginx fill:#009639
```
---
## 2. Network Topology & Nginx Routing
```mermaid
graph TB
subgraph "External Access"
Client[Client Browser/Desktop]
end
subgraph "Reverse Proxy - Nginx :80/:443"
Nginx[Nginx]
subgraph "Route Mappings"
R1["/ → front:8080"]
R2["/_accounts → account:3000"]
R3["/_transactor → transactor:3333"]
R4["/_collaborator → collaborator:3078"]
R5["/_rekoni → rekoni:4004"]
R6["/_stats → stats:4900"]
R7["/files → minio:9000"]
end
end
subgraph "Application Services"
Front[Front :8080]
Account[Account :3000]
Transactor[Transactor :3333]
Collaborator[Collaborator :3078]
Rekoni[Rekoni :4004]
Stats[Stats :4900]
Workspace[Workspace]
Fulltext[Fulltext :4700]
HulyKVS[HulyKVS :8094]
end
subgraph "Primary Database"
CockroachDB[(CockroachDB :26257)]
end
subgraph "Supporting Infrastructure"
Elasticsearch[(Elasticsearch :9200)]
Minio[(MinIO :9000/:9001)]
Redpanda[Redpanda :9092/:19092]
end
Client --> Nginx
Nginx --> R1 --> Front
Nginx --> R2 --> Account
Nginx --> R3 --> Transactor
Nginx --> R4 --> Collaborator
Nginx --> R5 --> Rekoni
Nginx --> R6 --> Stats
Nginx --> R7 --> Minio
style Nginx fill:#009639
style Front fill:#4A90E2
style Transactor fill:#E24A4A
style CockroachDB fill:#7ED321
style Redpanda fill:#F5A623
```
---
## 3. Event-Driven Architecture (Redpanda/Kafka)
```mermaid
graph LR
subgraph "Event Producers"
Transactor[Transactor<br/>Transaction Events]
Workspace[Workspace<br/>Workspace Events]
Account[Account<br/>Account Events]
end
subgraph "Event Bus"
Redpanda[Redpanda<br/>Kafka Topics<br/>:9092]
end
subgraph "Event Consumers"
Fulltext[Fulltext<br/>Indexing]
end
subgraph "Queue Configuration"
QC[QUEUE_CONFIG<br/>redpanda:9092]
end
Transactor -->|Document Events| Redpanda
Transactor -->|User Actions| Redpanda
Workspace -->|Workspace Events| Redpanda
Account -->|Account Events| Redpanda
Redpanda -->|Index Events| Fulltext
QC -.Config.-> Transactor
QC -.Config.-> Workspace
QC -.Config.-> Fulltext
QC -.Config.-> Account
style Redpanda fill:#F5A623
style Transactor fill:#E24A4A
```
---
## 4. Authentication & Authorization Flow
```mermaid
sequenceDiagram
participant Client
participant Nginx
participant Front
participant Account
participant Transactor
participant Workspace
participant CockroachDB
Client->>Nginx: Login Request
Nginx->>Front: Proxy
Front->>Account: Authenticate
Account->>CockroachDB: Verify Credentials
CockroachDB-->>Account: User Record
Account->>Account: Generate Token<br/>(SERVER_SECRET)
Account-->>Front: JWT Token
Front-->>Client: Token + Workspace List
Client->>Nginx: Connect to Workspace
Nginx->>Account: Verify Token (/_accounts)
Account-->>Nginx: Token Valid + User Info
Nginx->>Workspace: Get Workspace Info
Workspace->>CockroachDB: Query Workspace
CockroachDB-->>Workspace: Workspace Data
Workspace-->>Nginx: Workspace Config
Client->>Nginx: WebSocket Connect
Nginx->>Transactor: Proxy to /_transactor
Transactor->>Account: Verify Token
Account-->>Transactor: User Authorized
Transactor->>CockroachDB: Load User Permissions
Transactor-->>Client: Connected
Note over Client,CockroachDB: All services share SERVER_SECRET<br/>for internal authentication
```
---
## Service Summary Table
| Service | Container | Port | Purpose | Dependencies |
|---------|-----------|------|---------|--------------|
| **Reverse Proxy** | | | | |
| nginx | nginx:1.21.3 | 80/443 | Reverse proxy, SSL termination | all services |
| **Frontend** | | | | |
| front | hardcoreeng/front | 8080 | Web application server | account, minio |
| **Core** | | | | |
| account | hardcoreeng/account | 3000 | Authentication & user management | cockroach, redpanda |
| transactor | hardcoreeng/transactor | 3333 | Transaction processing (WebSocket) | cockroach, redpanda, fulltext |
| workspace | hardcoreeng/workspace | - | Workspace management | cockroach, redpanda, minio |
| collaborator | hardcoreeng/collaborator | 3078 | Real-time document collaboration | account, minio |
| **Storage** | | | | |
| kvs (HulyKVS) | hardcoreeng/hulykvs | 8094 | Key-value store | cockroach |
| **Search** | | | | |
| fulltext | hardcoreeng/fulltext | 4700 | Full-text search indexing | elasticsearch, cockroach, rekoni, redpanda |
| rekoni | hardcoreeng/rekoni-service | 4004 | AI/ML recognition service | - |
| **Monitoring** | | | | |
| stats | hardcoreeng/stats | 4900 | Metrics collection | - |
| **Primary Database** | | | | |
| cockroach | cockroachdb/cockroach | 26257 | Distributed SQL database | - |
| **Supporting Infrastructure** | | | | |
| elastic | elasticsearch:7.14.2 | 9200 | Search engine | - |
| minio | minio/minio | 9000/9001 | Object storage | - |
| redpanda | redpandadata/redpanda | 9092/19092 | Event streaming (Kafka) | - |
---
## Environment Variables Summary
### Common Configuration
- `SECRET` / `SERVER_SECRET`: Shared authentication secret
- `STORAGE_CONFIG`: `minio|minio?accessKey=minioadmin&secretKey=minioadmin`
- `QUEUE_CONFIG`: `redpanda:9092`
### Database Configuration
- `DB_URL`: CockroachDB connection string (e.g., `postgresql://user:pass@cockroach:26257/huly`)
- `FULLTEXT_DB_URL`: `http://elastic:9200`
### Service URLs (Internal)
- `ACCOUNTS_URL`: `http://account:3000`
- `TRANSACTOR_URL`: `ws://transactor:3333`
- `FULLTEXT_URL`: `http://fulltext:4700`
- `REKONI_URL`: `http://rekoni:4004`
- `STATS_URL`: `http://stats:4900`
### External URLs (via Nginx)
- Frontend: `http(s)://${HOST_ADDRESS}/`
- Accounts API: `http(s)://${HOST_ADDRESS}/_accounts`
- Transactor WebSocket: `ws(s)://${HOST_ADDRESS}/_transactor`
- Collaborator WebSocket: `ws(s)://${HOST_ADDRESS}/_collaborator`
- Files: `http(s)://${HOST_ADDRESS}/files`
- Rekoni: `http(s)://${HOST_ADDRESS}/_rekoni`
- Stats: `http(s)://${HOST_ADDRESS}/_stats`
---
## Services NOT Included in Self-Hosted
The following services are available in Huly Cloud/Enterprise but **not included** in the self-hosted deployment:
| Service | Purpose |
|---------|---------|
| Datalake | Advanced blob storage management |
| Hulylake | Storage adapter API |
| HulyPulse | WebSocket push notifications |
| HulyGun | Event processing |
| Redis | Cache & pub/sub |
| Rating | Content rating service |
| Print | PDF generation |
| Sign | Digital signatures |
| Payment | Billing integration |
| Export | Data export |
| Analytics | Analytics collection |
| Process | Workflow automation |
| Stream | Video streaming |
| Media | Media processing |
| Preview | Thumbnail generation |
| Backup/Backup-API | Backup services |
| Jaeger | Distributed tracing |