huly-platform/foundations/net/docs/QUICKSTART_HA.md

Quick Start: HA Stateless Containers

⚠️ Important: Network Service Limitation

The Network Server (central coordinator) does NOT support HA:

• ❌ Network service must run as a single instance
• ❌ No clustering or multiple network servers allowed
• ✅ Agents and containers DO support HA (explained below)

This guide covers HA for agents and containers only.

5-Minute Guide

What is it?

A feature that lets multiple agents compete to manage the same container UUID. The first agent wins, and others automatically take over if it fails.

When to use it?

• You need leader election
• You want only one instance of a service running
• You need automatic failover
• You want HA without external coordination services

Basic Example

import { AgentImpl } from '@hcengineering/network-core'
import { createNetworkClient, containerOnAgentEndpointRef } from '@hcengineering/network-client'

// 1. Create your container
class MyService implements Container {
  constructor(readonly uuid: ContainerUuid) {}

  async request(operation: string): Promise<any> {
    return { status: 'active', uuid: this.uuid }
  }

  async terminate(): Promise<void> {
    console.log('Service stopped')
  }

  // ... other required methods
}

// 2. Create agents with serveAgent (both will try to register same UUID)
const sharedUUID = 'my-service-001' as ContainerUuid

// Connect to network
const client = createNetworkClient('localhost:3737')
await client.waitConnection()

// Agent 1 (Primary) - uses serveAgent with stateless containers
await client.serveAgent(
  'localhost:3801',
  {}, // Container factories for dynamic containers
  (agentEndpoint) => {
    // Return stateless containers
    const service1 = new MyService(sharedUUID)
    return [
      {
        uuid: sharedUUID,
        kind: 'my-service' as ContainerKind,
        endpoint: containerOnAgentEndpointRef(agentEndpoint, sharedUUID),
        container: service1
      }
    ]
  }
)

// Agent 2 (Standby) - uses serveAgent with stateless containers
await client.serveAgent(
  'localhost:3802',
  {}, // Container factories for dynamic containers
  (agentEndpoint) => {
    // Return stateless containers
    const service2 = new MyService(sharedUUID)
    return [
      {
        uuid: sharedUUID,
        kind: 'my-service' as ContainerKind,
        endpoint: containerOnAgentEndpointRef(agentEndpoint, sharedUUID),
        container: service2
      }
    ]
  }
)

// Agent 1 accepted, Agent 2 rejected (agent1 already owns it)
// Failover happens automatically when container is removed

Key Methods

// Use serveAgent to create agents with stateless containers
await client.serveAgent(
  endpointUrl,
  containerFactories,
  (agentEndpoint) => [{
    uuid,
    kind,
    endpoint,
    container
  }]
)

// Monitor for failover events
client.onUpdate(async (event) => {
  for (const c of event.containers) {
    if (c.event === NetworkEventKind.removed) {
      console.log('Container removed - failover in progress')
    }
  }

### Key Methods

```typescript
// Add a stateless container to agent
agent.addStatelessContainer(uuid, kind, endpoint, container)

What Happens?

1. Both agents register via serveAgent → First one's container is accepted, second is rejected
2. Rejected agent → Terminates its container instance
3. Active container fails → Network broadcasts removal event
4. Standby agents → Automatically re-register (100ms delay)
5. First standby wins → Takes over as new active instance

Common Patterns

Leader Election

const leaderId = `cluster-${clusterId}-leader` as ContainerUuid

await client.serveAgent(`localhost:${port}`, {}, (agentEndpoint) => [
  {
    uuid: leaderId,
    kind: 'leader' as ContainerKind,
    endpoint: containerOnAgentEndpointRef(agentEndpoint, leaderId),
    container: leaderService
  }
])
// First agent to register becomes leader

Singleton Services

const singletonId = 'migration-service' as ContainerUuid

await client.serveAgent(`localhost:${port}`, {}, (agentEndpoint) => [
  {
    uuid: singletonId,
    kind: 'migration' as ContainerKind,
    endpoint: containerOnAgentEndpointRef(agentEndpoint, singletonId),
    container: migrationService
  }
])
// Only one instance runs across cluster

Database Replica

const dbId = `database-replica-${replicaId}` as ContainerUuid

await client.serveAgent(`localhost:${port}`, {}, (agentEndpoint) => [
  {
    uuid: dbId,
    kind: 'database' as ContainerKind,
    endpoint: containerOnAgentEndpointRef(agentEndpoint, dbId),
    container: databaseReplica
  }
])
// Multiple replicas with same UUID for HA

Gotchas

❌ Don't use different UUIDs on different agents (they won't compete) ✅ Do use the same UUID across all HA agents

❌ Don't manually call agent.addStatelessContainer() - use serveAgent instead ✅ Do use the stateless containers factory parameter in serveAgent

❌ Don't expect instant failover (there's a ~100ms delay) ✅ Do design for eventual consistency

Testing

# Terminal 1: Start network
cd pods/network-pod && rushx dev

# Terminal 2: Run example
npx ts-node examples/ha-stateless-container-example.ts

Next Steps

• Read full docs: docs/HA_STATELESS_CONTAINERS.md
• See working example: examples/ha-stateless-container-example.ts
• Run tests: cd packages/core && rushx test

Questions?

• How long is failover? ~100ms by default
• Can I have 3+ standbys? Yes, first to re-register wins
• Does state transfer? No, containers are stateless
• What about split-brain? No automatic protection (use network redundancy)