huly-platform/docs/QUICKSTART_HA.md

Quick Start: HA Stateless Containers

⚠️ Important: Network Service Limitation

The Network Server (central coordinator) does NOT support HA:

• ❌ Network service must run as a single instance
• ❌ No clustering or multiple network servers allowed
• ✅ Agents and containers DO support HA (explained below)

This guide covers HA for agents and containers only.

5-Minute Guide

What is it?

A feature that lets multiple agents compete to manage the same container UUID. The first agent wins, and others automatically take over if it fails.

When to use it?

• You need leader election
• You want only one instance of a service running
• You need automatic failover
• You want HA without external coordination services

Basic Example

import { AgentImpl } from '@hcengineering/network-core'
import { createNetworkClient } from '@hcengineering/network-client'

// 1. Create your container
class MyService implements Container {
  constructor(readonly uuid: ContainerUuid) {}

  async request(operation: string): Promise<any> {
    return { status: 'active', uuid: this.uuid }
  }

  async terminate(): Promise<void> {
    console.log('Service stopped')
  }

  // ... other required methods
}

// 2. Create agents (both will try to register same UUID)
const sharedUUID = 'my-service-001' as ContainerUuid

// Agent 1 (Primary) - Note: In production, use serveAgent() on the client
const agent1 = new AgentImpl('agent-1', {})
const service1 = new MyService(sharedUUID)
agent1.addStatelessContainer(
  sharedUUID,
  'my-service' as ContainerKind,
  'service://agent1/service-001' as ContainerEndpointRef,
  service1
)

// Agent 2 (Standby) - Note: In production, use serveAgent() on the client
const agent2 = new AgentImpl('agent-2', {})
const service2 = new MyService(sharedUUID)
agent2.addStatelessContainer(
  sharedUUID,
  'my-service' as ContainerKind,
  'service://agent2/service-001' as ContainerEndpointRef,
  service2
)

// 3. Connect and register
const client = createNetworkClient('localhost:3737')
await client.waitConnection()

await client.register(agent1) // ✅ Accepted
await client.register(agent2) // ❌ Rejected (agent1 already owns it)

// 4. Failover happens automatically
await agent1.terminate(sharedUUID) // Agent1 stops
// After ~100ms, agent2 automatically takes over

Key Methods

// Add a stateless container to agent
agent.addStatelessContainer(uuid, kind, endpoint, container)

// Remove from tracking (if needed)
agent.removeStatelessContainer(uuid)

// Register with network (handles conflicts automatically)
await client.register(agent)

// Monitor for failover events
client.onUpdate(async (event) => {
  for (const c of event.containers) {
    if (c.event === NetworkEventKind.removed) {
      console.log('Container removed - failover in progress')
    }
  }
})

What Happens?

1. Both agents register → First one is accepted, second is rejected
2. Rejected agent → Terminates its container instance
3. Active container fails → Network broadcasts removal event
4. Standby agents → Automatically re-register (100ms delay)
5. First standby wins → Takes over as new active instance

Common Patterns

Leader Election

const leaderId = `cluster-${clusterId}-leader` as ContainerUuid
agent.addStatelessContainer(leaderId, 'leader', endpoint, leaderService)
// First agent to register becomes leader

Singleton Service

const singletonId = 'migration-service' as ContainerUuid
agent.addStatelessContainer(singletonId, 'migration', endpoint, migrationService)
// Only one instance will run migrations

Active-Standby Database

const dbId = 'database-primary' as ContainerUuid
agent.addStatelessContainer(dbId, 'database', endpoint, databaseReplica)
// Primary serves writes, standby automatically promotes on failure

Gotchas

❌ Don't use different UUIDs on different agents (they won't compete) ✅ Do use the same UUID across all HA agents

❌ Don't forget to handle termination of rejected containers ✅ Do let the agent.register() method handle it automatically

❌ Don't expect instant failover (there's a ~100ms delay) ✅ Do design for eventual consistency

Testing

# Terminal 1: Start network
cd pods/network-pod && rushx dev

# Terminal 2: Run example
npx ts-node examples/ha-stateless-container-example.ts

Next Steps

• Read full docs: docs/HA_STATELESS_CONTAINERS.md
• See working example: examples/ha-stateless-container-example.ts
• Run tests: cd packages/core && rushx test

Questions?

• How long is failover? ~100ms by default
• Can I have 3+ standbys? Yes, first to re-register wins
• Does state transfer? No, containers are stateless
• What about split-brain? No automatic protection (use network redundancy)