openswarm/ASSISTANT_UI_MIGRATION/OVERVIEW.md

assistant-ui + Tool UI Migration Plan

Goal

Replace the custom chat UI in frontend/src/app/pages/AgentChat/ with assistant-ui (React chat primitives) and Tool UI (tool call rendering components). This replaces ~5,800 lines of custom chat code with maintained, accessible, schema-driven components.

Decisions

• Tailwind + MUI coexist: Tailwind is added for assistant-ui/Tool UI; MUI stays for non-chat pages (Dashboard, Settings, etc.)
• Clean swap: No feature flag — old code is replaced directly
• No custom styling: Use default assistant-ui/Tool UI appearance as-is (no theming to match MUI)
• SkillBuilderChat: Included in migration
• Sub-agent rendering: Kept as-is, wired into toolkit as custom tool UIs

Architecture: Before → After

Before

AgentChat.tsx
├── useAgentChat.ts (WS lifecycle, session state)
├── useMessageRendering.ts (branch resolution, render items)
├── ChatInput.tsx + CommandPicker (custom contentEditable + @/slash picker)
├── MessageBubble → UserBubbleContent / AssistantBubbleContent (custom markdown)
├── MessageActionBar (copy/edit/regen/branch)
├── BranchNavigator (custom branch picker)
├── ToolCallBubble + toolCallColors + toolCallUtils (custom terminal rendering)
├── ToolGroupBubble (custom accordion)
├── ApprovalBar / BatchApprovalBar / QuestionForm (custom HITL)
├── GmailCard / McpServiceCards (custom MCP rendering)
├── BrowserAgentInlineFeed (custom progress feed)
├── DiffViewer (custom diff rendering)
└── ThinkingBubble (custom reasoning display)

After

AgentChat.tsx
├── AssistantRuntimeProvider + ExternalStoreRuntime (bridges Redux + WS)
├── OpenSwarmThread (assistant-ui Thread + Message + ActionBar + BranchPicker)
├── OpenSwarmComposer (assistant-ui Composer + ComposerMentionPopover)
├── Toolkit registry
│   ├── native-tools.tsx → Tool UI Terminal, CodeBlock, CodeDiff
│   ├── approval-tools.tsx → Tool UI ApprovalCard, QuestionFlow, OptionList
│   ├── mcp-tools.tsx → Tool UI MessageDraft, DataTable, ProgressTracker
│   └── custom-tools.tsx → AgentToolBubble, ViewBubble (kept as-is)
├── ModelModeSelector (kept, MUI)
├── MessageQueue (kept, MUI)
└── ChatHeader (kept, MUI)

Phases & Dependency Graph

┌─────────────────────────────────────┐
│  PHASE 1 — Sequential (blocking)   │
│  Agent 1: Foundation & Packages     │
└──────────────┬──────────────────────┘
               │
     ┌─────────┼──────────┬──────────────┬──────────────┐
     │         │          │              │              │
     ▼         ▼          ▼              ▼              ▼
┌─────────┐┌─────────┐┌──────────┐┌──────────┐┌──────────────┐
│ Agent 2 ││ Agent 3 ││ Agent 4  ││ Agent 5  ││   Agent 6    │
│ Thread  ││Composer ││ Tool UI: ││ Tool UI: ││  Tool UI:    │
│   &     ││   &     ││ Native   ││Approvals ││  MCP Cards   │
│Messages ││Mentions ││  Tools   ││   &      ││  & Browser   │
│         ││         ││          ││Questions ││    Feed      │
└────┬────┘└────┬────┘└────┬─────┘└────┬─────┘└──────┬───────┘
     │         │          │            │             │
     │    PHASE 2 — All 5 agents run in parallel     │
     │         │          │            │             │
     └─────────┴──────────┴────────────┴─────────────┘
                           │
               ┌───────────▼───────────────┐
               │  PHASE 3 — Sequential     │
               │  Agent 7: Integration,    │
               │  SkillBuilderChat &       │
               │  Cleanup                  │
               └───────────────────────────┘

Agent Summary

Agent	Plan File	Phase	What It Does
1	MIGRATION_AGENT_1.md	1 (sequential)	Install Tailwind, shadcn, assistant-ui, Tool UI packages. Create ExternalStoreRuntime adapter, toolkit skeleton, new directory structure. Scaffold AgentChat.tsx with provider wrapper.
2	MIGRATION_AGENT_2.md	2 (parallel)	Replace message list with Thread. Replace bubbles with Message primitives. Replace MessageActionBar → ActionBar, BranchNavigator → BranchPicker, ThinkingBubble → Reasoning.
3	MIGRATION_AGENT_3.md	2 (parallel)	Replace ChatInput + CommandPicker with Composer + ComposerMentionPopover. Create MentionAdapter for templates/skills/modes/tools/files.
4	MIGRATION_AGENT_4.md	2 (parallel)	Register Tool UI components for native tools: Terminal (bash), CodeBlock (file read), CodeDiff (edit/diff). Replace ToolCallBubble, toolCallColors, toolCallUtils.
5	MIGRATION_AGENT_5.md	2 (parallel)	Register Tool UI components for approvals: ApprovalCard, QuestionFlow, OptionList. Replace ApprovalBar, BatchApprovalBar, QuestionForm, ToolPreview.
6	MIGRATION_AGENT_6.md	2 (parallel)	Register Tool UI for MCP services: MessageDraft (Gmail), DataTable (Calendar/Drive), ProgressTracker (browser feed). Wire AgentToolBubble and ViewBubble as custom toolkit entries.
7	MIGRATION_AGENT_7.md	3 (sequential)	Wire all pieces in AgentChat.tsx. Migrate SkillBuilderChat. Delete all dead files. Verify build.

File Ownership (Parallel Safety)

During Phase 2, each agent only touches files it owns. No conflicts.

Agent	Creates	Modifies	Deletes
2	thread/ directory	Nothing shared	MessageBubble, UserBubbleContent, AssistantBubbleContent, MessageActionBar, BranchNavigator, ThinkingBubble, MessageImageThumbnails, messageBubbleUtils, useMessageRendering
3	composer/ directory	Nothing shared	ChatInput, useChatSubmit, CommandPicker, commandPickerTypes, useCommandPickerItems, CommandPickerIcons, SlashCommandPicker, AttachmentChips, ImageAttachments, richEditorUtils, RichPromptEditor
4	toolkit/native-tools.tsx	Nothing shared	ToolCallBubble, toolCallColors, toolCallUtils, ElapsedTimer, ToolGroupBubble
5	toolkit/approval-tools.tsx	Nothing shared	ApprovalBar, BatchApprovalBar, QuestionForm, ToolPreview, approvalUtils
6	toolkit/mcp-tools.tsx, toolkit/custom-tools.tsx	Nothing shared	GmailCard, McpServiceCards, BrowserAgentInlineFeed, BrowserFeedEntryRow, browserFeedUtils, DiffViewer
7	(integration)	AgentChat.tsx, SkillBuilderChat.tsx	Remaining dead imports/files

Key Technical Notes

ExternalStoreRuntime Adapter

The runtime adapter (runtime/useOpenSwarmRuntime.ts) bridges Redux ↔ assistant-ui:

• Messages: Read from session.messages + session.streamingMessage in Redux
• isRunning: Derived from session.status === 'running'
• onNew: Dispatches sendMessage thunk → WebSocket
• onEdit: Dispatches editMessage thunk → REST API
• onCancel: Dispatches stopAgent thunk → REST API
• Branches: Mapped from session.branches + session.active_branch_id

Message Format Conversion

Redux AgentMessage → assistant-ui format:

• role: 'user' → { role: 'user', content: [{ type: 'text', text }] }
• role: 'assistant' → { role: 'assistant', content: [{ type: 'text', text }] }
• role: 'tool_call' → { role: 'assistant', content: [{ type: 'tool-call', toolCallId, toolName, args }] }
• role: 'tool_result' → { role: 'tool', content: [{ type: 'tool-result', toolCallId, result }] }

Tailwind + MUI Coexistence

Tailwind is configured with a prefix or scoped to assistant-ui components to avoid conflicts with MUI's global styles. The important selector strategy or Tailwind's prefix option may be needed.

Tool UI Registration Pattern

All Tool UI components are registered in toolkit files. Each toolkit file exports a partial Toolkit object. The toolkit/index.ts merges them:

import { nativeToolkit } from './native-tools';
import { approvalToolkit } from './approval-tools';
import { mcpToolkit } from './mcp-tools';
import { customToolkit } from './custom-tools';

export const toolkit: Toolkit = {
  ...nativeToolkit,
  ...approvalToolkit,
  ...mcpToolkit,
  ...customToolkit,
};

Lines of Code Impact (Estimated)

Category	Lines
Deleted (old components)	~5,800
New bridge/adapter code	~600–800
Tool UI components (installed, not authored)	~2,000 (maintained externally)
Net reduction in authored code	~5,000

Estimated Effort

Phase	Agents	Estimated Time
Phase 1	Agent 1	1–2 hours
Phase 2	Agents 2–6 (parallel)	2–4 hours each, ~4 hours wall clock
Phase 3	Agent 7	1–2 hours
Total wall clock		~6–8 hours