From 42c4ceac8022e32bfbc6ddad57c2397a2ae8bfa9 Mon Sep 17 00:00:00 2001 From: haikdc Date: Mon, 30 Mar 2026 04:51:12 -0700 Subject: [PATCH] [Haik]: ckpt, cleaned up readmes --- CLEANUP_AGENT_1.md | 202 --------------------------- CLEANUP_AGENT_2.md | 289 -------------------------------------- CLEANUP_AGENT_3.md | 253 --------------------------------- CLEANUP_AGENT_4.md | 340 --------------------------------------------- README.md | 1 + 5 files changed, 1 insertion(+), 1084 deletions(-) delete mode 100644 CLEANUP_AGENT_1.md delete mode 100644 CLEANUP_AGENT_2.md delete mode 100644 CLEANUP_AGENT_3.md delete mode 100644 CLEANUP_AGENT_4.md diff --git a/CLEANUP_AGENT_1.md b/CLEANUP_AGENT_1.md deleted file mode 100644 index 3156988a..00000000 --- a/CLEANUP_AGENT_1.md +++ /dev/null @@ -1,202 +0,0 @@ -# Agent 1: Foundations + Hygiene (Phase 0 + Phase 4) - -## Context - -You are cleaning up the OpenSwarm codebase. This is the first of 4 agents. Your job is to build shared infrastructure that the other agents will depend on, plus minor hygiene fixes. - -**Rules:** -- Every file you create or modify must be <250 lines of code -- Keep code DRY -- Do NOT touch the `9router/` or `debugger/` directories -- Do NOT touch tests -- Run the app after your changes to make sure nothing is broken: `cd backend && python -m backend.main` - ---- - -## Task 0A: Create `backend/apps/common/json_store.py` - -**Problem:** Every sub-app copy-pastes identical `_load_all()`, `_save()`, `_load()`, `_delete()` functions for JSON-file CRUD. This exists in: - -- `backend/apps/tools_lib/tools_lib.py` (lines 255-277) -- `backend/apps/outputs/outputs.py` (lines 103-134) -- `backend/apps/dashboards/dashboards.py` (lines 27-55) -- `backend/apps/agents/agent_manager.py` (lines 33-61) -- `backend/apps/modes/modes.py` -- `backend/apps/templates/templates.py` -- `backend/apps/skills/skills.py` - -**What to do:** - -1. Create `backend/apps/common/__init__.py` (empty) -2. Create `backend/apps/common/json_store.py` with a generic `JsonStore` class: - -```python -class JsonStore(Generic[T]): - def __init__(self, model_cls: type[T], data_dir: str, id_field: str = "id"): - ... - def load_all(self) -> list[T]: ... - def save(self, item: T) -> None: ... - def load(self, item_id: str) -> T: ... # raises HTTPException(404) if not found - def delete(self, item_id: str) -> None: ... - def exists(self, item_id: str) -> bool: ... -``` - -3. Replace the inline `_load_all`, `_save`, `_load`, `_delete` in **all 7 sub-apps** with `JsonStore` instances. -4. For the agents session store, it's slightly different (uses `session_id` not `id`, and has `_load_all_session_data` returning tuples). Create a small subclass or adapter — keep it clean. - -**Verify:** All existing imports of `_load_all`, `_save`, `_load` from these modules still work (they're imported in `agent_manager.py`, `outputs.py`, `dashboards.py`, etc). If external code imports them, keep backward-compatible aliases. - ---- - -## Task 0B: Create `backend/apps/common/model_registry.py` - -**Problem:** Model name-to-ID mappings are duplicated with inconsistent values: - -- `backend/apps/agents/browser_agent.py` lines 24-28: `MODEL_MAP` with `"sonnet": "claude-sonnet-4-6"` -- `backend/apps/outputs/outputs.py` lines 23-27: `MODEL_MAP` with `"sonnet": "claude-sonnet-4-20250514"` -- `backend/apps/agents/providers/registry.py` lines 25-31: `BUILTIN_MODELS` dict -- `backend/apps/agents/providers/registry.py` lines 284-309: `COST_PER_1M_TOKENS` dict - -**What to do:** - -1. Create `backend/apps/common/model_registry.py` with: - -```python -@dataclass -class ModelDef: - value: str # short name ("sonnet") - label: str # display name ("Claude Sonnet 4.6") - model_id: str # API model ID ("claude-sonnet-4-6") - provider: str # "Anthropic", "OpenAI", etc. - api: str # "anthropic", "openai", "gemini", "openrouter" - context_window: int - input_cost_per_1m: float - output_cost_per_1m: float - -ALL_MODELS: list[ModelDef] = [ ... ] # Single source of truth - -def resolve_model_id(short_name: str) -> str: ... -def get_cost_rates(provider: str, model: str) -> tuple[float, float] | None: ... -def calculate_cost(provider: str, model: str, input_tokens: int, output_tokens: int) -> float: ... -def get_context_window(model: str) -> int: ... -def get_builtin_models_by_provider() -> dict[str, list[dict]]: ... -``` - -2. Use the canonical model IDs from `providers/registry.py` BUILTIN_MODELS (the `model_id` field there is the correct one: `claude-sonnet-4-6`, `claude-opus-4-6`, etc). - -3. Delete `MODEL_MAP` from `browser_agent.py` and `outputs.py`, replace with `resolve_model_id()` import. - -4. Delete `BUILTIN_MODELS` and `COST_PER_1M_TOKENS` from `providers/registry.py`, replace with imports from `model_registry.py`. Update `get_available_models()`, `calculate_cost()`, `get_context_window()` in `registry.py` to delegate to the new module. - ---- - -## Task 0C: Create `backend/apps/common/mcp_utils.py` - -**Problem:** `_parse_sse_json()` is duplicated verbatim in: -- `backend/apps/tools_lib/tools_lib.py` (lines 773-787) -- `backend/apps/agents/mcp_client.py` (lines 234-249) - -Also, `_sanitize_server_name()` is defined in `tools_lib.py` (line 481) but imported into `agent_manager.py`. - -**What to do:** - -1. Create `backend/apps/common/mcp_utils.py` with: - - `parse_sse_json(text: str) -> dict | None` - - `sanitize_server_name(name: str) -> str` - -2. Update `tools_lib.py` to import from `common.mcp_utils` instead of defining locally. Keep `_sanitize_server_name` as a backward-compatible alias: `_sanitize_server_name = sanitize_server_name`. - -3. Update `mcp_client.py` to import `parse_sse_json` from `common.mcp_utils` instead of having its own `_parse_sse_json` static method. - -4. Update `agent_manager.py` imports to use `from backend.apps.common.mcp_utils import sanitize_server_name`. - ---- - -## Task 0D: Create `backend/apps/common/llm_helpers.py` - -**Problem:** Multiple files independently construct Anthropic clients, call `messages.create`, strip markdown fences from responses, and parse JSON — with identical error handling. This happens in: - -- `agent_manager.py` `generate_title()` (lines 1511-1539) -- `agent_manager.py` `generate_group_meta()` (lines 1541-1617) -- `dashboards.py` `generate_name()` (lines 133-190) -- `outputs.py` `vibe_code()` (lines 351-419) -- `outputs.py` `auto_run_output()` (lines 430-480) - -**What to do:** - -1. Create `backend/apps/common/llm_helpers.py` with: - -```python -async def quick_llm_call( - system: str, - user_content: str, - model: str = "claude-sonnet-4-20250514", - max_tokens: int = 300, -) -> str: - """Make a simple LLM call and return the text response. Handles client construction.""" - -async def quick_llm_json( - system: str, - user_content: str, - model: str = "claude-sonnet-4-20250514", - max_tokens: int = 300, -) -> dict: - """Make an LLM call expecting JSON. Strips markdown fences, parses JSON.""" - -def strip_markdown_fences(text: str) -> str: - """Remove ```json ... ``` or similar fences from LLM output.""" -``` - -2. These should use `get_anthropic_client` from `backend/apps/settings/credentials.py` internally. - -3. **Do not** refactor the callers yet — that's Agent 2's job. Just create the helpers so they're available. - ---- - -## Task 4A: Clean Up `modes/models.py` - -**Problem:** `backend/apps/modes/models.py` (171 lines) has ~120 lines of `BUILTIN_MODES` data mixed in with Pydantic schema definitions. - -**What to do:** - -1. Create `backend/apps/modes/builtin.py` and move the `BUILTIN_MODES` list there. -2. `modes/models.py` should only contain `Mode`, `ModeCreate`, `ModeUpdate` Pydantic models. -3. Update `modes/modes.py` to import `BUILTIN_MODES` from `builtin.py` instead of `models.py`. -4. Check for any other files that import `BUILTIN_MODES` from `models.py` and update them. - ---- - -## Task 4B: DRY Up `outputs/models.py` Migration Validators - -**Problem:** `backend/apps/outputs/models.py` (197 lines) has `_migrate_flat_fields` copy-pasted across 4 classes: `Output`, `OutputCreate`, `OutputUpdate`, `WorkspaceSeedRequest`. - -**What to do:** - -1. Extract a shared function at module level: - -```python -def _migrate_legacy_files(data: dict, allow_schema_json: bool = False) -> dict: - """Convert legacy frontend_code/backend_code fields into the files dict.""" - ... -``` - -2. Have each `@model_validator` call this shared function instead of duplicating the logic. - ---- - -## Task 4C: Remove Dead Code - -1. `backend/apps/tools_lib/tools_lib.py` lines 640-646: empty comment section `# OAuth2 flow for Google Workspace (and other OAuth providers)` followed by another empty comment `# MCP tool discovery`. Remove the stale comments. - ---- - -## Verification - -After all tasks are complete: - -1. `cd /Users/haikdecie/Desktop/openswarm-ai/openswarm/backend` -2. `python -c "from backend.apps.common.json_store import JsonStore; print('OK')"` -3. `python -c "from backend.apps.common.model_registry import resolve_model_id; print(resolve_model_id('sonnet'))"` -4. `python -c "from backend.apps.common.mcp_utils import parse_sse_json, sanitize_server_name; print('OK')"` -5. `python -c "from backend.apps.common.llm_helpers import quick_llm_call; print('OK')"` -6. Verify no existing imports are broken by running: `python -c "from backend.main import app; print('OK')"` diff --git a/CLEANUP_AGENT_2.md b/CLEANUP_AGENT_2.md deleted file mode 100644 index da917fbd..00000000 --- a/CLEANUP_AGENT_2.md +++ /dev/null @@ -1,289 +0,0 @@ -# Agent 2: Split Backend God Objects (Phase 1) - -## Context - -You are cleaning up the OpenSwarm codebase. This is agent 2 of 4. Agent 1 has already completed Phase 0 (foundations), creating these shared utilities that you should USE: - -- `backend/apps/common/json_store.py` — Generic `JsonStore[T]` for JSON file CRUD -- `backend/apps/common/model_registry.py` — Single source of truth for model definitions, `resolve_model_id()`, `calculate_cost()`, etc. -- `backend/apps/common/mcp_utils.py` — `parse_sse_json()`, `sanitize_server_name()` -- `backend/apps/common/llm_helpers.py` — `quick_llm_call()`, `quick_llm_json()`, `strip_markdown_fences()` - -**Rules:** -- Every file you create or modify must be <250 lines of code -- Keep code DRY — use the Phase 0 utilities -- Do NOT touch `9router/`, `debugger/`, or `frontend/` -- Do NOT touch tests -- Run the app after each major split to verify nothing is broken: `cd backend && python -m backend.main` - ---- - -## Task 1A: Split `agent_manager.py` (2093 lines → ~6 files) - -This is the most important task. `backend/apps/agents/agent_manager.py` is a 2093-line god object. - -Read it carefully first. The `AgentManager` class has these distinct responsibilities that should be separated: - -### New file: `backend/apps/agents/prompt_builder.py` (~200 lines) - -Extract these methods from `AgentManager`: -- `_resolve_mode()` -- `_compose_system_prompt()` -- `_build_connected_tools_context()` -- `_build_outputs_context()` -- `_build_browser_context()` -- `_get_pre_selected_browser_ids()` -- `_resolve_context_paths()` -- `_build_dir_tree()` -- `_resolve_forced_tools()` -- `_resolve_attached_skills()` -- `_build_prompt_content()` - -Make these standalone functions (not methods) that take the data they need as parameters. For example: -```python -def compose_system_prompt( - default_prompt: str | None, mode_prompt: str | None, - session_prompt: str | None, connected_tools_ctx: str | None = None, - outputs_ctx: str | None = None, browser_ctx: str | None = None, -) -> str | None: -``` - -### New file: `backend/apps/agents/mcp_builder.py` (~200 lines) - -Extract: -- `_build_mcp_servers()` — builds the mcp_servers dict for ClaudeAgentOptions -- `_get_effective_policy()` — the permission policy resolver (currently a nested function inside `_run_agent_loop`) -- `_get_denied_tool_names()`, `_get_all_known_tool_names()`, `_is_fully_denied()` — the module-level helper functions -- `get_all_tool_names()` — the module-level function -- `FULL_TOOLS` — the constant list -- Logic for building `effective_allowed` and `effective_disallowed` tool lists (currently ~50 lines inside `_run_agent_loop`) -- Logic for adding browser-agent and invoke-agent MCP servers - -### New file: `backend/apps/agents/session_store.py` (~200 lines) - -Extract session persistence and history. Use `JsonStore` from `backend/apps/common/json_store.py` where possible: -- `_save_session()`, `_load_session_data()`, `_delete_session_file()`, `_load_all_session_data()` -- `get_history()` method -- `_build_search_text()` static method -- `reconcile_on_startup()` -- `persist_all_sessions()` -- `restore_all_sessions()` -- `get_browser_agent_children()` - -### Refactor: `backend/apps/agents/agent_loop.py` (replace existing, ~250 lines) - -The existing `agent_loop.py` (331 lines) is an older/unused file. Replace it with the extracted `_run_agent_loop` method from `agent_manager.py`. - -Extract from `AgentManager`: -- `_run_agent_loop()` — the main SDK query loop (lines 501-1128). This is the biggest single method. -- The hook functions (`pre_tool_hook`, `post_tool_hook`, `can_use_tool`) that are currently nested inside `_run_agent_loop` -- `_run_mock_agent()` — the development mock (lines 1172-1259) -- `_stream_text()` and `_stream_tool_input()` — streaming helpers (lines 1130-1170) -- `_fire_session_completed()` — analytics for completed sessions - -Use `quick_llm_call` and `quick_llm_json` from `common/llm_helpers.py` for `generate_title()` and `generate_group_meta()`. - -### Slim down: `backend/apps/agents/agent_manager.py` (~250 lines) - -What remains in `AgentManager`: -- `__init__()` — holds `self.sessions` dict and `self.tasks` dict -- `launch_agent()` -- `send_message()` -- `stop_agent()` -- `handle_approval()` -- `edit_message()` -- `switch_branch()` -- `generate_title()` — refactor to use `quick_llm_call()` -- `generate_group_meta()` — refactor to use `quick_llm_json()` -- `update_session()` -- `close_session()` -- `delete_session()` -- `resume_session()` -- `duplicate_session()` -- `invoke_agent()` -- `get_all_sessions()`, `get_session()` - -Each of these methods becomes a thin coordinator that calls into the extracted modules. - -The `agent_manager = AgentManager()` singleton stays at the bottom of this file. - -### Important Notes for this split: - -- `_run_agent_loop` has deeply nested closures (`can_use_tool`, `pre_tool_hook`, `post_tool_hook`, `prompt_stream`). When extracting to `agent_loop.py`, you'll need to pass the session, ws_manager, and other dependencies as parameters to these functions. -- The `duplicate_session` and `invoke_agent` methods have duplicated message-copying logic (~30 lines each). Extract a shared `_copy_session_messages()` helper into `session_store.py`. -- Keep the `agent_manager` singleton import path unchanged: `from backend.apps.agents.agent_manager import agent_manager` must still work. - ---- - -## Task 1B: Split `tools_lib.py` (1153 lines → 5-6 files) - -Convert `backend/apps/tools_lib/` from a single file into a package. - -### Step 1: Create the package structure - -``` -backend/apps/tools_lib/ -├── __init__.py # SubApp instance + backward-compat imports -├── routes.py # Tool CRUD endpoints (~120 lines) -├── oauth.py # OAuth start, callback, disconnect, refresh (~200 lines) -├── oauth_providers.py # OAuthProvider dataclass + OAUTH_PROVIDERS registry (~180 lines) -├── mcp_config.py # derive_mcp_config, _resolve_command, _augmented_path, _extra_bin_dirs (~180 lines) -├── mcp_discovery.py # discover_tools endpoint + stdio/HTTP/SSE discovery (~200 lines) -├── classification.py # _SERVICE_RULES, _categorize_tool, _extract_service (~100 lines) -└── models.py # Already exists, keep as-is -``` - -### Step 2: What goes where - -**`oauth_providers.py`** — Pure data, no dependencies: -- `OAuthProvider` dataclass (lines 86-107) -- `OAUTH_PROVIDERS` dict (lines 109-239) -- `_resolve_oauth_provider()` helper (lines 242-248) - -**`oauth.py`** — OAuth flow logic: -- `_pending_oauth` and `_pending_pkce` state dicts (lines 251-252) -- `oauth_callback()` endpoint (lines 319-435) -- `oauth_start()` endpoint (lines 1062-1098) -- `oauth_disconnect()` endpoint (lines 1036-1059) -- `refresh_oauth_token()` function (lines 1102-1153) -- `refresh_google_token` alias - -**`mcp_config.py`** — MCP server config derivation: -- `_extra_bin_dirs()` (lines 486-513) -- `_resolve_command()` (lines 516-539) -- `_augmented_path()` (lines 542-552) -- `derive_mcp_config()` (lines 555-637) - -**`mcp_discovery.py`** — MCP tool discovery: -- `_discover_mcp_tools_http()` (lines 790-830) -- `_discover_mcp_tools_sse()` (lines 833-856) -- `_discover_mcp_tools_stdio()` (lines 859-941) -- `discover_tools()` endpoint (lines 944-1033) - -**`classification.py`** — Tool categorization (pure data + logic): -- `_READ_PREFIXES`, `_WRITE_PREFIXES` (lines 649-651) -- `_SERVICE_RULES` (lines 653-748) -- `_categorize_tool()` (lines 751-760) -- `_extract_service()` (lines 763-770) - -**`routes.py`** — CRUD endpoints: -- `list_builtin_tools()`, `list_tools()`, `get_tool()`, `create_tool()`, `update_tool()`, `delete_tool()` (lines 279-474) -- `load_builtin_permissions()`, `save_builtin_permissions()`, permission endpoints (lines 284-311) -- Use `JsonStore` from `backend/apps/common/json_store.py` for the `_load_all`, `_save`, `_load` functions - -**`__init__.py`** — Glue: -- `tools_lib` SubApp instance -- `tools_lib_lifespan` -- Backward-compatible imports so that `from backend.apps.tools_lib.tools_lib import _load_all, derive_mcp_config, ...` still works. Add a `tools_lib.py` shim file OR add these to `__init__.py`. -- Wire up all routes from the sub-modules - -### Critical: Maintain backward compatibility - -These imports exist in `agent_manager.py` and must keep working: -```python -from backend.apps.tools_lib.tools_lib import ( - _load_all as load_all_tools, - _sanitize_server_name, - derive_mcp_config, - load_builtin_permissions, - refresh_google_token, -) -``` - -Keep a `tools_lib.py` file that re-exports from the new package modules, OR update all import sites. - ---- - -## Task 1C: Split `outputs.py` (593 lines → 3 files) - -### New file: `backend/apps/outputs/helpers.py` (~80 lines) - -Extract: -- `_build_data_injection()` (lines 53-70) -- `_inject_data_into_html()` (lines 73-79) -- `_decode_data_param()` (lines 82-90) -- `_validate_against_schema()` (lines 41-48) -- `_walk_directory()` (lines 136-150) - -### New file: `backend/apps/outputs/ai_generation.py` (~200 lines) - -Extract: -- `VIBE_CODE_SYSTEM_PROMPT` (lines 334-348) -- `vibe_code()` endpoint (lines 351-419) — refactor to use `quick_llm_json` from `common/llm_helpers.py` -- `AUTO_RUN_SYSTEM_PROMPT` (lines 422-427) -- `auto_run_output()` endpoint (lines 430-480) — refactor to use `quick_llm_call` -- `AUTO_RUN_AGENT_SYSTEM_PROMPT` (lines 525-541) -- `auto_run_agent()` endpoint (lines 544-581) -- `cleanup_auto_run_agent()` endpoint (lines 584-593) -- `_get_anthropic_client()` helper (lines 34-38) — or replace with direct `quick_llm_*` usage -- `_resolve_model()` — replace with `resolve_model_id` from `common/model_registry.py` - -### Slim down: `backend/apps/outputs/outputs.py` (~200 lines) - -What remains: -- SubApp instance + lifespan -- CRUD endpoints (list, get, create, update, delete) -- Workspace endpoints (read, seed, write file, delete file) -- File serving endpoints (serve_workspace_file, serve_output_file) -- `_load_all`, `_save`, `_load`, `load_output` — replace with `JsonStore` - -Wire the ai_generation routes into the outputs router. - ---- - -## Task 1D: Split `browser_agent.py` (632 lines → 3 files) - -### New file: `backend/apps/agents/browser/schemas.py` (~100 lines) - -Extract: -- `BROWSER_TOOLS_SCHEMA` (lines 30-158) -- `ACTION_MAP` (lines 160-170) -- `SYSTEM_PROMPT` (lines 172-193) -- `MAX_TURNS` constant (line 195) - -Replace the local `MODEL_MAP` (lines 24-28) with `resolve_model_id` from `common/model_registry.py`. - -### New file: `backend/apps/agents/browser/executor.py` (~120 lines) - -Extract: -- `execute_browser_tool()` (lines 198-211) -- `_format_tool_result()` (lines 214-234) -- `_request_browser_approval()` (lines 237-274) - -### New file: `backend/apps/agents/browser/runner.py` (~250 lines) - -Extract: -- `run_browser_agent()` (lines 277-549) -- `_create_browser_card()` (lines 552-580) -- `run_browser_agents()` (lines 583-632) - -Create `backend/apps/agents/browser/__init__.py` that re-exports the public API: -```python -from backend.apps.agents.browser.runner import run_browser_agent, run_browser_agents -``` - -Update `backend/main.py` line 177 which imports `from backend.apps.agents.browser_agent import run_browser_agents`. - ---- - -## Verification - -After all tasks are complete: - -1. Verify no file in `backend/` exceeds 250 lines: - ```bash - find backend -name '*.py' -not -path '*/__pycache__/*' -not -path '*/test*' | xargs wc -l | sort -rn | head -20 - ``` - -2. Verify the app starts: - ```bash - cd backend && python -c "from backend.main import app; print('App created OK')" - ``` - -3. Verify key imports still work: - ```bash - python -c "from backend.apps.agents.agent_manager import agent_manager; print('OK')" - python -c "from backend.apps.tools_lib.tools_lib import _load_all, derive_mcp_config, load_builtin_permissions; print('OK')" - python -c "from backend.apps.agents.browser_agent import run_browser_agents; print('OK')" - python -c "from backend.apps.outputs.outputs import _load_all; print('OK')" - ``` diff --git a/CLEANUP_AGENT_3.md b/CLEANUP_AGENT_3.md deleted file mode 100644 index f5f5b572..00000000 --- a/CLEANUP_AGENT_3.md +++ /dev/null @@ -1,253 +0,0 @@ -# Agent 3: DRY Up Cross-Cutting Patterns (Phase 2) - -## Context - -You are cleaning up the OpenSwarm codebase. This is agent 3 of 4. Agents 1 and 2 have already completed: - -- **Phase 0:** Shared utilities in `backend/apps/common/` (json_store, model_registry, mcp_utils, llm_helpers) -- **Phase 1:** Split god objects — `agent_manager.py`, `tools_lib.py`, `outputs.py`, `browser_agent.py` are now modular packages/files - -**Rules:** -- Every file you create or modify must be <250 lines of code -- Keep code DRY -- Do NOT touch `9router/`, `debugger/`, or `frontend/` -- Do NOT touch tests -- Verify the app still starts after each major change - ---- - -## Task 2A: Extract Shared Approval Flow - -**Problem:** Two places implement independent HITL (human-in-the-loop) approval request patterns: - -1. **`agent_manager.py`** (now possibly in `agent_loop.py` after Phase 1 split) — `_request_user_approval()` function: - - Creates an `ApprovalRequest` - - Appends to `session.pending_approvals` - - Sets `session.status = "waiting_approval"` - - Sends WS status event - - Fires analytics event - - Waits for decision via `ws_manager.send_approval_request()` - - Fires another analytics event with latency - - Removes from pending approvals - - Restores `session.status = "running"` - - Sends WS status event - -2. **`browser_agent.py`** (now possibly in `browser/executor.py`) — `_request_browser_approval()`: - - Same pattern but without analytics events and with a timeout - -**What to do:** - -1. Create `backend/apps/agents/approval.py` (~80 lines): - -```python -async def request_approval( - session: AgentSession, - tool_name: str, - tool_input: dict, - timeout: float | None = None, - track_analytics: bool = True, -) -> dict: - """Unified HITL approval flow. - - Creates an ApprovalRequest, sends it via WebSocket, waits for the user's - decision, cleans up, and returns the decision dict. - - Returns: {"behavior": "allow"|"deny", "message": ..., "updated_input": ...} - """ -``` - -2. Replace both implementations with calls to this shared function. - -3. Make sure the analytics tracking is optional (browser agents don't currently track approval analytics). - ---- - -## Task 2B: Extract WebSocket Event Helpers - -**Problem:** Throughout the codebase (especially `agent_manager.py` / `agent_loop.py` / `browser_agent.py`), there are 30+ calls that look like: - -```python -await ws_manager.send_to_session(session_id, "agent:status", { - "session_id": session_id, - "status": "running", - "session": session.model_dump(mode="json"), -}) -``` - -The payload construction is repeated identically every time for each event type. - -**What to do:** - -Add typed convenience methods to `backend/apps/agents/ws_manager.py`. The file is currently 126 lines so there's room: - -```python -async def emit_status(self, session_id: str, status: str, session: AgentSession): - await self.send_to_session(session_id, "agent:status", { - "session_id": session_id, - "status": status, - "session": session.model_dump(mode="json"), - }) - -async def emit_message(self, session_id: str, message: Message): - await self.send_to_session(session_id, "agent:message", { - "session_id": session_id, - "message": message.model_dump(mode="json"), - }) - -async def emit_cost_update(self, session_id: str, cost_usd: float): - await self.send_to_session(session_id, "agent:cost_update", { - "session_id": session_id, - "cost_usd": cost_usd, - }) - -async def emit_stream_start(self, session_id: str, message_id: str, role: str, tool_name: str = ""): - payload = {"session_id": session_id, "message_id": message_id, "role": role} - if tool_name: - payload["tool_name"] = tool_name - await self.send_to_session(session_id, "agent:stream_start", payload) - -async def emit_stream_delta(self, session_id: str, message_id: str, delta: str): - await self.send_to_session(session_id, "agent:stream_delta", { - "session_id": session_id, - "message_id": message_id, - "delta": delta, - }) - -async def emit_stream_end(self, session_id: str, message_id: str): - await self.send_to_session(session_id, "agent:stream_end", { - "session_id": session_id, - "message_id": message_id, - }) -``` - -Then do a find-and-replace across all files that construct these payloads manually. Replace with the typed helpers. This is a mechanical change — just make sure every event type is covered. - -**Important:** If `ws_manager.py` exceeds 250 lines after adding these helpers, split it into `ws_manager.py` (connection management) and `ws_events.py` (typed event emitters). - ---- - -## Task 2C: Move Subscription Routes to New Sub-App - -**Problem:** `backend/apps/agents/agents.py` (305 lines) contains ~120 lines of 9Router/subscription endpoints (lines 186-305) that have nothing to do with agent sessions: -- `subscriptions_status()` -- `subscriptions_connect()` -- `subscriptions_poll()` -- `subscriptions_exchange()` -- `subscriptions_models()` -- `subscriptions_disconnect()` - -**What to do:** - -1. Create `backend/apps/subscriptions/__init__.py` (empty) -2. Create `backend/apps/subscriptions/subscriptions.py` (~130 lines): - - Move all 6 subscription endpoints here - - Create a new SubApp: `subscriptions = SubApp("subscriptions", subscriptions_lifespan)` - - The lifespan can be a simple no-op (or move the 9Router auto-start from analytics lifespan here if it makes more sense) - -3. Update `backend/main.py`: - - Import the new `subscriptions` sub-app - - Add it to the `MainApp` list - -4. Remove the subscription endpoints from `agents.py`. This should bring `agents.py` down to ~185 lines. - -5. **Frontend impact:** The frontend calls these endpoints at `/api/agents/subscriptions/*`. The new path will be `/api/subscriptions/*`. Search the frontend for these API paths and update them: - ```bash - grep -r "api/agents/subscriptions" frontend/src/ - ``` - Update all matches to use `/api/subscriptions/` instead. - ---- - -## Task 2D: Clean Up `main.py` - -**Problem:** `backend/main.py` (256 lines) has inline WebSocket handlers, OAuth callback endpoints, browser-agent HTTP endpoint, and invoke-agent HTTP endpoint that should live in their respective sub-apps. - -**What to do:** - -### Move WebSocket handlers - -The two WebSocket handlers (`websocket_session` and `websocket_dashboard`, lines 41-104) can't easily move to a SubApp router because FastAPI WebSocket routes need to be on the main app. However, the message dispatch logic inside them can be extracted. - -Create `backend/apps/agents/ws_routes.py` (~80 lines): - -```python -async def handle_session_message(session_id: str, event: str, payload: dict): - """Dispatch a WebSocket message for a session.""" - if event == "agent:send_message": - from backend.apps.agents.agent_manager import agent_manager - await agent_manager.send_message(...) - elif event == "agent:approval_response": - ... - elif event == "agent:edit_message": - ... - elif event == "agent:stop": - ... - -async def handle_dashboard_message(event: str, payload: dict): - """Dispatch a WebSocket message for the dashboard.""" - ... -``` - -Then `main.py`'s WebSocket handlers become thin wrappers: -```python -@app.websocket("/ws/agents/{session_id}") -async def websocket_session(websocket: WebSocket, session_id: str): - await ws_manager.connect_session(session_id, websocket) - try: - while True: - data = await websocket.receive_text() - msg = json.loads(data) - await handle_session_message(session_id, msg.get("event"), msg.get("data", {})) - except WebSocketDisconnect: - ws_manager.disconnect_session(session_id, websocket) -``` - -### Move HTTP endpoints - -- `/api/browser/command` (lines 107-122) → Move to `agents/browser/` as a route on the agents router, or keep in main.py if it needs to be at the root level -- `/api/browser-agent/run` (lines 172-196) → Move to agents router -- `/api/invoke-agent/run` (lines 199-227) → Move to agents router -- `/api/subscriptions/pending/{state}` (lines 125-136) → Move to subscriptions sub-app -- `/api/subscriptions/callback` (lines 139-169) → Move to subscriptions sub-app (or tools_lib OAuth if it's OAuth-related) - -### Target state for `main.py` (~80 lines): - -```python -# Imports -# Create MainApp with all sub-apps -# CORS middleware -# WebSocket routes (thin wrappers) -# if __name__ == "__main__": uvicorn -``` - ---- - -## Verification - -After all tasks are complete: - -1. Verify no file in `backend/` exceeds 250 lines: - ```bash - find backend -name '*.py' -not -path '*/__pycache__/*' -not -path '*/test*' | xargs wc -l | sort -rn | head -20 - ``` - -2. Verify the app starts: - ```bash - cd backend && python -c "from backend.main import app; print('App created OK')" - ``` - -3. Verify the new subscriptions sub-app is registered: - ```bash - python -c "from backend.main import app; routes = [r.path for r in app.routes]; print([r for r in routes if 'subscription' in r])" - ``` - -4. Check that no frontend API calls are broken by searching for old paths: - ```bash - grep -r "api/agents/subscriptions" frontend/src/ - ``` - This should return no results (all updated to `/api/subscriptions/`). - -5. Verify `main.py` is under 250 lines: - ```bash - wc -l backend/main.py - ``` diff --git a/CLEANUP_AGENT_4.md b/CLEANUP_AGENT_4.md deleted file mode 100644 index 82109e62..00000000 --- a/CLEANUP_AGENT_4.md +++ /dev/null @@ -1,340 +0,0 @@ -# Agent 4: Frontend Cleanup (Phase 3) - -## Context - -You are cleaning up the OpenSwarm frontend codebase. This is agent 4 of 4. You work **independently** from the backend agents — no backend changes are required for your work. - -The frontend is a React + TypeScript app in `frontend/src/`. It uses Redux Toolkit for state management and WebSockets for real-time communication with the backend. - -**Rules:** -- Every file you create or modify must be <250 lines of code -- Keep code DRY -- Do NOT touch `backend/`, `9router/`, `debugger/`, or `electron/` -- Do NOT touch tests -- Preserve all existing functionality — this is purely structural refactoring - ---- - -## Overview - -There are 40 TypeScript/TSX files over 250 lines. Here are the 13 worst offenders that need splitting: - -| File | Lines | Priority | -|------|-------|----------| -| `pages/Tools/Tools.tsx` | 2480 | P0 | -| `pages/AgentChat/ToolCallBubble.tsx` | 2182 | P0 | -| `pages/Dashboard/Dashboard.tsx` | 1594 | P0 | -| `pages/Views/ViewEditor.tsx` | 1591 | P0 | -| `pages/Settings/Settings.tsx` | 1567 | P0 | -| `pages/AgentChat/ChatInput.tsx` | 1278 | P1 | -| `pages/Dashboard/BrowserCard.tsx` | 1259 | P1 | -| `pages/AgentChat/ApprovalBar.tsx` | 1160 | P1 | -| `pages/AgentChat/AgentChat.tsx` | 1137 | P1 | -| `pages/Dashboard/AgentCard.tsx` | 1063 | P1 | -| `shared/state/agentsSlice.ts` | 1030 | P1 | -| `components/Layout/AppShell.tsx` | 1005 | P2 | -| `components/DynamicIsland.tsx` | 993 | P2 | - -All paths are relative to `frontend/src/app/` unless noted. - ---- - -## General Strategy - -For each oversized component: - -1. **Read the file** to understand its structure — identify logical sections, sub-components rendered inline, state management, event handlers, and utility functions. - -2. **Extract sub-components** into sibling files in the same directory. Each extracted component should: - - Be in its own file - - Accept props for data and callbacks - - Be <250 lines - -3. **Extract custom hooks** for complex stateful logic (e.g., `useDashboardDragDrop`, `useAgentChat`, `useChatSubmit`). - -4. **Extract utility functions** and constants into separate files. - -5. **The parent component** becomes a thin orchestrator that imports and composes the pieces. - ---- - -## P0: Critical Splits (do these first) - -### Tools.tsx (2480 lines) - -Read the file first. It likely contains: -- Tool list/grid view -- Individual tool cards with config panels -- OAuth connection flows -- MCP tool discovery UI -- Permission editors per tool -- Builtin tool permission toggles - -**Split into:** - -``` -pages/Tools/ -├── Tools.tsx # Main page: layout, tool list, state (~200) -├── ToolCard.tsx # Individual tool card (~200) -├── ToolConfigPanel.tsx # Config/edit panel for a tool (~200) -├── OAuthConnectFlow.tsx # OAuth button + status display (~150) -├── ToolPermissions.tsx # Per-tool permission toggles (~200) -├── McpDiscoveryPanel.tsx # MCP tool discovery results (~150) -├── BuiltinToolsList.tsx # Builtin tools section with permissions (~200) -└── hooks/ - └── useToolsState.ts # Tool loading, CRUD operations, OAuth state (~150) -``` - -### ToolCallBubble.tsx (2182 lines) - -This renders tool call + result bubbles in the chat. It likely has different renderers for different tool types. - -**Split into:** - -``` -pages/AgentChat/ -├── ToolCallBubble.tsx # Router component: picks renderer by tool type (~100) -├── toolRenderers/ -│ ├── DefaultToolRenderer.tsx # Generic tool call display (~150) -│ ├── ReadToolRenderer.tsx # File content display with line numbers (~150) -│ ├── EditToolRenderer.tsx # Diff view for edits (~150) -│ ├── BashToolRenderer.tsx # Terminal-style output (~150) -│ ├── SearchToolRenderer.tsx # Grep/Glob results (~100) -│ └── McpToolRenderer.tsx # MCP tool results (~100) -├── ToolInputDisplay.tsx # Formatted tool input JSON (~100) -└── ToolResultDisplay.tsx # Formatted tool result content (~100) -``` - -### Dashboard.tsx (1594 lines) - -**Split into:** - -``` -pages/Dashboard/ -├── Dashboard.tsx # Main canvas + layout orchestration (~200) -├── DashboardCanvas.tsx # The infinite canvas / drag-drop surface (~200) -├── CardRenderer.tsx # Routes card type to correct component (~80) -├── hooks/ -│ ├── useDashboardDragDrop.ts # Drag, drop, resize logic (~200) -│ └── useDashboardState.ts # Dashboard loading, saving (~150) -``` - -Keep existing `AgentCard.tsx`, `BrowserCard.tsx`, `DashboardViewCard.tsx` as-is (they'll be split separately). - -### ViewEditor.tsx (1591 lines) - -**Split into:** - -``` -pages/Views/ -├── ViewEditor.tsx # Main editor layout + state (~200) -├── EditorToolbar.tsx # Top toolbar (save, run, vibe-code button) (~100) -├── CodeEditorPanel.tsx # Code editor (Monaco or CodeMirror wrapper) (~150) -├── PreviewPane.tsx # Iframe preview with hot reload (~150) -├── SchemaEditorPanel.tsx # JSON schema editor for input_schema (~200) -├── FileTreePanel.tsx # Multi-file tree sidebar (~150) -├── BackendCodePanel.tsx # Backend Python code editor (~100) -└── hooks/ - └── useViewEditor.ts # Editor state, save, auto-run logic (~200) -``` - -### Settings.tsx (1567 lines) - -**Split into:** - -``` -pages/Settings/ -├── Settings.tsx # Main settings page with tabs/sections (~150) -├── ProviderSettings.tsx # API key inputs for each provider (~200) -├── SubscriptionSection.tsx # 9Router subscription management (~200) -├── SystemPromptEditor.tsx # Default system prompt editor (~150) -├── GeneralSettings.tsx # Default folder, theme, etc. (~150) -├── CustomProviderEditor.tsx # Custom OpenAI-compat provider form (~200) -├── AnalyticsOptInSection.tsx # Analytics toggle + info (~80) -└── hooks/ - └── useSettings.ts # Settings load/save logic (~100) -``` - ---- - -## P1: Important Splits - -### ChatInput.tsx (1278 lines) - -**Split into:** - -``` -pages/AgentChat/ -├── ChatInput.tsx # Main input container (~200) -├── AttachmentBar.tsx # File/image attachment display (~100) -├── ModeSelector.tsx # Mode dropdown (Agent/Ask/Plan/etc) (~100) -├── ModelPicker.tsx # Model selection dropdown (~100) -├── ContextAttachments.tsx # Context path + skill attachment UI (~150) -└── hooks/ - └── useChatSubmit.ts # Submit logic, validation, WS send (~150) -``` - -### BrowserCard.tsx (1259 lines) - -**Split into:** - -``` -pages/Dashboard/ -├── BrowserCard.tsx # Main browser card (~200) -├── BrowserToolbar.tsx # URL bar, navigation buttons (~150) -├── BrowserTabBar.tsx # Tab management strip (~100) -├── BrowserViewport.tsx # The webview/iframe wrapper (~200) -└── BrowserContextMenu.tsx # Right-click context menu (~100) -``` - -### ApprovalBar.tsx (1160 lines) - -**Split into:** - -``` -pages/AgentChat/ -├── ApprovalBar.tsx # Main approval container + queue (~150) -├── ApprovalCard.tsx # Individual approval request card (~200) -├── ToolInputEditor.tsx # Editable tool input JSON viewer (~200) -└── ApprovalActions.tsx # Allow/Deny/Edit buttons + logic (~100) -``` - -### AgentChat.tsx (1137 lines) - -**Split into:** - -``` -pages/AgentChat/ -├── AgentChat.tsx # Main chat page layout (~200) -├── MessageList.tsx # Scrollable message list (~200) -├── ChatHeader.tsx # Session name, model, status bar (~100) -└── hooks/ - └── useAgentChat.ts # Chat state, message handling, branch nav (~200) -``` - -### AgentCard.tsx (1063 lines) - -**Split into:** - -``` -pages/Dashboard/ -├── AgentCard.tsx # Main agent card (~200) -├── AgentCardHeader.tsx # Name, status badge, model tag (~100) -├── AgentCardMessages.tsx # Compact message list in card (~200) -├── AgentCardActions.tsx # Stop, close, resume, duplicate buttons (~100) -└── AgentCardToolGroup.tsx # Collapsed tool call groups (~150) -``` - -### agentsSlice.ts (1030 lines) - -**Split into:** - -``` -shared/state/ -├── agentsSlice.ts # Core session state: CRUD, status (~200) -├── agentMessagesSlice.ts # Message handling: add, edit, branch (~200) -├── agentStreamSlice.ts # Streaming state: deltas, stream start/end (~150) -└── agentWebSocket.ts # WebSocket message dispatch + handlers (~200) -``` - -Or if Redux Toolkit makes splitting slices difficult, at minimum extract the WebSocket handler logic and message processing into separate files, keeping a single slice that imports helper functions. - ---- - -## P2: Lower Priority Splits - -### AppShell.tsx (1005 lines) - -**Split into:** - -``` -components/Layout/ -├── AppShell.tsx # Main shell: sidebar + content area (~150) -├── Sidebar.tsx # Navigation sidebar (~200) -├── NavigationRail.tsx # Icon rail for collapsed sidebar (~100) -├── PageRouter.tsx # Route → page component mapping (~100) -└── hooks/ - └── useNavigation.ts # Route state, sidebar collapse (~100) -``` - -### DynamicIsland.tsx (993 lines) - -**Split into:** - -``` -components/ -├── DynamicIsland.tsx # Container + animation (~150) -├── IslandQuickActions.tsx # Quick action buttons/chips (~150) -├── IslandAgentStatus.tsx # Active agent status summary (~150) -├── IslandNotifications.tsx # Notification toasts (~100) -└── hooks/ - └── useIslandState.ts # Island expand/collapse, content logic (~150) -``` - ---- - -## P3: Remaining Files Over 250 Lines - -After completing P0-P2, check what's still over 250 lines: - -```bash -find frontend/src -type f \( -name '*.ts' -o -name '*.tsx' \) | xargs wc -l | sort -rn | awk '$1 > 250 {print}' -``` - -For each remaining file, apply the same decomposition strategy: -1. Extract sub-components -2. Extract hooks for complex logic -3. Extract constants/utils - -Target: **zero files over 250 lines**. - ---- - -## Shared Frontend Utilities (if time permits) - -### Typed API Client - -If you notice scattered `fetch()` calls with duplicated base URL construction and error handling, consider extracting into `shared/api.ts`: - -```typescript -const api = { - get: (path: string) => Promise, - post: (path: string, body?: any) => Promise, - put: (path: string, body?: any) => Promise, - delete: (path: string) => Promise, -} -``` - -### Common UI Patterns - -If you see repeated patterns (confirmation dialogs, loading spinners, error states), extract them into `shared/components/`. - ---- - -## Verification - -After all splits are complete: - -1. Verify no file exceeds 250 lines: - ```bash - find frontend/src -type f \( -name '*.ts' -o -name '*.tsx' \) | xargs wc -l | sort -rn | awk '$1 > 250' - ``` - This should return only the `total` line. - -2. Verify the frontend builds without errors: - ```bash - cd frontend && npm run build - ``` - (Or `npx webpack --mode production` — check `package.json` for the build command) - -3. Verify no TypeScript errors: - ```bash - cd frontend && npx tsc --noEmit - ``` - -4. Spot-check that key pages render correctly by starting the dev server and navigating to: - - Dashboard page - - Agent chat page - - Tools page - - Settings page - - Views/Editor page diff --git a/README.md b/README.md index a8e151c2..3018550c 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@

License + Getting Started Platform GitHub Stars PRs Welcome