diff --git a/CLEANUP_AGENT_1.md b/CLEANUP_AGENT_1.md new file mode 100644 index 00000000..3156988a --- /dev/null +++ b/CLEANUP_AGENT_1.md @@ -0,0 +1,202 @@ +# Agent 1: Foundations + Hygiene (Phase 0 + Phase 4) + +## Context + +You are cleaning up the OpenSwarm codebase. This is the first of 4 agents. Your job is to build shared infrastructure that the other agents will depend on, plus minor hygiene fixes. + +**Rules:** +- Every file you create or modify must be <250 lines of code +- Keep code DRY +- Do NOT touch the `9router/` or `debugger/` directories +- Do NOT touch tests +- Run the app after your changes to make sure nothing is broken: `cd backend && python -m backend.main` + +--- + +## Task 0A: Create `backend/apps/common/json_store.py` + +**Problem:** Every sub-app copy-pastes identical `_load_all()`, `_save()`, `_load()`, `_delete()` functions for JSON-file CRUD. This exists in: + +- `backend/apps/tools_lib/tools_lib.py` (lines 255-277) +- `backend/apps/outputs/outputs.py` (lines 103-134) +- `backend/apps/dashboards/dashboards.py` (lines 27-55) +- `backend/apps/agents/agent_manager.py` (lines 33-61) +- `backend/apps/modes/modes.py` +- `backend/apps/templates/templates.py` +- `backend/apps/skills/skills.py` + +**What to do:** + +1. Create `backend/apps/common/__init__.py` (empty) +2. Create `backend/apps/common/json_store.py` with a generic `JsonStore` class: + +```python +class JsonStore(Generic[T]): + def __init__(self, model_cls: type[T], data_dir: str, id_field: str = "id"): + ... + def load_all(self) -> list[T]: ... + def save(self, item: T) -> None: ... + def load(self, item_id: str) -> T: ... # raises HTTPException(404) if not found + def delete(self, item_id: str) -> None: ... + def exists(self, item_id: str) -> bool: ... +``` + +3. Replace the inline `_load_all`, `_save`, `_load`, `_delete` in **all 7 sub-apps** with `JsonStore` instances. +4. For the agents session store, it's slightly different (uses `session_id` not `id`, and has `_load_all_session_data` returning tuples). Create a small subclass or adapter — keep it clean. + +**Verify:** All existing imports of `_load_all`, `_save`, `_load` from these modules still work (they're imported in `agent_manager.py`, `outputs.py`, `dashboards.py`, etc). If external code imports them, keep backward-compatible aliases. + +--- + +## Task 0B: Create `backend/apps/common/model_registry.py` + +**Problem:** Model name-to-ID mappings are duplicated with inconsistent values: + +- `backend/apps/agents/browser_agent.py` lines 24-28: `MODEL_MAP` with `"sonnet": "claude-sonnet-4-6"` +- `backend/apps/outputs/outputs.py` lines 23-27: `MODEL_MAP` with `"sonnet": "claude-sonnet-4-20250514"` +- `backend/apps/agents/providers/registry.py` lines 25-31: `BUILTIN_MODELS` dict +- `backend/apps/agents/providers/registry.py` lines 284-309: `COST_PER_1M_TOKENS` dict + +**What to do:** + +1. Create `backend/apps/common/model_registry.py` with: + +```python +@dataclass +class ModelDef: + value: str # short name ("sonnet") + label: str # display name ("Claude Sonnet 4.6") + model_id: str # API model ID ("claude-sonnet-4-6") + provider: str # "Anthropic", "OpenAI", etc. + api: str # "anthropic", "openai", "gemini", "openrouter" + context_window: int + input_cost_per_1m: float + output_cost_per_1m: float + +ALL_MODELS: list[ModelDef] = [ ... ] # Single source of truth + +def resolve_model_id(short_name: str) -> str: ... +def get_cost_rates(provider: str, model: str) -> tuple[float, float] | None: ... +def calculate_cost(provider: str, model: str, input_tokens: int, output_tokens: int) -> float: ... +def get_context_window(model: str) -> int: ... +def get_builtin_models_by_provider() -> dict[str, list[dict]]: ... +``` + +2. Use the canonical model IDs from `providers/registry.py` BUILTIN_MODELS (the `model_id` field there is the correct one: `claude-sonnet-4-6`, `claude-opus-4-6`, etc). + +3. Delete `MODEL_MAP` from `browser_agent.py` and `outputs.py`, replace with `resolve_model_id()` import. + +4. Delete `BUILTIN_MODELS` and `COST_PER_1M_TOKENS` from `providers/registry.py`, replace with imports from `model_registry.py`. Update `get_available_models()`, `calculate_cost()`, `get_context_window()` in `registry.py` to delegate to the new module. + +--- + +## Task 0C: Create `backend/apps/common/mcp_utils.py` + +**Problem:** `_parse_sse_json()` is duplicated verbatim in: +- `backend/apps/tools_lib/tools_lib.py` (lines 773-787) +- `backend/apps/agents/mcp_client.py` (lines 234-249) + +Also, `_sanitize_server_name()` is defined in `tools_lib.py` (line 481) but imported into `agent_manager.py`. + +**What to do:** + +1. Create `backend/apps/common/mcp_utils.py` with: + - `parse_sse_json(text: str) -> dict | None` + - `sanitize_server_name(name: str) -> str` + +2. Update `tools_lib.py` to import from `common.mcp_utils` instead of defining locally. Keep `_sanitize_server_name` as a backward-compatible alias: `_sanitize_server_name = sanitize_server_name`. + +3. Update `mcp_client.py` to import `parse_sse_json` from `common.mcp_utils` instead of having its own `_parse_sse_json` static method. + +4. Update `agent_manager.py` imports to use `from backend.apps.common.mcp_utils import sanitize_server_name`. + +--- + +## Task 0D: Create `backend/apps/common/llm_helpers.py` + +**Problem:** Multiple files independently construct Anthropic clients, call `messages.create`, strip markdown fences from responses, and parse JSON — with identical error handling. This happens in: + +- `agent_manager.py` `generate_title()` (lines 1511-1539) +- `agent_manager.py` `generate_group_meta()` (lines 1541-1617) +- `dashboards.py` `generate_name()` (lines 133-190) +- `outputs.py` `vibe_code()` (lines 351-419) +- `outputs.py` `auto_run_output()` (lines 430-480) + +**What to do:** + +1. Create `backend/apps/common/llm_helpers.py` with: + +```python +async def quick_llm_call( + system: str, + user_content: str, + model: str = "claude-sonnet-4-20250514", + max_tokens: int = 300, +) -> str: + """Make a simple LLM call and return the text response. Handles client construction.""" + +async def quick_llm_json( + system: str, + user_content: str, + model: str = "claude-sonnet-4-20250514", + max_tokens: int = 300, +) -> dict: + """Make an LLM call expecting JSON. Strips markdown fences, parses JSON.""" + +def strip_markdown_fences(text: str) -> str: + """Remove ```json ... ``` or similar fences from LLM output.""" +``` + +2. These should use `get_anthropic_client` from `backend/apps/settings/credentials.py` internally. + +3. **Do not** refactor the callers yet — that's Agent 2's job. Just create the helpers so they're available. + +--- + +## Task 4A: Clean Up `modes/models.py` + +**Problem:** `backend/apps/modes/models.py` (171 lines) has ~120 lines of `BUILTIN_MODES` data mixed in with Pydantic schema definitions. + +**What to do:** + +1. Create `backend/apps/modes/builtin.py` and move the `BUILTIN_MODES` list there. +2. `modes/models.py` should only contain `Mode`, `ModeCreate`, `ModeUpdate` Pydantic models. +3. Update `modes/modes.py` to import `BUILTIN_MODES` from `builtin.py` instead of `models.py`. +4. Check for any other files that import `BUILTIN_MODES` from `models.py` and update them. + +--- + +## Task 4B: DRY Up `outputs/models.py` Migration Validators + +**Problem:** `backend/apps/outputs/models.py` (197 lines) has `_migrate_flat_fields` copy-pasted across 4 classes: `Output`, `OutputCreate`, `OutputUpdate`, `WorkspaceSeedRequest`. + +**What to do:** + +1. Extract a shared function at module level: + +```python +def _migrate_legacy_files(data: dict, allow_schema_json: bool = False) -> dict: + """Convert legacy frontend_code/backend_code fields into the files dict.""" + ... +``` + +2. Have each `@model_validator` call this shared function instead of duplicating the logic. + +--- + +## Task 4C: Remove Dead Code + +1. `backend/apps/tools_lib/tools_lib.py` lines 640-646: empty comment section `# OAuth2 flow for Google Workspace (and other OAuth providers)` followed by another empty comment `# MCP tool discovery`. Remove the stale comments. + +--- + +## Verification + +After all tasks are complete: + +1. `cd /Users/haikdecie/Desktop/openswarm-ai/openswarm/backend` +2. `python -c "from backend.apps.common.json_store import JsonStore; print('OK')"` +3. `python -c "from backend.apps.common.model_registry import resolve_model_id; print(resolve_model_id('sonnet'))"` +4. `python -c "from backend.apps.common.mcp_utils import parse_sse_json, sanitize_server_name; print('OK')"` +5. `python -c "from backend.apps.common.llm_helpers import quick_llm_call; print('OK')"` +6. Verify no existing imports are broken by running: `python -c "from backend.main import app; print('OK')"` diff --git a/CLEANUP_AGENT_2.md b/CLEANUP_AGENT_2.md new file mode 100644 index 00000000..da917fbd --- /dev/null +++ b/CLEANUP_AGENT_2.md @@ -0,0 +1,289 @@ +# Agent 2: Split Backend God Objects (Phase 1) + +## Context + +You are cleaning up the OpenSwarm codebase. This is agent 2 of 4. Agent 1 has already completed Phase 0 (foundations), creating these shared utilities that you should USE: + +- `backend/apps/common/json_store.py` — Generic `JsonStore[T]` for JSON file CRUD +- `backend/apps/common/model_registry.py` — Single source of truth for model definitions, `resolve_model_id()`, `calculate_cost()`, etc. +- `backend/apps/common/mcp_utils.py` — `parse_sse_json()`, `sanitize_server_name()` +- `backend/apps/common/llm_helpers.py` — `quick_llm_call()`, `quick_llm_json()`, `strip_markdown_fences()` + +**Rules:** +- Every file you create or modify must be <250 lines of code +- Keep code DRY — use the Phase 0 utilities +- Do NOT touch `9router/`, `debugger/`, or `frontend/` +- Do NOT touch tests +- Run the app after each major split to verify nothing is broken: `cd backend && python -m backend.main` + +--- + +## Task 1A: Split `agent_manager.py` (2093 lines → ~6 files) + +This is the most important task. `backend/apps/agents/agent_manager.py` is a 2093-line god object. + +Read it carefully first. The `AgentManager` class has these distinct responsibilities that should be separated: + +### New file: `backend/apps/agents/prompt_builder.py` (~200 lines) + +Extract these methods from `AgentManager`: +- `_resolve_mode()` +- `_compose_system_prompt()` +- `_build_connected_tools_context()` +- `_build_outputs_context()` +- `_build_browser_context()` +- `_get_pre_selected_browser_ids()` +- `_resolve_context_paths()` +- `_build_dir_tree()` +- `_resolve_forced_tools()` +- `_resolve_attached_skills()` +- `_build_prompt_content()` + +Make these standalone functions (not methods) that take the data they need as parameters. For example: +```python +def compose_system_prompt( + default_prompt: str | None, mode_prompt: str | None, + session_prompt: str | None, connected_tools_ctx: str | None = None, + outputs_ctx: str | None = None, browser_ctx: str | None = None, +) -> str | None: +``` + +### New file: `backend/apps/agents/mcp_builder.py` (~200 lines) + +Extract: +- `_build_mcp_servers()` — builds the mcp_servers dict for ClaudeAgentOptions +- `_get_effective_policy()` — the permission policy resolver (currently a nested function inside `_run_agent_loop`) +- `_get_denied_tool_names()`, `_get_all_known_tool_names()`, `_is_fully_denied()` — the module-level helper functions +- `get_all_tool_names()` — the module-level function +- `FULL_TOOLS` — the constant list +- Logic for building `effective_allowed` and `effective_disallowed` tool lists (currently ~50 lines inside `_run_agent_loop`) +- Logic for adding browser-agent and invoke-agent MCP servers + +### New file: `backend/apps/agents/session_store.py` (~200 lines) + +Extract session persistence and history. Use `JsonStore` from `backend/apps/common/json_store.py` where possible: +- `_save_session()`, `_load_session_data()`, `_delete_session_file()`, `_load_all_session_data()` +- `get_history()` method +- `_build_search_text()` static method +- `reconcile_on_startup()` +- `persist_all_sessions()` +- `restore_all_sessions()` +- `get_browser_agent_children()` + +### Refactor: `backend/apps/agents/agent_loop.py` (replace existing, ~250 lines) + +The existing `agent_loop.py` (331 lines) is an older/unused file. Replace it with the extracted `_run_agent_loop` method from `agent_manager.py`. + +Extract from `AgentManager`: +- `_run_agent_loop()` — the main SDK query loop (lines 501-1128). This is the biggest single method. +- The hook functions (`pre_tool_hook`, `post_tool_hook`, `can_use_tool`) that are currently nested inside `_run_agent_loop` +- `_run_mock_agent()` — the development mock (lines 1172-1259) +- `_stream_text()` and `_stream_tool_input()` — streaming helpers (lines 1130-1170) +- `_fire_session_completed()` — analytics for completed sessions + +Use `quick_llm_call` and `quick_llm_json` from `common/llm_helpers.py` for `generate_title()` and `generate_group_meta()`. + +### Slim down: `backend/apps/agents/agent_manager.py` (~250 lines) + +What remains in `AgentManager`: +- `__init__()` — holds `self.sessions` dict and `self.tasks` dict +- `launch_agent()` +- `send_message()` +- `stop_agent()` +- `handle_approval()` +- `edit_message()` +- `switch_branch()` +- `generate_title()` — refactor to use `quick_llm_call()` +- `generate_group_meta()` — refactor to use `quick_llm_json()` +- `update_session()` +- `close_session()` +- `delete_session()` +- `resume_session()` +- `duplicate_session()` +- `invoke_agent()` +- `get_all_sessions()`, `get_session()` + +Each of these methods becomes a thin coordinator that calls into the extracted modules. + +The `agent_manager = AgentManager()` singleton stays at the bottom of this file. + +### Important Notes for this split: + +- `_run_agent_loop` has deeply nested closures (`can_use_tool`, `pre_tool_hook`, `post_tool_hook`, `prompt_stream`). When extracting to `agent_loop.py`, you'll need to pass the session, ws_manager, and other dependencies as parameters to these functions. +- The `duplicate_session` and `invoke_agent` methods have duplicated message-copying logic (~30 lines each). Extract a shared `_copy_session_messages()` helper into `session_store.py`. +- Keep the `agent_manager` singleton import path unchanged: `from backend.apps.agents.agent_manager import agent_manager` must still work. + +--- + +## Task 1B: Split `tools_lib.py` (1153 lines → 5-6 files) + +Convert `backend/apps/tools_lib/` from a single file into a package. + +### Step 1: Create the package structure + +``` +backend/apps/tools_lib/ +├── __init__.py # SubApp instance + backward-compat imports +├── routes.py # Tool CRUD endpoints (~120 lines) +├── oauth.py # OAuth start, callback, disconnect, refresh (~200 lines) +├── oauth_providers.py # OAuthProvider dataclass + OAUTH_PROVIDERS registry (~180 lines) +├── mcp_config.py # derive_mcp_config, _resolve_command, _augmented_path, _extra_bin_dirs (~180 lines) +├── mcp_discovery.py # discover_tools endpoint + stdio/HTTP/SSE discovery (~200 lines) +├── classification.py # _SERVICE_RULES, _categorize_tool, _extract_service (~100 lines) +└── models.py # Already exists, keep as-is +``` + +### Step 2: What goes where + +**`oauth_providers.py`** — Pure data, no dependencies: +- `OAuthProvider` dataclass (lines 86-107) +- `OAUTH_PROVIDERS` dict (lines 109-239) +- `_resolve_oauth_provider()` helper (lines 242-248) + +**`oauth.py`** — OAuth flow logic: +- `_pending_oauth` and `_pending_pkce` state dicts (lines 251-252) +- `oauth_callback()` endpoint (lines 319-435) +- `oauth_start()` endpoint (lines 1062-1098) +- `oauth_disconnect()` endpoint (lines 1036-1059) +- `refresh_oauth_token()` function (lines 1102-1153) +- `refresh_google_token` alias + +**`mcp_config.py`** — MCP server config derivation: +- `_extra_bin_dirs()` (lines 486-513) +- `_resolve_command()` (lines 516-539) +- `_augmented_path()` (lines 542-552) +- `derive_mcp_config()` (lines 555-637) + +**`mcp_discovery.py`** — MCP tool discovery: +- `_discover_mcp_tools_http()` (lines 790-830) +- `_discover_mcp_tools_sse()` (lines 833-856) +- `_discover_mcp_tools_stdio()` (lines 859-941) +- `discover_tools()` endpoint (lines 944-1033) + +**`classification.py`** — Tool categorization (pure data + logic): +- `_READ_PREFIXES`, `_WRITE_PREFIXES` (lines 649-651) +- `_SERVICE_RULES` (lines 653-748) +- `_categorize_tool()` (lines 751-760) +- `_extract_service()` (lines 763-770) + +**`routes.py`** — CRUD endpoints: +- `list_builtin_tools()`, `list_tools()`, `get_tool()`, `create_tool()`, `update_tool()`, `delete_tool()` (lines 279-474) +- `load_builtin_permissions()`, `save_builtin_permissions()`, permission endpoints (lines 284-311) +- Use `JsonStore` from `backend/apps/common/json_store.py` for the `_load_all`, `_save`, `_load` functions + +**`__init__.py`** — Glue: +- `tools_lib` SubApp instance +- `tools_lib_lifespan` +- Backward-compatible imports so that `from backend.apps.tools_lib.tools_lib import _load_all, derive_mcp_config, ...` still works. Add a `tools_lib.py` shim file OR add these to `__init__.py`. +- Wire up all routes from the sub-modules + +### Critical: Maintain backward compatibility + +These imports exist in `agent_manager.py` and must keep working: +```python +from backend.apps.tools_lib.tools_lib import ( + _load_all as load_all_tools, + _sanitize_server_name, + derive_mcp_config, + load_builtin_permissions, + refresh_google_token, +) +``` + +Keep a `tools_lib.py` file that re-exports from the new package modules, OR update all import sites. + +--- + +## Task 1C: Split `outputs.py` (593 lines → 3 files) + +### New file: `backend/apps/outputs/helpers.py` (~80 lines) + +Extract: +- `_build_data_injection()` (lines 53-70) +- `_inject_data_into_html()` (lines 73-79) +- `_decode_data_param()` (lines 82-90) +- `_validate_against_schema()` (lines 41-48) +- `_walk_directory()` (lines 136-150) + +### New file: `backend/apps/outputs/ai_generation.py` (~200 lines) + +Extract: +- `VIBE_CODE_SYSTEM_PROMPT` (lines 334-348) +- `vibe_code()` endpoint (lines 351-419) — refactor to use `quick_llm_json` from `common/llm_helpers.py` +- `AUTO_RUN_SYSTEM_PROMPT` (lines 422-427) +- `auto_run_output()` endpoint (lines 430-480) — refactor to use `quick_llm_call` +- `AUTO_RUN_AGENT_SYSTEM_PROMPT` (lines 525-541) +- `auto_run_agent()` endpoint (lines 544-581) +- `cleanup_auto_run_agent()` endpoint (lines 584-593) +- `_get_anthropic_client()` helper (lines 34-38) — or replace with direct `quick_llm_*` usage +- `_resolve_model()` — replace with `resolve_model_id` from `common/model_registry.py` + +### Slim down: `backend/apps/outputs/outputs.py` (~200 lines) + +What remains: +- SubApp instance + lifespan +- CRUD endpoints (list, get, create, update, delete) +- Workspace endpoints (read, seed, write file, delete file) +- File serving endpoints (serve_workspace_file, serve_output_file) +- `_load_all`, `_save`, `_load`, `load_output` — replace with `JsonStore` + +Wire the ai_generation routes into the outputs router. + +--- + +## Task 1D: Split `browser_agent.py` (632 lines → 3 files) + +### New file: `backend/apps/agents/browser/schemas.py` (~100 lines) + +Extract: +- `BROWSER_TOOLS_SCHEMA` (lines 30-158) +- `ACTION_MAP` (lines 160-170) +- `SYSTEM_PROMPT` (lines 172-193) +- `MAX_TURNS` constant (line 195) + +Replace the local `MODEL_MAP` (lines 24-28) with `resolve_model_id` from `common/model_registry.py`. + +### New file: `backend/apps/agents/browser/executor.py` (~120 lines) + +Extract: +- `execute_browser_tool()` (lines 198-211) +- `_format_tool_result()` (lines 214-234) +- `_request_browser_approval()` (lines 237-274) + +### New file: `backend/apps/agents/browser/runner.py` (~250 lines) + +Extract: +- `run_browser_agent()` (lines 277-549) +- `_create_browser_card()` (lines 552-580) +- `run_browser_agents()` (lines 583-632) + +Create `backend/apps/agents/browser/__init__.py` that re-exports the public API: +```python +from backend.apps.agents.browser.runner import run_browser_agent, run_browser_agents +``` + +Update `backend/main.py` line 177 which imports `from backend.apps.agents.browser_agent import run_browser_agents`. + +--- + +## Verification + +After all tasks are complete: + +1. Verify no file in `backend/` exceeds 250 lines: + ```bash + find backend -name '*.py' -not -path '*/__pycache__/*' -not -path '*/test*' | xargs wc -l | sort -rn | head -20 + ``` + +2. Verify the app starts: + ```bash + cd backend && python -c "from backend.main import app; print('App created OK')" + ``` + +3. Verify key imports still work: + ```bash + python -c "from backend.apps.agents.agent_manager import agent_manager; print('OK')" + python -c "from backend.apps.tools_lib.tools_lib import _load_all, derive_mcp_config, load_builtin_permissions; print('OK')" + python -c "from backend.apps.agents.browser_agent import run_browser_agents; print('OK')" + python -c "from backend.apps.outputs.outputs import _load_all; print('OK')" + ``` diff --git a/CLEANUP_AGENT_3.md b/CLEANUP_AGENT_3.md new file mode 100644 index 00000000..f5f5b572 --- /dev/null +++ b/CLEANUP_AGENT_3.md @@ -0,0 +1,253 @@ +# Agent 3: DRY Up Cross-Cutting Patterns (Phase 2) + +## Context + +You are cleaning up the OpenSwarm codebase. This is agent 3 of 4. Agents 1 and 2 have already completed: + +- **Phase 0:** Shared utilities in `backend/apps/common/` (json_store, model_registry, mcp_utils, llm_helpers) +- **Phase 1:** Split god objects — `agent_manager.py`, `tools_lib.py`, `outputs.py`, `browser_agent.py` are now modular packages/files + +**Rules:** +- Every file you create or modify must be <250 lines of code +- Keep code DRY +- Do NOT touch `9router/`, `debugger/`, or `frontend/` +- Do NOT touch tests +- Verify the app still starts after each major change + +--- + +## Task 2A: Extract Shared Approval Flow + +**Problem:** Two places implement independent HITL (human-in-the-loop) approval request patterns: + +1. **`agent_manager.py`** (now possibly in `agent_loop.py` after Phase 1 split) — `_request_user_approval()` function: + - Creates an `ApprovalRequest` + - Appends to `session.pending_approvals` + - Sets `session.status = "waiting_approval"` + - Sends WS status event + - Fires analytics event + - Waits for decision via `ws_manager.send_approval_request()` + - Fires another analytics event with latency + - Removes from pending approvals + - Restores `session.status = "running"` + - Sends WS status event + +2. **`browser_agent.py`** (now possibly in `browser/executor.py`) — `_request_browser_approval()`: + - Same pattern but without analytics events and with a timeout + +**What to do:** + +1. Create `backend/apps/agents/approval.py` (~80 lines): + +```python +async def request_approval( + session: AgentSession, + tool_name: str, + tool_input: dict, + timeout: float | None = None, + track_analytics: bool = True, +) -> dict: + """Unified HITL approval flow. + + Creates an ApprovalRequest, sends it via WebSocket, waits for the user's + decision, cleans up, and returns the decision dict. + + Returns: {"behavior": "allow"|"deny", "message": ..., "updated_input": ...} + """ +``` + +2. Replace both implementations with calls to this shared function. + +3. Make sure the analytics tracking is optional (browser agents don't currently track approval analytics). + +--- + +## Task 2B: Extract WebSocket Event Helpers + +**Problem:** Throughout the codebase (especially `agent_manager.py` / `agent_loop.py` / `browser_agent.py`), there are 30+ calls that look like: + +```python +await ws_manager.send_to_session(session_id, "agent:status", { + "session_id": session_id, + "status": "running", + "session": session.model_dump(mode="json"), +}) +``` + +The payload construction is repeated identically every time for each event type. + +**What to do:** + +Add typed convenience methods to `backend/apps/agents/ws_manager.py`. The file is currently 126 lines so there's room: + +```python +async def emit_status(self, session_id: str, status: str, session: AgentSession): + await self.send_to_session(session_id, "agent:status", { + "session_id": session_id, + "status": status, + "session": session.model_dump(mode="json"), + }) + +async def emit_message(self, session_id: str, message: Message): + await self.send_to_session(session_id, "agent:message", { + "session_id": session_id, + "message": message.model_dump(mode="json"), + }) + +async def emit_cost_update(self, session_id: str, cost_usd: float): + await self.send_to_session(session_id, "agent:cost_update", { + "session_id": session_id, + "cost_usd": cost_usd, + }) + +async def emit_stream_start(self, session_id: str, message_id: str, role: str, tool_name: str = ""): + payload = {"session_id": session_id, "message_id": message_id, "role": role} + if tool_name: + payload["tool_name"] = tool_name + await self.send_to_session(session_id, "agent:stream_start", payload) + +async def emit_stream_delta(self, session_id: str, message_id: str, delta: str): + await self.send_to_session(session_id, "agent:stream_delta", { + "session_id": session_id, + "message_id": message_id, + "delta": delta, + }) + +async def emit_stream_end(self, session_id: str, message_id: str): + await self.send_to_session(session_id, "agent:stream_end", { + "session_id": session_id, + "message_id": message_id, + }) +``` + +Then do a find-and-replace across all files that construct these payloads manually. Replace with the typed helpers. This is a mechanical change — just make sure every event type is covered. + +**Important:** If `ws_manager.py` exceeds 250 lines after adding these helpers, split it into `ws_manager.py` (connection management) and `ws_events.py` (typed event emitters). + +--- + +## Task 2C: Move Subscription Routes to New Sub-App + +**Problem:** `backend/apps/agents/agents.py` (305 lines) contains ~120 lines of 9Router/subscription endpoints (lines 186-305) that have nothing to do with agent sessions: +- `subscriptions_status()` +- `subscriptions_connect()` +- `subscriptions_poll()` +- `subscriptions_exchange()` +- `subscriptions_models()` +- `subscriptions_disconnect()` + +**What to do:** + +1. Create `backend/apps/subscriptions/__init__.py` (empty) +2. Create `backend/apps/subscriptions/subscriptions.py` (~130 lines): + - Move all 6 subscription endpoints here + - Create a new SubApp: `subscriptions = SubApp("subscriptions", subscriptions_lifespan)` + - The lifespan can be a simple no-op (or move the 9Router auto-start from analytics lifespan here if it makes more sense) + +3. Update `backend/main.py`: + - Import the new `subscriptions` sub-app + - Add it to the `MainApp` list + +4. Remove the subscription endpoints from `agents.py`. This should bring `agents.py` down to ~185 lines. + +5. **Frontend impact:** The frontend calls these endpoints at `/api/agents/subscriptions/*`. The new path will be `/api/subscriptions/*`. Search the frontend for these API paths and update them: + ```bash + grep -r "api/agents/subscriptions" frontend/src/ + ``` + Update all matches to use `/api/subscriptions/` instead. + +--- + +## Task 2D: Clean Up `main.py` + +**Problem:** `backend/main.py` (256 lines) has inline WebSocket handlers, OAuth callback endpoints, browser-agent HTTP endpoint, and invoke-agent HTTP endpoint that should live in their respective sub-apps. + +**What to do:** + +### Move WebSocket handlers + +The two WebSocket handlers (`websocket_session` and `websocket_dashboard`, lines 41-104) can't easily move to a SubApp router because FastAPI WebSocket routes need to be on the main app. However, the message dispatch logic inside them can be extracted. + +Create `backend/apps/agents/ws_routes.py` (~80 lines): + +```python +async def handle_session_message(session_id: str, event: str, payload: dict): + """Dispatch a WebSocket message for a session.""" + if event == "agent:send_message": + from backend.apps.agents.agent_manager import agent_manager + await agent_manager.send_message(...) + elif event == "agent:approval_response": + ... + elif event == "agent:edit_message": + ... + elif event == "agent:stop": + ... + +async def handle_dashboard_message(event: str, payload: dict): + """Dispatch a WebSocket message for the dashboard.""" + ... +``` + +Then `main.py`'s WebSocket handlers become thin wrappers: +```python +@app.websocket("/ws/agents/{session_id}") +async def websocket_session(websocket: WebSocket, session_id: str): + await ws_manager.connect_session(session_id, websocket) + try: + while True: + data = await websocket.receive_text() + msg = json.loads(data) + await handle_session_message(session_id, msg.get("event"), msg.get("data", {})) + except WebSocketDisconnect: + ws_manager.disconnect_session(session_id, websocket) +``` + +### Move HTTP endpoints + +- `/api/browser/command` (lines 107-122) → Move to `agents/browser/` as a route on the agents router, or keep in main.py if it needs to be at the root level +- `/api/browser-agent/run` (lines 172-196) → Move to agents router +- `/api/invoke-agent/run` (lines 199-227) → Move to agents router +- `/api/subscriptions/pending/{state}` (lines 125-136) → Move to subscriptions sub-app +- `/api/subscriptions/callback` (lines 139-169) → Move to subscriptions sub-app (or tools_lib OAuth if it's OAuth-related) + +### Target state for `main.py` (~80 lines): + +```python +# Imports +# Create MainApp with all sub-apps +# CORS middleware +# WebSocket routes (thin wrappers) +# if __name__ == "__main__": uvicorn +``` + +--- + +## Verification + +After all tasks are complete: + +1. Verify no file in `backend/` exceeds 250 lines: + ```bash + find backend -name '*.py' -not -path '*/__pycache__/*' -not -path '*/test*' | xargs wc -l | sort -rn | head -20 + ``` + +2. Verify the app starts: + ```bash + cd backend && python -c "from backend.main import app; print('App created OK')" + ``` + +3. Verify the new subscriptions sub-app is registered: + ```bash + python -c "from backend.main import app; routes = [r.path for r in app.routes]; print([r for r in routes if 'subscription' in r])" + ``` + +4. Check that no frontend API calls are broken by searching for old paths: + ```bash + grep -r "api/agents/subscriptions" frontend/src/ + ``` + This should return no results (all updated to `/api/subscriptions/`). + +5. Verify `main.py` is under 250 lines: + ```bash + wc -l backend/main.py + ``` diff --git a/CLEANUP_AGENT_4.md b/CLEANUP_AGENT_4.md new file mode 100644 index 00000000..82109e62 --- /dev/null +++ b/CLEANUP_AGENT_4.md @@ -0,0 +1,340 @@ +# Agent 4: Frontend Cleanup (Phase 3) + +## Context + +You are cleaning up the OpenSwarm frontend codebase. This is agent 4 of 4. You work **independently** from the backend agents — no backend changes are required for your work. + +The frontend is a React + TypeScript app in `frontend/src/`. It uses Redux Toolkit for state management and WebSockets for real-time communication with the backend. + +**Rules:** +- Every file you create or modify must be <250 lines of code +- Keep code DRY +- Do NOT touch `backend/`, `9router/`, `debugger/`, or `electron/` +- Do NOT touch tests +- Preserve all existing functionality — this is purely structural refactoring + +--- + +## Overview + +There are 40 TypeScript/TSX files over 250 lines. Here are the 13 worst offenders that need splitting: + +| File | Lines | Priority | +|------|-------|----------| +| `pages/Tools/Tools.tsx` | 2480 | P0 | +| `pages/AgentChat/ToolCallBubble.tsx` | 2182 | P0 | +| `pages/Dashboard/Dashboard.tsx` | 1594 | P0 | +| `pages/Views/ViewEditor.tsx` | 1591 | P0 | +| `pages/Settings/Settings.tsx` | 1567 | P0 | +| `pages/AgentChat/ChatInput.tsx` | 1278 | P1 | +| `pages/Dashboard/BrowserCard.tsx` | 1259 | P1 | +| `pages/AgentChat/ApprovalBar.tsx` | 1160 | P1 | +| `pages/AgentChat/AgentChat.tsx` | 1137 | P1 | +| `pages/Dashboard/AgentCard.tsx` | 1063 | P1 | +| `shared/state/agentsSlice.ts` | 1030 | P1 | +| `components/Layout/AppShell.tsx` | 1005 | P2 | +| `components/DynamicIsland.tsx` | 993 | P2 | + +All paths are relative to `frontend/src/app/` unless noted. + +--- + +## General Strategy + +For each oversized component: + +1. **Read the file** to understand its structure — identify logical sections, sub-components rendered inline, state management, event handlers, and utility functions. + +2. **Extract sub-components** into sibling files in the same directory. Each extracted component should: + - Be in its own file + - Accept props for data and callbacks + - Be <250 lines + +3. **Extract custom hooks** for complex stateful logic (e.g., `useDashboardDragDrop`, `useAgentChat`, `useChatSubmit`). + +4. **Extract utility functions** and constants into separate files. + +5. **The parent component** becomes a thin orchestrator that imports and composes the pieces. + +--- + +## P0: Critical Splits (do these first) + +### Tools.tsx (2480 lines) + +Read the file first. It likely contains: +- Tool list/grid view +- Individual tool cards with config panels +- OAuth connection flows +- MCP tool discovery UI +- Permission editors per tool +- Builtin tool permission toggles + +**Split into:** + +``` +pages/Tools/ +├── Tools.tsx # Main page: layout, tool list, state (~200) +├── ToolCard.tsx # Individual tool card (~200) +├── ToolConfigPanel.tsx # Config/edit panel for a tool (~200) +├── OAuthConnectFlow.tsx # OAuth button + status display (~150) +├── ToolPermissions.tsx # Per-tool permission toggles (~200) +├── McpDiscoveryPanel.tsx # MCP tool discovery results (~150) +├── BuiltinToolsList.tsx # Builtin tools section with permissions (~200) +└── hooks/ + └── useToolsState.ts # Tool loading, CRUD operations, OAuth state (~150) +``` + +### ToolCallBubble.tsx (2182 lines) + +This renders tool call + result bubbles in the chat. It likely has different renderers for different tool types. + +**Split into:** + +``` +pages/AgentChat/ +├── ToolCallBubble.tsx # Router component: picks renderer by tool type (~100) +├── toolRenderers/ +│ ├── DefaultToolRenderer.tsx # Generic tool call display (~150) +│ ├── ReadToolRenderer.tsx # File content display with line numbers (~150) +│ ├── EditToolRenderer.tsx # Diff view for edits (~150) +│ ├── BashToolRenderer.tsx # Terminal-style output (~150) +│ ├── SearchToolRenderer.tsx # Grep/Glob results (~100) +│ └── McpToolRenderer.tsx # MCP tool results (~100) +├── ToolInputDisplay.tsx # Formatted tool input JSON (~100) +└── ToolResultDisplay.tsx # Formatted tool result content (~100) +``` + +### Dashboard.tsx (1594 lines) + +**Split into:** + +``` +pages/Dashboard/ +├── Dashboard.tsx # Main canvas + layout orchestration (~200) +├── DashboardCanvas.tsx # The infinite canvas / drag-drop surface (~200) +├── CardRenderer.tsx # Routes card type to correct component (~80) +├── hooks/ +│ ├── useDashboardDragDrop.ts # Drag, drop, resize logic (~200) +│ └── useDashboardState.ts # Dashboard loading, saving (~150) +``` + +Keep existing `AgentCard.tsx`, `BrowserCard.tsx`, `DashboardViewCard.tsx` as-is (they'll be split separately). + +### ViewEditor.tsx (1591 lines) + +**Split into:** + +``` +pages/Views/ +├── ViewEditor.tsx # Main editor layout + state (~200) +├── EditorToolbar.tsx # Top toolbar (save, run, vibe-code button) (~100) +├── CodeEditorPanel.tsx # Code editor (Monaco or CodeMirror wrapper) (~150) +├── PreviewPane.tsx # Iframe preview with hot reload (~150) +├── SchemaEditorPanel.tsx # JSON schema editor for input_schema (~200) +├── FileTreePanel.tsx # Multi-file tree sidebar (~150) +├── BackendCodePanel.tsx # Backend Python code editor (~100) +└── hooks/ + └── useViewEditor.ts # Editor state, save, auto-run logic (~200) +``` + +### Settings.tsx (1567 lines) + +**Split into:** + +``` +pages/Settings/ +├── Settings.tsx # Main settings page with tabs/sections (~150) +├── ProviderSettings.tsx # API key inputs for each provider (~200) +├── SubscriptionSection.tsx # 9Router subscription management (~200) +├── SystemPromptEditor.tsx # Default system prompt editor (~150) +├── GeneralSettings.tsx # Default folder, theme, etc. (~150) +├── CustomProviderEditor.tsx # Custom OpenAI-compat provider form (~200) +├── AnalyticsOptInSection.tsx # Analytics toggle + info (~80) +└── hooks/ + └── useSettings.ts # Settings load/save logic (~100) +``` + +--- + +## P1: Important Splits + +### ChatInput.tsx (1278 lines) + +**Split into:** + +``` +pages/AgentChat/ +├── ChatInput.tsx # Main input container (~200) +├── AttachmentBar.tsx # File/image attachment display (~100) +├── ModeSelector.tsx # Mode dropdown (Agent/Ask/Plan/etc) (~100) +├── ModelPicker.tsx # Model selection dropdown (~100) +├── ContextAttachments.tsx # Context path + skill attachment UI (~150) +└── hooks/ + └── useChatSubmit.ts # Submit logic, validation, WS send (~150) +``` + +### BrowserCard.tsx (1259 lines) + +**Split into:** + +``` +pages/Dashboard/ +├── BrowserCard.tsx # Main browser card (~200) +├── BrowserToolbar.tsx # URL bar, navigation buttons (~150) +├── BrowserTabBar.tsx # Tab management strip (~100) +├── BrowserViewport.tsx # The webview/iframe wrapper (~200) +└── BrowserContextMenu.tsx # Right-click context menu (~100) +``` + +### ApprovalBar.tsx (1160 lines) + +**Split into:** + +``` +pages/AgentChat/ +├── ApprovalBar.tsx # Main approval container + queue (~150) +├── ApprovalCard.tsx # Individual approval request card (~200) +├── ToolInputEditor.tsx # Editable tool input JSON viewer (~200) +└── ApprovalActions.tsx # Allow/Deny/Edit buttons + logic (~100) +``` + +### AgentChat.tsx (1137 lines) + +**Split into:** + +``` +pages/AgentChat/ +├── AgentChat.tsx # Main chat page layout (~200) +├── MessageList.tsx # Scrollable message list (~200) +├── ChatHeader.tsx # Session name, model, status bar (~100) +└── hooks/ + └── useAgentChat.ts # Chat state, message handling, branch nav (~200) +``` + +### AgentCard.tsx (1063 lines) + +**Split into:** + +``` +pages/Dashboard/ +├── AgentCard.tsx # Main agent card (~200) +├── AgentCardHeader.tsx # Name, status badge, model tag (~100) +├── AgentCardMessages.tsx # Compact message list in card (~200) +├── AgentCardActions.tsx # Stop, close, resume, duplicate buttons (~100) +└── AgentCardToolGroup.tsx # Collapsed tool call groups (~150) +``` + +### agentsSlice.ts (1030 lines) + +**Split into:** + +``` +shared/state/ +├── agentsSlice.ts # Core session state: CRUD, status (~200) +├── agentMessagesSlice.ts # Message handling: add, edit, branch (~200) +├── agentStreamSlice.ts # Streaming state: deltas, stream start/end (~150) +└── agentWebSocket.ts # WebSocket message dispatch + handlers (~200) +``` + +Or if Redux Toolkit makes splitting slices difficult, at minimum extract the WebSocket handler logic and message processing into separate files, keeping a single slice that imports helper functions. + +--- + +## P2: Lower Priority Splits + +### AppShell.tsx (1005 lines) + +**Split into:** + +``` +components/Layout/ +├── AppShell.tsx # Main shell: sidebar + content area (~150) +├── Sidebar.tsx # Navigation sidebar (~200) +├── NavigationRail.tsx # Icon rail for collapsed sidebar (~100) +├── PageRouter.tsx # Route → page component mapping (~100) +└── hooks/ + └── useNavigation.ts # Route state, sidebar collapse (~100) +``` + +### DynamicIsland.tsx (993 lines) + +**Split into:** + +``` +components/ +├── DynamicIsland.tsx # Container + animation (~150) +├── IslandQuickActions.tsx # Quick action buttons/chips (~150) +├── IslandAgentStatus.tsx # Active agent status summary (~150) +├── IslandNotifications.tsx # Notification toasts (~100) +└── hooks/ + └── useIslandState.ts # Island expand/collapse, content logic (~150) +``` + +--- + +## P3: Remaining Files Over 250 Lines + +After completing P0-P2, check what's still over 250 lines: + +```bash +find frontend/src -type f \( -name '*.ts' -o -name '*.tsx' \) | xargs wc -l | sort -rn | awk '$1 > 250 {print}' +``` + +For each remaining file, apply the same decomposition strategy: +1. Extract sub-components +2. Extract hooks for complex logic +3. Extract constants/utils + +Target: **zero files over 250 lines**. + +--- + +## Shared Frontend Utilities (if time permits) + +### Typed API Client + +If you notice scattered `fetch()` calls with duplicated base URL construction and error handling, consider extracting into `shared/api.ts`: + +```typescript +const api = { + get: (path: string) => Promise, + post: (path: string, body?: any) => Promise, + put: (path: string, body?: any) => Promise, + delete: (path: string) => Promise, +} +``` + +### Common UI Patterns + +If you see repeated patterns (confirmation dialogs, loading spinners, error states), extract them into `shared/components/`. + +--- + +## Verification + +After all splits are complete: + +1. Verify no file exceeds 250 lines: + ```bash + find frontend/src -type f \( -name '*.ts' -o -name '*.tsx' \) | xargs wc -l | sort -rn | awk '$1 > 250' + ``` + This should return only the `total` line. + +2. Verify the frontend builds without errors: + ```bash + cd frontend && npm run build + ``` + (Or `npx webpack --mode production` — check `package.json` for the build command) + +3. Verify no TypeScript errors: + ```bash + cd frontend && npx tsc --noEmit + ``` + +4. Spot-check that key pages render correctly by starting the dev server and navigating to: + - Dashboard page + - Agent chat page + - Tools page + - Settings page + - Views/Editor page diff --git a/backend/apps/agents/agent_manager.py b/backend/apps/agents/agent_manager.py index dceffc58..dc045645 100644 --- a/backend/apps/agents/agent_manager.py +++ b/backend/apps/agents/agent_manager.py @@ -15,14 +15,15 @@ from backend.apps.agents.ws_manager import ws_manager from backend.apps.modes.modes import load_mode from backend.apps.outputs.outputs import _load_all as load_all_outputs from backend.apps.settings.settings import load_settings +from backend.apps.common.mcp_utils import sanitize_server_name as _sanitize_server_name from backend.apps.tools_lib.tools_lib import ( _load_all as load_all_tools, - _sanitize_server_name, derive_mcp_config, load_builtin_permissions, refresh_google_token, ) from backend.config.paths import SESSIONS_DIR +from backend.apps.common.json_store import SessionStore from backend.apps.analytics.collector import record as _analytics logger = logging.getLogger(__name__) @@ -30,35 +31,12 @@ logger = logging.getLogger(__name__) os.environ.setdefault("CLAUDE_CODE_STREAM_CLOSE_TIMEOUT", "3600000") -def _save_session(session_id: str, doc_data: dict): - os.makedirs(SESSIONS_DIR, exist_ok=True) - with open(os.path.join(SESSIONS_DIR, f"{session_id}.json"), "w") as f: - json.dump(doc_data, f, indent=2) +_session_store = SessionStore(SESSIONS_DIR) - -def _load_session_data(session_id: str) -> dict | None: - path = os.path.join(SESSIONS_DIR, f"{session_id}.json") - if not os.path.exists(path): - return None - with open(path) as f: - return json.load(f) - - -def _delete_session_file(session_id: str): - path = os.path.join(SESSIONS_DIR, f"{session_id}.json") - if os.path.exists(path): - os.remove(path) - - -def _load_all_session_data() -> list[tuple[str, dict]]: - results = [] - if not os.path.exists(SESSIONS_DIR): - return results - for fname in os.listdir(SESSIONS_DIR): - if fname.endswith(".json"): - with open(os.path.join(SESSIONS_DIR, fname)) as f: - results.append((fname[:-5], json.load(f))) - return results +_save_session = _session_store.save +_load_session_data = _session_store.load +_delete_session_file = _session_store.delete +_load_all_session_data = _session_store.load_all FULL_TOOLS = [ "Read", "Edit", "Write", "Bash", "Glob", "Grep", "AskUserQuestion", diff --git a/backend/apps/agents/browser_agent.py b/backend/apps/agents/browser_agent.py index 7b9ee097..f6d9582a 100644 --- a/backend/apps/agents/browser_agent.py +++ b/backend/apps/agents/browser_agent.py @@ -17,16 +17,11 @@ import anthropic from backend.apps.agents.models import AgentSession, ApprovalRequest, Message from backend.apps.agents.ws_manager import ws_manager +from backend.apps.common.model_registry import resolve_model_id from backend.apps.tools_lib.tools_lib import load_builtin_permissions logger = logging.getLogger(__name__) -MODEL_MAP = { - "sonnet": "claude-sonnet-4-6", - "opus": "claude-opus-4-6", - "haiku": "claude-haiku-4-5-20251001", -} - BROWSER_TOOLS_SCHEMA = [ { "name": "BrowserScreenshot", @@ -321,7 +316,7 @@ async def run_browser_agent( ) logger.info(f"Browser agent {session_id}: navigated to {initial_url}: {nav_result.get('text', nav_result.get('error', ''))}") - api_model = MODEL_MAP.get(model, model) + api_model = resolve_model_id(model) from backend.apps.settings.settings import load_settings from backend.apps.settings.credentials import get_anthropic_client client = get_anthropic_client(load_settings()) diff --git a/backend/apps/agents/mcp_client.py b/backend/apps/agents/mcp_client.py index 303078ab..1ad69aa4 100644 --- a/backend/apps/agents/mcp_client.py +++ b/backend/apps/agents/mcp_client.py @@ -15,6 +15,7 @@ from dataclasses import dataclass, field from typing import Any from backend.apps.agents.providers.base import ToolSchema +from backend.apps.common.mcp_utils import parse_sse_json as _parse_sse_json logger = logging.getLogger(__name__) @@ -231,22 +232,7 @@ class MCPClientManager: conn._next_id = 3 # type: ignore[attr-defined] return conn - @staticmethod - def _parse_sse_json(text: str) -> dict | None: - """Extract JSON from an SSE response body.""" - for line in text.splitlines(): - stripped = line.strip() - if stripped.startswith("data:"): - payload = stripped[len("data:"):].strip() - if payload: - try: - return json.loads(payload) - except json.JSONDecodeError: - continue - try: - return json.loads(text) - except json.JSONDecodeError: - return None + _parse_sse_json = staticmethod(_parse_sse_json) async def call_tool( self, server_name: str, tool_name: str, arguments: dict, diff --git a/backend/apps/agents/providers/registry.py b/backend/apps/agents/providers/registry.py index 7ebe4c8e..09e38010 100644 --- a/backend/apps/agents/providers/registry.py +++ b/backend/apps/agents/providers/registry.py @@ -12,23 +12,18 @@ import logging from typing import Any, TYPE_CHECKING from backend.apps.agents.providers.base import BaseProvider +from backend.apps.common.model_registry import ( + get_builtin_models_by_provider, + get_context_window as _registry_get_context_window, + calculate_cost as _registry_calculate_cost, +) if TYPE_CHECKING: from backend.apps.settings.models import AppSettings logger = logging.getLogger(__name__) -# --------------------------------------------------------------------------- -# Tier 1: Built-in models (curated, we know their quirks) -# --------------------------------------------------------------------------- - -BUILTIN_MODELS: dict[str, list[dict[str, Any]]] = { - "Anthropic": [ - {"value": "sonnet", "label": "Claude Sonnet 4.6", "context_window": 1_000_000, "model_id": "claude-sonnet-4-6", "api": "anthropic"}, - {"value": "opus", "label": "Claude Opus 4.6", "context_window": 1_000_000, "model_id": "claude-opus-4-6", "api": "anthropic"}, - {"value": "haiku", "label": "Claude Haiku 4.5", "context_window": 200_000, "model_id": "claude-haiku-4-5", "api": "anthropic"}, - ], -} +BUILTIN_MODELS = get_builtin_models_by_provider() # --------------------------------------------------------------------------- # OpenRouter: built-in integration for 300+ models @@ -261,52 +256,17 @@ def get_available_models(settings: AppSettings) -> dict[str, list[dict]]: def get_context_window(provider: str, model: str, settings: AppSettings | None = None) -> int: """Look up context window for any model.""" - # Check built-in models first - for models in BUILTIN_MODELS.values(): - for m in models: - if m["value"] == model: - return m.get("context_window", 128_000) + result = _registry_get_context_window(model) + if result != 128_000: + return result - # Check custom providers if settings: for cp in getattr(settings, "custom_providers", []): for m in cp.models: if m.get("value") == model or m.get("id") == model: return m.get("context_window", 128_000) - return 128_000 # safe default - - -# --------------------------------------------------------------------------- -# Cost tracking -# --------------------------------------------------------------------------- - -COST_PER_1M_TOKENS: dict[tuple[str, str], tuple[float, float]] = { - # (provider, model): (input_cost_per_1M, output_cost_per_1M) - # Anthropic - ("Anthropic", "sonnet"): (3.0, 15.0), - ("Anthropic", "opus"): (5.0, 25.0), - ("Anthropic", "haiku"): (1.0, 5.0), - # OpenAI - ("OpenAI", "gpt-5.4"): (2.50, 15.0), - ("OpenAI", "gpt-5.4-mini"): (0.75, 3.0), - ("OpenAI", "o3"): (2.0, 8.0), - ("OpenAI", "o4-mini"): (1.10, 4.40), - # Google - ("Google", "gemini-2.5-flash"): (0.15, 0.60), - ("Google", "gemini-2.5-pro"): (1.25, 10.0), - # OpenRouter-backed (approximate) - ("xAI", "x-ai/grok-4-0214"): (3.0, 15.0), - ("Meta", "meta-llama/llama-4-maverick"): (0.50, 0.70), - ("Meta", "meta-llama/llama-4-scout"): (0.15, 0.40), - ("DeepSeek", "deepseek/deepseek-chat-v3-0324"): (0.30, 0.90), - ("DeepSeek", "deepseek/deepseek-r1"): (0.80, 2.40), - ("Mistral", "mistralai/mistral-large-2501"): (2.0, 6.0), - ("Mistral", "mistralai/mistral-small-3.1-24b-instruct"): (0.10, 0.30), - ("Qwen", "qwen/qwen3-coder"): (0.0, 0.0), - ("Qwen", "qwen/qwen3-235b-a22b"): (0.20, 0.70), - ("Cohere", "cohere/command-a-03-2025"): (2.50, 10.0), -} + return 128_000 def calculate_cost( @@ -314,16 +274,4 @@ def calculate_cost( input_tokens: int, output_tokens: int, ) -> float: """Calculate cost in USD from token counts.""" - # Direct lookup first - rates = COST_PER_1M_TOKENS.get((provider, model)) - if not rates: - # Case-insensitive provider lookup - lower = provider.lower() - for (p, m), r in COST_PER_1M_TOKENS.items(): - if p.lower() == lower and m == model: - rates = r - break - if not rates: - return 0.0 - input_rate, output_rate = rates - return (input_tokens * input_rate + output_tokens * output_rate) / 1_000_000 + return _registry_calculate_cost(provider, model, input_tokens, output_tokens) diff --git a/backend/apps/common/__init__.py b/backend/apps/common/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/apps/common/json_store.py b/backend/apps/common/json_store.py new file mode 100644 index 00000000..4fe0802b --- /dev/null +++ b/backend/apps/common/json_store.py @@ -0,0 +1,140 @@ +"""Generic JSON-file CRUD store for Pydantic models. + +Every sub-app in the backend persists entities as one JSON file per record +inside a data directory. This module eliminates that copy-paste. +""" + +from __future__ import annotations + +import json +import os +from typing import Generic, TypeVar + +from fastapi import HTTPException +from pydantic import BaseModel + +T = TypeVar("T", bound=BaseModel) + + +class JsonStore(Generic[T]): + """One-JSON-file-per-entity CRUD backed by a flat directory. + + Parameters + ---------- + model_cls: + The Pydantic model to (de)serialise. + data_dir: + Filesystem directory that holds ``{id}.json`` files. + id_field: + Name of the attribute used as the unique key (default ``"id"``). + dump_mode: + Passed to ``model_dump(mode=...)`` when serialising. Use ``"json"`` + for models that contain non-JSON-native types (e.g. ``datetime``). + not_found_detail: + Message for the ``HTTPException(404)`` raised by :meth:`load`. + """ + + def __init__( + self, + model_cls: type[T], + data_dir: str, + *, + id_field: str = "id", + dump_mode: str | None = None, + not_found_detail: str = "Not found", + ) -> None: + self._cls = model_cls + self._dir = data_dir + self._id = id_field + self._dump_mode = dump_mode + self._detail = not_found_detail + + # -- helpers ------------------------------------------------------------- + + def _path(self, item_id: str) -> str: + return os.path.join(self._dir, f"{item_id}.json") + + def _dump(self, item: T) -> dict: + if self._dump_mode: + return item.model_dump(mode=self._dump_mode) + return item.model_dump() + + # -- public API ---------------------------------------------------------- + + def load_all(self) -> list[T]: + result: list[T] = [] + if not os.path.exists(self._dir): + return result + for fname in os.listdir(self._dir): + if fname.endswith(".json"): + with open(os.path.join(self._dir, fname)) as f: + result.append(self._cls(**json.load(f))) + return result + + def save(self, item: T) -> None: + os.makedirs(self._dir, exist_ok=True) + item_id = getattr(item, self._id) + with open(self._path(item_id), "w") as f: + json.dump(self._dump(item), f, indent=2) + + def load(self, item_id: str) -> T: + path = self._path(item_id) + if not os.path.exists(path): + raise HTTPException(status_code=404, detail=self._detail) + with open(path) as f: + return self._cls(**json.load(f)) + + def load_or_none(self, item_id: str) -> T | None: + path = self._path(item_id) + if not os.path.exists(path): + return None + with open(path) as f: + return self._cls(**json.load(f)) + + def delete(self, item_id: str) -> None: + path = self._path(item_id) + if os.path.exists(path): + os.remove(path) + + def exists(self, item_id: str) -> bool: + return os.path.exists(self._path(item_id)) + + +class SessionStore: + """Specialised JSON store for agent session dicts (not Pydantic models). + + Sessions are stored as raw ``dict`` values keyed by ``session_id``. + """ + + def __init__(self, data_dir: str) -> None: + self._dir = data_dir + + def _path(self, session_id: str) -> str: + return os.path.join(self._dir, f"{session_id}.json") + + def save(self, session_id: str, doc_data: dict) -> None: + os.makedirs(self._dir, exist_ok=True) + with open(self._path(session_id), "w") as f: + json.dump(doc_data, f, indent=2) + + def load(self, session_id: str) -> dict | None: + path = self._path(session_id) + if not os.path.exists(path): + return None + with open(path) as f: + return json.load(f) + + def delete(self, session_id: str) -> None: + path = self._path(session_id) + if os.path.exists(path): + os.remove(path) + + def load_all(self) -> list[tuple[str, dict]]: + results: list[tuple[str, dict]] = [] + if not os.path.exists(self._dir): + return results + for fname in os.listdir(self._dir): + if fname.endswith(".json"): + with open(os.path.join(self._dir, fname)) as f: + results.append((fname[:-5], json.load(f))) + return results diff --git a/backend/apps/common/llm_helpers.py b/backend/apps/common/llm_helpers.py new file mode 100644 index 00000000..497aebd6 --- /dev/null +++ b/backend/apps/common/llm_helpers.py @@ -0,0 +1,54 @@ +"""Convenience wrappers for quick LLM calls. + +These helpers handle client construction, markdown fence stripping, and JSON +parsing so that callers don't need to repeat the same boilerplate. +""" + +from __future__ import annotations + +import json +import logging +import re + +logger = logging.getLogger(__name__) + + +def strip_markdown_fences(text: str) -> str: + """Remove ```json ... ``` or similar fences from LLM output.""" + stripped = text.strip() + if stripped.startswith("```"): + stripped = re.sub(r"^```[a-zA-Z]*\n?", "", stripped, count=1) + stripped = re.sub(r"\n?```\s*$", "", stripped) + return stripped.strip() + + +async def quick_llm_call( + system: str, + user_content: str, + model: str = "claude-sonnet-4-20250514", + max_tokens: int = 300, +) -> str: + """Make a simple LLM call and return the text response.""" + from backend.apps.settings.credentials import get_anthropic_client + from backend.apps.settings.settings import load_settings + + client = get_anthropic_client(load_settings()) + resp = await client.messages.create( + model=model, + max_tokens=max_tokens, + system=system, + messages=[{"role": "user", "content": user_content}], + ) + return resp.content[0].text.strip() + + +async def quick_llm_json( + system: str, + user_content: str, + model: str = "claude-sonnet-4-20250514", + max_tokens: int = 300, +) -> dict: + """Make an LLM call expecting JSON. Strips markdown fences, parses JSON.""" + raw = await quick_llm_call(system, user_content, model=model, max_tokens=max_tokens) + cleaned = strip_markdown_fences(raw) + return json.loads(cleaned) diff --git a/backend/apps/common/mcp_utils.py b/backend/apps/common/mcp_utils.py new file mode 100644 index 00000000..abe43750 --- /dev/null +++ b/backend/apps/common/mcp_utils.py @@ -0,0 +1,28 @@ +"""Shared MCP / SSE utilities used by tools_lib and mcp_client.""" + +from __future__ import annotations + +import json +import re + + +def parse_sse_json(text: str) -> dict | None: + """Extract JSON from an SSE response body (handles ``data: {...}`` lines).""" + for line in text.splitlines(): + stripped = line.strip() + if stripped.startswith("data:"): + payload = stripped[len("data:"):].strip() + if payload: + try: + return json.loads(payload) + except json.JSONDecodeError: + continue + try: + return json.loads(text) + except json.JSONDecodeError: + return None + + +def sanitize_server_name(name: str) -> str: + """Convert a tool name into a valid MCP server identifier (alphanumeric + hyphens).""" + return re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-") diff --git a/backend/apps/common/model_registry.py b/backend/apps/common/model_registry.py new file mode 100644 index 00000000..02cc42d9 --- /dev/null +++ b/backend/apps/common/model_registry.py @@ -0,0 +1,111 @@ +"""Single source of truth for model definitions, IDs, and cost rates. + +Other modules should import from here instead of maintaining their own +``MODEL_MAP`` / ``BUILTIN_MODELS`` / ``COST_PER_1M_TOKENS`` copies. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + + +@dataclass(frozen=True) +class ModelDef: + value: str + label: str + model_id: str + provider: str + api: str + context_window: int + input_cost_per_1m: float + output_cost_per_1m: float + + +# fmt: off +ALL_MODELS: list[ModelDef] = [ + # Anthropic + ModelDef("sonnet", "Claude Sonnet 4.6", "claude-sonnet-4-6", "Anthropic", "anthropic", 1_000_000, 3.0, 15.0), + ModelDef("opus", "Claude Opus 4.6", "claude-opus-4-6", "Anthropic", "anthropic", 1_000_000, 5.0, 25.0), + ModelDef("haiku", "Claude Haiku 4.5", "claude-haiku-4-5", "Anthropic", "anthropic", 200_000, 1.0, 5.0), + # OpenAI + ModelDef("gpt-5.4", "GPT-5.4", "gpt-5.4", "OpenAI", "openai", 128_000, 2.50, 15.0), + ModelDef("gpt-5.4-mini", "GPT-5.4 Mini", "gpt-5.4-mini", "OpenAI", "openai", 128_000, 0.75, 3.0), + ModelDef("o3", "o3", "o3", "OpenAI", "openai", 128_000, 2.0, 8.0), + ModelDef("o4-mini", "o4-mini", "o4-mini", "OpenAI", "openai", 128_000, 1.10, 4.40), + # Google + ModelDef("gemini-2.5-flash", "Gemini 2.5 Flash", "gemini-2.5-flash", "Google", "gemini", 1_000_000, 0.15, 0.60), + ModelDef("gemini-2.5-pro", "Gemini 2.5 Pro", "gemini-2.5-pro", "Google", "gemini", 1_000_000, 1.25, 10.0), + # OpenRouter-backed + ModelDef("x-ai/grok-4-0214", "Grok 4", "x-ai/grok-4-0214", "xAI", "openrouter", 128_000, 3.0, 15.0), + ModelDef("meta-llama/llama-4-maverick", "Llama 4 Maverick", "meta-llama/llama-4-maverick", "Meta", "openrouter", 128_000, 0.50, 0.70), + ModelDef("meta-llama/llama-4-scout", "Llama 4 Scout", "meta-llama/llama-4-scout", "Meta", "openrouter", 128_000, 0.15, 0.40), + ModelDef("deepseek/deepseek-chat-v3-0324", "DeepSeek V3", "deepseek/deepseek-chat-v3-0324", "DeepSeek", "openrouter", 128_000, 0.30, 0.90), + ModelDef("deepseek/deepseek-r1", "DeepSeek R1", "deepseek/deepseek-r1", "DeepSeek", "openrouter", 128_000, 0.80, 2.40), + ModelDef("mistralai/mistral-large-2501", "Mistral Large", "mistralai/mistral-large-2501", "Mistral", "openrouter", 128_000, 2.0, 6.0), + ModelDef("mistralai/mistral-small-3.1-24b-instruct", "Mistral Small 3.1", "mistralai/mistral-small-3.1-24b-instruct", "Mistral", "openrouter", 128_000, 0.10, 0.30), + ModelDef("qwen/qwen3-coder", "Qwen3 Coder", "qwen/qwen3-coder", "Qwen", "openrouter", 128_000, 0.0, 0.0), + ModelDef("qwen/qwen3-235b-a22b", "Qwen3 235B", "qwen/qwen3-235b-a22b", "Qwen", "openrouter", 128_000, 0.20, 0.70), + ModelDef("cohere/command-a-03-2025", "Command A", "cohere/command-a-03-2025", "Cohere", "openrouter", 128_000, 2.50, 10.0), +] +# fmt: on + +_BY_VALUE: dict[str, ModelDef] = {m.value: m for m in ALL_MODELS} +_BY_MODEL_ID: dict[str, ModelDef] = {m.model_id: m for m in ALL_MODELS} + + +def resolve_model_id(short_name: str) -> str: + """Map a short name (e.g. ``"sonnet"``) to the canonical API model ID. + + Returns *short_name* unchanged if no mapping exists. + """ + m = _BY_VALUE.get(short_name) + return m.model_id if m else short_name + + +def get_cost_rates(provider: str, model: str) -> tuple[float, float] | None: + """Return ``(input_cost_per_1m, output_cost_per_1m)`` or ``None``.""" + m = _BY_VALUE.get(model) + if m and m.provider.lower() == provider.lower(): + return (m.input_cost_per_1m, m.output_cost_per_1m) + for md in ALL_MODELS: + if md.value == model and md.provider.lower() == provider.lower(): + return (md.input_cost_per_1m, md.output_cost_per_1m) + return None + + +def calculate_cost( + provider: str, model: str, input_tokens: int, output_tokens: int, +) -> float: + """Calculate cost in USD from token counts.""" + rates = get_cost_rates(provider, model) + if not rates: + return 0.0 + input_rate, output_rate = rates + return (input_tokens * input_rate + output_tokens * output_rate) / 1_000_000 + + +def get_context_window(model: str) -> int: + """Look up context window for a model by its short value name.""" + m = _BY_VALUE.get(model) or _BY_MODEL_ID.get(model) + return m.context_window if m else 128_000 + + +def get_builtin_models_by_provider() -> dict[str, list[dict[str, Any]]]: + """Return built-in models grouped by provider, matching the legacy format. + + Only includes the curated built-in models (Anthropic) — not + OpenRouter-backed models which are exposed through custom providers. + """ + result: dict[str, list[dict[str, Any]]] = {} + for m in ALL_MODELS: + if m.api == "openrouter": + continue + result.setdefault(m.provider, []).append({ + "value": m.value, + "label": m.label, + "context_window": m.context_window, + "model_id": m.model_id, + "api": m.api, + }) + return result diff --git a/backend/apps/dashboards/dashboards.py b/backend/apps/dashboards/dashboards.py index 5a21d4a6..1c605b56 100644 --- a/backend/apps/dashboards/dashboards.py +++ b/backend/apps/dashboards/dashboards.py @@ -6,6 +6,7 @@ from datetime import datetime from uuid import uuid4 from backend.config.Apps import SubApp +from backend.apps.common.json_store import JsonStore from backend.apps.dashboards.models import ( Dashboard, DashboardCreate, @@ -24,34 +25,14 @@ from backend.config.paths import DASHBOARDS_DIR as DATA_DIR, SESSIONS_DIR, DASHB OLD_LAYOUT_FILE = os.path.join(OLD_LAYOUT_DIR, "layout.json") -def _load_all() -> list[Dashboard]: - result = [] - if not os.path.exists(DATA_DIR): - return result - for fname in os.listdir(DATA_DIR): - if fname.endswith(".json"): - with open(os.path.join(DATA_DIR, fname)) as f: - result.append(Dashboard(**json.load(f))) - return result +_store = JsonStore( + Dashboard, DATA_DIR, dump_mode="json", not_found_detail="Dashboard not found", +) - -def _save(dashboard: Dashboard): - with open(os.path.join(DATA_DIR, f"{dashboard.id}.json"), "w") as f: - json.dump(dashboard.model_dump(mode="json"), f, indent=2) - - -def _load(dashboard_id: str) -> Dashboard: - path = os.path.join(DATA_DIR, f"{dashboard_id}.json") - if not os.path.exists(path): - raise HTTPException(status_code=404, detail="Dashboard not found") - with open(path) as f: - return Dashboard(**json.load(f)) - - -def _delete(dashboard_id: str): - path = os.path.join(DATA_DIR, f"{dashboard_id}.json") - if os.path.exists(path): - os.remove(path) +_load_all = _store.load_all +_save = _store.save +_load = _store.load +_delete = _store.delete def _migrate_if_needed(): diff --git a/backend/apps/modes/builtin.py b/backend/apps/modes/builtin.py new file mode 100644 index 00000000..a7f3a0f7 --- /dev/null +++ b/backend/apps/modes/builtin.py @@ -0,0 +1,137 @@ +"""Built-in mode definitions. + +Separated from models.py to keep schema classes small and data separate. +""" + +from backend.apps.modes.models import Mode +from backend.config.paths import OUTPUTS_WORKSPACE_DIR as OUTPUTS_WORKSPACE, SKILLS_WORKSPACE_DIR as SKILLS_WORKSPACE + +BUILTIN_MODES: list[Mode] = [ + Mode( + id="agent", + name="Agent", + description="Full autonomous agent with read and write access to tools.", + system_prompt=None, + tools=None, + default_next_mode=None, + is_builtin=True, + icon="smart_toy", + color="#818cf8", + ), + Mode( + id="ask", + name="Ask", + description="Answer questions about the codebase. Read-only, no edits or changes.", + system_prompt="Answer questions about the codebase. Do not make any edits or changes.", + tools=["Read", "Glob", "Grep", "AskUserQuestion"], + default_next_mode=None, + is_builtin=True, + icon="question_answer", + color="#4ade80", + ), + Mode( + id="plan", + name="Plan", + description="Analyze requests and produce a detailed step-by-step plan without executing.", + system_prompt="Analyze the request and produce a detailed step-by-step plan. Do not execute the plan or make any changes.", + tools=["Read", "Glob", "Grep", "AskUserQuestion"], + default_next_mode="agent", + is_builtin=True, + icon="map", + color="#fbbf24", + ), + Mode( + id="view-builder", + name="App Builder", + description="Create and iterate on reusable App artifacts.", + system_prompt=( + "You are an App Builder — an AI assistant that creates self-contained " + "web apps rendered in an iframe preview.\n\n" + "Your working directory is a dedicated workspace folder pre-seeded with " + "template files. Read the existing files before making changes.\n\n" + "## Critical rules\n\n" + "- The entry point MUST be named `index.html`. Never rename it or create " + "a different HTML file as the main entry point.\n" + "- Write files immediately when you have code ready — the user sees a " + "live preview that auto-refreshes from these files.\n" + "- Always write the complete file content on first creation (do not use " + "Edit for partial patches on new files).\n" + "- For complex apps, split code into separate files (JS, CSS, etc.) " + "and reference them from index.html with relative paths.\n" + "- Always update meta.json with a short name and one-sentence description.\n" + "- Build beautiful, polished UIs with modern design — dark themes, smooth " + "transitions, proper spacing, and responsive layouts.\n\n" + "Read the SKILL.md reference in your workspace for the full technical " + "specification of the App platform (available globals, file conventions, " + "schema format, backend.py usage, and examples)." + ), + tools=None, + default_next_mode=None, + is_builtin=True, + icon="view_quilt", + color="#f472b6", + default_folder=OUTPUTS_WORKSPACE, + ), + Mode( + id="skill-builder", + name="Skill Builder", + description="Create and iterate on skills using AI-assisted vibe coding.", + system_prompt=( + "You are a Skill Builder — an AI assistant that helps users create, " + "refine, and iterate on Claude skills (SKILL.md files).\n\n" + "## How Skills Work\n\n" + "A skill is a Markdown file that teaches Claude how to perform a specific task. " + "Skills have YAML frontmatter with `name` and `description` fields, followed by " + "the skill body in Markdown. The description is the primary triggering mechanism — " + "it tells Claude when to use the skill.\n\n" + "## Your Working Directory\n\n" + "Your working directory is a dedicated workspace folder for this skill. " + "Write your output directly to these files using the Write tool:\n\n" + "1. **SKILL.md** — The complete skill file with YAML frontmatter and Markdown body. " + "Example frontmatter:\n" + " ```\n" + " ---\n" + " name: my-skill\n" + " description: When to trigger and what this skill does.\n" + " ---\n" + " ```\n\n" + "2. **meta.json** — Metadata for the skill builder UI. Always write this file. Example:\n" + ' {"name":"My Skill","description":"A short description","command":"my-skill"}\n\n' + "Write these files immediately when you have content ready. The user can see " + "a live preview that auto-refreshes from these files. Always write the " + "complete file content (do not use Edit for partial patches on first creation).\n\n" + "## Skill Creation Process\n\n" + "1. **Understand intent** — Ask what the skill should do, when it should trigger, " + "and what the expected output format is.\n" + "2. **Draft the skill** — Write a SKILL.md with clear instructions, examples, " + "and good progressive disclosure.\n" + "3. **Iterate** — Refine based on user feedback. Update the files each time.\n\n" + "## Skill Writing Best Practices\n\n" + "- Keep SKILL.md under 500 lines; use bundled reference files for large content.\n" + "- The `description` frontmatter is the primary trigger. Make it slightly \"pushy\" — " + "include both what the skill does AND specific contexts for when to use it.\n" + "- Use imperative form in instructions.\n" + "- Include examples with input/output pairs when helpful.\n" + "- Define output formats explicitly with templates.\n" + "- Use theory of mind — explain *why* things matter rather than just MUST directives.\n" + "- Think about edge cases, error handling, and progressive disclosure.\n\n" + "## Skill Anatomy\n\n" + "```\n" + "skill-name/\n" + "├── SKILL.md (required) — YAML frontmatter + Markdown instructions\n" + "└── Bundled Resources (optional)\n" + " ├── scripts/ — Executable code for repetitive tasks\n" + " ├── references/ — Docs loaded into context as needed\n" + " └── assets/ — Files used in output\n" + "```\n\n" + "Be collaborative and flexible. If the user wants to \"just vibe\", skip the formal " + "process and iterate freely. Always write updated files so the preview stays current." + ), + tools=None, + default_next_mode=None, + is_builtin=True, + icon="psychology", + color="#10b981", + default_folder=SKILLS_WORKSPACE, + ), +] diff --git a/backend/apps/modes/models.py b/backend/apps/modes/models.py index 0afc742c..f63b155d 100644 --- a/backend/apps/modes/models.py +++ b/backend/apps/modes/models.py @@ -2,8 +2,6 @@ from pydantic import BaseModel, Field from typing import Optional from uuid import uuid4 -from backend.config.paths import OUTPUTS_WORKSPACE_DIR as OUTPUTS_WORKSPACE, SKILLS_WORKSPACE_DIR as SKILLS_WORKSPACE - class Mode(BaseModel): id: str = Field(default_factory=lambda: uuid4().hex) @@ -38,134 +36,3 @@ class ModeUpdate(BaseModel): icon: Optional[str] = None color: Optional[str] = None default_folder: Optional[str] = None - - -BUILTIN_MODES: list[Mode] = [ - Mode( - id="agent", - name="Agent", - description="Full autonomous agent with read and write access to tools.", - system_prompt=None, - tools=None, - default_next_mode=None, - is_builtin=True, - icon="smart_toy", - color="#818cf8", - ), - Mode( - id="ask", - name="Ask", - description="Answer questions about the codebase. Read-only, no edits or changes.", - system_prompt="Answer questions about the codebase. Do not make any edits or changes.", - tools=["Read", "Glob", "Grep", "AskUserQuestion"], - default_next_mode=None, - is_builtin=True, - icon="question_answer", - color="#4ade80", - ), - Mode( - id="plan", - name="Plan", - description="Analyze requests and produce a detailed step-by-step plan without executing.", - system_prompt="Analyze the request and produce a detailed step-by-step plan. Do not execute the plan or make any changes.", - tools=["Read", "Glob", "Grep", "AskUserQuestion"], - default_next_mode="agent", - is_builtin=True, - icon="map", - color="#fbbf24", - ), - Mode( - id="view-builder", - name="App Builder", - description="Create and iterate on reusable App artifacts.", - system_prompt=( - "You are an App Builder — an AI assistant that creates self-contained " - "web apps rendered in an iframe preview.\n\n" - "Your working directory is a dedicated workspace folder pre-seeded with " - "template files. Read the existing files before making changes.\n\n" - "## Critical rules\n\n" - "- The entry point MUST be named `index.html`. Never rename it or create " - "a different HTML file as the main entry point.\n" - "- Write files immediately when you have code ready — the user sees a " - "live preview that auto-refreshes from these files.\n" - "- Always write the complete file content on first creation (do not use " - "Edit for partial patches on new files).\n" - "- For complex apps, split code into separate files (JS, CSS, etc.) " - "and reference them from index.html with relative paths.\n" - "- Always update meta.json with a short name and one-sentence description.\n" - "- Build beautiful, polished UIs with modern design — dark themes, smooth " - "transitions, proper spacing, and responsive layouts.\n\n" - "Read the SKILL.md reference in your workspace for the full technical " - "specification of the App platform (available globals, file conventions, " - "schema format, backend.py usage, and examples)." - ), - tools=None, - default_next_mode=None, - is_builtin=True, - icon="view_quilt", - color="#f472b6", - default_folder=OUTPUTS_WORKSPACE, - ), - Mode( - id="skill-builder", - name="Skill Builder", - description="Create and iterate on skills using AI-assisted vibe coding.", - system_prompt=( - "You are a Skill Builder — an AI assistant that helps users create, " - "refine, and iterate on Claude skills (SKILL.md files).\n\n" - "## How Skills Work\n\n" - "A skill is a Markdown file that teaches Claude how to perform a specific task. " - "Skills have YAML frontmatter with `name` and `description` fields, followed by " - "the skill body in Markdown. The description is the primary triggering mechanism — " - "it tells Claude when to use the skill.\n\n" - "## Your Working Directory\n\n" - "Your working directory is a dedicated workspace folder for this skill. " - "Write your output directly to these files using the Write tool:\n\n" - "1. **SKILL.md** — The complete skill file with YAML frontmatter and Markdown body. " - "Example frontmatter:\n" - " ```\n" - " ---\n" - " name: my-skill\n" - " description: When to trigger and what this skill does.\n" - " ---\n" - " ```\n\n" - "2. **meta.json** — Metadata for the skill builder UI. Always write this file. Example:\n" - ' {"name":"My Skill","description":"A short description","command":"my-skill"}\n\n' - "Write these files immediately when you have content ready. The user can see " - "a live preview that auto-refreshes from these files. Always write the " - "complete file content (do not use Edit for partial patches on first creation).\n\n" - "## Skill Creation Process\n\n" - "1. **Understand intent** — Ask what the skill should do, when it should trigger, " - "and what the expected output format is.\n" - "2. **Draft the skill** — Write a SKILL.md with clear instructions, examples, " - "and good progressive disclosure.\n" - "3. **Iterate** — Refine based on user feedback. Update the files each time.\n\n" - "## Skill Writing Best Practices\n\n" - "- Keep SKILL.md under 500 lines; use bundled reference files for large content.\n" - "- The `description` frontmatter is the primary trigger. Make it slightly \"pushy\" — " - "include both what the skill does AND specific contexts for when to use it.\n" - "- Use imperative form in instructions.\n" - "- Include examples with input/output pairs when helpful.\n" - "- Define output formats explicitly with templates.\n" - "- Use theory of mind — explain *why* things matter rather than just MUST directives.\n" - "- Think about edge cases, error handling, and progressive disclosure.\n\n" - "## Skill Anatomy\n\n" - "```\n" - "skill-name/\n" - "├── SKILL.md (required) — YAML frontmatter + Markdown instructions\n" - "└── Bundled Resources (optional)\n" - " ├── scripts/ — Executable code for repetitive tasks\n" - " ├── references/ — Docs loaded into context as needed\n" - " └── assets/ — Files used in output\n" - "```\n\n" - "Be collaborative and flexible. If the user wants to \"just vibe\", skip the formal " - "process and iterate freely. Always write updated files so the preview stays current." - ), - tools=None, - default_next_mode=None, - is_builtin=True, - icon="psychology", - color="#10b981", - default_folder=SKILLS_WORKSPACE, - ), -] diff --git a/backend/apps/modes/modes.py b/backend/apps/modes/modes.py index 37aabd90..b7c1a695 100644 --- a/backend/apps/modes/modes.py +++ b/backend/apps/modes/modes.py @@ -4,7 +4,9 @@ import logging from contextlib import asynccontextmanager from fastapi import HTTPException from backend.config.Apps import SubApp -from backend.apps.modes.models import Mode, ModeCreate, ModeUpdate, BUILTIN_MODES +from backend.apps.common.json_store import JsonStore +from backend.apps.modes.models import Mode, ModeCreate, ModeUpdate +from backend.apps.modes.builtin import BUILTIN_MODES logger = logging.getLogger(__name__) @@ -24,37 +26,12 @@ async def modes_lifespan(): modes = SubApp("modes", modes_lifespan) -def _load_all() -> list[Mode]: - result = [] - if not os.path.exists(DATA_DIR): - return result - for fname in os.listdir(DATA_DIR): - if fname.endswith(".json"): - with open(os.path.join(DATA_DIR, fname)) as f: - result.append(Mode(**json.load(f))) - return result +_store = JsonStore(Mode, DATA_DIR, not_found_detail="Mode not found") - -def _save(mode: Mode): - with open(os.path.join(DATA_DIR, f"{mode.id}.json"), "w") as f: - json.dump(mode.model_dump(), f, indent=2) - - -def _load(mode_id: str) -> Mode: - path = os.path.join(DATA_DIR, f"{mode_id}.json") - if not os.path.exists(path): - raise HTTPException(status_code=404, detail="Mode not found") - with open(path) as f: - return Mode(**json.load(f)) - - -def load_mode(mode_id: str) -> Mode | None: - """Public helper for other modules to resolve a mode by ID.""" - path = os.path.join(DATA_DIR, f"{mode_id}.json") - if not os.path.exists(path): - return None - with open(path) as f: - return Mode(**json.load(f)) +_load_all = _store.load_all +_save = _store.save +_load = _store.load +load_mode = _store.load_or_none @modes.router.get("/list") @@ -109,7 +86,5 @@ async def delete_mode(mode_id: str): mode = _load(mode_id) if mode.is_builtin: raise HTTPException(status_code=403, detail="Cannot delete built-in modes") - path = os.path.join(DATA_DIR, f"{mode_id}.json") - if os.path.exists(path): - os.remove(path) + _store.delete(mode_id) return {"ok": True} diff --git a/backend/apps/outputs/models.py b/backend/apps/outputs/models.py index 79e1b922..c1d595a9 100644 --- a/backend/apps/outputs/models.py +++ b/backend/apps/outputs/models.py @@ -4,6 +4,53 @@ from uuid import uuid4 from datetime import datetime +def _migrate_legacy_files( + data: dict, + *, + allow_schema_json: bool = False, + always_set_files: bool = False, +) -> dict: + """Convert legacy frontend_code/backend_code fields into the files dict. + + Parameters + ---------- + allow_schema_json: + Also migrate a ``schema_json`` field to ``files["schema.json"]``. + always_set_files: + When ``True``, set ``data["files"]`` even to an empty dict if no + legacy fields are found (used by Output / OutputCreate). When + ``False``, only set ``data["files"]`` if there are actual files to + migrate (used by OutputUpdate / WorkspaceSeedRequest). + """ + if not isinstance(data, dict): + return data + + files_present = "files" in data + files_truthy = files_present and data["files"] + + if not files_present or (always_set_files and not files_truthy): + files: dict[str, str] = {} + fc = data.pop("frontend_code", None) + bc = data.pop("backend_code", None) + if fc: + files["index.html"] = fc + if bc: + files["backend.py"] = bc + if allow_schema_json: + sj = data.pop("schema_json", None) + if sj: + files["schema.json"] = sj + if files or always_set_files: + data["files"] = files + else: + data.pop("frontend_code", None) + data.pop("backend_code", None) + if allow_schema_json: + data.pop("schema_json", None) + + return data + + class AutoRunConfig(BaseModel): enabled: bool = False prompt: str = "" @@ -33,22 +80,7 @@ class Output(BaseModel): @model_validator(mode="before") @classmethod def _migrate_flat_fields(cls, data: Any) -> Any: - """Migrate legacy frontend_code/backend_code fields into the files dict.""" - if not isinstance(data, dict): - return data - if "files" not in data or not data["files"]: - files: dict[str, str] = {} - fc = data.pop("frontend_code", None) - bc = data.pop("backend_code", None) - if fc: - files["index.html"] = fc - if bc: - files["backend.py"] = bc - data["files"] = files - else: - data.pop("frontend_code", None) - data.pop("backend_code", None) - return data + return _migrate_legacy_files(data, always_set_files=True) @property def frontend_code(self) -> str: @@ -75,21 +107,7 @@ class OutputCreate(BaseModel): @model_validator(mode="before") @classmethod def _migrate_flat_fields(cls, data: Any) -> Any: - if not isinstance(data, dict): - return data - if "files" not in data or not data["files"]: - files: dict[str, str] = {} - fc = data.pop("frontend_code", None) - bc = data.pop("backend_code", None) - if fc: - files["index.html"] = fc - if bc: - files["backend.py"] = bc - data["files"] = files - else: - data.pop("frontend_code", None) - data.pop("backend_code", None) - return data + return _migrate_legacy_files(data, always_set_files=True) class OutputUpdate(BaseModel): @@ -105,22 +123,7 @@ class OutputUpdate(BaseModel): @model_validator(mode="before") @classmethod def _migrate_flat_fields(cls, data: Any) -> Any: - if not isinstance(data, dict): - return data - if "files" not in data: - files: dict[str, str] = {} - fc = data.pop("frontend_code", None) - bc = data.pop("backend_code", None) - if fc: - files["index.html"] = fc - if bc: - files["backend.py"] = bc - if files: - data["files"] = files - else: - data.pop("frontend_code", None) - data.pop("backend_code", None) - return data + return _migrate_legacy_files(data) class OutputExecute(BaseModel): @@ -165,27 +168,7 @@ class WorkspaceSeedRequest(BaseModel): @model_validator(mode="before") @classmethod def _migrate_flat_fields(cls, data: Any) -> Any: - """Accept legacy frontend_code/backend_code/schema_json fields.""" - if not isinstance(data, dict): - return data - if "files" not in data: - files: dict[str, str] = {} - fc = data.pop("frontend_code", None) - bc = data.pop("backend_code", None) - sj = data.pop("schema_json", None) - if fc: - files["index.html"] = fc - if bc: - files["backend.py"] = bc - if sj: - files["schema.json"] = sj - if files: - data["files"] = files - else: - data.pop("frontend_code", None) - data.pop("backend_code", None) - data.pop("schema_json", None) - return data + return _migrate_legacy_files(data, allow_schema_json=True) class VibeCodeRequest(BaseModel): diff --git a/backend/apps/outputs/outputs.py b/backend/apps/outputs/outputs.py index 311b771f..2f7c9955 100644 --- a/backend/apps/outputs/outputs.py +++ b/backend/apps/outputs/outputs.py @@ -9,6 +9,7 @@ from fastapi import HTTPException, Query from fastapi.responses import Response from jsonschema import validate as schema_validate, ValidationError as SchemaValidationError from backend.config.Apps import SubApp +from backend.apps.common.json_store import JsonStore from backend.apps.outputs.models import ( Output, OutputCreate, OutputUpdate, OutputExecute, OutputExecuteResult, VibeCodeRequest, AutoRunRequest, AutoRunConfig, AutoRunAgentRequest, @@ -16,20 +17,11 @@ from backend.apps.outputs.models import ( ) from backend.apps.outputs.executor import execute_backend_code from backend.apps.outputs.view_builder_templates import VIEW_BUILDER_SKILL, VIEW_TEMPLATE_FILES +from backend.apps.common.model_registry import resolve_model_id as _resolve_model from backend.apps.settings.settings import load_settings logger = logging.getLogger(__name__) -MODEL_MAP = { - "sonnet": "claude-sonnet-4-20250514", - "opus": "claude-opus-4-20250514", - "haiku": "claude-haiku-4-5-20251001", -} - - -def _resolve_model(short_name: str) -> str: - return MODEL_MAP.get(short_name, short_name) - def _get_anthropic_client(): """Create an AsyncAnthropic client using the API key from app settings.""" @@ -100,37 +92,12 @@ async def outputs_lifespan(): outputs = SubApp("outputs", outputs_lifespan) -def _load_all() -> list[Output]: - result = [] - if not os.path.exists(DATA_DIR): - return result - for fname in os.listdir(DATA_DIR): - if fname.endswith(".json"): - with open(os.path.join(DATA_DIR, fname)) as f: - result.append(Output(**json.load(f))) - return result +_store = JsonStore(Output, DATA_DIR, not_found_detail="Output not found") - -def _save(output: Output): - with open(os.path.join(DATA_DIR, f"{output.id}.json"), "w") as f: - json.dump(output.model_dump(), f, indent=2) - - -def _load(output_id: str) -> Output: - path = os.path.join(DATA_DIR, f"{output_id}.json") - if not os.path.exists(path): - raise HTTPException(status_code=404, detail="Output not found") - with open(path) as f: - return Output(**json.load(f)) - - -def load_output(output_id: str) -> Output | None: - """Public helper for other modules to resolve an output by ID.""" - path = os.path.join(DATA_DIR, f"{output_id}.json") - if not os.path.exists(path): - return None - with open(path) as f: - return Output(**json.load(f)) +_load_all = _store.load_all +_save = _store.save +_load = _store.load +load_output = _store.load_or_none def _walk_directory(folder: str) -> dict[str, str]: diff --git a/backend/apps/templates/templates.py b/backend/apps/templates/templates.py index e3d9e96c..df7e522b 100644 --- a/backend/apps/templates/templates.py +++ b/backend/apps/templates/templates.py @@ -4,6 +4,7 @@ import logging from contextlib import asynccontextmanager from fastapi import HTTPException from backend.config.Apps import SubApp +from backend.apps.common.json_store import JsonStore from backend.apps.templates.models import PromptTemplate, PromptTemplateCreate, PromptTemplateUpdate logger = logging.getLogger(__name__) @@ -17,32 +18,12 @@ async def templates_lifespan(): templates = SubApp("templates", templates_lifespan) -def _load_all() -> list[PromptTemplate]: - result = [] - if not os.path.exists(DATA_DIR): - return result - for fname in os.listdir(DATA_DIR): - if fname.endswith(".json"): - with open(os.path.join(DATA_DIR, fname)) as f: - result.append(PromptTemplate(**json.load(f))) - return result +_store = JsonStore(PromptTemplate, DATA_DIR, not_found_detail="Template not found") -def _save(template: PromptTemplate): - path = os.path.join(DATA_DIR, f"{template.id}.json") - with open(path, "w") as f: - json.dump(template.model_dump(), f, indent=2) - -def _load(template_id: str) -> PromptTemplate: - path = os.path.join(DATA_DIR, f"{template_id}.json") - if not os.path.exists(path): - raise HTTPException(status_code=404, detail="Template not found") - with open(path) as f: - return PromptTemplate(**json.load(f)) - -def _delete(template_id: str): - path = os.path.join(DATA_DIR, f"{template_id}.json") - if os.path.exists(path): - os.remove(path) +_load_all = _store.load_all +_save = _store.save +_load = _store.load +_delete = _store.delete @templates.router.get("/list") async def list_templates(): diff --git a/backend/apps/tools_lib/tools_lib.py b/backend/apps/tools_lib/tools_lib.py index 316908cc..c1fe9288 100644 --- a/backend/apps/tools_lib/tools_lib.py +++ b/backend/apps/tools_lib/tools_lib.py @@ -18,6 +18,7 @@ from dotenv import load_dotenv from fastapi import HTTPException, Query from fastapi.responses import HTMLResponse from backend.config.Apps import SubApp +from backend.apps.common.json_store import JsonStore from backend.apps.tools_lib.models import ToolDefinition, ToolCreate, ToolUpdate, BUILTIN_TOOLS logger = logging.getLogger(__name__) @@ -252,28 +253,11 @@ _pending_oauth: dict[str, str] = {} _pending_pkce: dict[str, str] = {} # state -> code_verifier (for PKCE flows) -def _load_all() -> list[ToolDefinition]: - result = [] - if not os.path.exists(DATA_DIR): - return result - for fname in os.listdir(DATA_DIR): - if fname.endswith(".json"): - with open(os.path.join(DATA_DIR, fname)) as f: - result.append(ToolDefinition(**json.load(f))) - return result +_store = JsonStore(ToolDefinition, DATA_DIR, not_found_detail="Tool not found") - -def _save(tool: ToolDefinition): - with open(os.path.join(DATA_DIR, f"{tool.id}.json"), "w") as f: - json.dump(tool.model_dump(), f, indent=2) - - -def _load(tool_id: str) -> ToolDefinition: - path = os.path.join(DATA_DIR, f"{tool_id}.json") - if not os.path.exists(path): - raise HTTPException(status_code=404, detail="Tool not found") - with open(path) as f: - return ToolDefinition(**json.load(f)) +_load_all = _store.load_all +_save = _store.save +_load = _store.load @tools_lib.router.get("/builtin") @@ -478,9 +462,7 @@ async def delete_tool(tool_id: str): # MCP config derivation # --------------------------------------------------------------------------- -def _sanitize_server_name(name: str) -> str: - """Convert a tool name into a valid MCP server identifier (alphanumeric + hyphens).""" - return re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-") +from backend.apps.common.mcp_utils import sanitize_server_name as _sanitize_server_name def _extra_bin_dirs() -> list[str]: @@ -637,15 +619,6 @@ def derive_mcp_config(tool: ToolDefinition) -> Optional[dict]: return config -# --------------------------------------------------------------------------- -# OAuth2 flow for Google Workspace (and other OAuth providers) -# --------------------------------------------------------------------------- - - -# --------------------------------------------------------------------------- -# MCP tool discovery -# --------------------------------------------------------------------------- - _READ_PREFIXES = ("get", "list", "read", "search", "fetch", "find", "query", "count", "check", "describe", "show", "download", "browse", "analy", "explain") _WRITE_PREFIXES = ("create", "write", "delete", "update", "send", "remove", "modify", "add", "set", "put", "post", "patch", "insert", "move", "copy", "rename", "archive", "trash", "publish", "approve", "reject") @@ -770,21 +743,7 @@ def _extract_service(name: str) -> tuple[str, str]: return "Other", "" -def _parse_sse_json(text: str) -> dict | None: - """Extract JSON from an SSE response body (handles `data: {...}` lines).""" - for line in text.splitlines(): - stripped = line.strip() - if stripped.startswith("data:"): - payload = stripped[len("data:"):].strip() - if payload: - try: - return json.loads(payload) - except json.JSONDecodeError: - continue - try: - return json.loads(text) - except json.JSONDecodeError: - return None +from backend.apps.common.mcp_utils import parse_sse_json as _parse_sse_json async def _discover_mcp_tools_http(url: str, headers: dict | None = None) -> list[dict]: diff --git a/backend/run.sh b/backend/run.sh index a94240ba..f79a734c 100755 --- a/backend/run.sh +++ b/backend/run.sh @@ -2,15 +2,9 @@ # The comment above is shebang, DO NOT REMOVE DEV_ABSPATH="$(readlink -f "${BASH_SOURCE[0]}")" if [[ "$OSTYPE" == "darwin"* ]]; then - # echo "In macOS server sed START" - # echo "SERVER_ABSPATH: $SERVER_ABSPATH" sed -i '' 's/\r//g' "$DEV_ABSPATH" - # echo "In macOS server sed END" else - # echo "NOT in macOS server START" - # echo "SERVER_ABSPATH: $SERVER_ABSPATH" sed -i 's/\r//g' "$DEV_ABSPATH" - # echo "NOT in macOS server START" fi chmod +x "$DEV_ABSPATH" @@ -24,14 +18,47 @@ cleanup() { } trap cleanup EXIT INT TERM +# --- Find Python >= 3.10 --- +REQUIRED_PYTHON_MINOR=10 +PYTHON_BIN="" +for candidate in python3.13 python3.12 python3.11 python3.10 python3; do + if command -v "$candidate" &>/dev/null; then + ver=$("$candidate" -c "import sys; print(sys.version_info.minor)" 2>/dev/null) + if [[ -n "$ver" ]] && (( ver >= REQUIRED_PYTHON_MINOR )); then + PYTHON_BIN="$(command -v "$candidate")" + break + fi + fi +done + +if [[ -z "$PYTHON_BIN" ]]; then + echo "ERROR: Python >= 3.${REQUIRED_PYTHON_MINOR} is required but not found." + echo "Install it with: brew install python@3.13" + exit 1 +fi + # --- Create virtual environment if it doesn't exist --- VENV_DIR="$BACKEND_DIR_ABSPATH/.venv" if [[ ! -d "$VENV_DIR" ]]; then - echo "Creating virtual environment..." - python3 -m venv "$VENV_DIR" + echo "Creating virtual environment with $("$PYTHON_BIN" --version)..." + "$PYTHON_BIN" -m venv "$VENV_DIR" fi + +# --- Verify the venv Python meets the minimum version --- +VENV_PYTHON="$VENV_DIR/bin/python3" +VENV_VER=$("$VENV_PYTHON" -c "import sys; print(sys.version_info.minor)" 2>/dev/null) +if [[ -z "$VENV_VER" ]] || (( VENV_VER < REQUIRED_PYTHON_MINOR )); then + echo "Existing venv uses Python 3.${VENV_VER:-?}, need >= 3.${REQUIRED_PYTHON_MINOR}. Recreating..." + rm -rf "$VENV_DIR" + echo "Creating virtual environment with $("$PYTHON_BIN" --version)..." + "$PYTHON_BIN" -m venv "$VENV_DIR" +fi + source "$VENV_DIR/bin/activate" +# --- Upgrade pip if outdated --- +pip3 install --upgrade pip --quiet + # --- Install custom debugger module if not already installed --- DEBUGGER_DIR_ABSPATH="$PROJECT_ROOT_ABSPATH/debugger" if ! pip3 show debug > /dev/null 2>&1; then @@ -58,4 +85,4 @@ echo "Starting backend server on http://0.0.0.0:8324 ..." cd "$PROJECT_ROOT_ABSPATH" python3 -m uvicorn backend.main:app --host 0.0.0.0 --port 8324 --reload \ --reload-dir "$BACKEND_DIR_ABSPATH" \ - --reload-exclude '*.pyc' + --reload-exclude '*.pyc' \ No newline at end of file