mirror of
https://github.com/openswarm-ai/openswarm.git
synced 2026-09-05 17:27:42 +02:00
[Haik]: Phase 1 agentic refactor. Foundations + Hygiene
This commit is contained in:
haikdc
@@ -0,0 +1,202 @@
|
||||
# Agent 1: Foundations + Hygiene (Phase 0 + Phase 4)
|
||||
|
||||
## Context
|
||||
|
||||
You are cleaning up the OpenSwarm codebase. This is the first of 4 agents. Your job is to build shared infrastructure that the other agents will depend on, plus minor hygiene fixes.
|
||||
|
||||
**Rules:**
|
||||
- Every file you create or modify must be <250 lines of code
|
||||
- Keep code DRY
|
||||
- Do NOT touch the `9router/` or `debugger/` directories
|
||||
- Do NOT touch tests
|
||||
- Run the app after your changes to make sure nothing is broken: `cd backend && python -m backend.main`
|
||||
|
||||
---
|
||||
|
||||
## Task 0A: Create `backend/apps/common/json_store.py`
|
||||
|
||||
**Problem:** Every sub-app copy-pastes identical `_load_all()`, `_save()`, `_load()`, `_delete()` functions for JSON-file CRUD. This exists in:
|
||||
|
||||
- `backend/apps/tools_lib/tools_lib.py` (lines 255-277)
|
||||
- `backend/apps/outputs/outputs.py` (lines 103-134)
|
||||
- `backend/apps/dashboards/dashboards.py` (lines 27-55)
|
||||
- `backend/apps/agents/agent_manager.py` (lines 33-61)
|
||||
- `backend/apps/modes/modes.py`
|
||||
- `backend/apps/templates/templates.py`
|
||||
- `backend/apps/skills/skills.py`
|
||||
|
||||
**What to do:**
|
||||
|
||||
1. Create `backend/apps/common/__init__.py` (empty)
|
||||
2. Create `backend/apps/common/json_store.py` with a generic `JsonStore` class:
|
||||
|
||||
```python
|
||||
class JsonStore(Generic[T]):
|
||||
def __init__(self, model_cls: type[T], data_dir: str, id_field: str = "id"):
|
||||
...
|
||||
def load_all(self) -> list[T]: ...
|
||||
def save(self, item: T) -> None: ...
|
||||
def load(self, item_id: str) -> T: ... # raises HTTPException(404) if not found
|
||||
def delete(self, item_id: str) -> None: ...
|
||||
def exists(self, item_id: str) -> bool: ...
|
||||
```
|
||||
|
||||
3. Replace the inline `_load_all`, `_save`, `_load`, `_delete` in **all 7 sub-apps** with `JsonStore` instances.
|
||||
4. For the agents session store, it's slightly different (uses `session_id` not `id`, and has `_load_all_session_data` returning tuples). Create a small subclass or adapter — keep it clean.
|
||||
|
||||
**Verify:** All existing imports of `_load_all`, `_save`, `_load` from these modules still work (they're imported in `agent_manager.py`, `outputs.py`, `dashboards.py`, etc). If external code imports them, keep backward-compatible aliases.
|
||||
|
||||
---
|
||||
|
||||
## Task 0B: Create `backend/apps/common/model_registry.py`
|
||||
|
||||
**Problem:** Model name-to-ID mappings are duplicated with inconsistent values:
|
||||
|
||||
- `backend/apps/agents/browser_agent.py` lines 24-28: `MODEL_MAP` with `"sonnet": "claude-sonnet-4-6"`
|
||||
- `backend/apps/outputs/outputs.py` lines 23-27: `MODEL_MAP` with `"sonnet": "claude-sonnet-4-20250514"`
|
||||
- `backend/apps/agents/providers/registry.py` lines 25-31: `BUILTIN_MODELS` dict
|
||||
- `backend/apps/agents/providers/registry.py` lines 284-309: `COST_PER_1M_TOKENS` dict
|
||||
|
||||
**What to do:**
|
||||
|
||||
1. Create `backend/apps/common/model_registry.py` with:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ModelDef:
|
||||
value: str # short name ("sonnet")
|
||||
label: str # display name ("Claude Sonnet 4.6")
|
||||
model_id: str # API model ID ("claude-sonnet-4-6")
|
||||
provider: str # "Anthropic", "OpenAI", etc.
|
||||
api: str # "anthropic", "openai", "gemini", "openrouter"
|
||||
context_window: int
|
||||
input_cost_per_1m: float
|
||||
output_cost_per_1m: float
|
||||
|
||||
ALL_MODELS: list[ModelDef] = [ ... ] # Single source of truth
|
||||
|
||||
def resolve_model_id(short_name: str) -> str: ...
|
||||
def get_cost_rates(provider: str, model: str) -> tuple[float, float] | None: ...
|
||||
def calculate_cost(provider: str, model: str, input_tokens: int, output_tokens: int) -> float: ...
|
||||
def get_context_window(model: str) -> int: ...
|
||||
def get_builtin_models_by_provider() -> dict[str, list[dict]]: ...
|
||||
```
|
||||
|
||||
2. Use the canonical model IDs from `providers/registry.py` BUILTIN_MODELS (the `model_id` field there is the correct one: `claude-sonnet-4-6`, `claude-opus-4-6`, etc).
|
||||
|
||||
3. Delete `MODEL_MAP` from `browser_agent.py` and `outputs.py`, replace with `resolve_model_id()` import.
|
||||
|
||||
4. Delete `BUILTIN_MODELS` and `COST_PER_1M_TOKENS` from `providers/registry.py`, replace with imports from `model_registry.py`. Update `get_available_models()`, `calculate_cost()`, `get_context_window()` in `registry.py` to delegate to the new module.
|
||||
|
||||
---
|
||||
|
||||
## Task 0C: Create `backend/apps/common/mcp_utils.py`
|
||||
|
||||
**Problem:** `_parse_sse_json()` is duplicated verbatim in:
|
||||
- `backend/apps/tools_lib/tools_lib.py` (lines 773-787)
|
||||
- `backend/apps/agents/mcp_client.py` (lines 234-249)
|
||||
|
||||
Also, `_sanitize_server_name()` is defined in `tools_lib.py` (line 481) but imported into `agent_manager.py`.
|
||||
|
||||
**What to do:**
|
||||
|
||||
1. Create `backend/apps/common/mcp_utils.py` with:
|
||||
- `parse_sse_json(text: str) -> dict | None`
|
||||
- `sanitize_server_name(name: str) -> str`
|
||||
|
||||
2. Update `tools_lib.py` to import from `common.mcp_utils` instead of defining locally. Keep `_sanitize_server_name` as a backward-compatible alias: `_sanitize_server_name = sanitize_server_name`.
|
||||
|
||||
3. Update `mcp_client.py` to import `parse_sse_json` from `common.mcp_utils` instead of having its own `_parse_sse_json` static method.
|
||||
|
||||
4. Update `agent_manager.py` imports to use `from backend.apps.common.mcp_utils import sanitize_server_name`.
|
||||
|
||||
---
|
||||
|
||||
## Task 0D: Create `backend/apps/common/llm_helpers.py`
|
||||
|
||||
**Problem:** Multiple files independently construct Anthropic clients, call `messages.create`, strip markdown fences from responses, and parse JSON — with identical error handling. This happens in:
|
||||
|
||||
- `agent_manager.py` `generate_title()` (lines 1511-1539)
|
||||
- `agent_manager.py` `generate_group_meta()` (lines 1541-1617)
|
||||
- `dashboards.py` `generate_name()` (lines 133-190)
|
||||
- `outputs.py` `vibe_code()` (lines 351-419)
|
||||
- `outputs.py` `auto_run_output()` (lines 430-480)
|
||||
|
||||
**What to do:**
|
||||
|
||||
1. Create `backend/apps/common/llm_helpers.py` with:
|
||||
|
||||
```python
|
||||
async def quick_llm_call(
|
||||
system: str,
|
||||
user_content: str,
|
||||
model: str = "claude-sonnet-4-20250514",
|
||||
max_tokens: int = 300,
|
||||
) -> str:
|
||||
"""Make a simple LLM call and return the text response. Handles client construction."""
|
||||
|
||||
async def quick_llm_json(
|
||||
system: str,
|
||||
user_content: str,
|
||||
model: str = "claude-sonnet-4-20250514",
|
||||
max_tokens: int = 300,
|
||||
) -> dict:
|
||||
"""Make an LLM call expecting JSON. Strips markdown fences, parses JSON."""
|
||||
|
||||
def strip_markdown_fences(text: str) -> str:
|
||||
"""Remove ```json ... ``` or similar fences from LLM output."""
|
||||
```
|
||||
|
||||
2. These should use `get_anthropic_client` from `backend/apps/settings/credentials.py` internally.
|
||||
|
||||
3. **Do not** refactor the callers yet — that's Agent 2's job. Just create the helpers so they're available.
|
||||
|
||||
---
|
||||
|
||||
## Task 4A: Clean Up `modes/models.py`
|
||||
|
||||
**Problem:** `backend/apps/modes/models.py` (171 lines) has ~120 lines of `BUILTIN_MODES` data mixed in with Pydantic schema definitions.
|
||||
|
||||
**What to do:**
|
||||
|
||||
1. Create `backend/apps/modes/builtin.py` and move the `BUILTIN_MODES` list there.
|
||||
2. `modes/models.py` should only contain `Mode`, `ModeCreate`, `ModeUpdate` Pydantic models.
|
||||
3. Update `modes/modes.py` to import `BUILTIN_MODES` from `builtin.py` instead of `models.py`.
|
||||
4. Check for any other files that import `BUILTIN_MODES` from `models.py` and update them.
|
||||
|
||||
---
|
||||
|
||||
## Task 4B: DRY Up `outputs/models.py` Migration Validators
|
||||
|
||||
**Problem:** `backend/apps/outputs/models.py` (197 lines) has `_migrate_flat_fields` copy-pasted across 4 classes: `Output`, `OutputCreate`, `OutputUpdate`, `WorkspaceSeedRequest`.
|
||||
|
||||
**What to do:**
|
||||
|
||||
1. Extract a shared function at module level:
|
||||
|
||||
```python
|
||||
def _migrate_legacy_files(data: dict, allow_schema_json: bool = False) -> dict:
|
||||
"""Convert legacy frontend_code/backend_code fields into the files dict."""
|
||||
...
|
||||
```
|
||||
|
||||
2. Have each `@model_validator` call this shared function instead of duplicating the logic.
|
||||
|
||||
---
|
||||
|
||||
## Task 4C: Remove Dead Code
|
||||
|
||||
1. `backend/apps/tools_lib/tools_lib.py` lines 640-646: empty comment section `# OAuth2 flow for Google Workspace (and other OAuth providers)` followed by another empty comment `# MCP tool discovery`. Remove the stale comments.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
After all tasks are complete:
|
||||
|
||||
1. `cd /Users/haikdecie/Desktop/openswarm-ai/openswarm/backend`
|
||||
2. `python -c "from backend.apps.common.json_store import JsonStore; print('OK')"`
|
||||
3. `python -c "from backend.apps.common.model_registry import resolve_model_id; print(resolve_model_id('sonnet'))"`
|
||||
4. `python -c "from backend.apps.common.mcp_utils import parse_sse_json, sanitize_server_name; print('OK')"`
|
||||
5. `python -c "from backend.apps.common.llm_helpers import quick_llm_call; print('OK')"`
|
||||
6. Verify no existing imports are broken by running: `python -c "from backend.main import app; print('OK')"`
|
||||
@@ -0,0 +1,289 @@
|
||||
# Agent 2: Split Backend God Objects (Phase 1)
|
||||
|
||||
## Context
|
||||
|
||||
You are cleaning up the OpenSwarm codebase. This is agent 2 of 4. Agent 1 has already completed Phase 0 (foundations), creating these shared utilities that you should USE:
|
||||
|
||||
- `backend/apps/common/json_store.py` — Generic `JsonStore[T]` for JSON file CRUD
|
||||
- `backend/apps/common/model_registry.py` — Single source of truth for model definitions, `resolve_model_id()`, `calculate_cost()`, etc.
|
||||
- `backend/apps/common/mcp_utils.py` — `parse_sse_json()`, `sanitize_server_name()`
|
||||
- `backend/apps/common/llm_helpers.py` — `quick_llm_call()`, `quick_llm_json()`, `strip_markdown_fences()`
|
||||
|
||||
**Rules:**
|
||||
- Every file you create or modify must be <250 lines of code
|
||||
- Keep code DRY — use the Phase 0 utilities
|
||||
- Do NOT touch `9router/`, `debugger/`, or `frontend/`
|
||||
- Do NOT touch tests
|
||||
- Run the app after each major split to verify nothing is broken: `cd backend && python -m backend.main`
|
||||
|
||||
---
|
||||
|
||||
## Task 1A: Split `agent_manager.py` (2093 lines → ~6 files)
|
||||
|
||||
This is the most important task. `backend/apps/agents/agent_manager.py` is a 2093-line god object.
|
||||
|
||||
Read it carefully first. The `AgentManager` class has these distinct responsibilities that should be separated:
|
||||
|
||||
### New file: `backend/apps/agents/prompt_builder.py` (~200 lines)
|
||||
|
||||
Extract these methods from `AgentManager`:
|
||||
- `_resolve_mode()`
|
||||
- `_compose_system_prompt()`
|
||||
- `_build_connected_tools_context()`
|
||||
- `_build_outputs_context()`
|
||||
- `_build_browser_context()`
|
||||
- `_get_pre_selected_browser_ids()`
|
||||
- `_resolve_context_paths()`
|
||||
- `_build_dir_tree()`
|
||||
- `_resolve_forced_tools()`
|
||||
- `_resolve_attached_skills()`
|
||||
- `_build_prompt_content()`
|
||||
|
||||
Make these standalone functions (not methods) that take the data they need as parameters. For example:
|
||||
```python
|
||||
def compose_system_prompt(
|
||||
default_prompt: str | None, mode_prompt: str | None,
|
||||
session_prompt: str | None, connected_tools_ctx: str | None = None,
|
||||
outputs_ctx: str | None = None, browser_ctx: str | None = None,
|
||||
) -> str | None:
|
||||
```
|
||||
|
||||
### New file: `backend/apps/agents/mcp_builder.py` (~200 lines)
|
||||
|
||||
Extract:
|
||||
- `_build_mcp_servers()` — builds the mcp_servers dict for ClaudeAgentOptions
|
||||
- `_get_effective_policy()` — the permission policy resolver (currently a nested function inside `_run_agent_loop`)
|
||||
- `_get_denied_tool_names()`, `_get_all_known_tool_names()`, `_is_fully_denied()` — the module-level helper functions
|
||||
- `get_all_tool_names()` — the module-level function
|
||||
- `FULL_TOOLS` — the constant list
|
||||
- Logic for building `effective_allowed` and `effective_disallowed` tool lists (currently ~50 lines inside `_run_agent_loop`)
|
||||
- Logic for adding browser-agent and invoke-agent MCP servers
|
||||
|
||||
### New file: `backend/apps/agents/session_store.py` (~200 lines)
|
||||
|
||||
Extract session persistence and history. Use `JsonStore` from `backend/apps/common/json_store.py` where possible:
|
||||
- `_save_session()`, `_load_session_data()`, `_delete_session_file()`, `_load_all_session_data()`
|
||||
- `get_history()` method
|
||||
- `_build_search_text()` static method
|
||||
- `reconcile_on_startup()`
|
||||
- `persist_all_sessions()`
|
||||
- `restore_all_sessions()`
|
||||
- `get_browser_agent_children()`
|
||||
|
||||
### Refactor: `backend/apps/agents/agent_loop.py` (replace existing, ~250 lines)
|
||||
|
||||
The existing `agent_loop.py` (331 lines) is an older/unused file. Replace it with the extracted `_run_agent_loop` method from `agent_manager.py`.
|
||||
|
||||
Extract from `AgentManager`:
|
||||
- `_run_agent_loop()` — the main SDK query loop (lines 501-1128). This is the biggest single method.
|
||||
- The hook functions (`pre_tool_hook`, `post_tool_hook`, `can_use_tool`) that are currently nested inside `_run_agent_loop`
|
||||
- `_run_mock_agent()` — the development mock (lines 1172-1259)
|
||||
- `_stream_text()` and `_stream_tool_input()` — streaming helpers (lines 1130-1170)
|
||||
- `_fire_session_completed()` — analytics for completed sessions
|
||||
|
||||
Use `quick_llm_call` and `quick_llm_json` from `common/llm_helpers.py` for `generate_title()` and `generate_group_meta()`.
|
||||
|
||||
### Slim down: `backend/apps/agents/agent_manager.py` (~250 lines)
|
||||
|
||||
What remains in `AgentManager`:
|
||||
- `__init__()` — holds `self.sessions` dict and `self.tasks` dict
|
||||
- `launch_agent()`
|
||||
- `send_message()`
|
||||
- `stop_agent()`
|
||||
- `handle_approval()`
|
||||
- `edit_message()`
|
||||
- `switch_branch()`
|
||||
- `generate_title()` — refactor to use `quick_llm_call()`
|
||||
- `generate_group_meta()` — refactor to use `quick_llm_json()`
|
||||
- `update_session()`
|
||||
- `close_session()`
|
||||
- `delete_session()`
|
||||
- `resume_session()`
|
||||
- `duplicate_session()`
|
||||
- `invoke_agent()`
|
||||
- `get_all_sessions()`, `get_session()`
|
||||
|
||||
Each of these methods becomes a thin coordinator that calls into the extracted modules.
|
||||
|
||||
The `agent_manager = AgentManager()` singleton stays at the bottom of this file.
|
||||
|
||||
### Important Notes for this split:
|
||||
|
||||
- `_run_agent_loop` has deeply nested closures (`can_use_tool`, `pre_tool_hook`, `post_tool_hook`, `prompt_stream`). When extracting to `agent_loop.py`, you'll need to pass the session, ws_manager, and other dependencies as parameters to these functions.
|
||||
- The `duplicate_session` and `invoke_agent` methods have duplicated message-copying logic (~30 lines each). Extract a shared `_copy_session_messages()` helper into `session_store.py`.
|
||||
- Keep the `agent_manager` singleton import path unchanged: `from backend.apps.agents.agent_manager import agent_manager` must still work.
|
||||
|
||||
---
|
||||
|
||||
## Task 1B: Split `tools_lib.py` (1153 lines → 5-6 files)
|
||||
|
||||
Convert `backend/apps/tools_lib/` from a single file into a package.
|
||||
|
||||
### Step 1: Create the package structure
|
||||
|
||||
```
|
||||
backend/apps/tools_lib/
|
||||
├── __init__.py # SubApp instance + backward-compat imports
|
||||
├── routes.py # Tool CRUD endpoints (~120 lines)
|
||||
├── oauth.py # OAuth start, callback, disconnect, refresh (~200 lines)
|
||||
├── oauth_providers.py # OAuthProvider dataclass + OAUTH_PROVIDERS registry (~180 lines)
|
||||
├── mcp_config.py # derive_mcp_config, _resolve_command, _augmented_path, _extra_bin_dirs (~180 lines)
|
||||
├── mcp_discovery.py # discover_tools endpoint + stdio/HTTP/SSE discovery (~200 lines)
|
||||
├── classification.py # _SERVICE_RULES, _categorize_tool, _extract_service (~100 lines)
|
||||
└── models.py # Already exists, keep as-is
|
||||
```
|
||||
|
||||
### Step 2: What goes where
|
||||
|
||||
**`oauth_providers.py`** — Pure data, no dependencies:
|
||||
- `OAuthProvider` dataclass (lines 86-107)
|
||||
- `OAUTH_PROVIDERS` dict (lines 109-239)
|
||||
- `_resolve_oauth_provider()` helper (lines 242-248)
|
||||
|
||||
**`oauth.py`** — OAuth flow logic:
|
||||
- `_pending_oauth` and `_pending_pkce` state dicts (lines 251-252)
|
||||
- `oauth_callback()` endpoint (lines 319-435)
|
||||
- `oauth_start()` endpoint (lines 1062-1098)
|
||||
- `oauth_disconnect()` endpoint (lines 1036-1059)
|
||||
- `refresh_oauth_token()` function (lines 1102-1153)
|
||||
- `refresh_google_token` alias
|
||||
|
||||
**`mcp_config.py`** — MCP server config derivation:
|
||||
- `_extra_bin_dirs()` (lines 486-513)
|
||||
- `_resolve_command()` (lines 516-539)
|
||||
- `_augmented_path()` (lines 542-552)
|
||||
- `derive_mcp_config()` (lines 555-637)
|
||||
|
||||
**`mcp_discovery.py`** — MCP tool discovery:
|
||||
- `_discover_mcp_tools_http()` (lines 790-830)
|
||||
- `_discover_mcp_tools_sse()` (lines 833-856)
|
||||
- `_discover_mcp_tools_stdio()` (lines 859-941)
|
||||
- `discover_tools()` endpoint (lines 944-1033)
|
||||
|
||||
**`classification.py`** — Tool categorization (pure data + logic):
|
||||
- `_READ_PREFIXES`, `_WRITE_PREFIXES` (lines 649-651)
|
||||
- `_SERVICE_RULES` (lines 653-748)
|
||||
- `_categorize_tool()` (lines 751-760)
|
||||
- `_extract_service()` (lines 763-770)
|
||||
|
||||
**`routes.py`** — CRUD endpoints:
|
||||
- `list_builtin_tools()`, `list_tools()`, `get_tool()`, `create_tool()`, `update_tool()`, `delete_tool()` (lines 279-474)
|
||||
- `load_builtin_permissions()`, `save_builtin_permissions()`, permission endpoints (lines 284-311)
|
||||
- Use `JsonStore` from `backend/apps/common/json_store.py` for the `_load_all`, `_save`, `_load` functions
|
||||
|
||||
**`__init__.py`** — Glue:
|
||||
- `tools_lib` SubApp instance
|
||||
- `tools_lib_lifespan`
|
||||
- Backward-compatible imports so that `from backend.apps.tools_lib.tools_lib import _load_all, derive_mcp_config, ...` still works. Add a `tools_lib.py` shim file OR add these to `__init__.py`.
|
||||
- Wire up all routes from the sub-modules
|
||||
|
||||
### Critical: Maintain backward compatibility
|
||||
|
||||
These imports exist in `agent_manager.py` and must keep working:
|
||||
```python
|
||||
from backend.apps.tools_lib.tools_lib import (
|
||||
_load_all as load_all_tools,
|
||||
_sanitize_server_name,
|
||||
derive_mcp_config,
|
||||
load_builtin_permissions,
|
||||
refresh_google_token,
|
||||
)
|
||||
```
|
||||
|
||||
Keep a `tools_lib.py` file that re-exports from the new package modules, OR update all import sites.
|
||||
|
||||
---
|
||||
|
||||
## Task 1C: Split `outputs.py` (593 lines → 3 files)
|
||||
|
||||
### New file: `backend/apps/outputs/helpers.py` (~80 lines)
|
||||
|
||||
Extract:
|
||||
- `_build_data_injection()` (lines 53-70)
|
||||
- `_inject_data_into_html()` (lines 73-79)
|
||||
- `_decode_data_param()` (lines 82-90)
|
||||
- `_validate_against_schema()` (lines 41-48)
|
||||
- `_walk_directory()` (lines 136-150)
|
||||
|
||||
### New file: `backend/apps/outputs/ai_generation.py` (~200 lines)
|
||||
|
||||
Extract:
|
||||
- `VIBE_CODE_SYSTEM_PROMPT` (lines 334-348)
|
||||
- `vibe_code()` endpoint (lines 351-419) — refactor to use `quick_llm_json` from `common/llm_helpers.py`
|
||||
- `AUTO_RUN_SYSTEM_PROMPT` (lines 422-427)
|
||||
- `auto_run_output()` endpoint (lines 430-480) — refactor to use `quick_llm_call`
|
||||
- `AUTO_RUN_AGENT_SYSTEM_PROMPT` (lines 525-541)
|
||||
- `auto_run_agent()` endpoint (lines 544-581)
|
||||
- `cleanup_auto_run_agent()` endpoint (lines 584-593)
|
||||
- `_get_anthropic_client()` helper (lines 34-38) — or replace with direct `quick_llm_*` usage
|
||||
- `_resolve_model()` — replace with `resolve_model_id` from `common/model_registry.py`
|
||||
|
||||
### Slim down: `backend/apps/outputs/outputs.py` (~200 lines)
|
||||
|
||||
What remains:
|
||||
- SubApp instance + lifespan
|
||||
- CRUD endpoints (list, get, create, update, delete)
|
||||
- Workspace endpoints (read, seed, write file, delete file)
|
||||
- File serving endpoints (serve_workspace_file, serve_output_file)
|
||||
- `_load_all`, `_save`, `_load`, `load_output` — replace with `JsonStore`
|
||||
|
||||
Wire the ai_generation routes into the outputs router.
|
||||
|
||||
---
|
||||
|
||||
## Task 1D: Split `browser_agent.py` (632 lines → 3 files)
|
||||
|
||||
### New file: `backend/apps/agents/browser/schemas.py` (~100 lines)
|
||||
|
||||
Extract:
|
||||
- `BROWSER_TOOLS_SCHEMA` (lines 30-158)
|
||||
- `ACTION_MAP` (lines 160-170)
|
||||
- `SYSTEM_PROMPT` (lines 172-193)
|
||||
- `MAX_TURNS` constant (line 195)
|
||||
|
||||
Replace the local `MODEL_MAP` (lines 24-28) with `resolve_model_id` from `common/model_registry.py`.
|
||||
|
||||
### New file: `backend/apps/agents/browser/executor.py` (~120 lines)
|
||||
|
||||
Extract:
|
||||
- `execute_browser_tool()` (lines 198-211)
|
||||
- `_format_tool_result()` (lines 214-234)
|
||||
- `_request_browser_approval()` (lines 237-274)
|
||||
|
||||
### New file: `backend/apps/agents/browser/runner.py` (~250 lines)
|
||||
|
||||
Extract:
|
||||
- `run_browser_agent()` (lines 277-549)
|
||||
- `_create_browser_card()` (lines 552-580)
|
||||
- `run_browser_agents()` (lines 583-632)
|
||||
|
||||
Create `backend/apps/agents/browser/__init__.py` that re-exports the public API:
|
||||
```python
|
||||
from backend.apps.agents.browser.runner import run_browser_agent, run_browser_agents
|
||||
```
|
||||
|
||||
Update `backend/main.py` line 177 which imports `from backend.apps.agents.browser_agent import run_browser_agents`.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
After all tasks are complete:
|
||||
|
||||
1. Verify no file in `backend/` exceeds 250 lines:
|
||||
```bash
|
||||
find backend -name '*.py' -not -path '*/__pycache__/*' -not -path '*/test*' | xargs wc -l | sort -rn | head -20
|
||||
```
|
||||
|
||||
2. Verify the app starts:
|
||||
```bash
|
||||
cd backend && python -c "from backend.main import app; print('App created OK')"
|
||||
```
|
||||
|
||||
3. Verify key imports still work:
|
||||
```bash
|
||||
python -c "from backend.apps.agents.agent_manager import agent_manager; print('OK')"
|
||||
python -c "from backend.apps.tools_lib.tools_lib import _load_all, derive_mcp_config, load_builtin_permissions; print('OK')"
|
||||
python -c "from backend.apps.agents.browser_agent import run_browser_agents; print('OK')"
|
||||
python -c "from backend.apps.outputs.outputs import _load_all; print('OK')"
|
||||
```
|
||||
@@ -0,0 +1,253 @@
|
||||
# Agent 3: DRY Up Cross-Cutting Patterns (Phase 2)
|
||||
|
||||
## Context
|
||||
|
||||
You are cleaning up the OpenSwarm codebase. This is agent 3 of 4. Agents 1 and 2 have already completed:
|
||||
|
||||
- **Phase 0:** Shared utilities in `backend/apps/common/` (json_store, model_registry, mcp_utils, llm_helpers)
|
||||
- **Phase 1:** Split god objects — `agent_manager.py`, `tools_lib.py`, `outputs.py`, `browser_agent.py` are now modular packages/files
|
||||
|
||||
**Rules:**
|
||||
- Every file you create or modify must be <250 lines of code
|
||||
- Keep code DRY
|
||||
- Do NOT touch `9router/`, `debugger/`, or `frontend/`
|
||||
- Do NOT touch tests
|
||||
- Verify the app still starts after each major change
|
||||
|
||||
---
|
||||
|
||||
## Task 2A: Extract Shared Approval Flow
|
||||
|
||||
**Problem:** Two places implement independent HITL (human-in-the-loop) approval request patterns:
|
||||
|
||||
1. **`agent_manager.py`** (now possibly in `agent_loop.py` after Phase 1 split) — `_request_user_approval()` function:
|
||||
- Creates an `ApprovalRequest`
|
||||
- Appends to `session.pending_approvals`
|
||||
- Sets `session.status = "waiting_approval"`
|
||||
- Sends WS status event
|
||||
- Fires analytics event
|
||||
- Waits for decision via `ws_manager.send_approval_request()`
|
||||
- Fires another analytics event with latency
|
||||
- Removes from pending approvals
|
||||
- Restores `session.status = "running"`
|
||||
- Sends WS status event
|
||||
|
||||
2. **`browser_agent.py`** (now possibly in `browser/executor.py`) — `_request_browser_approval()`:
|
||||
- Same pattern but without analytics events and with a timeout
|
||||
|
||||
**What to do:**
|
||||
|
||||
1. Create `backend/apps/agents/approval.py` (~80 lines):
|
||||
|
||||
```python
|
||||
async def request_approval(
|
||||
session: AgentSession,
|
||||
tool_name: str,
|
||||
tool_input: dict,
|
||||
timeout: float | None = None,
|
||||
track_analytics: bool = True,
|
||||
) -> dict:
|
||||
"""Unified HITL approval flow.
|
||||
|
||||
Creates an ApprovalRequest, sends it via WebSocket, waits for the user's
|
||||
decision, cleans up, and returns the decision dict.
|
||||
|
||||
Returns: {"behavior": "allow"|"deny", "message": ..., "updated_input": ...}
|
||||
"""
|
||||
```
|
||||
|
||||
2. Replace both implementations with calls to this shared function.
|
||||
|
||||
3. Make sure the analytics tracking is optional (browser agents don't currently track approval analytics).
|
||||
|
||||
---
|
||||
|
||||
## Task 2B: Extract WebSocket Event Helpers
|
||||
|
||||
**Problem:** Throughout the codebase (especially `agent_manager.py` / `agent_loop.py` / `browser_agent.py`), there are 30+ calls that look like:
|
||||
|
||||
```python
|
||||
await ws_manager.send_to_session(session_id, "agent:status", {
|
||||
"session_id": session_id,
|
||||
"status": "running",
|
||||
"session": session.model_dump(mode="json"),
|
||||
})
|
||||
```
|
||||
|
||||
The payload construction is repeated identically every time for each event type.
|
||||
|
||||
**What to do:**
|
||||
|
||||
Add typed convenience methods to `backend/apps/agents/ws_manager.py`. The file is currently 126 lines so there's room:
|
||||
|
||||
```python
|
||||
async def emit_status(self, session_id: str, status: str, session: AgentSession):
|
||||
await self.send_to_session(session_id, "agent:status", {
|
||||
"session_id": session_id,
|
||||
"status": status,
|
||||
"session": session.model_dump(mode="json"),
|
||||
})
|
||||
|
||||
async def emit_message(self, session_id: str, message: Message):
|
||||
await self.send_to_session(session_id, "agent:message", {
|
||||
"session_id": session_id,
|
||||
"message": message.model_dump(mode="json"),
|
||||
})
|
||||
|
||||
async def emit_cost_update(self, session_id: str, cost_usd: float):
|
||||
await self.send_to_session(session_id, "agent:cost_update", {
|
||||
"session_id": session_id,
|
||||
"cost_usd": cost_usd,
|
||||
})
|
||||
|
||||
async def emit_stream_start(self, session_id: str, message_id: str, role: str, tool_name: str = ""):
|
||||
payload = {"session_id": session_id, "message_id": message_id, "role": role}
|
||||
if tool_name:
|
||||
payload["tool_name"] = tool_name
|
||||
await self.send_to_session(session_id, "agent:stream_start", payload)
|
||||
|
||||
async def emit_stream_delta(self, session_id: str, message_id: str, delta: str):
|
||||
await self.send_to_session(session_id, "agent:stream_delta", {
|
||||
"session_id": session_id,
|
||||
"message_id": message_id,
|
||||
"delta": delta,
|
||||
})
|
||||
|
||||
async def emit_stream_end(self, session_id: str, message_id: str):
|
||||
await self.send_to_session(session_id, "agent:stream_end", {
|
||||
"session_id": session_id,
|
||||
"message_id": message_id,
|
||||
})
|
||||
```
|
||||
|
||||
Then do a find-and-replace across all files that construct these payloads manually. Replace with the typed helpers. This is a mechanical change — just make sure every event type is covered.
|
||||
|
||||
**Important:** If `ws_manager.py` exceeds 250 lines after adding these helpers, split it into `ws_manager.py` (connection management) and `ws_events.py` (typed event emitters).
|
||||
|
||||
---
|
||||
|
||||
## Task 2C: Move Subscription Routes to New Sub-App
|
||||
|
||||
**Problem:** `backend/apps/agents/agents.py` (305 lines) contains ~120 lines of 9Router/subscription endpoints (lines 186-305) that have nothing to do with agent sessions:
|
||||
- `subscriptions_status()`
|
||||
- `subscriptions_connect()`
|
||||
- `subscriptions_poll()`
|
||||
- `subscriptions_exchange()`
|
||||
- `subscriptions_models()`
|
||||
- `subscriptions_disconnect()`
|
||||
|
||||
**What to do:**
|
||||
|
||||
1. Create `backend/apps/subscriptions/__init__.py` (empty)
|
||||
2. Create `backend/apps/subscriptions/subscriptions.py` (~130 lines):
|
||||
- Move all 6 subscription endpoints here
|
||||
- Create a new SubApp: `subscriptions = SubApp("subscriptions", subscriptions_lifespan)`
|
||||
- The lifespan can be a simple no-op (or move the 9Router auto-start from analytics lifespan here if it makes more sense)
|
||||
|
||||
3. Update `backend/main.py`:
|
||||
- Import the new `subscriptions` sub-app
|
||||
- Add it to the `MainApp` list
|
||||
|
||||
4. Remove the subscription endpoints from `agents.py`. This should bring `agents.py` down to ~185 lines.
|
||||
|
||||
5. **Frontend impact:** The frontend calls these endpoints at `/api/agents/subscriptions/*`. The new path will be `/api/subscriptions/*`. Search the frontend for these API paths and update them:
|
||||
```bash
|
||||
grep -r "api/agents/subscriptions" frontend/src/
|
||||
```
|
||||
Update all matches to use `/api/subscriptions/` instead.
|
||||
|
||||
---
|
||||
|
||||
## Task 2D: Clean Up `main.py`
|
||||
|
||||
**Problem:** `backend/main.py` (256 lines) has inline WebSocket handlers, OAuth callback endpoints, browser-agent HTTP endpoint, and invoke-agent HTTP endpoint that should live in their respective sub-apps.
|
||||
|
||||
**What to do:**
|
||||
|
||||
### Move WebSocket handlers
|
||||
|
||||
The two WebSocket handlers (`websocket_session` and `websocket_dashboard`, lines 41-104) can't easily move to a SubApp router because FastAPI WebSocket routes need to be on the main app. However, the message dispatch logic inside them can be extracted.
|
||||
|
||||
Create `backend/apps/agents/ws_routes.py` (~80 lines):
|
||||
|
||||
```python
|
||||
async def handle_session_message(session_id: str, event: str, payload: dict):
|
||||
"""Dispatch a WebSocket message for a session."""
|
||||
if event == "agent:send_message":
|
||||
from backend.apps.agents.agent_manager import agent_manager
|
||||
await agent_manager.send_message(...)
|
||||
elif event == "agent:approval_response":
|
||||
...
|
||||
elif event == "agent:edit_message":
|
||||
...
|
||||
elif event == "agent:stop":
|
||||
...
|
||||
|
||||
async def handle_dashboard_message(event: str, payload: dict):
|
||||
"""Dispatch a WebSocket message for the dashboard."""
|
||||
...
|
||||
```
|
||||
|
||||
Then `main.py`'s WebSocket handlers become thin wrappers:
|
||||
```python
|
||||
@app.websocket("/ws/agents/{session_id}")
|
||||
async def websocket_session(websocket: WebSocket, session_id: str):
|
||||
await ws_manager.connect_session(session_id, websocket)
|
||||
try:
|
||||
while True:
|
||||
data = await websocket.receive_text()
|
||||
msg = json.loads(data)
|
||||
await handle_session_message(session_id, msg.get("event"), msg.get("data", {}))
|
||||
except WebSocketDisconnect:
|
||||
ws_manager.disconnect_session(session_id, websocket)
|
||||
```
|
||||
|
||||
### Move HTTP endpoints
|
||||
|
||||
- `/api/browser/command` (lines 107-122) → Move to `agents/browser/` as a route on the agents router, or keep in main.py if it needs to be at the root level
|
||||
- `/api/browser-agent/run` (lines 172-196) → Move to agents router
|
||||
- `/api/invoke-agent/run` (lines 199-227) → Move to agents router
|
||||
- `/api/subscriptions/pending/{state}` (lines 125-136) → Move to subscriptions sub-app
|
||||
- `/api/subscriptions/callback` (lines 139-169) → Move to subscriptions sub-app (or tools_lib OAuth if it's OAuth-related)
|
||||
|
||||
### Target state for `main.py` (~80 lines):
|
||||
|
||||
```python
|
||||
# Imports
|
||||
# Create MainApp with all sub-apps
|
||||
# CORS middleware
|
||||
# WebSocket routes (thin wrappers)
|
||||
# if __name__ == "__main__": uvicorn
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
After all tasks are complete:
|
||||
|
||||
1. Verify no file in `backend/` exceeds 250 lines:
|
||||
```bash
|
||||
find backend -name '*.py' -not -path '*/__pycache__/*' -not -path '*/test*' | xargs wc -l | sort -rn | head -20
|
||||
```
|
||||
|
||||
2. Verify the app starts:
|
||||
```bash
|
||||
cd backend && python -c "from backend.main import app; print('App created OK')"
|
||||
```
|
||||
|
||||
3. Verify the new subscriptions sub-app is registered:
|
||||
```bash
|
||||
python -c "from backend.main import app; routes = [r.path for r in app.routes]; print([r for r in routes if 'subscription' in r])"
|
||||
```
|
||||
|
||||
4. Check that no frontend API calls are broken by searching for old paths:
|
||||
```bash
|
||||
grep -r "api/agents/subscriptions" frontend/src/
|
||||
```
|
||||
This should return no results (all updated to `/api/subscriptions/`).
|
||||
|
||||
5. Verify `main.py` is under 250 lines:
|
||||
```bash
|
||||
wc -l backend/main.py
|
||||
```
|
||||
@@ -0,0 +1,340 @@
|
||||
# Agent 4: Frontend Cleanup (Phase 3)
|
||||
|
||||
## Context
|
||||
|
||||
You are cleaning up the OpenSwarm frontend codebase. This is agent 4 of 4. You work **independently** from the backend agents — no backend changes are required for your work.
|
||||
|
||||
The frontend is a React + TypeScript app in `frontend/src/`. It uses Redux Toolkit for state management and WebSockets for real-time communication with the backend.
|
||||
|
||||
**Rules:**
|
||||
- Every file you create or modify must be <250 lines of code
|
||||
- Keep code DRY
|
||||
- Do NOT touch `backend/`, `9router/`, `debugger/`, or `electron/`
|
||||
- Do NOT touch tests
|
||||
- Preserve all existing functionality — this is purely structural refactoring
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
There are 40 TypeScript/TSX files over 250 lines. Here are the 13 worst offenders that need splitting:
|
||||
|
||||
| File | Lines | Priority |
|
||||
|------|-------|----------|
|
||||
| `pages/Tools/Tools.tsx` | 2480 | P0 |
|
||||
| `pages/AgentChat/ToolCallBubble.tsx` | 2182 | P0 |
|
||||
| `pages/Dashboard/Dashboard.tsx` | 1594 | P0 |
|
||||
| `pages/Views/ViewEditor.tsx` | 1591 | P0 |
|
||||
| `pages/Settings/Settings.tsx` | 1567 | P0 |
|
||||
| `pages/AgentChat/ChatInput.tsx` | 1278 | P1 |
|
||||
| `pages/Dashboard/BrowserCard.tsx` | 1259 | P1 |
|
||||
| `pages/AgentChat/ApprovalBar.tsx` | 1160 | P1 |
|
||||
| `pages/AgentChat/AgentChat.tsx` | 1137 | P1 |
|
||||
| `pages/Dashboard/AgentCard.tsx` | 1063 | P1 |
|
||||
| `shared/state/agentsSlice.ts` | 1030 | P1 |
|
||||
| `components/Layout/AppShell.tsx` | 1005 | P2 |
|
||||
| `components/DynamicIsland.tsx` | 993 | P2 |
|
||||
|
||||
All paths are relative to `frontend/src/app/` unless noted.
|
||||
|
||||
---
|
||||
|
||||
## General Strategy
|
||||
|
||||
For each oversized component:
|
||||
|
||||
1. **Read the file** to understand its structure — identify logical sections, sub-components rendered inline, state management, event handlers, and utility functions.
|
||||
|
||||
2. **Extract sub-components** into sibling files in the same directory. Each extracted component should:
|
||||
- Be in its own file
|
||||
- Accept props for data and callbacks
|
||||
- Be <250 lines
|
||||
|
||||
3. **Extract custom hooks** for complex stateful logic (e.g., `useDashboardDragDrop`, `useAgentChat`, `useChatSubmit`).
|
||||
|
||||
4. **Extract utility functions** and constants into separate files.
|
||||
|
||||
5. **The parent component** becomes a thin orchestrator that imports and composes the pieces.
|
||||
|
||||
---
|
||||
|
||||
## P0: Critical Splits (do these first)
|
||||
|
||||
### Tools.tsx (2480 lines)
|
||||
|
||||
Read the file first. It likely contains:
|
||||
- Tool list/grid view
|
||||
- Individual tool cards with config panels
|
||||
- OAuth connection flows
|
||||
- MCP tool discovery UI
|
||||
- Permission editors per tool
|
||||
- Builtin tool permission toggles
|
||||
|
||||
**Split into:**
|
||||
|
||||
```
|
||||
pages/Tools/
|
||||
├── Tools.tsx # Main page: layout, tool list, state (~200)
|
||||
├── ToolCard.tsx # Individual tool card (~200)
|
||||
├── ToolConfigPanel.tsx # Config/edit panel for a tool (~200)
|
||||
├── OAuthConnectFlow.tsx # OAuth button + status display (~150)
|
||||
├── ToolPermissions.tsx # Per-tool permission toggles (~200)
|
||||
├── McpDiscoveryPanel.tsx # MCP tool discovery results (~150)
|
||||
├── BuiltinToolsList.tsx # Builtin tools section with permissions (~200)
|
||||
└── hooks/
|
||||
└── useToolsState.ts # Tool loading, CRUD operations, OAuth state (~150)
|
||||
```
|
||||
|
||||
### ToolCallBubble.tsx (2182 lines)
|
||||
|
||||
This renders tool call + result bubbles in the chat. It likely has different renderers for different tool types.
|
||||
|
||||
**Split into:**
|
||||
|
||||
```
|
||||
pages/AgentChat/
|
||||
├── ToolCallBubble.tsx # Router component: picks renderer by tool type (~100)
|
||||
├── toolRenderers/
|
||||
│ ├── DefaultToolRenderer.tsx # Generic tool call display (~150)
|
||||
│ ├── ReadToolRenderer.tsx # File content display with line numbers (~150)
|
||||
│ ├── EditToolRenderer.tsx # Diff view for edits (~150)
|
||||
│ ├── BashToolRenderer.tsx # Terminal-style output (~150)
|
||||
│ ├── SearchToolRenderer.tsx # Grep/Glob results (~100)
|
||||
│ └── McpToolRenderer.tsx # MCP tool results (~100)
|
||||
├── ToolInputDisplay.tsx # Formatted tool input JSON (~100)
|
||||
└── ToolResultDisplay.tsx # Formatted tool result content (~100)
|
||||
```
|
||||
|
||||
### Dashboard.tsx (1594 lines)
|
||||
|
||||
**Split into:**
|
||||
|
||||
```
|
||||
pages/Dashboard/
|
||||
├── Dashboard.tsx # Main canvas + layout orchestration (~200)
|
||||
├── DashboardCanvas.tsx # The infinite canvas / drag-drop surface (~200)
|
||||
├── CardRenderer.tsx # Routes card type to correct component (~80)
|
||||
├── hooks/
|
||||
│ ├── useDashboardDragDrop.ts # Drag, drop, resize logic (~200)
|
||||
│ └── useDashboardState.ts # Dashboard loading, saving (~150)
|
||||
```
|
||||
|
||||
Keep existing `AgentCard.tsx`, `BrowserCard.tsx`, `DashboardViewCard.tsx` as-is (they'll be split separately).
|
||||
|
||||
### ViewEditor.tsx (1591 lines)
|
||||
|
||||
**Split into:**
|
||||
|
||||
```
|
||||
pages/Views/
|
||||
├── ViewEditor.tsx # Main editor layout + state (~200)
|
||||
├── EditorToolbar.tsx # Top toolbar (save, run, vibe-code button) (~100)
|
||||
├── CodeEditorPanel.tsx # Code editor (Monaco or CodeMirror wrapper) (~150)
|
||||
├── PreviewPane.tsx # Iframe preview with hot reload (~150)
|
||||
├── SchemaEditorPanel.tsx # JSON schema editor for input_schema (~200)
|
||||
├── FileTreePanel.tsx # Multi-file tree sidebar (~150)
|
||||
├── BackendCodePanel.tsx # Backend Python code editor (~100)
|
||||
└── hooks/
|
||||
└── useViewEditor.ts # Editor state, save, auto-run logic (~200)
|
||||
```
|
||||
|
||||
### Settings.tsx (1567 lines)
|
||||
|
||||
**Split into:**
|
||||
|
||||
```
|
||||
pages/Settings/
|
||||
├── Settings.tsx # Main settings page with tabs/sections (~150)
|
||||
├── ProviderSettings.tsx # API key inputs for each provider (~200)
|
||||
├── SubscriptionSection.tsx # 9Router subscription management (~200)
|
||||
├── SystemPromptEditor.tsx # Default system prompt editor (~150)
|
||||
├── GeneralSettings.tsx # Default folder, theme, etc. (~150)
|
||||
├── CustomProviderEditor.tsx # Custom OpenAI-compat provider form (~200)
|
||||
├── AnalyticsOptInSection.tsx # Analytics toggle + info (~80)
|
||||
└── hooks/
|
||||
└── useSettings.ts # Settings load/save logic (~100)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## P1: Important Splits
|
||||
|
||||
### ChatInput.tsx (1278 lines)
|
||||
|
||||
**Split into:**
|
||||
|
||||
```
|
||||
pages/AgentChat/
|
||||
├── ChatInput.tsx # Main input container (~200)
|
||||
├── AttachmentBar.tsx # File/image attachment display (~100)
|
||||
├── ModeSelector.tsx # Mode dropdown (Agent/Ask/Plan/etc) (~100)
|
||||
├── ModelPicker.tsx # Model selection dropdown (~100)
|
||||
├── ContextAttachments.tsx # Context path + skill attachment UI (~150)
|
||||
└── hooks/
|
||||
└── useChatSubmit.ts # Submit logic, validation, WS send (~150)
|
||||
```
|
||||
|
||||
### BrowserCard.tsx (1259 lines)
|
||||
|
||||
**Split into:**
|
||||
|
||||
```
|
||||
pages/Dashboard/
|
||||
├── BrowserCard.tsx # Main browser card (~200)
|
||||
├── BrowserToolbar.tsx # URL bar, navigation buttons (~150)
|
||||
├── BrowserTabBar.tsx # Tab management strip (~100)
|
||||
├── BrowserViewport.tsx # The webview/iframe wrapper (~200)
|
||||
└── BrowserContextMenu.tsx # Right-click context menu (~100)
|
||||
```
|
||||
|
||||
### ApprovalBar.tsx (1160 lines)
|
||||
|
||||
**Split into:**
|
||||
|
||||
```
|
||||
pages/AgentChat/
|
||||
├── ApprovalBar.tsx # Main approval container + queue (~150)
|
||||
├── ApprovalCard.tsx # Individual approval request card (~200)
|
||||
├── ToolInputEditor.tsx # Editable tool input JSON viewer (~200)
|
||||
└── ApprovalActions.tsx # Allow/Deny/Edit buttons + logic (~100)
|
||||
```
|
||||
|
||||
### AgentChat.tsx (1137 lines)
|
||||
|
||||
**Split into:**
|
||||
|
||||
```
|
||||
pages/AgentChat/
|
||||
├── AgentChat.tsx # Main chat page layout (~200)
|
||||
├── MessageList.tsx # Scrollable message list (~200)
|
||||
├── ChatHeader.tsx # Session name, model, status bar (~100)
|
||||
└── hooks/
|
||||
└── useAgentChat.ts # Chat state, message handling, branch nav (~200)
|
||||
```
|
||||
|
||||
### AgentCard.tsx (1063 lines)
|
||||
|
||||
**Split into:**
|
||||
|
||||
```
|
||||
pages/Dashboard/
|
||||
├── AgentCard.tsx # Main agent card (~200)
|
||||
├── AgentCardHeader.tsx # Name, status badge, model tag (~100)
|
||||
├── AgentCardMessages.tsx # Compact message list in card (~200)
|
||||
├── AgentCardActions.tsx # Stop, close, resume, duplicate buttons (~100)
|
||||
└── AgentCardToolGroup.tsx # Collapsed tool call groups (~150)
|
||||
```
|
||||
|
||||
### agentsSlice.ts (1030 lines)
|
||||
|
||||
**Split into:**
|
||||
|
||||
```
|
||||
shared/state/
|
||||
├── agentsSlice.ts # Core session state: CRUD, status (~200)
|
||||
├── agentMessagesSlice.ts # Message handling: add, edit, branch (~200)
|
||||
├── agentStreamSlice.ts # Streaming state: deltas, stream start/end (~150)
|
||||
└── agentWebSocket.ts # WebSocket message dispatch + handlers (~200)
|
||||
```
|
||||
|
||||
Or if Redux Toolkit makes splitting slices difficult, at minimum extract the WebSocket handler logic and message processing into separate files, keeping a single slice that imports helper functions.
|
||||
|
||||
---
|
||||
|
||||
## P2: Lower Priority Splits
|
||||
|
||||
### AppShell.tsx (1005 lines)
|
||||
|
||||
**Split into:**
|
||||
|
||||
```
|
||||
components/Layout/
|
||||
├── AppShell.tsx # Main shell: sidebar + content area (~150)
|
||||
├── Sidebar.tsx # Navigation sidebar (~200)
|
||||
├── NavigationRail.tsx # Icon rail for collapsed sidebar (~100)
|
||||
├── PageRouter.tsx # Route → page component mapping (~100)
|
||||
└── hooks/
|
||||
└── useNavigation.ts # Route state, sidebar collapse (~100)
|
||||
```
|
||||
|
||||
### DynamicIsland.tsx (993 lines)
|
||||
|
||||
**Split into:**
|
||||
|
||||
```
|
||||
components/
|
||||
├── DynamicIsland.tsx # Container + animation (~150)
|
||||
├── IslandQuickActions.tsx # Quick action buttons/chips (~150)
|
||||
├── IslandAgentStatus.tsx # Active agent status summary (~150)
|
||||
├── IslandNotifications.tsx # Notification toasts (~100)
|
||||
└── hooks/
|
||||
└── useIslandState.ts # Island expand/collapse, content logic (~150)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## P3: Remaining Files Over 250 Lines
|
||||
|
||||
After completing P0-P2, check what's still over 250 lines:
|
||||
|
||||
```bash
|
||||
find frontend/src -type f \( -name '*.ts' -o -name '*.tsx' \) | xargs wc -l | sort -rn | awk '$1 > 250 {print}'
|
||||
```
|
||||
|
||||
For each remaining file, apply the same decomposition strategy:
|
||||
1. Extract sub-components
|
||||
2. Extract hooks for complex logic
|
||||
3. Extract constants/utils
|
||||
|
||||
Target: **zero files over 250 lines**.
|
||||
|
||||
---
|
||||
|
||||
## Shared Frontend Utilities (if time permits)
|
||||
|
||||
### Typed API Client
|
||||
|
||||
If you notice scattered `fetch()` calls with duplicated base URL construction and error handling, consider extracting into `shared/api.ts`:
|
||||
|
||||
```typescript
|
||||
const api = {
|
||||
get: <T>(path: string) => Promise<T>,
|
||||
post: <T>(path: string, body?: any) => Promise<T>,
|
||||
put: <T>(path: string, body?: any) => Promise<T>,
|
||||
delete: (path: string) => Promise<void>,
|
||||
}
|
||||
```
|
||||
|
||||
### Common UI Patterns
|
||||
|
||||
If you see repeated patterns (confirmation dialogs, loading spinners, error states), extract them into `shared/components/`.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
After all splits are complete:
|
||||
|
||||
1. Verify no file exceeds 250 lines:
|
||||
```bash
|
||||
find frontend/src -type f \( -name '*.ts' -o -name '*.tsx' \) | xargs wc -l | sort -rn | awk '$1 > 250'
|
||||
```
|
||||
This should return only the `total` line.
|
||||
|
||||
2. Verify the frontend builds without errors:
|
||||
```bash
|
||||
cd frontend && npm run build
|
||||
```
|
||||
(Or `npx webpack --mode production` — check `package.json` for the build command)
|
||||
|
||||
3. Verify no TypeScript errors:
|
||||
```bash
|
||||
cd frontend && npx tsc --noEmit
|
||||
```
|
||||
|
||||
4. Spot-check that key pages render correctly by starting the dev server and navigating to:
|
||||
- Dashboard page
|
||||
- Agent chat page
|
||||
- Tools page
|
||||
- Settings page
|
||||
- Views/Editor page
|
||||
@@ -15,14 +15,15 @@ from backend.apps.agents.ws_manager import ws_manager
|
||||
from backend.apps.modes.modes import load_mode
|
||||
from backend.apps.outputs.outputs import _load_all as load_all_outputs
|
||||
from backend.apps.settings.settings import load_settings
|
||||
from backend.apps.common.mcp_utils import sanitize_server_name as _sanitize_server_name
|
||||
from backend.apps.tools_lib.tools_lib import (
|
||||
_load_all as load_all_tools,
|
||||
_sanitize_server_name,
|
||||
derive_mcp_config,
|
||||
load_builtin_permissions,
|
||||
refresh_google_token,
|
||||
)
|
||||
from backend.config.paths import SESSIONS_DIR
|
||||
from backend.apps.common.json_store import SessionStore
|
||||
from backend.apps.analytics.collector import record as _analytics
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -30,35 +31,12 @@ logger = logging.getLogger(__name__)
|
||||
os.environ.setdefault("CLAUDE_CODE_STREAM_CLOSE_TIMEOUT", "3600000")
|
||||
|
||||
|
||||
def _save_session(session_id: str, doc_data: dict):
|
||||
os.makedirs(SESSIONS_DIR, exist_ok=True)
|
||||
with open(os.path.join(SESSIONS_DIR, f"{session_id}.json"), "w") as f:
|
||||
json.dump(doc_data, f, indent=2)
|
||||
_session_store = SessionStore(SESSIONS_DIR)
|
||||
|
||||
|
||||
def _load_session_data(session_id: str) -> dict | None:
|
||||
path = os.path.join(SESSIONS_DIR, f"{session_id}.json")
|
||||
if not os.path.exists(path):
|
||||
return None
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def _delete_session_file(session_id: str):
|
||||
path = os.path.join(SESSIONS_DIR, f"{session_id}.json")
|
||||
if os.path.exists(path):
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def _load_all_session_data() -> list[tuple[str, dict]]:
|
||||
results = []
|
||||
if not os.path.exists(SESSIONS_DIR):
|
||||
return results
|
||||
for fname in os.listdir(SESSIONS_DIR):
|
||||
if fname.endswith(".json"):
|
||||
with open(os.path.join(SESSIONS_DIR, fname)) as f:
|
||||
results.append((fname[:-5], json.load(f)))
|
||||
return results
|
||||
_save_session = _session_store.save
|
||||
_load_session_data = _session_store.load
|
||||
_delete_session_file = _session_store.delete
|
||||
_load_all_session_data = _session_store.load_all
|
||||
|
||||
FULL_TOOLS = [
|
||||
"Read", "Edit", "Write", "Bash", "Glob", "Grep", "AskUserQuestion",
|
||||
|
||||
@@ -17,16 +17,11 @@ import anthropic
|
||||
|
||||
from backend.apps.agents.models import AgentSession, ApprovalRequest, Message
|
||||
from backend.apps.agents.ws_manager import ws_manager
|
||||
from backend.apps.common.model_registry import resolve_model_id
|
||||
from backend.apps.tools_lib.tools_lib import load_builtin_permissions
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
MODEL_MAP = {
|
||||
"sonnet": "claude-sonnet-4-6",
|
||||
"opus": "claude-opus-4-6",
|
||||
"haiku": "claude-haiku-4-5-20251001",
|
||||
}
|
||||
|
||||
BROWSER_TOOLS_SCHEMA = [
|
||||
{
|
||||
"name": "BrowserScreenshot",
|
||||
@@ -321,7 +316,7 @@ async def run_browser_agent(
|
||||
)
|
||||
logger.info(f"Browser agent {session_id}: navigated to {initial_url}: {nav_result.get('text', nav_result.get('error', ''))}")
|
||||
|
||||
api_model = MODEL_MAP.get(model, model)
|
||||
api_model = resolve_model_id(model)
|
||||
from backend.apps.settings.settings import load_settings
|
||||
from backend.apps.settings.credentials import get_anthropic_client
|
||||
client = get_anthropic_client(load_settings())
|
||||
|
||||
@@ -15,6 +15,7 @@ from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from backend.apps.agents.providers.base import ToolSchema
|
||||
from backend.apps.common.mcp_utils import parse_sse_json as _parse_sse_json
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -231,22 +232,7 @@ class MCPClientManager:
|
||||
conn._next_id = 3 # type: ignore[attr-defined]
|
||||
return conn
|
||||
|
||||
@staticmethod
|
||||
def _parse_sse_json(text: str) -> dict | None:
|
||||
"""Extract JSON from an SSE response body."""
|
||||
for line in text.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("data:"):
|
||||
payload = stripped[len("data:"):].strip()
|
||||
if payload:
|
||||
try:
|
||||
return json.loads(payload)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
try:
|
||||
return json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
_parse_sse_json = staticmethod(_parse_sse_json)
|
||||
|
||||
async def call_tool(
|
||||
self, server_name: str, tool_name: str, arguments: dict,
|
||||
|
||||
@@ -12,23 +12,18 @@ import logging
|
||||
from typing import Any, TYPE_CHECKING
|
||||
|
||||
from backend.apps.agents.providers.base import BaseProvider
|
||||
from backend.apps.common.model_registry import (
|
||||
get_builtin_models_by_provider,
|
||||
get_context_window as _registry_get_context_window,
|
||||
calculate_cost as _registry_calculate_cost,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from backend.apps.settings.models import AppSettings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tier 1: Built-in models (curated, we know their quirks)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
BUILTIN_MODELS: dict[str, list[dict[str, Any]]] = {
|
||||
"Anthropic": [
|
||||
{"value": "sonnet", "label": "Claude Sonnet 4.6", "context_window": 1_000_000, "model_id": "claude-sonnet-4-6", "api": "anthropic"},
|
||||
{"value": "opus", "label": "Claude Opus 4.6", "context_window": 1_000_000, "model_id": "claude-opus-4-6", "api": "anthropic"},
|
||||
{"value": "haiku", "label": "Claude Haiku 4.5", "context_window": 200_000, "model_id": "claude-haiku-4-5", "api": "anthropic"},
|
||||
],
|
||||
}
|
||||
BUILTIN_MODELS = get_builtin_models_by_provider()
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# OpenRouter: built-in integration for 300+ models
|
||||
@@ -261,52 +256,17 @@ def get_available_models(settings: AppSettings) -> dict[str, list[dict]]:
|
||||
|
||||
def get_context_window(provider: str, model: str, settings: AppSettings | None = None) -> int:
|
||||
"""Look up context window for any model."""
|
||||
# Check built-in models first
|
||||
for models in BUILTIN_MODELS.values():
|
||||
for m in models:
|
||||
if m["value"] == model:
|
||||
return m.get("context_window", 128_000)
|
||||
result = _registry_get_context_window(model)
|
||||
if result != 128_000:
|
||||
return result
|
||||
|
||||
# Check custom providers
|
||||
if settings:
|
||||
for cp in getattr(settings, "custom_providers", []):
|
||||
for m in cp.models:
|
||||
if m.get("value") == model or m.get("id") == model:
|
||||
return m.get("context_window", 128_000)
|
||||
|
||||
return 128_000 # safe default
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cost tracking
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
COST_PER_1M_TOKENS: dict[tuple[str, str], tuple[float, float]] = {
|
||||
# (provider, model): (input_cost_per_1M, output_cost_per_1M)
|
||||
# Anthropic
|
||||
("Anthropic", "sonnet"): (3.0, 15.0),
|
||||
("Anthropic", "opus"): (5.0, 25.0),
|
||||
("Anthropic", "haiku"): (1.0, 5.0),
|
||||
# OpenAI
|
||||
("OpenAI", "gpt-5.4"): (2.50, 15.0),
|
||||
("OpenAI", "gpt-5.4-mini"): (0.75, 3.0),
|
||||
("OpenAI", "o3"): (2.0, 8.0),
|
||||
("OpenAI", "o4-mini"): (1.10, 4.40),
|
||||
# Google
|
||||
("Google", "gemini-2.5-flash"): (0.15, 0.60),
|
||||
("Google", "gemini-2.5-pro"): (1.25, 10.0),
|
||||
# OpenRouter-backed (approximate)
|
||||
("xAI", "x-ai/grok-4-0214"): (3.0, 15.0),
|
||||
("Meta", "meta-llama/llama-4-maverick"): (0.50, 0.70),
|
||||
("Meta", "meta-llama/llama-4-scout"): (0.15, 0.40),
|
||||
("DeepSeek", "deepseek/deepseek-chat-v3-0324"): (0.30, 0.90),
|
||||
("DeepSeek", "deepseek/deepseek-r1"): (0.80, 2.40),
|
||||
("Mistral", "mistralai/mistral-large-2501"): (2.0, 6.0),
|
||||
("Mistral", "mistralai/mistral-small-3.1-24b-instruct"): (0.10, 0.30),
|
||||
("Qwen", "qwen/qwen3-coder"): (0.0, 0.0),
|
||||
("Qwen", "qwen/qwen3-235b-a22b"): (0.20, 0.70),
|
||||
("Cohere", "cohere/command-a-03-2025"): (2.50, 10.0),
|
||||
}
|
||||
return 128_000
|
||||
|
||||
|
||||
def calculate_cost(
|
||||
@@ -314,16 +274,4 @@ def calculate_cost(
|
||||
input_tokens: int, output_tokens: int,
|
||||
) -> float:
|
||||
"""Calculate cost in USD from token counts."""
|
||||
# Direct lookup first
|
||||
rates = COST_PER_1M_TOKENS.get((provider, model))
|
||||
if not rates:
|
||||
# Case-insensitive provider lookup
|
||||
lower = provider.lower()
|
||||
for (p, m), r in COST_PER_1M_TOKENS.items():
|
||||
if p.lower() == lower and m == model:
|
||||
rates = r
|
||||
break
|
||||
if not rates:
|
||||
return 0.0
|
||||
input_rate, output_rate = rates
|
||||
return (input_tokens * input_rate + output_tokens * output_rate) / 1_000_000
|
||||
return _registry_calculate_cost(provider, model, input_tokens, output_tokens)
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
"""Generic JSON-file CRUD store for Pydantic models.
|
||||
|
||||
Every sub-app in the backend persists entities as one JSON file per record
|
||||
inside a data directory. This module eliminates that copy-paste.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from typing import Generic, TypeVar
|
||||
|
||||
from fastapi import HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
T = TypeVar("T", bound=BaseModel)
|
||||
|
||||
|
||||
class JsonStore(Generic[T]):
|
||||
"""One-JSON-file-per-entity CRUD backed by a flat directory.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
model_cls:
|
||||
The Pydantic model to (de)serialise.
|
||||
data_dir:
|
||||
Filesystem directory that holds ``{id}.json`` files.
|
||||
id_field:
|
||||
Name of the attribute used as the unique key (default ``"id"``).
|
||||
dump_mode:
|
||||
Passed to ``model_dump(mode=...)`` when serialising. Use ``"json"``
|
||||
for models that contain non-JSON-native types (e.g. ``datetime``).
|
||||
not_found_detail:
|
||||
Message for the ``HTTPException(404)`` raised by :meth:`load`.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_cls: type[T],
|
||||
data_dir: str,
|
||||
*,
|
||||
id_field: str = "id",
|
||||
dump_mode: str | None = None,
|
||||
not_found_detail: str = "Not found",
|
||||
) -> None:
|
||||
self._cls = model_cls
|
||||
self._dir = data_dir
|
||||
self._id = id_field
|
||||
self._dump_mode = dump_mode
|
||||
self._detail = not_found_detail
|
||||
|
||||
# -- helpers -------------------------------------------------------------
|
||||
|
||||
def _path(self, item_id: str) -> str:
|
||||
return os.path.join(self._dir, f"{item_id}.json")
|
||||
|
||||
def _dump(self, item: T) -> dict:
|
||||
if self._dump_mode:
|
||||
return item.model_dump(mode=self._dump_mode)
|
||||
return item.model_dump()
|
||||
|
||||
# -- public API ----------------------------------------------------------
|
||||
|
||||
def load_all(self) -> list[T]:
|
||||
result: list[T] = []
|
||||
if not os.path.exists(self._dir):
|
||||
return result
|
||||
for fname in os.listdir(self._dir):
|
||||
if fname.endswith(".json"):
|
||||
with open(os.path.join(self._dir, fname)) as f:
|
||||
result.append(self._cls(**json.load(f)))
|
||||
return result
|
||||
|
||||
def save(self, item: T) -> None:
|
||||
os.makedirs(self._dir, exist_ok=True)
|
||||
item_id = getattr(item, self._id)
|
||||
with open(self._path(item_id), "w") as f:
|
||||
json.dump(self._dump(item), f, indent=2)
|
||||
|
||||
def load(self, item_id: str) -> T:
|
||||
path = self._path(item_id)
|
||||
if not os.path.exists(path):
|
||||
raise HTTPException(status_code=404, detail=self._detail)
|
||||
with open(path) as f:
|
||||
return self._cls(**json.load(f))
|
||||
|
||||
def load_or_none(self, item_id: str) -> T | None:
|
||||
path = self._path(item_id)
|
||||
if not os.path.exists(path):
|
||||
return None
|
||||
with open(path) as f:
|
||||
return self._cls(**json.load(f))
|
||||
|
||||
def delete(self, item_id: str) -> None:
|
||||
path = self._path(item_id)
|
||||
if os.path.exists(path):
|
||||
os.remove(path)
|
||||
|
||||
def exists(self, item_id: str) -> bool:
|
||||
return os.path.exists(self._path(item_id))
|
||||
|
||||
|
||||
class SessionStore:
|
||||
"""Specialised JSON store for agent session dicts (not Pydantic models).
|
||||
|
||||
Sessions are stored as raw ``dict`` values keyed by ``session_id``.
|
||||
"""
|
||||
|
||||
def __init__(self, data_dir: str) -> None:
|
||||
self._dir = data_dir
|
||||
|
||||
def _path(self, session_id: str) -> str:
|
||||
return os.path.join(self._dir, f"{session_id}.json")
|
||||
|
||||
def save(self, session_id: str, doc_data: dict) -> None:
|
||||
os.makedirs(self._dir, exist_ok=True)
|
||||
with open(self._path(session_id), "w") as f:
|
||||
json.dump(doc_data, f, indent=2)
|
||||
|
||||
def load(self, session_id: str) -> dict | None:
|
||||
path = self._path(session_id)
|
||||
if not os.path.exists(path):
|
||||
return None
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
|
||||
def delete(self, session_id: str) -> None:
|
||||
path = self._path(session_id)
|
||||
if os.path.exists(path):
|
||||
os.remove(path)
|
||||
|
||||
def load_all(self) -> list[tuple[str, dict]]:
|
||||
results: list[tuple[str, dict]] = []
|
||||
if not os.path.exists(self._dir):
|
||||
return results
|
||||
for fname in os.listdir(self._dir):
|
||||
if fname.endswith(".json"):
|
||||
with open(os.path.join(self._dir, fname)) as f:
|
||||
results.append((fname[:-5], json.load(f)))
|
||||
return results
|
||||
@@ -0,0 +1,54 @@
|
||||
"""Convenience wrappers for quick LLM calls.
|
||||
|
||||
These helpers handle client construction, markdown fence stripping, and JSON
|
||||
parsing so that callers don't need to repeat the same boilerplate.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def strip_markdown_fences(text: str) -> str:
|
||||
"""Remove ```json ... ``` or similar fences from LLM output."""
|
||||
stripped = text.strip()
|
||||
if stripped.startswith("```"):
|
||||
stripped = re.sub(r"^```[a-zA-Z]*\n?", "", stripped, count=1)
|
||||
stripped = re.sub(r"\n?```\s*$", "", stripped)
|
||||
return stripped.strip()
|
||||
|
||||
|
||||
async def quick_llm_call(
|
||||
system: str,
|
||||
user_content: str,
|
||||
model: str = "claude-sonnet-4-20250514",
|
||||
max_tokens: int = 300,
|
||||
) -> str:
|
||||
"""Make a simple LLM call and return the text response."""
|
||||
from backend.apps.settings.credentials import get_anthropic_client
|
||||
from backend.apps.settings.settings import load_settings
|
||||
|
||||
client = get_anthropic_client(load_settings())
|
||||
resp = await client.messages.create(
|
||||
model=model,
|
||||
max_tokens=max_tokens,
|
||||
system=system,
|
||||
messages=[{"role": "user", "content": user_content}],
|
||||
)
|
||||
return resp.content[0].text.strip()
|
||||
|
||||
|
||||
async def quick_llm_json(
|
||||
system: str,
|
||||
user_content: str,
|
||||
model: str = "claude-sonnet-4-20250514",
|
||||
max_tokens: int = 300,
|
||||
) -> dict:
|
||||
"""Make an LLM call expecting JSON. Strips markdown fences, parses JSON."""
|
||||
raw = await quick_llm_call(system, user_content, model=model, max_tokens=max_tokens)
|
||||
cleaned = strip_markdown_fences(raw)
|
||||
return json.loads(cleaned)
|
||||
@@ -0,0 +1,28 @@
|
||||
"""Shared MCP / SSE utilities used by tools_lib and mcp_client."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
|
||||
def parse_sse_json(text: str) -> dict | None:
|
||||
"""Extract JSON from an SSE response body (handles ``data: {...}`` lines)."""
|
||||
for line in text.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("data:"):
|
||||
payload = stripped[len("data:"):].strip()
|
||||
if payload:
|
||||
try:
|
||||
return json.loads(payload)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
try:
|
||||
return json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
|
||||
|
||||
def sanitize_server_name(name: str) -> str:
|
||||
"""Convert a tool name into a valid MCP server identifier (alphanumeric + hyphens)."""
|
||||
return re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-")
|
||||
@@ -0,0 +1,111 @@
|
||||
"""Single source of truth for model definitions, IDs, and cost rates.
|
||||
|
||||
Other modules should import from here instead of maintaining their own
|
||||
``MODEL_MAP`` / ``BUILTIN_MODELS`` / ``COST_PER_1M_TOKENS`` copies.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelDef:
|
||||
value: str
|
||||
label: str
|
||||
model_id: str
|
||||
provider: str
|
||||
api: str
|
||||
context_window: int
|
||||
input_cost_per_1m: float
|
||||
output_cost_per_1m: float
|
||||
|
||||
|
||||
# fmt: off
|
||||
ALL_MODELS: list[ModelDef] = [
|
||||
# Anthropic
|
||||
ModelDef("sonnet", "Claude Sonnet 4.6", "claude-sonnet-4-6", "Anthropic", "anthropic", 1_000_000, 3.0, 15.0),
|
||||
ModelDef("opus", "Claude Opus 4.6", "claude-opus-4-6", "Anthropic", "anthropic", 1_000_000, 5.0, 25.0),
|
||||
ModelDef("haiku", "Claude Haiku 4.5", "claude-haiku-4-5", "Anthropic", "anthropic", 200_000, 1.0, 5.0),
|
||||
# OpenAI
|
||||
ModelDef("gpt-5.4", "GPT-5.4", "gpt-5.4", "OpenAI", "openai", 128_000, 2.50, 15.0),
|
||||
ModelDef("gpt-5.4-mini", "GPT-5.4 Mini", "gpt-5.4-mini", "OpenAI", "openai", 128_000, 0.75, 3.0),
|
||||
ModelDef("o3", "o3", "o3", "OpenAI", "openai", 128_000, 2.0, 8.0),
|
||||
ModelDef("o4-mini", "o4-mini", "o4-mini", "OpenAI", "openai", 128_000, 1.10, 4.40),
|
||||
# Google
|
||||
ModelDef("gemini-2.5-flash", "Gemini 2.5 Flash", "gemini-2.5-flash", "Google", "gemini", 1_000_000, 0.15, 0.60),
|
||||
ModelDef("gemini-2.5-pro", "Gemini 2.5 Pro", "gemini-2.5-pro", "Google", "gemini", 1_000_000, 1.25, 10.0),
|
||||
# OpenRouter-backed
|
||||
ModelDef("x-ai/grok-4-0214", "Grok 4", "x-ai/grok-4-0214", "xAI", "openrouter", 128_000, 3.0, 15.0),
|
||||
ModelDef("meta-llama/llama-4-maverick", "Llama 4 Maverick", "meta-llama/llama-4-maverick", "Meta", "openrouter", 128_000, 0.50, 0.70),
|
||||
ModelDef("meta-llama/llama-4-scout", "Llama 4 Scout", "meta-llama/llama-4-scout", "Meta", "openrouter", 128_000, 0.15, 0.40),
|
||||
ModelDef("deepseek/deepseek-chat-v3-0324", "DeepSeek V3", "deepseek/deepseek-chat-v3-0324", "DeepSeek", "openrouter", 128_000, 0.30, 0.90),
|
||||
ModelDef("deepseek/deepseek-r1", "DeepSeek R1", "deepseek/deepseek-r1", "DeepSeek", "openrouter", 128_000, 0.80, 2.40),
|
||||
ModelDef("mistralai/mistral-large-2501", "Mistral Large", "mistralai/mistral-large-2501", "Mistral", "openrouter", 128_000, 2.0, 6.0),
|
||||
ModelDef("mistralai/mistral-small-3.1-24b-instruct", "Mistral Small 3.1", "mistralai/mistral-small-3.1-24b-instruct", "Mistral", "openrouter", 128_000, 0.10, 0.30),
|
||||
ModelDef("qwen/qwen3-coder", "Qwen3 Coder", "qwen/qwen3-coder", "Qwen", "openrouter", 128_000, 0.0, 0.0),
|
||||
ModelDef("qwen/qwen3-235b-a22b", "Qwen3 235B", "qwen/qwen3-235b-a22b", "Qwen", "openrouter", 128_000, 0.20, 0.70),
|
||||
ModelDef("cohere/command-a-03-2025", "Command A", "cohere/command-a-03-2025", "Cohere", "openrouter", 128_000, 2.50, 10.0),
|
||||
]
|
||||
# fmt: on
|
||||
|
||||
_BY_VALUE: dict[str, ModelDef] = {m.value: m for m in ALL_MODELS}
|
||||
_BY_MODEL_ID: dict[str, ModelDef] = {m.model_id: m for m in ALL_MODELS}
|
||||
|
||||
|
||||
def resolve_model_id(short_name: str) -> str:
|
||||
"""Map a short name (e.g. ``"sonnet"``) to the canonical API model ID.
|
||||
|
||||
Returns *short_name* unchanged if no mapping exists.
|
||||
"""
|
||||
m = _BY_VALUE.get(short_name)
|
||||
return m.model_id if m else short_name
|
||||
|
||||
|
||||
def get_cost_rates(provider: str, model: str) -> tuple[float, float] | None:
|
||||
"""Return ``(input_cost_per_1m, output_cost_per_1m)`` or ``None``."""
|
||||
m = _BY_VALUE.get(model)
|
||||
if m and m.provider.lower() == provider.lower():
|
||||
return (m.input_cost_per_1m, m.output_cost_per_1m)
|
||||
for md in ALL_MODELS:
|
||||
if md.value == model and md.provider.lower() == provider.lower():
|
||||
return (md.input_cost_per_1m, md.output_cost_per_1m)
|
||||
return None
|
||||
|
||||
|
||||
def calculate_cost(
|
||||
provider: str, model: str, input_tokens: int, output_tokens: int,
|
||||
) -> float:
|
||||
"""Calculate cost in USD from token counts."""
|
||||
rates = get_cost_rates(provider, model)
|
||||
if not rates:
|
||||
return 0.0
|
||||
input_rate, output_rate = rates
|
||||
return (input_tokens * input_rate + output_tokens * output_rate) / 1_000_000
|
||||
|
||||
|
||||
def get_context_window(model: str) -> int:
|
||||
"""Look up context window for a model by its short value name."""
|
||||
m = _BY_VALUE.get(model) or _BY_MODEL_ID.get(model)
|
||||
return m.context_window if m else 128_000
|
||||
|
||||
|
||||
def get_builtin_models_by_provider() -> dict[str, list[dict[str, Any]]]:
|
||||
"""Return built-in models grouped by provider, matching the legacy format.
|
||||
|
||||
Only includes the curated built-in models (Anthropic) — not
|
||||
OpenRouter-backed models which are exposed through custom providers.
|
||||
"""
|
||||
result: dict[str, list[dict[str, Any]]] = {}
|
||||
for m in ALL_MODELS:
|
||||
if m.api == "openrouter":
|
||||
continue
|
||||
result.setdefault(m.provider, []).append({
|
||||
"value": m.value,
|
||||
"label": m.label,
|
||||
"context_window": m.context_window,
|
||||
"model_id": m.model_id,
|
||||
"api": m.api,
|
||||
})
|
||||
return result
|
||||
@@ -6,6 +6,7 @@ from datetime import datetime
|
||||
from uuid import uuid4
|
||||
|
||||
from backend.config.Apps import SubApp
|
||||
from backend.apps.common.json_store import JsonStore
|
||||
from backend.apps.dashboards.models import (
|
||||
Dashboard,
|
||||
DashboardCreate,
|
||||
@@ -24,34 +25,14 @@ from backend.config.paths import DASHBOARDS_DIR as DATA_DIR, SESSIONS_DIR, DASHB
|
||||
OLD_LAYOUT_FILE = os.path.join(OLD_LAYOUT_DIR, "layout.json")
|
||||
|
||||
|
||||
def _load_all() -> list[Dashboard]:
|
||||
result = []
|
||||
if not os.path.exists(DATA_DIR):
|
||||
return result
|
||||
for fname in os.listdir(DATA_DIR):
|
||||
if fname.endswith(".json"):
|
||||
with open(os.path.join(DATA_DIR, fname)) as f:
|
||||
result.append(Dashboard(**json.load(f)))
|
||||
return result
|
||||
_store = JsonStore(
|
||||
Dashboard, DATA_DIR, dump_mode="json", not_found_detail="Dashboard not found",
|
||||
)
|
||||
|
||||
|
||||
def _save(dashboard: Dashboard):
|
||||
with open(os.path.join(DATA_DIR, f"{dashboard.id}.json"), "w") as f:
|
||||
json.dump(dashboard.model_dump(mode="json"), f, indent=2)
|
||||
|
||||
|
||||
def _load(dashboard_id: str) -> Dashboard:
|
||||
path = os.path.join(DATA_DIR, f"{dashboard_id}.json")
|
||||
if not os.path.exists(path):
|
||||
raise HTTPException(status_code=404, detail="Dashboard not found")
|
||||
with open(path) as f:
|
||||
return Dashboard(**json.load(f))
|
||||
|
||||
|
||||
def _delete(dashboard_id: str):
|
||||
path = os.path.join(DATA_DIR, f"{dashboard_id}.json")
|
||||
if os.path.exists(path):
|
||||
os.remove(path)
|
||||
_load_all = _store.load_all
|
||||
_save = _store.save
|
||||
_load = _store.load
|
||||
_delete = _store.delete
|
||||
|
||||
|
||||
def _migrate_if_needed():
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
"""Built-in mode definitions.
|
||||
|
||||
Separated from models.py to keep schema classes small and data separate.
|
||||
"""
|
||||
|
||||
from backend.apps.modes.models import Mode
|
||||
from backend.config.paths import OUTPUTS_WORKSPACE_DIR as OUTPUTS_WORKSPACE, SKILLS_WORKSPACE_DIR as SKILLS_WORKSPACE
|
||||
|
||||
BUILTIN_MODES: list[Mode] = [
|
||||
Mode(
|
||||
id="agent",
|
||||
name="Agent",
|
||||
description="Full autonomous agent with read and write access to tools.",
|
||||
system_prompt=None,
|
||||
tools=None,
|
||||
default_next_mode=None,
|
||||
is_builtin=True,
|
||||
icon="smart_toy",
|
||||
color="#818cf8",
|
||||
),
|
||||
Mode(
|
||||
id="ask",
|
||||
name="Ask",
|
||||
description="Answer questions about the codebase. Read-only, no edits or changes.",
|
||||
system_prompt="Answer questions about the codebase. Do not make any edits or changes.",
|
||||
tools=["Read", "Glob", "Grep", "AskUserQuestion"],
|
||||
default_next_mode=None,
|
||||
is_builtin=True,
|
||||
icon="question_answer",
|
||||
color="#4ade80",
|
||||
),
|
||||
Mode(
|
||||
id="plan",
|
||||
name="Plan",
|
||||
description="Analyze requests and produce a detailed step-by-step plan without executing.",
|
||||
system_prompt="Analyze the request and produce a detailed step-by-step plan. Do not execute the plan or make any changes.",
|
||||
tools=["Read", "Glob", "Grep", "AskUserQuestion"],
|
||||
default_next_mode="agent",
|
||||
is_builtin=True,
|
||||
icon="map",
|
||||
color="#fbbf24",
|
||||
),
|
||||
Mode(
|
||||
id="view-builder",
|
||||
name="App Builder",
|
||||
description="Create and iterate on reusable App artifacts.",
|
||||
system_prompt=(
|
||||
"You are an App Builder — an AI assistant that creates self-contained "
|
||||
"web apps rendered in an iframe preview.\n\n"
|
||||
"Your working directory is a dedicated workspace folder pre-seeded with "
|
||||
"template files. Read the existing files before making changes.\n\n"
|
||||
"## Critical rules\n\n"
|
||||
"- The entry point MUST be named `index.html`. Never rename it or create "
|
||||
"a different HTML file as the main entry point.\n"
|
||||
"- Write files immediately when you have code ready — the user sees a "
|
||||
"live preview that auto-refreshes from these files.\n"
|
||||
"- Always write the complete file content on first creation (do not use "
|
||||
"Edit for partial patches on new files).\n"
|
||||
"- For complex apps, split code into separate files (JS, CSS, etc.) "
|
||||
"and reference them from index.html with relative paths.\n"
|
||||
"- Always update meta.json with a short name and one-sentence description.\n"
|
||||
"- Build beautiful, polished UIs with modern design — dark themes, smooth "
|
||||
"transitions, proper spacing, and responsive layouts.\n\n"
|
||||
"Read the SKILL.md reference in your workspace for the full technical "
|
||||
"specification of the App platform (available globals, file conventions, "
|
||||
"schema format, backend.py usage, and examples)."
|
||||
),
|
||||
tools=None,
|
||||
default_next_mode=None,
|
||||
is_builtin=True,
|
||||
icon="view_quilt",
|
||||
color="#f472b6",
|
||||
default_folder=OUTPUTS_WORKSPACE,
|
||||
),
|
||||
Mode(
|
||||
id="skill-builder",
|
||||
name="Skill Builder",
|
||||
description="Create and iterate on skills using AI-assisted vibe coding.",
|
||||
system_prompt=(
|
||||
"You are a Skill Builder — an AI assistant that helps users create, "
|
||||
"refine, and iterate on Claude skills (SKILL.md files).\n\n"
|
||||
"## How Skills Work\n\n"
|
||||
"A skill is a Markdown file that teaches Claude how to perform a specific task. "
|
||||
"Skills have YAML frontmatter with `name` and `description` fields, followed by "
|
||||
"the skill body in Markdown. The description is the primary triggering mechanism — "
|
||||
"it tells Claude when to use the skill.\n\n"
|
||||
"## Your Working Directory\n\n"
|
||||
"Your working directory is a dedicated workspace folder for this skill. "
|
||||
"Write your output directly to these files using the Write tool:\n\n"
|
||||
"1. **SKILL.md** — The complete skill file with YAML frontmatter and Markdown body. "
|
||||
"Example frontmatter:\n"
|
||||
" ```\n"
|
||||
" ---\n"
|
||||
" name: my-skill\n"
|
||||
" description: When to trigger and what this skill does.\n"
|
||||
" ---\n"
|
||||
" ```\n\n"
|
||||
"2. **meta.json** — Metadata for the skill builder UI. Always write this file. Example:\n"
|
||||
' {"name":"My Skill","description":"A short description","command":"my-skill"}\n\n'
|
||||
"Write these files immediately when you have content ready. The user can see "
|
||||
"a live preview that auto-refreshes from these files. Always write the "
|
||||
"complete file content (do not use Edit for partial patches on first creation).\n\n"
|
||||
"## Skill Creation Process\n\n"
|
||||
"1. **Understand intent** — Ask what the skill should do, when it should trigger, "
|
||||
"and what the expected output format is.\n"
|
||||
"2. **Draft the skill** — Write a SKILL.md with clear instructions, examples, "
|
||||
"and good progressive disclosure.\n"
|
||||
"3. **Iterate** — Refine based on user feedback. Update the files each time.\n\n"
|
||||
"## Skill Writing Best Practices\n\n"
|
||||
"- Keep SKILL.md under 500 lines; use bundled reference files for large content.\n"
|
||||
"- The `description` frontmatter is the primary trigger. Make it slightly \"pushy\" — "
|
||||
"include both what the skill does AND specific contexts for when to use it.\n"
|
||||
"- Use imperative form in instructions.\n"
|
||||
"- Include examples with input/output pairs when helpful.\n"
|
||||
"- Define output formats explicitly with templates.\n"
|
||||
"- Use theory of mind — explain *why* things matter rather than just MUST directives.\n"
|
||||
"- Think about edge cases, error handling, and progressive disclosure.\n\n"
|
||||
"## Skill Anatomy\n\n"
|
||||
"```\n"
|
||||
"skill-name/\n"
|
||||
"├── SKILL.md (required) — YAML frontmatter + Markdown instructions\n"
|
||||
"└── Bundled Resources (optional)\n"
|
||||
" ├── scripts/ — Executable code for repetitive tasks\n"
|
||||
" ├── references/ — Docs loaded into context as needed\n"
|
||||
" └── assets/ — Files used in output\n"
|
||||
"```\n\n"
|
||||
"Be collaborative and flexible. If the user wants to \"just vibe\", skip the formal "
|
||||
"process and iterate freely. Always write updated files so the preview stays current."
|
||||
),
|
||||
tools=None,
|
||||
default_next_mode=None,
|
||||
is_builtin=True,
|
||||
icon="psychology",
|
||||
color="#10b981",
|
||||
default_folder=SKILLS_WORKSPACE,
|
||||
),
|
||||
]
|
||||
@@ -2,8 +2,6 @@ from pydantic import BaseModel, Field
|
||||
from typing import Optional
|
||||
from uuid import uuid4
|
||||
|
||||
from backend.config.paths import OUTPUTS_WORKSPACE_DIR as OUTPUTS_WORKSPACE, SKILLS_WORKSPACE_DIR as SKILLS_WORKSPACE
|
||||
|
||||
|
||||
class Mode(BaseModel):
|
||||
id: str = Field(default_factory=lambda: uuid4().hex)
|
||||
@@ -38,134 +36,3 @@ class ModeUpdate(BaseModel):
|
||||
icon: Optional[str] = None
|
||||
color: Optional[str] = None
|
||||
default_folder: Optional[str] = None
|
||||
|
||||
|
||||
BUILTIN_MODES: list[Mode] = [
|
||||
Mode(
|
||||
id="agent",
|
||||
name="Agent",
|
||||
description="Full autonomous agent with read and write access to tools.",
|
||||
system_prompt=None,
|
||||
tools=None,
|
||||
default_next_mode=None,
|
||||
is_builtin=True,
|
||||
icon="smart_toy",
|
||||
color="#818cf8",
|
||||
),
|
||||
Mode(
|
||||
id="ask",
|
||||
name="Ask",
|
||||
description="Answer questions about the codebase. Read-only, no edits or changes.",
|
||||
system_prompt="Answer questions about the codebase. Do not make any edits or changes.",
|
||||
tools=["Read", "Glob", "Grep", "AskUserQuestion"],
|
||||
default_next_mode=None,
|
||||
is_builtin=True,
|
||||
icon="question_answer",
|
||||
color="#4ade80",
|
||||
),
|
||||
Mode(
|
||||
id="plan",
|
||||
name="Plan",
|
||||
description="Analyze requests and produce a detailed step-by-step plan without executing.",
|
||||
system_prompt="Analyze the request and produce a detailed step-by-step plan. Do not execute the plan or make any changes.",
|
||||
tools=["Read", "Glob", "Grep", "AskUserQuestion"],
|
||||
default_next_mode="agent",
|
||||
is_builtin=True,
|
||||
icon="map",
|
||||
color="#fbbf24",
|
||||
),
|
||||
Mode(
|
||||
id="view-builder",
|
||||
name="App Builder",
|
||||
description="Create and iterate on reusable App artifacts.",
|
||||
system_prompt=(
|
||||
"You are an App Builder — an AI assistant that creates self-contained "
|
||||
"web apps rendered in an iframe preview.\n\n"
|
||||
"Your working directory is a dedicated workspace folder pre-seeded with "
|
||||
"template files. Read the existing files before making changes.\n\n"
|
||||
"## Critical rules\n\n"
|
||||
"- The entry point MUST be named `index.html`. Never rename it or create "
|
||||
"a different HTML file as the main entry point.\n"
|
||||
"- Write files immediately when you have code ready — the user sees a "
|
||||
"live preview that auto-refreshes from these files.\n"
|
||||
"- Always write the complete file content on first creation (do not use "
|
||||
"Edit for partial patches on new files).\n"
|
||||
"- For complex apps, split code into separate files (JS, CSS, etc.) "
|
||||
"and reference them from index.html with relative paths.\n"
|
||||
"- Always update meta.json with a short name and one-sentence description.\n"
|
||||
"- Build beautiful, polished UIs with modern design — dark themes, smooth "
|
||||
"transitions, proper spacing, and responsive layouts.\n\n"
|
||||
"Read the SKILL.md reference in your workspace for the full technical "
|
||||
"specification of the App platform (available globals, file conventions, "
|
||||
"schema format, backend.py usage, and examples)."
|
||||
),
|
||||
tools=None,
|
||||
default_next_mode=None,
|
||||
is_builtin=True,
|
||||
icon="view_quilt",
|
||||
color="#f472b6",
|
||||
default_folder=OUTPUTS_WORKSPACE,
|
||||
),
|
||||
Mode(
|
||||
id="skill-builder",
|
||||
name="Skill Builder",
|
||||
description="Create and iterate on skills using AI-assisted vibe coding.",
|
||||
system_prompt=(
|
||||
"You are a Skill Builder — an AI assistant that helps users create, "
|
||||
"refine, and iterate on Claude skills (SKILL.md files).\n\n"
|
||||
"## How Skills Work\n\n"
|
||||
"A skill is a Markdown file that teaches Claude how to perform a specific task. "
|
||||
"Skills have YAML frontmatter with `name` and `description` fields, followed by "
|
||||
"the skill body in Markdown. The description is the primary triggering mechanism — "
|
||||
"it tells Claude when to use the skill.\n\n"
|
||||
"## Your Working Directory\n\n"
|
||||
"Your working directory is a dedicated workspace folder for this skill. "
|
||||
"Write your output directly to these files using the Write tool:\n\n"
|
||||
"1. **SKILL.md** — The complete skill file with YAML frontmatter and Markdown body. "
|
||||
"Example frontmatter:\n"
|
||||
" ```\n"
|
||||
" ---\n"
|
||||
" name: my-skill\n"
|
||||
" description: When to trigger and what this skill does.\n"
|
||||
" ---\n"
|
||||
" ```\n\n"
|
||||
"2. **meta.json** — Metadata for the skill builder UI. Always write this file. Example:\n"
|
||||
' {"name":"My Skill","description":"A short description","command":"my-skill"}\n\n'
|
||||
"Write these files immediately when you have content ready. The user can see "
|
||||
"a live preview that auto-refreshes from these files. Always write the "
|
||||
"complete file content (do not use Edit for partial patches on first creation).\n\n"
|
||||
"## Skill Creation Process\n\n"
|
||||
"1. **Understand intent** — Ask what the skill should do, when it should trigger, "
|
||||
"and what the expected output format is.\n"
|
||||
"2. **Draft the skill** — Write a SKILL.md with clear instructions, examples, "
|
||||
"and good progressive disclosure.\n"
|
||||
"3. **Iterate** — Refine based on user feedback. Update the files each time.\n\n"
|
||||
"## Skill Writing Best Practices\n\n"
|
||||
"- Keep SKILL.md under 500 lines; use bundled reference files for large content.\n"
|
||||
"- The `description` frontmatter is the primary trigger. Make it slightly \"pushy\" — "
|
||||
"include both what the skill does AND specific contexts for when to use it.\n"
|
||||
"- Use imperative form in instructions.\n"
|
||||
"- Include examples with input/output pairs when helpful.\n"
|
||||
"- Define output formats explicitly with templates.\n"
|
||||
"- Use theory of mind — explain *why* things matter rather than just MUST directives.\n"
|
||||
"- Think about edge cases, error handling, and progressive disclosure.\n\n"
|
||||
"## Skill Anatomy\n\n"
|
||||
"```\n"
|
||||
"skill-name/\n"
|
||||
"├── SKILL.md (required) — YAML frontmatter + Markdown instructions\n"
|
||||
"└── Bundled Resources (optional)\n"
|
||||
" ├── scripts/ — Executable code for repetitive tasks\n"
|
||||
" ├── references/ — Docs loaded into context as needed\n"
|
||||
" └── assets/ — Files used in output\n"
|
||||
"```\n\n"
|
||||
"Be collaborative and flexible. If the user wants to \"just vibe\", skip the formal "
|
||||
"process and iterate freely. Always write updated files so the preview stays current."
|
||||
),
|
||||
tools=None,
|
||||
default_next_mode=None,
|
||||
is_builtin=True,
|
||||
icon="psychology",
|
||||
color="#10b981",
|
||||
default_folder=SKILLS_WORKSPACE,
|
||||
),
|
||||
]
|
||||
|
||||
@@ -4,7 +4,9 @@ import logging
|
||||
from contextlib import asynccontextmanager
|
||||
from fastapi import HTTPException
|
||||
from backend.config.Apps import SubApp
|
||||
from backend.apps.modes.models import Mode, ModeCreate, ModeUpdate, BUILTIN_MODES
|
||||
from backend.apps.common.json_store import JsonStore
|
||||
from backend.apps.modes.models import Mode, ModeCreate, ModeUpdate
|
||||
from backend.apps.modes.builtin import BUILTIN_MODES
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -24,37 +26,12 @@ async def modes_lifespan():
|
||||
modes = SubApp("modes", modes_lifespan)
|
||||
|
||||
|
||||
def _load_all() -> list[Mode]:
|
||||
result = []
|
||||
if not os.path.exists(DATA_DIR):
|
||||
return result
|
||||
for fname in os.listdir(DATA_DIR):
|
||||
if fname.endswith(".json"):
|
||||
with open(os.path.join(DATA_DIR, fname)) as f:
|
||||
result.append(Mode(**json.load(f)))
|
||||
return result
|
||||
_store = JsonStore(Mode, DATA_DIR, not_found_detail="Mode not found")
|
||||
|
||||
|
||||
def _save(mode: Mode):
|
||||
with open(os.path.join(DATA_DIR, f"{mode.id}.json"), "w") as f:
|
||||
json.dump(mode.model_dump(), f, indent=2)
|
||||
|
||||
|
||||
def _load(mode_id: str) -> Mode:
|
||||
path = os.path.join(DATA_DIR, f"{mode_id}.json")
|
||||
if not os.path.exists(path):
|
||||
raise HTTPException(status_code=404, detail="Mode not found")
|
||||
with open(path) as f:
|
||||
return Mode(**json.load(f))
|
||||
|
||||
|
||||
def load_mode(mode_id: str) -> Mode | None:
|
||||
"""Public helper for other modules to resolve a mode by ID."""
|
||||
path = os.path.join(DATA_DIR, f"{mode_id}.json")
|
||||
if not os.path.exists(path):
|
||||
return None
|
||||
with open(path) as f:
|
||||
return Mode(**json.load(f))
|
||||
_load_all = _store.load_all
|
||||
_save = _store.save
|
||||
_load = _store.load
|
||||
load_mode = _store.load_or_none
|
||||
|
||||
|
||||
@modes.router.get("/list")
|
||||
@@ -109,7 +86,5 @@ async def delete_mode(mode_id: str):
|
||||
mode = _load(mode_id)
|
||||
if mode.is_builtin:
|
||||
raise HTTPException(status_code=403, detail="Cannot delete built-in modes")
|
||||
path = os.path.join(DATA_DIR, f"{mode_id}.json")
|
||||
if os.path.exists(path):
|
||||
os.remove(path)
|
||||
_store.delete(mode_id)
|
||||
return {"ok": True}
|
||||
|
||||
@@ -4,6 +4,53 @@ from uuid import uuid4
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
def _migrate_legacy_files(
|
||||
data: dict,
|
||||
*,
|
||||
allow_schema_json: bool = False,
|
||||
always_set_files: bool = False,
|
||||
) -> dict:
|
||||
"""Convert legacy frontend_code/backend_code fields into the files dict.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
allow_schema_json:
|
||||
Also migrate a ``schema_json`` field to ``files["schema.json"]``.
|
||||
always_set_files:
|
||||
When ``True``, set ``data["files"]`` even to an empty dict if no
|
||||
legacy fields are found (used by Output / OutputCreate). When
|
||||
``False``, only set ``data["files"]`` if there are actual files to
|
||||
migrate (used by OutputUpdate / WorkspaceSeedRequest).
|
||||
"""
|
||||
if not isinstance(data, dict):
|
||||
return data
|
||||
|
||||
files_present = "files" in data
|
||||
files_truthy = files_present and data["files"]
|
||||
|
||||
if not files_present or (always_set_files and not files_truthy):
|
||||
files: dict[str, str] = {}
|
||||
fc = data.pop("frontend_code", None)
|
||||
bc = data.pop("backend_code", None)
|
||||
if fc:
|
||||
files["index.html"] = fc
|
||||
if bc:
|
||||
files["backend.py"] = bc
|
||||
if allow_schema_json:
|
||||
sj = data.pop("schema_json", None)
|
||||
if sj:
|
||||
files["schema.json"] = sj
|
||||
if files or always_set_files:
|
||||
data["files"] = files
|
||||
else:
|
||||
data.pop("frontend_code", None)
|
||||
data.pop("backend_code", None)
|
||||
if allow_schema_json:
|
||||
data.pop("schema_json", None)
|
||||
|
||||
return data
|
||||
|
||||
|
||||
class AutoRunConfig(BaseModel):
|
||||
enabled: bool = False
|
||||
prompt: str = ""
|
||||
@@ -33,22 +80,7 @@ class Output(BaseModel):
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def _migrate_flat_fields(cls, data: Any) -> Any:
|
||||
"""Migrate legacy frontend_code/backend_code fields into the files dict."""
|
||||
if not isinstance(data, dict):
|
||||
return data
|
||||
if "files" not in data or not data["files"]:
|
||||
files: dict[str, str] = {}
|
||||
fc = data.pop("frontend_code", None)
|
||||
bc = data.pop("backend_code", None)
|
||||
if fc:
|
||||
files["index.html"] = fc
|
||||
if bc:
|
||||
files["backend.py"] = bc
|
||||
data["files"] = files
|
||||
else:
|
||||
data.pop("frontend_code", None)
|
||||
data.pop("backend_code", None)
|
||||
return data
|
||||
return _migrate_legacy_files(data, always_set_files=True)
|
||||
|
||||
@property
|
||||
def frontend_code(self) -> str:
|
||||
@@ -75,21 +107,7 @@ class OutputCreate(BaseModel):
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def _migrate_flat_fields(cls, data: Any) -> Any:
|
||||
if not isinstance(data, dict):
|
||||
return data
|
||||
if "files" not in data or not data["files"]:
|
||||
files: dict[str, str] = {}
|
||||
fc = data.pop("frontend_code", None)
|
||||
bc = data.pop("backend_code", None)
|
||||
if fc:
|
||||
files["index.html"] = fc
|
||||
if bc:
|
||||
files["backend.py"] = bc
|
||||
data["files"] = files
|
||||
else:
|
||||
data.pop("frontend_code", None)
|
||||
data.pop("backend_code", None)
|
||||
return data
|
||||
return _migrate_legacy_files(data, always_set_files=True)
|
||||
|
||||
|
||||
class OutputUpdate(BaseModel):
|
||||
@@ -105,22 +123,7 @@ class OutputUpdate(BaseModel):
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def _migrate_flat_fields(cls, data: Any) -> Any:
|
||||
if not isinstance(data, dict):
|
||||
return data
|
||||
if "files" not in data:
|
||||
files: dict[str, str] = {}
|
||||
fc = data.pop("frontend_code", None)
|
||||
bc = data.pop("backend_code", None)
|
||||
if fc:
|
||||
files["index.html"] = fc
|
||||
if bc:
|
||||
files["backend.py"] = bc
|
||||
if files:
|
||||
data["files"] = files
|
||||
else:
|
||||
data.pop("frontend_code", None)
|
||||
data.pop("backend_code", None)
|
||||
return data
|
||||
return _migrate_legacy_files(data)
|
||||
|
||||
|
||||
class OutputExecute(BaseModel):
|
||||
@@ -165,27 +168,7 @@ class WorkspaceSeedRequest(BaseModel):
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def _migrate_flat_fields(cls, data: Any) -> Any:
|
||||
"""Accept legacy frontend_code/backend_code/schema_json fields."""
|
||||
if not isinstance(data, dict):
|
||||
return data
|
||||
if "files" not in data:
|
||||
files: dict[str, str] = {}
|
||||
fc = data.pop("frontend_code", None)
|
||||
bc = data.pop("backend_code", None)
|
||||
sj = data.pop("schema_json", None)
|
||||
if fc:
|
||||
files["index.html"] = fc
|
||||
if bc:
|
||||
files["backend.py"] = bc
|
||||
if sj:
|
||||
files["schema.json"] = sj
|
||||
if files:
|
||||
data["files"] = files
|
||||
else:
|
||||
data.pop("frontend_code", None)
|
||||
data.pop("backend_code", None)
|
||||
data.pop("schema_json", None)
|
||||
return data
|
||||
return _migrate_legacy_files(data, allow_schema_json=True)
|
||||
|
||||
|
||||
class VibeCodeRequest(BaseModel):
|
||||
|
||||
@@ -9,6 +9,7 @@ from fastapi import HTTPException, Query
|
||||
from fastapi.responses import Response
|
||||
from jsonschema import validate as schema_validate, ValidationError as SchemaValidationError
|
||||
from backend.config.Apps import SubApp
|
||||
from backend.apps.common.json_store import JsonStore
|
||||
from backend.apps.outputs.models import (
|
||||
Output, OutputCreate, OutputUpdate, OutputExecute, OutputExecuteResult,
|
||||
VibeCodeRequest, AutoRunRequest, AutoRunConfig, AutoRunAgentRequest,
|
||||
@@ -16,20 +17,11 @@ from backend.apps.outputs.models import (
|
||||
)
|
||||
from backend.apps.outputs.executor import execute_backend_code
|
||||
from backend.apps.outputs.view_builder_templates import VIEW_BUILDER_SKILL, VIEW_TEMPLATE_FILES
|
||||
from backend.apps.common.model_registry import resolve_model_id as _resolve_model
|
||||
from backend.apps.settings.settings import load_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
MODEL_MAP = {
|
||||
"sonnet": "claude-sonnet-4-20250514",
|
||||
"opus": "claude-opus-4-20250514",
|
||||
"haiku": "claude-haiku-4-5-20251001",
|
||||
}
|
||||
|
||||
|
||||
def _resolve_model(short_name: str) -> str:
|
||||
return MODEL_MAP.get(short_name, short_name)
|
||||
|
||||
|
||||
def _get_anthropic_client():
|
||||
"""Create an AsyncAnthropic client using the API key from app settings."""
|
||||
@@ -100,37 +92,12 @@ async def outputs_lifespan():
|
||||
outputs = SubApp("outputs", outputs_lifespan)
|
||||
|
||||
|
||||
def _load_all() -> list[Output]:
|
||||
result = []
|
||||
if not os.path.exists(DATA_DIR):
|
||||
return result
|
||||
for fname in os.listdir(DATA_DIR):
|
||||
if fname.endswith(".json"):
|
||||
with open(os.path.join(DATA_DIR, fname)) as f:
|
||||
result.append(Output(**json.load(f)))
|
||||
return result
|
||||
_store = JsonStore(Output, DATA_DIR, not_found_detail="Output not found")
|
||||
|
||||
|
||||
def _save(output: Output):
|
||||
with open(os.path.join(DATA_DIR, f"{output.id}.json"), "w") as f:
|
||||
json.dump(output.model_dump(), f, indent=2)
|
||||
|
||||
|
||||
def _load(output_id: str) -> Output:
|
||||
path = os.path.join(DATA_DIR, f"{output_id}.json")
|
||||
if not os.path.exists(path):
|
||||
raise HTTPException(status_code=404, detail="Output not found")
|
||||
with open(path) as f:
|
||||
return Output(**json.load(f))
|
||||
|
||||
|
||||
def load_output(output_id: str) -> Output | None:
|
||||
"""Public helper for other modules to resolve an output by ID."""
|
||||
path = os.path.join(DATA_DIR, f"{output_id}.json")
|
||||
if not os.path.exists(path):
|
||||
return None
|
||||
with open(path) as f:
|
||||
return Output(**json.load(f))
|
||||
_load_all = _store.load_all
|
||||
_save = _store.save
|
||||
_load = _store.load
|
||||
load_output = _store.load_or_none
|
||||
|
||||
|
||||
def _walk_directory(folder: str) -> dict[str, str]:
|
||||
|
||||
@@ -4,6 +4,7 @@ import logging
|
||||
from contextlib import asynccontextmanager
|
||||
from fastapi import HTTPException
|
||||
from backend.config.Apps import SubApp
|
||||
from backend.apps.common.json_store import JsonStore
|
||||
from backend.apps.templates.models import PromptTemplate, PromptTemplateCreate, PromptTemplateUpdate
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -17,32 +18,12 @@ async def templates_lifespan():
|
||||
|
||||
templates = SubApp("templates", templates_lifespan)
|
||||
|
||||
def _load_all() -> list[PromptTemplate]:
|
||||
result = []
|
||||
if not os.path.exists(DATA_DIR):
|
||||
return result
|
||||
for fname in os.listdir(DATA_DIR):
|
||||
if fname.endswith(".json"):
|
||||
with open(os.path.join(DATA_DIR, fname)) as f:
|
||||
result.append(PromptTemplate(**json.load(f)))
|
||||
return result
|
||||
_store = JsonStore(PromptTemplate, DATA_DIR, not_found_detail="Template not found")
|
||||
|
||||
def _save(template: PromptTemplate):
|
||||
path = os.path.join(DATA_DIR, f"{template.id}.json")
|
||||
with open(path, "w") as f:
|
||||
json.dump(template.model_dump(), f, indent=2)
|
||||
|
||||
def _load(template_id: str) -> PromptTemplate:
|
||||
path = os.path.join(DATA_DIR, f"{template_id}.json")
|
||||
if not os.path.exists(path):
|
||||
raise HTTPException(status_code=404, detail="Template not found")
|
||||
with open(path) as f:
|
||||
return PromptTemplate(**json.load(f))
|
||||
|
||||
def _delete(template_id: str):
|
||||
path = os.path.join(DATA_DIR, f"{template_id}.json")
|
||||
if os.path.exists(path):
|
||||
os.remove(path)
|
||||
_load_all = _store.load_all
|
||||
_save = _store.save
|
||||
_load = _store.load
|
||||
_delete = _store.delete
|
||||
|
||||
@templates.router.get("/list")
|
||||
async def list_templates():
|
||||
|
||||
@@ -18,6 +18,7 @@ from dotenv import load_dotenv
|
||||
from fastapi import HTTPException, Query
|
||||
from fastapi.responses import HTMLResponse
|
||||
from backend.config.Apps import SubApp
|
||||
from backend.apps.common.json_store import JsonStore
|
||||
from backend.apps.tools_lib.models import ToolDefinition, ToolCreate, ToolUpdate, BUILTIN_TOOLS
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -252,28 +253,11 @@ _pending_oauth: dict[str, str] = {}
|
||||
_pending_pkce: dict[str, str] = {} # state -> code_verifier (for PKCE flows)
|
||||
|
||||
|
||||
def _load_all() -> list[ToolDefinition]:
|
||||
result = []
|
||||
if not os.path.exists(DATA_DIR):
|
||||
return result
|
||||
for fname in os.listdir(DATA_DIR):
|
||||
if fname.endswith(".json"):
|
||||
with open(os.path.join(DATA_DIR, fname)) as f:
|
||||
result.append(ToolDefinition(**json.load(f)))
|
||||
return result
|
||||
_store = JsonStore(ToolDefinition, DATA_DIR, not_found_detail="Tool not found")
|
||||
|
||||
|
||||
def _save(tool: ToolDefinition):
|
||||
with open(os.path.join(DATA_DIR, f"{tool.id}.json"), "w") as f:
|
||||
json.dump(tool.model_dump(), f, indent=2)
|
||||
|
||||
|
||||
def _load(tool_id: str) -> ToolDefinition:
|
||||
path = os.path.join(DATA_DIR, f"{tool_id}.json")
|
||||
if not os.path.exists(path):
|
||||
raise HTTPException(status_code=404, detail="Tool not found")
|
||||
with open(path) as f:
|
||||
return ToolDefinition(**json.load(f))
|
||||
_load_all = _store.load_all
|
||||
_save = _store.save
|
||||
_load = _store.load
|
||||
|
||||
|
||||
@tools_lib.router.get("/builtin")
|
||||
@@ -478,9 +462,7 @@ async def delete_tool(tool_id: str):
|
||||
# MCP config derivation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _sanitize_server_name(name: str) -> str:
|
||||
"""Convert a tool name into a valid MCP server identifier (alphanumeric + hyphens)."""
|
||||
return re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-")
|
||||
from backend.apps.common.mcp_utils import sanitize_server_name as _sanitize_server_name
|
||||
|
||||
|
||||
def _extra_bin_dirs() -> list[str]:
|
||||
@@ -637,15 +619,6 @@ def derive_mcp_config(tool: ToolDefinition) -> Optional[dict]:
|
||||
return config
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# OAuth2 flow for Google Workspace (and other OAuth providers)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# MCP tool discovery
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_READ_PREFIXES = ("get", "list", "read", "search", "fetch", "find", "query", "count", "check", "describe", "show", "download", "browse", "analy", "explain")
|
||||
_WRITE_PREFIXES = ("create", "write", "delete", "update", "send", "remove", "modify", "add", "set", "put", "post", "patch", "insert", "move", "copy", "rename", "archive", "trash", "publish", "approve", "reject")
|
||||
|
||||
@@ -770,21 +743,7 @@ def _extract_service(name: str) -> tuple[str, str]:
|
||||
return "Other", ""
|
||||
|
||||
|
||||
def _parse_sse_json(text: str) -> dict | None:
|
||||
"""Extract JSON from an SSE response body (handles `data: {...}` lines)."""
|
||||
for line in text.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("data:"):
|
||||
payload = stripped[len("data:"):].strip()
|
||||
if payload:
|
||||
try:
|
||||
return json.loads(payload)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
try:
|
||||
return json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
from backend.apps.common.mcp_utils import parse_sse_json as _parse_sse_json
|
||||
|
||||
|
||||
async def _discover_mcp_tools_http(url: str, headers: dict | None = None) -> list[dict]:
|
||||
|
||||
+36
-9
@@ -2,15 +2,9 @@
|
||||
# The comment above is shebang, DO NOT REMOVE
|
||||
DEV_ABSPATH="$(readlink -f "${BASH_SOURCE[0]}")"
|
||||
if [[ "$OSTYPE" == "darwin"* ]]; then
|
||||
# echo "In macOS server sed START"
|
||||
# echo "SERVER_ABSPATH: $SERVER_ABSPATH"
|
||||
sed -i '' 's/\r//g' "$DEV_ABSPATH"
|
||||
# echo "In macOS server sed END"
|
||||
else
|
||||
# echo "NOT in macOS server START"
|
||||
# echo "SERVER_ABSPATH: $SERVER_ABSPATH"
|
||||
sed -i 's/\r//g' "$DEV_ABSPATH"
|
||||
# echo "NOT in macOS server START"
|
||||
fi
|
||||
chmod +x "$DEV_ABSPATH"
|
||||
|
||||
@@ -24,14 +18,47 @@ cleanup() {
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
# --- Find Python >= 3.10 ---
|
||||
REQUIRED_PYTHON_MINOR=10
|
||||
PYTHON_BIN=""
|
||||
for candidate in python3.13 python3.12 python3.11 python3.10 python3; do
|
||||
if command -v "$candidate" &>/dev/null; then
|
||||
ver=$("$candidate" -c "import sys; print(sys.version_info.minor)" 2>/dev/null)
|
||||
if [[ -n "$ver" ]] && (( ver >= REQUIRED_PYTHON_MINOR )); then
|
||||
PYTHON_BIN="$(command -v "$candidate")"
|
||||
break
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ -z "$PYTHON_BIN" ]]; then
|
||||
echo "ERROR: Python >= 3.${REQUIRED_PYTHON_MINOR} is required but not found."
|
||||
echo "Install it with: brew install python@3.13"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- Create virtual environment if it doesn't exist ---
|
||||
VENV_DIR="$BACKEND_DIR_ABSPATH/.venv"
|
||||
if [[ ! -d "$VENV_DIR" ]]; then
|
||||
echo "Creating virtual environment..."
|
||||
python3 -m venv "$VENV_DIR"
|
||||
echo "Creating virtual environment with $("$PYTHON_BIN" --version)..."
|
||||
"$PYTHON_BIN" -m venv "$VENV_DIR"
|
||||
fi
|
||||
|
||||
# --- Verify the venv Python meets the minimum version ---
|
||||
VENV_PYTHON="$VENV_DIR/bin/python3"
|
||||
VENV_VER=$("$VENV_PYTHON" -c "import sys; print(sys.version_info.minor)" 2>/dev/null)
|
||||
if [[ -z "$VENV_VER" ]] || (( VENV_VER < REQUIRED_PYTHON_MINOR )); then
|
||||
echo "Existing venv uses Python 3.${VENV_VER:-?}, need >= 3.${REQUIRED_PYTHON_MINOR}. Recreating..."
|
||||
rm -rf "$VENV_DIR"
|
||||
echo "Creating virtual environment with $("$PYTHON_BIN" --version)..."
|
||||
"$PYTHON_BIN" -m venv "$VENV_DIR"
|
||||
fi
|
||||
|
||||
source "$VENV_DIR/bin/activate"
|
||||
|
||||
# --- Upgrade pip if outdated ---
|
||||
pip3 install --upgrade pip --quiet
|
||||
|
||||
# --- Install custom debugger module if not already installed ---
|
||||
DEBUGGER_DIR_ABSPATH="$PROJECT_ROOT_ABSPATH/debugger"
|
||||
if ! pip3 show debug > /dev/null 2>&1; then
|
||||
@@ -58,4 +85,4 @@ echo "Starting backend server on http://0.0.0.0:8324 ..."
|
||||
cd "$PROJECT_ROOT_ABSPATH"
|
||||
python3 -m uvicorn backend.main:app --host 0.0.0.0 --port 8324 --reload \
|
||||
--reload-dir "$BACKEND_DIR_ABSPATH" \
|
||||
--reload-exclude '*.pyc'
|
||||
--reload-exclude '*.pyc'
|
||||
Reference in New Issue
Block a user