Commit Graph
20 Commits
Author SHA1 Message Date
j3ssie 0269cf4e26 feat: update Next.js build assets and add cloud setup E2E tests
- Update Next.js generated chunk hashes and build IDs reflecting latest dashboard build
- Update CSS stylesheet references in workflow upload page metadata
- Add comprehensive cloud setup E2E test suite (cloud_setup_test.go) with SSH password/key auth, post-command variable expansion, and Ansible integration
- Fix API priority levels to include 'medium' priority in test coverage
- Add agent-sdk test workflows (minimal, config, codex, multi-agent, session variants)
- Update E2E test utilities with runCLIInBase helper for multi-step cloud config tests
- Fix stderr/stdout capture in dependencies_target_types_test assertions
2026-04-04 13:57:34 +08:00
j3ssie 520977d1ea feat: Add agent-acp step type with ACP protocol support
- Implement agent-acp step type for spawning external ACP agent subprocesses via Agent Communication Protocol
- Add ACPExecutor with validation, field rendering, and subprocess lifecycle management
- Integrate agent-acp field rendering in StepDispatcher (batch and sequential modes)
- Add run_agent() utility function for workflows to execute ACP agents from steps and JS context
- Add osmedeus agent CLI command for interactive agent execution with --agent, --cwd, --timeout, --stdin, and --list flags
- Add /osm/api/agent/chat/completions REST endpoint with OpenAI-compatible chat format and concurrency control
- Support agent selection via: built-in names (claude-code, codex, opencode, gemini) or custom acp_config.command
- Add step-level configuration: cwd, allowed_paths, acp_config (command, args, env, write_enabled)
- Add comprehensive E2E tests for agent-acp workflows (basic, minimal, config, codex variants)
- Add test workflows in test/testdata/workflows/agent-and-llm/
- Update AGENTS.md documentation with agent-acp examples, CLI usage, and API endpoints
2026-02-27 16:18:06 +08:00
j3ssie 2bbf7295f6 refactor: enhance linter with comprehensive undefined variable detection
- Add platform variables (PlatformOS, PlatformArch, PlatformInDocker, PlatformInKubernetes, PlatformCloudProvider) to linter's built-in variables list
- Extract checkStepFieldsForUndefinedVars and collectReferencedVarsFromStep helper functions to support comprehensive scanning of all template-renderable fields including parallel_functions, speed_args, config_args, agent fields, HTTP headers, LLM messages, decision cases/conditions, and memory paths
- Add recursion for nested steps (foreach inner steps and parallel_steps) with proper variable scoping and support for loop-specific variables (_id_, foreach variable)
- Enhance message formatting with highlightQuoted() to make quoted terms bold+yellow in error output
- Enable UndefinedVariableRule by default in GetDefaultRules()
- Add comprehensive test coverage for newly supported fields and nested step types (TestUndefinedVariableRule_*, TestUnusedVariableRule_ReferencedInNewFields)
- Add test fixture and E2E test (TestWorkflow_Validate_UndefinedVarsComprehensive) validating detection across functions and foreach
2026-02-18 10:18:44 +07:00
j3ssie 0da63bf9b3 feat: improve decision conditions with param-based JS evaluation
- Add stripTemplateVarsForJS() to convert {{var}} outside quotes to bare JS variable names, enabling direct param/export access in condition expressions
- Add normalizeBoolStringsForJS() to convert string 'true'/'false' to actual booleans for correct JS truthiness evaluation
- Auto-classify asset types in db_import_asset_from_file() and default source to 'web' when empty
- Improve table display with terminal auto-width detection via term.GetSize(), custom column weights, and display name aliases (status_code → status)
- Change default --width to 0 (auto-detect) with weighted column distribution and minimum header-based sizing
- Add comprehensive tests for param-based conditions and JSON asset imports
- Update test data to use direct param references in conditions instead of exports
2026-02-17 22:05:18 +07:00
j3ssie fba43ba3b5 feat: add string utility functions and decision condition tests
- Add cut_to_file() and cut_space() utility functions for file processing and field extraction
- Add comprehensive E2E tests for decision condition routing with function/command execution
- Add test workflows for decision conditions and inline decision execution
- Add short-mode skip guards to all cloud E2E tests to allow quick test runs
- Register new functions in constants and goja runtime
2026-02-17 17:00:43 +07:00
j3ssie bd1434739e feat: add sudo authentication support with keepalive and variable renames
- Add sudo_auth() function for TTY prompts and credential management with optional 4-minute keepalive loop
- Add --sudo-aware CLI flag to detect workflows with sudo commands and offer authentication guidance
- Add sudo step scanner to detect sudo usage across all step types (bash, parallel, foreach) and nested structures
- Add parse_url_file() function to batch-process URLs with format directives, supporting bare IPs and CIDR notation
- Add portscan test data with realistic nmap JSONL samples
- Rename {{Workspace}} to {{TargetSpace}} in function examples for clarity
- Add sudo E2E tests covering tip message, flag acceptance, and non-sudo workflows
2026-02-16 23:46:33 +07:00
j3ssie baac7a016a feat: add worker management, hooks support, and db cleanup enhancements
- Add worker eval command for distributed function execution with Redis hooks registration
- Add worker set command to update worker fields (alias, public-ip, ssh-enabled, ssh-keys-path)
- Enhance worker status with JSON output, search filtering, and column selection (--columns, --exclude-columns, --search)
- Add --keep-setting flag to install base/validate commands to preserve osm-settings.yaml after base installation
- Fix binary installation in Nix: replace CopyInstalledBinaryToFolder with SymlinkInstalledBinaryToFolder
- Add --clean-ws flag to db clean command for removing workspace data
- Add HooksEnabled field to Run records when creating runs from CLI and API
- Add comprehensive test coverage for hook execution (pre/post hooks, execution order, failure handling)
- Add test coverage for worker commands (eval, set, status with JSON) and db clean operations
- Improve usage documentation for worker subcommands and db operations
2026-02-15 10:47:44 +07:00
j3ssie b6e9d12324 feat: implement path-friendly loop variables and improve asset merge logic
- Add auto-generated _<variable>_ path-friendly variables for foreach loops that sanitize unsafe filesystem characters (/, :, etc.) and deterministically truncate long values
- Refactor asset and vulnerability import/merge logic to preserve existing non-empty fields instead of full overwrite on conflict
- Add mergeAssetFields() and mergeVulnFields() helper functions for consistent field-level merge behavior across all import methods
- Add comprehensive unit tests for merge functions and path-friendly variable behavior in foreach loops
- Add E2E test module (test-foreach-path-friendly) validating sanitization, directory creation, truncation, and variable coexistence
2026-02-13 13:27:04 +07:00
j3ssie 4ac041fd73 feat: add workflow help metadata, artifact optional flag, and search/filter improvements
- Add WorkflowHelp struct with Usage and ExampleTargets for CLI documentation
- Add Optional field to Artifact model and database schema with migration support
- Implement workflow search functionality by name, description, and tags in CLI list command
- Add --usage and --search flags to workflow list command with multiple filtering options
- Display workflow usage info in show command when Help is defined
- Support help inheritance in workflow extends/inheritance resolver
- Update vulnerability counters from database after SARIF imports
- Add comprehensive Help unit tests covering parsing, cloning, and mutation isolation
- Improve test helpers with streaming output, diagnostics, and file validation utilities
- Add fourth general canary test for domain-list-recon flow with artifact validation
2026-02-12 01:04:03 +07:00
j3ssie 438d8ec138 feat: implement agent executor with tool calling, sub-agents, and comprehensive test suite
- Add AgentExecutor implementing LLM-based agentic loop with tool calling, max iterations, and stop conditions
- Introduce agent preset tools (bash, file_exists, http_get, run_module, etc.) with extensible registry pattern
- Add sub-agent spawning capability via spawn_agent tool call with recursive depth limits and validation
- Implement ToolExecutor for custom tool execution with template rendering and error handling
- Add agent session persistence and memory management with sliding window configuration
- Create comprehensive E2E test suite covering 15+ agent workflow scenarios (minimal, custom tools, planning, multi-goal, structured output, tracing hooks, file tools, orchestration, Python tools, sub-agents, nested sub-agents, and validation)
- Add agent-and-llm test data directory with 17 YAML workflow fixtures
- Update integration tests to include agent workflow directories
- Add AgentTool and AgentConfig types with validation for duplicate names and unknown presets
- Implement LLM streaming test utilities
- Update documentation (CLAUDE.md, HACKING.md, README.md) with agent features and CLI examples
2026-02-10 08:44:48 +07:00
j3ssie 7de3e7b2b2 feat: add performance optimizations and robustness improvements
- Add lock-free ResultCollector for parallel execution with atomic operations, eliminating mutex contention for pre-allocated slices
- Implement circuit breaker pattern (internal/retry/circuit_breaker) with configurable thresholds and half-open recovery state for fault tolerance
- Introduce json-iterator replacement (internal/json) for 2-6x faster JSON operations while maintaining stdlib compatibility
- Add lazy template rendering (RenderLazy) with variable reference caching for 50-80% faster rendering on large contexts
- Implement memory-efficient buffer pooling (bufpool) with 10MB pre-allocated reusable buffers for reduced GC pressure
- Add LoadFlowWithModules for parallel module pre-loading using errgroup, improving startup time for complex flows
- Add VarRefCache with LRU eviction for variable extraction caching
- Add streaming output support for foreach loops to process large datasets without memory accumulation
- Fix json import compatibility in llm_executor and db_functions
- Update test fixtures with correct YAML field names (call→function, run→command)
2026-01-31 01:26:28 +07:00
j3ssie 1a8ab856fa feat: add empty_target preference and FlowName variable, optimize concurrent file sync
- Add empty_target preference to allow running workflows without explicit target
- Implement FlowName template variable for module context awareness
- Optimize cdnDelete, SyncUpload, SyncDownload with worker pool concurrency
- Load workflow preferences early for --empty-target CLI flag support
- Add unit tests for preferences and E2E tests for FlowName functionality
- Update linter to recognize ModuleName and FlowName as built-in variables
2026-01-28 18:47:45 +08:00
j3ssie 890b7e3b90 feat: add inline module support and improve version output
- Add inline module support to ModuleRef allowing modules to be defined directly in flows without external files
- Implement IsInline() and ToWorkflow() methods to convert inline ModuleRef definitions to executable workflows
- Update ModuleRef.Clone() to properly duplicate all fields including Steps, Runner, RunnerConfig, and Description
- Add field alignment formatting and make path optional for inline modules
- Enhance parser validation to allow omitting path for inline modules while requiring at least one step
- Update executor's preloadModules() and ExecuteFlow() to handle inline modules during flow execution
- Improve version command output with JSON format support via --json flag and enhanced colored terminal output
2026-01-28 16:15:57 +08:00
j3ssie 5bb2ff7f0c build: update dashboard artifacts and add repeat-modules test workflow 2026-01-27 01:28:21 +08:00
j3ssie 7f339a69f0 feat: add run priority support and server submission API
- Add RunClient for submitting runs to server with priority support (low, normal, high, critical)
- Implement --run-priority and --server-url CLI flags for server submission mode
- Add RunPriority and RunMode fields to database Run model for persistence
- Update CreateRunRequest with priority and run_mode validation
- Implement runServerSubmission() to submit workflows with priority to server API
- Comprehensive E2E tests for run endpoints with priority validation and multiple targets
- Add foreach-preprocess test workflow with variable_pre_process support
2026-01-26 21:14:04 +08:00
j3ssie 777fb7c2bf chore: update ui build artifacts and refactor event trigger input syntax
- Update Next.js build IDs and chunk hashes in workflow upload static files
- Replace deprecated 'input' field syntax with new exports-style variables in event triggers (e.g., 'Target: event_data.value' instead of type/field/name)
- Simplify parameter override syntax in test workflows (remove nested 'default' structure)
2026-01-24 16:24:50 +08:00
j3ssie f5840272c5 feat: add run cancellation, event enhancements, and performance optimizations
Major features:
- Add run registry for tracking active runs with PID management
- Add API-based run cancellation with process termination
- Add event trigger input vars syntax for multi-variable extraction
- Add filter_functions with utility function support in triggers
- Add event envelope injection for full event context in workflows
- Add write coordinator for batched database operations

API improvements:
- Add logout endpoint and diffs endpoints for assets/vulnerabilities
- Add step-results listing endpoint
- Update schedule model with target, workspace, params fields
- Change run_id to run_uuid across API responses

Performance:
- Add compiled JS program caching for 60-80% faster loop conditions
- Add parallel shard rendering for 20-40% faster workflow startup
- Add memory-mapped I/O for large file line counting
- Add efficient output buffer combining in runners
- Add mtime-based cache invalidation for workflow loader

Other changes:
- Rename trigger field from trigger to triggers in workflow YAML
- Disable pongo2 HTML autoescape for shell command templates
- Update JWT expiration default to 1440 minutes (1 day)
- Change CORS default to reflect-origin for credentials support
- Add source_type field to events (run, eval, api)
- Skip copying core Unix tools to external-binaries
2026-01-24 01:11:33 +08:00
j3ssie e28c9dce50 refactor: split RunID into RunUUID and ID, compute workspace dynamically, improve error handling 2026-01-22 13:28:59 +08:00
j3ssie 1403d20a4d feat: add LLM step executor with vision and tool support, event workflow system, and inheritance
- Add LLM executor supporting OpenAI vision, tool calling, embeddings, and structured outputs
- Introduce event emitter/receiver workflows with deduplication and filtering (generate_event functions)
- Add workflow extends/override system enabling inheritance chains and step merge modes
- Update function naming to snake_case across all testdata (fileExists→file_exists, etc.)
- Add comprehensive test fixtures for linter, events, CDN, step dependencies, and extends workflows
2026-01-20 18:23:57 +08:00
j3ssie 7a2c5a5dc9 Complete rewrite and re-architecture Osmedeus Engine in v5 2026-01-18 19:32:24 +08:00