Commit Graph
34 Commits
Author SHA1 Message Date
j3ssie 896527421d fix(executor): preserve unrendered threads template and tidy run output
- Emit the raw `threads` value in step command summaries so template variables
  like `{{deparosParallel}}` survive until templateEngine.Render resolves them,
  instead of silently collapsing to 1 when Int() fails to parse the template
- Only print the Artifacts section in `printResultSummary` when -v/--verbose
  is set, removing state/log file clutter from default run output
- Update embedded UI build artifacts under public/ui/_next to the latest
  dashboard build (chunk renames, manifest updates, page txt/html refreshes)
2026-06-01 12:43:00 +08:00
j3ssie 8448791c3c feat: propagate run cancellation to tmux sessions and remote SSH processes
- Track tmux sessions in ActiveRun and kill them on run cancel via new
  TmuxHooks indirection; expose killed_tmux_sessions in CLI and API responses
- Add ExecuteSSHCommand with remote pidfile + process-group kill watcher so
  cancelling a run actually terminates remote scans (not just the local session)
- Route ssh_exec/ssh_rsync/sync_* through the run's cancellable context via
  new RunContextHooks
- Switch docker-publish to sequential per-arch buildx builds + imagetools
  manifest to avoid OOM on multi-arch builds; add docker-buildx-setup target
- Cross-compile Dockerfile via BUILDPLATFORM/TARGETOS/TARGETARCH and retry
  SAST binary installs to survive QEMU-flaky downloads
- Bump version to v5.0.3
2026-05-26 00:06:26 +08:00
Georg HeindlandCopilot 4ea9fda1bb Return artifact paths
Co-authored-by: Copilot <copilot@github.com>
2026-04-23 20:07:00 +02:00
Georg Heindl 771c000e92 Empty error panic on completion fixed 2026-04-23 19:53:24 +02:00
Georg HeindlandCopilot 39827b8b99 Included error, artifacts, message
Co-authored-by: Copilot <copilot@github.com>
2026-04-23 19:27:35 +02:00
Georg Heindl ce7199753f rm error 2026-04-23 17:52:00 +02:00
Georg HeindlandCopilot 67dc6b3de7 Added runUUID, error, steps, exports to webhook notification
Co-authored-by: Copilot <copilot@github.com>
2026-04-23 16:56:28 +02:00
Georg Heindl 29efc1b8e5 Remove duplicate condition check 2026-04-23 14:32:10 +02:00
Georg Heindl a4df58e063 Added on_error actions: Notify, Continue, Abort and run type function 2026-04-22 14:47:45 +02:00
Georg HeindlandGitHub fc827be975 Webhook Notification on every result path of ExecuteFlow/Module 2026-04-22 14:31:23 +02:00
j3ssie 0269cf4e26 feat: update Next.js build assets and add cloud setup E2E tests
- Update Next.js generated chunk hashes and build IDs reflecting latest dashboard build
- Update CSS stylesheet references in workflow upload page metadata
- Add comprehensive cloud setup E2E test suite (cloud_setup_test.go) with SSH password/key auth, post-command variable expansion, and Ansible integration
- Fix API priority levels to include 'medium' priority in test coverage
- Add agent-sdk test workflows (minimal, config, codex, multi-agent, session variants)
- Update E2E test utilities with runCLIInBase helper for multi-step cloud config tests
- Fix stderr/stdout capture in dependencies_target_types_test assertions
2026-04-04 13:57:34 +08:00
j3ssie 0da63bf9b3 feat: improve decision conditions with param-based JS evaluation
- Add stripTemplateVarsForJS() to convert {{var}} outside quotes to bare JS variable names, enabling direct param/export access in condition expressions
- Add normalizeBoolStringsForJS() to convert string 'true'/'false' to actual booleans for correct JS truthiness evaluation
- Auto-classify asset types in db_import_asset_from_file() and default source to 'web' when empty
- Improve table display with terminal auto-width detection via term.GetSize(), custom column weights, and display name aliases (status_code → status)
- Change default --width to 0 (auto-detect) with weighted column distribution and minimum header-based sizing
- Add comprehensive tests for param-based conditions and JSON asset imports
- Update test data to use direct param references in conditions instead of exports
2026-02-17 22:05:18 +07:00
j3ssie fba43ba3b5 feat: add string utility functions and decision condition tests
- Add cut_to_file() and cut_space() utility functions for file processing and field extraction
- Add comprehensive E2E tests for decision condition routing with function/command execution
- Add test workflows for decision conditions and inline decision execution
- Add short-mode skip guards to all cloud E2E tests to allow quick test runs
- Register new functions in constants and goja runtime
2026-02-17 17:00:43 +07:00
j3ssie 8d413aecb9 feat: table rendering, cron schedules, and assets command
- Refactor table display using tablewriter with box-drawing characters and proper Unicode width handling
- Add --as-cron flag to register cron schedules instead of executing immediately
- Implement new 'osmedeus assets' command for asset querying with filtering and statistics
- Enhance workflow parameter display with categorization and per-module summaries
- Improve column width handling and parameter documentation
2026-02-17 15:19:49 +07:00
j3ssie bd1434739e feat: add sudo authentication support with keepalive and variable renames
- Add sudo_auth() function for TTY prompts and credential management with optional 4-minute keepalive loop
- Add --sudo-aware CLI flag to detect workflows with sudo commands and offer authentication guidance
- Add sudo step scanner to detect sudo usage across all step types (bash, parallel, foreach) and nested structures
- Add parse_url_file() function to batch-process URLs with format directives, supporting bare IPs and CIDR notation
- Add portscan test data with realistic nmap JSONL samples
- Rename {{Workspace}} to {{TargetSpace}} in function examples for clarity
- Add sudo E2E tests covering tip message, flag acceptance, and non-sudo workflows
2026-02-16 23:46:33 +07:00
j3ssie d5b5564149 feat: add queue system for delayed task execution
- Add QueuePoller for dual-source (DB + Redis) task polling with concurrent execution
- Add worker queue subcommands (list, new, run) for managing queued tasks
- Add uninstall command to safely remove Osmedeus installation
- Add input validation to prevent command injection via dangerous shell characters
- Add queue run support via --queue and --queue-run flags in osmedeus run
- Add queue polling to server with --no-queue-polling flag to disable it
- Support queuing tasks from both CLI and API with deduplication across sources
2026-02-15 18:13:57 +07:00
j3ssie baac7a016a feat: add worker management, hooks support, and db cleanup enhancements
- Add worker eval command for distributed function execution with Redis hooks registration
- Add worker set command to update worker fields (alias, public-ip, ssh-enabled, ssh-keys-path)
- Enhance worker status with JSON output, search filtering, and column selection (--columns, --exclude-columns, --search)
- Add --keep-setting flag to install base/validate commands to preserve osm-settings.yaml after base installation
- Fix binary installation in Nix: replace CopyInstalledBinaryToFolder with SymlinkInstalledBinaryToFolder
- Add --clean-ws flag to db clean command for removing workspace data
- Add HooksEnabled field to Run records when creating runs from CLI and API
- Add comprehensive test coverage for hook execution (pre/post hooks, execution order, failure handling)
- Add test coverage for worker commands (eval, set, status with JSON) and db clean operations
- Improve usage documentation for worker subcommands and db operations
2026-02-15 10:47:44 +07:00
j3ssie b0736ab0ed feat: add skip() function and fuzzy module exclusion support
- Add skip() function to terminate remaining steps in current module while continuing to next module, with optional message parameter
- Implement isFuzzyModuleExcluded() for substring-based module filtering in ExecuteFlow
- Add fuzzy_exclude_modules CLI flag (-X) to both run and scan commands for flexible module exclusion
- Handle ErrSkipModule sentinel error throughout executor (executeStep, executeStepsDAG, ExecuteModule, ExecuteFlow) with proper status propagation
- Update function registry and Goja runtime to register skip() function
- Add comprehensive unit tests for skip() behavior, SkipModuleError, and fuzzy module matching
- Update snapshot tests to use generic example.com instead of shopee.vn
2026-02-13 15:56:02 +07:00
j3ssie 167fff791d feat: add repo slug extraction for code hosting URLs and enhance archive format detection
- Extract 'owner__repo' slugs from code hosting platforms (GitHub, GitLab, Bitbucket, Codeberg) for workspace naming preference
- Implement magic byte detection for archive formats in extractTo() function to support headerless archives
- Implement GetRunSteps() and GetRunArtifacts() API endpoints with database queries
- Format code for consistency in constants, type mappings, and test files
2026-02-10 18:32:06 +07:00
j3ssie 438d8ec138 feat: implement agent executor with tool calling, sub-agents, and comprehensive test suite
- Add AgentExecutor implementing LLM-based agentic loop with tool calling, max iterations, and stop conditions
- Introduce agent preset tools (bash, file_exists, http_get, run_module, etc.) with extensible registry pattern
- Add sub-agent spawning capability via spawn_agent tool call with recursive depth limits and validation
- Implement ToolExecutor for custom tool execution with template rendering and error handling
- Add agent session persistence and memory management with sliding window configuration
- Create comprehensive E2E test suite covering 15+ agent workflow scenarios (minimal, custom tools, planning, multi-goal, structured output, tracing hooks, file tools, orchestration, Python tools, sub-agents, nested sub-agents, and validation)
- Add agent-and-llm test data directory with 17 YAML workflow fixtures
- Update integration tests to include agent workflow directories
- Add AgentTool and AgentConfig types with validation for duplicate names and unknown presets
- Implement LLM streaming test utilities
- Update documentation (CLAUDE.md, HACKING.md, README.md) with agent features and CLI examples
2026-02-10 08:44:48 +07:00
j3ssie 87e20967f1 feat: add file type checking, git subfolder cloning, and parallel wget download
- Add is_file, is_dir, is_git functions for type validation
- Add is_compress function to detect compressed file extensions
- Implement pure Go wget with parallel segmented downloads (>1MB files)
- Add git_clone_subfolder with ZIP fallback for GitHub repos
- Add helper utilities: ZIP extraction, directory/file copying
- Add SkipWorkspace option to skip output dirs in empty-target mode
- Fix workflow preference loading order for --module-url flag
- Add 20+ comprehensive tests for all new functions
2026-02-08 00:34:47 +07:00
j3ssieandAmp b80c8b6ecd feat: add SSH test infrastructure, improve executor temp handling, enhance CLI workflow install
Amp-Thread-ID: https://ampcode.com/threads/T-019c195d-0f3b-724a-946a-a3a93dd7d09b
Co-authored-by: Amp <amp@ampcode.com>
2026-02-01 20:21:05 +07:00
j3ssie 7de3e7b2b2 feat: add performance optimizations and robustness improvements
- Add lock-free ResultCollector for parallel execution with atomic operations, eliminating mutex contention for pre-allocated slices
- Implement circuit breaker pattern (internal/retry/circuit_breaker) with configurable thresholds and half-open recovery state for fault tolerance
- Introduce json-iterator replacement (internal/json) for 2-6x faster JSON operations while maintaining stdlib compatibility
- Add lazy template rendering (RenderLazy) with variable reference caching for 50-80% faster rendering on large contexts
- Implement memory-efficient buffer pooling (bufpool) with 10MB pre-allocated reusable buffers for reduced GC pressure
- Add LoadFlowWithModules for parallel module pre-loading using errgroup, improving startup time for complex flows
- Add VarRefCache with LRU eviction for variable extraction caching
- Add streaming output support for foreach loops to process large datasets without memory accumulation
- Fix json import compatibility in llm_executor and db_functions
- Update test fixtures with correct YAML field names (call→function, run→command)
2026-01-31 01:26:28 +07:00
j3ssie 1a8ab856fa feat: add empty_target preference and FlowName variable, optimize concurrent file sync
- Add empty_target preference to allow running workflows without explicit target
- Implement FlowName template variable for module context awareness
- Optimize cdnDelete, SyncUpload, SyncDownload with worker pool concurrency
- Load workflow preferences early for --empty-target CLI flag support
- Add unit tests for preferences and E2E tests for FlowName functionality
- Update linter to recognize ModuleName and FlowName as built-in variables
2026-01-28 18:47:45 +08:00
j3ssie 890b7e3b90 feat: add inline module support and improve version output
- Add inline module support to ModuleRef allowing modules to be defined directly in flows without external files
- Implement IsInline() and ToWorkflow() methods to convert inline ModuleRef definitions to executable workflows
- Update ModuleRef.Clone() to properly duplicate all fields including Steps, Runner, RunnerConfig, and Description
- Add field alignment formatting and make path optional for inline modules
- Enhance parser validation to allow omitting path for inline modules while requiring at least one step
- Update executor's preloadModules() and ExecuteFlow() to handle inline modules during flow execution
- Improve version command output with JSON format support via --json flag and enhanced colored terminal output
2026-01-28 16:15:57 +08:00
j3ssie 7f339a69f0 feat: add run priority support and server submission API
- Add RunClient for submitting runs to server with priority support (low, normal, high, critical)
- Implement --run-priority and --server-url CLI flags for server submission mode
- Add RunPriority and RunMode fields to database Run model for persistence
- Update CreateRunRequest with priority and run_mode validation
- Implement runServerSubmission() to submit workflows with priority to server API
- Comprehensive E2E tests for run endpoints with priority validation and multiple targets
- Add foreach-preprocess test workflow with variable_pre_process support
2026-01-26 21:14:04 +08:00
j3ssie bef52a2223 feat: add github-action make target and refactor builtin variables initialization
- Add github-action target to trigger manual and nightly release workflows
- Extract BuildBuiltinVariables() function for reusable context setup
- Simplify RandomString from 8-char alphanumeric to 6-char lowercase
- Update documentation for RandomString variable description
2026-01-25 23:37:01 +08:00
j3ssie e7eea69a40 feat: expose platform variables in osmedeus eval command
Export platform detection functions (DetectDocker, DetectKubernetes,
DetectCloudProvider) from internal/executor to allow reuse in CLI.
Inject PlatformOS, PlatformArch, PlatformInDocker, PlatformInKubernetes,
and PlatformCloudProvider variables in func eval command context.
2026-01-24 16:22:44 +08:00
j3ssie f5840272c5 feat: add run cancellation, event enhancements, and performance optimizations
Major features:
- Add run registry for tracking active runs with PID management
- Add API-based run cancellation with process termination
- Add event trigger input vars syntax for multi-variable extraction
- Add filter_functions with utility function support in triggers
- Add event envelope injection for full event context in workflows
- Add write coordinator for batched database operations

API improvements:
- Add logout endpoint and diffs endpoints for assets/vulnerabilities
- Add step-results listing endpoint
- Update schedule model with target, workspace, params fields
- Change run_id to run_uuid across API responses

Performance:
- Add compiled JS program caching for 60-80% faster loop conditions
- Add parallel shard rendering for 20-40% faster workflow startup
- Add memory-mapped I/O for large file line counting
- Add efficient output buffer combining in runners
- Add mtime-based cache invalidation for workflow loader

Other changes:
- Rename trigger field from trigger to triggers in workflow YAML
- Disable pongo2 HTML autoescape for shell command templates
- Update JWT expiration default to 1440 minutes (1 day)
- Change CORS default to reflect-origin for credentials support
- Add source_type field to events (run, eval, api)
- Skip copying core Unix tools to external-binaries
2026-01-24 01:11:33 +08:00
j3ssie e28c9dce50 refactor: split RunID into RunUUID and ID, compute workspace dynamically, improve error handling 2026-01-22 13:28:59 +08:00
j3ssie 344e6117ca feat: enhance target handling and improve release/install workflows
- Add file target support to heuristics with automatic TargetSpace derivation and comprehensive unit tests for file path parsing
- Display target space folder location during workflow execution for both module and flow runs
- Fix IP address handling in URL parsing to correctly identify and extract root domain for IP targets
- Add goreleaser --mark-latest flag to manual release workflow for consistent release tagging
- Improve install binary help text and add progress feedback for silent mode installations
- Replace deprecated --list-registry-binaries with --list-registry-direct-fetch and --list-registry-nix-build examples
2026-01-21 17:20:52 +08:00
j3ssie 1403d20a4d feat: add LLM step executor with vision and tool support, event workflow system, and inheritance
- Add LLM executor supporting OpenAI vision, tool calling, embeddings, and structured outputs
- Introduce event emitter/receiver workflows with deduplication and filtering (generate_event functions)
- Add workflow extends/override system enabling inheritance chains and step merge modes
- Update function naming to snake_case across all testdata (fileExists→file_exists, etc.)
- Add comprehensive test fixtures for linter, events, CDN, step dependencies, and extends workflows
2026-01-20 18:23:57 +08:00
j3ssie 9ed02e7eee refactor: code quality improvements, Docker updates, and setup enhancements
- chore: fix code style and remove unused error handling (add blank checks for closed file handles)
- chore: align struct field padding in multiple files for consistency
- fix: add 386 architecture support to goreleaser build config
- feat: add docker-publish target to Makefile for publishing to Docker Hub
- feat: refactor first-time setup into reusable installRequiredBinaries helper function
- feat: add initialization marker tracking for first-time setup completion
- feat: enhance workflow YAML detection to skip non-workflow files and hidden directories
- feat: improve database column display defaults (assets and vulnerabilities tables)
- feat: add fallback mechanism to install.sh for version detection failures
- fix: correct tarball filename generation by stripping 'v' prefix
- chore: update Docker base image from golang:1.22 to golang:1.25
- chore: update goreleaser release flags and simplify Docker build naming
- chore: fix import ordering across multiple files (alphabetical consistency)
- chore: improve install script with better version display formatting
- chore: reduce binaries per row from 10 to 6 in CLI output for better readability
2026-01-19 01:17:04 +08:00
j3ssie 7a2c5a5dc9 Complete rewrite and re-architecture Osmedeus Engine in v5 2026-01-18 19:32:24 +08:00