Commit Graph
22 Commits
Author SHA1 Message Date
j3ssie 896527421d fix(executor): preserve unrendered threads template and tidy run output
- Emit the raw `threads` value in step command summaries so template variables
  like `{{deparosParallel}}` survive until templateEngine.Render resolves them,
  instead of silently collapsing to 1 when Int() fails to parse the template
- Only print the Artifacts section in `printResultSummary` when -v/--verbose
  is set, removing state/log file clutter from default run output
- Update embedded UI build artifacts under public/ui/_next to the latest
  dashboard build (chunk renames, manifest updates, page txt/html refreshes)
2026-06-01 12:43:00 +08:00
j3ssie 8ec1de4f84 feat(db): add db_import_vigolium importer and harden CLI run
- Add db_import_vigolium JS function that routes vigolium JSONL records
  by envelope type: http_record -> assets, finding -> vulnerabilities
  (deduped on new finding_hash column), skipping scan/oast_interaction
- Add FindingHash field to Vulnerability model with idempotent ALTER
  TABLE migration and matching index
- Suppress run errors when using --silent --empty-target placeholder mode
- Retry transient fetchURLContent failures (network errors, 408/429/5xx)
  with exponential backoff; leave 4xx untouched so GitHub auth fallback
  can engage
- Add unit tests for the vigolium importer (import + idempotency) and
  the HTTP retry behavior, plus a vigolium juice-shop sample fixture
- Bump katana, naabu, kingfisher, bearer registry entries and add
  vigolium to the direct-fetch registry
2026-05-29 23:31:56 +08:00
j3ssie 0269cf4e26 feat: update Next.js build assets and add cloud setup E2E tests
- Update Next.js generated chunk hashes and build IDs reflecting latest dashboard build
- Update CSS stylesheet references in workflow upload page metadata
- Add comprehensive cloud setup E2E test suite (cloud_setup_test.go) with SSH password/key auth, post-command variable expansion, and Ansible integration
- Fix API priority levels to include 'medium' priority in test coverage
- Add agent-sdk test workflows (minimal, config, codex, multi-agent, session variants)
- Update E2E test utilities with runCLIInBase helper for multi-step cloud config tests
- Fix stderr/stdout capture in dependencies_target_types_test assertions
2026-04-04 13:57:34 +08:00
j3ssie 8d413aecb9 feat: table rendering, cron schedules, and assets command
- Refactor table display using tablewriter with box-drawing characters and proper Unicode width handling
- Add --as-cron flag to register cron schedules instead of executing immediately
- Implement new 'osmedeus assets' command for asset querying with filtering and statistics
- Enhance workflow parameter display with categorization and per-module summaries
- Improve column width handling and parameter documentation
2026-02-17 15:19:49 +07:00
j3ssie d01bd326df feat: add jsonl_rename_key utility function and remove tablewriter dependency
- Add jsonl_rename_key() function to rename keys in JSONL files with mapping syntax 'old1:new1,old2:new2'
- Implements fast JSON parsing with fastjson and efficient buffered I/O for large files
- Include comprehensive test coverage with 7 test cases for edge cases and valid mappings
- Remove unused tablewriter dependency from go.mod/go.sum and internal/terminal/table.go
- Refactor printResultSummary() to use markdown table printing instead of tablewriter
- Add test data file ffuf-result.jsonl for integration testing
- Fix function list width default calculation in function.go
2026-02-17 01:43:04 +07:00
j3ssie bd1434739e feat: add sudo authentication support with keepalive and variable renames
- Add sudo_auth() function for TTY prompts and credential management with optional 4-minute keepalive loop
- Add --sudo-aware CLI flag to detect workflows with sudo commands and offer authentication guidance
- Add sudo step scanner to detect sudo usage across all step types (bash, parallel, foreach) and nested structures
- Add parse_url_file() function to batch-process URLs with format directives, supporting bare IPs and CIDR notation
- Add portscan test data with realistic nmap JSONL samples
- Rename {{Workspace}} to {{TargetSpace}} in function examples for clarity
- Add sudo E2E tests covering tip message, flag acceptance, and non-sudo workflows
2026-02-16 23:46:33 +07:00
j3ssie 459d5939fb feat: add cloud infrastructure and nmap/tmux function support
- Implement cloud provider infrastructure (DigitalOcean, AWS, GCP, Linode, Azure) with Pulumi integration for distributed scanning
- Add nmap and tmux utility functions for port scanning results processing and long-running background session management
- Introduce webhook-triggered run execution with unique UUID and authentication key support for external integrations
2026-02-16 15:23:24 +07:00
j3ssie d5b5564149 feat: add queue system for delayed task execution
- Add QueuePoller for dual-source (DB + Redis) task polling with concurrent execution
- Add worker queue subcommands (list, new, run) for managing queued tasks
- Add uninstall command to safely remove Osmedeus installation
- Add input validation to prevent command injection via dangerous shell characters
- Add queue run support via --queue and --queue-run flags in osmedeus run
- Add queue polling to server with --no-queue-polling flag to disable it
- Support queuing tasks from both CLI and API with deduplication across sources
2026-02-15 18:13:57 +07:00
j3ssie baac7a016a feat: add worker management, hooks support, and db cleanup enhancements
- Add worker eval command for distributed function execution with Redis hooks registration
- Add worker set command to update worker fields (alias, public-ip, ssh-enabled, ssh-keys-path)
- Enhance worker status with JSON output, search filtering, and column selection (--columns, --exclude-columns, --search)
- Add --keep-setting flag to install base/validate commands to preserve osm-settings.yaml after base installation
- Fix binary installation in Nix: replace CopyInstalledBinaryToFolder with SymlinkInstalledBinaryToFolder
- Add --clean-ws flag to db clean command for removing workspace data
- Add HooksEnabled field to Run records when creating runs from CLI and API
- Add comprehensive test coverage for hook execution (pre/post hooks, execution order, failure handling)
- Add test coverage for worker commands (eval, set, status with JSON) and db clean operations
- Improve usage documentation for worker subcommands and db operations
2026-02-15 10:47:44 +07:00
j3ssie b0736ab0ed feat: add skip() function and fuzzy module exclusion support
- Add skip() function to terminate remaining steps in current module while continuing to next module, with optional message parameter
- Implement isFuzzyModuleExcluded() for substring-based module filtering in ExecuteFlow
- Add fuzzy_exclude_modules CLI flag (-X) to both run and scan commands for flexible module exclusion
- Handle ErrSkipModule sentinel error throughout executor (executeStep, executeStepsDAG, ExecuteModule, ExecuteFlow) with proper status propagation
- Update function registry and Goja runtime to register skip() function
- Add comprehensive unit tests for skip() behavior, SkipModuleError, and fuzzy module matching
- Update snapshot tests to use generic example.com instead of shopee.vn
2026-02-13 15:56:02 +07:00
j3ssie 87e20967f1 feat: add file type checking, git subfolder cloning, and parallel wget download
- Add is_file, is_dir, is_git functions for type validation
- Add is_compress function to detect compressed file extensions
- Implement pure Go wget with parallel segmented downloads (>1MB files)
- Add git_clone_subfolder with ZIP fallback for GitHub repos
- Add helper utilities: ZIP extraction, directory/file copying
- Add SkipWorkspace option to skip output dirs in empty-target mode
- Fix workflow preference loading order for --module-url flag
- Add 20+ comprehensive tests for all new functions
2026-02-08 00:34:47 +07:00
j3ssie 641cf568aa docs: add security disclaimer and annotate intentional code execution points
- Add disclaimer section in README.md warning users about arbitrary code execution risks
- Annotate exec.Command calls throughout codebase with @NOTE comments explaining intentional design
- Improve step result table formatting: add column alignment, padding for status symbols, and remove unnecessary borders
2026-02-04 16:39:19 +07:00
j3ssieandAmp b80c8b6ecd feat: add SSH test infrastructure, improve executor temp handling, enhance CLI workflow install
Amp-Thread-ID: https://ampcode.com/threads/T-019c195d-0f3b-724a-946a-a3a93dd7d09b
Co-authored-by: Amp <amp@ampcode.com>
2026-02-01 20:21:05 +07:00
j3ssie 1a8ab856fa feat: add empty_target preference and FlowName variable, optimize concurrent file sync
- Add empty_target preference to allow running workflows without explicit target
- Implement FlowName template variable for module context awareness
- Optimize cdnDelete, SyncUpload, SyncDownload with worker pool concurrency
- Load workflow preferences early for --empty-target CLI flag support
- Add unit tests for preferences and E2E tests for FlowName functionality
- Update linter to recognize ModuleName and FlowName as built-in variables
2026-01-28 18:47:45 +08:00
j3ssie 7f339a69f0 feat: add run priority support and server submission API
- Add RunClient for submitting runs to server with priority support (low, normal, high, critical)
- Implement --run-priority and --server-url CLI flags for server submission mode
- Add RunPriority and RunMode fields to database Run model for persistence
- Update CreateRunRequest with priority and run_mode validation
- Implement runServerSubmission() to submit workflows with priority to server API
- Comprehensive E2E tests for run endpoints with priority validation and multiple targets
- Add foreach-preprocess test workflow with variable_pre_process support
2026-01-26 21:14:04 +08:00
j3ssie 777fb7c2bf chore: update ui build artifacts and refactor event trigger input syntax
- Update Next.js build IDs and chunk hashes in workflow upload static files
- Replace deprecated 'input' field syntax with new exports-style variables in event triggers (e.g., 'Target: event_data.value' instead of type/field/name)
- Simplify parameter override syntax in test workflows (remove nested 'default' structure)
2026-01-24 16:24:50 +08:00
j3ssie f5840272c5 feat: add run cancellation, event enhancements, and performance optimizations
Major features:
- Add run registry for tracking active runs with PID management
- Add API-based run cancellation with process termination
- Add event trigger input vars syntax for multi-variable extraction
- Add filter_functions with utility function support in triggers
- Add event envelope injection for full event context in workflows
- Add write coordinator for batched database operations

API improvements:
- Add logout endpoint and diffs endpoints for assets/vulnerabilities
- Add step-results listing endpoint
- Update schedule model with target, workspace, params fields
- Change run_id to run_uuid across API responses

Performance:
- Add compiled JS program caching for 60-80% faster loop conditions
- Add parallel shard rendering for 20-40% faster workflow startup
- Add memory-mapped I/O for large file line counting
- Add efficient output buffer combining in runners
- Add mtime-based cache invalidation for workflow loader

Other changes:
- Rename trigger field from trigger to triggers in workflow YAML
- Disable pongo2 HTML autoescape for shell command templates
- Update JWT expiration default to 1440 minutes (1 day)
- Change CORS default to reflect-origin for credentials support
- Add source_type field to events (run, eval, api)
- Skip copying core Unix tools to external-binaries
2026-01-24 01:11:33 +08:00
j3ssie e28c9dce50 refactor: split RunID into RunUUID and ID, compute workspace dynamically, improve error handling 2026-01-22 13:28:59 +08:00
j3ssie d44c94ef4a feat: enhance documentation and server startup flow
- Add comprehensive API and CLI documentation for event receiver, asset/vulnerability diffing, settings management, and function evaluation
- Improve event receiver startup by separating StartEventReceiver() from StartListener() for better control over trigger registration timing
- Update README with new command examples for db list, func eval, and event-driven triggers
- Refactor PrintStartupInfo to display detailed trigger information with tree formatting and better visual hierarchy
- Change trigger registration logging from Info to Debug level to reduce noise
- Clarify --workspace flag documentation in run/scan commands
- Add Type field to EventReceiverTriggerInfo for better trigger visibility in API responses
2026-01-20 21:24:52 +08:00
j3ssie 1403d20a4d feat: add LLM step executor with vision and tool support, event workflow system, and inheritance
- Add LLM executor supporting OpenAI vision, tool calling, embeddings, and structured outputs
- Introduce event emitter/receiver workflows with deduplication and filtering (generate_event functions)
- Add workflow extends/override system enabling inheritance chains and step merge modes
- Update function naming to snake_case across all testdata (fileExists→file_exists, etc.)
- Add comprehensive test fixtures for linter, events, CDN, step dependencies, and extends workflows
2026-01-20 18:23:57 +08:00
j3ssie 9ed02e7eee refactor: code quality improvements, Docker updates, and setup enhancements
- chore: fix code style and remove unused error handling (add blank checks for closed file handles)
- chore: align struct field padding in multiple files for consistency
- fix: add 386 architecture support to goreleaser build config
- feat: add docker-publish target to Makefile for publishing to Docker Hub
- feat: refactor first-time setup into reusable installRequiredBinaries helper function
- feat: add initialization marker tracking for first-time setup completion
- feat: enhance workflow YAML detection to skip non-workflow files and hidden directories
- feat: improve database column display defaults (assets and vulnerabilities tables)
- feat: add fallback mechanism to install.sh for version detection failures
- fix: correct tarball filename generation by stripping 'v' prefix
- chore: update Docker base image from golang:1.22 to golang:1.25
- chore: update goreleaser release flags and simplify Docker build naming
- chore: fix import ordering across multiple files (alphabetical consistency)
- chore: improve install script with better version display formatting
- chore: reduce binaries per row from 10 to 6 in CLI output for better readability
2026-01-19 01:17:04 +08:00
j3ssie 7a2c5a5dc9 Complete rewrite and re-architecture Osmedeus Engine in v5 2026-01-18 19:32:24 +08:00