Commit Graph
22 Commits
Author SHA1 Message Date
j3ssie 23ede677c4 feat: add extract_to function and optimize docker builds
- Add extract_to() utility function for auto-detecting archive formats (.zip, .tar.gz, .tar.bz2, .tar.xz, .tgz) and extracting with automatic destination cleanup
- Refactor Dockerfile to use slim debian base with install script, reducing image complexity and improving build reproducibility
- Update Dockerfile.dev to match production environment (Go 1.25 with full toolchain) while adding preset workflow initialization
- Update trivy binary registry metadata from v0.69.0 to v0.69.1 with direct platform-specific download URLs replacing dual curl installation script
2026-02-10 17:09:27 +07:00
j3ssie 2bc0e1e70a feat: add SARIF parsing and analysis functions for security tool integration
- Add sarif_functions.go with SARIF format parsing and vulnerability extraction (parse_sarif, extract_sarif_results, count_sarif_by_level)
- Add comprehensive sarif_functions_test.go with 239 lines of test coverage
- Add SARIF-sourced vulnerabilities to seed database including findings from Semgrep, Trivy, Kingfisher, and Bearer tools
- Add Bearer API key detection SARIF sample for test data
- Support asset_type='repo' for code-level findings from static analysis tools
- Integrate sarif functions into goja_runtime and function constants
- Test data includes real-world SARIF reports demonstrating container, dependency, and secret detection findings
2026-02-10 14:02:44 +07:00
j3ssie 9347795cbf feat: add detect_language utility function for source code language detection
- Implemented detect_language() function that analyzes directory structure to identify dominant programming language
- Detects language by file extension mapping (26+ supported languages: Go, Python, JS, TS, Rust, Java, etc.)
- Falls back to shebang detection for extensionless files (e.g., shell scripts)
- Intelligently skips non-source directories (node_modules, vendor, test, .git, etc.)
- Includes comprehensive test coverage for pure/mixed projects, edge cases, and shebang patterns
- Updated function registry and goja runtime to expose detect_language to workflows
2026-02-10 12:05:24 +07:00
j3ssie 438d8ec138 feat: implement agent executor with tool calling, sub-agents, and comprehensive test suite
- Add AgentExecutor implementing LLM-based agentic loop with tool calling, max iterations, and stop conditions
- Introduce agent preset tools (bash, file_exists, http_get, run_module, etc.) with extensible registry pattern
- Add sub-agent spawning capability via spawn_agent tool call with recursive depth limits and validation
- Implement ToolExecutor for custom tool execution with template rendering and error handling
- Add agent session persistence and memory management with sliding window configuration
- Create comprehensive E2E test suite covering 15+ agent workflow scenarios (minimal, custom tools, planning, multi-goal, structured output, tracing hooks, file tools, orchestration, Python tools, sub-agents, nested sub-agents, and validation)
- Add agent-and-llm test data directory with 17 YAML workflow fixtures
- Update integration tests to include agent workflow directories
- Add AgentTool and AgentConfig types with validation for duplicate names and unknown presets
- Implement LLM streaming test utilities
- Update documentation (CLAUDE.md, HACKING.md, README.md) with agent features and CLI examples
2026-02-10 08:44:48 +07:00
j3ssie 87e20967f1 feat: add file type checking, git subfolder cloning, and parallel wget download
- Add is_file, is_dir, is_git functions for type validation
- Add is_compress function to detect compressed file extensions
- Implement pure Go wget with parallel segmented downloads (>1MB files)
- Add git_clone_subfolder with ZIP fallback for GitHub repos
- Add helper utilities: ZIP extraction, directory/file copying
- Add SkipWorkspace option to skip output dirs in empty-target mode
- Fix workflow preference loading order for --module-url flag
- Add 20+ comprehensive tests for all new functions
2026-02-08 00:34:47 +07:00
j3ssie 641cf568aa docs: add security disclaimer and annotate intentional code execution points
- Add disclaimer section in README.md warning users about arbitrary code execution risks
- Annotate exec.Command calls throughout codebase with @NOTE comments explaining intentional design
- Improve step result table formatting: add column alignment, padding for status symbols, and remove unnecessary borders
2026-02-04 16:39:19 +07:00
j3ssie 7de3e7b2b2 feat: add performance optimizations and robustness improvements
- Add lock-free ResultCollector for parallel execution with atomic operations, eliminating mutex contention for pre-allocated slices
- Implement circuit breaker pattern (internal/retry/circuit_breaker) with configurable thresholds and half-open recovery state for fault tolerance
- Introduce json-iterator replacement (internal/json) for 2-6x faster JSON operations while maintaining stdlib compatibility
- Add lazy template rendering (RenderLazy) with variable reference caching for 50-80% faster rendering on large contexts
- Implement memory-efficient buffer pooling (bufpool) with 10MB pre-allocated reusable buffers for reduced GC pressure
- Add LoadFlowWithModules for parallel module pre-loading using errgroup, improving startup time for complex flows
- Add VarRefCache with LRU eviction for variable extraction caching
- Add streaming output support for foreach loops to process large datasets without memory accumulation
- Fix json import compatibility in llm_executor and db_functions
- Update test fixtures with correct YAML field names (call→function, run→command)
2026-01-31 01:26:28 +07:00
j3ssie 903689ce2a feat: add json mode to cdnDelete function for suppressed console output
- Add optional 'json' mode parameter to cdnDelete() to suppress console output when set
- Implement thread-safe console output with mutex coordination during parallel deletion
- Add detailed progress messages (started, deleting, deleted/error, summary) with terminal styling
2026-01-28 21:08:24 +08:00
j3ssie 1a8ab856fa feat: add empty_target preference and FlowName variable, optimize concurrent file sync
- Add empty_target preference to allow running workflows without explicit target
- Implement FlowName template variable for module context awareness
- Optimize cdnDelete, SyncUpload, SyncDownload with worker pool concurrency
- Load workflow preferences early for --empty-target CLI flag support
- Add unit tests for preferences and E2E tests for FlowName functionality
- Update linter to recognize ModuleName and FlowName as built-in variables
2026-01-28 18:47:45 +08:00
j3ssie 7f339a69f0 feat: add run priority support and server submission API
- Add RunClient for submitting runs to server with priority support (low, normal, high, critical)
- Implement --run-priority and --server-url CLI flags for server submission mode
- Add RunPriority and RunMode fields to database Run model for persistence
- Update CreateRunRequest with priority and run_mode validation
- Implement runServerSubmission() to submit workflows with priority to server API
- Comprehensive E2E tests for run endpoints with priority validation and multiple targets
- Add foreach-preprocess test workflow with variable_pre_process support
2026-01-26 21:14:04 +08:00
j3ssie 1fd3f09a9b feat(cdn): enhance delete with folder support, add sync progress events, and bash() function
- Enhanced cdnDelete to recursively delete folders with per-file tracking and error counts
- Added optional 'json' mode parameter to cdnSyncUpload/cdnSyncDownload for output format control
- Implemented real-time sync event callbacks for progress visualization with colored terminal output
- Added bash() as primary function with exec_cmd() as alias for backward compatibility
- Introduced SyncEvent type for action tracking (uploading/downloaded/skipped/deleted/error)
- Updated function tests to support new mode parameter
2026-01-25 21:07:58 +08:00
j3ssie 8ffbc91d43 feat: add cdn tree listing and webhook config setters 2026-01-25 16:43:36 +08:00
j3ssie ddcfac47c1 feat: enhance CDN functions with JSON serialization and glob pattern support
- Convert all CDN function return values to JSON strings for consistent Goja interop (cdnSyncUpload, cdnSyncDownload, cdnList, cdnStat)
- Add glob pattern matching support to cdnList with wildcard character support (*, ?, [])
- Implement new cdnRead function to read file content directly from cloud storage
2026-01-25 15:34:18 +08:00
j3ssie b6f97c7774 feat: add telegram channel mapping and multi-channel notification functions
- Add TelegramChannelMap config field to support channel aliases (#channel_name)
- Implement 4 new notification functions for sending to specific channels:
  - notify_telegram_channel(channel, message): send markdown message to channel
  - send_telegram_file_channel(channel, path, caption?): send file to channel
  - notify_message_as_file_telegram(path): read and send file as message
  - notify_message_as_file_telegram_channel(channel, path): read and send file to channel
- Enhance telegram.go with channel resolution logic supporting both named channels and numeric IDs
- Update function registry with documentation and examples for new functions
- Add path expansion support (~, $HOME) to file operations for better usability
- Update example config with telegram_channel_map documentation
2026-01-25 14:53:50 +08:00
j3ssie 777fb7c2bf chore: update ui build artifacts and refactor event trigger input syntax
- Update Next.js build IDs and chunk hashes in workflow upload static files
- Replace deprecated 'input' field syntax with new exports-style variables in event triggers (e.g., 'Target: event_data.value' instead of type/field/name)
- Simplify parameter override syntax in test workflows (remove nested 'default' structure)
2026-01-24 16:24:50 +08:00
j3ssie 9494f74942 feat: add db_reset_event_logs utility function
Add new utility function to reset processed event logs back to
unprocessed state, enabling event reprocessing with optional
filtering by workspace and topic pattern (glob syntax).
2026-01-24 13:14:50 +08:00
j3ssie f5840272c5 feat: add run cancellation, event enhancements, and performance optimizations
Major features:
- Add run registry for tracking active runs with PID management
- Add API-based run cancellation with process termination
- Add event trigger input vars syntax for multi-variable extraction
- Add filter_functions with utility function support in triggers
- Add event envelope injection for full event context in workflows
- Add write coordinator for batched database operations

API improvements:
- Add logout endpoint and diffs endpoints for assets/vulnerabilities
- Add step-results listing endpoint
- Update schedule model with target, workspace, params fields
- Change run_id to run_uuid across API responses

Performance:
- Add compiled JS program caching for 60-80% faster loop conditions
- Add parallel shard rendering for 20-40% faster workflow startup
- Add memory-mapped I/O for large file line counting
- Add efficient output buffer combining in runners
- Add mtime-based cache invalidation for workflow loader

Other changes:
- Rename trigger field from trigger to triggers in workflow YAML
- Disable pongo2 HTML autoescape for shell command templates
- Update JWT expiration default to 1440 minutes (1 day)
- Change CORS default to reflect-origin for credentials support
- Add source_type field to events (run, eval, api)
- Skip copying core Unix tools to external-binaries
2026-01-24 01:11:33 +08:00
j3ssie e28c9dce50 refactor: split RunID into RunUUID and ID, compute workspace dynamically, improve error handling 2026-01-22 13:28:59 +08:00
j3ssie 40d191e382 feat: add LLM function support and expand configuration options 2026-01-20 23:38:42 +08:00
j3ssie 1403d20a4d feat: add LLM step executor with vision and tool support, event workflow system, and inheritance
- Add LLM executor supporting OpenAI vision, tool calling, embeddings, and structured outputs
- Introduce event emitter/receiver workflows with deduplication and filtering (generate_event functions)
- Add workflow extends/override system enabling inheritance chains and step merge modes
- Update function naming to snake_case across all testdata (fileExists→file_exists, etc.)
- Add comprehensive test fixtures for linter, events, CDN, step dependencies, and extends workflows
2026-01-20 18:23:57 +08:00
j3ssie 9ed02e7eee refactor: code quality improvements, Docker updates, and setup enhancements
- chore: fix code style and remove unused error handling (add blank checks for closed file handles)
- chore: align struct field padding in multiple files for consistency
- fix: add 386 architecture support to goreleaser build config
- feat: add docker-publish target to Makefile for publishing to Docker Hub
- feat: refactor first-time setup into reusable installRequiredBinaries helper function
- feat: add initialization marker tracking for first-time setup completion
- feat: enhance workflow YAML detection to skip non-workflow files and hidden directories
- feat: improve database column display defaults (assets and vulnerabilities tables)
- feat: add fallback mechanism to install.sh for version detection failures
- fix: correct tarball filename generation by stripping 'v' prefix
- chore: update Docker base image from golang:1.22 to golang:1.25
- chore: update goreleaser release flags and simplify Docker build naming
- chore: fix import ordering across multiple files (alphabetical consistency)
- chore: improve install script with better version display formatting
- chore: reduce binaries per row from 10 to 6 in CLI output for better readability
2026-01-19 01:17:04 +08:00
j3ssie 7a2c5a5dc9 Complete rewrite and re-architecture Osmedeus Engine in v5 2026-01-18 19:32:24 +08:00