Workers now propagate RunUUID from the submitted task so the master
can resolve the correct DB run record and associate step results.
Also increments completed_steps on successful insert and connects
the worker to the database for db_import_* functions.
- Add QueuePoller for dual-source (DB + Redis) task polling with concurrent execution
- Add worker queue subcommands (list, new, run) for managing queued tasks
- Add uninstall command to safely remove Osmedeus installation
- Add input validation to prevent command injection via dangerous shell characters
- Add queue run support via --queue and --queue-run flags in osmedeus run
- Add queue polling to server with --no-queue-polling flag to disable it
- Support queuing tasks from both CLI and API with deduplication across sources
- Add worker eval command for distributed function execution with Redis hooks registration
- Add worker set command to update worker fields (alias, public-ip, ssh-enabled, ssh-keys-path)
- Enhance worker status with JSON output, search filtering, and column selection (--columns, --exclude-columns, --search)
- Add --keep-setting flag to install base/validate commands to preserve osm-settings.yaml after base installation
- Fix binary installation in Nix: replace CopyInstalledBinaryToFolder with SymlinkInstalledBinaryToFolder
- Add --clean-ws flag to db clean command for removing workspace data
- Add HooksEnabled field to Run records when creating runs from CLI and API
- Add comprehensive test coverage for hook execution (pre/post hooks, execution order, failure handling)
- Add test coverage for worker commands (eval, set, status with JSON) and db clean operations
- Improve usage documentation for worker subcommands and db operations
- Add Docker Compose distributed-e2e.yaml with Redis, master, and worker services
- Add Make targets: distributed-e2e-up, distributed-e2e-run, distributed-e2e-down
- Replace zap logger with terminal printer for consistent CLI output in master/worker
- Add Redis connection info and task details to startup and lifecycle logs
- Fix DNS records field: use 'dns_records' key with backward compat for 'a' key
- Add optional asset_type and source parameters to db_import_custom_asset
- Update Swagger docs for LLM embeddings schema
- Set run_uuid tracking in worker task execution for distributed runs
- Register distributed database hooks and ensure external binaries in PATH
- Improve terminal formatting with colors for worker/master IDs and log messages
- Add lock-free ResultCollector for parallel execution with atomic operations, eliminating mutex contention for pre-allocated slices
- Implement circuit breaker pattern (internal/retry/circuit_breaker) with configurable thresholds and half-open recovery state for fault tolerance
- Introduce json-iterator replacement (internal/json) for 2-6x faster JSON operations while maintaining stdlib compatibility
- Add lazy template rendering (RenderLazy) with variable reference caching for 50-80% faster rendering on large contexts
- Implement memory-efficient buffer pooling (bufpool) with 10MB pre-allocated reusable buffers for reduced GC pressure
- Add LoadFlowWithModules for parallel module pre-loading using errgroup, improving startup time for complex flows
- Add VarRefCache with LRU eviction for variable extraction caching
- Add streaming output support for foreach loops to process large datasets without memory accumulation
- Fix json import compatibility in llm_executor and db_functions
- Update test fixtures with correct YAML field names (call→function, run→command)
Major features:
- Add run registry for tracking active runs with PID management
- Add API-based run cancellation with process termination
- Add event trigger input vars syntax for multi-variable extraction
- Add filter_functions with utility function support in triggers
- Add event envelope injection for full event context in workflows
- Add write coordinator for batched database operations
API improvements:
- Add logout endpoint and diffs endpoints for assets/vulnerabilities
- Add step-results listing endpoint
- Update schedule model with target, workspace, params fields
- Change run_id to run_uuid across API responses
Performance:
- Add compiled JS program caching for 60-80% faster loop conditions
- Add parallel shard rendering for 20-40% faster workflow startup
- Add memory-mapped I/O for large file line counting
- Add efficient output buffer combining in runners
- Add mtime-based cache invalidation for workflow loader
Other changes:
- Rename trigger field from trigger to triggers in workflow YAML
- Disable pongo2 HTML autoescape for shell command templates
- Update JWT expiration default to 1440 minutes (1 day)
- Change CORS default to reflect-origin for credentials support
- Add source_type field to events (run, eval, api)
- Skip copying core Unix tools to external-binaries
Shai Rod