Fixes four reported issues.
#321 / #322 — POST /osm/api/runs was unusable when the server was started
with --settings-file. That branch in pkg/cli/root.go loaded the settings via
config.LoadFromFile (a plain YAML unmarshal) and never called ResolvePaths(),
so every derived runtime path stayed empty. An empty WorkflowsPath made
workflow lookups scan "" (reported as "Workflow not found") and made a flow's
relative module refs resolve against the process working directory.
- root.go now resolves paths (and applies env overrides) on that branch
- ResolvePaths backfills any environments.* key a partial settings file omits,
sourced from the new defaultEnvironments() so DefaultConfig and the backfill
can no longer drift
- parser.ErrWorkflowsDirNotConfigured replaces the silent cwd fallback, so all
loader call sites report the misconfiguration instead of a misleading 404
#323 — a run submitted with run_mode:distributed could be lost permanently.
BRPOP is at-most-once: a task popped but not yet recorded in osm:tasks:running
existed nowhere, and the master's recovery sweep only reads that hash.
- workers now claim via BLMOVE onto osm:tasks:processing:{worker_id}
- the claim is acked only once SetTaskRunning succeeds; a failure requeues
instead of executing the task untracked
- recovery on worker startup, on dead workers, and for processing lists left
by workers that are no longer registered
- requires Redis 6.2+
#320 — settings values can now be overridden by OSM_* environment variables so
secrets need not live in osm-settings.yaml. The mapping is derived from the YAML
tags by reflection, so new settings are overridable with no extra code. Applied
in config.Load, hotreload and the --settings-file branch, but deliberately not
in LoadFromFile: `osmedeus config set` round-trips through it and writes back,
which would persist env secrets to disk.
Known gaps: pkg/cli/worker_queue.go still consumes the pending queue with the
at-most-once PopTask, and two distributed e2e tests (TaskSubmission,
FullWorkflow) fail on main independently of these changes.
- Fix spacing in Makefile run-github-action target
- Add run-homebrew-action target to Makefile for workflow orchestration
- Detect explicit local paths (./foo.yaml, ../foo.yaml) and prevent fallback to workflows directory when file not found
- Add debug logging when workflow not found at relative path before trying workflows directory
- Add comprehensive test for explicit local path handling with error cases
- Add lock-free ResultCollector for parallel execution with atomic operations, eliminating mutex contention for pre-allocated slices
- Implement circuit breaker pattern (internal/retry/circuit_breaker) with configurable thresholds and half-open recovery state for fault tolerance
- Introduce json-iterator replacement (internal/json) for 2-6x faster JSON operations while maintaining stdlib compatibility
- Add lazy template rendering (RenderLazy) with variable reference caching for 50-80% faster rendering on large contexts
- Implement memory-efficient buffer pooling (bufpool) with 10MB pre-allocated reusable buffers for reduced GC pressure
- Add LoadFlowWithModules for parallel module pre-loading using errgroup, improving startup time for complex flows
- Add VarRefCache with LRU eviction for variable extraction caching
- Add streaming output support for foreach loops to process large datasets without memory accumulation
- Fix json import compatibility in llm_executor and db_functions
- Update test fixtures with correct YAML field names (call→function, run→command)
Major features:
- Add run registry for tracking active runs with PID management
- Add API-based run cancellation with process termination
- Add event trigger input vars syntax for multi-variable extraction
- Add filter_functions with utility function support in triggers
- Add event envelope injection for full event context in workflows
- Add write coordinator for batched database operations
API improvements:
- Add logout endpoint and diffs endpoints for assets/vulnerabilities
- Add step-results listing endpoint
- Update schedule model with target, workspace, params fields
- Change run_id to run_uuid across API responses
Performance:
- Add compiled JS program caching for 60-80% faster loop conditions
- Add parallel shard rendering for 20-40% faster workflow startup
- Add memory-mapped I/O for large file line counting
- Add efficient output buffer combining in runners
- Add mtime-based cache invalidation for workflow loader
Other changes:
- Rename trigger field from trigger to triggers in workflow YAML
- Disable pongo2 HTML autoescape for shell command templates
- Update JWT expiration default to 1440 minutes (1 day)
- Change CORS default to reflect-origin for credentials support
- Add source_type field to events (run, eval, api)
- Skip copying core Unix tools to external-binaries
j3ssie