Commit Graph
19 Commits
Author SHA1 Message Date
j3ssie bd1434739e feat: add sudo authentication support with keepalive and variable renames
- Add sudo_auth() function for TTY prompts and credential management with optional 4-minute keepalive loop
- Add --sudo-aware CLI flag to detect workflows with sudo commands and offer authentication guidance
- Add sudo step scanner to detect sudo usage across all step types (bash, parallel, foreach) and nested structures
- Add parse_url_file() function to batch-process URLs with format directives, supporting bare IPs and CIDR notation
- Add portscan test data with realistic nmap JSONL samples
- Rename {{Workspace}} to {{TargetSpace}} in function examples for clarity
- Add sudo E2E tests covering tip message, flag acceptance, and non-sudo workflows
2026-02-16 23:46:33 +07:00
j3ssie 459d5939fb feat: add cloud infrastructure and nmap/tmux function support
- Implement cloud provider infrastructure (DigitalOcean, AWS, GCP, Linode, Azure) with Pulumi integration for distributed scanning
- Add nmap and tmux utility functions for port scanning results processing and long-running background session management
- Introduce webhook-triggered run execution with unique UUID and authentication key support for external integrations
2026-02-16 15:23:24 +07:00
j3ssie 801583b0c8 feat: distributed E2E testing stack and improve worker/master logging
- Add Docker Compose distributed-e2e.yaml with Redis, master, and worker services
- Add Make targets: distributed-e2e-up, distributed-e2e-run, distributed-e2e-down
- Replace zap logger with terminal printer for consistent CLI output in master/worker
- Add Redis connection info and task details to startup and lifecycle logs
- Fix DNS records field: use 'dns_records' key with backward compat for 'a' key
- Add optional asset_type and source parameters to db_import_custom_asset
- Update Swagger docs for LLM embeddings schema
- Set run_uuid tracking in worker task execution for distributed runs
- Register distributed database hooks and ensure external binaries in PATH
- Improve terminal formatting with colors for worker/master IDs and log messages
2026-02-14 03:00:30 +07:00
j3ssie bb15efdf60 feat: db replay and snapshot improvements
- Add replayDBOperations to reinitialize databases from imported workspaces, with proper parameter resolution and pre-condition skipping
- Implement ISO 8601 timestamp format for snapshots (2026-02-13T18-20-34Z) instead of Unix epoch
- Add --include-heavy flag to include large fields (raw_response, screenshot, blob_content) in database queries
- Fix asset table default columns (url, status_code, content_length, title) for better UX
- Skip heavy initialization for lightweight commands (help, version, completion) to avoid ~50MB overhead
- Optimize database function execution via lazy config initialization
- Fix binary installation via Nix to copy already-installed binaries to binaries folder
2026-02-13 09:06:51 +07:00
j3ssie 4ac041fd73 feat: add workflow help metadata, artifact optional flag, and search/filter improvements
- Add WorkflowHelp struct with Usage and ExampleTargets for CLI documentation
- Add Optional field to Artifact model and database schema with migration support
- Implement workflow search functionality by name, description, and tags in CLI list command
- Add --usage and --search flags to workflow list command with multiple filtering options
- Display workflow usage info in show command when Help is defined
- Support help inheritance in workflow extends/inheritance resolver
- Update vulnerability counters from database after SARIF imports
- Add comprehensive Help unit tests covering parsing, cloning, and mutation isolation
- Improve test helpers with streaming output, diagnostics, and file validation utilities
- Add fourth general canary test for domain-list-recon flow with artifact validation
2026-02-12 01:04:03 +07:00
j3ssie ef00c176e5 feat: add canary tests for real-world scan workflows
- Add docker-compose.canary.yaml with multi-stage Dockerfile to build osmedeus from source and layer onto toolbox base, ensuring canary tests exercise current code not released binaries
- Implement three canary test scenarios (repo SAST scan, domain-lite reconnaissance, CIDR IP scanning) with lifecycle management (container startup/cleanup) and comprehensive assertions on filesystem artifacts and API records
- Add Makefile targets (test-canary-all, test-canary-repo, test-canary-domain, test-canary-ip, canary-up, canary-down) for granular test execution with configurable timeouts (20-60 minutes)
2026-02-10 17:25:15 +07:00
j3ssie 4ace076c98 docs: update security disclaimer and fix homebrew workflow command
- Update README disclaimer to reference the Security Warning documentation page
- Fix homebrew release workflow to run from the homebrew-tap directory context
- Improve CLI/API security guidance by linking to comprehensive security documentation
2026-02-05 10:54:21 +07:00
j3ssie 78c470791e fix(parser): improve explicit local path handling in workflow loading
- Fix spacing in Makefile run-github-action target
- Add run-homebrew-action target to Makefile for workflow orchestration
- Detect explicit local paths (./foo.yaml, ../foo.yaml) and prevent fallback to workflows directory when file not found
- Add debug logging when workflow not found at relative path before trying workflows directory
- Add comprehensive test for explicit local path handling with error cases
2026-02-02 15:31:28 +07:00
j3ssie dcb71d3b1d feat: add build time and commit hash to server info and improve nightly release workflow
- Expose BuildTime and CommitHash in core constants and health endpoints (/api/info)
- Refactor nightly release workflow to delete only assets instead of the entire release tag
- Rename github-action make target to run-github-action for clarity and consistency
2026-01-26 02:11:26 +08:00
j3ssie bbba13501f fix: ensure loader initialization in server handlers and gh token isolation
- Initialize workflow loader in executeRunsConcurrently for proper path resolution
- Set loader on all executor instances (CreateRun, StartRun, executeRunsConcurrently)
- Unset GH_TOKEN in github-action Makefile target to prevent token conflicts
2026-01-26 01:52:24 +08:00
j3ssie bef52a2223 feat: add github-action make target and refactor builtin variables initialization
- Add github-action target to trigger manual and nightly release workflows
- Extract BuildBuiltinVariables() function for reusable context setup
- Simplify RandomString from 8-char alphanumeric to 6-char lowercase
- Update documentation for RandomString variable description
2026-01-25 23:37:01 +08:00
j3ssie f5840272c5 feat: add run cancellation, event enhancements, and performance optimizations
Major features:
- Add run registry for tracking active runs with PID management
- Add API-based run cancellation with process termination
- Add event trigger input vars syntax for multi-variable extraction
- Add filter_functions with utility function support in triggers
- Add event envelope injection for full event context in workflows
- Add write coordinator for batched database operations

API improvements:
- Add logout endpoint and diffs endpoints for assets/vulnerabilities
- Add step-results listing endpoint
- Update schedule model with target, workspace, params fields
- Change run_id to run_uuid across API responses

Performance:
- Add compiled JS program caching for 60-80% faster loop conditions
- Add parallel shard rendering for 20-40% faster workflow startup
- Add memory-mapped I/O for large file line counting
- Add efficient output buffer combining in runners
- Add mtime-based cache invalidation for workflow loader

Other changes:
- Rename trigger field from trigger to triggers in workflow YAML
- Disable pongo2 HTML autoescape for shell command templates
- Update JWT expiration default to 1440 minutes (1 day)
- Change CORS default to reflect-origin for credentials support
- Add source_type field to events (run, eval, api)
- Skip copying core Unix tools to external-binaries
2026-01-24 01:11:33 +08:00
j3ssie e4fa681a9b feat: add eval command shorthand and reorder snapshot-release tasks
- Add top-level 'eval' command as shorthand for 'func eval' with multiple aliases (e, ev, evl, evla)
- Reorder Makefile snapshot-release to copy registry metadata before building binary
- Update contact URL in Swagger docs from osmedeus/osmedeus to osmedeus
- Update registry metadata with katana v1.4.0 (web crawling and spidering framework)
2026-01-21 18:13:47 +08:00
j3ssie da0085df7b docs: enhance Makefile help text and add install target
- Add dedicated 'install' target that builds binary if not already present
- Reorganize help output: move install-related targets under 'BUILD & INSTALL' section
- Expand test command descriptions with specific test types and requirements
- Clarify platform specifications in build-all documentation
- Reorder help sections for better discoverability of Docker and test targets
2026-01-21 17:39:16 +08:00
j3ssie 88e3552a63 refactor: simplify release workflows and add local-release target
- Remove unnecessary goreleaser skip flags from manual-release workflow
- Add local-release make target for testing macOS and Linux ARM64 builds
- Refactor Dockerfile.toolbox to use direct-fetch with Ubuntu 24.04 base
- Create Dockerfile.toolbox-nix for Nix-based installation variant
2026-01-21 14:41:02 +08:00
j3ssie 40d191e382 feat: add LLM function support and expand configuration options 2026-01-20 23:38:42 +08:00
j3ssie 1403d20a4d feat: add LLM step executor with vision and tool support, event workflow system, and inheritance
- Add LLM executor supporting OpenAI vision, tool calling, embeddings, and structured outputs
- Introduce event emitter/receiver workflows with deduplication and filtering (generate_event functions)
- Add workflow extends/override system enabling inheritance chains and step merge modes
- Update function naming to snake_case across all testdata (fileExists→file_exists, etc.)
- Add comprehensive test fixtures for linter, events, CDN, step dependencies, and extends workflows
2026-01-20 18:23:57 +08:00
j3ssie 9ed02e7eee refactor: code quality improvements, Docker updates, and setup enhancements
- chore: fix code style and remove unused error handling (add blank checks for closed file handles)
- chore: align struct field padding in multiple files for consistency
- fix: add 386 architecture support to goreleaser build config
- feat: add docker-publish target to Makefile for publishing to Docker Hub
- feat: refactor first-time setup into reusable installRequiredBinaries helper function
- feat: add initialization marker tracking for first-time setup completion
- feat: enhance workflow YAML detection to skip non-workflow files and hidden directories
- feat: improve database column display defaults (assets and vulnerabilities tables)
- feat: add fallback mechanism to install.sh for version detection failures
- fix: correct tarball filename generation by stripping 'v' prefix
- chore: update Docker base image from golang:1.22 to golang:1.25
- chore: update goreleaser release flags and simplify Docker build naming
- chore: fix import ordering across multiple files (alphabetical consistency)
- chore: improve install script with better version display formatting
- chore: reduce binaries per row from 10 to 6 in CLI output for better readability
2026-01-19 01:17:04 +08:00
j3ssie 7a2c5a5dc9 Complete rewrite and re-architecture Osmedeus Engine in v5 2026-01-18 19:32:24 +08:00