Commit Graph
127 Commits
Author SHA1 Message Date
j3ssie df0683b607 fix(api): don't run validate commands from a caller-supplied registry_url
#311 let callers point GET /osm/api/registry-info at any registry via the
registry_url query param. Both modes then called IsBinaryInstalled() on every
entry, which runs `sh -c <valide-command>` — so a GET with a hostile registry
executed arbitrary shell on the server. With SameSite=Lax session cookies and
reflect-all CORS, that was reachable by CSRF from any page an operator visits.

- add installer.IsBinaryInstalledNoExec() and use it whenever registry_url is
  set; only the embedded registry is trusted to run validate commands
- match isGitHubURL() on the parsed hostname so a lookalike host such as
  evil.tld/?x=github.com no longer receives the GitHub token
- cap remote registry reads at 32MB instead of an unbounded io.ReadAll
- surface LoadRegistry errors in nix-build mode rather than returning a
  success response with all metadata silently missing
- report the same registry_url semantics in both modes, and document the
  no-exec behaviour in docs/api/install.mdx
2026-08-08 14:10:26 +08:00
Jessie Ho (j3ssie)andGitHub ee80ef873e Merge pull request #311 from sissl0/main
feat: added registry_url query param to /osm/api/registry-info endpoint
2026-08-08 13:56:28 +08:00
j3ssie 5d9ff490f2 chore: update Docker tooling and sponsor links 2026-07-21 23:30:09 +07:00
Jessie Ho (j3ssie)andGitHub 2edef9f4b8 Merge pull request #314 from Friday-s/add-bloome-sponsor
docs: add Bloome sponsor section
2026-07-13 21:52:12 +08:00
Friday-s 6ac9b6f14e docs: add Bloome sponsor section before Installation 2026-07-13 18:28:30 +08:00
Georg Heindl 0b93b4f546 Added registry_url parameter for /osm/api/registry-info endpoint 2026-06-11 22:34:52 +02:00
j3ssie d5aa39ba30 fix(db_import_vigolium): stream JSONL with bufio.Reader to handle oversized lines
Vigolium http_record and finding entries can embed multi-MB response bodies,
producing JSONL lines that exceed bufio.Scanner's max token size and fail
with 'token too long'. Switch to bufio.Reader.ReadString, which grows as
needed, and extract per-line logic into a processLine closure. Add a
regression test that imports a finding with a 12MB response body.
2026-06-01 13:13:40 +08:00
j3ssie 896527421d fix(executor): preserve unrendered threads template and tidy run output
- Emit the raw `threads` value in step command summaries so template variables
  like `{{deparosParallel}}` survive until templateEngine.Render resolves them,
  instead of silently collapsing to 1 when Int() fails to parse the template
- Only print the Artifacts section in `printResultSummary` when -v/--verbose
  is set, removing state/log file clutter from default run output
- Update embedded UI build artifacts under public/ui/_next to the latest
  dashboard build (chunk renames, manifest updates, page txt/html refreshes)
2026-06-01 12:43:00 +08:00
j3ssie ec19282e64 fix(database): run column migrations before index creation
- Reorder Migrate() to add columns to existing tables BEFORE creating
  indexes, since CREATE TABLE IF NOT EXISTS is a no-op on existing
  databases and some indexes (e.g. finding_hash) reference migrated
  columns that would otherwise be missing
- Add explanatory comment in internal/database/database.go documenting
  why column migrations must precede index creation
- Surface a helpful CLI hint in pkg/cli/root.go pointing users to
  'osmedeus db migrate' when startup fails with 'failed to run
  migrations', so stale schemas are easier to recover from
2026-05-31 01:08:43 +08:00
j3ssie 8ec1de4f84 feat(db): add db_import_vigolium importer and harden CLI run
- Add db_import_vigolium JS function that routes vigolium JSONL records
  by envelope type: http_record -> assets, finding -> vulnerabilities
  (deduped on new finding_hash column), skipping scan/oast_interaction
- Add FindingHash field to Vulnerability model with idempotent ALTER
  TABLE migration and matching index
- Suppress run errors when using --silent --empty-target placeholder mode
- Retry transient fetchURLContent failures (network errors, 408/429/5xx)
  with exponential backoff; leave 4xx untouched so GitHub auth fallback
  can engage
- Add unit tests for the vigolium importer (import + idempotency) and
  the HTTP retry behavior, plus a vigolium juice-shop sample fixture
- Bump katana, naabu, kingfisher, bearer registry entries and add
  vigolium to the direct-fetch registry
v5.0.3
2026-05-29 23:31:56 +08:00
j3ssie 8448791c3c feat: propagate run cancellation to tmux sessions and remote SSH processes
- Track tmux sessions in ActiveRun and kill them on run cancel via new
  TmuxHooks indirection; expose killed_tmux_sessions in CLI and API responses
- Add ExecuteSSHCommand with remote pidfile + process-group kill watcher so
  cancelling a run actually terminates remote scans (not just the local session)
- Route ssh_exec/ssh_rsync/sync_* through the run's cancellable context via
  new RunContextHooks
- Switch docker-publish to sequential per-arch buildx builds + imagetools
  manifest to avoid OOM on multi-arch builds; add docker-buildx-setup target
- Cross-compile Dockerfile via BUILDPLATFORM/TARGETOS/TARGETARCH and retry
  SAST binary installs to survive QEMU-flaky downloads
- Bump version to v5.0.3
2026-05-26 00:06:26 +08:00
Jessie Ho (j3ssie)andGitHub f82afde4f4 Merge pull request #305 from sissl0/main
Webhook Notification on all return paths AND extended on_error actions
2026-05-12 01:26:55 +08:00
Jessie Ho (j3ssie)andGitHub 200f4ee732 Merge pull request #307 from NightRang3r/main
🐛 fix(distributed): link worker step results to master run via UUID
2026-05-12 00:52:15 +08:00
j3ssie 8136c20fa3 chore: fix the Discord link 2026-05-11 21:45:22 +08:00
Shai Rod df8ad1299b 🐛 fix(distributed): link worker step results to master run via UUID
Workers now propagate RunUUID from the submitted task so the master
can resolve the correct DB run record and associate step results.
Also increments completed_steps on successful insert and connects
the worker to the database for db_import_* functions.
2026-05-07 20:29:26 +03:00
j3ssie 9f95043356 chore(registry): bump tool versions and rename httpx to pd-httpx
- Bump amass, subfinder, nuclei, naabu, kingfisher, trivy, gosec, cariddi, cloudfox to latest releases
- Rename httpx registry entry to pd-httpx to disambiguate from other tools
- Refresh kingfisher description and update registry _last_update_at timestamp
2026-05-01 03:17:09 +08:00
Georg HeindlandCopilot 4ea9fda1bb Return artifact paths
Co-authored-by: Copilot <copilot@github.com>
2026-04-23 20:07:00 +02:00
Georg Heindl 771c000e92 Empty error panic on completion fixed 2026-04-23 19:53:24 +02:00
Georg HeindlandCopilot 39827b8b99 Included error, artifacts, message
Co-authored-by: Copilot <copilot@github.com>
2026-04-23 19:27:35 +02:00
Georg Heindl ce7199753f rm error 2026-04-23 17:52:00 +02:00
Georg HeindlandCopilot 67dc6b3de7 Added runUUID, error, steps, exports to webhook notification
Co-authored-by: Copilot <copilot@github.com>
2026-04-23 16:56:28 +02:00
Georg Heindl 29efc1b8e5 Remove duplicate condition check 2026-04-23 14:32:10 +02:00
Georg Heindl a4df58e063 Added on_error actions: Notify, Continue, Abort and run type function 2026-04-22 14:47:45 +02:00
Georg HeindlandGitHub fc827be975 Webhook Notification on every result path of ExecuteFlow/Module 2026-04-22 14:31:23 +02:00
j3ssie f467fa3c08 ci: mirror nightly OOM fixes in manual release (parallelism=1, GOMEMLIMIT, -trimpath) 2026-04-15 22:12:07 +08:00
j3ssie 1c9963f223 ci: drop nightly to parallelism=1, add GOMEMLIMIT and -trimpath, remove tidy hook 2026-04-15 21:46:15 +08:00
j3ssie 418c52f465 ci: fix nightly release OOM by limiting goreleaser parallelism
Build was being killed by the runner ~29min in (OOM on 7GB ubuntu-latest)
because goreleaser builds all targets in parallel by default. Cap to 2,
drop linux/386, and tune GOGC for lower peak memory.
2026-04-15 21:05:58 +08:00
j3ssie cda49b3897 ci: upgrade goreleaser action to v7, improve nightly CI, and tag optional binaries
- Upgrade goreleaser/goreleaser-action from v6 to v7 in manual and nightly release workflows
- Add timeout-minutes and Go cache to nightly release for faster, safer builds
- Normalize YAML string quoting from single to double quotes
- Add 'optional' tag to jq and massdns in direct-fetch registry metadata
2026-04-15 20:33:07 +08:00
j3ssie 52b4f05709 feat: auto-install pulumi cli and adjust ssh retry configuration
- Auto-install Pulumi CLI when not found instead of returning error
- Add installed binary to PATH for immediate availability
- Update SSH pool retry config to 5 attempts with 10s fixed delays
- Improve user feedback with progress indicators
2026-04-06 23:09:01 +08:00
j3ssie 69e3be23eb feat: add passive security scanning modules for frameworks and APIs
- Add 40+ new passive vulnerability detection modules covering:
  - Build/deployment misconfigurations (Next.js, Vite, webpack source maps)
  - Content Security Policy weakness auditing
  - HSTS preload readiness and secure header validation
  - API version detection, gRPC-Web, and WebAssembly discovery
  - Framework fingerprinting (WordPress, Drupal, Joomla, Firebase, Laravel, ASP.NET, Spring, Express, Rails, Django, Flask, FastAPI)
  - Cloud storage detection and signed URL/SAS token leaks
  - REST API analysis with pagination and sensitive field detection
  - GraphQL introspection and error information leakage
  - Debug information exposure (Python, Rails, Django)
  - Session management auditing for Express.js
  - CORS and cache poisoning vulnerabilities

All modules are passive-only with no external HTTP requests sent beyond initial scanning traffic.
2026-04-05 14:32:02 +08:00
j3ssie 9f9f107d95 build: update docker-build target to use explicit Dockerfile path 2026-04-04 15:21:41 +08:00
j3ssie 7804579dbd docs: add cloud setup and query commands documentation
- Add cloud setup section documenting SSH-based machine configuration
- Add query commands for vulnerabilities, runs, and execution steps
- Document Go install fallback for packages without go.mod
- Update README with cloud and query command examples
- Add reference to cloud setup E2E tests
2026-04-04 14:21:14 +08:00
Ai Ho (j3ssie)andGitHub a0a198d31e Merge pull request #302 from tair-m/patch-1
fix: change chat_id type to integer in docker config
2026-04-04 14:10:28 +08:00
j3ssie 0269cf4e26 feat: update Next.js build assets and add cloud setup E2E tests
- Update Next.js generated chunk hashes and build IDs reflecting latest dashboard build
- Update CSS stylesheet references in workflow upload page metadata
- Add comprehensive cloud setup E2E test suite (cloud_setup_test.go) with SSH password/key auth, post-command variable expansion, and Ansible integration
- Fix API priority levels to include 'medium' priority in test coverage
- Add agent-sdk test workflows (minimal, config, codex, multi-agent, session variants)
- Update E2E test utilities with runCLIInBase helper for multi-step cloud config tests
- Fix stderr/stdout capture in dependencies_target_types_test assertions
v5.0.2
2026-04-04 13:57:34 +08:00
tairandGitHub 3e5bad31d0 fix: change chat_id type to integer in docker config
Empty string for chat_id causes unmarshal error in Go. Changing to 0 fixes parsing failure
2026-04-04 02:55:27 +05:00
j3ssie 51bb8bfbac chore: update tool versions in registry metadata
- Update subfinder from 2.12.0 to 2.13.0 with corresponding download URLs
- Update nuclei from 3.7.0 to 3.7.1 with updated release links
- Update httpx from 1.8.1 to 1.9.0 across all platforms
- Update katana from 1.4.0 to 1.5.0 with new download links
- Update naabu from 2.4.0 to 2.5.0 with refreshed URLs
- Update kingfisher from 1.76.0 to 1.88.0 with new release binaries
- Update bearer from 2.0.0 to 2.0.1 with updated links
- Update gosec from 2.22.11 to 2.24.7
- Update interactsh from 1.3.0 to 1.3.1
- Update cloudfox from 1.17.0 to 2.0.1
- Update last_update_at timestamp to 2026-03-13T15:11:36.292828+00:00
2026-03-14 00:45:54 +08:00
Ai Ho (j3ssie)andGitHub b7eeaa524a Merge pull request #300 from YouChenJun/fix/trivy-404
fix: update trivy download link to v0.69.3 to fix 404 error
2026-03-13 22:12:04 +08:00
YouChenJun 16464a0533 fix: update trivy download link from v0.69.1 to v0.69.3 to fix 404 error 2026-03-12 21:23:21 +08:00
j3ssie 2dbec22627 fix: prevent git clone failures by removing existing destinations
- Add extractGitCloneDest() function to parse destination from git clone commands
- Remove existing git clone destination directories before recloning
- Add 'make' build tool to binary registry with platform-specific install commands
2026-03-08 14:00:20 +08:00
j3ssie 90ce5f4a16 refactor: consolidate process killing logic and ACP agent defaults
- Extract KillProcessAndChildren into core.types for reuse across CLI and server handlers
- Replace hardcoded 'claude-code' strings with core.DefaultACPAgent constant
- Precompute absolute allowed paths in ACP client to avoid repeated filepath.Abs calls
- Simplify runQuerySteps to delegate to runQueryTable, reducing code duplication
- Refactor agent_chat concurrency guard using sync.Mutex.TryLock for cleaner code
- Use request context for agent timeout instead of background context
2026-03-07 16:49:48 +08:00
Ai Ho (j3ssie)andGitHub cf98cab616 Merge pull request #299 from anhtn512/main
fix(installer): fix shallow clone and add go install fallback for binary installation
2026-03-07 15:50:40 +08:00
j3ssie ecc77b9a30 refactor: optimize ACP agent initialization and remove dead code
- Remove unreachable custom agent command validation from ACPExecutor.Execute()
- Add IsBuiltinAgent() utility function for cleaner agent name resolution
- Replace inline agent list iteration with IsBuiltinAgent() check in agent_chat.go
- Add explicit stdin pipe cleanup in RunAgentACP() defer block
- Simplify CLI output handling by removing redundant stream output fallback
2026-03-07 14:13:19 +08:00
j3ssie 9bdb3260b6 feat(cli): add query command and enhanced asset/run management
- Add query subcommands (vulns, runs, steps) with JSON output and flexible filtering
- Enhance assets command with --where, --search, --value filters (fuzzy matching)
- Expand asset searchable columns (status_code, content_type, title, dns_records, tls, tech)
- Add run status and run cancel subcommands with process termination
- Support control-plane and PID-based cancellation with database updates
- Implement fuzzyFilters in GetTableRecords for case-insensitive substring matching
2026-03-01 10:54:49 +08:00
j3ssie ca1c0d443a feat: add OSM_IGNORE_REGISTRY environment variable to skip binary installation
- Add OSM_IGNORE_REGISTRY env var to skip auto binary installation and health checks
- Update health check to display warning when binary folder is empty but OSM_IGNORE_REGISTRY is set
- Update installRequiredBinaries to skip installation when OSM_IGNORE_REGISTRY=true
- Enhance DownloadFile with retry logic and wget/curl fallback for resilient downloads
- Refactor download function to validate content length and handle transient failures
- Refactor test code to use switch statements instead of if-else chains
2026-02-28 17:29:42 +08:00
anhtn512 63653076c0 feat: enhance binary installation process with fallback to go install 2026-02-27 15:20:07 +07:00
anhtn512 e753f94877 fix: ensure go-getter uses depth=1 and sets ref=HEAD when version is not specified 2026-02-27 15:19:44 +07:00
j3ssie 520977d1ea feat: Add agent-acp step type with ACP protocol support
- Implement agent-acp step type for spawning external ACP agent subprocesses via Agent Communication Protocol
- Add ACPExecutor with validation, field rendering, and subprocess lifecycle management
- Integrate agent-acp field rendering in StepDispatcher (batch and sequential modes)
- Add run_agent() utility function for workflows to execute ACP agents from steps and JS context
- Add osmedeus agent CLI command for interactive agent execution with --agent, --cwd, --timeout, --stdin, and --list flags
- Add /osm/api/agent/chat/completions REST endpoint with OpenAI-compatible chat format and concurrency control
- Support agent selection via: built-in names (claude-code, codex, opencode, gemini) or custom acp_config.command
- Add step-level configuration: cwd, allowed_paths, acp_config (command, args, env, write_enabled)
- Add comprehensive E2E tests for agent-acp workflows (basic, minimal, config, codex variants)
- Add test workflows in test/testdata/workflows/agent-and-llm/
- Update AGENTS.md documentation with agent-acp examples, CLI usage, and API endpoints
v5.0.1
2026-02-27 16:18:06 +08:00
j3ssie 375c159f48 build: update dashboard UI artifacts from latest Next.js build 2026-02-20 00:34:15 +07:00
j3ssie 45c7ea0bbc feat: downgrade trigger input variables to info-level in linter
- Add tracking of trigger input variables and recognize their availability at runtime
- Downgrade undefined variable warnings to info-level when variables come from event trigger inputs (both new and legacy syntax)
- Add event envelope variables (EventTopic, EventSource, EventTimestamp, etc.) as recognized built-ins when workflow has event triggers
- Include comprehensive test coverage for trigger input scenarios, event triggers, and mixed variable definitions
2026-02-18 11:26:44 +07:00
j3ssie 2bbf7295f6 refactor: enhance linter with comprehensive undefined variable detection
- Add platform variables (PlatformOS, PlatformArch, PlatformInDocker, PlatformInKubernetes, PlatformCloudProvider) to linter's built-in variables list
- Extract checkStepFieldsForUndefinedVars and collectReferencedVarsFromStep helper functions to support comprehensive scanning of all template-renderable fields including parallel_functions, speed_args, config_args, agent fields, HTTP headers, LLM messages, decision cases/conditions, and memory paths
- Add recursion for nested steps (foreach inner steps and parallel_steps) with proper variable scoping and support for loop-specific variables (_id_, foreach variable)
- Enhance message formatting with highlightQuoted() to make quoted terms bold+yellow in error output
- Enable UndefinedVariableRule by default in GetDefaultRules()
- Add comprehensive test coverage for newly supported fields and nested step types (TestUndefinedVariableRule_*, TestUnusedVariableRule_ReferencedInNewFields)
- Add test fixture and E2E test (TestWorkflow_Validate_UndefinedVarsComprehensive) validating detection across functions and foreach
2026-02-18 10:18:44 +07:00