mirror of
https://github.com/affaan-m/ECC.git
synced 2026-08-17 21:15:40 +02:00
Add bounded harness evaluation and rollback loop (#2686)
* feat(ecc2): add bounded harness evaluation loop * fix(ecc2): preserve harness evidence and legacy IDs
This commit is contained in:
Affaan Mustafa
@@ -509,9 +509,9 @@ Kimi Code discovers the installed `.kimi/AGENTS.md` instructions and `.kimi/skil
|
||||
|
||||
### Itô compute CLI bridge
|
||||
|
||||
`ecc ito` delegates to the separately installed canonical Itô client; ECC does not maintain a second API client or browser handoff. The available operations are `ecc ito auth`, `ecc ito find`, `ecc ito status`, and the separately gated `ecc ito evals`. The matching MCP tools remain `ito_auth`, `ito_find`, and `ito_status`; node qualification is CLI-only.
|
||||
`ecc ito` delegates to the separately installed canonical Itô client; ECC does not maintain a second API client. `ecc ito login [--no-browser]` performs device authorization, opens the Itô verification page by default, and persists a device token in macOS Keychain; `--no-browser` suppresses the page handoff. ECC itself does no browser automation. `ecc ito auth` is validation-only and rejects `--no-browser`. The available operations are `ecc ito login`, `ecc ito auth`, `ecc ito find`, `ecc ito status`, and the separately gated `ecc ito evals`. The matching MCP tools remain `ito_auth`, `ito_find`, and `ito_status`; `ito_auth` validates existing credentials and node qualification is CLI-only.
|
||||
|
||||
The `ito-compute-cli` package is currently unpublished. Build it locally from the Itô runtime repo (private while the desk hardens; design partners get access) under `cli/ito-compute-cli`, run `npm ci` and `npm run check`, then set `ECC_ITO_CLI_EXECUTABLE` to that build's absolute `dist/bin/ito.js` path. Inject `ITO_API_KEY` from 1Password or the launching environment. ECC does not discover this credential-bearing client through `PATH`. See the [`ito-compute` skill](skills/ito-compute/SKILL.md) for the full RFQ authority and MCP setup contract.
|
||||
The `ito-compute-cli` package is currently unpublished. Build it locally from the Itô runtime repo (private while the desk hardens; design partners get access) under `cli/ito-compute-cli`, run `npm ci` and `npm run check`, then set `ECC_ITO_CLI_EXECUTABLE` to that build's absolute `dist/bin/ito.js` path. Login never inherits `ITO_API_KEY`; auth, find, and status forward `ITO_API_KEY` directly when configured, and `ITO_AUTH_MODE=legacy` is not required. Device tokens use macOS Keychain by default; explicit file fallback must retain owner-only directory/file permissions. ECC does not discover this credential-bearing client through `PATH`. See the [`ito-compute` skill](skills/ito-compute/SKILL.md) for the full RFQ authority and MCP setup contract.
|
||||
|
||||
`find` submits a live authenticated RFQ. It does not reserve capacity. `evals` requires both `ITO_ENABLE_SIXTYTWO_LIVE=1` and `--live-sixtytwo`, a separately installed `sixtytwo-cli==0.3.33`, an explicit node list, and an existing absolute configuration directory. It cannot rent, launch, recover, repair, or purchase. ECC exposes no quote lock, purchase, workload, or inference path, and it never replaces a missing client or failed live call with a local result.
|
||||
</details>
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
# Evaluator RAG Prototype
|
||||
|
||||
ECC 2.0 needs a self-improving harness loop that can learn from real work
|
||||
without blindly mutating a user's Claude, Codex, OpenCode, dmux, Zed, or
|
||||
terminal setup. This prototype defines the smallest read-only artifact set for
|
||||
that loop.
|
||||
ECC 2.0 needs an evidence-driven harness evaluation loop that can compare
|
||||
operator-supplied candidates from real work without implying model learning or
|
||||
blindly mutating a user's Claude, Codex, OpenCode, dmux, Zed, or terminal
|
||||
setup. This prototype defines the smallest read-only artifact set for that
|
||||
loop.
|
||||
|
||||
The fixture set lives in
|
||||
[`examples/evaluator-rag-prototype/`](../../examples/evaluator-rag-prototype/).
|
||||
|
||||
@@ -24,9 +24,10 @@ ECC delegates to the canonical Itô package in
|
||||
`Ito-Markets/ito-cloud-runtime/cli/ito-compute-cli`. ECC does not maintain a
|
||||
second API client or response schema.
|
||||
|
||||
The wrapper exposes only the canonical CLI's `auth`, `find`, `status`, and `evals`
|
||||
The wrapper exposes only the canonical CLI's `login`, `auth`, `find`, `status`, and `evals`
|
||||
operations:
|
||||
|
||||
ecc ito login [--no-browser]
|
||||
ecc ito auth
|
||||
ecc ito find <all required RFQ constraints>
|
||||
ecc ito status
|
||||
@@ -36,8 +37,12 @@ The canonical MCP server exposes only `ito_auth`, `ito_find`, and `ito_status`.
|
||||
ECC includes an opt-in configuration template pointing to the local built MCP
|
||||
entry. It does not enable the server by default.
|
||||
|
||||
The former browser/manual-copy command is retired. `ecc ito` performs no
|
||||
browser navigation and stores no economic state.
|
||||
The former browser/manual-copy command is retired. `ecc ito login` delegates to
|
||||
the canonical CLI's device authorization, which opens the Itô verification page
|
||||
by default and persists a device token in macOS Keychain. `--no-browser`
|
||||
suppresses that page handoff. ECC itself performs no browser automation and
|
||||
stores no economic state. `ecc ito auth` is validation-only, never starts
|
||||
device login, and rejects `--no-browser`.
|
||||
|
||||
## Local install
|
||||
|
||||
@@ -53,19 +58,25 @@ Set `ECC_ITO_CLI_EXECUTABLE` to the explicit absolute built entry:
|
||||
/absolute/path/to/ito-cloud-runtime/cli/ito-compute-cli/dist/bin/ito.js
|
||||
|
||||
ECC does not resolve the credential-bearing client through `PATH`; this avoids
|
||||
forwarding `ITO_API_KEY` to an unrelated executable with the same name.
|
||||
forwarding authentication material to an unrelated executable with the same
|
||||
name.
|
||||
|
||||
For MCP, configure `node` with:
|
||||
|
||||
/absolute/path/to/ito-cloud-runtime/cli/ito-compute-cli/dist/bin/ito-mcp.js
|
||||
|
||||
Inject `ITO_API_KEY` with 1Password or the launching environment. ECC forwards
|
||||
only `ITO_API_KEY`, optional Itô endpoint overrides, and the minimum process
|
||||
environment. It does not inspect or log the key.
|
||||
Device login forwards only required authorization settings, optional Itô
|
||||
endpoint overrides, and the minimum process environment; it never inherits
|
||||
`ITO_API_KEY`. The `auth`, `find`, and `status` commands forward `ITO_API_KEY`
|
||||
directly when configured; `ITO_AUTH_MODE=legacy` is not required. Device tokens
|
||||
use macOS Keychain by default. Explicit file fallback retains owner-only 0700
|
||||
directory and 0600 token-file permissions. ECC does not inspect or log secrets.
|
||||
|
||||
## Authority and economics
|
||||
|
||||
- `auth` validates the configured Itô API key.
|
||||
- `login` starts canonical device authorization, with `--no-browser` available
|
||||
when the operator does not want the CLI to open the verification page.
|
||||
- `auth` validates existing credentials only.
|
||||
- `find` reads live inventory and submits a live authenticated RFQ. An operator
|
||||
or agent must gather every hard topology/economic constraint and obtain
|
||||
explicit buyer authority before invoking it.
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
# ECC × Itô Real CLI Bridge — TDD Evidence
|
||||
|
||||
Date: 2026-07-23
|
||||
Date: 2026-08-05
|
||||
|
||||
Source plan: requirements were derived from the approved implementation
|
||||
handoff. No external plan file was executed.
|
||||
|
||||
## User journeys
|
||||
|
||||
1. As an ECC operator, I can invoke the canonical local Itô `auth`, `find`, and
|
||||
`status` operations without a duplicate client or browser workflow.
|
||||
1. As an ECC operator, I can explicitly invoke streaming device `login`, then
|
||||
use validation-only `auth`, `find`, and `status` without a duplicate client.
|
||||
2. As a security reviewer, I can prove unsupported operations, missing local
|
||||
installs, and ECC dry-run requests fail before any child process or network
|
||||
operation.
|
||||
@@ -21,62 +21,46 @@ Before production changes:
|
||||
|
||||
```text
|
||||
node tests/scripts/ito-cli-bridge.test.js
|
||||
Passed: 0
|
||||
Failed: 9
|
||||
Passed: 13
|
||||
Failed: 8
|
||||
|
||||
node tests/ci/ito-compute-skill.test.js
|
||||
Passed: 0
|
||||
Failed: 4
|
||||
Passed: 2
|
||||
Failed: 3
|
||||
```
|
||||
|
||||
The failures were caused by the old browser-only `rent` command and the missing
|
||||
real skill/install/MCP surfaces.
|
||||
The failures captured the old combined auth/login surface, legacy-mode API-key
|
||||
gate, buffered login output, and stale help, skill, MCP, and integration wording.
|
||||
|
||||
## GREEN evidence
|
||||
|
||||
```text
|
||||
node tests/scripts/ito-cli-bridge.test.js
|
||||
Passed: 9
|
||||
Passed: 21
|
||||
Failed: 0
|
||||
|
||||
node tests/ci/ito-compute-skill.test.js
|
||||
Passed: 4
|
||||
Passed: 5
|
||||
Failed: 0
|
||||
|
||||
NODE_PATH=<existing-ecc-checkout>/node_modules \
|
||||
node scripts/ci/validate-install-manifests.js
|
||||
Validated 33 install modules, 80 install components, and 7 profiles
|
||||
|
||||
npm test
|
||||
Total Tests: 3159
|
||||
Passed: 3159
|
||||
Failed: 0
|
||||
|
||||
npm run coverage
|
||||
Statements: 89.21%
|
||||
Branches: 79.71%
|
||||
Functions: 93.96%
|
||||
Lines: 89.21%
|
||||
|
||||
npm run security:ioc-scan
|
||||
Supply-chain IOC scan passed
|
||||
node scripts/ci/validate-skills.js
|
||||
Validated 281 skill directories
|
||||
```
|
||||
|
||||
The isolated worktree temporarily reused the canonical ECC checkout's existing
|
||||
`node_modules` through an untracked local symlink. The symlink was removed
|
||||
after validation; no dependency installation or source change was made in the
|
||||
canonical checkout.
|
||||
ESLint and Markdown lint also pass for every changed source file. The complete
|
||||
package dry-run contains the wrapper, environment boundary, skill, and MCP
|
||||
configuration.
|
||||
`node tests/scripts/ito-compute-sponsor.test.js` reached 11 passes and 2 failures;
|
||||
both failures are setup failures because the current worktree lacks `ajv`.
|
||||
`node scripts/ci/validate-install-manifests.js` is blocked by the same missing
|
||||
module. No dependency installation was performed.
|
||||
|
||||
## Test specification
|
||||
|
||||
| Guarantee | Test | Type | Result |
|
||||
|---|---|---|---|
|
||||
| Only `auth`, `find`, and `status` spawn | `tests/scripts/ito-cli-bridge.test.js` | end-to-end process contract | PASS |
|
||||
| `login`, `auth`, `find`, and `status` forward only their reviewed surfaces | `tests/scripts/ito-cli-bridge.test.js` | end-to-end process contract | PASS |
|
||||
| Login output streams before completion and its exit status propagates | `tests/scripts/ito-cli-bridge.test.js` | async process contract | PASS |
|
||||
| `auth --no-browser` fails before spawn | `tests/scripts/ito-cli-bridge.test.js` | negative process contract | PASS |
|
||||
| Full RFQ arguments cross unchanged | `tests/scripts/ito-cli-bridge.test.js` | integration | PASS |
|
||||
| Only required Itô settings cross the child boundary | `tests/scripts/ito-cli-bridge.test.js` | security integration | PASS |
|
||||
| Login scrubs the API key; auth/find/status forward it directly; evals stays isolated | `tests/scripts/ito-cli-bridge.test.js` | security integration | PASS |
|
||||
| Unsupported and dry-run operations fail before spawn | `tests/scripts/ito-cli-bridge.test.js` | negative end-to-end | PASS |
|
||||
| Missing/relative executables fail with exact local guidance | `tests/scripts/ito-cli-bridge.test.js` | negative end-to-end | PASS |
|
||||
| Child output and exit code are preserved | `tests/scripts/ito-cli-bridge.test.js` | end-to-end process contract | PASS |
|
||||
|
||||
@@ -70,6 +70,21 @@ cargo run -- resume <session-id>
|
||||
cargo run -- daemon
|
||||
```
|
||||
|
||||
## Bounded Harness Evaluation
|
||||
|
||||
ECC2 now has an operator-driven configuration registry and promotion gate. Candidate JSON is canonicalized and addressed by its SHA-256 digest, with immutable trace/evidence references. Evaluation uses the same explicit unique seeds for candidate and active baseline through a pluggable Rust trait. The CLI exposes only a deterministic local recorded-measurements evaluator; it makes no network or process calls.
|
||||
|
||||
```bash
|
||||
cargo run -- harness-eval record --config candidate.json --trace-ref trace://run-1 --evidence-ref evidence://review-1
|
||||
cargo run -- harness-eval activate-initial <sha256> --evidence-ref evidence://baseline-approval
|
||||
cargo run -- harness-eval run --candidate <sha256> --baseline <sha256> --seed 1 --seed 2 --measurements measurements.json --evidence-ref evidence://evaluation-1 --min-samples 2 --min-mean-delta 0.05 --min-win-rate 0.5
|
||||
cargo run -- harness-eval audit
|
||||
```
|
||||
|
||||
`measurements.json` contains `{"evaluator":"recorded-v1","scores":{"<candidate>":{"1":0.9},"<baseline>":{"1":0.7}},"health":{"<candidate>":true}}` (with every requested seed present). Promotion requires minimum paired samples, arithmetic-mean delta, and per-seed win rate. SQLite transactions update the active pointer and append audit evidence atomically; a failed or errored candidate-keyed recorded health assertion restores the prior pointer and records rollback evidence. Database triggers reject update/deletion of candidate, evaluation, and audit rows.
|
||||
|
||||
Limitations: this performs one bounded deterministic comparison. It does not autonomously rewrite prompts or `ecc2.toml`, train/fine-tune a model, implement or claim reinforcement learning, call a network service, or run shell-command evaluators. It does not alter running sessions. Evidence references and scores are operator assertions, not authenticated truth. Arithmetic gates do not establish statistical significance. The active pointer is registry state only; it is not automatic deployment into a harness runtime.
|
||||
|
||||
## Validate
|
||||
|
||||
```bash
|
||||
|
||||
@@ -0,0 +1,579 @@
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use serde_json::json;
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
#[test]
|
||||
fn candidate_id_addresses_canonical_config_and_normalized_references() {
|
||||
let first = CandidateSpec::new(
|
||||
json!({"model": "fixed", "limits": {"steps": 3, "tools": ["read"]}}),
|
||||
vec![" trace://two ".into(), "trace://one".into()],
|
||||
vec!["evidence://two".into(), " evidence://one ".into()],
|
||||
)
|
||||
.unwrap();
|
||||
let second = CandidateSpec::new(
|
||||
json!({"limits": {"tools": ["read"], "steps": 3}, "model": "fixed"}),
|
||||
vec!["trace://one".into(), "trace://two".into()],
|
||||
vec!["evidence://one".into(), "evidence://two".into()],
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(first.id, second.id);
|
||||
assert_eq!(first.canonical_config, second.canonical_config);
|
||||
assert_eq!(first.trace_refs, vec!["trace://one", "trace://two"]);
|
||||
assert_eq!(
|
||||
first.evidence_refs,
|
||||
vec!["evidence://one", "evidence://two"]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn candidate_id_changes_when_any_immutable_reference_changes() {
|
||||
let original = CandidateSpec::new(
|
||||
json!({"model": "fixed"}),
|
||||
vec!["trace://one".into()],
|
||||
vec!["evidence://one".into()],
|
||||
)
|
||||
.unwrap();
|
||||
let changed_trace = CandidateSpec::new(
|
||||
json!({"model": "fixed"}),
|
||||
vec!["trace://two".into()],
|
||||
vec!["evidence://one".into()],
|
||||
)
|
||||
.unwrap();
|
||||
let changed_evidence = CandidateSpec::new(
|
||||
json!({"model": "fixed"}),
|
||||
vec!["trace://one".into()],
|
||||
vec!["evidence://two".into()],
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_ne!(original.id, changed_trace.id);
|
||||
assert_ne!(original.id, changed_evidence.id);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn candidate_integrity_rejects_reference_tampering() {
|
||||
let mut candidate = CandidateSpec::new(
|
||||
json!({"model": "fixed"}),
|
||||
vec!["trace://one".into()],
|
||||
vec!["evidence://one".into()],
|
||||
)
|
||||
.unwrap();
|
||||
candidate.trace_refs = vec!["trace://tampered".into()];
|
||||
|
||||
assert!(candidate.verify_integrity().is_err());
|
||||
|
||||
let mut noncanonical = CandidateSpec::new(
|
||||
json!({"model": "fixed"}),
|
||||
vec!["trace://one".into(), "trace://two".into()],
|
||||
vec!["evidence://one".into()],
|
||||
)
|
||||
.unwrap();
|
||||
noncanonical.trace_refs.reverse();
|
||||
assert!(noncanonical.verify_integrity().is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn persisted_candidate_integrity_accepts_only_exact_v1_or_v2_ids() {
|
||||
let candidate = CandidateSpec::new(
|
||||
json!({"model": "fixed", "limits": {"steps": 3}}),
|
||||
vec!["trace://one".into()],
|
||||
vec!["evidence://one".into()],
|
||||
)
|
||||
.unwrap();
|
||||
let legacy_id = candidate.legacy_id();
|
||||
|
||||
candidate.verify_persisted_id(&candidate.id).unwrap();
|
||||
candidate.verify_persisted_id(&legacy_id).unwrap();
|
||||
assert!(candidate
|
||||
.verify_persisted_id(&"a".repeat(64))
|
||||
.unwrap_err()
|
||||
.to_string()
|
||||
.contains("content address"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn policy_requires_explicit_unique_seeds_and_minimum_samples() {
|
||||
let policy = PromotionPolicy {
|
||||
min_samples: 3,
|
||||
min_mean_delta: 0.05,
|
||||
min_win_rate: 2.0 / 3.0,
|
||||
};
|
||||
let duplicate = vec![
|
||||
paired(7, 1.0, 0.0),
|
||||
paired(7, 1.0, 0.0),
|
||||
paired(9, 1.0, 0.0),
|
||||
];
|
||||
assert!(policy.compare(&duplicate).is_err());
|
||||
|
||||
let too_few = vec![paired(7, 1.0, 0.0), paired(8, 1.0, 0.0)];
|
||||
let decision = policy.compare(&too_few).unwrap();
|
||||
assert!(!decision.passed);
|
||||
assert!(decision
|
||||
.failures
|
||||
.iter()
|
||||
.any(|failure| failure.contains("minimum sample")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn thresholds_are_deterministic_and_all_must_pass() {
|
||||
let policy = PromotionPolicy {
|
||||
min_samples: 3,
|
||||
min_mean_delta: 0.1,
|
||||
min_win_rate: 0.75,
|
||||
};
|
||||
let samples = vec![
|
||||
paired(1, 0.9, 0.7),
|
||||
paired(2, 0.8, 0.7),
|
||||
paired(3, 0.6, 0.7),
|
||||
paired(4, 0.8, 0.7),
|
||||
];
|
||||
let first = policy.compare(&samples).unwrap();
|
||||
let second = policy.compare(&samples).unwrap();
|
||||
|
||||
assert_eq!(first, second);
|
||||
assert!(!first.passed);
|
||||
assert_eq!(first.win_rate, 0.75);
|
||||
assert!(first
|
||||
.failures
|
||||
.iter()
|
||||
.any(|failure| failure.contains("mean delta")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn evaluator_is_called_for_each_explicit_seed_in_order() {
|
||||
let mut evaluator = RecordedEvaluator::new(
|
||||
BTreeMap::from([
|
||||
(("candidate".into(), 4), 0.9),
|
||||
(("baseline".into(), 4), 0.5),
|
||||
(("candidate".into(), 2), 0.8),
|
||||
(("baseline".into(), 2), 0.6),
|
||||
]),
|
||||
true,
|
||||
);
|
||||
|
||||
let samples = evaluate_paired(&mut evaluator, "candidate", "baseline", &[4, 2]).unwrap();
|
||||
assert_eq!(samples, vec![paired(4, 0.9, 0.5), paired(2, 0.8, 0.6)]);
|
||||
assert_eq!(
|
||||
evaluator.calls(),
|
||||
&[
|
||||
("candidate".into(), 4),
|
||||
("baseline".into(), 4),
|
||||
("candidate".into(), 2),
|
||||
("baseline".into(), 2)
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
fn paired(seed: u64, candidate_score: f64, baseline_score: f64) -> PairedSample {
|
||||
PairedSample {
|
||||
seed,
|
||||
candidate_score,
|
||||
baseline_score,
|
||||
}
|
||||
}
|
||||
}
|
||||
use anyhow::{bail, Context, Result};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::Value;
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct CandidateSpec {
|
||||
pub id: String,
|
||||
pub canonical_config: String,
|
||||
pub trace_refs: Vec<String>,
|
||||
pub evidence_refs: Vec<String>,
|
||||
}
|
||||
|
||||
impl CandidateSpec {
|
||||
pub fn new(config: Value, trace_refs: Vec<String>, evidence_refs: Vec<String>) -> Result<Self> {
|
||||
let trace_refs = normalize_refs("trace", trace_refs)?;
|
||||
let evidence_refs = normalize_refs("evidence", evidence_refs)?;
|
||||
let canonical_config = serde_json::to_string(&canonicalize(config))?;
|
||||
if canonical_config.len() > 1024 * 1024 {
|
||||
bail!("candidate configuration exceeds 1 MiB");
|
||||
}
|
||||
let artifact = serde_json::to_string(&CanonicalCandidateArtifact {
|
||||
config: serde_json::from_str(&canonical_config)?,
|
||||
trace_refs: &trace_refs,
|
||||
evidence_refs: &evidence_refs,
|
||||
})?;
|
||||
let id = sha256_hex(artifact.as_bytes());
|
||||
Ok(Self {
|
||||
id,
|
||||
canonical_config,
|
||||
trace_refs,
|
||||
evidence_refs,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn verify_integrity(&self) -> Result<()> {
|
||||
self.verify_persisted_id(&self.id)?;
|
||||
if self.id != self.id_for_v2()? {
|
||||
bail!("candidate content address or canonical configuration is invalid");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn legacy_id(&self) -> String {
|
||||
sha256_hex(self.canonical_config.as_bytes())
|
||||
}
|
||||
|
||||
pub fn verify_persisted_id(&self, persisted_id: &str) -> Result<()> {
|
||||
let value: Value = serde_json::from_str(&self.canonical_config)?;
|
||||
let rebuilt = Self::new(value, self.trace_refs.clone(), self.evidence_refs.clone())?;
|
||||
let is_v1 = persisted_id == self.legacy_id();
|
||||
let is_v2 = persisted_id == rebuilt.id;
|
||||
if rebuilt.canonical_config != self.canonical_config
|
||||
|| (!is_v1 && !is_v2)
|
||||
|| (is_v2
|
||||
&& (rebuilt.trace_refs != self.trace_refs
|
||||
|| rebuilt.evidence_refs != self.evidence_refs))
|
||||
{
|
||||
bail!("candidate content address or canonical configuration is invalid");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn id_for_v2(&self) -> Result<String> {
|
||||
Ok(Self::new(
|
||||
serde_json::from_str(&self.canonical_config)?,
|
||||
self.trace_refs.clone(),
|
||||
self.evidence_refs.clone(),
|
||||
)?
|
||||
.id)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct CanonicalCandidateArtifact<'a> {
|
||||
config: Value,
|
||||
trace_refs: &'a [String],
|
||||
evidence_refs: &'a [String],
|
||||
}
|
||||
|
||||
fn normalize_refs(kind: &str, refs: Vec<String>) -> Result<Vec<String>> {
|
||||
if refs.is_empty() || refs.iter().any(|reference| reference.trim().is_empty()) {
|
||||
bail!("at least one non-empty {kind} reference is required");
|
||||
}
|
||||
if refs.len() > 100 || refs.iter().any(|reference| reference.len() > 4096) {
|
||||
bail!("{kind} references exceed bounded limits");
|
||||
}
|
||||
let mut normalized = refs
|
||||
.into_iter()
|
||||
.map(|reference| reference.trim().to_string())
|
||||
.collect::<Vec<_>>();
|
||||
normalized.sort();
|
||||
normalized.dedup();
|
||||
Ok(normalized)
|
||||
}
|
||||
|
||||
fn sha256_hex(bytes: &[u8]) -> String {
|
||||
Sha256::digest(bytes)
|
||||
.iter()
|
||||
.map(|byte| format!("{byte:02x}"))
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn canonicalize(value: Value) -> Value {
|
||||
match value {
|
||||
Value::Object(entries) => Value::Object(
|
||||
entries
|
||||
.into_iter()
|
||||
.map(|(key, value)| (key, canonicalize(value)))
|
||||
.collect::<BTreeMap<_, _>>()
|
||||
.into_iter()
|
||||
.collect(),
|
||||
),
|
||||
Value::Array(values) => Value::Array(values.into_iter().map(canonicalize).collect()),
|
||||
other => other,
|
||||
}
|
||||
}
|
||||
|
||||
pub trait Evaluator {
|
||||
fn name(&self) -> &str;
|
||||
fn evaluate(&mut self, candidate_id: &str, seed: u64) -> Result<f64>;
|
||||
fn health_check(&mut self, candidate_id: &str) -> Result<bool>;
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct RecordedEvidence {
|
||||
pub evaluator: String,
|
||||
pub scores: BTreeMap<String, BTreeMap<u64, f64>>,
|
||||
pub health: BTreeMap<String, bool>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct HealthEvidenceSnapshot {
|
||||
pub schema_version: u8,
|
||||
pub evaluator: String,
|
||||
pub candidate_id: String,
|
||||
pub asserted_healthy: bool,
|
||||
}
|
||||
|
||||
impl HealthEvidenceSnapshot {
|
||||
pub fn new(evaluator: &str, candidate_id: &str, asserted_healthy: bool) -> Result<Self> {
|
||||
let snapshot = Self {
|
||||
schema_version: 1,
|
||||
evaluator: evaluator.to_string(),
|
||||
candidate_id: candidate_id.to_string(),
|
||||
asserted_healthy,
|
||||
};
|
||||
snapshot.verify()?;
|
||||
Ok(snapshot)
|
||||
}
|
||||
|
||||
pub fn canonical_json(&self) -> Result<String> {
|
||||
self.verify()?;
|
||||
Ok(serde_json::to_string(self)?)
|
||||
}
|
||||
|
||||
pub fn digest(&self) -> Result<String> {
|
||||
Ok(sha256_hex(self.canonical_json()?.as_bytes()))
|
||||
}
|
||||
|
||||
pub fn verify(&self) -> Result<()> {
|
||||
if self.schema_version != 1
|
||||
|| self.evaluator != "recorded-v1"
|
||||
|| self.candidate_id.len() != 64
|
||||
|| !self
|
||||
.candidate_id
|
||||
.bytes()
|
||||
.all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte))
|
||||
{
|
||||
bail!("invalid canonical health evidence snapshot");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
pub struct RecordedEvaluator {
|
||||
name: String,
|
||||
scores: BTreeMap<(String, u64), f64>,
|
||||
health_ok: bool,
|
||||
health_candidate: Option<String>,
|
||||
calls: Vec<(String, u64)>,
|
||||
}
|
||||
|
||||
impl RecordedEvaluator {
|
||||
#[cfg(test)]
|
||||
pub fn new(scores: BTreeMap<(String, u64), f64>, health_ok: bool) -> Self {
|
||||
Self {
|
||||
name: "recorded-v1".into(),
|
||||
scores,
|
||||
health_ok,
|
||||
health_candidate: None,
|
||||
calls: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn from_evidence(evidence: RecordedEvidence) -> Result<Self> {
|
||||
if evidence.evaluator != "recorded-v1" {
|
||||
bail!("CLI evidence evaluator must be recorded-v1");
|
||||
}
|
||||
let score_count = evidence.scores.values().map(BTreeMap::len).sum::<usize>();
|
||||
if score_count > 20_000 || evidence.scores.keys().any(|id| id.len() != 64) {
|
||||
bail!("recorded evidence exceeds bounded score or candidate limits");
|
||||
}
|
||||
if evidence.health.len() != 1 {
|
||||
bail!("exactly one candidate-keyed health assertion is required");
|
||||
}
|
||||
let (health_candidate, health_ok) = evidence
|
||||
.health
|
||||
.into_iter()
|
||||
.next()
|
||||
.context("candidate-keyed health evidence is required")?;
|
||||
let scores = evidence
|
||||
.scores
|
||||
.into_iter()
|
||||
.flat_map(|(id, values)| {
|
||||
values
|
||||
.into_iter()
|
||||
.map(move |(seed, score)| ((id.clone(), seed), score))
|
||||
})
|
||||
.collect();
|
||||
Ok(Self {
|
||||
name: evidence.evaluator,
|
||||
scores,
|
||||
health_ok,
|
||||
health_candidate: Some(health_candidate),
|
||||
calls: Vec::new(),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn health_evidence_snapshot(&self) -> Result<HealthEvidenceSnapshot> {
|
||||
HealthEvidenceSnapshot::new(
|
||||
&self.name,
|
||||
self.health_candidate
|
||||
.as_deref()
|
||||
.context("candidate-keyed health evidence is required")?,
|
||||
self.health_ok,
|
||||
)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub fn calls(&self) -> &[(String, u64)] {
|
||||
&self.calls
|
||||
}
|
||||
}
|
||||
|
||||
impl Evaluator for RecordedEvaluator {
|
||||
fn name(&self) -> &str {
|
||||
&self.name
|
||||
}
|
||||
|
||||
fn evaluate(&mut self, candidate_id: &str, seed: u64) -> Result<f64> {
|
||||
self.calls.push((candidate_id.to_string(), seed));
|
||||
let score = *self
|
||||
.scores
|
||||
.get(&(candidate_id.to_string(), seed))
|
||||
.with_context(|| format!("missing recorded score for {candidate_id} seed {seed}"))?;
|
||||
if !score.is_finite() || !(0.0..=1.0).contains(&score) {
|
||||
bail!("score must be finite and between 0 and 1");
|
||||
}
|
||||
Ok(score)
|
||||
}
|
||||
|
||||
fn health_check(&mut self, candidate_id: &str) -> Result<bool> {
|
||||
if self
|
||||
.health_candidate
|
||||
.as_deref()
|
||||
.is_some_and(|expected| expected != candidate_id)
|
||||
{
|
||||
bail!("health evidence does not match promoted candidate");
|
||||
}
|
||||
Ok(self.health_ok)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct PairedSample {
|
||||
pub seed: u64,
|
||||
pub candidate_score: f64,
|
||||
pub baseline_score: f64,
|
||||
}
|
||||
|
||||
pub fn evaluate_paired(
|
||||
evaluator: &mut dyn Evaluator,
|
||||
candidate_id: &str,
|
||||
baseline_id: &str,
|
||||
seeds: &[u64],
|
||||
) -> Result<Vec<PairedSample>> {
|
||||
if seeds.is_empty() {
|
||||
bail!("at least one explicit seed is required");
|
||||
}
|
||||
if seeds.len() > 10_000 {
|
||||
bail!("seed count exceeds 10000");
|
||||
}
|
||||
if seeds.iter().copied().collect::<BTreeSet<_>>().len() != seeds.len() {
|
||||
bail!("seeds must be unique");
|
||||
}
|
||||
seeds
|
||||
.iter()
|
||||
.map(|seed| {
|
||||
Ok(PairedSample {
|
||||
seed: *seed,
|
||||
candidate_score: evaluator.evaluate(candidate_id, *seed)?,
|
||||
baseline_score: evaluator.evaluate(baseline_id, *seed)?,
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]
|
||||
pub struct PromotionPolicy {
|
||||
pub min_samples: usize,
|
||||
pub min_mean_delta: f64,
|
||||
pub min_win_rate: f64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct Comparison {
|
||||
pub passed: bool,
|
||||
pub sample_count: usize,
|
||||
pub candidate_mean: f64,
|
||||
pub baseline_mean: f64,
|
||||
pub mean_delta: f64,
|
||||
pub win_rate: f64,
|
||||
pub failures: Vec<String>,
|
||||
}
|
||||
|
||||
impl PromotionPolicy {
|
||||
pub fn validate(self) -> Result<()> {
|
||||
if self.min_samples == 0 {
|
||||
bail!("minimum samples must be positive");
|
||||
}
|
||||
if !self.min_mean_delta.is_finite() {
|
||||
bail!("minimum mean delta must be finite");
|
||||
}
|
||||
if !self.min_win_rate.is_finite() || !(0.0..=1.0).contains(&self.min_win_rate) {
|
||||
bail!("minimum win rate must be between 0 and 1");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn compare(self, samples: &[PairedSample]) -> Result<Comparison> {
|
||||
self.validate()?;
|
||||
if samples.is_empty() {
|
||||
bail!("samples cannot be empty");
|
||||
}
|
||||
if samples
|
||||
.iter()
|
||||
.map(|sample| sample.seed)
|
||||
.collect::<BTreeSet<_>>()
|
||||
.len()
|
||||
!= samples.len()
|
||||
{
|
||||
bail!("sample seeds must be unique");
|
||||
}
|
||||
if samples.iter().any(|s| {
|
||||
!s.candidate_score.is_finite()
|
||||
|| !s.baseline_score.is_finite()
|
||||
|| !(0.0..=1.0).contains(&s.candidate_score)
|
||||
|| !(0.0..=1.0).contains(&s.baseline_score)
|
||||
}) {
|
||||
bail!("scores must be finite and between 0 and 1");
|
||||
}
|
||||
let count = samples.len();
|
||||
let candidate_mean = samples.iter().map(|s| s.candidate_score).sum::<f64>() / count as f64;
|
||||
let baseline_mean = samples.iter().map(|s| s.baseline_score).sum::<f64>() / count as f64;
|
||||
let mean_delta = candidate_mean - baseline_mean;
|
||||
let win_rate = samples
|
||||
.iter()
|
||||
.filter(|s| s.candidate_score > s.baseline_score)
|
||||
.count() as f64
|
||||
/ count as f64;
|
||||
let mut failures = Vec::new();
|
||||
if count < self.min_samples {
|
||||
failures.push(format!(
|
||||
"minimum sample count is {}, got {count}",
|
||||
self.min_samples
|
||||
));
|
||||
}
|
||||
if mean_delta < self.min_mean_delta {
|
||||
failures.push(format!(
|
||||
"mean delta {mean_delta:.6} is below {:.6}",
|
||||
self.min_mean_delta
|
||||
));
|
||||
}
|
||||
if win_rate < self.min_win_rate {
|
||||
failures.push(format!(
|
||||
"win rate {win_rate:.6} is below {:.6}",
|
||||
self.min_win_rate
|
||||
));
|
||||
}
|
||||
Ok(Comparison {
|
||||
passed: failures.is_empty(),
|
||||
sample_count: count,
|
||||
candidate_mean,
|
||||
baseline_mean,
|
||||
mean_delta,
|
||||
win_rate,
|
||||
failures,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
mod comms;
|
||||
mod config;
|
||||
mod harness_eval;
|
||||
mod notifications;
|
||||
mod observability;
|
||||
mod session;
|
||||
@@ -108,6 +109,11 @@ impl OptionalWorktreePolicyArgs {
|
||||
|
||||
#[derive(clap::Subcommand, Debug)]
|
||||
enum Commands {
|
||||
/// Run bounded, deterministic harness configuration evaluations
|
||||
HarnessEval {
|
||||
#[command(subcommand)]
|
||||
command: HarnessEvalCommands,
|
||||
},
|
||||
/// Launch the TUI dashboard
|
||||
Dashboard,
|
||||
/// Start a new agent session
|
||||
@@ -437,6 +443,46 @@ enum Commands {
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(clap::Subcommand, Debug)]
|
||||
enum HarnessEvalCommands {
|
||||
/// Record an immutable content-addressed candidate from a local JSON file
|
||||
Record {
|
||||
#[arg(long)]
|
||||
config: PathBuf,
|
||||
#[arg(long = "trace-ref", required = true)]
|
||||
trace_refs: Vec<String>,
|
||||
#[arg(long = "evidence-ref", required = true)]
|
||||
evidence_refs: Vec<String>,
|
||||
},
|
||||
/// Set the first baseline; subsequent changes require evaluation
|
||||
ActivateInitial {
|
||||
candidate_id: String,
|
||||
#[arg(long)]
|
||||
evidence_ref: String,
|
||||
},
|
||||
/// Evaluate paired scores and conditionally promote with a health gate
|
||||
Run {
|
||||
#[arg(long)]
|
||||
candidate: String,
|
||||
#[arg(long)]
|
||||
baseline: String,
|
||||
#[arg(long = "seed", required = true)]
|
||||
seeds: Vec<u64>,
|
||||
#[arg(long)]
|
||||
measurements: PathBuf,
|
||||
#[arg(long)]
|
||||
evidence_ref: String,
|
||||
#[arg(long)]
|
||||
min_samples: usize,
|
||||
#[arg(long)]
|
||||
min_mean_delta: f64,
|
||||
#[arg(long)]
|
||||
min_win_rate: f64,
|
||||
},
|
||||
/// Show append-only promotion audit entries
|
||||
Audit,
|
||||
}
|
||||
|
||||
#[derive(clap::Subcommand, Debug)]
|
||||
enum MessageCommands {
|
||||
/// Send a structured message between sessions
|
||||
@@ -1345,6 +1391,37 @@ struct DotenvMemoryEntry {
|
||||
details: BTreeMap<String, String>,
|
||||
}
|
||||
|
||||
fn read_bounded_file(path: &Path, max_bytes: u64, label: &str) -> Result<Vec<u8>> {
|
||||
let mut options = File::options();
|
||||
options.read(true);
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
options.custom_flags(libc::O_NONBLOCK);
|
||||
}
|
||||
let file = options
|
||||
.open(path)
|
||||
.with_context(|| format!("Failed to open {}", path.display()))?;
|
||||
let metadata = file
|
||||
.metadata()
|
||||
.with_context(|| format!("Failed to inspect {}", path.display()))?;
|
||||
if !metadata.is_file() {
|
||||
anyhow::bail!("{label} must be a regular file");
|
||||
}
|
||||
|
||||
let read_limit = max_bytes
|
||||
.checked_add(1)
|
||||
.context("bounded input byte limit is too large")?;
|
||||
let mut content = Vec::new();
|
||||
file.take(read_limit)
|
||||
.read_to_end(&mut content)
|
||||
.with_context(|| format!("Failed to read {}", path.display()))?;
|
||||
if content.len() as u64 > max_bytes {
|
||||
anyhow::bail!("{label} exceeds the {max_bytes}-byte limit");
|
||||
}
|
||||
Ok(content)
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
tracing_subscriber::fmt()
|
||||
@@ -1357,6 +1434,75 @@ async fn main() -> Result<()> {
|
||||
let db = session::store::StateStore::open(&cfg.db_path)?;
|
||||
|
||||
match cli.command {
|
||||
Some(Commands::HarnessEval { command }) => match command {
|
||||
HarnessEvalCommands::Record {
|
||||
config,
|
||||
trace_refs,
|
||||
evidence_refs,
|
||||
} => {
|
||||
let value: serde_json::Value = serde_json::from_slice(&read_bounded_file(
|
||||
&config,
|
||||
1_048_576,
|
||||
"candidate configuration",
|
||||
)?)
|
||||
.with_context(|| format!("Invalid JSON in {}", config.display()))?;
|
||||
let candidate = harness_eval::CandidateSpec::new(value, trace_refs, evidence_refs)?;
|
||||
db.record_harness_candidate(&candidate)?;
|
||||
println!("{}", candidate.id);
|
||||
}
|
||||
HarnessEvalCommands::ActivateInitial {
|
||||
candidate_id,
|
||||
evidence_ref,
|
||||
} => {
|
||||
db.activate_initial_harness(&candidate_id, &evidence_ref)?;
|
||||
println!("Activated initial baseline: {candidate_id}");
|
||||
}
|
||||
HarnessEvalCommands::Run {
|
||||
candidate,
|
||||
baseline,
|
||||
seeds,
|
||||
measurements,
|
||||
evidence_ref,
|
||||
min_samples,
|
||||
min_mean_delta,
|
||||
min_win_rate,
|
||||
} => {
|
||||
use harness_eval::Evaluator;
|
||||
let evidence: harness_eval::RecordedEvidence = serde_json::from_slice(
|
||||
&read_bounded_file(&measurements, 8_388_608, "recorded measurements")?,
|
||||
)
|
||||
.with_context(|| {
|
||||
format!("Invalid recorded evidence in {}", measurements.display())
|
||||
})?;
|
||||
let mut evaluator = harness_eval::RecordedEvaluator::from_evidence(evidence)?;
|
||||
let evaluator_name = evaluator.name().to_string();
|
||||
let health_evidence = evaluator.health_evidence_snapshot()?;
|
||||
let samples =
|
||||
harness_eval::evaluate_paired(&mut evaluator, &candidate, &baseline, &seeds)?;
|
||||
let policy = harness_eval::PromotionPolicy {
|
||||
min_samples,
|
||||
min_mean_delta,
|
||||
min_win_rate,
|
||||
};
|
||||
let outcome = db.evaluate_promote_and_health_check(
|
||||
&candidate,
|
||||
&baseline,
|
||||
&evaluator_name,
|
||||
&samples,
|
||||
policy,
|
||||
&evidence_ref,
|
||||
&health_evidence,
|
||||
|id| evaluator.health_check(id),
|
||||
)?;
|
||||
println!("{}", serde_json::to_string_pretty(&outcome)?);
|
||||
}
|
||||
HarnessEvalCommands::Audit => {
|
||||
println!(
|
||||
"{}",
|
||||
serde_json::to_string_pretty(&db.harness_audit_entries()?)?
|
||||
);
|
||||
}
|
||||
},
|
||||
Some(Commands::Dashboard) | None => {
|
||||
tui::app::run(db, cfg).await?;
|
||||
}
|
||||
@@ -8533,6 +8679,96 @@ mod tests {
|
||||
assert!(!policy.resolve(&cfg));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn harness_eval_cli_requires_explicit_bounded_inputs() {
|
||||
let cli = Cli::try_parse_from([
|
||||
"ecc",
|
||||
"harness-eval",
|
||||
"run",
|
||||
"--candidate",
|
||||
"candidate",
|
||||
"--baseline",
|
||||
"baseline",
|
||||
"--seed",
|
||||
"1",
|
||||
"--seed",
|
||||
"2",
|
||||
"--measurements",
|
||||
"scores.json",
|
||||
"--evidence-ref",
|
||||
"evidence://run",
|
||||
"--min-samples",
|
||||
"2",
|
||||
"--min-mean-delta",
|
||||
"0.1",
|
||||
"--min-win-rate",
|
||||
"0.5",
|
||||
])
|
||||
.expect("valid harness evaluation command");
|
||||
match cli.command {
|
||||
Some(Commands::HarnessEval {
|
||||
command:
|
||||
HarnessEvalCommands::Run {
|
||||
seeds, min_samples, ..
|
||||
},
|
||||
}) => {
|
||||
assert_eq!(seeds, vec![1, 2]);
|
||||
assert_eq!(min_samples, 2);
|
||||
}
|
||||
other => panic!("unexpected command: {other:?}"),
|
||||
}
|
||||
assert!(Cli::try_parse_from([
|
||||
"ecc",
|
||||
"harness-eval",
|
||||
"run",
|
||||
"--candidate",
|
||||
"c",
|
||||
"--baseline",
|
||||
"b"
|
||||
])
|
||||
.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn harness_eval_bounded_input_rejects_content_over_limit() -> Result<()> {
|
||||
let tempdir = TestDir::new("harness-eval-oversized-input")?;
|
||||
let input = tempdir.path().join("measurements.json");
|
||||
fs::write(&input, b"12345")?;
|
||||
|
||||
let error = read_bounded_file(&input, 4, "recorded measurements")
|
||||
.expect_err("input larger than the byte limit must fail");
|
||||
|
||||
assert_eq!(
|
||||
error.to_string(),
|
||||
"recorded measurements exceeds the 4-byte limit"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn harness_eval_bounded_input_rejects_non_regular_file() -> Result<()> {
|
||||
use std::ffi::CString;
|
||||
use std::os::unix::ffi::OsStrExt;
|
||||
|
||||
let tempdir = TestDir::new("harness-eval-non-regular-input")?;
|
||||
let input = tempdir.path().join("measurements.fifo");
|
||||
let input_c = CString::new(input.as_os_str().as_bytes())?;
|
||||
// SAFETY: `input_c` is a valid, NUL-terminated path and the mode is valid.
|
||||
let result = unsafe { libc::mkfifo(input_c.as_ptr(), 0o600) };
|
||||
if result != 0 {
|
||||
return Err(std::io::Error::last_os_error().into());
|
||||
}
|
||||
let error = read_bounded_file(&input, 4, "recorded measurements")
|
||||
.expect_err("non-regular input must fail");
|
||||
|
||||
assert_eq!(
|
||||
error.to_string(),
|
||||
"recorded measurements must be a regular file"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn worktree_policy_explicit_flags_override_config_setting() {
|
||||
let mut cfg = Config::default();
|
||||
|
||||
+1236
-83
File diff suppressed because it is too large
Load Diff
@@ -8,7 +8,7 @@
|
||||
"ito-compute": {
|
||||
"command": "node",
|
||||
"args": ["/absolute/path/to/ito-cloud-runtime/cli/ito-compute-cli/dist/bin/ito-mcp.js"],
|
||||
"description": "Opt-in local Itô compute MCP. The canonical package is unpublished and must be built from Ito-Markets/ito-cloud-runtime/cli/ito-compute-cli. Exposes only ito_auth, ito_find, and ito_status; inject ITO_API_KEY from the launching environment."
|
||||
"description": "Opt-in local Itô compute MCP. The canonical package is unpublished and must be built from Ito-Markets/ito-cloud-runtime/cli/ito-compute-cli. Exposes only ito_auth, ito_find, and ito_status. ito_auth validates existing credentials; it does not start device login. Use ecc ito login [--no-browser] for device authorization, which stores tokens in macOS Keychain by default; explicit file fallback must retain owner-only settings. ECC itself performs no browser automation. ITO_API_KEY is forwarded directly to auth, find, and status when configured; ITO_AUTH_MODE=legacy is not required."
|
||||
},
|
||||
"jira": {
|
||||
"command": "uvx",
|
||||
|
||||
+6
-2
@@ -4,7 +4,7 @@ const { spawnSync } = require('child_process');
|
||||
const path = require('path');
|
||||
const { listAvailableLanguages } = require('./lib/install-executor');
|
||||
const { getComputeSponsorCopy } = require('./lib/compute-sponsor');
|
||||
const { createSafeItoInvocationEnvironment } = require('./lib/ito-environment');
|
||||
const { createSafeItoInvocationEnvironment, getInvocationCommand } = require('./lib/ito-environment');
|
||||
|
||||
const COMMANDS = {
|
||||
install: {
|
||||
@@ -148,6 +148,7 @@ Examples:
|
||||
ecc catalog show framework:nextjs
|
||||
ecc consult "security reviews"
|
||||
ecc control-pane --port 8765
|
||||
ecc ito login [--no-browser]
|
||||
ecc ito auth
|
||||
ecc ito find --gpu h200 --count 8 --nodes 1 --gpus-per-node 8 --days 30 --storage-tb 1 --start-window 2099-08-15 --max-rate 3.00 --form-factor bare_metal --contract-type reservation --fabric infiniband --region us-east-1
|
||||
ecc ito status --json
|
||||
@@ -241,6 +242,7 @@ function runCommand(commandName, args) {
|
||||
if (!command) {
|
||||
throw new Error(`Unknown command: ${commandName}`);
|
||||
}
|
||||
const isItoLogin = commandName === 'ito' && getInvocationCommand(args) === 'login';
|
||||
const result = spawnSync(
|
||||
process.execPath,
|
||||
[path.join(__dirname, command.script), ...args],
|
||||
@@ -253,7 +255,9 @@ function runCommand(commandName, args) {
|
||||
}),
|
||||
}
|
||||
: process.env,
|
||||
stdio: commandName === 'memory'
|
||||
stdio: isItoLogin
|
||||
? 'inherit'
|
||||
: commandName === 'memory'
|
||||
? ['inherit', 'pipe', 'pipe']
|
||||
: ['pipe', 'pipe', 'pipe'],
|
||||
encoding: 'utf8',
|
||||
|
||||
+20
-8
@@ -10,7 +10,7 @@ const {
|
||||
getInvocationCommand,
|
||||
} = require("./lib/ito-environment");
|
||||
|
||||
const SUPPORTED_COMMANDS = Object.freeze(["auth", "find", "status", "evals"]);
|
||||
const SUPPORTED_COMMANDS = Object.freeze(["login", "auth", "find", "status", "evals"]);
|
||||
const CANONICAL_REPOSITORY = "https://github.com/Ito-Markets/ito-cloud-runtime.git";
|
||||
const CANONICAL_PACKAGE_PATH = "cli/ito-compute-cli";
|
||||
const CANONICAL_ENTRY_SEGMENTS = Object.freeze([
|
||||
@@ -28,15 +28,20 @@ function showHelp() {
|
||||
ECC × Itô local CLI bridge
|
||||
|
||||
Usage:
|
||||
ecc ito login [--no-browser]
|
||||
ecc ito auth
|
||||
ecc ito find <all required RFQ options>
|
||||
ecc ito status
|
||||
ecc ito evals --cluster <id> --live-sixtytwo --nodes <list> --config-dir <dir>
|
||||
ecc ito <auth|find|status|evals> --json
|
||||
ecc ito <login|auth|find|status|evals> --json
|
||||
|
||||
The bridge invokes the separately installed canonical Itô CLI and returns its
|
||||
real stdout, stderr, and exit code unchanged. It performs no browser navigation
|
||||
and adds no lock, workload, inference, or purchase path.
|
||||
real stdout, stderr, and exit code unchanged. "ecc ito login" delegates to the
|
||||
canonical CLI's device authorization. It opens the Itô verification page by default
|
||||
and persists its device token in macOS Keychain. Pass --no-browser to
|
||||
suppress that handoff. ECC itself performs no browser automation and adds no
|
||||
lock, workload, inference, or purchase path.
|
||||
"ecc ito auth" is validation-only and never starts device login.
|
||||
|
||||
Important:
|
||||
- "find" reads live inventory and submits an authenticated RFQ.
|
||||
@@ -67,9 +72,11 @@ The same package's MCP server exposes only:
|
||||
Configure the MCP command as "node" with this absolute argument:
|
||||
/absolute/path/to/ito-cloud-runtime/${CANONICAL_PACKAGE_PATH}/dist/bin/ito-mcp.js
|
||||
|
||||
For auth, find, and status, inject ITO_API_KEY into the child process from
|
||||
1Password or the launching environment. Never put the key in arguments,
|
||||
tracked files, or chat.
|
||||
Device login never inherits ITO_API_KEY. The auth, find, and status commands
|
||||
forward ITO_API_KEY directly when configured; ITO_AUTH_MODE=legacy is not
|
||||
required. The canonical client stores device credentials in macOS Keychain by
|
||||
default; file-token fallback remains explicit and must use restrictive settings.
|
||||
Never put a key or token in arguments, tracked files, or chat.
|
||||
|
||||
Live node qualification requires ITO_ENABLE_SIXTYTWO_LIVE=1,
|
||||
--live-sixtytwo, an explicit node list, and an existing absolute config
|
||||
@@ -154,9 +161,12 @@ function parseArgs(argv, environment = process.env) {
|
||||
const command = withoutJson.shift();
|
||||
if (!SUPPORTED_COMMANDS.includes(command)) {
|
||||
throw new Error(
|
||||
`Unsupported Itô command "${command || "(missing)"}"; ECC permits only auth, find, status, and evals.`
|
||||
`Unsupported Itô command "${command || "(missing)"}"; ECC permits only login, auth, find, status, and evals.`
|
||||
);
|
||||
}
|
||||
if (command === "auth" && withoutJson.includes("--no-browser")) {
|
||||
throw new Error("--no-browser is valid only for ecc ito login; auth is validation-only.");
|
||||
}
|
||||
if (command === "evals") {
|
||||
validateNodeQualificationArgs(withoutJson, environment);
|
||||
}
|
||||
@@ -255,12 +265,14 @@ function invokeIto(executable, args, environment = process.env) {
|
||||
const invocation = buildInvocation(executable, args);
|
||||
const command = getInvocationCommand(args);
|
||||
const isNodeQualification = command === "evals";
|
||||
const isDeviceLogin = command === "login";
|
||||
const result = spawnSync(invocation.executable, invocation.args, {
|
||||
cwd: process.cwd(),
|
||||
encoding: "utf8",
|
||||
// Keep policy helpers immutable for callers, but give child-process
|
||||
// instrumentation its own mutable copy (for example NODE_V8_COVERAGE).
|
||||
env: { ...createSafeItoInvocationEnvironment(environment, args) },
|
||||
stdio: isDeviceLogin ? "inherit" : ["pipe", "pipe", "pipe"],
|
||||
maxBuffer: MAX_OUTPUT_BYTES,
|
||||
timeout: isNodeQualification ? NODE_QUALIFICATION_TIMEOUT_MS : undefined,
|
||||
shell: false,
|
||||
|
||||
@@ -27,6 +27,9 @@ const ITO_RUNTIME_ENVIRONMENT_KEYS = Object.freeze([
|
||||
"ITO_API_KEY",
|
||||
"ITO_API_URL",
|
||||
"ITO_INVENTORY_URL",
|
||||
"ITO_AUTH_MODE",
|
||||
"ITO_ALLOW_FILE_TOKEN",
|
||||
"ITO_TOKEN_FILE",
|
||||
]);
|
||||
|
||||
const ITO_EVAL_ENVIRONMENT_KEYS = Object.freeze([
|
||||
@@ -42,7 +45,7 @@ const ECC_ITO_CONTROL_KEYS = Object.freeze([
|
||||
"ECC_ITO_CLI_EXECUTABLE",
|
||||
"NODE_ENV",
|
||||
]);
|
||||
const ITO_RUNTIME_COMMANDS = new Set(["auth", "find", "status"]);
|
||||
const ITO_RUNTIME_COMMANDS = new Set(["login", "auth", "find", "status"]);
|
||||
|
||||
function copyDefined(source, target, key) {
|
||||
if (typeof source[key] === "string") {
|
||||
@@ -61,6 +64,7 @@ function createSafeItoEnvironment(source = process.env, options = {}) {
|
||||
|
||||
if (options.includeItoRuntime) {
|
||||
for (const key of ITO_RUNTIME_ENVIRONMENT_KEYS) {
|
||||
if (key === "ITO_API_KEY" && options.includeItoApiKey !== true) continue;
|
||||
copyDefined(source, safe, key);
|
||||
}
|
||||
}
|
||||
@@ -93,6 +97,7 @@ function createSafeItoInvocationEnvironment(
|
||||
return createSafeItoEnvironment(source, {
|
||||
includeControls: options.includeControls === true,
|
||||
includeItoRuntime: ITO_RUNTIME_COMMANDS.has(command),
|
||||
includeItoApiKey: ["auth", "find", "status"].includes(command),
|
||||
includeItoEvals: command === "evals",
|
||||
});
|
||||
}
|
||||
|
||||
@@ -8,8 +8,8 @@ metadata:
|
||||
# Itô Compute
|
||||
|
||||
Use the canonical Itô compute CLI or MCP server. ECC does not implement a
|
||||
parallel client, browser handoff, local simulation, reservation, workload
|
||||
runner, or inference server.
|
||||
parallel client, local simulation, reservation, workload runner, or inference
|
||||
server. ECC itself does no browser automation.
|
||||
|
||||
## Install the canonical local package
|
||||
|
||||
@@ -30,20 +30,30 @@ Set `ECC_ITO_CLI_EXECUTABLE` to the explicit absolute built entry:
|
||||
```
|
||||
|
||||
ECC never discovers this credential-bearing client through `PATH`.
|
||||
Inject `ITO_API_KEY` through 1Password or the launching process environment.
|
||||
Never put it in arguments, tracked files, MCP results, logs, or chat.
|
||||
`ecc ito login` performs device authorization and never inherits `ITO_API_KEY`.
|
||||
The validation-only `auth`, plus `find` and `status`, forward `ITO_API_KEY`
|
||||
directly when configured; `ITO_AUTH_MODE=legacy` is not required. Never put a
|
||||
key or token in arguments, tracked files, MCP results, logs, or chat.
|
||||
|
||||
## CLI workflow
|
||||
|
||||
1. Run `ecc ito auth` before the first operation.
|
||||
2. Before `ecc ito find`, obtain explicit buyer authority to submit an RFQ.
|
||||
1. Run `ecc ito login` before the first operation. ECC delegates this to the
|
||||
canonical CLI's device authorization, which opens the Itô verification page
|
||||
by default and persists a device token in macOS Keychain. Use
|
||||
`ecc ito login --no-browser` to suppress the page handoff. ECC itself does no
|
||||
browser automation.
|
||||
Device tokens use macOS Keychain by default. File-token fallback is explicit
|
||||
and its directory and token file must remain owner-only (0700 and 0600).
|
||||
2. Run `ecc ito auth` to validate existing credentials; it never starts login
|
||||
and rejects `--no-browser`.
|
||||
3. Before `ecc ito find`, obtain explicit buyer authority to submit an RFQ.
|
||||
- Require `gpu`, `count`, whole `days`, `max-rate`, `nodes`,
|
||||
`gpus-per-node`, `storage-tb`, `start-window`, `form-factor`,
|
||||
`contract-type`, `fabric`, `region`, and the split-fill decision.
|
||||
- Require `count == nodes * gpus-per-node`; never derive topology.
|
||||
- Use `any` only when the buyer explicitly accepts any fabric or region.
|
||||
- Omitted `--allow-split` means false.
|
||||
3. Run the live RFQ command:
|
||||
4. Run the live RFQ command:
|
||||
|
||||
```sh
|
||||
ecc ito find \
|
||||
@@ -61,7 +71,7 @@ Never put it in arguments, tracked files, MCP results, logs, or chat.
|
||||
--region us-east-1
|
||||
```
|
||||
|
||||
4. Run `ecc ito status` to inspect RFQs and procurement orders.
|
||||
5. Run `ecc ito status` to inspect RFQs and procurement orders.
|
||||
After an ambiguous transport failure, check status before repeating `find`.
|
||||
|
||||
Inventory prices are indicative. An RFQ is not reserved capacity. Treat a rate
|
||||
@@ -117,7 +127,8 @@ The server exposes only:
|
||||
- `ito_find`
|
||||
- `ito_status`
|
||||
|
||||
Use `ito_auth`, gather explicit buyer authority and every hard constraint, call
|
||||
`ito_auth` validates existing credentials; it does not start device login. Use
|
||||
`ito_auth`, gather explicit buyer authority and every hard constraint, call
|
||||
`ito_find`, then poll with `ito_status` when needed.
|
||||
|
||||
## Unsupported operations
|
||||
|
||||
@@ -35,6 +35,7 @@ function main() {
|
||||
["documents only the real CLI commands and MCP tools", () => {
|
||||
const skill = read("skills/ito-compute/SKILL.md");
|
||||
for (const command of [
|
||||
"ecc ito login",
|
||||
"ecc ito auth",
|
||||
"ecc ito find",
|
||||
"ecc ito status",
|
||||
@@ -57,12 +58,34 @@ function main() {
|
||||
assert.match(skill, /ECC_ITO_CLI_EXECUTABLE/);
|
||||
assert.match(skill, /explicit absolute built entry/);
|
||||
assert.match(skill, /never discovers[^\n]*through `PATH`/);
|
||||
assert.match(skill, /ecc ito login --no-browser/);
|
||||
assert.match(skill, /auth.*validat/i);
|
||||
assert.match(skill, /--no-browser/);
|
||||
assert.match(skill, /macOS Keychain/i);
|
||||
assert.match(skill, /(?:auth|find|status).*ITO_API_KEY/i);
|
||||
assert.match(skill, /ITO_AUTH_MODE=legacy[^.]*not required/i);
|
||||
assert.match(skill, /ECC (?:itself )?(?:does|performs) no browser automation/i);
|
||||
assert.match(skill, /ITO_ENABLE_SIXTYTWO_LIVE/);
|
||||
assert.match(skill, /sixtytwo-cli==0\.3\.33/);
|
||||
assert.match(skill, /explicit node/i);
|
||||
assert.match(skill, /cannot (?:rent|launch|recover|repair)/i);
|
||||
assert.doesNotMatch(skill, /npm link/);
|
||||
}],
|
||||
["keeps README and integration docs aligned with the separated auth contract", () => {
|
||||
for (const relativePath of [
|
||||
"README.md",
|
||||
"docs/design/ecc-ito-compute-integration.md",
|
||||
]) {
|
||||
const source = read(relativePath);
|
||||
assert.match(source, /ecc ito login \[?--no-browser\]?/i, relativePath);
|
||||
assert.match(source, /ecc ito auth/i, relativePath);
|
||||
assert.match(source, /auth.*validat/i, relativePath);
|
||||
assert.match(source, /login.*(?:Keychain|device authorization)/is, relativePath);
|
||||
assert.doesNotMatch(source, /ecc ito auth --no-browser/i, relativePath);
|
||||
assert.match(source, /ITO_API_KEY.*(?:auth|find|status)/is, relativePath);
|
||||
assert.match(source, /ITO_AUTH_MODE=legacy[^.]*not required/i, relativePath);
|
||||
}
|
||||
}],
|
||||
["registers one opt-in install module and capability", () => {
|
||||
const modules = readJson("manifests/install-modules.json").modules;
|
||||
const module = modules.find((candidate) => candidate.id === "ito-compute");
|
||||
@@ -106,6 +129,9 @@ function main() {
|
||||
assert.doesNotMatch(JSON.stringify(server), /npx|ito_lock|ito_run|paper|simulat/i);
|
||||
assert.match(server.description, /ito_auth, ito_find, and ito_status/);
|
||||
assert.match(server.description, /unpublished/i);
|
||||
assert.match(server.description, /ito_auth.*validat/i);
|
||||
assert.match(server.description, /macOS Keychain/i);
|
||||
assert.match(server.description, /no browser automation/i);
|
||||
}],
|
||||
];
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ const assert = require("assert");
|
||||
const fs = require("fs");
|
||||
const os = require("os");
|
||||
const path = require("path");
|
||||
const { spawnSync } = require("child_process");
|
||||
const { spawn, spawnSync } = require("child_process");
|
||||
|
||||
const REPO_ROOT = path.join(__dirname, "..", "..");
|
||||
const ECC_SCRIPT = path.join(REPO_ROOT, "scripts", "ecc.js");
|
||||
@@ -21,6 +21,7 @@ const {
|
||||
const {
|
||||
createSafeItoInvocationEnvironment,
|
||||
getInvocationCommand,
|
||||
ITO_RUNTIME_ENVIRONMENT_KEYS,
|
||||
} = require("../../scripts/lib/ito-environment");
|
||||
|
||||
function runCli(args, environment = {}) {
|
||||
@@ -35,6 +36,34 @@ function runCli(args, environment = {}) {
|
||||
});
|
||||
}
|
||||
|
||||
function runCliAndObserveFirstOutput(args, environment = {}) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const child = spawn(process.execPath, [ECC_SCRIPT, ...args], {
|
||||
cwd: REPO_ROOT,
|
||||
env: { ...process.env, NODE_ENV: "test", ...environment },
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
let firstOutputAt;
|
||||
const startedAt = Date.now();
|
||||
child.stdout.on("data", (chunk) => {
|
||||
if (firstOutputAt === undefined) firstOutputAt = Date.now();
|
||||
stdout += chunk;
|
||||
});
|
||||
child.stderr.on("data", (chunk) => { stderr += chunk; });
|
||||
child.once("error", reject);
|
||||
child.once("close", (status) => resolve({
|
||||
status,
|
||||
stdout,
|
||||
stderr,
|
||||
startedAt,
|
||||
firstOutputAt,
|
||||
closedAt: Date.now(),
|
||||
}));
|
||||
});
|
||||
}
|
||||
|
||||
function makeItoProbe(exitCode = 0) {
|
||||
const directory = fs.mkdtempSync(path.join(os.tmpdir(), "ecc-ito-cli-"));
|
||||
const log = path.join(directory, "invocation.json");
|
||||
@@ -72,9 +101,9 @@ function readInvocation(probe) {
|
||||
return JSON.parse(fs.readFileSync(probe.log, "utf8"));
|
||||
}
|
||||
|
||||
function runTest(name, fn) {
|
||||
async function runTest(name, fn) {
|
||||
try {
|
||||
fn();
|
||||
await fn();
|
||||
console.log(` ✓ ${name}`);
|
||||
return true;
|
||||
} catch (error) {
|
||||
@@ -84,12 +113,12 @@ function runTest(name, fn) {
|
||||
}
|
||||
}
|
||||
|
||||
function main() {
|
||||
async function main() {
|
||||
console.log("\n=== Testing ECC × Itô real CLI bridge ===\n");
|
||||
|
||||
const tests = [
|
||||
["forwards only the reviewed RFQ CLI surface to an explicit local executable", () => {
|
||||
for (const command of ["auth", "find", "status"]) {
|
||||
for (const command of ["login", "auth", "find", "status"]) {
|
||||
const probe = makeItoProbe();
|
||||
try {
|
||||
const result = runCli(["ito", command], {
|
||||
@@ -103,6 +132,31 @@ function main() {
|
||||
}
|
||||
}
|
||||
}],
|
||||
["forwards the canonical login browser opt-out without performing browser automation", () => {
|
||||
const probe = makeItoProbe();
|
||||
try {
|
||||
const result = runCli(["ito", "login", "--no-browser"], {
|
||||
ECC_ITO_CLI_EXECUTABLE: probe.executable,
|
||||
});
|
||||
assert.strictEqual(result.status, 0, result.stderr);
|
||||
assert.deepStrictEqual(readInvocation(probe).argv, ["login", "--no-browser"]);
|
||||
} finally {
|
||||
fs.rmSync(probe.directory, { recursive: true, force: true });
|
||||
}
|
||||
}],
|
||||
["rejects --no-browser on validation-only auth before spawning", () => {
|
||||
const probe = makeItoProbe();
|
||||
try {
|
||||
const result = runCli(["ito", "auth", "--no-browser"], {
|
||||
ECC_ITO_CLI_EXECUTABLE: probe.executable,
|
||||
});
|
||||
assert.notStrictEqual(result.status, 0);
|
||||
assert.match(result.stderr, /--no-browser.*only.*login/i);
|
||||
assert.ok(!fs.existsSync(probe.log));
|
||||
} finally {
|
||||
fs.rmSync(probe.directory, { recursive: true, force: true });
|
||||
}
|
||||
}],
|
||||
["normalizes JSON and forwards every RFQ constraint without interpretation", () => {
|
||||
const probe = makeItoProbe();
|
||||
try {
|
||||
@@ -135,12 +189,15 @@ function main() {
|
||||
fs.rmSync(probe.directory, { recursive: true, force: true });
|
||||
}
|
||||
}],
|
||||
["passes only the required Itô runtime settings across the process boundary", () => {
|
||||
["login never inherits ITO_API_KEY but preserves secure token settings", () => {
|
||||
const probe = makeItoProbe();
|
||||
try {
|
||||
const result = runCli(["ito", "auth"], {
|
||||
const result = runCli(["ito", "login"], {
|
||||
ECC_ITO_CLI_EXECUTABLE: probe.executable,
|
||||
ITO_API_KEY: "ito_test_key",
|
||||
ITO_API_KEY: "must-not-cross-without-legacy-mode",
|
||||
ITO_AUTH_MODE: "device",
|
||||
ITO_ALLOW_FILE_TOKEN: "1",
|
||||
ITO_TOKEN_FILE: "/tmp/ito-device-token",
|
||||
ITO_API_URL: "https://compute.example.test",
|
||||
ITO_INVENTORY_URL: "https://edge.example.test",
|
||||
AWS_SECRET_ACCESS_KEY: "must-not-cross",
|
||||
@@ -149,7 +206,10 @@ function main() {
|
||||
});
|
||||
assert.strictEqual(result.status, 0, result.stderr);
|
||||
const childEnvironment = readInvocation(probe).env;
|
||||
assert.strictEqual(childEnvironment.ITO_API_KEY, "ito_test_key");
|
||||
assert.strictEqual(childEnvironment.ITO_API_KEY, undefined);
|
||||
assert.strictEqual(childEnvironment.ITO_AUTH_MODE, "device");
|
||||
assert.strictEqual(childEnvironment.ITO_ALLOW_FILE_TOKEN, "1");
|
||||
assert.strictEqual(childEnvironment.ITO_TOKEN_FILE, "/tmp/ito-device-token");
|
||||
assert.strictEqual(childEnvironment.ITO_API_URL, "https://compute.example.test");
|
||||
assert.strictEqual(childEnvironment.ITO_INVENTORY_URL, "https://edge.example.test");
|
||||
assert.strictEqual(childEnvironment.AWS_SECRET_ACCESS_KEY, undefined);
|
||||
@@ -160,6 +220,46 @@ function main() {
|
||||
fs.rmSync(probe.directory, { recursive: true, force: true });
|
||||
}
|
||||
}],
|
||||
["forwards ITO_API_KEY directly to auth, find, and status without legacy mode", () => {
|
||||
for (const command of ["auth", "find", "status"]) {
|
||||
const probe = makeItoProbe();
|
||||
try {
|
||||
const result = runCli(["ito", command], {
|
||||
ECC_ITO_CLI_EXECUTABLE: probe.executable,
|
||||
ITO_API_KEY: "ito_test_key",
|
||||
});
|
||||
assert.strictEqual(result.status, 0, result.stderr);
|
||||
assert.strictEqual(readInvocation(probe).env.ITO_API_KEY, "ito_test_key");
|
||||
} finally {
|
||||
fs.rmSync(probe.directory, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
}],
|
||||
["streams device login output before completion and propagates its exit status", async () => {
|
||||
const probe = makeItoProbe(7);
|
||||
try {
|
||||
fs.writeFileSync(
|
||||
probe.executable,
|
||||
[
|
||||
'"use strict";',
|
||||
'process.stdout.write("device-code-now\\n");',
|
||||
'setTimeout(() => process.exit(7), 500);',
|
||||
"",
|
||||
].join("\n")
|
||||
);
|
||||
const result = await runCliAndObserveFirstOutput(["ito", "login"], {
|
||||
ECC_ITO_CLI_EXECUTABLE: probe.executable,
|
||||
});
|
||||
assert.strictEqual(result.status, 7, result.stderr);
|
||||
assert.match(result.stdout, /device-code-now/);
|
||||
assert.ok(
|
||||
result.closedAt - result.firstOutputAt >= 350,
|
||||
"login output was buffered until process completion",
|
||||
);
|
||||
} finally {
|
||||
fs.rmSync(probe.directory, { recursive: true, force: true });
|
||||
}
|
||||
}],
|
||||
["isolates live node qualification from Itô and unrelated credentials", () => {
|
||||
const probe = makeItoProbe();
|
||||
try {
|
||||
@@ -176,6 +276,9 @@ function main() {
|
||||
], {
|
||||
ECC_ITO_CLI_EXECUTABLE: probe.executable,
|
||||
ITO_API_KEY: "must-not-cross-into-node-qualification",
|
||||
ITO_AUTH_MODE: "legacy",
|
||||
ITO_ALLOW_FILE_TOKEN: "1",
|
||||
ITO_TOKEN_FILE: "/tmp/must-not-cross-token-file",
|
||||
ITO_API_URL: "https://compute.example.test",
|
||||
ITO_INVENTORY_URL: "https://edge.example.test",
|
||||
ITO_ENABLE_SIXTYTWO_LIVE: "1",
|
||||
@@ -201,6 +304,9 @@ function main() {
|
||||
assert.strictEqual(invocation.env.SIXTYTWO_TOKEN, "sixtytwo-legacy-test-token");
|
||||
assert.strictEqual(invocation.env.SSH_AUTH_SOCK, "/tmp/ecc-test-agent.sock");
|
||||
assert.strictEqual(invocation.env.ITO_API_KEY, undefined);
|
||||
assert.strictEqual(invocation.env.ITO_AUTH_MODE, undefined);
|
||||
assert.strictEqual(invocation.env.ITO_ALLOW_FILE_TOKEN, undefined);
|
||||
assert.strictEqual(invocation.env.ITO_TOKEN_FILE, undefined);
|
||||
assert.strictEqual(invocation.env.ITO_API_URL, undefined);
|
||||
assert.strictEqual(invocation.env.ITO_INVENTORY_URL, undefined);
|
||||
assert.strictEqual(invocation.env.ITO_CLI_DEMO, undefined);
|
||||
@@ -310,6 +416,14 @@ function main() {
|
||||
}
|
||||
}],
|
||||
["classifies Itō child environments once and fails closed on unknown prefixes", () => {
|
||||
assert.deepStrictEqual(ITO_RUNTIME_ENVIRONMENT_KEYS, [
|
||||
"ITO_API_KEY",
|
||||
"ITO_API_URL",
|
||||
"ITO_INVENTORY_URL",
|
||||
"ITO_AUTH_MODE",
|
||||
"ITO_ALLOW_FILE_TOKEN",
|
||||
"ITO_TOKEN_FILE",
|
||||
]);
|
||||
const safe = createSafeItoInvocationEnvironment(
|
||||
{
|
||||
PATH: process.env.PATH,
|
||||
@@ -338,14 +452,14 @@ function main() {
|
||||
);
|
||||
}],
|
||||
["rejects unsupported browser, paper, and execution operations before spawning", () => {
|
||||
for (const command of ["rent", "lock", "run", "inference", "mcp"]) {
|
||||
for (const command of ["rent", "lock", "purchase", "run", "inference", "mcp"]) {
|
||||
const probe = makeItoProbe();
|
||||
try {
|
||||
const result = runCli(["ito", command], {
|
||||
ECC_ITO_CLI_EXECUTABLE: probe.executable,
|
||||
});
|
||||
assert.notStrictEqual(result.status, 0, command);
|
||||
assert.match(result.stderr, /only auth, find, status, and evals/i);
|
||||
assert.match(result.stderr, /only login, auth, find, status, and evals/i);
|
||||
assert.ok(!fs.existsSync(probe.log), `${command} must not spawn the Itô CLI`);
|
||||
} finally {
|
||||
fs.rmSync(probe.directory, { recursive: true, force: true });
|
||||
@@ -510,13 +624,14 @@ function main() {
|
||||
fs.rmSync(probe.directory, { recursive: true, force: true });
|
||||
}
|
||||
}],
|
||||
["help exposes the truthful CLI and MCP surface without a browser path", () => {
|
||||
["help separates device login from auth validation", () => {
|
||||
const probe = makeItoProbe();
|
||||
try {
|
||||
const result = runCli(["ito", "--help"], {
|
||||
ECC_ITO_CLI_EXECUTABLE: probe.executable,
|
||||
});
|
||||
assert.strictEqual(result.status, 0, result.stderr);
|
||||
assert.match(result.stdout, /ecc ito login \[--no-browser\]/);
|
||||
assert.match(result.stdout, /ecc ito auth/);
|
||||
assert.match(result.stdout, /ecc ito find/);
|
||||
assert.match(result.stdout, /ecc ito status/);
|
||||
@@ -528,9 +643,15 @@ function main() {
|
||||
assert.match(result.stdout, new RegExp(CANONICAL_PACKAGE.replaceAll("/", "\\/")));
|
||||
assert.match(result.stdout, /unpublished/i);
|
||||
assert.match(result.stdout, /never discovers[^\n]*through PATH/i);
|
||||
assert.match(result.stdout, /device authorization/i);
|
||||
assert.match(result.stdout, /opens the Itô verification page by default/i);
|
||||
assert.match(result.stdout, /macOS Keychain/i);
|
||||
assert.match(result.stdout, /ECC itself performs no browser automation/i);
|
||||
assert.match(result.stdout, /auth.*validat/i);
|
||||
assert.match(result.stdout, /ITO_AUTH_MODE=legacy is not\s+required/i);
|
||||
assert.doesNotMatch(
|
||||
result.stdout,
|
||||
/manual copy|open(?:s)? (?:a )?browser|ito_lock|ito_run|npm link|paper|simulat/i
|
||||
/manual copy|ito_lock|ito_run|npm link|paper|simulat/i
|
||||
);
|
||||
assert.ok(!fs.existsSync(probe.log));
|
||||
} finally {
|
||||
@@ -542,7 +663,7 @@ function main() {
|
||||
let passed = 0;
|
||||
let failed = 0;
|
||||
for (const [name, fn] of tests) {
|
||||
if (runTest(name, fn)) passed += 1;
|
||||
if (await runTest(name, fn)) passed += 1;
|
||||
else failed += 1;
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user