mirror of
https://github.com/affaan-m/ECC.git
synced 2026-08-24 00:12:25 +02:00
fix(continuous-learning): /evolve never produces skill or agent candidates (#2664)
* fix(continuous-learning): cluster instincts by keyword overlap in /evolve
`cmd_evolve` grouped instincts by exact string equality of the whole
normalized trigger sentence. Triggers are free-form sentences, so every
instinct landed in its own bucket and `skill_candidates` was always empty.
`agent_candidates` is derived from `skill_candidates`, so agents never
generated either — `/evolve --generate` could only ever emit commands.
Measured on a 42-instinct project: 42 instincts produced 42 unique cluster
keys, largest cluster size 1.
Group on keyword overlap instead. Jaccard is the wrong metric here — trigger
keyword sets average ~7 words, so even clearly related pairs top out around
0.33 — so this uses the overlap coefficient (shared / smaller set) at 0.5,
plus a floor of 2 shared keywords so one incidental word cannot pull
unrelated instincts together. The same 42 instincts now yield 4 clusters.
Also unify the command/agent slug used by the preview and the writer. The
preview called `.replace('a ', '')`, which strips "a " anywhere in the
string, mangling "extracting data from Reddit" into
`/extracting-datfrom-R` while `--generate` wrote `extracting-data-from.md`.
Both paths now share `_evolved_command_name()` / `_evolved_agent_name()`.
Adds tests/scripts/instinct-cli-evolve.test.js, which fails on the previous
implementation (0 clusters instead of 1; preview name `extracting-datfrom-R`)
and covers the negative cases so unrelated triggers still stay apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(continuous-learning): correct clustering metric name in docstring
The docstring said "Jaccard" while the implementation uses the overlap
coefficient, which is the point of the change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(continuous-learning): generate every evolve candidate and cut slugs on word boundaries
_generate_evolved() wrote only skill_candidates[:5], workflow_instincts[:5]
and agent_candidates[:3]. On a project with 36 command candidates that meant
5 files and no warning, so the output read as complete while 86% of the
candidates were dropped.
Generation is now unbounded by default and takes a --limit N flag for callers
that want a cap. A cap that truncates says so:
Note: writing 3 of 36 command candidates (--limit 3); 33 skipped.
The analysis preview keeps showing five per kind but now names the remainder
("... and 31 more command candidates not shown") instead of presenting a
sample as the whole set.
Slugs were also cut with a hard slice, which split words mid-token and
produced /investigating-comple, /learning-about-compl and
/researching-mechanis. _truncate_slug() retreats to the last separator that
fits, and keeps the full head when the cut already lands on one, so
"analyzing large text files" stays /analyzing-large-text rather than losing
a word. A first word longer than the limit still falls back to a hard cut
because no boundary is available.
Shorter slugs collide more easily, and a collision used to mean one file
silently overwriting another. _assign_unique_slugs() suffixes duplicates
(-2, -3) and is called by both the preview and the writer over the same
ordered list, so advertised names and written names cannot drift apart.
Skill directory naming moved to _evolved_skill_name(); it previously used its
own inline slug expression, so it was the one truncation the shared helper
did not cover.
Adds tests/scripts/instinct-cli-evolve-generate.test.js: 7 cases covering
word-boundary cuts, the separator-aligned cut, unbounded generation, --limit
reporting, collision dedup, preview remainder and preview/writer agreement.
Six of the seven fail against the previous implementation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: haelyra <49814733+haelyra@users.noreply.github.com>
This commit is contained in:
co-authored by
Claude Opus 5
haelyra
parent
b2bc8dcd14
commit
8a97868b5b
@@ -1145,6 +1145,163 @@ def cmd_export(args) -> int:
|
||||
# Evolve Command
|
||||
# ─────────────────────────────────────────────
|
||||
|
||||
# Words carrying no topical signal in a trigger sentence.
|
||||
TRIGGER_STOP_WORDS = {
|
||||
'when', 'while', 'the', 'and', 'or', 'to', 'of', 'in', 'on', 'for', 'with',
|
||||
'that', 'this', 'from', 'into', 'at', 'by', 'as', 'is', 'are', 'be', 'it',
|
||||
'its', 'they', 'them', 'their', 'you', 'your', 'new', 'any', 'all', 'about',
|
||||
'after', 'before', 'over', 'via', 'use', 'using', 'need', 'needs', 'not',
|
||||
}
|
||||
|
||||
# Overlap coefficient (shared / smaller set) two triggers need to cluster.
|
||||
# Jaccard is the wrong metric here: trigger keyword sets average ~7 words, so
|
||||
# even clearly-related pairs top out near 0.33 and nothing ever groups.
|
||||
TRIGGER_SIMILARITY_THRESHOLD = 0.5
|
||||
|
||||
# Guard against one incidental shared word pulling unrelated instincts together.
|
||||
TRIGGER_MIN_SHARED_KEYWORDS = 2
|
||||
|
||||
|
||||
# Evolved artefact slugs are trimmed to keep file names short. The cut has to
|
||||
# land on a word boundary: a hard slice produced names like
|
||||
# "investigating-comple" and "learning-about-compl", which read as typos.
|
||||
EVOLVED_SKILL_SLUG_LENGTH = 30
|
||||
EVOLVED_COMMAND_SLUG_LENGTH = 20
|
||||
EVOLVED_AGENT_SLUG_LENGTH = 20
|
||||
|
||||
|
||||
def _truncate_slug(slug: str, max_length: int) -> str:
|
||||
"""Trim a slug to max_length without splitting a word.
|
||||
|
||||
Falls back to a hard cut only when the first word is already longer than
|
||||
the limit, because then there is no boundary left to retreat to.
|
||||
"""
|
||||
if len(slug) <= max_length:
|
||||
return slug
|
||||
head = slug[:max_length]
|
||||
# The cut can already land on a separator, in which case head is a whole
|
||||
# sequence of words and dropping one more would lose a word for nothing.
|
||||
if slug[max_length] == '-':
|
||||
return head.rstrip('-')
|
||||
boundary = head.rfind('-')
|
||||
if boundary > 0:
|
||||
return head[:boundary]
|
||||
return head.strip('-')
|
||||
|
||||
|
||||
def _evolved_skill_name(trigger: str) -> str:
|
||||
"""Slug used for a generated skill directory. Shared by preview and writer."""
|
||||
return _truncate_slug(
|
||||
re.sub(r'[^a-z0-9]+', '-', str(trigger or '').lower()).strip('-'),
|
||||
EVOLVED_SKILL_SLUG_LENGTH,
|
||||
)
|
||||
|
||||
|
||||
def _evolved_command_name(trigger: str) -> str:
|
||||
"""Slug used for a generated command file. Shared by preview and writer."""
|
||||
stripped = str(trigger or 'unknown').lower().replace('when ', '').replace('implementing ', '')
|
||||
return _truncate_slug(
|
||||
re.sub(r'[^a-z0-9]+', '-', stripped).strip('-'),
|
||||
EVOLVED_COMMAND_SLUG_LENGTH,
|
||||
)
|
||||
|
||||
|
||||
def _evolved_agent_name(trigger: str) -> str:
|
||||
"""Slug used for a generated agent file. Shared by preview and writer."""
|
||||
return _truncate_slug(
|
||||
re.sub(r'[^a-z0-9]+', '-', str(trigger or '').lower()).strip('-'),
|
||||
EVOLVED_AGENT_SLUG_LENGTH,
|
||||
)
|
||||
|
||||
|
||||
# How many candidates of each kind the analysis prints before summarising the
|
||||
# rest. The preview is a sample, never the whole set, so it always says so.
|
||||
PREVIEW_LIMIT = 5
|
||||
|
||||
|
||||
def _print_preview_remainder(total: int, shown: int, noun: str) -> None:
|
||||
"""State how many candidates the preview left out.
|
||||
|
||||
Without this the truncated list reads as the complete set.
|
||||
"""
|
||||
if total > shown:
|
||||
print(f" ... and {total - shown} more {noun} not shown\n")
|
||||
|
||||
|
||||
def _assign_unique_slugs(items: list, slug_fn) -> list:
|
||||
"""Pair every item with a collision-free slug, preserving input order.
|
||||
|
||||
Word-boundary trimming makes collisions more likely because two triggers
|
||||
can now share a whole prefix, and a collision previously meant one
|
||||
generated file silently overwriting another. Preview and writer both call
|
||||
this over the same ordered list, so the names shown and the names written
|
||||
stay identical.
|
||||
"""
|
||||
used = set()
|
||||
assigned = []
|
||||
for item in items:
|
||||
base = slug_fn(item)
|
||||
if not base:
|
||||
assigned.append((item, ''))
|
||||
continue
|
||||
name = base
|
||||
suffix = 2
|
||||
while name in used:
|
||||
name = f"{base}-{suffix}"
|
||||
suffix += 1
|
||||
used.add(name)
|
||||
assigned.append((item, name))
|
||||
return assigned
|
||||
|
||||
|
||||
def _trigger_keywords(trigger: str) -> set:
|
||||
"""Reduce a trigger sentence to the words that carry its topic."""
|
||||
words = re.findall(r'[a-z0-9]+', str(trigger or '').lower())
|
||||
return {w for w in words if len(w) > 2 and w not in TRIGGER_STOP_WORDS}
|
||||
|
||||
|
||||
def _cluster_by_keyword_overlap(instincts: list) -> dict:
|
||||
"""Group instincts whose triggers share enough keywords.
|
||||
|
||||
Triggers are free-form sentences, so grouping on the whole normalized
|
||||
string puts every instinct in its own bucket and no skill or agent
|
||||
candidate is ever produced. Greedy clustering on keyword overlap groups
|
||||
the near-duplicate instincts that accumulate in a project.
|
||||
"""
|
||||
clusters = [] # [(shared_keywords, [instincts])]
|
||||
|
||||
for inst in instincts:
|
||||
keywords = _trigger_keywords(inst.get('trigger', ''))
|
||||
if not keywords:
|
||||
continue
|
||||
|
||||
best_index, best_score, best_shared = -1, 0.0, 0
|
||||
for index, (cluster_keywords, _members) in enumerate(clusters):
|
||||
shared = len(keywords & cluster_keywords)
|
||||
smaller = min(len(keywords), len(cluster_keywords))
|
||||
score = shared / smaller if smaller else 0.0
|
||||
if score > best_score:
|
||||
best_index, best_score, best_shared = index, score, shared
|
||||
|
||||
if (best_index >= 0
|
||||
and best_score >= TRIGGER_SIMILARITY_THRESHOLD
|
||||
and best_shared >= TRIGGER_MIN_SHARED_KEYWORDS):
|
||||
cluster_keywords, members = clusters[best_index]
|
||||
members.append(inst)
|
||||
# Keep the shared core so a cluster stays on one topic.
|
||||
clusters[best_index] = (cluster_keywords & keywords, members)
|
||||
else:
|
||||
clusters.append((keywords, [inst]))
|
||||
|
||||
grouped = {}
|
||||
for cluster_keywords, members in clusters:
|
||||
label = ' '.join(sorted(cluster_keywords)[:4]) or 'general'
|
||||
while label in grouped:
|
||||
label += ' +'
|
||||
grouped[label] = members
|
||||
return grouped
|
||||
|
||||
|
||||
def cmd_evolve(args) -> int:
|
||||
"""Analyze instincts and suggest evolutions to skills/commands/agents."""
|
||||
project = detect_project()
|
||||
@@ -1175,14 +1332,7 @@ def cmd_evolve(args) -> int:
|
||||
print(f"High confidence instincts (>=80%): {len(high_conf)}")
|
||||
|
||||
# Find clusters (instincts with similar triggers)
|
||||
trigger_clusters = defaultdict(list)
|
||||
for inst in instincts:
|
||||
trigger = inst.get('trigger', '')
|
||||
# Normalize trigger
|
||||
trigger_key = trigger.lower()
|
||||
for keyword in ['when', 'creating', 'writing', 'adding', 'implementing', 'testing']:
|
||||
trigger_key = trigger_key.replace(keyword, '').strip()
|
||||
trigger_clusters[trigger_key].append(inst)
|
||||
trigger_clusters = _cluster_by_keyword_overlap(instincts)
|
||||
|
||||
# Find clusters with 2+ instincts (good skill candidates)
|
||||
skill_candidates = []
|
||||
@@ -1203,8 +1353,8 @@ def cmd_evolve(args) -> int:
|
||||
print(f"\nPotential skill clusters found: {len(skill_candidates)}")
|
||||
|
||||
if skill_candidates:
|
||||
print(f"\n## SKILL CANDIDATES\n")
|
||||
for i, cand in enumerate(skill_candidates[:5], 1):
|
||||
print(f"\n## SKILL CANDIDATES ({len(skill_candidates)})\n")
|
||||
for i, cand in enumerate(skill_candidates[:PREVIEW_LIMIT], 1):
|
||||
scope_info = ', '.join(cand['scopes'])
|
||||
print(f"{i}. Cluster: \"{cand['trigger']}\"")
|
||||
print(f" Instincts: {len(cand['instincts'])}")
|
||||
@@ -1215,37 +1365,51 @@ def cmd_evolve(args) -> int:
|
||||
for inst in cand['instincts'][:3]:
|
||||
print(f" - {inst.get('id')} [{inst.get('scope', '?')}]")
|
||||
print()
|
||||
_print_preview_remainder(len(skill_candidates), PREVIEW_LIMIT, 'skill clusters')
|
||||
|
||||
# Command candidates (workflow instincts with high confidence)
|
||||
workflow_instincts = [i for i in instincts if i.get('domain') == 'workflow' and i.get('confidence', 0) >= 0.7]
|
||||
if workflow_instincts:
|
||||
print(f"\n## COMMAND CANDIDATES ({len(workflow_instincts)})\n")
|
||||
for inst in workflow_instincts[:5]:
|
||||
trigger = inst.get('trigger', 'unknown')
|
||||
cmd_name = trigger.replace('when ', '').replace('implementing ', '').replace('a ', '')
|
||||
cmd_name = cmd_name.replace(' ', '-')[:20]
|
||||
# Slugs come from the same helper the writer uses, over the same ordered
|
||||
# list, or the preview advertises names that differ from the files
|
||||
# --generate actually writes.
|
||||
for inst, cmd_name in _assign_unique_slugs(
|
||||
workflow_instincts,
|
||||
lambda i: _evolved_command_name(i.get('trigger', 'unknown')),
|
||||
)[:PREVIEW_LIMIT]:
|
||||
print(f" /{cmd_name}")
|
||||
print(f" From: {inst.get('id')} [{inst.get('scope', '?')}]")
|
||||
print(f" Confidence: {inst.get('confidence', 0.5):.0%}")
|
||||
print()
|
||||
_print_preview_remainder(len(workflow_instincts), PREVIEW_LIMIT, 'command candidates')
|
||||
|
||||
# Agent candidates (complex multi-step patterns)
|
||||
agent_candidates = [c for c in skill_candidates if len(c['instincts']) >= 3 and c['avg_confidence'] >= 0.75]
|
||||
if agent_candidates:
|
||||
print(f"\n## AGENT CANDIDATES ({len(agent_candidates)})\n")
|
||||
for cand in agent_candidates[:3]:
|
||||
agent_name = cand['trigger'].replace(' ', '-')[:20] + '-agent'
|
||||
for cand, agent_name in _assign_unique_slugs(
|
||||
agent_candidates,
|
||||
lambda c: _evolved_agent_name(str(c.get('trigger', '')).strip()),
|
||||
)[:PREVIEW_LIMIT]:
|
||||
print(f" {agent_name}")
|
||||
print(f" Covers {len(cand['instincts'])} instincts")
|
||||
print(f" Avg confidence: {cand['avg_confidence']:.0%}")
|
||||
print()
|
||||
_print_preview_remainder(len(agent_candidates), PREVIEW_LIMIT, 'agent candidates')
|
||||
|
||||
# Promotion candidates (project instincts that could be global)
|
||||
_show_promotion_candidates(project)
|
||||
|
||||
if args.generate:
|
||||
evolved_dir = project["evolved_dir"] if project["id"] != "global" else GLOBAL_EVOLVED_DIR
|
||||
generated = _generate_evolved(skill_candidates, workflow_instincts, agent_candidates, evolved_dir)
|
||||
generated = _generate_evolved(
|
||||
skill_candidates,
|
||||
workflow_instincts,
|
||||
agent_candidates,
|
||||
evolved_dir,
|
||||
limit=max(0, getattr(args, 'limit', 0) or 0),
|
||||
)
|
||||
if generated:
|
||||
print(f"\nGenerated {len(generated)} evolved structures:")
|
||||
for path in generated:
|
||||
@@ -1770,17 +1934,33 @@ def _cmd_projects_merge(args) -> int:
|
||||
# Generate Evolved Structures
|
||||
# ─────────────────────────────────────────────
|
||||
|
||||
def _generate_evolved(skill_candidates: list, workflow_instincts: list, agent_candidates: list, evolved_dir: Path) -> list[str]:
|
||||
"""Generate skill/command/agent files from analyzed instinct clusters."""
|
||||
def _generate_evolved(skill_candidates: list, workflow_instincts: list, agent_candidates: list, evolved_dir: Path, limit: int = 0) -> list[str]:
|
||||
"""Generate skill/command/agent files from analyzed instinct clusters.
|
||||
|
||||
``limit`` caps how many candidates of each kind are written; 0 writes them
|
||||
all. Anything a cap leaves out is reported, because the previous fixed
|
||||
caps (5 skills, 5 commands, 3 agents) discarded most candidates without
|
||||
saying a word — 35 command candidates produced 5 files and no warning.
|
||||
"""
|
||||
generated = []
|
||||
|
||||
# Generate skills from top candidates
|
||||
for cand in skill_candidates[:5]:
|
||||
def bounded(assigned: list, kind: str) -> list:
|
||||
if limit and len(assigned) > limit:
|
||||
print(f"\nNote: writing {limit} of {len(assigned)} {kind} candidates "
|
||||
f"(--limit {limit}); {len(assigned) - limit} skipped.")
|
||||
return assigned[:limit]
|
||||
return assigned
|
||||
|
||||
# Generate skills from candidate clusters
|
||||
for cand, name in bounded(
|
||||
_assign_unique_slugs(
|
||||
skill_candidates,
|
||||
lambda c: _evolved_skill_name(str(c.get('trigger', '')).strip()),
|
||||
),
|
||||
'skill',
|
||||
):
|
||||
trigger = cand['trigger'].strip()
|
||||
if not trigger:
|
||||
continue
|
||||
name = re.sub(r'[^a-z0-9]+', '-', trigger.lower()).strip('-')[:30]
|
||||
if not name:
|
||||
if not trigger or not name:
|
||||
continue
|
||||
|
||||
skill_dir = evolved_dir / "skills" / name
|
||||
@@ -1802,10 +1982,13 @@ def _generate_evolved(skill_candidates: list, workflow_instincts: list, agent_ca
|
||||
generated.append(str(skill_dir / "SKILL.md"))
|
||||
|
||||
# Generate commands from workflow instincts
|
||||
for inst in workflow_instincts[:5]:
|
||||
trigger = inst.get('trigger', 'unknown')
|
||||
cmd_name = re.sub(r'[^a-z0-9]+', '-', trigger.lower().replace('when ', '').replace('implementing ', ''))
|
||||
cmd_name = cmd_name.strip('-')[:20]
|
||||
for inst, cmd_name in bounded(
|
||||
_assign_unique_slugs(
|
||||
workflow_instincts,
|
||||
lambda i: _evolved_command_name(i.get('trigger', 'unknown')),
|
||||
),
|
||||
'command',
|
||||
):
|
||||
if not cmd_name:
|
||||
continue
|
||||
|
||||
@@ -1819,9 +2002,13 @@ def _generate_evolved(skill_candidates: list, workflow_instincts: list, agent_ca
|
||||
generated.append(str(cmd_file))
|
||||
|
||||
# Generate agents from complex clusters
|
||||
for cand in agent_candidates[:3]:
|
||||
trigger = cand['trigger'].strip()
|
||||
agent_name = re.sub(r'[^a-z0-9]+', '-', trigger.lower()).strip('-')[:20]
|
||||
for cand, agent_name in bounded(
|
||||
_assign_unique_slugs(
|
||||
agent_candidates,
|
||||
lambda c: _evolved_agent_name(str(c.get('trigger', '')).strip()),
|
||||
),
|
||||
'agent',
|
||||
):
|
||||
if not agent_name:
|
||||
continue
|
||||
|
||||
@@ -2018,6 +2205,8 @@ def main() -> int:
|
||||
# Evolve
|
||||
evolve_parser = subparsers.add_parser('evolve', help='Analyze and evolve instincts')
|
||||
evolve_parser.add_argument('--generate', action='store_true', help='Generate evolved structures')
|
||||
evolve_parser.add_argument('--limit', type=int, default=0, metavar='N',
|
||||
help='Max candidates of each kind to generate (default: 0 = all)')
|
||||
|
||||
# Promote (new in v2.1)
|
||||
promote_parser = subparsers.add_parser('promote', help='Promote project instincts to global scope')
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
const assert = require('assert');
|
||||
const fs = require('fs');
|
||||
const os = require('os');
|
||||
const path = require('path');
|
||||
const { spawnSync } = require('child_process');
|
||||
|
||||
let passed = 0;
|
||||
let failed = 0;
|
||||
|
||||
const repoRoot = path.resolve(__dirname, '..', '..');
|
||||
const cliPath = path.join(
|
||||
repoRoot,
|
||||
'skills',
|
||||
'continuous-learning-v2',
|
||||
'scripts',
|
||||
'instinct-cli.py'
|
||||
);
|
||||
|
||||
function detectPython3() {
|
||||
for (const bin of ['python3', 'python']) {
|
||||
const r = spawnSync(bin, ['--version'], { encoding: 'utf8' });
|
||||
if (r.status === 0 && /Python 3/.test(r.stdout + r.stderr)) return bin;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
const PYTHON3 = detectPython3();
|
||||
if (!PYTHON3) {
|
||||
console.log('\n=== Testing instinct-cli.py evolve generation ===\n');
|
||||
console.log(' - skipped: Python 3 not found in PATH');
|
||||
console.log('\nPassed: 0');
|
||||
console.log('Failed: 0');
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
function test(name, fn) {
|
||||
try {
|
||||
fn();
|
||||
console.log(` ✓ ${name}`);
|
||||
passed += 1;
|
||||
} catch (error) {
|
||||
console.log(` ✗ ${name}`);
|
||||
console.log(` Error: ${error.message}`);
|
||||
failed += 1;
|
||||
}
|
||||
}
|
||||
|
||||
function createTempDir() {
|
||||
return fs.mkdtempSync(path.join(os.tmpdir(), 'ecc-instinct-cli-evolve-'));
|
||||
}
|
||||
|
||||
function cleanupDir(dir) {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
function writeInstinct(root, id, trigger, confidence = 0.8, domain = 'workflow') {
|
||||
const dir = path.join(root, 'instincts', 'personal');
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(dir, `${id}.yaml`),
|
||||
[
|
||||
'---',
|
||||
`id: ${id}`,
|
||||
`trigger: "${trigger}"`,
|
||||
`confidence: ${confidence}`,
|
||||
`domain: ${domain}`,
|
||||
'---',
|
||||
'',
|
||||
`## Action`,
|
||||
'',
|
||||
`Action for ${id}.`,
|
||||
'',
|
||||
].join('\n')
|
||||
);
|
||||
}
|
||||
|
||||
// CLV2_NO_PROJECT pins the run to global scope, so seeded instincts live in
|
||||
// <root>/instincts/personal and generated files land in <root>/evolved.
|
||||
function runCli(root, args) {
|
||||
return spawnSync(PYTHON3, [cliPath, ...args], {
|
||||
cwd: repoRoot,
|
||||
encoding: 'utf8',
|
||||
env: {
|
||||
...process.env,
|
||||
CLV2_HOMUNCULUS_DIR: root,
|
||||
CLV2_NO_PROJECT: '1',
|
||||
HOME: path.join(root, 'home'),
|
||||
USERPROFILE: path.join(root, 'home'),
|
||||
CLAUDE_PROJECT_DIR: '',
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function generatedCommands(root) {
|
||||
const dir = path.join(root, 'evolved', 'commands');
|
||||
if (!fs.existsSync(dir)) return [];
|
||||
return fs.readdirSync(dir).sort();
|
||||
}
|
||||
|
||||
// Eight unrelated workflow triggers: no two share enough keywords to cluster,
|
||||
// so each one is its own command candidate.
|
||||
const EIGHT_TRIGGERS = [
|
||||
['run-tests', 'when running tests'],
|
||||
['build-images', 'when building images'],
|
||||
['deploy-services', 'when deploying services'],
|
||||
['profile-memory', 'when profiling memory'],
|
||||
['rotate-secrets', 'when rotating secrets'],
|
||||
['tag-releases', 'when tagging releases'],
|
||||
['prune-caches', 'when pruning caches'],
|
||||
['review-requests', 'when reviewing pull requests'],
|
||||
];
|
||||
|
||||
function seedEight(root) {
|
||||
for (const [id, trigger] of EIGHT_TRIGGERS) {
|
||||
writeInstinct(root, id, trigger);
|
||||
}
|
||||
}
|
||||
|
||||
console.log('\n=== Testing instinct-cli.py evolve generation ===\n');
|
||||
|
||||
test('generated command names are cut on a word boundary', () => {
|
||||
const root = createTempDir();
|
||||
try {
|
||||
writeInstinct(root, 'archaeology', 'when investigating complex systems');
|
||||
writeInstinct(root, 'codebases', 'when learning about complex codebases');
|
||||
writeInstinct(root, 'large-text', 'when analyzing large text files');
|
||||
|
||||
const result = runCli(root, ['evolve', '--generate']);
|
||||
assert.strictEqual(result.status, 0, result.stderr);
|
||||
|
||||
const names = generatedCommands(root);
|
||||
// A hard slice used to yield investigating-comple.md and
|
||||
// learning-about-compl.md, which read as typos.
|
||||
assert.deepStrictEqual(names, [
|
||||
'analyzing-large-text.md',
|
||||
'investigating.md',
|
||||
'learning-about.md',
|
||||
]);
|
||||
} finally {
|
||||
cleanupDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
test('a cut landing on a separator keeps the whole word', () => {
|
||||
const root = createTempDir();
|
||||
try {
|
||||
writeInstinct(root, 'a', 'when analyzing large text files');
|
||||
writeInstinct(root, 'b', 'when running tests');
|
||||
writeInstinct(root, 'c', 'when building images');
|
||||
|
||||
assert.strictEqual(runCli(root, ['evolve', '--generate']).status, 0);
|
||||
// "analyzing-large-text" is exactly the slug limit and ends on a word, so
|
||||
// nothing further may be dropped.
|
||||
assert.ok(generatedCommands(root).includes('analyzing-large-text.md'));
|
||||
} finally {
|
||||
cleanupDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
test('every command candidate is generated, not just the first five', () => {
|
||||
const root = createTempDir();
|
||||
try {
|
||||
seedEight(root);
|
||||
|
||||
const result = runCli(root, ['evolve', '--generate']);
|
||||
assert.strictEqual(result.status, 0, result.stderr);
|
||||
assert.strictEqual(
|
||||
generatedCommands(root).length,
|
||||
EIGHT_TRIGGERS.length,
|
||||
'a fixed cap silently dropped candidates'
|
||||
);
|
||||
} finally {
|
||||
cleanupDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
test('--limit caps generation and reports what it skipped', () => {
|
||||
const root = createTempDir();
|
||||
try {
|
||||
seedEight(root);
|
||||
|
||||
const result = runCli(root, ['evolve', '--generate', '--limit', '3']);
|
||||
assert.strictEqual(result.status, 0, result.stderr);
|
||||
assert.strictEqual(generatedCommands(root).length, 3);
|
||||
assert.match(result.stdout, /writing 3 of 8 command candidates/);
|
||||
assert.match(result.stdout, /5 skipped/);
|
||||
} finally {
|
||||
cleanupDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
test('colliding slugs produce distinct files instead of overwriting', () => {
|
||||
const root = createTempDir();
|
||||
try {
|
||||
// Both triggers trim to "investigating".
|
||||
writeInstinct(root, 'first', 'when investigating complex systems');
|
||||
writeInstinct(root, 'second', 'when investigating extraordinarily convoluted pipelines');
|
||||
writeInstinct(root, 'third', 'when running tests');
|
||||
|
||||
assert.strictEqual(runCli(root, ['evolve', '--generate']).status, 0);
|
||||
|
||||
const names = generatedCommands(root);
|
||||
assert.ok(names.includes('investigating.md'), `missing base name in ${names}`);
|
||||
assert.ok(names.includes('investigating-2.md'), `missing deduped name in ${names}`);
|
||||
assert.strictEqual(new Set(names).size, names.length);
|
||||
} finally {
|
||||
cleanupDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
test('preview states how many candidates it left out', () => {
|
||||
const root = createTempDir();
|
||||
try {
|
||||
seedEight(root);
|
||||
|
||||
const result = runCli(root, ['evolve']);
|
||||
assert.strictEqual(result.status, 0, result.stderr);
|
||||
assert.match(result.stdout, /COMMAND CANDIDATES \(8\)/);
|
||||
assert.match(result.stdout, /and 3 more command candidates not shown/);
|
||||
} finally {
|
||||
cleanupDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
test('preview names match the files --generate writes', () => {
|
||||
const root = createTempDir();
|
||||
try {
|
||||
writeInstinct(root, 'first', 'when investigating complex systems');
|
||||
writeInstinct(root, 'second', 'when investigating extraordinarily convoluted pipelines');
|
||||
writeInstinct(root, 'third', 'when running tests');
|
||||
|
||||
const preview = runCli(root, ['evolve']);
|
||||
assert.strictEqual(preview.status, 0, preview.stderr);
|
||||
assert.match(preview.stdout, /\/investigating\b/);
|
||||
assert.match(preview.stdout, /\/investigating-2\b/);
|
||||
|
||||
assert.strictEqual(runCli(root, ['evolve', '--generate']).status, 0);
|
||||
const names = generatedCommands(root);
|
||||
assert.ok(names.includes('investigating.md'));
|
||||
assert.ok(names.includes('investigating-2.md'));
|
||||
} finally {
|
||||
cleanupDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
console.log(`\nPassed: ${passed}`);
|
||||
console.log(`Failed: ${failed}`);
|
||||
|
||||
process.exit(failed > 0 ? 1 : 0);
|
||||
@@ -0,0 +1,188 @@
|
||||
const assert = require('assert');
|
||||
const fs = require('fs');
|
||||
const os = require('os');
|
||||
const path = require('path');
|
||||
const { spawnSync } = require('child_process');
|
||||
|
||||
let passed = 0;
|
||||
let failed = 0;
|
||||
|
||||
const repoRoot = path.resolve(__dirname, '..', '..');
|
||||
const cliPath = path.join(
|
||||
repoRoot,
|
||||
'skills',
|
||||
'continuous-learning-v2',
|
||||
'scripts',
|
||||
'instinct-cli.py'
|
||||
);
|
||||
|
||||
function detectPython3() {
|
||||
for (const bin of ['python3', 'python']) {
|
||||
const r = spawnSync(bin, ['--version'], { encoding: 'utf8' });
|
||||
if (r.status === 0 && /Python 3/.test(r.stdout + r.stderr)) return bin;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
const PYTHON3 = detectPython3();
|
||||
if (!PYTHON3) {
|
||||
console.log('\n=== Testing instinct-cli.py evolve clustering ===\n');
|
||||
console.log(' - skipped: Python 3 not found in PATH');
|
||||
console.log('\nPassed: 0');
|
||||
console.log('Failed: 0');
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
function test(name, fn) {
|
||||
try {
|
||||
fn();
|
||||
console.log(` ✓ ${name}`);
|
||||
passed += 1;
|
||||
} catch (error) {
|
||||
console.log(` ✗ ${name}`);
|
||||
console.log(` Error: ${error.message}`);
|
||||
failed += 1;
|
||||
}
|
||||
}
|
||||
|
||||
function createTempDir() {
|
||||
return fs.mkdtempSync(path.join(os.tmpdir(), 'ecc-instinct-cli-evolve-'));
|
||||
}
|
||||
|
||||
function cleanupDir(dir) {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
function writeInstinct(root, id, trigger, confidence = 0.8, domain = 'workflow') {
|
||||
const filePath = path.join(root, 'instincts', 'personal', `${id}.yaml`);
|
||||
fs.mkdirSync(path.dirname(filePath), { recursive: true });
|
||||
fs.writeFileSync(
|
||||
filePath,
|
||||
[
|
||||
'---',
|
||||
`id: ${id}`,
|
||||
`trigger: ${trigger}`,
|
||||
`confidence: ${confidence}`,
|
||||
`domain: ${domain}`,
|
||||
'scope: global',
|
||||
'---',
|
||||
'',
|
||||
'## Action',
|
||||
'',
|
||||
`Action for ${id}.`,
|
||||
'',
|
||||
].join('\n')
|
||||
);
|
||||
}
|
||||
|
||||
// cwd is the temp dir (not a git repo) so project detection falls back to
|
||||
// global scope and the fixture instincts are the only ones loaded.
|
||||
function runEvolve(root, args = []) {
|
||||
return spawnSync(PYTHON3, [cliPath, 'evolve', ...args], {
|
||||
cwd: root,
|
||||
encoding: 'utf8',
|
||||
env: {
|
||||
...process.env,
|
||||
CLV2_HOMUNCULUS_DIR: root,
|
||||
HOME: path.join(root, 'home'),
|
||||
USERPROFILE: path.join(root, 'home'),
|
||||
CLAUDE_PROJECT_DIR: '',
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function clusterCount(stdout) {
|
||||
const match = stdout.match(/Potential skill clusters found:\s*(\d+)/);
|
||||
assert.ok(match, `cluster count missing from output:\n${stdout}`);
|
||||
return Number(match[1]);
|
||||
}
|
||||
|
||||
console.log('\n=== Testing instinct-cli.py evolve clustering ===\n');
|
||||
|
||||
// evolve refuses to analyze fewer than 3 instincts, so each fixture adds a
|
||||
// filler whose trigger shares no keywords with the pair under test.
|
||||
const FILLER = ['filler-unrelated', 'when rotating expired signing certificates'];
|
||||
|
||||
test('instincts with overlapping trigger keywords form one cluster', () => {
|
||||
const root = createTempDir();
|
||||
try {
|
||||
writeInstinct(root, 'l10n-merge', 'when adding a language to the localization dictionary');
|
||||
writeInstinct(root, 'l10n-verify', 'when verifying the localization dictionary for a language');
|
||||
writeInstinct(root, ...FILLER);
|
||||
|
||||
const result = runEvolve(root);
|
||||
assert.strictEqual(result.status, 0, result.stderr);
|
||||
assert.strictEqual(clusterCount(result.stdout), 1);
|
||||
} finally {
|
||||
cleanupDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
test('unrelated triggers do not cluster together', () => {
|
||||
const root = createTempDir();
|
||||
try {
|
||||
writeInstinct(root, 'docker-build', 'when building container images for deployment');
|
||||
writeInstinct(root, 'sql-index', 'when tuning slow database queries');
|
||||
writeInstinct(root, ...FILLER);
|
||||
|
||||
const result = runEvolve(root);
|
||||
assert.strictEqual(result.status, 0, result.stderr);
|
||||
assert.strictEqual(clusterCount(result.stdout), 0);
|
||||
} finally {
|
||||
cleanupDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
test('a single shared keyword is not enough to cluster', () => {
|
||||
const root = createTempDir();
|
||||
try {
|
||||
writeInstinct(root, 'bash-archives', 'when sampling compressed archives with bash pipelines');
|
||||
writeInstinct(root, 'bash-signing', 'when inspecting bash exit codes after failures');
|
||||
writeInstinct(root, ...FILLER);
|
||||
|
||||
const result = runEvolve(root);
|
||||
assert.strictEqual(result.status, 0, result.stderr);
|
||||
assert.strictEqual(clusterCount(result.stdout), 0);
|
||||
} finally {
|
||||
cleanupDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
test('preview command name matches the file --generate writes', () => {
|
||||
const root = createTempDir();
|
||||
try {
|
||||
writeInstinct(root, 'reddit-scrape', 'when extracting data from Reddit pages');
|
||||
writeInstinct(root, 'filler-one', 'when rotating expired signing certificates', 0.8, 'testing');
|
||||
writeInstinct(root, 'filler-two', 'when pruning stale feature branches', 0.8, 'testing');
|
||||
|
||||
const preview = runEvolve(root);
|
||||
assert.strictEqual(preview.status, 0, preview.stderr);
|
||||
|
||||
const match = preview.stdout.match(/^\s+\/(\S+)$/m);
|
||||
assert.ok(match, `no command candidate in output:\n${preview.stdout}`);
|
||||
const previewName = match[1];
|
||||
|
||||
// The old preview stripped every "a " occurrence, mangling "data from"
|
||||
// into "datfrom" and advertising a name --generate never wrote.
|
||||
assert.ok(
|
||||
!previewName.includes('datfrom'),
|
||||
`preview mangled the trigger: ${previewName}`
|
||||
);
|
||||
|
||||
const generated = runEvolve(root, ['--generate']);
|
||||
assert.strictEqual(generated.status, 0, generated.stderr);
|
||||
|
||||
const commandFile = path.join(root, 'evolved', 'commands', `${previewName}.md`);
|
||||
assert.ok(
|
||||
fs.existsSync(commandFile),
|
||||
`expected ${commandFile}, got: ${fs.readdirSync(path.join(root, 'evolved', 'commands')).join(', ')}`
|
||||
);
|
||||
} finally {
|
||||
cleanupDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
console.log(`\nPassed: ${passed}`);
|
||||
console.log(`Failed: ${failed}`);
|
||||
|
||||
process.exit(failed > 0 ? 1 : 0);
|
||||
Reference in New Issue
Block a user