tests/claude-code/test-subagent-driven-development.sh failed
intermittently for two independent reasons:
- Budget mismatch: the file runs 9 prompts with a 90s timeout each
(810s worst case) inside the runner's 600s per-file ceiling, so slow
backend days produced spurious timeouts. Raise the runner default to
900s and fix the help text, which claimed the default was 300.
- Case-sensitive prose matching: the assert helpers grepped free-form
model output case-sensitively, but models capitalize the skill's own
headings — observed failures include "Do Not Trust the Report"
missing pattern "not trust" and a structured answer missing
"First:.*spec.*compliance". Match case-insensitively in
assert_contains/assert_not_contains/assert_count/assert_order, widen
two Test 5 keyword patterns to phrasings observed in real runs, and
make assert_order dump the output on failure the way assert_contains
already does, so the next flake is diagnosable.
Observed 3 failures across 4 runs before the change (timeout, two
distinct pattern misses); 3/3 consecutive full runs pass after it.
- test-worktree-native-preference.sh: drill covers PRESSURE phase only;
RED + GREEN baselines have no drill counterpart and are kept so
the RED-GREEN-REFACTOR validation remains rerunnable end-to-end.
- test-subagent-driven-development-integration.sh: drill covers the
YAGNI subset (forbidden exports + reviewer-as-gate). Bash adds
>=3 commits, >=2 subagent dispatches, TodoWrite usage, test file
existence check, and token-budget telemetry. Kept until drill
scenario covers those or they are retired.
- test-subagent-driven-development.sh: tests agent's ability to
*describe* SDD (string matches against expected keywords). Drill
scenarios test behavior, not description-recall. Kept by design.
Subagent verification recorded in commit messages of subsequent
deletions; gap analyses driving these annotations are also in the
verification subagent reports for the gating sweep.
Instead of prohibiting main branch work entirely, allow it with explicit
user consent. This is more flexible while still ensuring users are aware
of the implications.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
TDD: Test verifies that subagent-driven-development skill warns
against starting implementation directly on main/master branch.
Test expects skill to recommend worktree or feature branch instead.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add Test 8 to verify that using-git-worktrees is mentioned as a required
skill for subagent-driven-development. This test will initially fail per
TDD approach - the skill file needs to be updated to pass this test.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The assertion now matches "subagent-driven-development", "Subagent-Driven
Development", and "Subagent Driven" since Claude's responses may use
different casing and formatting styles.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Created automated test suite for testing superpowers skills using
Claude Code CLI in headless mode.
New files:
- tests/claude-code/run-skill-tests.sh - Main test runner
- tests/claude-code/test-helpers.sh - Helper functions for testing
- tests/claude-code/test-subagent-driven-development.sh - First test
- tests/claude-code/README.md - Documentation
Test framework features:
- Run Claude Code with prompts and capture output
- Assertion helpers (contains, not_contains, count, order)
- Test project creation helpers
- Timeout support (default 5 minutes)
- Verbose mode for debugging
- Specific test selection
First test verifies subagent-driven-development skill:
- Skill loading
- Workflow ordering (spec compliance before code quality)
- Self-review requirements
- Plan reading efficiency (read once)
- Spec compliance reviewer skepticism
- Review loops
- Task context provision
Run with: cd tests/claude-code && ./run-skill-tests.sh
Jesse Vincent