Compare commits

...

5 Commits

Author SHA1 Message Date
Jesse Vincent cda4b15fd6 fix(tests): stop the SDD skill test flaking on timing and prose case
tests/claude-code/test-subagent-driven-development.sh failed
intermittently for two independent reasons:

- Budget mismatch: the file runs 9 prompts with a 90s timeout each
  (810s worst case) inside the runner's 600s per-file ceiling, so slow
  backend days produced spurious timeouts. Raise the runner default to
  900s and fix the help text, which claimed the default was 300.
- Case-sensitive prose matching: the assert helpers grepped free-form
  model output case-sensitively, but models capitalize the skill's own
  headings — observed failures include "Do Not Trust the Report"
  missing pattern "not trust" and a structured answer missing
  "First:.*spec.*compliance". Match case-insensitively in
  assert_contains/assert_not_contains/assert_count/assert_order, widen
  two Test 5 keyword patterns to phrasings observed in real runs, and
  make assert_order dump the output on failure the way assert_contains
  already does, so the next flake is diagnosable.

Observed 3 failures across 4 runs before the change (timeout, two
distinct pattern misses); 3/3 consecutive full runs pass after it.
2026-07-16 04:00:15 +00:00
Jesse Vincent 93b250b155 fix(codex): make package script and its test portable beyond macOS/bsdtar
The packaging pipeline only worked on a Mac with default umask, for
three stacked reasons:

- The deterministic-metadata tar flags (--uid/--gid/--uname/--gname)
  are bsdtar spellings; GNU tar rejects them, so the tar.gz archive
  step died on Linux. Detect the tar flavor and use --owner=:0
  --group=:0 --numeric-owner on GNU tar, which writes byte-identical
  ustar headers (uid/gid 0, empty uname/gname).
- Staged file modes depended on two umasks canceling out: git archive
  masks entry modes with tar.umask (git default 0002 -> 775), and the
  unflagged tar extraction re-masked with the process umask (022 on
  macOS -> 755, but 002 elsewhere -> 775). Pin tar.umask=0022 on the
  archive call and extract with -p so staged modes are canonical
  755/644 on every machine.
- The test's timestamp assertion parsed bsdtar's -tv column layout and
  expected epoch 0 rendered in a US timezone ("Dec 31 1969"); GNU tar
  uses different columns and UTC hosts render "1970-01-01". Assert
  mtime == 0 via python3 tarfile instead, matching how the test
  already checks zip timestamps.

tests/codex/test-package-codex-plugin.sh now passes on Linux/GNU tar;
the bsdtar branch preserves the exact flags that passed on macOS.
2026-07-16 03:32:25 +00:00
Jesse Vincent fb7b07088e docs: fix dead references to pruned claude-code-tools.md/copilot-tools.md
e7ddc25 deleted claude-code-tools.md and copilot-tools.md but left
writing-skills and the porting guide's reference-integration table
pointing at them. State the current architecture instead: Claude Code's
personal-skills path inline, and "no adapter file needed" for the
harnesses that ride the Claude Code-compatible tool surface.

Reported by @rasibintang (#1969, with a fix proposed in #1970).

Fixes #1969
2026-07-15 19:15:16 +00:00
Gaurav Dubey 7a81eb7177 test(pi): scope mapping assertions to the table, not whole file
The pi tokens (subagent, pi-subagents, Task, TODO.md) also appear in the
surrounding prose, so matching the whole file passed even with the mapping
table deleted — the exact regression this test exists to catch. Filter to
table rows (lines starting with '|') so the assertion fails when the table
is gone and passes on dev.

Reported by @muunkky on #1987 (approach from #1983); verified failing-first
by stripping the table rows from pi-tools.md.
2026-07-15 11:10:55 -07:00
Gaurav Dubey 2b1c06a849 test: realign antigravity + pi mapping assertions with pruned references
Commit e7ddc25 ('Prune per-harness tool-mapping boilerplate') deliberately
removed the skill-loading explainers and generic action->tool tables from
antigravity-tools.md and pi-tools.md, keeping only the harness-specific
notes (subagent dispatch, task tracking). It did not touch tests/, so two
content-assertion tests kept asserting the removed tokens and now fail on
both dev and main:

  - tests/antigravity/test-antigravity-tools.sh: asserted view_file,
    IsSkillFile, run_command, grep_search (all pruned)
  - tests/pi/test-pi-extension.mjs: asserted read/write/edit/bash (pruned)

Update both to assert only the surviving harness-specific mappings. No
reference or skill content is changed; only the stale test assertions.
2026-07-15 11:10:55 -07:00
9 changed files with 57 additions and 34 deletions
+3 -3
View File
@@ -784,10 +784,10 @@ Use this as the live index; when in doubt, read the files, not this table.
| Harness | Entry point | Bootstrap mechanism | Tool mapping | Tests | Distribution |
|---|---|---|---|---|---|
| Claude Code | `.claude-plugin/plugin.json` + `hooks/hooks.json` | shell hook → `hooks/session-start` (`hookSpecificOutput.additionalContext`) | native `Skill` tool; `references/claude-code-tools.md` | `tests/hooks/` | marketplace |
| Claude Code | `.claude-plugin/plugin.json` + `hooks/hooks.json` | shell hook → `hooks/session-start` (`hookSpecificOutput.additionalContext`) | native `Skill` tool; no adapter file needed | `tests/hooks/` | marketplace |
| Codex | `.codex-plugin/plugin.json` (declares empty `hooks`) | native skill discovery (no session-start hook) | `references/codex-tools.md` | `tests/codex/`, `tests/codex-plugin-sync/` | fork sync (`scripts/sync-to-codex-plugin.sh`) |
| Cursor | `.cursor-plugin/plugin.json` + `hooks/hooks-cursor.json` | shell hook → `hooks/session-start` (`additional_context`) | `references/claude-code-tools.md` | `tests/hooks/` | hand-authored |
| Copilot CLI | (shares Claude Code hook path; `COPILOT_CLI` env) | shell hook → `hooks/session-start` (`additionalContext`) | `references/copilot-tools.md` | `tests/hooks/` | — |
| Cursor | `.cursor-plugin/plugin.json` + `hooks/hooks-cursor.json` | shell hook → `hooks/session-start` (`additional_context`) | none needed (Claude Codecompatible tool surface) | `tests/hooks/` | hand-authored |
| Copilot CLI | (shares Claude Code hook path; `COPILOT_CLI` env) | shell hook → `hooks/session-start` (`additionalContext`) | none needed (Claude Codecompatible tool surface) | `tests/hooks/` | — |
| Gemini CLI | `gemini-extension.json` + `GEMINI.md` | instructions file `@`-includes bootstrap + mapping | `references/gemini-tools.md` | — | `gemini extensions install` |
| Kimi Code | `.kimi-plugin/plugin.json` | manifest `sessionStart.skill` loads `using-superpowers` | inline `skillInstructions` in manifest | `tests/kimi/` | marketplace or `/plugins install` GitHub URL |
| OpenCode | `.opencode/plugins/superpowers.js` (declared via root `package.json` `main`) | in-process: `config` hook registers skills dir; `experimental.chat.messages.transform` injects user message | inline in `superpowers.js` | `tests/opencode/` | `opencode.json` plugin git URL |
+13 -4
View File
@@ -230,14 +230,16 @@ prepare_metadata_root() {
METADATA_ROOT="$(prepare_metadata_root "$METADATA_SOURCE")"
git -C "$REPO_ROOT" archive --format=tar "$REF" -- \
# Pin tar.umask and extract with -p so staged modes are canonical 755/644
# regardless of the builder's git config or process umask.
git -C "$REPO_ROOT" -c tar.umask=0022 archive --format=tar "$REF" -- \
.codex-plugin \
CODE_OF_CONDUCT.md \
LICENSE \
README.md \
assets \
skills \
| tar -xf - -C "$STAGE"
| tar -xpf - -C "$STAGE"
VERSION="$(jq -r '.version // empty' "$STAGE/.codex-plugin/plugin.json")"
[[ -n "$VERSION" ]] || die "could not read version from .codex-plugin/plugin.json"
@@ -298,12 +300,19 @@ case "$FORMAT" in
)
;;
tar.gz)
# Match the prior official archive's deterministic tar entry metadata.
# Match the prior official archive's deterministic tar entry metadata:
# ustar entries with uid/gid 0 and empty uname/gname. GNU tar and bsdtar
# (macOS) spell those flags differently.
if tar --version 2>/dev/null | grep -q 'GNU tar'; then
TAR_METADATA_FLAGS=(--owner=:0 --group=:0 --numeric-owner)
else
TAR_METADATA_FLAGS=(--uid 0 --gid 0 --uname '' --gname '')
fi
TZ=UTC find "$STAGE" -exec touch -t 197001010000 {} +
(
cd "$STAGE"
rm -f "$OUTPUT"
COPYFILE_DISABLE=1 tar -cf - --no-recursion --format ustar --uid 0 --gid 0 --uname '' --gname '' -T "$ARCHIVE_LIST" |
COPYFILE_DISABLE=1 tar -cf - --no-recursion --format ustar "${TAR_METADATA_FLAGS[@]}" -T "$ARCHIVE_LIST" |
gzip -9n >"$OUTPUT"
)
;;
+1 -1
View File
@@ -9,7 +9,7 @@ description: Use when creating new skills, editing existing skills, or verifying
**Writing skills IS Test-Driven Development applied to process documentation.**
**Personal skills live in your runtime's skills directory** — see [claude-code-tools.md](../using-superpowers/references/claude-code-tools.md), [codex-tools.md](../using-superpowers/references/codex-tools.md), [copilot-tools.md](../using-superpowers/references/copilot-tools.md), or [gemini-tools.md](../using-superpowers/references/gemini-tools.md) for the path on your runtime. Codex, Copilot CLI, and Gemini CLI all also recognize `~/.agents/skills/` as a cross-runtime alias.
**Personal skills live in your runtime's skills directory** (`~/.claude/skills/` on Claude Code) — see [codex-tools.md](../using-superpowers/references/codex-tools.md) or [gemini-tools.md](../using-superpowers/references/gemini-tools.md) for the path on those runtimes. Codex, Copilot CLI, and Gemini CLI all also recognize `~/.agents/skills/` as a cross-runtime alias.
You write test cases (pressure scenarios with subagents), watch them fail (baseline behavior), write the skill (documentation), watch tests pass (agents comply), and refactor (close loopholes).
+5 -12
View File
@@ -2,8 +2,9 @@
# Validate the Antigravity (agy) integration. agy installs the existing plugin
# directly (`agy plugin install <repo-url>`): it loads the bundled skills and
# runs the SessionStart hook for bootstrap, so there is no agy-specific scaffold
# to test. What IS agy-specific is the tool mapping — agy has no `Skill` tool and
# loads skills by reading SKILL.md with view_file — and SKILL.md pointing at it.
# to test. What IS agy-specific is the tool mapping — subagent dispatch via
# invoke_subagent (self/research types) and task tracking via a task artifact —
# and SKILL.md pointing at it.
#
# Mirrors tests/pi/test-pi-extension.mjs's "tools reference documents
# harness-specific mappings" check. CI-safe: does not require `agy` installed.
@@ -22,16 +23,8 @@ echo "test-antigravity-tools: checking Antigravity tool mapping"
# --- Mapping exists ---------------------------------------------------------
[ -f "$MAPPING" ] || fail "tool mapping missing at $MAPPING"
# --- Skill-load mechanism: view_file on SKILL.md (IsSkillFile), no Skill tool -
grep -qiE "view_file" "$MAPPING" \
|| fail "mapping does not document view_file as the file/skill-read tool"
grep -qiE "SKILL\.md" "$MAPPING" \
|| fail "mapping does not document reading SKILL.md as the skill-load path"
grep -q "IsSkillFile" "$MAPPING" \
|| fail "mapping does not document setting IsSkillFile when loading a skill"
# --- Core action→tool mappings are documented -------------------------------
for tool in write_to_file replace_file_content run_command grep_search invoke_subagent; do
for tool in write_to_file replace_file_content invoke_subagent; do
grep -q "$tool" "$MAPPING" \
|| fail "mapping does not document the '$tool' tool"
done
@@ -50,4 +43,4 @@ grep -qE 'ArtifactType.*task|task. artifact' "$MAPPING" \
grep -q "antigravity-tools.md" "$SKILL" \
|| fail "SKILL.md Platform Adaptation does not reference antigravity-tools.md"
echo "PASS: Antigravity tool mapping valid (view_file skill-load, agy tools, SKILL.md link)"
echo "PASS: Antigravity tool mapping valid (subagent dispatch, task artifact, SKILL.md link)"
+3 -2
View File
@@ -25,7 +25,8 @@ fi
# Parse command line arguments
VERBOSE=false
SPECIFIC_TEST=""
TIMEOUT=600 # Default 10 minute timeout per test
TIMEOUT=900 # Per-test-file budget; must exceed the file's worst case
# (test-subagent-driven-development.sh: 9 prompts x 90s each)
RUN_INTEGRATION=false
while [[ $# -gt 0 ]]; do
@@ -52,7 +53,7 @@ while [[ $# -gt 0 ]]; do
echo "Options:"
echo " --verbose, -v Show verbose output"
echo " --test, -t NAME Run only the specified test"
echo " --timeout SECONDS Set timeout per test (default: 300)"
echo " --timeout SECONDS Set timeout per test (default: 900)"
echo " --integration, -i Run integration tests (slow, 10-30 min)"
echo " --help, -h Show this help"
echo ""
+11 -5
View File
@@ -30,12 +30,14 @@ run_claude() {
# Check if output contains a pattern
# Usage: assert_contains "output" "pattern" "test name"
# Matching is case-insensitive: patterns are prose keywords, and models
# freely capitalize skill terms ("Do Not Trust", "Spec Compliance").
assert_contains() {
local output="$1"
local pattern="$2"
local test_name="${3:-test}"
if echo "$output" | grep -q "$pattern"; then
if echo "$output" | grep -qi "$pattern"; then
echo " [PASS] $test_name"
return 0
else
@@ -54,7 +56,7 @@ assert_not_contains() {
local pattern="$2"
local test_name="${3:-test}"
if echo "$output" | grep -q "$pattern"; then
if echo "$output" | grep -qi "$pattern"; then
echo " [FAIL] $test_name"
echo " Did not expect to find: $pattern"
echo " In output:"
@@ -74,7 +76,7 @@ assert_count() {
local expected="$3"
local test_name="${4:-test}"
local actual=$(echo "$output" | grep -c "$pattern" || echo "0")
local actual=$(echo "$output" | grep -ci "$pattern" || echo "0")
if [ "$actual" -eq "$expected" ]; then
echo " [PASS] $test_name (found $actual instances)"
@@ -98,16 +100,20 @@ assert_order() {
local test_name="${4:-test}"
# Get line numbers where patterns appear
local line_a=$(echo "$output" | grep -n "$pattern_a" | head -1 | cut -d: -f1)
local line_b=$(echo "$output" | grep -n "$pattern_b" | head -1 | cut -d: -f1)
local line_a=$(echo "$output" | grep -ni "$pattern_a" | head -1 | cut -d: -f1)
local line_b=$(echo "$output" | grep -ni "$pattern_b" | head -1 | cut -d: -f1)
if [ -z "$line_a" ]; then
echo " [FAIL] $test_name: pattern A not found: $pattern_a"
echo " In output:"
echo "$output" | sed 's/^/ /'
return 1
fi
if [ -z "$line_b" ]; then
echo " [FAIL] $test_name: pattern B not found: $pattern_b"
echo " In output:"
echo "$output" | sed 's/^/ /'
return 1
fi
@@ -96,13 +96,13 @@ echo "Test 5: Spec compliance reviewer mindset..."
output=$(run_claude "What is the spec compliance reviewer's attitude toward the implementer's report in subagent-driven-development?" "$CLAUDE_PROMPT_TIMEOUT")
if assert_contains "$output" "not trust\|don't trust\|skeptical\|verify.*independently\|suspiciously" "Reviewer is skeptical"; then
if assert_contains "$output" "not.*trust\|don't trust\|skeptical\|verify.*independently\|suspiciously" "Reviewer is skeptical"; then
: # pass
else
exit 1
fi
if assert_contains "$output" "read.*code\|inspect.*code\|verify.*code" "Reviewer reads code"; then
if assert_contains "$output" "read.*code\|inspect.*code\|verify.*code\|read.*diff\|trust.*diff" "Reviewer reads code"; then
: # pass
else
exit 1
+7 -2
View File
@@ -210,8 +210,13 @@ assert_equals "$tar_archive_paths" "$archive_paths" "zip and tar.gz archives con
tar_task_brief_mode="$(tar -tzvf "$tar_archive" skills/subagent-driven-development/scripts/task-brief | awk '{print $1}')"
assert_equals "$tar_task_brief_mode" "-rwxr-xr-x" "tar.gz archive preserves executable script mode"
tar_metadata_times="$(tar -tzvf "$tar_archive" | awk '{print $6, $7, $8}' | sort -u)"
assert_equals "$tar_metadata_times" "Dec 31 1969" "tar.gz archive normalizes entry timestamps"
tar_metadata_times="$(python3 - "$tar_archive" <<'PY'
import sys, tarfile
with tarfile.open(sys.argv[1]) as archive:
print(sorted({member.mtime for member in archive.getmembers()}))
PY
)"
assert_equals "$tar_metadata_times" "[0]" "tar.gz archive normalizes entry timestamps"
metadata_archive="$TEST_ROOT/metadata-source.tar.gz"
metadata_zip="$TEST_ROOT/metadata-source.zip"
+12 -3
View File
@@ -122,7 +122,16 @@ test('pi tools reference documents pi-specific mappings', async () => {
assert.equal(existsSync(piToolsPath), true, 'pi-tools.md should exist');
const text = await readFile(piToolsPath, 'utf8');
for (const expected of ['Skill', 'Task', 'TodoWrite', 'read', 'write', 'edit', 'bash']) {
assert.match(text, new RegExp(expected));
}
// Assert against the mapping-table rows only. The surrounding prose mentions
// these same tokens, so matching the whole file would still pass if the table
// were deleted — the exact regression this test exists to catch.
const rows = text.split('\n').filter((line) => line.startsWith('|'));
assert.ok(
rows.some((row) => /subagent/i.test(row)),
'mapping table documents subagent dispatch',
);
assert.ok(
rows.some((row) => /todo|task/i.test(row)),
'mapping table documents task tracking',
);
});