chore(llms): gate live model tests behind test:live (#610)

* chore(llms): gate live model tests behind test:live

Keep *.live.test.ts out of npm test so local CI stays fast; run them
explicitly via npm run test:live when checking provider compatibility.

* chore: skill gh permission

.agents/skills/submit-pr-from-current-changes/SKILL.md

@@ -24,6 +24,13 @@ Before attempting to push or open a PR, verify that the necessary tools are avai
 - Check that `gh` CLI is installed and authenticated (`gh auth status`). If not available, stop and ask the user to install and authenticate GitHub CLI first.
 - If the workflow uses MCP tools for GitHub operations, verify the MCP server is accessible.
 
+## Shell Permissions
+
+`gh` reads credentials from the OS keyring. A sandboxed shell cannot reach the keyring and will report a false auth failure.
+
+- Run every `gh` command (`gh auth status`, `gh pr create`, `gh pr view`, …) and every `git push` / `git fetch` that talks to GitHub with full permissions on the **first** attempt (`required_permissions: ["all"]` for the Shell tool). Do not probe these in the sandbox first.
+- Local read-only git inspection (`git status`, `git diff`, `git log`, `git branch`) may stay sandboxed.
+
 ## Procedure
 
 1. **Read contribution rules.** Read `CONTRIBUTING.md` and any package-level instructions (e.g. `packages/*/AGENTS.md`) relevant to the changed files.

AGENTS.md

@@ -132,6 +132,7 @@ const pageInfo = await this.pageController.getPageInfo()
 - **Location**: co-located, `src/foo.test.ts` next to `src/foo.ts`
 - **Coverage today**: `packages/llms` only — other packages will follow incrementally
 - **Adding tests to a new package**: create `vitest.config.ts` in the package and add a `"test": "vitest run"` script. Root `npm test` and `node scripts/ci.js` pick it up through npm workspaces.
+- **Live tests** (hit real external APIs, slow/costly): name them `*.live.test.ts`, exclude them from the package's `test` script, and expose them via a `test:live` script. Root `npm run test:live` runs all of them; they never run in `npm test` or CI. Template: `packages/llms`.
 - **Template**: See @page-agent/llms
 
 ```bash

package.json

@@ -37,6 +37,7 @@
         "postpublish": "npm run postpublish --workspaces --if-present",
         "typecheck": "tsc --noEmit -p tsconfig.typecheck.json && tsc --noEmit -p packages/extension/tsconfig.json",
         "test": "npm test --workspaces --if-present",
+        "test:live": "npm run test:live --workspaces --if-present",
         "lint": "eslint .",
         "ci": "node scripts/ci.js",
         "cleanup": "rm -rf packages/*/dist && rm -rf packages/*/.output",

packages/llms/package.json

@@ -43,7 +43,8 @@
     "homepage": "https://alibaba.github.io/page-agent/",
     "scripts": {
         "build": "vite build",
-        "test": "vitest run",
+        "test": "vitest run --project llms",
+        "test:live": "vitest run --project llms:live",
         "prepublishOnly": "node ../../scripts/pre-publish.js",
         "postpublish": "node ../../scripts/post-publish.js"
     },

packages/llms/src/models.live.test.ts

@@ -10,9 +10,10 @@
  * Tests `OpenAIClient` directly (not the `LLM` retry wrapper), so a failure
  * always reflects the very first request/response — no retry can mask it.
  *
- * Each provider's tests skip (not fail) when its `TESTING_*_KEY` env var is
+ * Per the `*.live.test.ts` convention this suite is excluded from `npm test`
- * absent, so this stays CI-safe. To actually run these, put keys in the
+ * (slow, costs tokens) — run it manually with `npm run test:live`. Each
- * repo-root `.env`:
+ * provider's tests skip (not fail) when its `TESTING_*_KEY` env var is
+ * absent. Keys live in the repo-root `.env`:
  *
  *   TESTING_OPENROUTER_KEY=...
  *   TESTING_DEEPSEEK_KEY=...

packages/llms/vitest.config.js

@@ -1,12 +1,27 @@
-import { defineConfig } from 'vitest/config'
+import { configDefaults, defineConfig } from 'vitest/config'
 
+// Convention: `*.live.test.ts` hits real provider APIs (slow, costs tokens)
+// and only runs via `npm run test:live`. Everything else runs on `npm test`.
 export default defineConfig({
 	test: {
-		name: 'llms',
-		include: ['src/**/*.test.ts'],
-		// Keep live provider suites under OpenRouter's ~20 req/min free-route cap.
-		maxConcurrency: 2,
 		// Suppress console output from passing tests; failed tests still get their logs.
 		silent: 'passed-only',
+		projects: [
+			{
+				test: {
+					name: 'llms',
+					include: ['src/**/*.test.ts'],
+					exclude: [...configDefaults.exclude, 'src/**/*.live.test.ts'],
+				},
+			},
+			{
+				test: {
+					name: 'llms:live',
+					include: ['src/**/*.live.test.ts'],
+					// Keep live provider suites under OpenRouter's ~20 req/min free-route cap.
+					maxConcurrency: 2,
+				},
+			},
+		],
 	},
 })