diff --git a/CHANGELOG.md b/CHANGELOG.md index 258ba47..8f2d346 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,38 @@ --- +## [0.3.4] - 2026-05-12 + +### ๐Ÿ› ไฟฎๅค + +- **ๅ…ผๅฎน OpenClaw v2026.4.7 ไปฅไธ‹็‰ˆๆœฌ L1 ๆŠฝๅ–็ฉบ่พ“ๅ‡บ**๏ผšๆ—งๅฎฟไธปไธๆ”ฏๆŒ `systemPromptOverride`๏ผŒ้€š่ฟ‡ `extraSystemPrompt` ๅ›ž้€€ๆณจๅ…ฅ็ณป็ปŸๆ็คบ๏ผŒ็กฎไฟ LLM ๆŒ‰ๆ•ฐๆฎๆๅ–ๅŠฉๆ‰‹่บซไปฝๅทฅไฝœใ€‚ +- **TCVDB hybrid ๅฌๅ›žๅ†—ไฝ™ๅŒ้‡ HTTP ่ฐƒ็”จ**๏ผš`auto-recall` ๅฏน TCVDB ๅ‘ไธคๆฌก็›ธๅŒ็š„ `hybridSearch` ่ฏทๆฑ‚๏ผˆไธ” keyword ่ทฏๅพ„ๅฐ† FTS5 OR ่กจ่พพๅผ้”™่ฏฏไผ ๅ…ฅ BM25 ็ผ–็ ๅ™จ๏ผ‰ใ€‚ๆ–ฐๅขž `nativeHybridSearch` ็Ÿญ่ทฏ๏ผŒTCVDB ๅ•ๆฌก่ฐƒ็”จๅณๅฏๅฎŒๆˆ dense + sparse + RRF๏ผŒrecall ่€—ๆ—ถๅ‡ๅŠ๏ผˆ~50-120ms๏ผ‰ใ€‚ +- **L2 parser ๅฏน้ฝ Go ๅŽ็ซฏ**๏ผšๅขžๅŠ  mermaid fallback๏ผŒไฟฎๅค `first{...last}` JSON ๆๅ–้€ป่พ‘ใ€‚ + +### โœจ ๆ”น่ฟ› + +- **VDB HTTP ่ฏทๆฑ‚็บง่ฎกๆ—ถ**๏ผš`tcvdb-client` ๆฏๆฌก่ฏทๆฑ‚ๆ‰“ไธ€ๆก info ่ฎกๆ—ถๆ—ฅๅฟ—๏ผˆ`/document/hybridSearch 85ms`๏ผ‰๏ผŒretry/ๅคฑ่ดฅ็ป†่Š‚ไฟๆŒ debug ็บงๅˆซใ€‚ +- **ๅฏๅŠจ่ทฏๅพ„่ฏฏๅฏผๆ€งๆ—ฅๅฟ—้™็บงไธบ DEBUG**๏ผšstore manifest ไธไธ€่‡ดใ€sqlite schema migrationใ€profile-sync MD5 mismatch ็ญ‰ๆญฃๅธธๅœบๆ™ฏไธๅ†ๆ‰“ warn/info๏ผŒ้ฟๅ… AI ่ฏฏๅˆคใ€‚ +- **L1 ๆๅ–่ฐƒ่ฏ•ๆ—ฅๅฟ—**๏ผšๆ–ฐๅขž `[l1-debug]` ็ณปๅˆ—๏ผˆRESOLVE / INVOKE / RESULT / EMPTY_DUMP / ENTRY / NO_JSON๏ผ‰๏ผŒๆ–นไพฟๅฎšไฝ LLM ่ฐƒ็”จ้“พ้—ฎ้ข˜ใ€‚ + +### ๐Ÿ”ง ๅ…ผๅฎนๆ€ง้€‚้… + +- **OC 2026.4.23 Zod schema ๅ…ผๅฎน patch ่„šๆœฌ**๏ผˆ`scripts/bugfix-20260423/`๏ผ‰๏ผšไธ€้”ฎไฟฎๅค `allowConversationAccess` ่ขซ `.strict()` ๆ‹’็ป็š„้—ฎ้ข˜๏ผŒๅซ่ฝป้‡็‰ˆ่„šๆœฌใ€ๅ…จ่‡ชๅŠจ่„šๆœฌใ€ๆ‰‹ๅŠจ SOP ๆ–‡ๆกฃใ€‚ +- Offload ๆ—ฅๅฟ—ๅŽปๆމ `Backend` ๅ‰็ผ€๏ผŒ้ป˜่ฎค่ถ…ๆ—ถไธบ 120sใ€‚ + +### ๐Ÿ“ฆ ๆ–ฐๅŠŸ่ƒฝ + +- **Offload Local Mode**๏ผšๆ”ฏๆŒๆœฌๅœฐๆจกๅผ่ฟ่กŒ offload๏ผˆไธไพ่ต–่ฟœ็ซฏๅŽ็ซฏ๏ผ‰ใ€‚ +- **Docker ไธ€ไฝ“ๅŒ–้•œๅƒ**๏ผˆ`Dockerfile.hermes`๏ผ‰๏ผšๅ•ๅฎนๅ™จๆ†็ป‘ Hermes Agent + memory_tencentdb ๆ’ไปถ + TDAI Memory Gateway๏ผŒ็ปŸไธ€ `MODEL_*` ็Žฏๅขƒๅ˜้‡้ฉฑๅŠจใ€‚ + +### โœ… ๆต‹่ฏ• + +- ไฟฎๅค `fault-injection` FI-05 mock config ็ผบ `embedding` ๅญ—ๆฎต +- ไฟฎๅค `cli.test` dependencies ๆ–ญ่จ€้€‚้…ๆ–ฐๅขžไพ่ต– +- ่ทณ่ฟ‡ `patch-effectiveness` ๅทฒๅˆ ้™ค็š„ `install-plugin.sh` ๆต‹่ฏ• + +--- + ## [0.3.3] - 2026-05-08 ### ๐Ÿ› ไฟฎๅค diff --git a/package.json b/package.json index 7d54d85..2d74158 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@tencentdb-agent-memory/memory-tencentdb", - "version": "0.3.3", + "version": "0.3.4", "description": "Four-layer local memory system plugin for OpenClaw โ€” auto-captures, structures, and profiles conversational knowledge using local LLM + SQLite vector search (L0โ†’L1โ†’L2โ†’L3 pipeline)", "type": "module", "main": "./dist/index.mjs", diff --git a/scripts/bugfix-20260423/BUGFIX-20260423-SOP.md b/scripts/bugfix-20260423/BUGFIX-20260423-SOP.md new file mode 100644 index 0000000..c1a6d3c --- /dev/null +++ b/scripts/bugfix-20260423/BUGFIX-20260423-SOP.md @@ -0,0 +1,88 @@ +# Bugfix-20260423 ๆ‰“้•œๅƒ SOP + +> **้€‚็”จ็‰ˆๆœฌ**: OpenClaw 2026.4.23 +> **ไฟฎๅค้—ฎ้ข˜**: Issue #73806 โ€” Zod schema `.strict()` ๆ‹’็ป `hooks.allowConversationAccess`๏ผŒๅฏผ่‡ด้žๆ†็ป‘ๆ’ไปถๆ— ๆณ•ๆณจๅ†Œไผš่ฏ้’ฉๅญ +> **่„šๆœฌไฝ็ฝฎ**: `scripts/bugfix-20260423.sh` + +--- + +## ๆญฅ้ชคไธ€๏ผšๅœๆญข Gateway + +```bash +openclaw gateway stop +``` + +็กฎ่ฎคๅทฒๅœๆญข๏ผš + +```bash +ps aux | grep gateway +``` + +็กฎไฟๆฒกๆœ‰ `openclaw-gateway` ่ฟ›็จ‹ๅœจ่ฟ่กŒใ€‚ + +--- + +## ๆญฅ้ชคไบŒ๏ผšๆ‰ง่กŒ Patch + +```bash +cd /path/to/memory-tdai/scripts +bash bugfix-20260423.sh +``` + +--- + +## ๆญฅ้ชคไธ‰๏ผš้ชŒ่ฏ + +### 3.1 ้ชŒ่ฏ openclaw.json ้…็ฝฎ + +```bash +cat ~/.openclaw/openclaw.json | python3 -m json.tool | grep allowConversationAccess +``` + +้ข„ๆœŸ่พ“ๅ‡บ๏ผš + +``` +"allowConversationAccess": true +``` + +็กฎ่ฎคๅœจ `plugins.entries.memory-tencentdb.hooks` ไธ‹ใ€‚ + +### 3.2 ้ชŒ่ฏ Zod Schema dist ๆ–‡ไปถ + +ๅ…ˆๅฎšไฝ OpenClaw ๅฎ‰่ฃ…็›ฎๅฝ•๏ผˆ่ทฏๅพ„ๅ› ็Žฏๅขƒ่€Œๅผ‚๏ผŒไปฅไธ‹ไป…ไธบ็คบไพ‹๏ผ‰๏ผš + +```bash +# ๆ–นๅผไธ€๏ผš้€š่ฟ‡ which ่‡ชๅŠจๅฎšไฝ +OC_DIR=$(node -e "const p=require('path'),f=require('fs'); \ + const bin=require('child_process').execSync('which openclaw',{encoding:'utf8'}).trim(); \ + let d=p.dirname(f.realpathSync(bin)); \ + while(d!=p.dirname(d)){if(f.existsSync(p.join(d,'package.json'))){console.log(d);break;}d=p.dirname(d);}") +echo "$OC_DIR" + +# ๆ–นๅผไบŒ๏ผšๆ‰‹ๅŠจๆŒ‡ๅฎš๏ผˆ็คบไพ‹่ทฏๅพ„๏ผŒ่ฏทๆ นๆฎๅฎž้™…็Žฏๅขƒๆ›ฟๆข๏ผ‰ +# OC_DIR=~/.local/share/pnpm/global/5/.pnpm/openclaw@2026.4.23_@napi-rs+canvas@0.1.100/node_modules/openclaw +``` + +็„ถๅŽๆฃ€ๆŸฅ `zod-schema-BhKK4qYw.js`๏ผš + +```bash +cat "$OC_DIR/dist/zod-schema-BhKK4qYw.js" | grep allowConversationAccess -n +``` + +้ชŒ่ฏ่ฆ็‚น๏ผš + +1. `allowConversationAccess` ๅทฒๅ‡บ็Žฐๅœจ่พ“ๅ‡บไธญ +2. **ๅชๅ‡บ็Žฐไบ†ไธ€ๆฌก**๏ผˆๅชๆœ‰ไธ€่กŒๅŒน้…๏ผ‰ +3. ๆ‰€ๅœจ่กŒ็š„ไธŠไธ‹ๆ–‡ๅฝขๅฆ‚๏ผš`allowPromptInjection:z.boolean().optional(),allowConversationAccess:z.boolean().optional()}).strict().optional()` + + + +--- + +## ้ชŒ่ฏ้€š่ฟ‡ๅŽ + +ไธค้กน้ชŒ่ฏๅ‡้€š่ฟ‡ๅณๅฏ้‡ๆ–ฐๅฏๅŠจ Gateway๏ผš + +```bash +openclaw gateway run +``` diff --git a/scripts/bugfix-20260423/bugfix-20260423-full.sh b/scripts/bugfix-20260423/bugfix-20260423-full.sh new file mode 100644 index 0000000..a24a630 --- /dev/null +++ b/scripts/bugfix-20260423/bugfix-20260423-full.sh @@ -0,0 +1,298 @@ +#!/usr/bin/env bash +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• +# bugfix-20260423.sh โ€” OC 2026.4.23 allowConversationAccess ไฟฎๅค +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• +# Issue #73806: OC 2026.4.23 ็š„ Zod schema ไฝฟ็”จ .strict() ๆ‹’็ป +# hooks.allowConversationAccess ๅญ—ๆฎต๏ผŒๅฏผ่‡ด้žๆ†็ป‘ๆ’ไปถๆ— ๆณ•ๆณจๅ†Œไผš่ฏ้’ฉๅญ +# (llm_input, llm_output, agent_end)ใ€‚PR #71221 ๅœจ 4.24 ไฟฎๅคใ€‚ +# +# ๆœฌ่„šๆœฌๅšไธคไปถไบ‹๏ผˆๅ‡ๅน‚็ญ‰๏ผŒๅฏๅฎ‰ๅ…จ้‡ๅคๆ‰ง่กŒ๏ผ‰๏ผš +# 1. Patch dist JS: ็ป™ hooks zod schema ๆณจๅ…ฅ allowConversationAccess ๅญ—ๆฎต +# 2. ๅ†™ openclaw.json: ่ฎพ็ฝฎ hooks.allowConversationAccess = true +# +# ็‰ˆๆœฌ้™ๅˆถ๏ผšไป…ๅœจ OC 2026.4.23 ไธŠๆ‰ง่กŒ Part 1๏ผŒๅ…ถไป–็‰ˆๆœฌๅฎ‰ๅ…จ่ทณ่ฟ‡ dist patchใ€‚ +# Part 2 (้…็ฝฎๅ†™ๅ…ฅ) ไธ้™็‰ˆๆœฌ๏ผŒๅง‹็ปˆ็กฎไฟ้…็ฝฎๅญ˜ๅœจใ€‚ +# +# ็”จๆณ•๏ผš +# bash bugfix-20260423.sh [/path/to/openclaw] +# +# ็Žฏๅขƒๅ˜้‡๏ผš +# OPENCLAW_DIR โ€” ่ฆ†็›– openclaw ๅฎ‰่ฃ…่ทฏๅพ„๏ผˆไผ˜ๅ…ˆไบŽๅ‚ๆ•ฐ๏ผ‰ +# OPENCLAW_JSON โ€” ่ฆ†็›–้…็ฝฎๆ–‡ไปถ่ทฏๅพ„๏ผˆ้ป˜่ฎค ~/.openclaw/openclaw.json๏ผ‰ +# DEBUG=1 โ€” ๅผ€ๅฏ่ฐƒ่ฏ•่พ“ๅ‡บ +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• +set -euo pipefail + +RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m' +info() { echo -e "${CYAN}[INFO]${NC} $*"; } +ok() { echo -e "${GREEN}[OK]${NC} $*"; } +warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } +fail() { echo -e "${RED}[FAIL]${NC} $*" >&2; exit 1; } +debug() { [[ "${DEBUG:-}" == "1" ]] && echo -e "${CYAN}[DEBUG]${NC} $*" || true; } + +PLUGIN_ID="memory-tencentdb" +OPENCLAW_JSON="${OPENCLAW_JSON:-${HOME}/.openclaw/openclaw.json}" + +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• +# Part 1: Patch dist JS (ไป… 2026.4.23) +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• + +_resolve_openclaw_dir() { + # ๅ‚ๆ•ฐ > ็Žฏๅขƒๅ˜้‡ > ่‡ชๅŠจๆฃ€ๆต‹ + if [[ -n "${1:-}" ]]; then + echo "$1"; return 0 + fi + if [[ -n "${OPENCLAW_DIR:-}" && -d "${OPENCLAW_DIR}" ]]; then + echo "$OPENCLAW_DIR"; return 0 + fi + # ่‡ชๅŠจๅฎšไฝ + node -e " + const {dirname, join} = require('path'); + const {realpathSync, existsSync, readFileSync, statSync} = require('fs'); + function walkUp(start) { + let dir = statSync(start).isDirectory() ? start : dirname(start); + for (let i = 0; i < 10; i++) { + const pj = join(dir, 'package.json'); + if (existsSync(pj)) { + try { if (JSON.parse(readFileSync(pj,'utf8')).name==='openclaw') { console.log(dir); process.exit(0); } } catch {} + } + const parent = dirname(dir); + if (parent === dir) break; + dir = parent; + } + return null; + } + try { + const {execSync} = require('child_process'); + const bin = execSync('which openclaw',{encoding:'utf8'}).trim(); + const real = realpathSync(bin); + const found = walkUp(real); + if (found) { console.log(found); process.exit(0); } + const content = readFileSync(bin,'utf8'); + const m = content.match(/['\"]([^'\"]*openclaw[^'\"]*\\.(?:js|mjs))['\"]/) || + content.match(/['\"]([^'\"]*openclaw[^'\"]*)['\"].*node/); + if (m) { const f = walkUp(realpathSync(m[1])); if (f) { console.log(f); process.exit(0); } } + } catch {} + const searchDirs = [ + join(process.env.HOME||'/root','.local/share/pnpm'), + join(process.env.HOME||'/root','.local/node/lib/node_modules'), + '/usr/local/lib/node_modules','/usr/lib/node_modules', + ]; + for (const base of searchDirs) { + if (!existsSync(base)) continue; + try { + const {execSync:e2} = require('child_process'); + const out = e2('find '+JSON.stringify(base)+' -maxdepth 8 -name package.json -path \"*/openclaw/package.json\" 2>/dev/null',{encoding:'utf8',timeout:5000}).trim(); + for (const line of out.split('\\n')) { + if (!line) continue; + try { if (JSON.parse(readFileSync(line,'utf8')).name==='openclaw') { console.log(dirname(line)); process.exit(0); } } catch {} + } + } catch {} + } + process.exit(1); + " 2>/dev/null +} + +patch_dist_js() { + local oc_dir + oc_dir="$(_resolve_openclaw_dir "${1:-}")" || { + warn "[Part 1] ๆ‰พไธๅˆฐ OpenClaw ๅฎ‰่ฃ…็›ฎๅฝ•๏ผŒ่ทณ่ฟ‡ dist patch" + return 0 + } + + local dist_dir="$oc_dir/dist" + [[ -d "$dist_dir" ]] || { warn "[Part 1] dist ็›ฎๅฝ•ไธๅญ˜ๅœจ: $dist_dir๏ผŒ่ทณ่ฟ‡"; return 0; } + + local version + version=$(grep -oP '"version"\s*:\s*"\K[^"]+' "$oc_dir/package.json" 2>/dev/null || echo "unknown") + info "[Part 1] OpenClaw ็‰ˆๆœฌ: $version" + + # ็‰ˆๆœฌ้—จๆŽง๏ผšไป… 2026.4.23 + if [[ ! "$version" =~ ^2026\.4\.23($|[-\.]) ]]; then + ok "[Part 1] ็‰ˆๆœฌ $version ไธ้œ€่ฆ schema patch๏ผŒ่ทณ่ฟ‡" + return 0 + fi + + # ็ฒพ็กฎๅฎšไฝ๏ผšhooks zod schema ็š„ๅ”ฏไธ€็‰นๅพ + local -a candidates + mapfile -t candidates < <( + grep -rl 'allowPromptInjection' "$dist_dir" --include='*.js' 2>/dev/null | while read -r _f; do + if perl -0777 -ne 'exit(0) if /allowPromptInjection\s*:\s*[a-zA-Z_\$][a-zA-Z0-9_\$]*\s*\.\s*boolean\s*\(\s*\)\s*\.\s*optional\s*\(\s*\)\s*[,\s]*\}\s*\)\s*\.\s*strict\s*\(\s*\)/; exit(1)' "$_f" 2>/dev/null; then + echo "$_f" + fi + done + ) + + if [[ ${#candidates[@]} -eq 0 ]]; then + warn "[Part 1] ๆœชๆ‰พๅˆฐ hooks zod schema ็›ฎๆ ‡ๆ–‡ไปถ๏ผŒ่ทณ่ฟ‡" + return 0 + elif [[ ${#candidates[@]} -gt 1 ]]; then + warn "[Part 1] ๅ‘็Žฐ ${#candidates[@]} ไธชๅŒน้…ๆ–‡ไปถ๏ผˆ้ข„ๆœŸ 1๏ผ‰๏ผŒๅฎ‰ๅ…จ่ตท่ง่ทณ่ฟ‡" + return 0 + fi + + local target="${candidates[0]}" + local relpath="${target#$dist_dir/}" + debug "[Part 1] ็›ฎๆ ‡: $relpath" + + # ๅน‚็ญ‰๏ผš็›ฎๆ ‡ๆ–‡ไปถไธญๅทฒๅŒ…ๅซ allowConversationAccess โ†’ ่ทณ่ฟ‡ + if grep -q 'allowConversationAccess' "$target" 2>/dev/null; then + ok "[Part 1] allowConversationAccess ๅทฒๅญ˜ๅœจไบŽ $relpath๏ผŒ่ทณ่ฟ‡" + return 0 + fi + + # ๅค‡ไปฝ + [[ -f "${target}.pre-aca-patch.bak" ]] || cp "$target" "${target}.pre-aca-patch.bak" + + # ๆณจๅ…ฅ๏ผš็”จ็ฒพ็กฎๅ˜้‡ๅๅŒน้…๏ผŒ้ฟๅ…่ดชๅฉชๅ›žๆบฏ + perl -0777 -i -pe ' + s/(allowPromptInjection\s*:\s*[a-zA-Z_\$][a-zA-Z0-9_\$]*\s*\.\s*boolean\s*\(\s*\)\s*\.\s*optional\s*\(\s*\))(\s*\}\s*\)\s*\.\s*strict\s*\(\s*\))/$1,allowConversationAccess:z.boolean().optional()$2/ + ' "$target" + + # ้ชŒ่ฏ + if grep -q 'allowConversationAccess' "$target" 2>/dev/null; then + ok "[Part 1] $relpath โ€” patch ๆˆๅŠŸ" + else + warn "[Part 1] patch ้ชŒ่ฏๅคฑ่ดฅ๏ผŒๆขๅคๅค‡ไปฝ" + cp "${target}.pre-aca-patch.bak" "$target" + return 1 + fi +} + +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• +# Part 2: ๅ†™ๅ…ฅ openclaw.json (ไธ้™็‰ˆๆœฌ๏ผŒๅง‹็ปˆ็กฎไฟ้…็ฝฎๅญ˜ๅœจ) +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• + +patch_config_json() { + if [[ ! -f "$OPENCLAW_JSON" ]]; then + warn "[Part 2] openclaw.json ไธๅญ˜ๅœจ: $OPENCLAW_JSON๏ผŒ่ทณ่ฟ‡" + return 0 + fi + + # ๅน‚็ญ‰ๆฃ€ๆต‹ + local exists + exists=$(python3 -c " +import json +try: + with open('$OPENCLAW_JSON') as f: + cfg = json.load(f) + val = cfg.get('plugins',{}).get('entries',{}).get('$PLUGIN_ID',{}).get('hooks',{}).get('allowConversationAccess') + print('yes' if val is True else 'no') +except Exception: + print('no') +" 2>/dev/null || echo "no") + + if [[ "$exists" == "yes" ]]; then + ok "[Part 2] hooks.allowConversationAccess ๅทฒๅญ˜ๅœจ๏ผŒ่ทณ่ฟ‡" + return 0 + fi + + # ๅ†™ๅ…ฅ + python3 -c " +import json + +with open('$OPENCLAW_JSON') as f: + cfg = json.load(f) + +entry = cfg.setdefault('plugins', {}).setdefault('entries', {}).setdefault('$PLUGIN_ID', {}) +hooks = entry.setdefault('hooks', {}) +hooks['allowConversationAccess'] = True + +with open('$OPENCLAW_JSON', 'w') as f: + json.dump(cfg, f, indent=2, ensure_ascii=False) + f.write('\n') +" + ok "[Part 2] hooks.allowConversationAccess = true ๅทฒๅ†™ๅ…ฅ" +} + +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• +# ไธปๅ…ฅๅฃ +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• + +echo "" +echo -e "${CYAN}โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•${NC}" +echo -e "${CYAN} bugfix-20260423: allowConversationAccess ไธ€้”ฎไฟฎๅค${NC}" +echo -e "${CYAN} Issue #73806 | ้€‚็”จ็‰ˆๆœฌ: OC 2026.4.23${NC}" +echo -e "${CYAN}โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•${NC}" +echo "" + +# โ”€โ”€ Step 1: ๅœๆญข Gateway โ”€โ”€ +info "[Step 1] ๅœๆญข Gateway..." +openclaw gateway stop 2>/dev/null || true +sleep 10 + +# ็กฎ่ฎคๅทฒๅœๆญข +if ps aux | grep -v grep | grep -q 'openclaw-gateway'; then + warn "ๆฃ€ๆต‹ๅˆฐ openclaw-gateway ไปๅœจ่ฟ่กŒ๏ผŒๅฐ่ฏ•ๅผบๅˆถๅœๆญข..." + pkill -9 -f 'openclaw-gateway' 2>/dev/null || true + sleep 3 +fi + +if ps aux | grep -v grep | grep -q 'openclaw-gateway'; then + fail "[Step 1] ๆ— ๆณ•ๅœๆญข openclaw-gateway๏ผŒ่ฏทๆ‰‹ๅŠจๅค„็†ๅŽ้‡่ฏ•" +fi +ok "[Step 1] Gateway ๅทฒๅœๆญข" +echo "" + +# โ”€โ”€ Step 2: ๆ‰ง่กŒ Patch โ”€โ”€ +info "[Step 2] ๆ‰ง่กŒ Patch..." +if ! patch_dist_js "${1:-}"; then + fail "[Step 2] dist JS patch ๅคฑ่ดฅ" +fi +if ! patch_config_json; then + fail "[Step 2] openclaw.json ๅ†™ๅ…ฅๅคฑ่ดฅ" +fi +echo "" + +# โ”€โ”€ Step 3: ้ชŒ่ฏ โ”€โ”€ +info "[Step 3] ้ชŒ่ฏ็ป“ๆžœ..." +echo "" + +# 3.1 ้ชŒ่ฏ openclaw.json +info " [3.1] ๆฃ€ๆŸฅ openclaw.json" +if grep -q '"allowConversationAccess"' "$OPENCLAW_JSON" 2>/dev/null; then + _json_line=$(grep -n 'allowConversationAccess' "$OPENCLAW_JSON") + ok " openclaw.json: allowConversationAccess โœ“" + echo -e " ${CYAN}${_json_line}${NC}" +else + fail "[Step 3.1] openclaw.json ไธญๆœชๆ‰พๅˆฐ allowConversationAccess" +fi +echo "" + +# 3.2 ้ชŒ่ฏ dist JS โ€” ๅชๆฃ€ๆŸฅ zod-schema-BhKK4qYw.js +info " [3.2] ๆฃ€ๆŸฅ zod-schema-BhKK4qYw.js" +_oc_dir="$(_resolve_openclaw_dir "${1:-}" 2>/dev/null)" || _oc_dir="" +_zod_file="$_oc_dir/dist/zod-schema-BhKK4qYw.js" +if [[ -n "$_oc_dir" && -f "$_zod_file" ]]; then + _match_count=$(grep -c 'allowConversationAccess' "$_zod_file" 2>/dev/null || echo "0") + + if [[ "$_match_count" -eq 0 ]]; then + fail "[Step 3.2] zod-schema-BhKK4qYw.js ไธญๆœชๆ‰พๅˆฐ allowConversationAccess" + elif [[ "$_match_count" -eq 1 ]]; then + ok " zod-schema-BhKK4qYw.js: allowConversationAccess ๅ‡บ็Žฐ 1 ๆฌก โœ“" + echo -e " ${CYAN}ๅŒน้…่กŒ:${NC}" + grep -n 'allowConversationAccess' "$_zod_file" | head -1 | sed 's/^/ /' + else + fail "[Step 3.2] zod-schema-BhKK4qYw.js ไธญ allowConversationAccess ๅ‡บ็Žฐ $_match_count ๆฌก๏ผˆ้ข„ๆœŸ 1 ๆฌก๏ผ‰๏ผŒๅฏ่ƒฝ้‡ๅคๆณจๅ…ฅ" + fi +else + fail "[Step 3.2] ๆ–‡ไปถไธๅญ˜ๅœจ: $_zod_file" +fi +echo "" + +# โ”€โ”€ Step 4: ้‡ๅฏ Gateway โ”€โ”€ +info "[Step 4] ้‡ๅฏ Gateway..." +openclaw gateway start +sleep 10 + +if ps aux | grep -v grep | grep -q 'openclaw-gateway'; then + ok "[Step 4] Gateway ๅทฒๅฏๅŠจ (pid=$(pgrep -f 'openclaw-gateway' | head -1))" +else + fail "[Step 4] Gateway ๅฏๅŠจๅคฑ่ดฅ๏ผŒ่ฏทๆฃ€ๆŸฅๆ—ฅๅฟ—" +fi + +echo "" +echo -e "${GREEN}โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•${NC}" +echo -e "${GREEN} โœ… bugfix-20260423 ไฟฎๅคๅฎŒๆˆ๏ผŒGateway ๅทฒ้‡ๅฏ${NC}" +echo -e "${GREEN}โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•${NC}" +echo "" diff --git a/scripts/bugfix-20260423/bugfix-20260423.sh b/scripts/bugfix-20260423/bugfix-20260423.sh new file mode 100644 index 0000000..046d06a --- /dev/null +++ b/scripts/bugfix-20260423/bugfix-20260423.sh @@ -0,0 +1,218 @@ +#!/usr/bin/env bash +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• +# bugfix-20260423.sh โ€” OC 2026.4.23 allowConversationAccess ไฟฎๅค +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• +# Issue #73806: OC 2026.4.23 ็š„ Zod schema ไฝฟ็”จ .strict() ๆ‹’็ป +# hooks.allowConversationAccess ๅญ—ๆฎต๏ผŒๅฏผ่‡ด้žๆ†็ป‘ๆ’ไปถๆ— ๆณ•ๆณจๅ†Œไผš่ฏ้’ฉๅญ +# (llm_input, llm_output, agent_end)ใ€‚PR #71221 ๅœจ 4.24 ไฟฎๅคใ€‚ +# +# ๆœฌ่„šๆœฌๅšไธคไปถไบ‹๏ผˆๅ‡ๅน‚็ญ‰๏ผŒๅฏๅฎ‰ๅ…จ้‡ๅคๆ‰ง่กŒ๏ผ‰๏ผš +# 1. Patch dist JS: ็ป™ hooks zod schema ๆณจๅ…ฅ allowConversationAccess ๅญ—ๆฎต +# 2. ๅ†™ openclaw.json: ่ฎพ็ฝฎ hooks.allowConversationAccess = true +# +# ็‰ˆๆœฌ้™ๅˆถ๏ผšไป…ๅœจ OC 2026.4.23 ไธŠๆ‰ง่กŒ Part 1๏ผŒๅ…ถไป–็‰ˆๆœฌๅฎ‰ๅ…จ่ทณ่ฟ‡ dist patchใ€‚ +# Part 2 (้…็ฝฎๅ†™ๅ…ฅ) ไธ้™็‰ˆๆœฌ๏ผŒๅง‹็ปˆ็กฎไฟ้…็ฝฎๅญ˜ๅœจใ€‚ +# +# ็”จๆณ•๏ผš +# bash bugfix-20260423.sh [/path/to/openclaw] +# +# ็Žฏๅขƒๅ˜้‡๏ผš +# OPENCLAW_DIR โ€” ่ฆ†็›– openclaw ๅฎ‰่ฃ…่ทฏๅพ„๏ผˆไผ˜ๅ…ˆไบŽๅ‚ๆ•ฐ๏ผ‰ +# OPENCLAW_JSON โ€” ่ฆ†็›–้…็ฝฎๆ–‡ไปถ่ทฏๅพ„๏ผˆ้ป˜่ฎค ~/.openclaw/openclaw.json๏ผ‰ +# DEBUG=1 โ€” ๅผ€ๅฏ่ฐƒ่ฏ•่พ“ๅ‡บ +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• +set -euo pipefail + +RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m' +info() { echo -e "${CYAN}[INFO]${NC} $*"; } +ok() { echo -e "${GREEN}[OK]${NC} $*"; } +warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } +fail() { echo -e "${RED}[FAIL]${NC} $*" >&2; exit 1; } +debug() { [[ "${DEBUG:-}" == "1" ]] && echo -e "${CYAN}[DEBUG]${NC} $*" || true; } + +PLUGIN_ID="memory-tencentdb" +OPENCLAW_JSON="${OPENCLAW_JSON:-${HOME}/.openclaw/openclaw.json}" + +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• +# Part 1: Patch dist JS (ไป… 2026.4.23) +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• + +_resolve_openclaw_dir() { + # ๅ‚ๆ•ฐ > ็Žฏๅขƒๅ˜้‡ > ่‡ชๅŠจๆฃ€ๆต‹ + if [[ -n "${1:-}" ]]; then + echo "$1"; return 0 + fi + if [[ -n "${OPENCLAW_DIR:-}" && -d "${OPENCLAW_DIR}" ]]; then + echo "$OPENCLAW_DIR"; return 0 + fi + # ่‡ชๅŠจๅฎšไฝ + node -e " + const {dirname, join} = require('path'); + const {realpathSync, existsSync, readFileSync, statSync} = require('fs'); + function walkUp(start) { + let dir = statSync(start).isDirectory() ? start : dirname(start); + for (let i = 0; i < 10; i++) { + const pj = join(dir, 'package.json'); + if (existsSync(pj)) { + try { if (JSON.parse(readFileSync(pj,'utf8')).name==='openclaw') { console.log(dir); process.exit(0); } } catch {} + } + const parent = dirname(dir); + if (parent === dir) break; + dir = parent; + } + return null; + } + try { + const {execSync} = require('child_process'); + const bin = execSync('which openclaw',{encoding:'utf8'}).trim(); + const real = realpathSync(bin); + const found = walkUp(real); + if (found) { console.log(found); process.exit(0); } + const content = readFileSync(bin,'utf8'); + const m = content.match(/['\"]([^'\"]*openclaw[^'\"]*\\.(?:js|mjs))['\"]/) || + content.match(/['\"]([^'\"]*openclaw[^'\"]*)['\"].*node/); + if (m) { const f = walkUp(realpathSync(m[1])); if (f) { console.log(f); process.exit(0); } } + } catch {} + const searchDirs = [ + join(process.env.HOME||'/root','.local/share/pnpm'), + join(process.env.HOME||'/root','.local/node/lib/node_modules'), + '/usr/local/lib/node_modules','/usr/lib/node_modules', + ]; + for (const base of searchDirs) { + if (!existsSync(base)) continue; + try { + const {execSync:e2} = require('child_process'); + const out = e2('find '+JSON.stringify(base)+' -maxdepth 8 -name package.json -path \"*/openclaw/package.json\" 2>/dev/null',{encoding:'utf8',timeout:5000}).trim(); + for (const line of out.split('\\n')) { + if (!line) continue; + try { if (JSON.parse(readFileSync(line,'utf8')).name==='openclaw') { console.log(dirname(line)); process.exit(0); } } catch {} + } + } catch {} + } + process.exit(1); + " 2>/dev/null +} + +patch_dist_js() { + local oc_dir + oc_dir="$(_resolve_openclaw_dir "${1:-}")" || { + warn "[Part 1] ๆ‰พไธๅˆฐ OpenClaw ๅฎ‰่ฃ…็›ฎๅฝ•๏ผŒ่ทณ่ฟ‡ dist patch" + return 0 + } + + local dist_dir="$oc_dir/dist" + [[ -d "$dist_dir" ]] || { warn "[Part 1] dist ็›ฎๅฝ•ไธๅญ˜ๅœจ: $dist_dir๏ผŒ่ทณ่ฟ‡"; return 0; } + + local version + version=$(grep -oP '"version"\s*:\s*"\K[^"]+' "$oc_dir/package.json" 2>/dev/null || echo "unknown") + info "[Part 1] OpenClaw ็‰ˆๆœฌ: $version" + + # ็‰ˆๆœฌ้—จๆŽง๏ผšไป… 2026.4.23 + if [[ ! "$version" =~ ^2026\.4\.23($|[-\.]) ]]; then + ok "[Part 1] ็‰ˆๆœฌ $version ไธ้œ€่ฆ schema patch๏ผŒ่ทณ่ฟ‡" + return 0 + fi + + # ็ฒพ็กฎๅฎšไฝ๏ผšhooks zod schema ็š„ๅ”ฏไธ€็‰นๅพ + local -a candidates + mapfile -t candidates < <( + grep -rl 'allowPromptInjection' "$dist_dir" --include='*.js' 2>/dev/null | while read -r _f; do + if perl -0777 -ne 'exit(0) if /allowPromptInjection\s*:\s*[a-zA-Z_\$][a-zA-Z0-9_\$]*\s*\.\s*boolean\s*\(\s*\)\s*\.\s*optional\s*\(\s*\)\s*[,\s]*\}\s*\)\s*\.\s*strict\s*\(\s*\)/; exit(1)' "$_f" 2>/dev/null; then + echo "$_f" + fi + done + ) + + if [[ ${#candidates[@]} -eq 0 ]]; then + warn "[Part 1] ๆœชๆ‰พๅˆฐ hooks zod schema ็›ฎๆ ‡ๆ–‡ไปถ๏ผŒ่ทณ่ฟ‡" + return 0 + elif [[ ${#candidates[@]} -gt 1 ]]; then + warn "[Part 1] ๅ‘็Žฐ ${#candidates[@]} ไธชๅŒน้…ๆ–‡ไปถ๏ผˆ้ข„ๆœŸ 1๏ผ‰๏ผŒๅฎ‰ๅ…จ่ตท่ง่ทณ่ฟ‡" + return 0 + fi + + local target="${candidates[0]}" + local relpath="${target#$dist_dir/}" + debug "[Part 1] ็›ฎๆ ‡: $relpath" + + # ๅน‚็ญ‰๏ผš็›ฎๆ ‡ๆ–‡ไปถไธญๅทฒๅŒ…ๅซ allowConversationAccess โ†’ ่ทณ่ฟ‡ + if grep -q 'allowConversationAccess' "$target" 2>/dev/null; then + ok "[Part 1] allowConversationAccess ๅทฒๅญ˜ๅœจไบŽ $relpath๏ผŒ่ทณ่ฟ‡" + return 0 + fi + + # ๅค‡ไปฝ + [[ -f "${target}.pre-aca-patch.bak" ]] || cp "$target" "${target}.pre-aca-patch.bak" + + # ๆณจๅ…ฅ๏ผš็”จ็ฒพ็กฎๅ˜้‡ๅๅŒน้…๏ผŒ้ฟๅ…่ดชๅฉชๅ›žๆบฏ + perl -0777 -i -pe ' + s/(allowPromptInjection\s*:\s*[a-zA-Z_\$][a-zA-Z0-9_\$]*\s*\.\s*boolean\s*\(\s*\)\s*\.\s*optional\s*\(\s*\))(\s*\}\s*\)\s*\.\s*strict\s*\(\s*\))/$1,allowConversationAccess:z.boolean().optional()$2/ + ' "$target" + + # ้ชŒ่ฏ + if grep -q 'allowConversationAccess' "$target" 2>/dev/null; then + ok "[Part 1] $relpath โ€” patch ๆˆๅŠŸ" + else + warn "[Part 1] patch ้ชŒ่ฏๅคฑ่ดฅ๏ผŒๆขๅคๅค‡ไปฝ" + cp "${target}.pre-aca-patch.bak" "$target" + return 1 + fi +} + +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• +# Part 2: ๅ†™ๅ…ฅ openclaw.json (ไธ้™็‰ˆๆœฌ๏ผŒๅง‹็ปˆ็กฎไฟ้…็ฝฎๅญ˜ๅœจ) +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• + +patch_config_json() { + if [[ ! -f "$OPENCLAW_JSON" ]]; then + warn "[Part 2] openclaw.json ไธๅญ˜ๅœจ: $OPENCLAW_JSON๏ผŒ่ทณ่ฟ‡" + return 0 + fi + + # ๅน‚็ญ‰ๆฃ€ๆต‹ + local exists + exists=$(python3 -c " +import json +try: + with open('$OPENCLAW_JSON') as f: + cfg = json.load(f) + val = cfg.get('plugins',{}).get('entries',{}).get('$PLUGIN_ID',{}).get('hooks',{}).get('allowConversationAccess') + print('yes' if val is True else 'no') +except Exception: + print('no') +" 2>/dev/null || echo "no") + + if [[ "$exists" == "yes" ]]; then + ok "[Part 2] hooks.allowConversationAccess ๅทฒๅญ˜ๅœจ๏ผŒ่ทณ่ฟ‡" + return 0 + fi + + # ๅ†™ๅ…ฅ + python3 -c " +import json + +with open('$OPENCLAW_JSON') as f: + cfg = json.load(f) + +entry = cfg.setdefault('plugins', {}).setdefault('entries', {}).setdefault('$PLUGIN_ID', {}) +hooks = entry.setdefault('hooks', {}) +hooks['allowConversationAccess'] = True + +with open('$OPENCLAW_JSON', 'w') as f: + json.dump(cfg, f, indent=2, ensure_ascii=False) + f.write('\n') +" + ok "[Part 2] hooks.allowConversationAccess = true ๅทฒๅ†™ๅ…ฅ" +} + +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• +# ไธปๅ…ฅๅฃ +# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• + +info "โ”€โ”€ bugfix-20260423: allowConversationAccess โ”€โ”€" + +patch_dist_js "${1:-}" +patch_config_json + +echo "" +ok "bugfix-20260423 ๅฎŒๆˆ" diff --git a/scripts/test-tdai-memory.mjs b/scripts/test-tdai-memory.mjs deleted file mode 100644 index 24c77d5..0000000 --- a/scripts/test-tdai-memory.mjs +++ /dev/null @@ -1,319 +0,0 @@ -/** - * TDAI Memory Test Script - * - * ๆต‹่ฏ• TDAI ่ฎฐๅฟ†ๅŠŸ่ƒฝๆ˜ฏๅฆๆญฃๅธธๅทฅไฝœใ€‚ - * ๅฏๅŠจ Gateway๏ผŒ็„ถๅŽๅ‘้€ๆต‹่ฏ•่ฏทๆฑ‚้ชŒ่ฏ่ฎฐๅฟ†ๆ•่Žทๅ’Œๅฌๅ›žใ€‚ - */ - -import { spawn } from "node:child_process"; -import { createServer } from "node:http"; -import { dirname, join } from "node:path"; -import { fileURLToPath } from "node:url"; -import { EventEmitter } from "node:events"; -import { tmpdir } from "node:os"; -import { mkdir, rm, writeFile } from "node:fs/promises"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -const PROJECT_ROOT = join(__dirname, ".."); - -// ๆต‹่ฏ•้…็ฝฎ -const TEST_CONFIG = { - server: { - // Use 8422 by default to avoid clashing with the hermes sidecar on 8420. - port: Number(process.env.TDAI_TEST_PORT || 8422), - host: "127.0.0.1", - }, - data: { - baseDir: join(tmpdir(), "tdai-memory-test"), - }, - llm: { - baseUrl: process.env.TDAI_LLM_BASE_URL || "https://vdbteam.openai.azure.com/openai/v1", - apiKey: process.env.TDAI_LLM_API_KEY || "", - model: process.env.TDAI_LLM_MODEL || "gpt-5.2-chat", - maxTokens: 4096, - timeoutMs: 120000, - }, -}; - -// ็”จไบŽๆŽฅๆ”ถ Gateway ๅ›ž่ฐƒ็š„ๆœฌๅœฐๆœๅŠกๅ™จ -const callbackServer = createServer((req, res) => { - console.log(`[Callback] ${req.method} ${req.url}`); - res.writeHead(200, { "Content-Type": "application/json" }); - res.end(JSON.stringify({ status: "ok" })); -}); - -let gatewayProcess = null; - -async function ensureDataDir() { - await mkdir(TEST_CONFIG.data.baseDir, { recursive: true }); - const configPath = join(TEST_CONFIG.data.baseDir, "tdai-gateway.yaml"); - - const configContent = `# TDAI Gateway Test Config -server: - port: ${TEST_CONFIG.server.port} - host: ${TEST_CONFIG.server.host} -data: - baseDir: ${TEST_CONFIG.data.baseDir} -llm: - baseUrl: ${TEST_CONFIG.llm.baseUrl} - apiKey: ${TEST_CONFIG.llm.apiKey || "sk-test-key"} - model: ${TEST_CONFIG.llm.model} - maxTokens: ${TEST_CONFIG.llm.maxTokens} - timeoutMs: ${TEST_CONFIG.llm.timeoutMs} -memory: - capture: - enabled: true - l0l1RetentionDays: 7 - extraction: - enabled: true - enableDedup: true - recall: - enabled: true - maxResults: 5 - scoreThreshold: 0.1 - strategy: hybrid - pipeline: - everyNConversations: 3 - enableWarmup: true -`; - - await writeFile(configPath, configContent); - console.log(`[Config] Written to ${configPath}`); -} - -function startCallbackServer() { - const port = 18420; - return new Promise((resolve, reject) => { - callbackServer.listen(port, "127.0.0.1", () => { - console.log(`[CallbackServer] Started on port ${port}`); - resolve(port); - }); - callbackServer.on("error", reject); - }); -} - -function startGateway(callbackPort) { - return new Promise((resolve, reject) => { - const configPath = join(TEST_CONFIG.data.baseDir, "tdai-gateway.yaml"); - - // Launch the TS source directly via npx/tsx (no dist build step required). - const gatewayArgs = [ - "npx", - "--yes", - "tsx", - "src/gateway/server.ts", - ]; - - console.log(`[Gateway] Starting with args:`, gatewayArgs); - - const env = { - ...process.env, - TDAI_GATEWAY_PORT: String(TEST_CONFIG.server.port), - TDAI_GATEWAY_HOST: TEST_CONFIG.server.host, - TDAI_DATA_DIR: TEST_CONFIG.data.baseDir, - TDAI_LLM_BASE_URL: TEST_CONFIG.llm.baseUrl, - TDAI_LLM_API_KEY: TEST_CONFIG.llm.apiKey, - TDAI_LLM_MODEL: TEST_CONFIG.llm.model, - TDAI_CALLBACK_PORT: String(callbackPort), - }; - - gatewayProcess = spawn(gatewayArgs[0], gatewayArgs.slice(1), { - cwd: PROJECT_ROOT, - env, - stdio: ["ignore", "pipe", "pipe"], - }); - - gatewayProcess.stdout?.on("data", (data) => { - console.log(`[Gateway] ${data.toString().trim()}`); - }); - - gatewayProcess.stderr?.on("data", (data) => { - console.error(`[Gateway Error] ${data.toString().trim()}`); - }); - - let healthCheckCount = 0; - const maxHealthChecks = 30; - const checkInterval = 1000; - - const checkHealth = () => { - healthCheckCount++; - - const req = new Request(`http://${TEST_CONFIG.server.host}:${TEST_CONFIG.server.port}/health`); - fetch(req) - .then((res) => { - if (res.ok) { - resolve({ process: gatewayProcess, port: TEST_CONFIG.server.port }); - return; - } - }) - .catch(() => {}); - - if (healthCheckCount >= maxHealthChecks) { - reject(new Error("Gateway failed to start")); - } - - setTimeout(checkHealth, checkInterval); - }; - - setTimeout(checkHealth, checkInterval); - - gatewayProcess.on("error", (err) => { - reject(new Error(`Failed to start gateway: ${err.message}`)); - }); - }); -} - -async function testHealthCheck(gatewayPort) { - console.log("\n[Test] Health check..."); - try { - const res = await fetch(`http://127.0.0.1:${gatewayPort}/health`); - const data = await res.json(); - console.log(`[OK] Health check passed:`, JSON.stringify(data, null, 2)); - return true; - } catch (err) { - console.error(`[FAIL] Health check failed:`, err.message); - return false; - } -} - -async function testCapture(gatewayPort) { - console.log("\n[Test] Capture conversation..."); - try { - const res = await fetch(`http://127.0.0.1:${gatewayPort}/capture`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ - user_content: "ไฝ ๅฅฝ๏ผŒๆˆ‘ๆ˜ฏไธ€ไธชๆต‹่ฏ•็”จๆˆทใ€‚ๆˆ‘ๅ–œๆฌข็ผ–็จ‹ๅ’Œๅ’–ๅ•กใ€‚", - assistant_content: "ไฝ ๅฅฝ๏ผๅพˆ้ซ˜ๅ…ด่ฎค่ฏ†ไฝ ใ€‚็ผ–็จ‹ๅ’Œๅ’–ๅ•ก้ƒฝๆ˜ฏๅพˆๆฃ’็š„็ˆฑๅฅฝ๏ผ", - session_key: "test-session-001", - session_id: "test-session-id-001", - user_id: "test-user", - }), - }); - const data = await res.json(); - console.log(`[OK] Capture response:`, JSON.stringify(data, null, 2)); - return true; - } catch (err) { - console.error(`[FAIL] Capture failed:`, err.message); - return false; - } -} - -async function testRecall(gatewayPort) { - console.log("\n[Test] Recall memories..."); - try { - const res = await fetch(`http://127.0.0.1:${gatewayPort}/recall`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ - query: "็”จๆˆท็š„็ˆฑๅฅฝๆ˜ฏไป€ไนˆ๏ผŸ", - session_key: "test-session-001", - user_id: "test-user", - }), - }); - const data = await res.json(); - console.log(`[OK] Recall response:`, JSON.stringify(data, null, 2)); - return true; - } catch (err) { - console.error(`[FAIL] Recall failed:`, err.message); - return false; - } -} - -async function testSearchMemories(gatewayPort) { - console.log("\n[Test] Search memories..."); - try { - const res = await fetch(`http://127.0.0.1:${gatewayPort}/search/memories`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ - query: "็ผ–็จ‹", - limit: 5, - type: "episode", - }), - }); - const data = await res.json(); - console.log(`[OK] Search memories response:`, JSON.stringify(data, null, 2)); - return true; - } catch (err) { - console.error(`[FAIL] Search memories failed:`, err.message); - return false; - } -} - -async function stopGateway() { - if (gatewayProcess) { - console.log("\n[Cleanup] Stopping gateway..."); - gatewayProcess.kill("SIGTERM"); - gatewayProcess = null; - } - if (callbackServer.listening) { - callbackServer.close(); - } -} - -async function cleanup() { - await stopGateway(); - // ไธๅˆ ้™คๆต‹่ฏ•ๆ•ฐๆฎ็›ฎๅฝ•๏ผŒไฟ็•™็”จไบŽ่ฐƒ่ฏ• - console.log("[Cleanup] Test data preserved at:", TEST_CONFIG.data.baseDir); -} - -async function main() { - console.log("=".repeat(60)); - console.log("TDAI Memory Test"); - console.log("=".repeat(60)); - - const results = { - health: false, - capture: false, - recall: false, - search: false, - }; - - try { - // 1. ็กฎไฟๆ•ฐๆฎ็›ฎๅฝ•ๅญ˜ๅœจ - await ensureDataDir(); - - // 2. ๅฏๅŠจๅ›ž่ฐƒๆœๅŠกๅ™จ - const callbackPort = await startCallbackServer(); - - // 3. ๅฏๅŠจ Gateway - const { port: gatewayPort } = await startGateway(callbackPort); - - // ็ญ‰ๅพ… Gateway ๅฎŒๅ…จๅฏๅŠจ - await new Promise((resolve) => setTimeout(resolve, 2000)); - - // 4. ่ฟ่กŒๆต‹่ฏ• - results.health = await testHealthCheck(gatewayPort); - if (results.health) { - // ็ญ‰ๅพ…ไธ€ไธ‹่ฎฉ Gateway ๅˆๅง‹ๅŒ– - await new Promise((resolve) => setTimeout(resolve, 1000)); - results.capture = await testCapture(gatewayPort); - if (results.capture) { - // ็ญ‰ๅพ…ๆๅ–ๅฎŒๆˆ - await new Promise((resolve) => setTimeout(resolve, 3000)); - results.recall = await testRecall(gatewayPort); - results.search = await testSearchMemories(gatewayPort); - } - } - - } catch (err) { - console.error("\n[Test] Error:", err.message); - } finally { - await cleanup(); - - // ๆ‰“ๅฐๆต‹่ฏ•็ป“ๆžœ - console.log("\n" + "=".repeat(60)); - console.log("Test Results:"); - console.log("=".repeat(60)); - console.log(` Health Check: ${results.health ? "PASS" : "FAIL"}`); - console.log(` Capture: ${results.capture ? "PASS" : "FAIL"}`); - console.log(` Recall: ${results.recall ? "PASS" : "FAIL"}`); - console.log(` Search: ${results.search ? "PASS" : "FAIL"}`); - - const passed = Object.values(results).filter(Boolean).length; - console.log(`\nTotal: ${passed}/4 passed`); - } -} - -main().catch(console.error); diff --git a/src/cli/README.md b/src/cli/README.md new file mode 100644 index 0000000..f314e6e --- /dev/null +++ b/src/cli/README.md @@ -0,0 +1,174 @@ +# memory-tdai CLI + +`openclaw memory-tdai` ๅ‘ฝไปค็ฉบ้—ด๏ผŒๆไพ›็ฆป็บฟๆ•ฐๆฎ็ฎก็†ๅทฅๅ…ทใ€‚ + +## seed โ€” ๅฏผๅ…ฅๅކๅฒๅฏน่ฏๆ•ฐๆฎ + +ๅฐ†ๅކๅฒๅฏน่ฏ JSON ๆ–‡ไปถๅฏผๅ…ฅๅˆฐ่ฎฐๅฟ†็ฎก็บฟไธญ๏ผŒๅฎŒๆ•ดๆ‰ง่กŒ L0โ†’L1โ†’L2โ†’L3 ๆต็จ‹ใ€‚้€‚็”จไบŽ๏ผš + +- ๅฐ†ๅทฒๆœ‰ๅฏน่ฏๆ•ฐๆฎ็Œๅ…ฅ่ฎฐๅฟ†็ณป็ปŸ +- ๆ‰น้‡ๆต‹่ฏ•่ฎฐๅฟ†ๆๅ–ๆ•ˆๆžœ +- ่ฟ็งป/ๆขๅค่ฎฐๅฟ†ๆ•ฐๆฎ + +### ็”จๆณ• + +```bash +openclaw memory-tdai seed --input [options] +``` + +### ๅ‚ๆ•ฐ + +| ๅ‚ๆ•ฐ | ๅฟ…ๅกซ | ่ฏดๆ˜Ž | +|------|------|------| +| `--input ` | โœ… | ่พ“ๅ…ฅ JSON ๆ–‡ไปถ่ทฏๅพ„ | +| `--output-dir ` | โ€” | ่พ“ๅ‡บ็›ฎๅฝ•๏ผˆ้ป˜่ฎค่‡ชๅŠจ็”Ÿๆˆๅธฆๆ—ถ้—ดๆˆณ็š„็›ฎๅฝ•๏ผ‰ | +| `--session-key ` | โ€” | ๅ›ž้€€ session key๏ผˆๅฝ“่พ“ๅ…ฅๆ•ฐๆฎ็ผบๅฐ‘ๆ—ถไฝฟ็”จ๏ผ‰ | +| `--config ` | โ€” | ้…็ฝฎ่ฆ†็›–ๆ–‡ไปถ๏ผˆJSON๏ผŒไธŽ openclaw.json ๆ’ไปถ้…็ฝฎๆทฑๅบฆๅˆๅนถ๏ผ‰ | +| `--strict-round-role` | โ€” | ไธฅๆ ผๆ ก้ชŒๆฏ่ฝฎๅฏน่ฏๅฟ…้กปๅŒ…ๅซ user ๅ’Œ assistant ๆถˆๆฏ | +| `--yes` | โ€” | ่ทณ่ฟ‡ไบคไบ’็กฎ่ฎค๏ผˆๅฆ‚ๆ—ถ้—ดๆˆณ่‡ชๅŠจๅกซๅ……็กฎ่ฎค๏ผ‰ | + +### ็คบไพ‹ + +```bash +# ๅŸบๆœฌ็”จๆณ• +openclaw memory-tdai seed --input conversations.json + +# ๆŒ‡ๅฎš่พ“ๅ‡บ็›ฎๅฝ• +openclaw memory-tdai seed --input data.json --output-dir ./seed-output + +# ไฝฟ็”จ่‡ชๅฎšไน‰้…็ฝฎ่ฆ†็›–๏ผˆๅฆ‚่ฐƒๆ•ด pipeline ๅ‚ๆ•ฐ๏ผ‰ +openclaw memory-tdai seed --input data.json --config seed-config.json + +# ่ทณ่ฟ‡ๆ‰€ๆœ‰็กฎ่ฎค +openclaw memory-tdai seed --input data.json --yes + +# ไธฅๆ ผๆจกๅผ + ่‡ชๅฎšไน‰้…็ฝฎ +openclaw memory-tdai seed --input data.json --config seed-config.json --strict-round-role --yes +``` + +### ่พ“ๅ…ฅๆ–‡ไปถๆ ผๅผ + +ๆ”ฏๆŒไธค็ง JSON ๆ ผๅผ๏ผš + +#### Format A๏ผšๅฏน่ฑกๅŒ…่ฃ… + +```json +{ + "sessions": [ + { + "sessionKey": "user-alice", + "sessionId": "conv-001", + "conversations": [ + [ + { "role": "user", "content": "ไฝ ๅฅฝ", "timestamp": 1711929600000 }, + { "role": "assistant", "content": "ไฝ ๅฅฝ๏ผๆœ‰ไป€ไนˆๅฏไปฅๅธฎไฝ ๏ผŸ", "timestamp": 1711929601000 } + ], + [ + { "role": "user", "content": "ไปŠๅคฉๅคฉๆฐ”ๆ€Žไนˆๆ ท๏ผŸ" }, + { "role": "assistant", "content": "ไปŠๅคฉๆ™ดๅคฉ๏ผŒ้€‚ๅˆๅ‡บ้—จใ€‚" } + ] + ] + } + ] +} +``` + +#### Format B๏ผš้กถๅฑ‚ๆ•ฐ็ป„ + +```json +[ + { + "sessionKey": "user-alice", + "conversations": [ + [ + { "role": "user", "content": "ไฝ ๅฅฝ" }, + { "role": "assistant", "content": "ไฝ ๅฅฝ๏ผ" } + ] + ] + } +] +``` + +#### ๅญ—ๆฎต่ฏดๆ˜Ž + +| ๅญ—ๆฎต | ็ฑปๅž‹ | ๅฟ…ๅกซ | ่ฏดๆ˜Ž | +|------|------|------|------| +| `sessionKey` | string | โœ… | Session ๆ ‡่ฏ†๏ผˆๅฆ‚็”จๆˆท IDใ€้ข‘้“ๅ๏ผ‰ | +| `sessionId` | string | โ€” | ไผš่ฏๅฎžไพ‹ ID๏ผˆๅŒไธ€ sessionKey ไธ‹ๅฏๆœ‰ๅคšไธช sessionId๏ผ‰ | +| `conversations` | message[][] | โœ… | ๅฏน่ฏ่ฝฎๆฌกๆ•ฐ็ป„๏ผŒๆฏไธช่ฝฎๆฌกๆ˜ฏไธ€็ป„ๆถˆๆฏ | +| `role` | string | โœ… | ๆถˆๆฏ่ง’่‰ฒ๏ผš`user` ๆˆ– `assistant` | +| `content` | string | โœ… | ๆถˆๆฏๅ†…ๅฎน | +| `timestamp` | number \| string | โ€” | ๆ—ถ้—ดๆˆณ๏ผšepoch ๆฏซ็ง’ๆˆ– ISO 8601 ๅญ—็ฌฆไธฒใ€‚็ผบๅคฑๆ—ถ seed ไผšๆ็คบ่‡ชๅŠจๅกซๅ…… | + +### ้…็ฝฎ่ฆ†็›– + +`--config` ๆŽฅๅ—ไธ€ไธช JSON ๆ–‡ไปถ๏ผŒไธŽ `openclaw.json` ไธญ็š„ๆ’ไปถ้…็ฝฎ**ไธค็บงๆทฑๅบฆๅˆๅนถ**๏ผš + +- ้กถๅฑ‚ key ๅฆ‚ๆžœไธค่พน้ƒฝๆ˜ฏๅฏน่ฑก โ†’ ๆต…ๅˆๅนถ๏ผˆไฟ็•™ base ไธญๆœช่ฆ†็›–็š„ๅญ—ๆฎต๏ผ‰ +- ๅ…ถไป–็ฑปๅž‹ โ†’ ็›ดๆŽฅ่ฆ†็›– + +ๅธธ่งๅœบๆ™ฏ๏ผšseed ๆ—ถไฝฟ็”จๆ›ดๆฟ€่ฟ›็š„ pipeline ๅ‚ๆ•ฐไปฅๅŠ ้€Ÿๅค„็†๏ผš + +```json +{ + "pipeline": { + "everyNConversations": 3, + "enableWarmup": false, + "l1IdleTimeoutSeconds": 2, + "l2DelayAfterL1Seconds": 1, + "l2MinIntervalSeconds": 1, + "l2MaxIntervalSeconds": 10 + } +} +``` + +ๅฆ‚ๆžœ้œ€่ฆ seed ๅˆฐ็‹ฌ็ซ‹็š„ TCVDB ๆ•ฐๆฎๅบ“๏ผš + +```json +{ + "storeBackend": "tcvdb", + "tcvdb": { + "database": "my_seed_test_db" + }, + "pipeline": { + "everyNConversations": 3, + "enableWarmup": false, + "l1IdleTimeoutSeconds": 2 + } +} +``` + +### ่พ“ๅ‡บ็›ฎๅฝ•็ป“ๆž„ + +``` +/ +โ”œโ”€โ”€ conversations/ โ€” L0 JSONL ๆ–‡ไปถ +โ”œโ”€โ”€ records/ โ€” L1 JSONL ๆ–‡ไปถ +โ”œโ”€โ”€ scene_blocks/ โ€” L2 ๅœบๆ™ฏๅ— +โ”œโ”€โ”€ vectors.db โ€” SQLite ๅ‘้‡ๆ•ฐๆฎๅบ“๏ผˆไป… sqlite ๅŽ็ซฏ๏ผ‰ +โ”œโ”€โ”€ .metadata/ +โ”‚ โ”œโ”€โ”€ manifest.json โ€” ๅ…ƒๆ•ฐๆฎ๏ผˆstore ็ป‘ๅฎš + seed ่ฟ่กŒ่ฎฐๅฝ•๏ผ‰ +โ”‚ โ””โ”€โ”€ checkpoint.json โ€” ็ฎก็บฟ่ฟ›ๅบฆ +โ””โ”€โ”€ .backup/ โ€” ๆปšๅŠจๅค‡ไปฝ +``` + +Seed ๅฎŒๆˆๅŽ๏ผŒ`manifest.json` ไผš่ฎฐๅฝ•ๆœฌๆฌก่ฟ่กŒไฟกๆฏ๏ผš + +```json +{ + "version": 1, + "createdAt": "2026-04-01T22:00:00.000Z", + "store": { + "type": "sqlite", + "sqlite": { "path": "vectors.db" } + }, + "seed": { + "inputFile": "conversations.json", + "sessions": 3, + "rounds": 42, + "messages": 128, + "startedAt": "2026-04-01T22:00:00.000Z", + "completedAt": "2026-04-01T22:05:30.000Z" + } +} +``` diff --git a/src/cli/commands/seed.ts b/src/cli/commands/seed.ts new file mode 100644 index 0000000..611af05 --- /dev/null +++ b/src/cli/commands/seed.ts @@ -0,0 +1,281 @@ +/** + * `openclaw memory-tdai seed` command definition. + * + * Responsibilities: + * - Define CLI parameters and help text + * - Interactive confirmation for timestamp auto-fill + * - Output directory resolution and checkpoint detection + * - Delegate to seed-runtime for actual execution + */ + +import fs from "node:fs"; +import path from "node:path"; +import readline from "node:readline"; +import type { Command } from "commander"; +import type { SeedCliContext } from "../index.ts"; +import type { SeedCommandOptions } from "../../core/seed/types.js"; +import { loadAndValidateInput, fillTimestamps, SeedValidationError } from "../../core/seed/input.js"; +import { executeSeed } from "../../core/seed/seed-runtime.js"; + +const TAG = "[memory-tdai] [seed-cmd]"; + +/** + * Register the `seed` subcommand under the memory-tdai CLI namespace. + */ +export function registerSeedCommand(parent: Command, ctx: SeedCliContext): void { + parent + .command("seed") + .description("Seed historical conversation data into the memory pipeline (L0 โ†’ L1)") + .requiredOption("--input ", "Path to input JSON file") + .option("--output-dir ", "Output directory for pipeline data (default: auto-generated)") + .option("--session-key ", "Fallback session key when input lacks one") + .option("--config ", "Path to memory-tdai config override file (JSON, deep-merged on top of current plugin config)") + .option("--strict-round-role", "Require each round to have both user and assistant messages", false) + .option("--yes", "Skip interactive confirmations (e.g. timestamp auto-fill)", false) + .addHelpText("after", ` +Examples: + openclaw memory-tdai seed --input conversations.json + openclaw memory-tdai seed --input data.json --output-dir ./seed-output --strict-round-role + openclaw memory-tdai seed --input data.json --config ./seed-config.json + openclaw memory-tdai seed --input data.json --yes +`) + .action(async (rawOpts: Record) => { + const opts: SeedCommandOptions = { + input: rawOpts.input as string, + outputDir: rawOpts.outputDir as string | undefined, + sessionKey: rawOpts.sessionKey as string | undefined, + strictRoundRole: rawOpts.strictRoundRole === true, + yes: rawOpts.yes === true, + configFile: rawOpts.config as string | undefined, + }; + + await runSeedCommand(opts, ctx); + }); +} + +// ============================ +// Command handler +// ============================ + +async function runSeedCommand(opts: SeedCommandOptions, ctx: SeedCliContext): Promise { + const { logger } = ctx; + + logger.info(`${TAG} Starting seed command...`); + logger.info(`${TAG} input: ${opts.input}`); + logger.info(`${TAG} outputDir: ${opts.outputDir ?? "(auto)"}`); + logger.info(`${TAG} sessionKey: ${opts.sessionKey ?? "(from input)"}`); + logger.info(`${TAG} config: ${opts.configFile ?? "(default)"}`); + logger.info(`${TAG} strict: ${opts.strictRoundRole}`); + logger.info(`${TAG} yes: ${opts.yes}`); + + // 0. Load config override file and deep-merge with base plugin config + const mergedPluginConfig = loadAndMergePluginConfig( + ctx.pluginConfig as Record | undefined, + opts.configFile, + logger, + ); + + // 1. Load and validate input + let loadResult; + try { + loadResult = loadAndValidateInput(opts); + } catch (err) { + if (err instanceof SeedValidationError) { + console.error(`\nโŒ ${err.message}\n`); + process.exit(1); + } + throw err; + } + + const { input, needsTimestampConfirmation } = loadResult; + + console.log( + `\n๐Ÿ“ฅ Input loaded: ${input.sessions.length} session(s), ` + + `${input.totalRounds} round(s), ${input.totalMessages} message(s)` + + `${input.hasTimestamps ? "" : " (no timestamps)"}`, + ); + + // 2. Timestamp confirmation (if all messages lack timestamps) + if (needsTimestampConfirmation) { + if (opts.yes) { + console.log(" Timestamps missing โ€” auto-filling with current time (--yes)"); + fillTimestamps(input); + } else { + const confirmed = await askConfirmation( + "All messages have no timestamp. Use current time for each conversation round? [y/N] ", + ); + if (!confirmed) { + console.log("Aborted."); + process.exit(0); + } + fillTimestamps(input); + } + } + + // 3. Resolve output directory + const outputDir = resolveOutputDir(opts.outputDir, ctx.stateDir); + logger.info(`${TAG} Output directory: ${outputDir}`); + + // 4. Check for existing directory / checkpoint (resume detection) + if (fs.existsSync(outputDir)) { + const checkpointPath = path.join(outputDir, ".metadata", "checkpoint.json"); + if (fs.existsSync(checkpointPath)) { + // Checkpoint exists โ†’ resume scenario โ†’ P0 not implemented + console.error( + "\nโŒ Resume from checkpoint is not implemented in P0 yet. " + + "Please use a new output directory.\n" + + ` Existing: ${outputDir}\n`, + ); + process.exit(1); + } + + // Directory exists but no checkpoint โ†’ might have stale data + const entries = fs.readdirSync(outputDir); + if (entries.length > 0) { + console.error( + `\nโŒ Output directory already exists and is not empty: ${outputDir}\n` + + " Please use a new directory or clean the existing one.\n", + ); + process.exit(1); + } + } + + // 5. Execute seed pipeline + console.log(`\n๐Ÿ”ง Output: ${outputDir}`); + console.log(`โ–ถ๏ธ Starting seed pipeline...\n`); + + const summary = await executeSeed(input, { + outputDir, + openclawConfig: ctx.config, + pluginConfig: mergedPluginConfig, + inputFile: opts.input, + logger, + onProgress: (progress) => { + const pct = ((progress.currentRound / progress.totalRounds) * 100).toFixed(0); + process.stdout.write( + `\r [${progress.currentRound}/${progress.totalRounds}] ${pct}% ` + + `session=${progress.sessionKey} stage=${progress.stage} `, + ); + }, + }); + + // 6. Print summary + console.log("\n"); + console.log("โ•”โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•—"); + console.log("โ•‘ Seed Summary โ•‘"); + console.log("โ• โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•ฃ"); + console.log(`โ•‘ Sessions: ${String(summary.sessionsProcessed).padStart(11)} โ•‘`); + console.log(`โ•‘ Rounds: ${String(summary.roundsProcessed).padStart(11)} โ•‘`); + console.log(`โ•‘ Messages: ${String(summary.messagesProcessed).padStart(11)} โ•‘`); + console.log(`โ•‘ L0 recorded: ${String(summary.l0RecordedCount).padStart(11)} โ•‘`); + console.log(`โ•‘ Duration: ${(summary.durationMs / 1000).toFixed(1).padStart(10)}s โ•‘`); + console.log("โ•šโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•"); + console.log(`\n๐Ÿ“ Output: ${summary.outputDir}\n`); +} + +// ============================ +// Helpers +// ============================ + +/** + * Load an optional config override file and deep-merge it on top of the + * base plugin config from openclaw.json. + * + * Returns the merged config, or the base config unchanged if no override + * file is specified. + */ +function loadAndMergePluginConfig( + base: Record | undefined, + configFile: string | undefined, + logger: { info: (msg: string) => void }, +): Record | undefined { + if (!configFile) return base; + + const resolved = path.resolve(configFile); + if (!fs.existsSync(resolved)) { + console.error(`\nโŒ Config override file not found: ${resolved}\n`); + process.exit(1); + } + + let override: Record; + try { + const raw = fs.readFileSync(resolved, "utf-8"); + override = JSON.parse(raw) as Record; + } catch (err) { + console.error( + `\nโŒ Failed to parse config override file: ${resolved}\n` + + ` ${err instanceof Error ? err.message : String(err)}\n`, + ); + process.exit(1); + } + + if (typeof override !== "object" || override === null || Array.isArray(override)) { + console.error(`\nโŒ Config override file must contain a JSON object: ${resolved}\n`); + process.exit(1); + } + + logger.info(`${TAG} Config override loaded from: ${resolved}`); + return deepMerge(base ?? {}, override); +} + +/** + * Simple two-level deep merge: for each key in `override`, if both base + * and override values are plain objects, merge them; otherwise override wins. + * + * This is sufficient for the memory-tdai config shape: + * { capture: {...}, extraction: {...}, pipeline: {...}, ... } + */ +function deepMerge( + base: Record, + override: Record, +): Record { + const result: Record = { ...base }; + + for (const key of Object.keys(override)) { + const baseVal = base[key]; + const overVal = override[key]; + + if (isPlainObject(baseVal) && isPlainObject(overVal)) { + result[key] = { ...baseVal, ...overVal }; + } else { + result[key] = overVal; + } + } + + return result; +} + +function isPlainObject(v: unknown): v is Record { + return v !== null && typeof v === "object" && !Array.isArray(v); +} + +function resolveOutputDir(explicit: string | undefined, stateDir: string): string { + if (explicit) return path.resolve(explicit); + + // Default: /memory-tdai-seed- + const now = new Date(); + const pad = (n: number) => String(n).padStart(2, "0"); + const ts = + `${now.getFullYear()}${pad(now.getMonth() + 1)}${pad(now.getDate())}-` + + `${pad(now.getHours())}${pad(now.getMinutes())}${pad(now.getSeconds())}`; + + return path.join(stateDir, `memory-tdai-seed-${ts}`); +} + +function askConfirmation(prompt: string): Promise { + return new Promise((resolve) => { + // Delay slightly to let async plugin logs flush before showing the prompt. + // Without this, the prompt gets buried under registration logs. + setTimeout(() => { + console.log("\n" + "โ”€".repeat(60)); + const rl = readline.createInterface({ + input: process.stdin, + output: process.stdout, + }); + rl.question(`โš ๏ธ ${prompt}`, (answer) => { + rl.close(); + resolve(answer.trim().toLowerCase() === "y"); + }); + }, 2000); + }); +} diff --git a/src/cli/index.ts b/src/cli/index.ts new file mode 100644 index 0000000..6015d68 --- /dev/null +++ b/src/cli/index.ts @@ -0,0 +1,60 @@ +/** + * memory-tdai CLI entry point. + * + * Registers the `memory-tdai` namespace under the OpenClaw CLI and + * wires up all subcommands (currently: `seed`). + * + * Integration path: + * index.ts โ†’ api.registerCli() โ†’ registerMemoryTdaiCli() โ†’ registerSeedCommand() + */ + +import type { Command } from "commander"; +import { registerSeedCommand } from "./commands/seed.js"; + +// ============================ +// Context type +// ============================ + +/** + * Minimal context needed by seed CLI commands. + * + * Derived from OpenClawPluginCliContext but scoped to what seed actually needs, + * avoiding a hard dependency on the full plugin CLI context type. + */ +export interface SeedCliContext { + /** OpenClaw config (for LLM calls in L1 extraction). */ + config: unknown; + /** Raw plugin config (same shape as api.pluginConfig). */ + pluginConfig: unknown; + /** State directory root (e.g. ~/.openclaw). */ + stateDir: string; + /** Logger instance. */ + logger: { + debug?: (message: string) => void; + info: (message: string) => void; + warn: (message: string) => void; + error: (message: string) => void; + }; +} + +// ============================ +// Top-level registration +// ============================ + +/** + * Register all memory-tdai CLI subcommands under the given Commander program. + * + * This function is called by the plugin's `api.registerCli()` registrar. + * It creates the `memory-tdai` namespace and delegates to individual + * command registrars. + * + * @param program - The `memory-tdai` Commander command (already created by the registrar) + * @param ctx - CLI context with config, state dir, and logger + */ +export function registerMemoryTdaiCli(program: Command, ctx: SeedCliContext): void { + // Register subcommands + registerSeedCommand(program, ctx); + + // Future: registerQueryCommand(program, ctx); + // Future: registerStatsCommand(program, ctx); +} diff --git a/src/config.ts b/src/config.ts index ca663b7..30a68e3 100644 --- a/src/config.ts +++ b/src/config.ts @@ -198,7 +198,14 @@ export interface StandaloneLLMOverrideConfig { export interface OffloadConfig { /** Enable context offload (default: false) */ enabled: boolean; - /** LLM model for offload tasks, format: "provider/model-id" */ + /** + * LLM execution mode for L1/L1.5/L2 tasks. + * - "local": call LLM directly via AI SDK (uses offload.model or main agent model) + * - "backend": route through remote backend service (requires backendUrl) + * Default: "local" (auto-detects based on backendUrl presence for backward compat) + */ + mode: "local" | "backend"; + /** LLM model for offload tasks, format: "provider/model-id". Falls back to agents.defaults.model when omitted. */ model?: string; /** LLM temperature (default: 0.2) */ temperature: number; @@ -419,8 +426,15 @@ export function parseConfig(raw: Record | undefined): MemoryTda // --- Offload --- const offloadGroup = obj(c, "offload"); + const offloadMode: "local" | "backend" = (() => { + const raw = optStr(offloadGroup, "mode"); + if (raw === "local" || raw === "backend") return raw; + return optStr(offloadGroup, "backendUrl") ? "backend" : "local"; + })(); + const offload: OffloadConfig = { enabled: bool(offloadGroup, "enabled") ?? false, + mode: offloadMode, model: optStr(offloadGroup, "model"), temperature: num(offloadGroup, "temperature") ?? 0.2, forceTriggerThreshold: num(offloadGroup, "forceTriggerThreshold") ?? 4, @@ -434,7 +448,7 @@ export function parseConfig(raw: Record | undefined): MemoryTda mmdMaxTokenRatio: num(offloadGroup, "mmdMaxTokenRatio") ?? 0.2, backendUrl: optStr(offloadGroup, "backendUrl"), backendApiKey: optStr(offloadGroup, "backendApiKey"), - backendTimeoutMs: num(offloadGroup, "backendTimeoutMs") ?? 10000, + backendTimeoutMs: num(offloadGroup, "backendTimeoutMs") ?? 120000, offloadRetentionDays: normalizeOffloadRetentionDays(num(offloadGroup, "offloadRetentionDays") ?? 0), logMaxSizeMb: num(offloadGroup, "logMaxSizeMb") ?? 50, userId: optStr(offloadGroup, "userId"), diff --git a/src/conversation/l0-recorder.ts b/src/conversation/l0-recorder.ts deleted file mode 100644 index 6eb9274..0000000 --- a/src/conversation/l0-recorder.ts +++ /dev/null @@ -1,582 +0,0 @@ -/** - * L0 Conversation Recorder: records raw conversation messages to local JSONL files. - * - * Triggered from agent_end hook. Receives the conversation messages directly from - * the hook context (no file I/O needed), sanitizes them, filters out noise, and - * writes to ~/.openclaw/memory-tdai/conversations/YYYY-MM-DD.jsonl - * - * Design decisions: - * - Uses JSONL format (**one message per line** โ€” flat, easy to grep/stream) - * - One file per day (all sessions merged into the same daily file) - * - sessionKey is stored as a field in each JSONL line, not in the filename - * - Independent from system session files โ€” format fully controlled by plugin - * - Messages are sanitized to remove injected tags (prevent feedback loops) - * - Short/long/command messages are filtered out - */ - -import fs from "node:fs/promises"; -import path from "node:path"; -import crypto from "node:crypto"; -import { sanitizeText, stripCodeBlocks, shouldCaptureL0 } from "../utils/sanitize.js"; - -// ============================ -// Types -// ============================ - -export interface ConversationMessage { - /** Unique message ID (used by L1 prompt for source_message_ids tracking) */ - id: string; - role: "user" | "assistant"; - content: string; - timestamp: number; // epoch ms -} - -/** - * Generate a short unique message ID. - */ -function generateMessageId(): string { - return `msg_${Date.now()}_${crypto.randomBytes(3).toString("hex")}`; -} - -/** - * New flat format: one message per JSONL line. - */ -export interface L0MessageRecord { - sessionKey: string; - sessionId: string; - recordedAt: string; // ISO timestamp - id: string; - role: "user" | "assistant"; - content: string; - timestamp: number; // epoch ms -} - -/** - * A group of conversation messages (used by downstream consumers). - * Each L0ConversationRecord represents one or more messages from the same recording event. - */ -export interface L0ConversationRecord { - sessionKey: string; - sessionId: string; - recordedAt: string; // ISO timestamp - messageCount: number; - messages: ConversationMessage[]; -} - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -const TAG = "[memory-tdai][l0]"; - -// ============================ -// Core function -// ============================ - -/** - * Record a conversation round to the L0 JSONL file. - * - * Only records **incremental** messages (new since the last capture). - * Uses `afterTimestamp` as the primary filter to skip already-captured history. - * - * @param sessionKey - The session key for this conversation - * @param rawMessages - Raw messages from the agent_end hook context (full session history) - * @param baseDir - Base data directory (~/.openclaw/memory-tdai/) - * @param logger - Optional logger - * @param originalUserText - Clean original user prompt (pre-prependContext) - * @param afterTimestamp - Epoch ms cursor: only messages with timestamp > this are new. - * Pass 0 or omit for the first capture of a session. - * @returns Filtered messages (for L1 to use directly), or empty array if nothing worth recording - */ -export async function recordConversation(params: { - sessionKey: string; - sessionId?: string; - rawMessages: unknown[]; - baseDir: string; - logger?: Logger; - /** Clean original user prompt (pre-prependContext) */ - originalUserText?: string; - /** Epoch ms cursor: only process messages with timestamp strictly greater than this. */ - afterTimestamp?: number; - /** - * Number of messages in the session at before_prompt_build time. - * Used to locate the exact user message that originalUserText corresponds to: - * rawMessages[originalUserMessageCount] is the user message appended by the framework - * AFTER before_prompt_build, i.e. the one whose content was polluted by prependContext. - */ - originalUserMessageCount?: number; -}): Promise { - const { sessionKey, sessionId, rawMessages, baseDir, logger, originalUserText, afterTimestamp, originalUserMessageCount } = params; - - // Step 1: Position slice + extract user/assistant messages. - // - // Dual protection against duplicate capture: - // Layer 1 (position slice): Use originalUserMessageCount (cached at before_prompt_build) - // to slice rawMessages โ€” only keep messages added AFTER the prompt build, i.e. this - // turn's new messages. This is immune to timestamp drift after gateway restarts. - // Layer 2 (timestamp cursor): The existing afterTimestamp filter below acts as a fallback - // when the position slice is unavailable (cache expired, process restart, etc.). - const usePositionSlice = originalUserMessageCount != null && originalUserMessageCount > 0 - && originalUserMessageCount <= rawMessages.length; - const slicedMessages = usePositionSlice - ? rawMessages.slice(originalUserMessageCount) - : rawMessages; - - const allExtracted = extractUserAssistantMessages(slicedMessages); - - if (usePositionSlice) { - logger?.debug?.( - `${TAG} Position slice: ${rawMessages.length} raw โ†’ ${slicedMessages.length} new (sliceStart=${originalUserMessageCount})`, - ); - } - - // Diagnostic: check whether the framework actually provides timestamp on raw messages. - // If all raw timestamps are missing, the timestamp cursor is effectively useless and - // position slice becomes the sole incremental mechanism. - if (slicedMessages.length > 0) { - const firstRaw = slicedMessages[0] as Record | undefined; - const rawTs = firstRaw?.timestamp; - const hasRawTs = typeof rawTs === "number"; - logger?.debug?.( - `${TAG} Raw message[0] timestamp probe: ${hasRawTs ? `present (${rawTs})` : `missing (type=${typeof rawTs}, value=${String(rawTs)})`}`, - ); - } - - logger?.debug?.(`${TAG} Extracted ${allExtracted.length} user/assistant messages from ${slicedMessages.length} total`); - - // Step 1.5: Incremental filter โ€” only keep messages newer than the cursor. - // - // Uses strict greater-than (>) which is safe because: - // - The cursor is set to max(timestamps) of the LAST recorded batch. - // - The next agent turn's messages will have timestamps strictly greater than - // the previous turn (there's at least one LLM API call between turns, which - // takes hundreds of milliseconds minimum โ€” no same-millisecond collision). - // - All messages within a single turn are captured together as one batch, - // so even if multiple messages share the same timestamp, they are either - // all included (new batch) or all excluded (already captured). - // - If a message lacks a timestamp field, extractUserAssistantMessages() - // assigns Date.now() at extraction time, which is always > previous cursor. - const cursor = afterTimestamp ?? 0; - const extracted = cursor !== 0 - ? allExtracted.filter((m) => m.timestamp > cursor) - : allExtracted; - - if (extracted.length > 0) { - const first = extracted[0]; - logger?.debug?.( - `${TAG} First captured message: role=${first.role}, ts=${first.timestamp}, ` + - `date=${new Date(first.timestamp).toISOString()}, content=${first.content.slice(0, 80)}${first.content.length > 80 ? "โ€ฆ" : ""}`, - ); - } - - if (cursor > 0) { - logger?.debug?.( - `${TAG} Incremental filter: ${allExtracted.length} total โ†’ ${extracted.length} new (cursor=${cursor})`, - ); - - // Safety valve: if timestamp filter passed everything through and position slice - // was not available, this likely indicates timestamp drift after a gateway restart. - if (!usePositionSlice && extracted.length === allExtracted.length && allExtracted.length > 8) { - logger?.warn?.( - `${TAG} โš  Safety valve: all ${allExtracted.length} messages passed timestamp filter (cursor=${cursor}) โ€” ` + - `possible timestamp drift after gateway restart. Position slice was not available (no cached messageCount).`, - ); - } - } - - if (extracted.length === 0) { - logger?.debug?.(`${TAG} No new user/assistant messages to record`); - return []; - } - - // Step 2: Replace polluted user messages with cached original prompt. - // - // Background: - // The framework appends the user's message to the session after before_prompt_build, - // then injects prependContext into it. So the user message in rawMessages is polluted. - // We cached the clean prompt (originalUserText) and the message count at - // before_prompt_build time (originalUserMessageCount) to identify which raw message - // is the real user input. - // - // Strategy: - // When position slice is active, the polluted user message is slicedMessages[0]. - // Otherwise, fall back to rawMessages[originalUserMessageCount]. - // In both cases, find the timestamp and match it in `extracted` for replacement. - // If matching fails, skip replacement โ€” sanitizeText() in Step 3 is the safety net. - if (originalUserText) { - // Determine the target raw message that contains the polluted user prompt - const targetRaw: Record | undefined = usePositionSlice - ? slicedMessages[0] as Record | undefined - : (originalUserMessageCount != null && originalUserMessageCount >= 0 && originalUserMessageCount < rawMessages.length) - ? rawMessages[originalUserMessageCount] as Record | undefined - : undefined; - - const targetTs = targetRaw && typeof targetRaw.timestamp === "number" ? targetRaw.timestamp : undefined; - - if (targetTs != null) { - let replaced = false; - for (let i = 0; i < extracted.length; i++) { - if (extracted[i].role === "user" && extracted[i].timestamp === targetTs) { - logger?.debug?.( - `${TAG} Replacing user message at timestamp=${targetTs} with cached original prompt ` + - `(${originalUserText.length} chars, was ${extracted[i].content.length} chars) [positionSlice=${usePositionSlice}]`, - ); - extracted[i] = { ...extracted[i], content: originalUserText }; - replaced = true; - break; - } - } - if (!replaced) { - logger?.warn?.( - `${TAG} Target user message (ts=${targetTs}) not found in extracted batch โ€” ` + - `possibly filtered by cursor. Skipping replacement, will rely on sanitizeText().`, - ); - } - } else if (targetRaw) { - logger?.warn?.( - `${TAG} Target raw message has no valid timestamp โ€” ` + - `skipping replacement, will rely on sanitizeText().`, - ); - } else { - logger?.warn?.( - `${TAG} Have originalUserText but cannot locate target raw message โ€” ` + - `skipping replacement, will rely on sanitizeText().`, - ); - } - } - - // Step 3: Sanitize and filter - const filtered = extracted - .map((m) => { - let content = sanitizeText(m.content); - // Strip fenced code blocks from assistant replies to reduce embedding noise - if (m.role === "assistant") { - content = stripCodeBlocks(content); - } - return { id: m.id, role: m.role, content, timestamp: m.timestamp }; - }) - .filter((m) => shouldCaptureL0(m.content)); - - logger?.debug?.(`${TAG} After sanitize+filter: ${filtered.length} messages (from ${extracted.length})`); - - if (filtered.length === 0) { - logger?.debug?.(`${TAG} All messages filtered out, skipping L0 write`); - return []; - } - - // Step 4: Write to JSONL file โ€” one message per line (flat format) - const now = new Date().toISOString(); - const lines: string[] = []; - for (const msg of filtered) { - const record: L0MessageRecord = { - sessionKey, - sessionId: sessionId || "", - recordedAt: now, - id: msg.id, - role: msg.role, - content: msg.content, - timestamp: msg.timestamp, - }; - lines.push(JSON.stringify(record)); - } - - const shardDate = formatLocalDate(new Date()); - const outDir = path.join(baseDir, "conversations"); - const outPath = path.join(outDir, `${shardDate}.jsonl`); - - try { - await fs.mkdir(outDir, { recursive: true }); - // Append each message as its own JSONL line - await fs.appendFile(outPath, lines.join("\n") + "\n", "utf-8"); - logger?.debug?.(`${TAG} Recorded ${filtered.length} messages to ${outPath}`); - } catch (err) { - logger?.error(`${TAG} Failed to write L0 file: ${err instanceof Error ? err.message : String(err)}`); - // Return filtered messages anyway so L1 can still process them - } - - return filtered; -} - -/** - * Read all L0 conversation records for a session. - * Returns records in chronological order. - * - * File format: `YYYY-MM-DD.jsonl` (daily files, all sessions merged). - * Each line is an L0MessageRecord; filtered by sessionKey at line level. - */ -export async function readConversationRecords( - sessionKey: string, - baseDir: string, - logger?: Logger, -): Promise { - const conversationsDir = path.join(baseDir, "conversations"); - - // Daily file pattern: YYYY-MM-DD.jsonl - const dateFilePattern = /^\d{4}-\d{2}-\d{2}\.jsonl$/; - - let entries: string[]; - try { - const dirEntries = await fs.readdir(conversationsDir, { withFileTypes: true }); - entries = dirEntries - .filter((entry) => entry.isFile()) - .map((entry) => entry.name); - } catch { - // Directory doesn't exist yet โ€” normal for first conversation - return []; - } - - const targetFiles = entries - .filter((name) => dateFilePattern.test(name)) - .sort(); - - if (targetFiles.length === 0) { - return []; - } - - const records: L0ConversationRecord[] = []; - - for (const fileName of targetFiles) { - const filePath = path.join(conversationsDir, fileName); - - let raw: string; - try { - raw = await fs.readFile(filePath, "utf-8"); - } catch { - logger?.warn?.(`${TAG} Failed to read L0 file: ${filePath}`); - continue; - } - - const lines = raw.split("\n").filter((line: string) => line.trim()); - for (let i = 0; i < lines.length; i++) { - const line = lines[i]; - try { - const parsed = JSON.parse(line) as Record; - - // Filter by sessionKey at line level - const lineSessionKey = parsed.sessionKey as string | undefined; - if (lineSessionKey !== sessionKey) continue; - - if (typeof parsed.role === "string" && typeof parsed.content === "string") { - // Flat format: { sessionKey, sessionId, recordedAt, id, role, content, timestamp } - // Wrap into L0ConversationRecord for uniform downstream consumption - const msg: ConversationMessage = { - id: (typeof parsed.id === "string" && parsed.id) ? parsed.id : generateMessageId(), - role: parsed.role as "user" | "assistant", - content: parsed.content as string, - timestamp: typeof parsed.timestamp === "number" ? parsed.timestamp : Date.now(), - }; - records.push({ - sessionKey: (parsed.sessionKey as string) || sessionKey, - sessionId: (parsed.sessionId as string) || "", - recordedAt: (parsed.recordedAt as string) || new Date().toISOString(), - messageCount: 1, - messages: [msg], - }); - } else { - logger?.warn?.(`${TAG} Unrecognized JSONL line format in ${filePath}:${i + 1}`); - } - } catch { - logger?.warn?.(`${TAG} Skipping malformed JSONL line in ${filePath}:${i + 1}`); - } - } - } - - records.sort((a, b) => { - const ta = Date.parse(a.recordedAt); - const tb = Date.parse(b.recordedAt); - const na = Number.isFinite(ta) ? ta : Number.POSITIVE_INFINITY; - const nb = Number.isFinite(tb) ? tb : Number.POSITIVE_INFINITY; - return na - nb; - }); - - return records; -} - -/** - * Read L0 messages across all conversation records for a session, - * optionally filtered by a cursor timestamp (messages after the cursor). - * - * When `limit` is provided, only the **newest** `limit` messages are returned - * (matching the DB path's `ORDER BY timestamp DESC LIMIT ?` behavior). - * Returned messages are always in chronological order (oldest โ†’ newest). - * - * NOTE: potential optimization โ€” records are chronologically ordered (append-only JSONL), - * so a reverse scan could skip entire old records. Deferred for now; see Issue 5 in - * docs/05-known-issues.md. - */ -export async function readConversationMessages( - sessionKey: string, - baseDir: string, - afterTimestamp?: number, - logger?: Logger, - limit?: number, -): Promise { - const records = await readConversationRecords(sessionKey, baseDir, logger); - const allMessages: ConversationMessage[] = []; - - for (const record of records) { - for (const msg of record.messages) { - if (afterTimestamp && msg.timestamp <= afterTimestamp) continue; - allMessages.push(msg); - } - } - - // Truncate to newest `limit` messages (keep tail, since array is chronological) - if (limit != null && limit > 0 && allMessages.length > limit) { - logger?.debug?.( - `${TAG} readConversationMessages: truncating ${allMessages.length} โ†’ ${limit} (newest)`, - ); - return allMessages.slice(-limit); - } - - return allMessages; -} - -/** - * A group of conversation messages sharing the same sessionId. - */ -export interface SessionIdMessageGroup { - sessionId: string; - messages: Array; -} - -/** - * Read L0 messages for a session, grouped by sessionId. - * - * Within the same sessionKey, different sessionIds represent different conversation - * instances (e.g. after /reset). L1 extraction should process each group independently - * so that each group's sessionId is correctly associated with its extracted memories. - * - * When `limit` is provided, only the **newest** `limit` messages (across all groups) - * are retained โ€” matching the DB path's `ORDER BY recorded_at DESC LIMIT ?` behavior. - * Groups that become empty after truncation are dropped. - * - * Groups are returned in chronological order (by earliest message timestamp). - * Messages within each group are also in chronological order. - * - * @param afterRecordedAtMs - Epoch ms cursor: only messages with recordedAt > this are included. - */ -export async function readConversationMessagesGroupedBySessionId( - sessionKey: string, - baseDir: string, - afterRecordedAtMs?: number, - logger?: Logger, - limit?: number, -): Promise { - const records = await readConversationRecords(sessionKey, baseDir, logger); - - // Collect all messages with their sessionId, filtering by recorded_at cursor - const allMessages: Array<{ sessionId: string; msg: ConversationMessage & { recordedAtMs: number } }> = []; - - for (const record of records) { - const sid = record.sessionId || ""; - const recMs = Date.parse(record.recordedAt) || 0; - if (afterRecordedAtMs && recMs <= afterRecordedAtMs) continue; - for (const msg of record.messages) { - allMessages.push({ sessionId: sid, msg: { ...msg, recordedAtMs: recMs } }); - } - } - - // Sort by timestamp ASC (chronological) โ€” records are already roughly ordered - // by recordedAt, but messages within may not be perfectly sorted by timestamp. - allMessages.sort((a, b) => a.msg.timestamp - b.msg.timestamp); - - // Truncate to newest `limit` messages (keep tail) - let selected = allMessages; - if (limit != null && limit > 0 && allMessages.length > limit) { - logger?.debug?.( - `${TAG} readConversationMessagesGroupedBySessionId: truncating ${allMessages.length} โ†’ ${limit} (newest)`, - ); - selected = allMessages.slice(-limit); - } - - // Re-group by sessionId - const groupMap = new Map>(); - for (const { sessionId, msg } of selected) { - let group = groupMap.get(sessionId); - if (!group) { - group = []; - groupMap.set(sessionId, group); - } - group.push(msg); - } - - // Convert to array, sorted by earliest message timestamp in each group - const groups: SessionIdMessageGroup[] = []; - for (const [sessionId, messages] of groupMap) { - if (messages.length > 0) { - groups.push({ sessionId, messages }); - } - } - groups.sort((a, b) => a.messages[0].timestamp - b.messages[0].timestamp); - - return groups; -} - -// ============================ -// Helpers -// ============================ - -/** - * Extract user and assistant messages from raw hook message array. - */ -function extractUserAssistantMessages(messages: unknown[]): ConversationMessage[] { - const result: ConversationMessage[] = []; - - for (const msg of messages) { - if (!msg || typeof msg !== "object") continue; - const m = msg as Record; - const role = m.role as string | undefined; - - if (role !== "user" && role !== "assistant") continue; - - let content: string | undefined; - if (typeof m.content === "string") { - content = m.content; - } else if (Array.isArray(m.content)) { - const textParts: string[] = []; - for (const part of m.content) { - if ( - part && - typeof part === "object" && - (part as Record).type === "text" - ) { - const text = (part as Record).text; - if (typeof text === "string") textParts.push(text); - } - } - content = textParts.join("\n"); - } - - // Strip inline base64 image data URIs that some providers embed in string content. - // These are not useful for memory and would pollute FTS / embedding indexes. - if (content && /data:image\/[a-z+]+;base64,/i.test(content)) { - content = content.replace(/data:image\/[a-z+]+;base64,[A-Za-z0-9+/=]+/gi, "[image]"); - } - - if (content && content.trim()) { - const ts = typeof m.timestamp === "number" ? m.timestamp : Date.now(); - result.push({ - id: (typeof m.id === "string" && m.id) ? m.id : generateMessageId(), - role: role as "user" | "assistant", - content: content.trim(), - timestamp: ts, - }); - } - } - - return result; -} - -/** - * Format local date as YYYY-MM-DD. - */ -function formatLocalDate(d: Date): string { - const y = d.getFullYear(); - const m = String(d.getMonth() + 1).padStart(2, "0"); - const day = String(d.getDate()).padStart(2, "0"); - return `${y}-${m}-${day}`; -} diff --git a/src/core/hooks/auto-recall.ts b/src/core/hooks/auto-recall.ts index edb8449..cccb864 100644 --- a/src/core/hooks/auto-recall.ts +++ b/src/core/hooks/auto-recall.ts @@ -356,7 +356,7 @@ async function searchMemories( // Resolve per-call embedding timeout for recall path. // Falls back to global embedding.timeoutMs when recallTimeoutMs is not configured. - const recallEmbeddingTimeoutMs = cfg.embedding.recallTimeoutMs ?? cfg.embedding.timeoutMs; + const recallEmbeddingTimeoutMs = cfg.embedding?.recallTimeoutMs ?? cfg.embedding?.timeoutMs; const embeddingCallOpts: EmbeddingCallOptions = { timeoutMs: recallEmbeddingTimeoutMs }; try { @@ -372,7 +372,19 @@ async function searchMemories( return { lines, timing: { ftsMs: 0, embeddingMs: performance.now() - tEmb, ftsHits: 0, embeddingHits: lines.length } }; } - // Hybrid: run both keyword and embedding, merge with RRF + // Hybrid: if the store natively supports hybrid search (e.g. TCVDB does + // server-side dense + sparse + RRF in a single API call), short-circuit + // to avoid a redundant second HTTP request and a wasted local embed(). + if (vectorStore?.getCapabilities().nativeHybridSearch) { + const tNative = performance.now(); + const results = await vectorStore.searchL1Hybrid({ query: cleanText, topK: maxResults }); + const nativeMs = performance.now() - tNative; + logger?.debug?.(`${TAG} [hybrid-native] Single-call hybrid: ${results.length} results in ${nativeMs.toFixed(0)}ms`); + const lines = results.map((r) => formatMemoryLine(vectorResultToFormatable(r))); + return { lines, timing: { ftsMs: 0, embeddingMs: nativeMs, ftsHits: 0, embeddingHits: results.length } }; + } + + // Fallback: run keyword + embedding in parallel, merge with client-side RRF (SQLite path) return await searchHybrid(cleanText, pluginDataDir, maxResults, threshold, vectorStore!, embeddingService!, logger, embeddingCallOpts); } catch (err) { logger?.warn?.(`${TAG} Memory search failed (strategy=${effectiveStrategy}): ${err instanceof Error ? err.message : String(err)}`); diff --git a/src/core/profile/profile-sync.ts b/src/core/profile/profile-sync.ts index 260dc86..c921265 100644 --- a/src/core/profile/profile-sync.ts +++ b/src/core/profile/profile-sync.ts @@ -142,7 +142,7 @@ export async function pullProfilesToLocal( await fs.writeFile(target, record.content, "utf-8"); if (md5(record.content) !== record.contentMd5) { await fs.rm(target, { force: true }); - logger.warn(`[memory-tdai][profile-sync] MD5 mismatch for ${record.filename}`); + logger.debug?.(`[memory-tdai][profile-sync] MD5 mismatch for ${record.filename} (will re-pull on next sync)`); } continue; } @@ -152,7 +152,7 @@ export async function pullProfilesToLocal( await fs.writeFile(path.join(tempDir, "persona.md"), body, "utf-8"); if (md5(body) !== record.contentMd5) { await fs.rm(path.join(tempDir, "persona.md"), { force: true }); - logger.warn(`[memory-tdai][profile-sync] MD5 mismatch for ${record.filename}`); + logger.debug?.(`[memory-tdai][profile-sync] MD5 mismatch for ${record.filename} (will re-pull on next sync)`); } } } diff --git a/src/core/record/l1-extractor.ts b/src/core/record/l1-extractor.ts index 99a6431..ad0a8f1 100644 --- a/src/core/record/l1-extractor.ts +++ b/src/core/record/l1-extractor.ts @@ -318,6 +318,11 @@ async function callLlmExtraction(params: { previousSceneName, }); + // [l1-debug] ENTRY โ€” what are we about to ask the LLM to extract? + logger?.debug?.( + `${TAG} [l1-debug] ENTRY taskId=l1-extraction, newMsgs=${newMessages.length}, bgMsgs=${backgroundMessages.length}, userPromptLen=${userPrompt.length}, sysPromptLen=${EXTRACT_MEMORIES_SYSTEM_PROMPT.length}, model=${model ?? "(default)"}, previousSceneName=${previousSceneName ? JSON.stringify(previousSceneName) : "(none)"}, runnerKind=${llmRunner ? "llmRunner" : "CleanContextRunner"}`, + ); + let result: string; if (llmRunner) { @@ -364,6 +369,11 @@ function parseExtractionResult(raw: string, logger?: Logger): SceneSegment[] { const arrayMatch = cleaned.match(/\[[\s\S]*\]/); if (!arrayMatch) { logger?.warn?.(`${TAG} No JSON array found in extraction response`); + // [l1-debug] NO_JSON โ€” dump the full raw so we can see what the LLM actually said + const rawPreview = raw.slice(0, 2048); + logger?.warn?.( + `${TAG} [l1-debug] NO_JSON taskId=l1-extraction, rawLen=${raw.length}, cleanedLen=${cleaned.length}, rawFull=${JSON.stringify(rawPreview)}${raw.length > 2048 ? `โ€ฆ(+${raw.length - 2048})` : ""}`, + ); return []; } diff --git a/src/core/store/sqlite.ts b/src/core/store/sqlite.ts index a3922a3..6252419 100644 --- a/src/core/store/sqlite.ts +++ b/src/core/store/sqlite.ts @@ -665,7 +665,7 @@ export class VectorStore implements IMemoryStore { // Migration: add timestamp column if missing (existing DBs pre-v3.x) try { this.db.exec("ALTER TABLE l0_conversations ADD COLUMN timestamp INTEGER DEFAULT 0"); - this.logger?.info(`${TAG} Migrated l0_conversations: added timestamp column`); + this.logger?.debug?.(`${TAG} Migrated l0_conversations: added timestamp column`); } catch { // Column already exists โ€” expected on non-first run } diff --git a/src/core/store/tcvdb-client.ts b/src/core/store/tcvdb-client.ts index 3594a59..41d173b 100644 --- a/src/core/store/tcvdb-client.ts +++ b/src/core/store/tcvdb-client.ts @@ -120,10 +120,12 @@ export class TcvdbClient { */ async request(path: string, body: Record): Promise { let lastError: Error | undefined; + const t0 = performance.now(); for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) { + const tAttempt = performance.now(); try { - this.logger?.debug?.(`${TAG} โ†’ ${path} body=${JSON.stringify(body).slice(0, 500)}`); + this.logger?.debug?.(`${TAG} โ†’ ${path} attempt=${attempt} body=${JSON.stringify(body).slice(0, 500)}`); const { statusCode, body: respBody } = await undiciRequest(`${this.baseUrl}${path}`, { method: "POST", headers: { @@ -137,7 +139,8 @@ export class TcvdbClient { const text = await respBody.text(); const json = JSON.parse(text) as ApiResponse; - this.logger?.debug?.(`${TAG} โ† ${path} status=${statusCode} code=${json.code} msg=${json.msg} keys=[${Object.keys(json).join(",")}]`); + const attemptMs = Math.round(performance.now() - tAttempt); + this.logger?.debug?.(`${TAG} โ† ${path} status=${statusCode} code=${json.code} attemptMs=${attemptMs} attempt=${attempt}`); if (json.code !== 0) { const err = new TcvdbApiError(path, json.code, json.msg); @@ -146,18 +149,25 @@ export class TcvdbClient { continue; } + // Always log completion at info level (one line per request) + const totalMs = Math.round(performance.now() - t0); + this.logger?.info(`${TAG} ${path} ${totalMs}ms${attempt > 0 ? ` (${attempt + 1} attempts)` : ""}`); + return json as unknown as T; } catch (err) { + const attemptMs = Math.round(performance.now() - tAttempt); if (err instanceof TcvdbApiError && err.apiCode !== 0) throw err; lastError = err instanceof Error ? err : new Error(String(err)); if (attempt < MAX_RETRIES) { const delay = 500 * (attempt + 1); - this.logger?.debug?.(`${TAG} ${path} retry ${attempt + 1}/${MAX_RETRIES} in ${delay}ms`); + this.logger?.debug?.(`${TAG} ${path} retry ${attempt + 1}/${MAX_RETRIES} in ${delay}ms (lastAttemptMs=${attemptMs}, error=${lastError.message})`); await new Promise((r) => setTimeout(r, delay)); } } } + const totalMs = Math.round(performance.now() - t0); + this.logger?.debug?.(`${TAG} โœ— ${path} totalMs=${totalMs} attempts=${MAX_RETRIES + 1} error=${lastError?.message}`); throw lastError ?? new Error(`${TAG} ${path} failed after retries`); } diff --git a/src/hooks/auto-capture.ts b/src/hooks/auto-capture.ts deleted file mode 100644 index 6fd50d7..0000000 --- a/src/hooks/auto-capture.ts +++ /dev/null @@ -1,319 +0,0 @@ -/** - * auto-capture hook (v3): records conversation messages locally (L0), - * then notifies the MemoryPipelineManager for L1/L2/L3 scheduling. - * - * Key design decisions: - * - Always write L0 locally via l0-recorder. - * - When VectorStore + EmbeddingService are available, also write L0 vector index. - * - Notify MemoryPipelineManager for L1/L2/L3 trigger evaluation. - * - L1 Runner reads from VectorStore DB (primary) or L0 JSONL files (fallback). - * - Extraction is NOT triggered here. The pipeline manager decides when. - */ - -import crypto from "node:crypto"; -import type { MemoryTdaiConfig } from "../config.js"; -import { CheckpointManager } from "../utils/checkpoint.js"; -import type { MemoryPipelineManager } from "../utils/pipeline-manager.js"; -import { recordConversation } from "../conversation/l0-recorder.js"; -import type { ConversationMessage } from "../conversation/l0-recorder.js"; -import type { IMemoryStore, L0Record } from "../store/types.js"; -import type { EmbeddingService } from "../store/embedding.js"; - -const TAG = "[memory-tdai] [capture]"; - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -export interface AutoCaptureResult { - /** Whether the scheduler was notified (conversation count incremented) */ - schedulerNotified: boolean; - /** Number of messages recorded to L0 */ - l0RecordedCount: number; - /** Number of L0 message vectors written */ - l0VectorsWritten: number; - /** Filtered messages for L1 immediate use */ - filteredMessages: ConversationMessage[]; -} - -/** - * Generate a unique L0 record ID for vector indexing. - * Includes an index to distinguish multiple messages within the same round. - */ -function generateL0RecordId(sessionKey: string, index: number): string { - return `l0_${sessionKey}_${Date.now()}_${index}_${crypto.randomBytes(3).toString("hex")}`; -} - -export async function performAutoCapture(params: { - messages: unknown[]; - sessionKey: string; - sessionId?: string; - cfg: MemoryTdaiConfig; - pluginDataDir: string; - logger?: Logger; - scheduler?: MemoryPipelineManager; - /** Clean original user prompt from before_prompt_build cache (pre-prependContext). */ - originalUserText?: string; - /** - * Number of messages in the session at before_prompt_build time. - * Used by l0-recorder to locate the exact user message that originalUserText - * corresponds to: rawMessages[originalUserMessageCount] is the polluted user message. - */ - originalUserMessageCount?: number; - /** Epoch ms when the plugin was registered (cold-start time). - * Used as fallback cursor when checkpoint has no prior timestamp โ€” - * prevents the first agent_end from dumping all session history into L0. */ - pluginStartTimestamp?: number; - /** VectorStore for L0 vector indexing (optional). */ - vectorStore?: IMemoryStore; - /** EmbeddingService for L0 vector indexing (optional). */ - embeddingService?: EmbeddingService; -}): Promise { - const { - messages, sessionKey, sessionId, cfg, pluginDataDir, logger, scheduler, - originalUserText, originalUserMessageCount, pluginStartTimestamp, - vectorStore, embeddingService, - } = params; - const tCaptureStart = performance.now(); - - const checkpoint = new CheckpointManager(pluginDataDir, logger); - - // ============================ - // Step 1 + 2: L0 recording + checkpoint update (ATOMIC) - // ============================ - // These steps are combined inside captureAtomically() to prevent the race - // condition where two concurrent agent_end events both read the same stale - // cursor and produce duplicate L0 records. The file lock is held for the - // entire read-cursor โ†’ recordConversation โ†’ advance-cursor sequence. - const tL0RecordStart = performance.now(); - let filteredMessages: ConversationMessage[] = []; - try { - await checkpoint.captureAtomically( - sessionKey, - pluginStartTimestamp, - async (afterTimestamp) => { - logger?.debug?.(`${TAG} L0 capture cursor (per-session, atomic): afterTimestamp=${afterTimestamp} session=${sessionKey}`); - - if (afterTimestamp === pluginStartTimestamp && pluginStartTimestamp && pluginStartTimestamp > 0) { - logger?.debug?.( - `${TAG} No per-session checkpoint cursor found for session=${sessionKey} โ€” ` + - `using pluginStartTimestamp as floor: ` + - `${afterTimestamp} (${new Date(afterTimestamp).toISOString()})`, - ); - } - - filteredMessages = await recordConversation({ - sessionKey, - sessionId, - rawMessages: messages, - baseDir: pluginDataDir, - logger, - originalUserText, - afterTimestamp, - originalUserMessageCount, - }); - - if (filteredMessages.length === 0) { - return null; // Nothing captured โ€” cursor stays unchanged - } - - logger?.debug?.(`${TAG} L0 recorded: ${filteredMessages.length} messages for session ${sessionKey}`); - const maxTs = Math.max(...filteredMessages.map((m) => m.timestamp)); - return { maxTimestamp: maxTs, messageCount: filteredMessages.length }; - }, - ); - } catch (err) { - logger?.error(`${TAG} L0 recording failed: ${err instanceof Error ? err.message : String(err)}`); - } - const tL0RecordEnd = performance.now(); - - // ============================ - // Step 1.5: L0 vector indexing - // ============================ - // Two paths depending on store capabilities: - // - // A) Store supports updateL0Embedding (sqlite): - // - Write metadata + FTS immediately WITHOUT embedding (~ms) - // - Fire-and-forget background task: embedBatch + updateL0Embedding - // - PERF: avoids blocking agent_end with 2-3s embedding calls - // - // B) Store does NOT support updateL0Embedding (VDB / remote): - // - Embed synchronously, then upsertL0 with embedding in one call - // - VDB backends handle embedding server-side or need it upfront - const tL0VecStart = performance.now(); - let l0VectorsWritten = 0; - let l0EmbedTotalMs = 0; - let l0UpsertTotalMs = 0; - logger?.debug?.( - `${TAG} [L0-vec-index] Check: filteredMessages=${filteredMessages.length}, ` + - `vectorStore=${vectorStore ? "available" : "UNAVAILABLE"}, ` + - `embeddingService=${embeddingService ? "available" : "UNAVAILABLE"}`, - ); - - const supportsBgEmbed = vectorStore?.supportsDeferredEmbedding === true; - - if (filteredMessages.length > 0 && vectorStore) { - const now = new Date().toISOString(); - const bgRecords: Array<{ recordId: string; content: string }> = []; - logger?.debug?.( - `${TAG} [L0-vec-index] START indexing ${filteredMessages.length} message(s) for session ${sessionKey} ` + - `(mode=${supportsBgEmbed ? "async-bg" : "sync"})`, - ); - - for (let i = 0; i < filteredMessages.length; i++) { - const msg = filteredMessages[i]; - try { - const l0Record: L0Record = { - id: generateL0RecordId(sessionKey, i), - sessionKey, - sessionId: sessionId || "", - role: msg.role, - messageText: msg.content, - recordedAt: now, - timestamp: msg.timestamp, - }; - - let embedding: Float32Array | undefined; - - if (!supportsBgEmbed && embeddingService) { - // Path B (VDB): embed synchronously โ€” needed for upsertL0 - // Skip local embed when using server-side embedding (NoopEmbeddingService, dims=0) - if (embeddingService.getDimensions() === 0) { - logger?.debug?.( - `${TAG} [L0-vec-index] Server-side embedding (dims=0), skipping local embed for message ${i}`, - ); - } else { - const tEmbedStart = performance.now(); - try { - embedding = await embeddingService.embed(msg.content); - l0EmbedTotalMs += performance.now() - tEmbedStart; - logger?.debug?.( - `${TAG} [L0-vec-index] Embedding OK: dims=${embedding.length}, ` + - `norm=${Math.sqrt(Array.from(embedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}`, - ); - } catch (embedErr) { - l0EmbedTotalMs += performance.now() - tEmbedStart; - logger?.warn( - `${TAG} [L0-vec-index] Embedding FAILED for message ${i}, ` + - `will write metadata only: ${embedErr instanceof Error ? embedErr.message : String(embedErr)}`, - ); - } - } - } - - // Path A (sqlite): pass undefined embedding โ€” metadata + FTS only - // Path B (VDB): pass embedding (may be undefined on failure) - const tUpsertStart = performance.now(); - const upsertOk = await vectorStore.upsertL0(l0Record, supportsBgEmbed ? undefined : embedding); - l0UpsertTotalMs += performance.now() - tUpsertStart; - - if (upsertOk) { - l0VectorsWritten++; - if (supportsBgEmbed) { - bgRecords.push({ recordId: l0Record.id, content: msg.content }); - } - } else { - logger?.warn(`${TAG} [L0-vec-index] upsertL0 returned false for message ${i}`); - } - } catch (err) { - logger?.warn?.(`${TAG} [L0-vec-index] FAILED for message ${i} (non-blocking): ${err instanceof Error ? err.message : String(err)}`); - } - } - - const modeLabel = supportsBgEmbed ? "metadata-only, embed=background" : `embed=${l0EmbedTotalMs.toFixed(0)}ms, upsert=${l0UpsertTotalMs.toFixed(0)}ms`; - logger?.debug?.(`${TAG} [L0-vec-index] DONE: ${l0VectorsWritten}/${filteredMessages.length} records written (${modeLabel})`); - - // Path A only: fire-and-forget background embedding for sqlite stores - if (supportsBgEmbed && bgRecords.length > 0 && embeddingService) { - const bgVectorStore = vectorStore; - const bgEmbeddingService = embeddingService; - const bgSnapshot = [...bgRecords]; - const bgLogger = logger; - - // Do NOT await โ€” runs in background after response is sent - void (async () => { - const tBgStart = performance.now(); - try { - const texts = bgSnapshot.map((r) => r.content); - const embeddings = await bgEmbeddingService.embedBatch(texts); - - let bgUpdated = 0; - for (let i = 0; i < bgSnapshot.length; i++) { - try { - const ok = await bgVectorStore.updateL0Embedding!(bgSnapshot[i].recordId, embeddings[i]); - if (ok) bgUpdated++; - } catch (err) { - bgLogger?.warn?.( - `${TAG} [L0-vec-index-bg] Failed to update embedding for ${bgSnapshot[i].recordId}: ` + - `${err instanceof Error ? err.message : String(err)}`, - ); - } - } - const bgMs = performance.now() - tBgStart; - bgLogger?.debug?.( - `${TAG} [L0-vec-index-bg] Background embedding complete: ${bgUpdated}/${bgSnapshot.length} vectors updated (${bgMs.toFixed(0)}ms)`, - ); - } catch (err) { - const bgMs = performance.now() - tBgStart; - bgLogger?.warn?.( - `${TAG} [L0-vec-index-bg] Background embedding failed (${bgMs.toFixed(0)}ms, non-fatal): ` + - `${err instanceof Error ? err.message : String(err)}`, - ); - } - })(); - } - } else if (filteredMessages.length > 0) { - logger?.warn(`${TAG} [L0-vec-index] SKIPPED: vectorStore not available`); - } - const tL0VecEnd = performance.now(); - - // ============================ - // Step 3: Notify scheduler of this conversation round - // ============================ - const tNotifyStart = performance.now(); - // Pass empty array: L1 Runner reads from VectorStore DB (or L0 JSONL fallback), not from in-memory buffers. - if (scheduler) { - await scheduler.notifyConversation(sessionKey, []); - logger?.debug?.(`${TAG} Scheduler notified of conversation round (sessionKey=${sessionKey})`); - - const totalMs = performance.now() - tCaptureStart; - const vecDetail = supportsBgEmbed - ? `metadata-only, embed=background, msgs=${filteredMessages.length}` - : `embed=${l0EmbedTotalMs.toFixed(0)}ms, upsert=${l0UpsertTotalMs.toFixed(0)}ms, msgs=${filteredMessages.length}`; - logger?.info( - `${TAG} โฑ Capture timing: total=${totalMs.toFixed(0)}ms, ` + - `l0Record+checkpoint=${(tL0RecordEnd - tL0RecordStart).toFixed(0)}ms, ` + - `l0VecIndex=${(tL0VecEnd - tL0VecStart).toFixed(0)}ms (${vecDetail}), ` + - `notify=${(performance.now() - tNotifyStart).toFixed(0)}ms`, - ); - - return { - schedulerNotified: true, - l0RecordedCount: filteredMessages.length, - l0VectorsWritten, - filteredMessages, - }; - } - - const totalMs = performance.now() - tCaptureStart; - const vecDetail = supportsBgEmbed - ? `metadata-only, embed=background, msgs=${filteredMessages.length}` - : `embed=${l0EmbedTotalMs.toFixed(0)}ms, upsert=${l0UpsertTotalMs.toFixed(0)}ms, msgs=${filteredMessages.length}`; - logger?.info( - `${TAG} โฑ Capture timing: total=${totalMs.toFixed(0)}ms, ` + - `l0Record+checkpoint=${(tL0RecordEnd - tL0RecordStart).toFixed(0)}ms, ` + - `l0VecIndex=${(tL0VecEnd - tL0VecStart).toFixed(0)}ms (${vecDetail}), ` + - `notify=${(performance.now() - tNotifyStart).toFixed(0)}ms`, - ); - - logger?.debug?.(`${TAG} No scheduler provided, skipping notification`); - return { - schedulerNotified: false, - l0RecordedCount: filteredMessages.length, - l0VectorsWritten, - filteredMessages, - }; -} diff --git a/src/hooks/auto-recall.ts b/src/hooks/auto-recall.ts deleted file mode 100644 index d94230a..0000000 --- a/src/hooks/auto-recall.ts +++ /dev/null @@ -1,769 +0,0 @@ -/** - * auto-recall hook (v3): injects relevant memories + persona into agent context - * before the agent starts processing. - * - * - Searches L1 memories using configurable strategy (keyword / embedding / hybrid) - * - keyword: FTS5 BM25 (requires FTS5; returns empty if unavailable) - * - embedding: VectorStore cosine similarity - * - hybrid: keyword + embedding merged with RRF - * - L3 persona injection - * - L2 scene navigation (full injection, LLM decides relevance) - */ - -import fs from "node:fs/promises"; -import path from "node:path"; -import type { MemoryTdaiConfig } from "../config.js"; -import { readSceneIndex } from "../scene/scene-index.js"; -import { generateSceneNavigation, stripSceneNavigation } from "../scene/scene-navigation.js"; -import type { MemoryRecord } from "../record/l1-reader.js"; -import type { IMemoryStore, L1SearchResult, L1FtsResult } from "../store/types.js"; -import { buildFtsQuery } from "../store/sqlite.js"; -import type { EmbeddingService } from "../store/embedding.js"; -import { sanitizeText } from "../utils/sanitize.js"; - -const TAG = "[memory-tdai] [recall]"; - -/** - * Memory tools usage guide โ€” injected at the end of memory context so the - * main agent knows how to actively retrieve deeper information. - */ -const MEMORY_TOOLS_GUIDE = ` -## ่ฎฐๅฟ†ๅทฅๅ…ท่ฐƒ็”จๆŒ‡ๅ— - -ๅฝ“ไธŠๆ–นๆณจๅ…ฅ็š„่ฎฐๅฟ†็‰‡ๆฎตไธ่ถณไปฅๅ›ž็ญ”็”จๆˆท้—ฎ้ข˜ๆ—ถ๏ผŒๅฏไธปๅŠจ่ฐƒ็”จไปฅไธ‹ๅทฅๅ…ท่Žทๅ–ๆ›ดๅคšไฟกๆฏ๏ผš - -- **tdai_memory_search**๏ผšๆœ็ดข็ป“ๆž„ๅŒ–่ฎฐๅฟ†๏ผˆL1๏ผ‰๏ผŒ้€‚็”จไบŽๅ›žๅฟ†็”จๆˆทๅๅฅฝใ€ๅކๅฒไบ‹ไปถ่Š‚็‚นใ€่ง„ๅˆ™็ญ‰ๅ…ณ้”ฎไฟกๆฏใ€‚ -- **tdai_conversation_search**๏ผšๆœ็ดขๅŽŸๅง‹ๅฏน่ฏ๏ผˆL0๏ผ‰๏ผŒ้€‚็”จไบŽๆŸฅๆ‰พๅ…ทไฝ“ๆถˆๆฏๅŽŸๆ–‡ใ€ๆ—ถ้—ด็บฟใ€ไธŠไธ‹ๆ–‡็ป†่Š‚๏ผ›ไนŸๅฏ็”จไบŽ่กฅๅ……ๆˆ–ๆ ก้ชŒ memory_search ็š„็ป“ๆžœใ€‚ -- **read_file**๏ผˆScene Navigation ไธญ็š„่ทฏๅพ„๏ผ‰๏ผšๅฝ“ๅทฒๅฎšไฝๅˆฐ็›ธๅ…ณๆƒ…ๅขƒ๏ผŒไธ”้œ€่ฆ่ฏฅๅœบๆ™ฏ็š„ๅฎŒๆ•ด็”ปๅƒใ€ไบ‹ไปถ็ป่ฟ‡ๆˆ–้˜ถๆฎต็ป“่ฎบๆ—ถไฝฟ็”จใ€‚ - -### โš ๏ธ ่ฐƒ็”จๆฌกๆ•ฐ้™ๅˆถ -ๆฏ่ฝฎๅฏน่ฏไธญ๏ผŒtdai_memory_search ๅ’Œ tdai_conversation_search **ๅˆ่ฎกๆœ€ๅคš่ฐƒ็”จ 3 ๆฌก**ใ€‚ -- ้ฆ–ๆฌกๆœ็ดขๆ— ็ป“ๆžœๆ—ถ๏ผŒๅฏๆขๅ…ณ้”ฎ่ฏๆˆ–ๆขๅทฅๅ…ท้‡่ฏ•๏ผŒไฝ†ๆ€ป่ฐƒ็”จๆฌกๆ•ฐไธ่ฆ่ถ…่ฟ‡ 3 ๆฌกใ€‚ -- ่‹ฅ 3 ๆฌกๆœ็ดขๅŽไปๆ— ็ป“ๆžœ๏ผŒ่ฏดๆ˜Ž่ฏฅไฟกๆฏไธๅœจ่ฎฐๅฟ†ไธญ๏ผŒ่ฏท็›ดๆŽฅๆ นๆฎๅทฒๆœ‰ไฟกๆฏๅ›žๅค็”จๆˆท๏ผŒไธ่ฆ็ปง็ปญๆœ็ดขใ€‚ -` - -/** - * Build the dynamic scene-navigation read_file hint. - * Tells the agent how to resolve relative paths in scene navigation - * by prepending the actual pluginDataDir. - */ -function buildScenePathHint(pluginDataDir: string): string { - return `โš ๏ธ Scene Navigation ่ทฏๅพ„ๆ็คบ๏ผšไธŠๆ–น Scene Navigation ไธญ็š„ Path๏ผˆๅฆ‚ \`scene_blocks/xxx.md\`๏ผ‰ๆ˜ฏ็›ธๅฏน่ทฏๅพ„๏ผŒไฝฟ็”จ read_file ่ฏปๅ–ๆ—ถ้œ€ๆ‹ผๆŽฅไธบ็ปๅฏน่ทฏๅพ„๏ผš\`${pluginDataDir}/scene_blocks/xxx.md\``; -} - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -/** A single recalled L1 memory with its search score and type. */ -export interface RecalledMemory { - content: string; - score: number; - type: string; -} - -export interface RecallResult { - /** Injected before user message (prepended to the user's prompt text by openclaw) */ - prependContext?: string; - /** Appended to system prompt (all memory context: persona, scene navigation, relevant memories) */ - appendSystemContext?: string; - - // โ”€โ”€ Metric payload (for pendingRecallCache in index.ts) โ”€โ”€ - /** L1 memories that were recalled (with scores), for metric reporting */ - recalledL1Memories?: RecalledMemory[]; - /** L3 Persona raw content loaded during recall (null if none) */ - recalledL3Persona?: string | null; - /** Effective search strategy used */ - recallStrategy?: string; -} - -export async function performAutoRecall(params: { - userText: string; - actorId: string; - sessionKey: string; - cfg: MemoryTdaiConfig; - pluginDataDir: string; - logger?: Logger; - vectorStore?: IMemoryStore; - embeddingService?: EmbeddingService; -}): Promise { - const { cfg, logger } = params; - const timeoutMs = cfg.recall.timeoutMs ?? 5000; - - let timer: ReturnType | undefined; - - return Promise.race([ - performAutoRecallInner(params).finally(() => { - if (timer) clearTimeout(timer); - }), - new Promise((resolve) => { - timer = setTimeout(() => { - logger?.warn?.( - `${TAG} โš ๏ธ Recall timed out after ${timeoutMs}ms โ€” skipping memory injection to avoid blocking the user`, - ); - resolve(undefined); - }, timeoutMs); - }), - ]); -} - -async function performAutoRecallInner(params: { - userText: string; - actorId: string; - sessionKey: string; - cfg: MemoryTdaiConfig; - pluginDataDir: string; - logger?: Logger; - vectorStore?: IMemoryStore; - embeddingService?: EmbeddingService; -}): Promise { - const { userText, cfg, pluginDataDir, logger, vectorStore, embeddingService } = params; - const tRecallStart = performance.now(); - - // Search relevant memories (L1 layer) โ€” skip only when userText is empty/undefined - const tSearchStart = performance.now(); - let memoryLines: string[] = []; - let effectiveStrategy = "skipped"; - let recalledL1Memories: RecalledMemory[] = []; - let searchTiming: SearchTiming = { ftsMs: 0, embeddingMs: 0, ftsHits: 0, embeddingHits: 0 }; - if (!userText || userText.length === 0) { - logger?.debug?.(`${TAG} User text empty/undefined, skipping memory search (persona/scene still injected)`); - } else { - effectiveStrategy = cfg.recall.strategy ?? "hybrid"; - const searchResult = await searchMemories(userText, pluginDataDir, cfg, logger, effectiveStrategy as "keyword" | "embedding" | "hybrid", vectorStore, embeddingService); - memoryLines = searchResult.lines; - searchTiming = searchResult.timing; - - // Extract structured RecalledMemory from formatted lines for metric reporting - recalledL1Memories = memoryLines.map((line) => { - const match = line.match(/^-\s+\[([^\]]+)\]\s+(.+?)(?:\s*\(ๆดปๅŠจๆ—ถ้—ด:.*\))?$/); - if (match) { - const tag = match[1]; - const content = match[2].trim(); - const typePart = tag.includes("|") ? tag.split("|")[0] : tag; - return { content, score: 0, type: typePart }; - } - return { content: line, score: 0, type: "unknown" }; - }); - } - const tSearchEnd = performance.now(); - - // Read persona (L3 layer) - const tPersonaStart = performance.now(); - let personaContent: string | undefined; - try { - const personaPath = path.join(pluginDataDir, "persona.md"); - const raw = await fs.readFile(personaPath, "utf-8"); - personaContent = stripSceneNavigation(raw).trim(); - if (!personaContent) personaContent = undefined; - logger?.debug?.(`${TAG} Persona loaded: ${personaContent ? `${personaContent.length} chars` : "empty"}`); - } catch { - logger?.debug?.(`${TAG} No persona file found (expected for new users)`); - } - const tPersonaEnd = performance.now(); - - // Load full scene navigation (L2 layer) - const tSceneStart = performance.now(); - let sceneNavigation: string | undefined; - try { - const sceneIndex = await readSceneIndex(pluginDataDir); - if (sceneIndex.length > 0) { - sceneNavigation = generateSceneNavigation(sceneIndex); - logger?.debug?.(`${TAG} Scene navigation generated: ${sceneIndex.length} scenes`); - } - } catch { - logger?.debug?.(`${TAG} No scene index found`); - } - const tSceneEnd = performance.now(); - - if (memoryLines.length === 0 && !personaContent && !sceneNavigation) { - const totalMs = performance.now() - tRecallStart; - logger?.info( - `${TAG} โฑ Recall timing: total=${totalMs.toFixed(0)}ms, ` + - `search=${(tSearchEnd - tSearchStart).toFixed(0)}ms(strategy=${effectiveStrategy},hits=${memoryLines.length},` + - `fts=${searchTiming.ftsMs.toFixed(0)}ms/${searchTiming.ftsHits}hits,` + - `vec=${searchTiming.embeddingMs.toFixed(0)}ms/${searchTiming.embeddingHits}hits), ` + - `persona=${(tPersonaEnd - tPersonaStart).toFixed(0)}ms, ` + - `scene=${(tSceneEnd - tSceneStart).toFixed(0)}ms โ€” no context to inject`, - ); - logger?.debug?.(`${TAG} No memories/persona/scenes to inject`); - return undefined; - } - - // All memory context โ†’ appendSystemContext (system prompt end) - // Order: user-persona โ†’ scene-navigation โ†’ relevant-memories โ†’ tools-guide - const systemParts: string[] = []; - if (personaContent) { - systemParts.push(`\n${personaContent}\n`); - } - if (sceneNavigation) { - const pathHint = buildScenePathHint(pluginDataDir); - systemParts.push(`\n${sceneNavigation}\n\n${pathHint}\n`); - } - if (memoryLines.length > 0) { - systemParts.push( - `\nไปฅไธ‹ๆ˜ฏๅฝ“ๅ‰ๅฏน่ฏๅฌๅ›ž็š„็›ธๅ…ณ่ฎฐๅฟ†๏ผŒไธไปฃ่กจๅฝ“ๅ‰ไปปๅŠก่ฟ›็จ‹๏ผŒไป…ไฝœไธบๅ‚่€ƒ๏ผš\n\n${memoryLines.join("\n")}\n` - ); - } - - // Append memory tools usage guide so the agent knows how to actively - // retrieve deeper context when the injected snippets are not enough. - if (systemParts.length > 0) { - systemParts.push(MEMORY_TOOLS_GUIDE); - } - - const appendSystemContext = systemParts.length > 0 ? systemParts.join("\n\n") : undefined; - - const totalMs = performance.now() - tRecallStart; - logger?.info( - `${TAG} โฑ Recall timing: total=${totalMs.toFixed(0)}ms, ` + - `search=${(tSearchEnd - tSearchStart).toFixed(0)}ms(strategy=${effectiveStrategy},hits=${memoryLines.length},` + - `fts=${searchTiming.ftsMs.toFixed(0)}ms/${searchTiming.ftsHits}hits,` + - `vec=${searchTiming.embeddingMs.toFixed(0)}ms/${searchTiming.embeddingHits}hits), ` + - `persona=${(tPersonaEnd - tPersonaStart).toFixed(0)}ms(${personaContent ? `${personaContent.length}chars` : "none"}), ` + - `scene=${(tSceneEnd - tSceneStart).toFixed(0)}ms(${sceneNavigation ? "loaded" : "none"})`, - ); - - if (!appendSystemContext) { - return undefined; - } - - return { - appendSystemContext, - recalledL1Memories, - recalledL3Persona: personaContent ?? null, - recallStrategy: effectiveStrategy, - }; -} - -// ============================ -// Multi-strategy search dispatcher -// ============================ - -interface ScoredRecord { - record: MemoryRecord; - score: number; -} - -/** Timing breakdown from memory search */ -interface SearchTiming { - ftsMs: number; - embeddingMs: number; - ftsHits: number; - embeddingHits: number; -} - -interface SearchResult { - lines: string[]; - timing: SearchTiming; -} - -/** - * Search memories and return both formatted lines and structured details. - * - * This is a thin wrapper around `searchMemories` that also captures - * the recalled memory metadata for metric reporting (agent_turn event). - * It parses the returned formatted lines to extract type/content info. - */ -async function searchMemoriesWithDetails( - userText: string, - pluginDataDir: string, - cfg: MemoryTdaiConfig, - logger: Logger | undefined, - strategy: "keyword" | "embedding" | "hybrid", - vectorStore?: IMemoryStore, - embeddingService?: EmbeddingService, -): Promise<{ lines: string[]; memories: RecalledMemory[]; timing: SearchTiming }> { - const result = await searchMemories(userText, pluginDataDir, cfg, logger, strategy, vectorStore, embeddingService); - - // Extract structured data from formatted memory lines. - // Format: "- [type|scene] content (ๆดปๅŠจๆ—ถ้—ด: ...)" or "- [type] content" - const memories: RecalledMemory[] = result.lines.map((line) => { - const match = line.match(/^-\s+\[([^\]]+)\]\s+(.+?)(?:\s*\(ๆดปๅŠจๆ—ถ้—ด:.*\))?$/); - if (match) { - const tag = match[1]; - const content = match[2].trim(); - const typePart = tag.includes("|") ? tag.split("|")[0] : tag; - return { content, score: 0, type: typePart }; - } - return { content: line, score: 0, type: "unknown" }; - }); - - return { lines: result.lines, memories, timing: result.timing }; -} - -/** - * Search memories using the configured strategy. - * - * - "keyword": JSONL keyword-based (Jaccard similarity) โ€” no embedding needed - * - "embedding": VectorStore cosine similarity โ€” requires vectorStore + embeddingService - * - "hybrid": merge both keyword and embedding results with RRF (Reciprocal Rank Fusion) - * - * Falls back to keyword if embedding resources are unavailable. - */ -async function searchMemories( - userText: string, - pluginDataDir: string, - cfg: MemoryTdaiConfig, - logger: Logger | undefined, - strategy: "keyword" | "embedding" | "hybrid", - vectorStore?: IMemoryStore, - embeddingService?: EmbeddingService, -): Promise { - const emptyResult: SearchResult = { lines: [], timing: { ftsMs: 0, embeddingMs: 0, ftsHits: 0, embeddingHits: 0 } }; - // Strip gateway-injected inbound metadata (Sender, timestamps, media markers, - // base64 image data, etc.) so FTS / embedding queries are based on pure user intent. - const cleanText = sanitizeText(userText); - - if (cleanText.length < 2) { - logger?.debug?.(`${TAG} Query too short for memory search (raw=${userText.length}, clean=${cleanText.length})`); - return emptyResult; - } - - if (cleanText.length !== userText.length) { - logger?.debug?.( - `${TAG} userText sanitized: ${userText.length} โ†’ ${cleanText.length} chars`, - ); - } - - const maxResults = cfg.recall.maxResults ?? 5; - const threshold = cfg.recall.scoreThreshold ?? 0.3; - - const embeddingAvailable = !!vectorStore && !!embeddingService; - - logger?.debug?.( - `${TAG} [searchMemories] strategy=${strategy}, embeddingAvailable=${embeddingAvailable}, ` + - `vectorStore=${vectorStore ? "available" : "UNAVAILABLE"}, ` + - `embeddingService=${embeddingService ? "available" : "UNAVAILABLE"}, ` + - `maxResults=${maxResults}, threshold=${threshold}`, - ); - - // Determine effective strategy (fall back to keyword if embedding not available) - let effectiveStrategy = strategy; - if ((strategy === "embedding" || strategy === "hybrid") && !embeddingAvailable) { - logger?.warn?.( - `${TAG} Strategy "${strategy}" requested but EmbeddingService not available, falling back to keyword`, - ); - effectiveStrategy = "keyword"; - } - - logger?.debug?.(`${TAG} Search strategy: ${effectiveStrategy} (configured: ${strategy})`); - - try { - if (effectiveStrategy === "keyword") { - const tFts = performance.now(); - const lines = await searchByKeyword(cleanText, pluginDataDir, maxResults, threshold, logger, vectorStore); - return { lines, timing: { ftsMs: performance.now() - tFts, embeddingMs: 0, ftsHits: lines.length, embeddingHits: 0 } }; - } - - if (effectiveStrategy === "embedding") { - const tEmb = performance.now(); - const lines = await searchByEmbedding(cleanText, maxResults, threshold, vectorStore!, embeddingService!, logger); - return { lines, timing: { ftsMs: 0, embeddingMs: performance.now() - tEmb, ftsHits: 0, embeddingHits: lines.length } }; - } - - // Hybrid: run both keyword and embedding, merge with RRF - return await searchHybrid(cleanText, pluginDataDir, maxResults, threshold, vectorStore!, embeddingService!, logger); - } catch (err) { - logger?.warn?.(`${TAG} Memory search failed (strategy=${effectiveStrategy}): ${err instanceof Error ? err.message : String(err)}`); - return emptyResult; - } -} - -// ============================ -// Strategy: Keyword (FTS5 BM25, no in-memory fallback) -// ============================ - -async function searchByKeyword( - userText: string, - _pluginDataDir: string, - maxResults: number, - threshold: number, - logger?: Logger, - vectorStore?: IMemoryStore, -): Promise { - // Prefer FTS5 if available - if (vectorStore?.isFtsAvailable()) { - const ftsQuery = buildFtsQuery(userText); - if (ftsQuery) { - logger?.debug?.(`${TAG} [keyword-fts] Using FTS5 BM25 search: query="${ftsQuery}"`); - const ftsResults = await vectorStore.searchL1Fts(ftsQuery, maxResults * 2); - if (ftsResults.length > 0) { - logger?.debug?.( - `${TAG} [keyword-fts] FTS5 raw results (${ftsResults.length}): ` + - ftsResults.map((r) => `id=${r.record_id} score=${r.score.toFixed(6)}`).join(", "), - ); - const filtered = ftsResults - .filter((r) => r.score >= threshold) - .slice(0, maxResults); - - if (filtered.length > 0) { - logger?.debug?.(`${TAG} [keyword-fts] FTS5 found ${filtered.length} results (from ${ftsResults.length} raw, threshold=${threshold})`); - return filtered.map((r) => formatMemoryLine(ftsResultToFormatable(r))); - } - - // BM25 absolute scores are unreliable when the document set is very - // small (e.g. 1โ€“3 records) because IDF approaches 0. In that case, - // trust FTS5's MATCH + rank ordering and return the top results anyway. - if (ftsResults.length <= maxResults) { - logger?.debug?.( - `${TAG} [keyword-fts] All ${ftsResults.length} results below threshold=${threshold} ` + - `but document set is small โ€” returning all matched results`, - ); - return ftsResults.slice(0, maxResults).map((r) => formatMemoryLine(ftsResultToFormatable(r))); - } - logger?.debug?.(`${TAG} [keyword-fts] FTS5 returned 0 results above threshold (from ${ftsResults.length} raw)`); - } - } - } - - // FTS5 not available or returned no results โ€” skip in-memory fallback to avoid O(N) full scan - logger?.debug?.(`${TAG} [keyword] FTS5 unavailable or no results, skipping keyword search`); - return []; -} - -// ============================ -// Strategy: Embedding (VectorStore cosine) -// ============================ - -async function searchByEmbedding( - userText: string, - maxResults: number, - threshold: number, - vectorStore: IMemoryStore, - embeddingService: EmbeddingService, - logger?: Logger, -): Promise { - logger?.debug?.( - `${TAG} [embedding-search] START query="${userText.slice(0, 80)}...", maxResults=${maxResults}, threshold=${threshold}`, - ); - const queryEmbedding = await embeddingService.embed(userText); - logger?.debug?.( - `${TAG} [embedding-search] Query embedding OK: dims=${queryEmbedding.length}, ` + - `norm=${Math.sqrt(Array.from(queryEmbedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}, ` + - `searching top-${maxResults * 2}...`, - ); - // Retrieve more candidates for subsequent filtering - const vecResults: L1SearchResult[] = await vectorStore.searchL1Vector(queryEmbedding, maxResults * 2); - - if (vecResults.length === 0) { - logger?.debug?.(`${TAG} [embedding-search] Returned 0 results`); - return []; - } - - logger?.debug?.(`${TAG} [embedding-search] Got ${vecResults.length} candidates, filtering by threshold=${threshold}`); - for (const r of vecResults) { - logger?.debug?.( - `${TAG} [embedding-search] candidate id=${r.record_id}, score=${r.score.toFixed(4)}, ` + - `type=${r.type}, content="${r.content.slice(0, 60)}..."`, - ); - } - - const filtered = vecResults - .filter((r) => r.score >= threshold) - .slice(0, maxResults); - - if (filtered.length > 0) { - logger?.debug?.(`${TAG} [embedding-search] Found ${filtered.length} relevant memories above threshold (from ${vecResults.length} candidates)`); - return filtered.map((r) => formatMemoryLine(vectorResultToFormatable(r))); - } - - logger?.debug?.(`${TAG} [embedding-search] No results above threshold ${threshold}`); - return []; -} - -// ============================ -// Strategy: Hybrid (Keyword + Embedding + RRF) -// ============================ - -/** - * Hybrid search: run keyword (FTS5) and embedding in parallel, merge with - * Reciprocal Rank Fusion (RRF) to combine rank lists. - * - * RRF score for a record at rank r = 1 / (k + r), where k=60 is a constant. - * If a record appears in both lists, its RRF scores are summed. - * - * If FTS5 is unavailable, the keyword side returns empty and RRF uses - * embedding results only. - */ -async function searchHybrid( - userText: string, - _pluginDataDir: string, - maxResults: number, - _threshold: number, - vectorStore: IMemoryStore, - embeddingService: EmbeddingService, - logger?: Logger, -): Promise { - // Run keyword and embedding searches in parallel - const candidateK = maxResults * 3; // retrieve more for merging - - const [keywordResult, embeddingResult] = await Promise.all([ - // Keyword search: FTS5 only (no in-memory fallback) - (async () => { - const tStart = performance.now(); - try { - // Try FTS5 first - if (vectorStore.isFtsAvailable()) { - const ftsQuery = buildFtsQuery(userText); - if (ftsQuery) { - const ftsResults = await vectorStore.searchL1Fts(ftsQuery, candidateK); - if (ftsResults.length > 0) { - logger?.debug?.(`${TAG} [hybrid-keyword-fts] FTS5 found ${ftsResults.length} candidates`); - // Convert FtsSearchResult to ScoredRecord for RRF merge - const records = ftsResults.map((r): ScoredRecord => ({ - record: { - id: r.record_id, - content: r.content, - type: r.type as MemoryRecord["type"], - priority: r.priority, - scene_name: r.scene_name, - source_message_ids: [], - metadata: r.metadata_json ? (() => { try { return JSON.parse(r.metadata_json); } catch { return {}; } })() : {}, - timestamps: [r.timestamp_str].filter(Boolean), - createdAt: "", - updatedAt: "", - sessionKey: r.session_key, - sessionId: r.session_id, - }, - score: r.score, - })); - return { records, ms: performance.now() - tStart }; - } - } - } - // FTS5 not available or returned no results โ€” skip in-memory fallback - logger?.debug?.(`${TAG} [hybrid-keyword] FTS5 unavailable or no results, skipping keyword part`); - return { records: [] as ScoredRecord[], ms: performance.now() - tStart }; - } catch (err) { - logger?.warn?.(`${TAG} Hybrid: keyword part failed: ${err instanceof Error ? err.message : String(err)}`); - return { records: [] as ScoredRecord[], ms: performance.now() - tStart }; - } - })(), - // Embedding search - (async () => { - const tStart = performance.now(); - try { - logger?.debug?.(`${TAG} [hybrid-embedding] Generating query embedding...`); - const queryEmbedding = await embeddingService.embed(userText); - logger?.debug?.( - `${TAG} [hybrid-embedding] Embedding OK, dims=${queryEmbedding.length}, searching top-${candidateK}...`, - ); - const results = await vectorStore.searchL1Vector(queryEmbedding, candidateK, userText); - logger?.debug?.(`${TAG} [hybrid-embedding] Got ${results.length} candidates`); - return { results, ms: performance.now() - tStart }; - } catch (err) { - logger?.warn?.(`${TAG} Hybrid: embedding part failed: ${err instanceof Error ? err.message : String(err)}`); - return { results: [] as L1SearchResult[], ms: performance.now() - tStart }; - } - })(), - ]); - - const keywordResults = keywordResult.records; - const embeddingResults = embeddingResult.results; - const timing: SearchTiming = { - ftsMs: keywordResult.ms, - embeddingMs: embeddingResult.ms, - ftsHits: keywordResults.length, - embeddingHits: embeddingResults.length, - }; - - if (keywordResults.length === 0 && embeddingResults.length === 0) { - logger?.debug?.(`${TAG} Hybrid search: both strategies returned 0 results`); - return { lines: [], timing }; - } - - // RRF merge: k=60 is a standard constant from the RRF paper - const RRF_K = 60; - - // Map: record_id โ†’ { rrfScore, formatable } - const mergedMap = new Map(); - - // Process keyword results - for (let rank = 0; rank < keywordResults.length; rank++) { - const r = keywordResults[rank]; - const id = r.record.id; - const rrfScore = 1 / (RRF_K + rank + 1); - const existing = mergedMap.get(id); - if (existing) { - existing.rrfScore += rrfScore; - } else { - mergedMap.set(id, { rrfScore, formatable: recordToFormatable(r.record) }); - } - } - - // Process embedding results - for (let rank = 0; rank < embeddingResults.length; rank++) { - const r = embeddingResults[rank]; - const id = r.record_id; - const rrfScore = 1 / (RRF_K + rank + 1); - const existing = mergedMap.get(id); - if (existing) { - existing.rrfScore += rrfScore; - } else { - mergedMap.set(id, { rrfScore, formatable: vectorResultToFormatable(r) }); - } - } - - // Sort by combined RRF score and take top results - const sorted = [...mergedMap.entries()] - .sort((a, b) => b[1].rrfScore - a[1].rrfScore) - .slice(0, maxResults); - - if (sorted.length > 0) { - logger?.debug?.( - `${TAG} Hybrid search found ${sorted.length} results ` + - `(keyword=${keywordResults.length}, embedding=${embeddingResults.length})`, - ); - return { lines: sorted.map(([, { formatable }]) => formatMemoryLine(formatable)), timing }; - } - - logger?.debug?.(`${TAG} Hybrid search: no results after merge`); - return { lines: [], timing }; -} - -// ============================ -// Unified memory line formatter -// ============================ - -/** - * Format a single memory record into a rich natural-language line for prompt injection. - * - * Time semantics: - * - timestamp (็‚นๆ—ถ้—ด): when the activity/event happened, e.g. "2025-03-01 mentioned something" - * - activity_start_time / activity_end_time (ๆฎตๆ—ถ้—ด): activity time range, e.g. "trip from 2025-05-01 to 2025-05-10" - * - All three time fields may be empty/undefined โ€” handled gracefully. - * - * Output examples: - * - [persona] ็”จๆˆทๅซ็Ž‹ๅฐๆ˜Ž๏ผŒ30ๅฒ๏ผŒๆ˜ฏไธ€ๅ่ฝฏไปถๅทฅ็จ‹ๅธˆใ€‚ - * - [episodic|ๆ—…่กŒ่ฎกๅˆ’] ็”จๆˆท่ฎกๅˆ’ไบ”ๆœˆๅŽปๆ—ฅๆœฌๆ—…่กŒใ€‚(ๆดปๅŠจๆ—ถ้—ด: 2025-05-01 ~ 2025-05-10) - * - [episodic] ็”จๆˆทไปŠๅคฉๅŠ ็ญๅˆฐๅพˆๆ™šใ€‚(ๆดปๅŠจๆ—ถ้—ด: 2025-03-01) - * - [instruction] ็”จๆˆท่ฆๆฑ‚ๅ›ž็ญ”ๆ—ถไฝฟ็”จไธญๆ–‡๏ผŒไฟๆŒ็ฎ€ๆดใ€‚ - */ -interface FormatableMemory { - type: string; - content: string; - scene_name?: string; - /** Activity time range start (ๆฎตๆ—ถ้—ด start), may be empty */ - activity_start_time?: string; - /** Activity time range end (ๆฎตๆ—ถ้—ด end), may be empty */ - activity_end_time?: string; - /** Activity point-in-time (็‚นๆ—ถ้—ด: when it happened), may be empty */ - timestamp?: string; -} - -function formatMemoryLine(m: FormatableMemory): string { - // 1. Type tag + optional scene name - const tag = m.scene_name ? `${m.type}|${m.scene_name}` : m.type; - - // 2. Content (core) - let line = `- [${tag}] ${m.content}`; - - // 3. Time info โ€” prefer activity_start/end range; fall back to timestamp as point-in-time - const start = formatTimestamp(m.activity_start_time); - const end = formatTimestamp(m.activity_end_time); - const point = formatTimestamp(m.timestamp); - - if (start && end) { - // ๆฎตๆ—ถ้—ด: both start and end - line += ` (ๆดปๅŠจๆ—ถ้—ด: ${start} ~ ${end})`; - } else if (start) { - // ๆฎตๆ—ถ้—ด: only start - line += ` (ๆดปๅŠจๆ—ถ้—ด: ${start}่ตท)`; - } else if (end) { - // ๆฎตๆ—ถ้—ด: only end - line += ` (ๆดปๅŠจๆ—ถ้—ด: ่‡ณ${end})`; - } else if (point) { - // ็‚นๆ—ถ้—ด: single timestamp - line += ` (ๆดปๅŠจๆ—ถ้—ด: ${point})`; - } - // If all three are empty โ†’ no time info appended (graceful) - - return line; -} - -/** - * Format an ISO 8601 timestamp to a concise date or datetime string. - * - If the time part is 00:00:00 โ†’ show date only (e.g. "2025-03-01") - * - Otherwise โ†’ show date + time (e.g. "2025-03-01 14:30") - * - Returns undefined for empty/invalid inputs. - */ -function formatTimestamp(ts: string | undefined): string | undefined { - if (!ts) return undefined; - // Try to parse ISO format: "2025-03-01T14:30:00.000Z" or "2025-03-01" - const match = ts.match(/^(\d{4}-\d{2}-\d{2})(?:T(\d{2}:\d{2})(?::\d{2})?)?/); - if (!match) return undefined; - const datePart = match[1]; - const timePart = match[2]; - if (!timePart || timePart === "00:00") { - return datePart; - } - return `${datePart} ${timePart}`; -} - -/** - * Build a FormatableMemory from a full MemoryRecord (keyword search path). - * Handles empty metadata, empty timestamps array gracefully. - */ -function recordToFormatable(record: MemoryRecord): FormatableMemory { - const meta = record.metadata as { activity_start_time?: string; activity_end_time?: string } | undefined; - return { - type: record.type, - content: record.content, - scene_name: record.scene_name || undefined, - activity_start_time: meta?.activity_start_time || undefined, - activity_end_time: meta?.activity_end_time || undefined, - timestamp: (record.timestamps && record.timestamps.length > 0) ? record.timestamps[0] : undefined, - }; -} - -/** - * Build a FormatableMemory from a VectorSearchResult (embedding search path). - * Handles empty/invalid metadata_json, empty timestamp_str gracefully. - */ -function vectorResultToFormatable(r: L1SearchResult): FormatableMemory { - let activityStart: string | undefined; - let activityEnd: string | undefined; - if (r.metadata_json && r.metadata_json !== "{}") { - try { - const meta = typeof r.metadata_json === "string" ? JSON.parse(r.metadata_json) : r.metadata_json; - activityStart = meta?.activity_start_time || undefined; - activityEnd = meta?.activity_end_time || undefined; - } catch { /* ignore parse errors โ€” treat as no metadata */ } - } - return { - type: r.type, - content: r.content, - scene_name: r.scene_name || undefined, - activity_start_time: activityStart, - activity_end_time: activityEnd, - timestamp: r.timestamp_str || undefined, - }; -} - -/** - * Build a FormatableMemory from an FtsSearchResult (FTS5 keyword search path). - * Handles empty/invalid metadata_json, empty timestamp_str gracefully. - */ -function ftsResultToFormatable(r: L1FtsResult): FormatableMemory { - let activityStart: string | undefined; - let activityEnd: string | undefined; - if (r.metadata_json && r.metadata_json !== "{}") { - try { - const meta = typeof r.metadata_json === "string" ? JSON.parse(r.metadata_json) : r.metadata_json; - activityStart = meta?.activity_start_time || undefined; - activityEnd = meta?.activity_end_time || undefined; - } catch { /* ignore parse errors โ€” treat as no metadata */ } - } - return { - type: r.type, - content: r.content, - scene_name: r.scene_name || undefined, - activity_start_time: activityStart, - activity_end_time: activityEnd, - timestamp: r.timestamp_str || undefined, - }; -} diff --git a/src/offload/hooks/llm-input-l3.ts b/src/offload/hooks/llm-input-l3.ts index 0671997..56a064c 100644 --- a/src/offload/hooks/llm-input-l3.ts +++ b/src/offload/hooks/llm-input-l3.ts @@ -494,7 +494,8 @@ export function compressByScoreCascade( for (const tuId of c.allToolUseIds) { replacedIds.add(tuId); replacedToolCallIdList.push(tuId); - const tuEntry = c.allOffloadEntries.find((e: OffloadEntry) => e.tool_call_id === tuId); + const tuIdNorm = normalizeToolCallIdForLookup(tuId); + const tuEntry = c.allOffloadEntries.find((e: OffloadEntry) => e.tool_call_id === tuId || e.tool_call_id === tuIdNorm || normalizeToolCallIdForLookup(e.tool_call_id) === tuIdNorm); replacedDetails.push({ toolCallId: tuId, score: c.score, summaryPreview: (tuEntry?.summary ?? "").slice(0, 120) }); } for (let ei = 0; ei < c.allToolUseIds.length; ei++) { diff --git a/src/offload/index.ts b/src/offload/index.ts index 349f364..33283df 100644 --- a/src/offload/index.ts +++ b/src/offload/index.ts @@ -50,6 +50,7 @@ import { findHistoryMmdInsertionPoint } from "./mmd-injector.js"; import type { OffloadConfig } from "../config.js"; import type { PluginConfig, PluginLogger } from "./types.js"; import { BackendClient } from "./backend-client.js"; +import { LocalLlmClient } from "./local-llm/index.js"; import type { L1Request, L15Request, L2Request } from "./backend-client.js"; import { parseMmdMeta } from "./mmd-meta.js"; import { sanitizeText, writeRefMd } from "./storage.js"; @@ -299,36 +300,91 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { } const sessions = _sharedSessions; - // Resolve LLM Configuration โ€” backend mode only - // Backend Client (required for L1/L1.5/L2/L4 โ€” but context engine still registers for L3) + // Resolve LLM Configuration โ€” mode-based selection + // - "backend": use remote backend service (requires backendUrl) + // - "local": call LLM directly via AI SDK (uses offload.model or main agent model) // // User identity: prefer offloadConfig.userId; fall back to the host's - // primary non-loopback IPv4 address. The backend `/offload/v1/store` - // endpoint upserts state keyed by this value (as Mongo `_id`). + // primary non-loopback IPv4 address. const _resolvedUserId = resolveUserId(offloadConfig.userId ?? null); logger.debug?.( `[context-offload] user-id resolved: "${_resolvedUserId}" (source=${getUserIdSource() ?? "?"})`, ); - const backendClient = offloadConfig.backendUrl - ? new BackendClient( + let backendClient: BackendClient | LocalLlmClient | null = null; + + if (offloadConfig.mode === "backend") { + // Remote backend mode + if (!offloadConfig.backendUrl) { + logger.error("[context-offload] mode=backend but backendUrl not configured. L1/L1.5/L2/L4 disabled."); + } else { + backendClient = new BackendClient( offloadConfig.backendUrl, logger, offloadConfig.backendApiKey, offloadConfig.backendTimeoutMs, () => _lastActiveSessionKey, () => _resolvedUserId, - () => _lastActiveSessionKey, // use sessionKey as task scope โ€” coarse but stable - ) - : null; + () => _lastActiveSessionKey, + ); + } + } else { + // Local LLM mode โ€” resolve model from offload.model or fall back to agents.defaults.model + let resolvedModelRef = offloadConfig.model; + if (!resolvedModelRef) { + // Fallback: use main agent model from openclaw.json agents.defaults.model + const mainConfig = api.config as Record | undefined; + const agents = mainConfig?.agents as Record | undefined; + const defaults = agents?.defaults as Record | undefined; + const modelCfg = defaults?.model; + if (typeof modelCfg === "string" && modelCfg.includes("/")) { + resolvedModelRef = modelCfg; + } else if (modelCfg && typeof modelCfg === "object") { + const primary = (modelCfg as Record).primary; + if (typeof primary === "string" && primary.includes("/")) { + resolvedModelRef = primary; + } + } + if (resolvedModelRef) { + logger.info(`[context-offload] offload.model not set, using main agent model: ${resolvedModelRef}`); + } + } + + if (resolvedModelRef) { + const modelParts = resolvedModelRef.split("/", 2); + const providerKey = modelParts[0]; + const modelId = modelParts[1] ?? resolvedModelRef; + const models = (api.config as any)?.models; + const providerCfg = models?.providers?.[providerKey]; + const baseUrl = providerCfg?.baseUrl ?? providerCfg?.baseURL; + const apiKey = providerCfg?.apiKey; + + if (baseUrl && apiKey) { + backendClient = new LocalLlmClient( + { baseUrl, apiKey, model: modelId, temperature: offloadConfig.temperature, timeoutMs: offloadConfig.backendTimeoutMs }, + logger, + ); + } else { + logger.error( + `[context-offload] Local LLM mode failed: provider "${providerKey}" not found or missing baseUrl/apiKey in models.providers. ` + + `L1/L1.5/L2 disabled.`, + ); + } + } else { + logger.warn("[context-offload] No model resolved (offload.model not set, agents.defaults.model not found). L1/L1.5/L2 disabled."); + } + } // Track last active session key for BackendClient header let _lastActiveSessionKey: string | null = null; - if (backendClient) { - logger.debug?.(`[context-offload] Backend mode: ${offloadConfig.backendUrl}`); + if (backendClient && offloadConfig.mode === "backend") { + logger.debug?.(`[context-offload] LLM mode: backend (${offloadConfig.backendUrl})`); + } else if (backendClient) { + logger.debug?.(`[context-offload] LLM mode: local (${offloadConfig.model ?? "main-agent-model"})`); + } else { - logger.warn("[context-offload] backendUrl not configured. L1/L1.5/L2/L4 disabled (L3 compression still active)."); + logger.warn("[context-offload] LLM client not available. L1/L1.5/L2/L4 disabled (L3 compression still active)."); } // โ”€โ”€โ”€ Fault tolerance constants โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ @@ -361,7 +417,7 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { const beforeFilter = takenPairs.length; const pairs = takenPairs.filter((p) => !isHeartbeat(p)); if (beforeFilter > pairs.length) { - logger.info(`[context-offload] Backend L1: filtered ${beforeFilter - pairs.length} heartbeat pair(s)`); + logger.info(`[context-offload] L1: filtered ${beforeFilter - pairs.length} heartbeat pair(s)`); } if (pairs.length === 0) return; @@ -376,7 +432,7 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { const refPath = await writeRefMd(stateManager.ctx, p.timestamp, p.toolName, content); refByToolCallId.set(p.toolCallId, refPath); } catch (err) { - logger.error(`[context-offload] Backend L1.1 ref write error (${p.toolCallId}): ${err}`); + logger.error(`[context-offload] L1.1 ref write error (${p.toolCallId}): ${err}`); } } @@ -385,7 +441,7 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { for (let i = 0; i < pairs.length; i += L1_BATCH_SIZE) { batches.push(pairs.slice(i, i + L1_BATCH_SIZE)); } - logger.info(`[context-offload] Backend L1 (${triggerSource}): ${pairs.length} pairs โ†’ ${batches.length} batch(es) of โ‰ค${L1_BATCH_SIZE}`); + logger.info(`[context-offload] L1 (${triggerSource}): ${pairs.length} pairs โ†’ ${batches.length} batch(es) of โ‰ค${L1_BATCH_SIZE}`); const recentMessages = _buildL1RecentContext(stateManager); logger.info(`[context-offload] L1 recentMessages (${recentMessages.length} chars):\n${recentMessages}`); @@ -417,15 +473,15 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { } await appendOffloadEntries(stateManager.ctx, resp.entries, undefined, logger); stateManager.entryCounter += resp.entries.length; - logger.info(`[context-offload] Backend L1 batch OK: ${resp.entries.length} entries from ${chunk.length} pairs (entryCounter=${stateManager.entryCounter})`); + logger.info(`[context-offload] L1 batch OK: ${resp.entries.length} entries from ${chunk.length} pairs (entryCounter=${stateManager.entryCounter})`); } } catch (err) { const newFails = prevFails + 1; - logger.warn(`[context-offload] Backend L1 batch FAILED (${chunkKey}, attempt ${newFails}/${MAX_L1_CHUNK_RETRIES}): ${err}`); + logger.warn(`[context-offload] L1 batch FAILED (${chunkKey}, attempt ${newFails}/${MAX_L1_CHUNK_RETRIES}): ${err}`); if (newFails >= MAX_L1_CHUNK_RETRIES) { // Exceeded retry limit โ€” generate local fallback entries (no LLM summary) - logger.warn(`[context-offload] Backend L1 batch DEGRADED: ${chunk.length} pairs โ†’ fallback entries (no LLM summary)`); + logger.warn(`[context-offload] L1 batch DEGRADED: ${chunk.length} pairs โ†’ fallback entries (no LLM summary)`); stateManager._l1ChunkFailCounts.delete(chunkKey); const fallbackEntries: import("./types.js").OffloadEntry[] = []; for (const p of chunk) { @@ -446,7 +502,7 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { } await appendOffloadEntries(stateManager.ctx, fallbackEntries, undefined, logger); stateManager.entryCounter += fallbackEntries.length; - logger.info(`[context-offload] Backend L1 fallback: wrote ${fallbackEntries.length} degraded entries`); + logger.info(`[context-offload] L1 fallback: wrote ${fallbackEntries.length} degraded entries`); } else { // Under retry limit โ€” re-enqueue this chunk for next flush stateManager._l1ChunkFailCounts.set(chunkKey, newFails); @@ -454,7 +510,7 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { stateManager.processedToolCallIds.delete(p.toolCallId); stateManager.pendingToolPairs.push(p as any); } - logger.info(`[context-offload] Backend L1 batch: re-enqueued ${chunk.length} pairs (retry ${newFails}/${MAX_L1_CHUNK_RETRIES})`); + logger.info(`[context-offload] L1 batch: re-enqueued ${chunk.length} pairs (retry ${newFails}/${MAX_L1_CHUNK_RETRIES})`); } } } @@ -512,13 +568,13 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { // Normalize backend response (handles null fields from fallback) const judgment = normalizeJudgment(resp as unknown as Record); if (!judgment) { - logger.warn("[context-offload] Backend L1.5: all-null response (backend LLM unavailable)"); + logger.warn("[context-offload] L1.5: all-null response (backend LLM unavailable)"); return false; // trigger retry } // Success logger.info( - `[context-offload] Backend L1.5: completed=${judgment.taskCompleted}, continuation=${judgment.isContinuation}, longTask=${judgment.isLongTask}, label=${judgment.newTaskLabel ?? "none"}, contFile=${judgment.continuationMmdFile ?? "none"}`, + `[context-offload] L1.5: completed=${judgment.taskCompleted}, continuation=${judgment.isContinuation}, longTask=${judgment.isLongTask}, label=${judgment.newTaskLabel ?? "none"}, contFile=${judgment.continuationMmdFile ?? "none"}`, ); // โ”€โ”€ Flush residual null entries for the OLD mmd before task transition โ”€โ”€ @@ -578,10 +634,10 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { await stateManager.save(); stateManager.setMmdInjectionReady(true); stateManager.l15Settled = true; - logger.info("[context-offload] Backend L1.5: settled, MMD injection ready"); + logger.info("[context-offload] L1.5: settled, MMD injection ready"); return true; } catch (err) { - logger.warn(`[context-offload] Backend L1.5 attempt failed: ${err}`); + logger.warn(`[context-offload] L1.5 attempt failed: ${err}`); return false; } }; @@ -634,7 +690,7 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { for (let i = 0; i < mmdEntries.length; i += L2_BATCH_SIZE) { batches.push(mmdEntries.slice(i, i + L2_BATCH_SIZE)); } - logger.info(`[context-offload] Backend L2 (${triggerSource}): mmd=${mmdFile}, ${mmdEntries.length} entries โ†’ ${batches.length} batch(es) of โ‰ค${L2_BATCH_SIZE}`); + logger.info(`[context-offload] L2 (${triggerSource}): mmd=${mmdFile}, ${mmdEntries.length} entries โ†’ ${batches.length} batch(es) of โ‰ค${L2_BATCH_SIZE}`); for (let bIdx = 0; bIdx < batches.length; bIdx++) { const batch = batches[bIdx]; @@ -678,7 +734,7 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { // Handle backend degraded response (empty fileAction = LLM unavailable) if (!resp.fileAction) { - logger.warn(`[context-offload] Backend L2 [${mmdFile}] batch ${bIdx + 1}/${batches.length}: degraded response, applying fallback backfill`); + logger.warn(`[context-offload] L2 [${mmdFile}] batch ${bIdx + 1}/${batches.length}: degraded response, applying fallback backfill`); await backfillNodeIds(stateManager.ctx, resp.nodeMapping ?? {}, batchWaitIds, logger, { mmdFallbackText: existingMmd ?? "", mmdPrefix, @@ -689,14 +745,14 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { // Apply MMD file changes if (resp.fileAction === "replace" && resp.replaceBlocks && resp.replaceBlocks.length > 0) { const patchOk = await patchMmd(stateManager.ctx, mmdFile, resp.replaceBlocks); - logger.info(`[context-offload] Backend L2 [${mmdFile}] batch ${bIdx + 1}/${batches.length}: patchMmd: ${patchOk ? "ok" : "FAILED"} (${resp.replaceBlocks.length} blocks)`); + logger.info(`[context-offload] L2 [${mmdFile}] batch ${bIdx + 1}/${batches.length}: patchMmd: ${patchOk ? "ok" : "FAILED"} (${resp.replaceBlocks.length} blocks)`); if (!patchOk && resp.mmdContent) { await writeMmd(stateManager.ctx, mmdFile, resp.mmdContent); - logger.info(`[context-offload] Backend L2 [${mmdFile}] batch ${bIdx + 1}/${batches.length}: fallback writeMmd: ${resp.mmdContent.length} chars`); + logger.info(`[context-offload] L2 [${mmdFile}] batch ${bIdx + 1}/${batches.length}: fallback writeMmd: ${resp.mmdContent.length} chars`); } } else if (resp.mmdContent) { await writeMmd(stateManager.ctx, mmdFile, resp.mmdContent); - logger.info(`[context-offload] Backend L2 [${mmdFile}] batch ${bIdx + 1}/${batches.length}: writeMmd: ${resp.mmdContent.length} chars`); + logger.info(`[context-offload] L2 [${mmdFile}] batch ${bIdx + 1}/${batches.length}: writeMmd: ${resp.mmdContent.length} chars`); } // Backfill node_ids @@ -712,15 +768,15 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { mmdPrefix, }); - logger.info(`[context-offload] Backend L2 [${mmdFile}] batch ${bIdx + 1}/${batches.length} (${triggerSource}): applied, action=${resp.fileAction}, mapping=${Object.keys(resp.nodeMapping ?? {}).length}`); + logger.info(`[context-offload] L2 [${mmdFile}] batch ${bIdx + 1}/${batches.length} (${triggerSource}): applied, action=${resp.fileAction}, mapping=${Object.keys(resp.nodeMapping ?? {}).length}`); } catch (err) { - logger.error(`[context-offload] Backend L2 [${mmdFile}] batch ${bIdx + 1}/${batches.length} failed: ${err}`); + logger.error(`[context-offload] L2 [${mmdFile}] batch ${bIdx + 1}/${batches.length} failed: ${err}`); // Continue with remaining batches โ€” failed entries stay as "wait" for retry } } } } catch (err) { - logger.error(`[context-offload] Backend L2 failed: ${err}`); + logger.error(`[context-offload] L2 failed: ${err}`); } }; @@ -745,12 +801,13 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { nodeIds.add(match[1]); } const filtered = allEntries.filter((e) => e.node_id && nodeIds.has(e.node_id)); - const resp = await backendClient.l4Generate({ + const resp = await (backendClient as any).l4Generate({ mmdFilename, mmdContent, offloadEntries: filtered, skillFocus: skillCommand.skillFocus, }); + if (!resp) return null; // Write skill file locally const { mkdir, writeFile } = await import("node:fs/promises"); const { join } = await import("node:path"); @@ -960,7 +1017,7 @@ export function registerOffload(api: any, offloadConfig: OffloadConfig): void { logger.warn(`[context-offload] <<< after_tool_call SKIP: no session manager (${Date.now() - _atcStart}ms)`); return; } - const afterToolCallHandler = createAfterToolCallHandler(_mgr, logger, getContextWindow, pCfg, backendClient); + const afterToolCallHandler = createAfterToolCallHandler(_mgr, logger, getContextWindow, pCfg, backendClient as any); await afterToolCallHandler(event, ctx); const _handlerDone = Date.now(); logger.info(`[context-offload] after_tool_call handler done: ${_handlerDone - _atcStart}ms`); diff --git a/src/offload/local-llm/index.ts b/src/offload/local-llm/index.ts new file mode 100644 index 0000000..78b4782 --- /dev/null +++ b/src/offload/local-llm/index.ts @@ -0,0 +1,170 @@ +/** + * LocalLlmClient โ€” local-mode offload LLM client. + * + * Implements the same interface as BackendClient (l1Summarize, l15Judge, l2Generate) + * but calls the LLM directly via AI SDK instead of routing through a remote backend. + * + * Used when `offload.model` is configured and `offload.backendUrl` is not set. + */ +import { callLlm, type LlmCallerConfig } from "./llm-caller.js"; +import { L1_SYSTEM_PROMPT, buildL1UserPrompt, type L1ToolPair } from "./prompts/l1-prompt.js"; +import { L15_SYSTEM_PROMPT, buildL15UserPrompt, type L15CurrentMmd, type L15MmdMeta } from "./prompts/l15-prompt.js"; +import { L2_SYSTEM_PROMPT, buildL2UserPrompt, type L2NewEntry } from "./prompts/l2-prompt.js"; +import { parseL1Response } from "./parsers/l1-parser.js"; +import { parseL15Response } from "./parsers/l15-parser.js"; +import { parseL2Response, type L2ParsedResponse } from "./parsers/l2-parser.js"; +import type { OffloadEntry, TaskJudgment, PluginLogger } from "../types.js"; +import type { L1Request, L1Response, L15Request, L15Response, L2Request, L2Response } from "../backend-client.js"; + +const TAG = "[context-offload] [local-llm]"; + +export interface LocalLlmClientConfig { + baseUrl: string; + apiKey: string; + model: string; + temperature?: number; + timeoutMs?: number; +} + +export class LocalLlmClient { + private config: LlmCallerConfig; + private logger?: PluginLogger; + + constructor(cfg: LocalLlmClientConfig, logger?: PluginLogger) { + this.config = { + baseUrl: cfg.baseUrl, + apiKey: cfg.apiKey, + model: cfg.model, + temperature: cfg.temperature ?? 0.2, + timeoutMs: cfg.timeoutMs ?? 120_000, + }; + this.logger = logger; + logger?.info?.(`${TAG} Initialized: model=${cfg.model}, baseUrl=${cfg.baseUrl}`); + } + + // โ”€โ”€โ”€ L1 Summarize โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + async l1Summarize(req: L1Request): Promise { + const pairs: L1ToolPair[] = req.toolPairs.map((p) => ({ + toolName: p.toolName, + toolCallId: p.toolCallId, + params: p.params, + result: p.result, + timestamp: p.timestamp, + })); + + const userPrompt = buildL1UserPrompt(req.recentMessages, pairs); + + const raw = await callLlm(this.config, { + systemPrompt: L1_SYSTEM_PROMPT, + userPrompt, + label: "L1", + }, this.logger); + + const entries = parseL1Response(raw); + if (entries.length === 0) { + this.logger?.warn?.(`${TAG} L1: parsed 0 entries from LLM response (${raw.length} chars)`); + } + + return { entries }; + } + + // โ”€โ”€โ”€ L1.5 Judge โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + async l15Judge(req: L15Request): Promise { + const currentMmd: L15CurrentMmd | null = req.currentMmd + ? { filename: req.currentMmd.filename, content: req.currentMmd.content, path: req.currentMmd.path } + : null; + + const metas: L15MmdMeta[] = req.availableMmdMetas.map((m) => ({ + filename: m.filename, + path: m.path, + taskGoal: m.taskGoal, + doneCount: m.doneCount, + doingCount: m.doingCount, + todoCount: m.todoCount, + updatedTime: m.updatedTime, + nodeSummaries: m.nodeSummaries?.map((n) => ({ + nodeId: n.nodeId, + status: n.status, + summary: n.summary, + })), + })); + + const userPrompt = buildL15UserPrompt(req.recentMessages, currentMmd, metas); + + const raw = await callLlm(this.config, { + systemPrompt: L15_SYSTEM_PROMPT, + userPrompt, + label: "L1.5", + }, this.logger); + + const result = parseL15Response(raw); + if (!result) { + this.logger?.warn?.(`${TAG} L1.5: failed to parse judgment from LLM response (${raw.length} chars)`); + // Return all-null to trigger normalizeJudgment's "LLM unavailable" path + return { + taskCompleted: false, + isContinuation: false, + isLongTask: false, + } as L15Response; + } + + return result as L15Response; + } + + // โ”€โ”€โ”€ L2 Generate โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + async l2Generate(req: L2Request): Promise { + const entries: L2NewEntry[] = req.newEntries.map((e) => ({ + toolCallId: e.tool_call_id, + toolCall: e.tool_call, + summary: e.summary, + timestamp: e.timestamp, + })); + + const userPrompt = buildL2UserPrompt({ + existingMmd: req.existingMmd, + entries, + recentHistory: req.recentHistory, + currentTurn: req.currentTurn, + taskLabel: req.taskLabel, + mmdPrefix: req.mmdPrefix, + charCount: req.mmdCharCount, + }); + + const raw = await callLlm(this.config, { + systemPrompt: L2_SYSTEM_PROMPT, + userPrompt, + label: "L2", + timeoutMs: 120_000, // L2 may take longer due to complex prompts + }, this.logger); + + const result = parseL2Response(raw); + if (!result) { + this.logger?.error?.(`${TAG} L2: failed to parse response (${raw.length} chars)`); + throw new Error("L2 response parsing failed"); + } + + return { + fileAction: result.fileAction, + mmdContent: result.mmdContent, + replaceBlocks: result.replaceBlocks?.map((b) => ({ + startLine: b.startLine, + endLine: b.endLine, + content: b.content, + })), + nodeMapping: result.nodeMapping, + }; + } + + // โ”€โ”€โ”€ Stubs (not applicable in local mode) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + /** No-op in local mode โ€” state reporting requires a remote backend. */ + async storeState(_payload: unknown): Promise {} + + /** L4 Skill generation is not supported in local mode. */ + async l4Generate(_req: unknown): Promise { + return null; + } +} diff --git a/src/offload/local-llm/llm-caller.ts b/src/offload/local-llm/llm-caller.ts new file mode 100644 index 0000000..95ae691 --- /dev/null +++ b/src/offload/local-llm/llm-caller.ts @@ -0,0 +1,80 @@ +/** + * Unified LLM caller for offload local mode. + * + * Uses Vercel AI SDK (`ai` + `@ai-sdk/openai`) with "compatible" mode + * to support any OpenAI-compatible backend. + */ +import { generateText } from "ai"; +import { createOpenAI } from "@ai-sdk/openai"; +import type { PluginLogger } from "../types.js"; + +const TAG = "[context-offload] [local-llm]"; + +export interface LlmCallerConfig { + baseUrl: string; + apiKey: string; + model: string; + temperature: number; + timeoutMs: number; +} + +export interface CallLlmOpts { + systemPrompt: string; + userPrompt: string; + /** Override temperature for this call */ + temperature?: number; + /** Override timeout for this call */ + timeoutMs?: number; + /** Label for logging (e.g. "L1", "L1.5", "L2") */ + label?: string; +} + +/** + * Call LLM with the given prompts and return the text response. + * Throws on timeout or API errors. + */ +export async function callLlm( + config: LlmCallerConfig, + opts: CallLlmOpts, + logger?: PluginLogger, +): Promise { + const startMs = Date.now(); + const label = opts.label ?? "call"; + const temperature = opts.temperature ?? config.temperature; + const timeoutMs = opts.timeoutMs ?? config.timeoutMs; + + logger?.info?.( + `${TAG} ${label} >>> model=${config.model}, temp=${temperature}, timeout=${timeoutMs}ms, ` + + `systemLen=${opts.systemPrompt.length}, userLen=${opts.userPrompt.length}`, + ); + + const provider = createOpenAI({ + baseURL: config.baseUrl, + apiKey: config.apiKey, + compatibility: "compatible", + }); + + try { + const result = await generateText({ + model: provider.chat(config.model), + system: opts.systemPrompt, + prompt: opts.userPrompt, + temperature, + abortSignal: AbortSignal.timeout(timeoutMs), + }); + + const text = result.text.trim(); + const elapsedMs = Date.now() - startMs; + + logger?.info?.( + `${TAG} ${label} <<< ${elapsedMs}ms, output=${text.length} chars`, + ); + + return text; + } catch (err) { + const elapsedMs = Date.now() - startMs; + const errMsg = err instanceof Error ? err.message : String(err); + logger?.error?.(`${TAG} ${label} FAILED (${elapsedMs}ms): ${errMsg}`); + throw err; + } +} diff --git a/src/offload/local-llm/parsers/json-utils.ts b/src/offload/local-llm/parsers/json-utils.ts new file mode 100644 index 0000000..c2ac51f --- /dev/null +++ b/src/offload/local-llm/parsers/json-utils.ts @@ -0,0 +1,85 @@ +/** + * Tolerant JSON parsing utilities for LLM responses. + * + * LLMs often wrap JSON in markdown code fences, include trailing commas, + * or prepend explanatory text. These utilities handle common deviations. + */ + +/** + * Extract JSON from LLM output โ€” handles code fences, prefix text, etc. + * Returns the parsed object/array, or null if parsing fails. + */ +export function extractJson(raw: string): T | null { + if (!raw || typeof raw !== "string") return null; + + const trimmed = raw.trim(); + + // Strategy 1: Direct parse (ideal case) + const direct = tryParse(trimmed); + if (direct !== null) return direct; + + // Strategy 2: Extract from markdown code fence (```json ... ``` or ``` ... ```) + const fenceMatch = trimmed.match(/```(?:json)?\s*\n?([\s\S]*?)```/); + if (fenceMatch) { + const inner = fenceMatch[1].trim(); + const parsed = tryParse(inner); + if (parsed !== null) return parsed; + } + + // Strategy 3: Find first { to last } (or first [ to last ]) + const firstBrace = trimmed.indexOf("{"); + const lastBrace = trimmed.lastIndexOf("}"); + if (firstBrace >= 0 && lastBrace > firstBrace) { + const candidate = trimmed.slice(firstBrace, lastBrace + 1); + const parsed = tryParse(candidate); + if (parsed !== null) return parsed; + + // Try with trailing comma fix + const fixed = fixTrailingCommas(candidate); + const parsedFixed = tryParse(fixed); + if (parsedFixed !== null) return parsedFixed; + } + + const firstBracket = trimmed.indexOf("["); + const lastBracket = trimmed.lastIndexOf("]"); + if (firstBracket >= 0 && lastBracket > firstBracket) { + const candidate = trimmed.slice(firstBracket, lastBracket + 1); + const parsed = tryParse(candidate); + if (parsed !== null) return parsed; + } + + // Strategy 4: Try fixing the entire string + const fixed = fixTrailingCommas(trimmed); + const parsedFixed = tryParse(fixed); + if (parsedFixed !== null) return parsedFixed; + + return null; +} + +/** + * Extract mermaid content from a code fence. + * Returns the raw mermaid text (without fence markers). + */ +export function extractMermaidFromFence(text: string): string | null { + if (!text) return null; + const match = text.match(/```mermaid\s*\n?([\s\S]*?)```/); + if (match) return match[1].trim(); + // Fallback: if no fence, return as-is (might already be raw mermaid) + if (text.includes("flowchart") || text.includes("graph")) return text.trim(); + return null; +} + +// โ”€โ”€โ”€ Internal Helpers โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +function tryParse(s: string): T | null { + try { + return JSON.parse(s) as T; + } catch { + return null; + } +} + +function fixTrailingCommas(s: string): string { + // Remove trailing commas before } or ] + return s.replace(/,\s*([}\]])/g, "$1"); +} diff --git a/src/offload/local-llm/parsers/l1-parser.ts b/src/offload/local-llm/parsers/l1-parser.ts new file mode 100644 index 0000000..4e79021 --- /dev/null +++ b/src/offload/local-llm/parsers/l1-parser.ts @@ -0,0 +1,41 @@ +/** + * L1 Response Parser โ€” extracts summarization results from LLM output. + */ +import { extractJson } from "./json-utils.js"; +import type { OffloadEntry } from "../../types.js"; + +interface RawL1Entry { + tool_call?: string; + summary?: string; + tool_call_id?: string; + timestamp?: string; + score?: number; +} + +/** + * Parse L1 LLM response into OffloadEntry array. + * Tolerant of markdown wrapping, missing fields, etc. + */ +export function parseL1Response(raw: string): OffloadEntry[] { + const parsed = extractJson(raw); + if (!parsed || !Array.isArray(parsed)) return []; + + const entries: OffloadEntry[] = []; + for (const item of parsed) { + if (!item || typeof item !== "object") continue; + + const toolCallId = item.tool_call_id ?? ""; + if (!toolCallId) continue; // tool_call_id is required + + entries.push({ + tool_call_id: toolCallId, + tool_call: item.tool_call ?? "", + summary: item.summary ?? "", + timestamp: item.timestamp ?? "", + score: typeof item.score === "number" ? item.score : 5, + node_id: null, + }); + } + + return entries; +} diff --git a/src/offload/local-llm/parsers/l15-parser.ts b/src/offload/local-llm/parsers/l15-parser.ts new file mode 100644 index 0000000..1f477ff --- /dev/null +++ b/src/offload/local-llm/parsers/l15-parser.ts @@ -0,0 +1,37 @@ +/** + * L1.5 Response Parser โ€” extracts task judgment from LLM output. + */ +import { extractJson } from "./json-utils.js"; +import type { TaskJudgment } from "../../types.js"; + +interface RawL15Response { + taskCompleted?: boolean | null; + isContinuation?: boolean | null; + isLongTask?: boolean | null; + continuationMmdFile?: string | null; + newTaskLabel?: string | null; +} + +/** + * Parse L1.5 LLM response into TaskJudgment. + * Returns null if the response is completely unparseable or all-null (backend unavailable). + */ +export function parseL15Response(raw: string): TaskJudgment | null { + const parsed = extractJson(raw); + if (!parsed || typeof parsed !== "object") return null; + + // All-null check (mirrors normalizeJudgment logic) + if (parsed.taskCompleted == null && parsed.isContinuation == null && parsed.isLongTask == null) { + return null; + } + + return { + taskCompleted: Boolean(parsed.taskCompleted), + isContinuation: Boolean(parsed.isContinuation), + isLongTask: Boolean(parsed.isLongTask), + continuationMmdFile: + typeof parsed.continuationMmdFile === "string" ? parsed.continuationMmdFile : undefined, + newTaskLabel: + typeof parsed.newTaskLabel === "string" ? parsed.newTaskLabel : undefined, + }; +} diff --git a/src/offload/local-llm/parsers/l2-parser.ts b/src/offload/local-llm/parsers/l2-parser.ts new file mode 100644 index 0000000..3e50b5f --- /dev/null +++ b/src/offload/local-llm/parsers/l2-parser.ts @@ -0,0 +1,92 @@ +/** + * L2 Response Parser โ€” extracts MMD generation results from LLM output. + */ +import { extractJson, extractMermaidFromFence } from "./json-utils.js"; + +export interface L2ParsedResponse { + fileAction: "write" | "replace"; + mmdContent?: string; + replaceBlocks?: Array<{ + startLine: number; + endLine: number; + content: string; + }>; + nodeMapping: Record; +} + +interface RawL2Response { + file_action?: string; + mmd_content?: string | null; + replace_blocks?: Array<{ + start_line?: number | string; + end_line?: number | string; + content?: string; + }> | null; + node_mapping?: Record; +} + +/** + * Parse L2 LLM response into structured L2 result. + * Returns null if parsing fails completely. + */ +export function parseL2Response(raw: string): L2ParsedResponse | null { + const parsed = extractJson(raw); + if (!parsed || typeof parsed !== "object") { + // Fallback: try extracting ```mermaid ... ``` code block (same as Go backend) + const mmd = extractMermaidFromFence(raw); + if (mmd) { + return { fileAction: "write", mmdContent: mmd, nodeMapping: {} }; + } + return null; + } + + const fileAction = parsed.file_action === "replace" ? "replace" : "write"; + + // Extract mmd_content (may be wrapped in code fence) + let mmdContent: string | undefined; + if (fileAction === "write") { + if (parsed.mmd_content) { + mmdContent = extractMermaidFromFence(parsed.mmd_content) ?? parsed.mmd_content; + } else { + // mmd_content missing in write mode โ€” try extracting from raw response + const fallbackMmd = extractMermaidFromFence(raw); + if (fallbackMmd) mmdContent = fallbackMmd; + } + } + + // Parse replace_blocks + let replaceBlocks: L2ParsedResponse["replaceBlocks"] | undefined; + if (fileAction === "replace" && Array.isArray(parsed.replace_blocks)) { + replaceBlocks = []; + for (const block of parsed.replace_blocks) { + if (!block || typeof block !== "object") continue; + const startLine = Number(block.start_line); + const endLine = Number(block.end_line); + if (isNaN(startLine) || isNaN(endLine)) continue; + + let content = block.content ?? ""; + // Extract mermaid from fence if present + const extracted = extractMermaidFromFence(content); + if (extracted) content = extracted; + + replaceBlocks.push({ startLine, endLine, content }); + } + } + + // Parse node_mapping + const nodeMapping: Record = {}; + if (parsed.node_mapping && typeof parsed.node_mapping === "object") { + for (const [key, value] of Object.entries(parsed.node_mapping)) { + if (typeof value === "string") { + nodeMapping[key] = value; + } + } + } + + return { + fileAction, + mmdContent, + replaceBlocks, + nodeMapping, + }; +} diff --git a/src/offload/local-llm/prompts/l1-prompt.ts b/src/offload/local-llm/prompts/l1-prompt.ts new file mode 100644 index 0000000..154961b --- /dev/null +++ b/src/offload/local-llm/prompts/l1-prompt.ts @@ -0,0 +1,98 @@ +/** + * L1 Summarization Prompt โ€” migrated from context-offload-server. + * + * Converts tool call/result pairs into high-density JSON summaries. + */ + +// โ”€โ”€โ”€ System Prompt โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +export const L1_SYSTEM_PROMPT = `ไฝ ๆ˜ฏไธ€ไธชไธ“ไธบ AI ็ผ–็ ๅŠฉๆ‰‹ๆไพ›ๆ”ฏๆŒ็š„"ๅทฅๅ…ท็ป“ๆžœๆ‘˜่ฆๅ™จ"ใ€‚ไฝ ็š„ๆ ธๅฟƒไปปๅŠกๆ˜ฏๆทฑๅบฆ็†่งฃๅฝ“ๅ‰็š„ๅฏน่ฏไธŠไธ‹ๆ–‡๏ผŒๅนถๅฐ†็นๆ‚็š„ๅทฅๅ…ท่ฐƒ็”จไธŽๆ‰ง่กŒ็ป“ๆžœ๏ผˆไธ€ๅฏนtoolcallๅ’Œtool resultๆ•ดๅˆๆˆไธ€ๆกsummary่พ“ๅ‡บ๏ผ‰๏ผŒๆ็‚ผไธบ้ซ˜ไฟกๆฏๅฏ†ๅบฆ็š„ JSON ๆ•ฐ็ป„ใ€‚ + +ๅœจ็”Ÿๆˆๆ‘˜่ฆๅ‰๏ผŒ่ฏทๅŠกๅฟ…่ฟ›่กŒไปฅไธ‹ๅ†…้ƒจๆ€่€ƒ๏ผš +1. ไปปๅŠกๅฏน้ฝ๏ผš็ป“ๅˆๆœ€่ฟ‘็š„ๅฏน่ฏ่ฎฐๅฝ•๏ผŒ่ฏ†ๅˆซ็”จๆˆทๅฝ“ๅ‰็š„ๆ ธๅฟƒ็›ฎๆ ‡ๅ’Œๆœ€ๆ–ฐๆ„ๅ›พใ€‚่‹ฅไธŠไธ‹ๆ–‡ๅญ˜ๅœจๅ†ฒ็ช๏ผŒๅง‹็ปˆไปฅๆœ€ๆ–ฐ็š„็”จๆˆทๆ„ๅ›พไธบๅ‡†ใ€‚ +2. ไปทๅ€ผ่ฟ‡ๆปค๏ผšๅฟฝ็•ฅๅทฅๅ…ทๅฆ‚ไฝ•ๅทฅไฝœ็š„ๅ†—ไฝ™็ป†่Š‚๏ผŒ็›ดๆŽฅๆๅ–"ๅ‘็Žฐไบ†ไป€ไนˆๅ…ณ้”ฎ็บฟ็ดข"ใ€"ๅšไบ†ไป€ไนˆๅ…ณ้”ฎๅŠจไฝœ"ใ€"ไฟฎๆ”นไบ†ไป€ไนˆๅ…ทไฝ“ๅ†…ๅฎน"ๆˆ–"้‡ๅˆฐไบ†ไป€ไนˆๅ…ทไฝ“ๆŠฅ้”™"ใ€‚ +3. ๅฝฑๅ“่ฏ„ไผฐ๏ผšๅˆคๆ–ญ่ฏฅ็ป“ๆžœๅฏนๅฝ“ๅ‰ไปปๅŠก็š„ๅฎž่ดจๆ€งๅฝฑๅ“๏ผˆไพ‹ๅฆ‚๏ผš่ฏๅฎžไบ†ๆŸไธชๅ‡่ฎพใ€ๆŽจ่ฟ›ไบ†ๅ“ชไธ€ๆญฅใ€ๅšๅ‡บไบ†ไป€ไนˆๅ†ณ็ญ–๏ผŒๆˆ–ๅ› ไธบไป€ไนˆๆŠฅ้”™ๅฏผ่‡ดไบ†้˜ปๅกž๏ผ‰ใ€‚ + +ใ€่พ“ๅ‡บๆ ผๅผ่ฆๆฑ‚ใ€‘ +ไฝ ๅฟ…้กปไธ”ๅช่ƒฝ่พ“ๅ‡บไธ€ไธชๅˆๆณ•็š„ JSON ๅฏน่ฑกๆ•ฐ็ป„ [{...}]๏ผŒๆฏไธชๅฏน่ฑก**ๅฟ…้กป**ๅŒ…ๅซไปฅไธ‹ๅญ—ๆฎต๏ผš +- "tool_call": ๅทฅๅ…ท่ฐƒ็”จ็š„็ฎ€ๆดๆ่ฟฐใ€‚ๅค„็†่ง„ๅˆ™ๅฆ‚ไธ‹๏ผš + ยท ๅฆ‚ๆžœ่พ“ๅ…ฅไธญ่ฏฅ tool pair ๆ ‡่ฎฐไบ† [NEEDS_COMPRESS]๏ผŒไฝ ๅฟ…้กปๅฐ†ๅทฅๅ…ทๅ+ๅ…ณ้”ฎๅ‚ๆ•ฐๅŽ‹็ผฉไธบไธ€ๅฅ็ฎ€ๆด็š„ๆ่ฟฐ๏ผˆโ‰ค150ๅญ—็ฌฆ๏ผ‰๏ผŒไฟ็•™ๅทฅๅ…ทๅใ€ๆ“ไฝœ็›ฎๆ ‡๏ผˆๅฆ‚ๆ–‡ไปถ่ทฏๅพ„ใ€ๅ‘ฝไปคๆ„ๅ›พ๏ผ‰๏ผŒ็œ็•ฅๅ†…่”่„šๆœฌ/ๅคงๆฎตๅ†…ๅฎน็š„็ป†่Š‚ใ€‚ + ็คบไพ‹๏ผšexec({"command":"python3 -c 'import csv; ...200่กŒ่„šๆœฌ...'"}) โ†’ "exec: ่ฟ่กŒ Python ๏ผˆxx/xx/xx.sh๏ผŒๆ ‡ๆ˜Žๅ…ทไฝ“่ทฏๅพ„ๅ’Œๆ–‡ไปถ๏ผ‰่„šๆœฌๅˆ†ๆž sales_channels.csv ๆ•ฐๆฎ่ดจ้‡" + ็คบไพ‹๏ผšwrite_file({"path":"/root/app.py","content":"...5000ๅญ—็ฌฆ..."}) โ†’ "write_file: ๅ†™ๅ…ฅ /root/app.py (Flask ๅบ”็”จไธปๆ–‡ไปถ)๏ผŒๅคง่‡ดๅ†…ๅฎนๆ˜ฏโ€ฆโ€ฆ" + ยท ๅฆ‚ๆžœๆœชๆ ‡่ฎฐ [NEEDS_COMPRESS]๏ผŒ็›ดๆŽฅ็ฎ€่ฟฐๅทฅๅ…ทไธŽๅ‚ๆ•ฐๅณๅฏ๏ผˆ็ณป็ปŸไผš็”จๅŽŸๅง‹ๅ€ผ่ฆ†็›–๏ผ‰ใ€‚ +- "summary": ่žๅˆไธŠ่ฟฐๆ€่€ƒ็š„็ฒพ็‚ผๆ€ป็ป“๏ผˆโ‰ค200ไธชๅญ—็ฌฆ๏ผ‰ใ€‚ๅฟ…้กปไธ€้’ˆ่ง่ก€ๅœฐ่ฏดๆธ…ๆฅš็ป“ๆžœ็š„ไธšๅŠกไปทๅ€ผ๏ผŒไปฅๅŠๅฎƒๅฏนไปปๅŠก็š„ๆŽจ่ฟ›/้˜ปๅกžไฝœ็”จใ€‚ +- "tool_call_id": ๅŽŸๅง‹็š„ tool_call_id๏ผˆๅฟ…้กปๅŽŸๆ ท้€ไผ ๏ผ‰ใ€‚ +- "timestamp": ๅŽŸๅง‹็š„ไธญๅ›ฝๆ ‡ๅ‡†ๆ—ถ้—ด๏ผˆ+08:00๏ผ‰ISO 8601 ๆ—ถ้—ดๆˆณ๏ผˆๅฟ…้กปๅŽŸๆ ท้€ไผ ๏ผ‰ใ€‚ +- "score"๏ผˆ**ๅฟ…ๅกซ**๏ผ‰: ็ป“ๅˆไฟกๆฏๅฏ†ๅบฆๅ’ŒไปปๅŠก็›ฎ็š„ๅˆ†ๆžsummaryๅฏนไบŽๅŽŸๆ–‡็š„ๅฏๆ›ฟไปฃๆ€ง๏ผŒ่Œƒๅ›ดๅœจ0-10ไน‹้—ด๏ผŒ่ถŠๆŽฅ่ฟ‘10่กจ็คบsummary่ถŠ่ƒฝๆ›ฟไปฃๅŽŸๆ–‡ใ€‚ + +ใ€ไธฅๆ ผ่ง„ๅˆ™ใ€‘ +ๅชๅ…่ฎธ่พ“ๅ‡บ็บฏ JSON ๆ•ฐ็ป„๏ผŒไธฅ็ฆ่พ“ๅ‡บๆ€่€ƒ่ฟ‡็จ‹ๆˆ–ๅ…ถไป–่งฃ้‡Šๆ€งๆ–‡ๆœฌใ€‚`; + +// โ”€โ”€โ”€ Constants โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +const PARAMS_MAX_LEN = 500; +const RESULT_MAX_LEN = 2000; +const COMPRESS_THRESHOLD = 200; + +// โ”€โ”€โ”€ Types โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +export interface L1ToolPair { + toolName: string; + toolCallId: string; + params: unknown; + result: unknown; + timestamp: string; +} + +// โ”€โ”€โ”€ User Prompt Builder โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +/** + * Build the L1 user prompt for summarization. + * Mirrors context-offload-server/internal/service/prompt/BuildL1UserPrompt. + */ +export function buildL1UserPrompt(recentMessages: string, pairs: L1ToolPair[]): string { + const parts: string[] = []; + + parts.push("## ๆœ€่ฟ‘็š„ๅฏน่ฏไธŠไธ‹ๆ–‡๏ผˆ็”จไบŽ็†่งฃๅฝ“ๅ‰ไปปๅŠก๏ผ‰๏ผš"); + parts.push(recentMessages); + parts.push("\n## Tool call/result pairs to summarize:"); + + for (let i = 0; i < pairs.length; i++) { + const p = pairs[i]; + const paramsStr = truncate(stringify(p.params), PARAMS_MAX_LEN); + const resultStr = truncate(stringify(p.result), RESULT_MAX_LEN); + const canonical = `${p.toolName}(${stringify(p.params)})`; + const needsCompress = canonical.length > COMPRESS_THRESHOLD; + + parts.push(`--- Tool Pair ${i + 1} ---`); + parts.push(`tool_call_id: ${p.toolCallId}`); + parts.push(`timestamp: ${p.timestamp}`); + if (needsCompress) { + parts.push(`Tool: ${p.toolName} [NEEDS_COMPRESS]`); + } else { + parts.push(`Tool: ${p.toolName}`); + } + parts.push(`Params: ${paramsStr}`); + parts.push(`Result: ${resultStr}\n`); + } + + parts.push("Summarize each pair into the JSON array format described."); + return parts.join("\n"); +} + +// โ”€โ”€โ”€ Helpers โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +function stringify(value: unknown): string { + if (value == null) return ""; + if (typeof value === "string") return value; + try { + return JSON.stringify(value); + } catch { + return String(value); + } +} + +function truncate(s: string, maxLen: number): string { + if (s.length <= maxLen) return s; + return s.slice(0, maxLen) + "..."; +} diff --git a/src/offload/local-llm/prompts/l15-prompt.ts b/src/offload/local-llm/prompts/l15-prompt.ts new file mode 100644 index 0000000..16c8c40 --- /dev/null +++ b/src/offload/local-llm/prompts/l15-prompt.ts @@ -0,0 +1,101 @@ +/** + * L1.5 Task Judgment Prompt โ€” migrated from context-offload-server. + * + * Determines task lifecycle: completion, continuation, new task detection. + */ + +// โ”€โ”€โ”€ System Prompt โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +export const L15_SYSTEM_PROMPT = `ไฝ ๆ˜ฏไธ€ไธช้ขๅ‘ AI ็ผ–็ ๅŠฉๆ‰‹็š„"ไปปๅŠก็”Ÿๅ‘ฝๅ‘จๆœŸ้—จ็ฅž"ใ€‚ +ไฝ ็š„่Œ่ดฃๆ˜ฏไบคๅ‰ๅˆ†ๆžๆไพ›็š„ไธ‰ไธช่พ“ๅ…ฅๆบ๏ผŒ็ฒพๅ‡†็ ”ๅˆคไปปๅŠก็Šถๆ€๏ผŒๅนถ่พ“ๅ‡บ็บฏ JSON ๅฏน่ฑกใ€‚ + +ใ€่พ“ๅ…ฅๆ•ฐๆฎๅˆฉ็”จๆŒ‡ๅ—๏ผˆๅฟ…้กป้ตๅพช็š„ๆ€่€ƒ้“พ่ทฏ๏ผ‰ใ€‘ +1. ็ฌฌไธ€ๆญฅ - ๅ‰–ๆž recentMessages๏ผˆ่ฏ†ๅˆซๆ„ๅ›พ๏ผ‰๏ผšๆ นๆฎๅฝ“ๅ‰ๅ’Œๅކๅฒๅฏน่ฏ๏ผŒๆๅ–็”จๆˆทๆœ€ๆ–ฐๅ›žๅค็š„ๆ ธๅฟƒ่ฏ‰ๆฑ‚ใ€‚ๅˆคๆ–ญๆ˜ฏ"็ปง็ปญๆŽ’ๆŸฅ"ใ€"ๅฎฃๅธƒๅฎŒๅทฅ๏ผˆๅฆ‚๏ผš่ท‘้€šไบ†๏ผ‰"ใ€"ๅ•่ฝฎ้—ฒ่Š้—ฎ็ญ”"่ฟ˜ๆ˜ฏ"ๅผ€ๅฏๅ…จๆ–ฐ้œ€ๆฑ‚"ใ€‚ +2. ็ฌฌไบŒๆญฅ - ๅฏน้ฝ currentMmd๏ผˆ่ฏ„ไผฐๅฝ“ๅ‰ๅŸบ็บฟ๏ผ‰๏ผšๅฐ†็”จๆˆท็š„ๆœ€ๆ–ฐๆ„ๅ›พไธŽ currentMmd ็š„ๅฎŒๆ•ด Mermaid ๅ†…ๅฎน่ฟ›่กŒๆฏ”ๅฏนโ€”โ€”ๅ…ณๆณจ taskGoalใ€ๅ„่Š‚็‚น็š„ status๏ผˆdone/doing/todo๏ผ‰ไปฅๅŠ summaryใ€‚ๅฆ‚ๆžœ่ฏ‰ๆฑ‚ๅฎŒๅ…จ่ถ…ๅ‡บไบ†ๅฝ“ๅ‰ๅ›พ่กจ็š„่Œƒ็•ดๆˆ–็›ฎๆ ‡ๅทฒๅฎž็Žฐ๏ผˆๆ‰€ๆœ‰่Š‚็‚น done ไธ”ๆ— ๅŽ็ปญ๏ผ‰๏ผŒๅˆ™ taskCompleted ไธบ trueใ€‚่‹ฅไปๅœจ่งฃๅ†ณๅ›พ่กจไธญ็š„ๅญ้—ฎ้ข˜๏ผˆๅŒ…ๆ‹ฌ doing ่Š‚็‚นๆˆ–ไฟฎ bug๏ผ‰๏ผŒๅˆ™ไธบ falseใ€‚(ๅฆ‚ๆžœๆฒกๆœ‰currentMmd๏ผŒๅฐฑๅชๆ นๆฎๅฝ“ๅ‰ๅฏน่ฏๅ’Œๅކๅฒๅฏน่ฏๆฅๅˆคๆ–ญๆ˜ฏๅฆ็ปง็ปญไปปๅŠก) +3. ็ฌฌไธ‰ๆญฅ - ๆฃ€็ดข availableMmds๏ผˆๅˆคๆ–ญๆ˜ฏๅฆๅปถ็ปญ๏ผ‰๏ผšๅฆ‚ๆžœๅˆคๅฎš่ฆๅผ€ๅฏๆ–ฐไปปๅŠก๏ผˆisLongTask=true ไธ” taskCompleted=true/ๅฝ“ๅ‰ๆ— ไปปๅŠก๏ผ‰๏ผŒๅฟ…้กปๆ‰ซๆ availableMmds ็š„ taskGoal ๅ’Œๆ—ถ้—ดไฟกๆฏใ€‚่‹ฅๆ–ฐ่ฏ‰ๆฑ‚ไธŽๅˆ—่กจไธญๆŸไธชๆ—งไปปๅŠก้ซ˜ๅบฆ้‡ๅˆ๏ผˆๅฆ‚ๅ›žๅˆฐๆ˜จๅคฉๆฒกๅšๅฎŒ็š„ๆจกๅ—๏ผ‰๏ผŒๅˆ™ๆ˜ฏๅปถ็ปญ๏ผˆisContinuation=true๏ผ‰ใ€‚ + +ใ€ไธฅๆ ผ JSON ่พ“ๅ‡บๆ ผๅผใ€‘ +ๅŠกๅฟ…่พ“ๅ‡บๅˆๆณ•็š„็บฏ JSON ๅฏน่ฑก๏ผŒๆ ผๅผๅฆ‚ไธ‹๏ผš +{ + "taskCompleted": boolean, // ๅฝ“ๅ‰ไปปๅŠกๆ˜ฏๅฆๅทฒ็ป“ๆŸ๏ผˆๅฆ‚ๆžœ currentMmd ไธบ none๏ผŒ่ฟ™้‡Œๅฟ…้กปๅกซ true๏ผ‰ + "isLongTask": boolean, // ๆœ€ๆ–ฐ่ฏ‰ๆฑ‚ๆ˜ฏๅฆๆ˜ฏ้œ€่ฆๅคšๆญฅๆ“ไฝœ็š„ๅคๆ‚ๅทฅ็จ‹๏ผˆๆ™ฎ้€šๆŠ€ๆœฏ้—ฎ็ญ”ใ€้—ฒ่Šๅกซ false๏ผ‰ + "isContinuation": boolean, // ๆ˜ฏๅฆๅœจๅปถ็ปญ availableMmds ไธญ็š„ๅކๅฒไปปๅŠก + "continuationMmdFile": "string|null", // ่‹ฅๅปถ็ปญๆ—งไปปๅŠก๏ผŒ็ฒพ็กฎๅกซๅ…ฅ availableMmds ไธญ็š„ๆ–‡ไปถๅ๏ผˆไธๅซ่ทฏๅพ„ๅ‰็ผ€๏ผ‰๏ผŒๅฆๅˆ™ไธบ null + "newTaskLabel": "string|null" // ่‹ฅๆ˜ฏๅ…จๆ–ฐ้•ฟไปปๅŠก๏ผŒ็”Ÿๆˆ็ฎ€็Ÿญๆ ‡็ญพ๏ผˆโ‰ค30ๅญ—็ฌฆ๏ผŒkebab-case๏ผŒๅฆ‚ "refactor-api"๏ผ‰๏ผŒๅฆๅˆ™ไธบ null +} + +ๅช่พ“ๅ‡บ็บฏ JSON ๅฏน่ฑก๏ผŒ็ปไธๅ…่ฎธๅŒ…ๅซ่งฃ้‡Šๆ–‡ๅญ—ใ€‚`; + +// โ”€โ”€โ”€ Types โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +export interface L15CurrentMmd { + filename: string; + content: string; + path: string; +} + +export interface L15MmdMeta { + filename: string; + path: string; + taskGoal: string; + doneCount: number; + doingCount: number; + todoCount: number; + updatedTime?: string | null; + nodeSummaries?: Array<{ nodeId: string; status: string; summary: string }>; +} + +// โ”€โ”€โ”€ User Prompt Builder โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +/** + * Build the L1.5 user prompt for task judgment. + * Mirrors context-offload-server/internal/service/prompt/BuildL15UserPrompt. + */ +export function buildL15UserPrompt( + recentMessages: string, + currentMmd: L15CurrentMmd | null, + metas: L15MmdMeta[], +): string { + const parts: string[] = []; + + parts.push("## 1. ๆœ€่ฟ‘็š„ๅฏน่ฏไธŠไธ‹ๆ–‡ (Recent 6 messages):"); + parts.push(recentMessages); + parts.push("\n## 2. ๅฝ“ๅ‰ๆŒ‚่ฝฝ็š„ไปปๅŠกๅ›พ (Active Mermaid โ€” ๅฎŒๆ•ดๅ†…ๅฎน):"); + + if (currentMmd && currentMmd.filename) { + parts.push(`**File:** ${currentMmd.filename}`); + if (currentMmd.path) { + parts.push(`**Path:** \`${currentMmd.path}\``); + } + parts.push(`\n\`\`\`mermaid\n${currentMmd.content}\n\`\`\``); + } else { + parts.push("(none - ๅฝ“ๅ‰ๅค„ไบŽ้—ฒ็ฝฎ็Šถๆ€๏ผŒๆ— ๆดป่ทƒไปปๅŠก)"); + } + + parts.push("\n## 3. ๅކๅฒๅฏ็”จ็š„ไปปๅŠกๅ›พ (Available Mermaid task files):"); + + if (metas.length === 0) { + parts.push("(none - ๆš‚ๆ— ๅކๅฒ้•ฟไปปๅŠก)"); + } else { + for (const m of metas) { + parts.push(`- **${m.filename}**`); + parts.push(` path: \`${m.path}\``); + parts.push(` taskGoal: ${m.taskGoal}`); + const total = m.doneCount + m.doingCount + m.todoCount; + parts.push(` progress: ${m.doneCount}/${total} done, ${m.doingCount} doing, ${m.todoCount} todo`); + if (m.updatedTime) { + parts.push(` lastUpdated: ${m.updatedTime}`); + } + if (m.nodeSummaries && m.nodeSummaries.length > 0) { + parts.push(" recentNodes:"); + for (const n of m.nodeSummaries) { + parts.push(` - [${n.nodeId}] (${n.status}) ${n.summary}`); + } + } + parts.push(""); + } + } + + parts.push("่ฏทไธฅๆ ผๆ นๆฎ็ณป็ปŸๆŒ‡ไปค็š„ใ€ไธ‰ๆญฅๆ€่€ƒ้“พ่ทฏใ€‘่ฟ›่กŒ็ ”ๅˆค๏ผŒๅนถ่พ“ๅ‡บๅˆๆณ•็š„ JSON ๅฏน่ฑกใ€‚"); + return parts.join("\n"); +} diff --git a/src/offload/local-llm/prompts/l2-prompt.ts b/src/offload/local-llm/prompts/l2-prompt.ts new file mode 100644 index 0000000..b0fd3a5 --- /dev/null +++ b/src/offload/local-llm/prompts/l2-prompt.ts @@ -0,0 +1,127 @@ +/** + * L2 MMD Generation Prompt โ€” migrated from context-offload-server. + * + * Generates/updates Mermaid flowchart diagrams from offload entries. + */ + +// โ”€โ”€โ”€ System Prompt โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +export const L2_SYSTEM_PROMPT = `ไฝ ๆ˜ฏไธ€ไธช็ฉถๆžๅฎž็”จไธปไน‰็š„ AI ไปปๅŠกๆ‹“ๆ‰‘ๆžถๆž„ๅธˆไธŽ่ง†่ง‰ๅ™ไบ‹่€…ใ€‚ +ไฝ ็š„ๆ ธๅฟƒ้€ป่พ‘ๆ˜ฏ็”จๅฐฝ้‡ๅฐ‘็š„ๅญ—็ฌฆ่กจ่พพๅฐฝ้‡ๅคš็š„ไฟกๆฏ๏ผŒ่ฎฉLLMๆจกๅž‹่ƒฝ็œ‹ๆ‡‚๏ผŒไธๆ˜ฏไธบไบบ็ฑปๆœๅŠก๏ผŒๅฐฝ้‡ๅ‡ๅฐ‘ๆ— ็”จ็š„่ง†่ง‰็ฌฆๅทใ€‚ไปปๅŠกๆ˜ฏๅฐ†ๅบ•ๅฑ‚ๅทฅๅ…ท่ฐƒ็”จ่ฎฐๅฝ•๏ผŒๅ‡็ปดๆ˜ ๅฐ„ไธบไธ€ๅผ ้ซ˜ๅบฆ่ฏญไน‰ๅŒ–ใ€่กจ็ŽฐๅŠ›ไธฐๅฏŒไธ”ๆžๅบฆๅ…‹ๅˆถ็š„ Mermaid (flowchart TD) ่ฎค็Ÿฅ็Šถๆ€ๆœบใ€‚ไฝ ่ฆๆ นๆฎๅฝ“ๅ‰ไปปๅŠกๅ’Œๆ„ๅ›พ๏ผŒๅฝ’็บณ"่ฟ‡ๅŽป"๏ผŒ่ฆๆ€่€ƒ"ๆœชๆฅ"ๅฆ‚ไฝ•็”จ่ฟ™ไบ›ๅทฒๆœ‰็š„ไฟกๆฏ๏ผˆไฝ ๅช้œ€่ฆ่ฎฐๅฝ•ๅทฒๆœ‰ไฟกๆฏ๏ผŒไธ้œ€่ฆๅ†™ไธ‹ไธ€ๆญฅ่ง„ๅˆ’๏ผ‰ๅนถๆ ‡่ฎฐ"้›ทๅŒบ"ใ€‚ไฟๆŒๅ›พ่กจ็š„้ซ˜ๅบฆๆฆ‚ๆ‹ฌๆ€งใ€‚ + +ใ€้ซ˜้˜ถ่ฎค็ŸฅไธŽๆ‹“ๆ‰‘ๆŒ‡ๅ—๏ผˆไฝ ็š„่‡ชไธปๆƒไธŽๆž็ฎ€ๅŽŸๅˆ™๏ผ‰ใ€‘ +1. ๅผนๆ€ง่šๅˆ๏ผšไฝ ๆ‹ฅๆœ‰ๅ†ณๅฎš่Š‚็‚นๆ‹†ๅˆ็š„ๅฎŒๅ…จ่‡ชไธปๆƒใ€‚ๅฏนไบŽ่ฟž็ปญ็š„ใ€ๆ„ๅ›พ็›ธๅŒ็š„ๅธธ่ง„ๅŠจไฝœ๏ผˆๅฆ‚่ฟž็ปญๆŸฅ็œ‹ๅคšไธชๆ–‡ไปถไปฅไบ†่งฃไธŠไธ‹ๆ–‡๏ผ‰๏ผŒๅปบ่ฎฎๅˆๅนถไธบไธ€ไธชๅฎ่ง‚่Š‚็‚น๏ผ›๏ผŒไฝ†ไฟ็•™ๅ…ณ้”ฎ่ฝฌๆŠ˜็‚นๆˆ–้‡ๅคงๅ‘็Žฐไธบ็‹ฌ็ซ‹่Š‚็‚นใ€‚ๅ›พ่กจๅฟ…้กปไฟๆŒๅฎ่ง‚ๅ’Œๅ…‹ๅˆถ๏ผŒ็ปไธไบ‹ๆ— ๅทจ็ป†ๅœฐ่ฎฐๆตๆฐด่ดฆใ€‚ +2. ่ฎค็Ÿฅๅข“็ข‘ (้˜ฒ้‡่นˆ่ฆ†่พ™)๏ผš้‡ๅˆฐๅฝปๅบ•่ตฐไธ้€š็š„ๆญป่ƒกๅŒๆˆ–ๅผ•ๅ‘ไธฅ้‡ๆŠฅ้”™็š„ๅบŸๅผƒๆ–นๆกˆ๏ผŒๅฏไปฅๅปบ็ซ‹่ญฆ็คบ่Š‚็‚น๏ผˆstatus: blocked๏ผ‰๏ผˆๅฆ‚ๆžœๆ˜ฏไปทๅ€ผไธ้ซ˜็š„failไฟกๆฏๅˆ™ไธ้œ€่ฆ่ฎฐๅฝ•๏ผ‰ใ€‚ +3. ็ป“่ฎบๅฏผๅ‘็š„ๆ‘˜่ฆ๏ผš่Š‚็‚น็š„ summary๏ผˆๆณจๆ„๏ผšๅฐฝ้‡ๅฐไบŽ150ๅญ—๏ผ‰ๅบ”่š็„ฆไบŽ"ๅพ—ๅ‡บไบ†ไป€ไนˆ็ป“่ฎบ"ๆˆ–"ๅ‘็”Ÿไบ†ไป€ไนˆๅฎž่ดจๆ”นๅ˜"๏ผŒ่€Œ้ž็ฝ—ๅˆ—็็ขŽ็š„ๆ•ฐๆฎๆˆ–ๅ‚ๆ•ฐ๏ผŒ่ฎฐๅพ—ไฟๆŒๆž็ฎ€ๅŽŸๅˆ™ใ€‚ +4. ่ฆๅฎžไบ‹ๆฑ‚ๆ˜ฏ๏ผŒไฝ ็š„ไปปๅŠกๆ˜ฏ่ฎฐๅฝ•ๅนถๅฝ’็บณๅทฒ็ปๅ‘็”Ÿ็š„ไบ‹ๆƒ…๏ผŒไธๆ˜ฏ่ง„ๅˆ’ๆœชๆฅ็š„ๅ…ทไฝ“ๆ“ไฝœ๏ผŒๆœชๅ‘็”Ÿ็š„่Š‚็‚นไธ่ฆๅ†™๏ผŒ่ฎฐๅฝ•็š„ๅทฒๅ‘็”Ÿ่Š‚็‚น่ฆๆœ‰ๅฏนๅบ”็š„ๆถˆๆฏๆฅๆบ๏ผˆๅฏนๅบ”ๆ ‡ๆณจnode_id๏ผ‰ใ€‚ +ใ€็ฌฆๅทๅณ่ฏญไน‰๏ผš้ซ˜็ปด่ฎค็Ÿฅๅญ—ๅ…ธ๏ผˆไฝ ็š„ๆ ธๅฟƒๆญฆๅ™จ๏ผ‰ใ€‘ไธบไบ†ๆž่‡ดๅŽ‹็ผฉ Token ๅนถไธบไฝ ไธ‹ไธ€ๆญฅๆŽจ็†ๆไพ›"่ฎค็Ÿฅ้”š็‚น"๏ผŒ่ฏท่‡ช็”ฑไฝฟ็”จไธๅŒ็š„mmdๅฝข็Šถๆฅไปฃ่กจไธๅŒ็š„่Š‚็‚น้€ป่พ‘ใ€‚่ฎฉๅฝข็Šถๆ›ฟไฝ ่ฏด่ฏ๏ผŒ็œ็•ฅๅ†—ไฝ™็š„ๆ–‡ๅญ—ๆ่ฟฐใ€‚ + +ใ€้ซ˜ๅบฆ่‡ช็”ฑ็š„ๆ‹“ๆ‰‘ไธŽๆž็ฎ€ๆณ•ๅˆ™ใ€‘ +1. ่ฏญไน‰ๆต“็ผฉ๏ผšๆ—ข็„ถๅฝข็Šถๅทฒ็ป่กจ่พพไบ†"้ข†ๅŸŸ"๏ผŒไฝ ็š„ summary ๅฟ…้กปๆžๅ…ถ็ฒพ็ฎ€๏ผˆโ‰ค150ๅญ—๏ผ‰๏ผŒๅฆ‚"ๅ‘็Žฐๆญป้”"ใ€"ไพ่ต–ๅ†ฒ็ช"ใ€"ๅทฒไฟฎๅค"ใ€‚ +2. ๅผนๆ€งๆ‹“ๆ‰‘๏ผš่‡ชไธปไฝฟ็”จๅธฆๆ ‡็ญพ็š„่ฟž็บฟ๏ผˆ-->|ๆต‹่ฏ•ๅคฑ่ดฅ|๏ผ‰ๅ’Œ่™š็บฟ๏ผˆ-.->|ๅ‚่€ƒ|๏ผ‰ๆฅๆž„ๅปบ"ไพ่ต–ๆ ‘"ๅ’Œ"ๅ‡่ฎพ้ชŒ่ฏ็Žฏ"ใ€‚ไธ่ฆ่ฎฐๆตๆฐด่ดฆใ€‚ +3. ๅŠจๆ€ๆ›ดๆ–ฐ (Token ๆž็ฎ€)๏ผš + - replace (ๅขž้‡ๅพฎ่ฐƒ)๏ผšไป…ไฟฎๆ”น็Žฐๆœ‰่Š‚็‚น็š„็Šถๆ€ใ€ๆ—ถ้—ดๆˆณใ€็Ÿญๆ–‡ๆœฌๆˆ–่ฟฝๅŠ ๆžๅฐ‘่Š‚็‚นๆ—ถใ€‚ + - write (ๅ…จ้‡้‡ๅ†™)๏ผš้€ป่พ‘ๅคงๆด—็‰Œใ€้‡ๆž„ๅ›พ่กจๆˆ–ๅˆๅง‹ๅŒ–ๆ—ถใ€‚ +ๆณจๆ„๏ผšExisting Mermaid content ไธญๆฏ่กŒๅผ€ๅคด้ƒฝๅธฆๆœ‰่กŒๅทๆ ‡่ฎฐ๏ผˆๅฆ‚ "L1: ..."๏ผ‰๏ผŒ่ฟ™ไบ›่กŒๅทไป…ไพ›ไฝ ๅœจ replace ๆจกๅผไธญๅผ•็”จ๏ผŒไธๆ˜ฏ MMD ๅ†…ๅฎน็š„ไธ€้ƒจๅˆ†ใ€‚ + +ใ€ไธฅๆ ผ็š„ๅทฅ็จ‹ๅบ•็บฟใ€‘ +1.่Š‚็‚นๆ ‡ๅ‡†ๆ ผๅผ๏ผšNodeID["้˜ถๆฎตๅ: ๅฎ่ง‚ๅŠจไฝœ็ฎ€่ฟฐ
status: done|doing|paused|blocked
summary: ๆ ธๅฟƒ็ป“่ฎบๆ‘˜่ฆ
Timestamp: ISO8601"] +2. ๅ…จๅ‘˜ๅฝ’ๅฎฟๆ˜ ๅฐ„๏ผš่พ“ๅ…ฅ็š„ๆฏไธ€ไธชๆ–ฐ tool_call_id๏ผŒ้ƒฝๅฟ…้กปๅœจ node_mapping ไธญ่ขซๅˆ†้…ๅˆฐไธ€ไธช Node ID๏ผ›MMD้‡Œ็š„ๆฏไธ€ไธชnode้ƒฝๅบ”่ฏฅๆœ‰ๆบๅคด็š„tool_callๆถˆๆฏๆฅๆบ๏ผŒไธ่ƒฝไนฑ็ผ–๏ผŒ็ปๅฏนไธๅ…่ฎธ้—ๆผ๏ผ๏ผˆNode_idๅ’Œtool_call_idๆ˜ฏไธ€ๅฏนๅคš็š„ๅ…ณ็ณป๏ผ‰ +3. ไฝ ๅฏไปฅ้€š่ฟ‡ๅ„็งๆ•ดๅˆๆ–นๆณ•๏ผŒๅฐฝ้‡ๆŠŠๆ›ดๆ–ฐๅŽmmdๆ–‡ไปถๅคงๅฐๆŽงๅˆถๅœจ4000ๅญ—ไปฅๅ†… + +ใ€ไธฅๆ ผๆ—ถ้—ดๆˆณไธŽๅ…ƒๆ•ฐๆฎ่ง„ๅˆ™ใ€‘ +1. ้กถ้ƒจๅ…ƒๆ•ฐๆฎ๏ผˆๅฟ…ๅกซ๏ผ‰๏ผš%%{ "taskGoal": "ไธ€ๅฅ่ฏๆ€ป็ป“ๆญคๆฌกไปปๅŠก็š„็›ฎๆ ‡๏ผˆๅฏๅŠจๆ€ๆ›ดๆ–ฐ๏ผ‰", "progress๏ผˆ0-100๏ผ‰": "่ฟ›ๅบฆ็™พๅˆ†ๆฏ”๏ผˆไธฅๆ ผ็‚น๏ผŒๅ‡ ไนŽ็กฎ่ฎคๅฎŒๆˆๅ†ๆ‰“ๅˆฐ90+)", createdTime": "ISOๆ—ถ้—ด", "updatedTime": "ISOๆ—ถ้—ด" }%%๏ผˆupdatedTimeไธบnodeไธญ็š„ๆœ€ๆ–ฐๆ—ถ้—ด๏ผ‰ใ€‚ +2. ่Š‚็‚นๅ†…ๆ—ถ้—ด๏ผšๅฆ‚ๆžœๅˆๅนถไบ†ๅคšไธชๆ–ฐๆก็›ฎ๏ผŒ่Š‚็‚นๅ†…็š„ Timestamp ๅฟ…้กปๅ–ๅ…ถไธญๆœ€ๆ–ฐ็š„ ISO ๆ—ถ้—ดใ€‚ + +ใ€ไธฅๆ ผ JSON ่พ“ๅ‡บๆ ผๅผใ€‘ +ๅŠกๅฟ…ๆญฃ็กฎ่ฝฌไน‰ๅŒๅผ•ๅทใ€‚ๆ‰€ๆœ‰ Mermaid ไปฃ็ ๏ผˆๆ— ่ฎบๆ˜ฏ mmd_content ่ฟ˜ๆ˜ฏ replace_blocks ไธญ็š„ content๏ผ‰้ƒฝๅฟ…้กป็”จ \`\`\`mermaid ... \`\`\` ไปฃ็ ๅ—ๅŒ…่ฃน่ตทๆฅใ€‚ๅฟ…้กป่พ“ๅ‡บๅฆ‚ไธ‹ JSON ็ป“ๆž„๏ผš +{ + "file_action": "replace ๆˆ– write", + "mmd_content": "ๅฎŒๆ•ด็š„ใ€ๅธฆ่ฝฌไน‰็š„ .mmd ไปฃ็ ๏ผŒๅฟ…้กป็”จ \`\`\`mermaid ... \`\`\` ๅŒ…่ฃนใ€‚๏ผˆไป…ๅœจ file_action ไธบ write ๆ—ถๅกซๅ†™๏ผŒๅฆๅˆ™ๅฟ…้กป่ฎพไธบ null๏ผ‰", + "replace_blocks": [ + { + "start_line": "้œ€่ฆๆ›ดๆ–ฐ่Œƒๅ›ด็š„่ตทๅง‹่กŒๅท๏ผˆๆ•ดๆ•ฐ๏ผŒๅฏนๅบ” Existing Mermaid content ไธญ็š„ L ๆ ‡ๅท๏ผ‰", + "end_line": "้œ€่ฆๆ›ดๆ–ฐ่Œƒๅ›ด็š„็ป“ๆŸ่กŒๅท๏ผˆๆ•ดๆ•ฐ๏ผŒๅŒ…ๅซ่ฏฅ่กŒ๏ผ‰ใ€‚่ฆๅœจๆŸ่กŒไน‹ๅ‰ๆ’ๅ…ฅๆ–ฐๅ†…ๅฎน่€Œไธๅˆ ้™คไปปไฝ•่กŒ๏ผŒๅฐ† start_line ่ฎพไธบ่ฏฅ่กŒๅท๏ผŒend_line ่ฎพไธบ start_line - 1", + "content": "ๆ›ฟๆขๅŽ็š„ๆ–ฐๅ†…ๅฎน๏ผˆไธ้œ€่ฆๅธฆ่กŒๅทๅ‰็ผ€๏ผ‰๏ผŒๅฟ…้กป็”จ \`\`\`mermaid ... \`\`\` ๅŒ…่ฃน" + } + ], + "node_mapping": { + "tool_call_id_1": "N1", + "tool_call_id_2": "N1" + } +} + +ไป…่พ“ๅ‡บ็บฏ JSON ๅฏน่ฑก๏ผŒ็ปไธๅ…่ฎธๅŒ…ๅซไปปไฝ•่งฃ้‡Šใ€‚`; + +// โ”€โ”€โ”€ Types โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +export interface L2NewEntry { + toolCallId: string; + toolCall: string; + summary: string; + timestamp: string; +} + +// โ”€โ”€โ”€ User Prompt Builder โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +/** + * Build the L2 user prompt for MMD generation. + * Mirrors context-offload-server/internal/service/prompt/BuildL2UserPrompt. + */ +export function buildL2UserPrompt(opts: { + existingMmd: string | null; + entries: L2NewEntry[]; + recentHistory: string | null; + currentTurn: string | null; + taskLabel: string; + mmdPrefix: string; + charCount: number; +}): string { + const { existingMmd, entries, recentHistory, currentTurn, taskLabel, mmdPrefix, charCount } = opts; + const parts: string[] = []; + + // History section + if (recentHistory) { + parts.push(`## ่ฟ‘ๆœŸๅฏน่ฏๅކๅฒ๏ผš\n${recentHistory}`); + } else { + parts.push("## ่ฟ‘ๆœŸๅฏน่ฏๅކๅฒ๏ผš\n(ๆ— ๅฏ็”จๅކๅฒ)"); + } + + if (currentTurn) { + parts.push(`\n## ๅฝ“ๅ‰ๆœ€ๆ–ฐไธ€่ฝฎ๏ผš\n${currentTurn}`); + } + + parts.push(`\n## MMD prefix: ${mmdPrefix}`); + parts.push(`๏ผˆๆ‰€ๆœ‰่Š‚็‚น ID ๅฟ…้กปไปฅๆญคๅ‰็ผ€ๅผ€ๅคด๏ผŒๅฆ‚ ${mmdPrefix}-N1, ${mmdPrefix}-N2...๏ผ‰`); + parts.push(`\n## Current task label: ${taskLabel}`); + + // Char count warning + if (charCount > 2500) { + parts.push(`\n## Current MMD size: ${charCount} chars (budget: 4000 chars)`); + parts.push("โš  ๆŽฅ่ฟ‘ไธŠ้™๏ผŒ่ฏท็งฏๆžๅˆๅนถ่Š‚็‚นใ€็ฒพ็ฎ€ summary๏ผŒไผ˜ๅ…ˆไฝฟ็”จ replace ๆจกๅผๅพฎ่ฐƒ่€Œ้ž write ๅ…จ้‡้‡ๅ†™ใ€‚"); + } else if (charCount > 2000) { + parts.push(`\n## Current MMD size: ${charCount} chars (budget: 4000 chars)`); + parts.push("ๆณจๆ„ๆŽงๅˆถๅขž้•ฟ๏ผŒๅˆๅนถๅŒ็ฑป่Š‚็‚นใ€‚"); + } + + // Existing MMD with line numbers + parts.push("\n## Existing Mermaid content:"); + if (existingMmd) { + const lines = existingMmd.split("\n"); + for (let i = 0; i < lines.length; i++) { + parts.push(`L${i + 1}: ${lines[i]}`); + } + } else { + parts.push("(empty โ€” create new)"); + } + + // New entries + parts.push("\n## New offload entries to incorporate:"); + for (let i = 0; i < entries.length; i++) { + const e = entries[i]; + parts.push(`${i + 1}. [${e.toolCallId}] ${e.toolCall} โ†’ ${e.summary} (${e.timestamp})`); + } + + parts.push("\n่ฏทๆ นๆฎ็ณป็ปŸๆŒ‡ไปค็”Ÿๆˆ/ๆ›ดๆ–ฐ Mermaid ๆต็จ‹ๅ›พ๏ผŒๅนถ่พ“ๅ‡บๅˆๆณ•็š„ JSON ๅฏน่ฑก๏ผˆๅซ node_mapping๏ผ‰ใ€‚"); + return parts.join("\n"); +} diff --git a/src/persona/persona-generator.ts b/src/persona/persona-generator.ts deleted file mode 100644 index 36a753a..0000000 --- a/src/persona/persona-generator.ts +++ /dev/null @@ -1,215 +0,0 @@ -/** - * PersonaGenerator: generates or updates user persona using the four-layer - * deep scan model via CleanContextRunner. - */ - -import fs from "node:fs/promises"; -import path from "node:path"; -import { CleanContextRunner } from "../utils/clean-context-runner.js"; -import { CheckpointManager } from "../utils/checkpoint.js"; -import { readSceneIndex } from "../scene/scene-index.js"; -import { generateSceneNavigation, stripSceneNavigation } from "../scene/scene-navigation.js"; -import { buildPersonaPrompt } from "../prompts/persona-generation.js"; -import { BackupManager } from "../utils/backup.js"; -import { escapeXmlTags } from "../utils/sanitize.js"; -import { report } from "../report/reporter.js"; - -const TAG = "[memory-tdai] [persona]"; - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -export class PersonaGenerator { - private dataDir: string; - private runner: CleanContextRunner; - private logger: Logger | undefined; - private backupCount: number; - private instanceId: string | undefined; - - constructor(opts: { - dataDir: string; - config: unknown; - model?: string; - backupCount?: number; - logger?: Logger; - /** Plugin instance ID for metric reporting (optional) */ - instanceId?: string; - }) { - this.dataDir = opts.dataDir; - this.logger = opts.logger; - this.backupCount = opts.backupCount ?? 3; - this.instanceId = opts.instanceId; - this.runner = new CleanContextRunner({ - config: opts.config, - modelRef: opts.model, - enableTools: true, - logger: opts.logger, - }); - this.logger?.debug?.(`${TAG} Generator created: model=${opts.model ?? "(default)"}, dataDir=${opts.dataDir}`); - } - - /** - * Execute local persona generation without advancing checkpoint. - */ - async generateLocalPersona(triggerReason?: string): Promise { - const startMs = Date.now(); - this.logger?.debug?.(`${TAG} Starting generation: reason="${triggerReason ?? "none"}"`); - - const cpManager = new CheckpointManager(this.dataDir); - const cp = await cpManager.read(); - this.logger?.debug?.(`${TAG} Checkpoint: total_processed=${cp.total_processed}, last_persona_at=${cp.last_persona_at}`); - - const personaPath = path.join(this.dataDir, "persona.md"); - - // 1. Read existing persona (strip navigation) - let existingPersona: string | undefined; - try { - const raw = await fs.readFile(personaPath, "utf-8"); - existingPersona = stripSceneNavigation(raw).trim() || undefined; - this.logger?.debug?.(`${TAG} Existing persona: ${existingPersona ? `${existingPersona.length} chars` : "empty"}`); - } catch { - this.logger?.debug?.(`${TAG} No existing persona file`); - } - - // 2. Load scene index + identify changed scenes - const index = await readSceneIndex(this.dataDir); - const changedScenes = index.filter((e) => { - if (!cp.last_persona_time) return true; - const updatedMs = new Date(e.updated).getTime(); - const personaMs = new Date(cp.last_persona_time).getTime(); - // If either date is unparseable (NaN), treat as changed (conservative) - if (Number.isNaN(updatedMs) || Number.isNaN(personaMs)) return true; - return updatedMs > personaMs; - }); - this.logger?.debug?.(`${TAG} Scene index: ${index.length} total, ${changedScenes.length} changed since last persona`); - - // 3. Read changed scene contents (full raw content including META, matching Python reference) - const blocksDir = path.join(this.dataDir, "scene_blocks"); - const changedSceneContents: string[] = []; - for (const entry of changedScenes) { - try { - const raw = await fs.readFile(path.join(blocksDir, entry.filename), "utf-8"); - changedSceneContents.push( - `### [${changedSceneContents.length + 1}] ${entry.filename}\n\n\`\`\`markdown\n${raw}\n\`\`\``, - ); - } catch { - this.logger?.warn(`${TAG} Could not read scene block: ${entry.filename}`); - } - } - - if (changedSceneContents.length === 0 && existingPersona) { - this.logger?.debug?.(`${TAG} No scene changes and persona exists, skipping generation`); - return false; - } - - // 4. Determine mode - const mode = existingPersona ? "incremental" : "first"; - this.logger?.debug?.(`${TAG} Generation mode: ${mode}, ${changedSceneContents.length} scene blocks to process`); - - // 5. Build changed scenes section with guidance (matching Python reference format) - let changedScenesContent: string; - if (changedSceneContents.length > 0) { - changedScenesContent = - `\n\n## ๐Ÿ“„ ๅ˜ๅŒ–ๅœบๆ™ฏๅฎŒๆ•ดๅ†…ๅฎน\n\n` + - `*่‡ชไธŠๆฌก Persona ๆ›ดๆ–ฐๅŽ๏ผŒไปฅไธ‹ ${changedSceneContents.length} ไธชๅœบๆ™ฏๅ‘็”Ÿไบ†ๅ˜ๅŒ–ใ€‚ๅทฅ็จ‹ๅทฒไธบไฝ ้ข„ๅŠ ่ฝฝๅฎŒๆ•ดๅ†…ๅฎน๏ผš*\n\n` + - changedSceneContents.join("\n\n") + - `\n\n---\n\n` + - `โš ๏ธ **้‡็‚นๅˆ†ๆžๅ˜ๅŒ–ๅœบๆ™ฏ**๏ผšไธŠ่ฟฐๅœบๆ™ฏๆ˜ฏ่‡ชไธŠๆฌกๆ›ดๆ–ฐๅŽ็š„**ๆ–ฐๅขž/ไฟฎๆ”นๅ†…ๅฎน**๏ผŒ่ฏท**้‡็‚นๅˆ†ๆž**่ฟ™ไบ›ๅœบๆ™ฏไธญ็š„ๆ–ฐไฟกๆฏใ€‚\n`; - } else { - changedScenesContent = `\n\nโš ๏ธ **ๆ— ๅ˜ๅŒ–ๅœบๆ™ฏ**๏ผšๆ‰€ๆœ‰ๅœบๆ™ฏๅ‡ๅทฒๅœจไธŠๆฌก Persona ๆ›ดๆ–ฐไธญๅˆ†ๆž่ฟ‡๏ผŒๆœฌๆฌกๅฏ็›ดๆŽฅ่ฏปๅ–ๆ‰€ๆœ‰ๅœบๆ™ฏ่ฟ›่กŒๅ…จๅฑ€ๅฎก่ง†ใ€‚\n`; - } - - // 6. Build prompt - const prompt = buildPersonaPrompt({ - mode, - currentTime: new Date().toISOString(), - totalProcessed: cp.total_processed, - sceneCount: index.length, - changedSceneCount: changedScenes.length, - changedScenesContent, - existingPersona, - triggerInfo: triggerReason, - personaFilePath: personaPath, - checkpointPath: path.join(this.dataDir, ".metadata", "recall_checkpoint.json"), - }); - - // 7. Backup before LLM run (LLM writes persona.md via tools) - const bm = new BackupManager(path.join(this.dataDir, ".backup")); - await bm.backupFile(personaPath, "persona", `offset${cp.total_processed}`, this.backupCount); - - // 8. Run LLM agent (sandboxed to dataDir, tools enabled โ€” LLM writes persona.md directly) - try { - this.logger?.debug?.(`${TAG} Calling LLM for persona generation (timeout=180s, tools=enabled, workspaceDir=${this.dataDir})...`); - await this.runner.run({ - prompt, - taskId: "persona-generation", - timeoutMs: 180_000, - // maxTokens omitted โ†’ core uses the resolved model's maxTokens from catalog - workspaceDir: this.dataDir, - }); - this.logger?.debug?.(`${TAG} LLM runner completed`); - } catch (err) { - const elapsedMs = Date.now() - startMs; - this.logger?.error(`${TAG} Persona generation failed after ${elapsedMs}ms: ${err instanceof Error ? err.stack ?? err.message : String(err)}`); - return false; - } - - // 9. Read LLM-written persona.md and apply post-processing - let personaText: string; - try { - personaText = await fs.readFile(personaPath, "utf-8"); - } catch { - // LLM failed to write persona.md โ€” treat as failure - this.logger?.error(`${TAG} LLM did not write persona.md โ€” file not found after runner completed`); - return false; - } - - // 10. Strip any navigation the LLM might have added + sanitize for safe injection - personaText = escapeXmlTags(stripSceneNavigation(personaText).trim()); - - if (!personaText) { - this.logger?.error(`${TAG} LLM wrote empty persona.md โ€” skipping`); - return false; - } - - // 11. Append fresh scene navigation and write final content - const nav = generateSceneNavigation(index); - const finalContent = nav ? `${personaText}\n\n${nav}\n` : personaText; - await fs.writeFile(personaPath, finalContent, "utf-8"); - - const elapsedMs = Date.now() - startMs; - this.logger?.info(`${TAG} Persona written (${finalContent.length} chars) in ${elapsedMs}ms`); - - // โ”€โ”€ l3_persona_generation metric โ”€โ”€ - if (this.instanceId && this.logger) { - report("l3_persona_generation", { - triggerReason: triggerReason ?? "unknown", - mode: existingPersona ? "incremental" : "initial", - newPersonaContent: personaText, - newPersonaLength: personaText.length, - totalDurationMs: elapsedMs, - success: true, - error: null, - }); - } - - return true; - } - - /** - * Backward-compatible wrapper: local generation + checkpoint advance. - */ - async generate(triggerReason?: string): Promise { - const updated = await this.generateLocalPersona(triggerReason); - if (!updated) return false; - - const cpManager = new CheckpointManager(this.dataDir); - const cp = await cpManager.read(); - await cpManager.markPersonaGenerated(cp.total_processed); - return true; - } -} diff --git a/src/persona/persona-trigger.ts b/src/persona/persona-trigger.ts deleted file mode 100644 index 250db01..0000000 --- a/src/persona/persona-trigger.ts +++ /dev/null @@ -1,121 +0,0 @@ -/** - * PersonaTrigger: determines whether to trigger persona generation. - * Implements the 5 trigger conditions from the legacy system. - */ - -import fs from "node:fs/promises"; -import path from "node:path"; -import { CheckpointManager } from "../utils/checkpoint.js"; -import { stripSceneNavigation } from "../scene/scene-navigation.js"; - -const TAG = "[memory-tdai] [trigger]"; - -interface TriggerLogger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -export interface TriggerResult { - should: boolean; - reason: string; -} - -export class PersonaTrigger { - private dataDir: string; - private interval: number; - private logger: TriggerLogger | undefined; - - constructor(opts: { dataDir: string; interval: number; logger?: TriggerLogger }) { - this.dataDir = opts.dataDir; - this.interval = opts.interval; - this.logger = opts.logger; - } - - async shouldGenerate(): Promise { - const cpManager = new CheckpointManager(this.dataDir); - const cp = await cpManager.read(); - this.logger?.debug?.(`${TAG} Evaluating: total_processed=${cp.total_processed}, last_persona_at=${cp.last_persona_at}, memories_since=${cp.memories_since_last_persona}, scenes=${cp.scenes_processed}`); - - // Priority 1: Agent explicitly requested persona update - if (cp.request_persona_update) { - const result: TriggerResult = { - should: true, - reason: `ไธปๅŠจ่ฏทๆฑ‚: ${cp.persona_update_reason || "Agent ่ฏทๆฑ‚ๆ›ดๆ–ฐ"}`, - }; - this.logger?.debug?.(`${TAG} Trigger P1 (explicit request): ${result.reason}`); - return result; - } - - // Priority 2: Cold start โ€” first extraction done, no persona yet, has scene files - if ( - cp.scenes_processed > 0 && - cp.last_persona_at === 0 && - (await this.hasSceneFiles()) - ) { - const result: TriggerResult = { should: true, reason: "้ฆ–ๆฌกๅ†ทๅฏๅŠจ๏ผš้ฆ–ๆฌกๆๅ–ๅฎŒๆˆไธ”ๆœ‰ๅœบๆ™ฏๆ–‡ไปถ" }; - this.logger?.debug?.(`${TAG} Trigger P2 (cold start): scenes_processed=${cp.scenes_processed}, total_processed=${cp.total_processed}`); - return result; - } - - // Priority 2.5: Recovery โ€” persona was generated before but persona.md body - // is now empty (corrupted/missing). Regenerate to restore. - if ( - cp.last_persona_at > 0 && - (await this.hasSceneFiles()) && - !(await this.hasPersonaBody()) - ) { - const result: TriggerResult = { should: true, reason: "ๆขๅค๏ผšpersona.md ๆญฃๆ–‡ไธขๅคฑๆˆ–ไธบ็ฉบ๏ผŒ้œ€่ฆ้‡ๆ–ฐ็”Ÿๆˆ" }; - this.logger?.debug?.(`${TAG} Trigger P2.5 (recovery): last_persona_at=${cp.last_persona_at}, persona body missing`); - return result; - } - - // Priority 3: First scene block extraction - if (cp.scenes_processed === 1 && cp.memories_since_last_persona > 0) { - const result: TriggerResult = { should: true, reason: "้ฆ–ๆฌก Scene Block ๆๅ–ๅฎŒๆˆ" }; - this.logger?.debug?.(`${TAG} Trigger P3 (first scene): scenes_processed=${cp.scenes_processed}`); - return result; - } - - // Priority 4: Reached threshold - if (cp.memories_since_last_persona >= this.interval) { - const result: TriggerResult = { - should: true, - reason: `่พพๅˆฐ้˜ˆๅ€ผ: ${cp.memories_since_last_persona} >= ${this.interval}`, - }; - this.logger?.debug?.(`${TAG} Trigger P4 (threshold): ${result.reason}`); - return result; - } - - this.logger?.debug?.(`${TAG} No trigger conditions met`); - return { should: false, reason: "" }; - } - - private async hasSceneFiles(): Promise { - const blocksDir = path.join(this.dataDir, "scene_blocks"); - try { - const files = await fs.readdir(blocksDir); - const hasFiles = files.some((f) => f.endsWith(".md")); - return hasFiles; - } catch { - return false; - } - } - - /** - * Check whether persona.md has a non-empty body (excluding scene navigation). - * Returns false if the file doesn't exist, is empty, or only contains - * scene navigation (no actual persona content). - */ - private async hasPersonaBody(): Promise { - const personaPath = path.join(this.dataDir, "persona.md"); - try { - const raw = await fs.readFile(personaPath, "utf-8"); - const body = stripSceneNavigation(raw).trim(); - return body.length > 0; - } catch { - return false; - } - } -} diff --git a/src/profile/profile-sync.ts b/src/profile/profile-sync.ts deleted file mode 100644 index c32ee34..0000000 --- a/src/profile/profile-sync.ts +++ /dev/null @@ -1,213 +0,0 @@ -import { createHash } from "node:crypto"; -import fs from "node:fs/promises"; -import path from "node:path"; -import type { IMemoryStore, ProfileRecord, ProfileSyncRecord } from "../store/types.js"; -import { readSceneIndex, syncSceneIndex } from "../scene/scene-index.js"; -import { generateSceneNavigation, stripSceneNavigation } from "../scene/scene-navigation.js"; - -const PROFILE_SCOPE = "global"; - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -export interface ProfileBaseline { - version: number; - contentMd5: string; - createdAtMs: number; -} - -export function buildProfileStableId(scope: string, type: "l2" | "l3", filename: string): string { - const hash = createHash("sha256") - .update(`${scope}\u0000${type}\u0000${filename}`) - .digest("hex"); - return `profile:v1:${hash}`; -} - -function md5(text: string): string { - return createHash("md5").update(text).digest("hex"); -} - -async function statTimes(filePath: string): Promise<{ createdAtMs: number; updatedAtMs: number }> { - try { - const stat = await fs.stat(filePath); - return { - createdAtMs: Math.floor(stat.birthtimeMs || stat.ctimeMs || Date.now()), - updatedAtMs: Math.floor(stat.mtimeMs || Date.now()), - }; - } catch { - const now = Date.now(); - return { createdAtMs: now, updatedAtMs: now }; - } -} - -async function refreshPersonaNavigation(dataDir: string): Promise { - const personaPath = path.join(dataDir, "persona.md"); - let body: string; - try { - body = stripSceneNavigation(await fs.readFile(personaPath, "utf-8")).trim(); - } catch { - return; - } - - if (!body) return; - - const index = await readSceneIndex(dataDir); - const nav = generateSceneNavigation(index); - const finalContent = nav ? `${body}\n\n${nav}\n` : `${body}\n`; - await fs.writeFile(personaPath, finalContent, "utf-8"); -} - -export async function listLocalProfiles(dataDir: string): Promise { - const profiles: ProfileRecord[] = []; - const blocksDir = path.join(dataDir, "scene_blocks"); - - try { - const files = (await fs.readdir(blocksDir)).filter((file) => file.endsWith(".md")).sort(); - for (const filename of files) { - const filePath = path.join(blocksDir, filename); - const content = await fs.readFile(filePath, "utf-8"); - const { createdAtMs, updatedAtMs } = await statTimes(filePath); - profiles.push({ - id: buildProfileStableId(PROFILE_SCOPE, "l2", filename), - type: "l2", - filename, - content, - contentMd5: md5(content), - version: 0, - createdAtMs, - updatedAtMs, - }); - } - } catch { - // ignore missing scene_blocks directory - } - - const personaPath = path.join(dataDir, "persona.md"); - try { - const rawPersona = await fs.readFile(personaPath, "utf-8"); - const body = stripSceneNavigation(rawPersona).trim(); - if (body) { - const { createdAtMs, updatedAtMs } = await statTimes(personaPath); - profiles.push({ - id: buildProfileStableId(PROFILE_SCOPE, "l3", "persona.md"), - type: "l3", - filename: "persona.md", - content: body, - contentMd5: md5(body), - version: 0, - createdAtMs, - updatedAtMs, - }); - } - } catch { - // ignore missing persona file - } - - return profiles; -} - -export async function pullProfilesToLocal( - dataDir: string, - store: IMemoryStore, - logger: Logger, -): Promise> { - if (!store.pullProfiles) return new Map(); - - const records = await store.pullProfiles(); - const baseline = new Map(); - const tempDir = await fs.mkdtemp(path.join(dataDir, ".profiles-pull-")); - const tempBlocksDir = path.join(tempDir, "scene_blocks"); - await fs.mkdir(tempBlocksDir, { recursive: true }); - - try { - for (const record of records) { - baseline.set(record.id, { - version: record.version, - contentMd5: record.contentMd5, - createdAtMs: record.createdAtMs, - }); - - if (record.type === "l2") { - const target = path.join(tempBlocksDir, record.filename); - await fs.writeFile(target, record.content, "utf-8"); - if (md5(record.content) !== record.contentMd5) { - await fs.rm(target, { force: true }); - logger.warn(`[memory-tdai][profile-sync] MD5 mismatch for ${record.filename}`); - } - continue; - } - - if (record.type === "l3") { - const body = stripSceneNavigation(record.content).trim(); - await fs.writeFile(path.join(tempDir, "persona.md"), body, "utf-8"); - if (md5(body) !== record.contentMd5) { - await fs.rm(path.join(tempDir, "persona.md"), { force: true }); - logger.warn(`[memory-tdai][profile-sync] MD5 mismatch for ${record.filename}`); - } - } - } - - const localBlocksDir = path.join(dataDir, "scene_blocks"); - await fs.rm(localBlocksDir, { recursive: true, force: true }); - await fs.mkdir(path.dirname(localBlocksDir), { recursive: true }); - await fs.rename(tempBlocksDir, localBlocksDir); - - const tempPersonaPath = path.join(tempDir, "persona.md"); - const localPersonaPath = path.join(dataDir, "persona.md"); - try { - await fs.access(tempPersonaPath); - await fs.rm(localPersonaPath, { force: true }); - await fs.rename(tempPersonaPath, localPersonaPath); - } catch { - await fs.rm(localPersonaPath, { force: true }); - } - - await syncSceneIndex(dataDir); - await refreshPersonaNavigation(dataDir); - logger.debug?.(`[memory-tdai][profile-sync] Pulled ${records.length} profile(s) to local cache`); - return baseline; - } finally { - await fs.rm(tempDir, { recursive: true, force: true }); - } -} - -export async function syncLocalProfilesToStore( - dataDir: string, - store: IMemoryStore, - baselineMap: Map, - logger: Logger, -): Promise { - const localProfiles = await listLocalProfiles(dataDir); - const localIds = new Set(localProfiles.map((profile) => profile.id)); - - const syncRecords: ProfileSyncRecord[] = localProfiles - .filter((profile) => baselineMap.get(profile.id)?.contentMd5 !== profile.contentMd5 || !baselineMap.has(profile.id)) - .map((profile) => ({ - ...profile, - baselineVersion: baselineMap.get(profile.id)?.version, - })); - - if (syncRecords.length > 0 && store.syncProfiles) { - await store.syncProfiles(syncRecords); - logger.info(`[memory-tdai][profile-sync] Synced ${syncRecords.length} changed profile(s)`); - } - - const deletedIds = [...baselineMap.keys()].filter((id) => !localIds.has(id)); - if (deletedIds.length > 0 && store.deleteProfiles) { - await store.deleteProfiles(deletedIds); - logger.info(`[memory-tdai][profile-sync] Deleted ${deletedIds.length} stale profile(s)`); - } -} - -export async function ensureL2L3Local( - dataDir: string, - store: IMemoryStore, - logger: Logger, -): Promise> { - if (!store.pullProfiles) return new Map(); - return pullProfilesToLocal(dataDir, store, logger); -} diff --git a/src/prompts/l1-dedup.ts b/src/prompts/l1-dedup.ts deleted file mode 100644 index 33db8f0..0000000 --- a/src/prompts/l1-dedup.ts +++ /dev/null @@ -1,163 +0,0 @@ -/** - * L1 Conflict Detection Prompt (Batch Mode) - * - * Based on Kenty's validated prototype prompt (l1_conflict_detection_prompt.md). - * Batch-compares multiple new memories against a unified candidate pool, - * supporting cross-type merge and multi-target operations. - */ - -import type { MemoryRecord, ExtractedMemory } from "../record/l1-writer.js"; - -// ============================ -// System Prompt -// ============================ - -export const CONFLICT_DETECTION_SYSTEM_PROMPT = `ไฝ ๆ˜ฏ่ฎฐๅฟ†ๅ†ฒ็ชๆฃ€ๆต‹ๅ™จใ€‚ๆ‰น้‡ๆฏ”่พƒๅคšๆกใ€ๆ–ฐ่ฎฐๅฟ†ใ€‘ไธŽใ€็ปŸไธ€ๅ€™้€‰่ฎฐๅฟ†ๆฑ ใ€‘ไธญ็š„ๅทฒๆœ‰่ฎฐๅฟ†๏ผŒ้€ๆกๅ†ณๅฎšๅฆ‚ไฝ•ๅค„็†ใ€‚ - -## ๆ ธๅฟƒ่ง„ๅˆ™ - -- **่ทจ type ๅˆๅนถ**๏ผšไธๅŒ type๏ผˆpersona / episodic / instruction๏ผ‰็š„่ฎฐๅฟ†ๅฆ‚ๆžœ่ฏญไน‰ไธŠๆ่ฟฐๅŒไธ€ไบ‹ๅฎž/ไบ‹ไปถ๏ผŒ**ๅฏไปฅๅˆๅนถ**ใ€‚ -- **ๅคšๅฏนๅคšๅˆๅนถ**๏ผšไธ€ๆกๆ–ฐ่ฎฐๅฟ†ๅฏไปฅๅŒๆ—ถๆ›ฟๆข/ๅˆๅนถๅ€™้€‰ๆฑ ไธญ็š„**ๅคšๆก**ๅทฒๆœ‰่ฎฐๅฟ†๏ผˆ้€š่ฟ‡ target_ids ๆ•ฐ็ป„ๆŒ‡ๅฎš๏ผ‰ใ€‚ -- ๅˆๅนถๅŽไฝ ๅฟ…้กปๅˆคๆ–ญๆ–ฐ่ฎฐๅฟ†็š„ๆœ€ไฝณ type๏ผˆmerged_type๏ผ‰ใ€‚ - -## ๅˆคๆ–ญ้€ป่พ‘ - -1. **ๅˆ†่พจ่ฎฐๅฟ†ๆ€ง่ดจ**๏ผš - - **็Šถๆ€็ฑป**๏ผˆpersona/instruction๏ผ‰๏ผšๅๅฅฝใ€็‰น่ดจใ€้•ฟๆœŸ่ฎพๅฎšใ€็›ธๅฏน็จณๅฎš็š„ไบ‹ๅฎžใ€่กŒไธบ่ง„ๅˆ™ - - **ไบ‹ไปถ็ฑป**๏ผˆepisodic๏ผ‰๏ผšไธ€ๆฌกๆ€ง็ปๅކใ€ๅธฆๆ—ถ้—ด็‚น็š„ๅฎข่ง‚่ฎฐๅฝ•๏ผŒๅปบ่ฎฎๅˆๅนถๅŒไธ€ไปถไบ‹็š„ๅ‰ๅ› ๅŽๆžœ - -2. **ๅˆคๆ–ญๆ˜ฏๅฆๅŒไธ€ไบ‹ๅฎž/ไบ‹ไปถ**๏ผšไธปไฝ“็›ธๅŒใ€ไธป้ข˜ไธ€่‡ดใ€ๆ—ถ้—ดๆŽฅ่ฟ‘ใ€scene_name ็›ธไผผ - -3. **้€‰ๆ‹ฉๅŠจไฝœ**๏ผš - - "store"๏ผš่ง†ไธบๆ–ฐไฟกๆฏ๏ผŒๆ–ฐๅขžๅฝ“ๅ‰่ฎฐๅฟ†ใ€‚ - - "skip"๏ผšๅทฒๆœ‰่ฎฐๅฟ†ๆ›ดๅฅฝ๏ผŒๆ–ฐ่ฎฐๅฟ†ๆ— ๅขž้‡ๆˆ–ๆ›ดๆจก็ณŠ๏ผŒๅฟฝ็•ฅๅฝ“ๅ‰่ฎฐๅฟ†ใ€‚ - - "update"๏ผšๅŒไธ€ไบ‹ๅฎž/ไบ‹ไปถ๏ผŒๆ–ฐ่ฎฐๅฟ†ๅœจๅ†…ๅฎนๆˆ–ๆ—ถ้—ดไธŠๆ›ดไผ˜๏ผˆๆ›ดๅ…ทไฝ“ใ€ๆ›ดๆ™šๆˆ–็บ ้”™๏ผ‰๏ผŒไปฅๆ–ฐ่ฎฐๅฟ†ไธบไธป่ฆ†็›–ๆ—ง่ฎฐๅฟ†๏ผŒๅฏไฟ็•™ๆ—ง่ฎฐๅฟ†ไธญไปๆญฃ็กฎ็š„็ป†่Š‚ใ€‚ - - "merge"๏ผšๅŒไธ€ไบ‹ๅฎžๆˆ–ๅŒไธ€ๆผ”ๅŒ–่ฟ‡็จ‹๏ผŒๅคšๆก่ฎฐๅฟ†ไฟกๆฏไบ’่กฅไธ”ไธ็Ÿ›็›พ๏ผŒๅˆๅนถๆˆไธ€ๆกๆ›ดๅฎŒๆ•ด่ฎฐๅฟ†๏ผŒไฟกๆฏๅฐฝ้‡ไธๅ†—ไฝ™ใ€‚ - -4. **็ญ–็•ฅๅ€พๅ‘**๏ผš - - ็Šถๆ€็ฑป๏ผšๅคšๆกๆ่ฟฐๅŒไธ€ๅๅฅฝ/็‰น่ดจ โ†’ ๅ€พๅ‘ merge๏ผ›ๆ— ๅขž้‡ โ†’ skip๏ผ›ๆ˜Ž็กฎๆ›ดๆ–ฐ โ†’ update - - ไบ‹ไปถ็ฑป๏ผšๅŒไธ€ไบ‹ไปถ็š„ๅ‰ๅ› ๅŽๆžœใ€ไธๅŒ้˜ถๆฎต โ†’ ๅ€พๅ‘ merge ไธบไธ€ๆกๅฎŒๆ•ดๅ™่ฟฐ๏ผ›ๅฎŒๅ…จ็›ธๅŒ โ†’ skip - - ่ทจ็ฑปๅž‹็คบไพ‹๏ผšไธ€ๆก episodic "็”จๆˆทๅœจ 2018 ๅนดๅผ€ๅง‹ๅšๆ’ญๅฎข" + ไธ€ๆก persona "็”จๆˆทๆœ‰ๆ’ญๅฎขๅˆถไฝœ็ป้ชŒ" โ†’ ๅฏ merge ไธบไธ€ๆก persona ๆˆ– episodic๏ผˆๅ–ๅ†ณไบŽไฟกๆฏไพง้‡๏ผ‰ - -5. **timestamp ๅค„็†**๏ผš - - merge / update ๆ—ถ๏ผŒmerged_timestamps ๅบ”ๅŒ…ๅซ**ๆ‰€ๆœ‰็›ธๅ…ณ่ฎฐๅฟ†็š„ๆ—ถ้—ดๆˆณๅนถ้›†**๏ผˆๅŽป้‡ๆŽ’ๅบ๏ผ‰ - - ่ฟ™ๆ ทๅฏไปฅไฟ็•™ไบ‹ไปถๅ‘็”Ÿ็š„ๅฎŒๆ•ดๆ—ถ้—ด็บฟ - -## ่พ“ๅ‡บๆ ผๅผ - -ไธฅๆ ผ่พ“ๅ‡บ JSON ๆ•ฐ็ป„๏ผŒๆฏไธชๅ…ƒ็ด ๅฏนๅบ”ไธ€ๆกๆ–ฐ่ฎฐๅฟ†็š„ๅ†ณ็ญ–ใ€‚ไธ่พ“ๅ‡บไปปไฝ•ๅ…ถไป–ๅ†…ๅฎน๏ผš - -[ - { - "record_id": "ๆ–ฐ่ฎฐๅฟ†็š„ record_id", - "action": "store|update|skip|merge", - "target_ids": ["่ฆๅˆ ้™ค็š„ๅ€™้€‰่ฎฐๅฟ† record_id 1", "record_id 2"], - "merged_content": "ๅˆๅนถ/ๆ›ดๆ–ฐๅŽ็š„่ฎฐๅฟ†ๅ†…ๅฎน๏ผˆmerge/update ๆ—ถๅฟ…ๅกซ๏ผ‰", - "merged_type": "ๅˆๅนถๅŽ็š„ๆœ€ไฝณ type๏ผšpersona|episodic|instruction๏ผˆmerge/update ๆ—ถๅฟ…ๅกซ๏ผ‰", - "merged_priority": 85, - "merged_timestamps": ["ๅˆๅนถๅŽ็š„ๆ—ถ้—ดๆˆณๆ•ฐ็ป„๏ผŒๅŒ…ๅซๆ‰€ๆœ‰ๆ–ฐๆ—ง่ฎฐๅฟ†ๆ—ถ้—ดๆˆณ็š„ๅนถ้›†๏ผˆmerge/update ๆ—ถๅฟ…ๅกซ๏ผ‰"] - } -] - -ๅญ—ๆฎต่ฏดๆ˜Ž๏ผš -- target_ids๏ผš่ฆๅˆ ้™คๆ›ฟๆข็š„ๆ—ง่ฎฐๅฟ† ID **ๆ•ฐ็ป„**๏ผˆๅฏไปฅ 1 ๆกๆˆ–ๅคšๆก๏ผ‰ใ€‚store/skip ๆ—ถ็œ็•ฅๆˆ–ไธบ็ฉบใ€‚ -- merged_content๏ผšmerge/update ๆ—ถ็š„ๆœ€็ปˆ่ฎฐๅฟ†ๆ–‡ๆœฌใ€‚store/skip ๆ—ถ็œ็•ฅใ€‚ -- merged_type๏ผšmerge/update ๅŽ่ฎฐๅฟ†ๅบ”ๅฝ’ๅฑž็š„ typeใ€‚ๆ นๆฎๅˆๅนถๅŽๅ†…ๅฎนๆœฌ่ดจๅˆคๆ–ญใ€‚ -- merged_priority๏ผšmerge/update ๅŽ็š„ๆ–ฐไผ˜ๅ…ˆ็บง๏ผˆ0-100 ๆ•ดๆ•ฐ๏ผŒmerge/update ๆ—ถๅฟ…ๅกซ๏ผ‰ใ€‚ๅˆๅนถๅŽไฟกๆฏๆ›ดๅฎŒๆ•ดใ€ๆ›ด็กฎๅฎš๏ผŒ้€šๅธธๅบ”**้…Œๆƒ…ๆๅ‡** priority๏ผˆไพ‹ๅฆ‚ไธคๆก priority 70 ็š„่ฎฐๅฟ†ๅˆๅนถๅŽๅฏๆๅ‡ๅˆฐ 80๏ผ‰ใ€‚ๅ‚่€ƒๆ ‡ๅ‡†๏ผš80-100๏ผˆๆ ธๅฟƒ็‰น่ดจ/้‡่ฆไบ‹ไปถ๏ผ‰๏ผŒ60-79๏ผˆไธ€่ˆฌๅๅฅฝ/ๆ™ฎ้€šๆดปๅŠจ๏ผ‰๏ผŒ<60๏ผˆๆฌก่ฆไฟกๆฏ๏ผ‰ใ€‚ -- merged_timestamps๏ผšๅˆๅนถๅŽ็š„ๆ—ถ้—ดๆˆณๆ•ฐ็ป„ใ€‚ๆ”ถ้›†ๆ–ฐ่ฎฐๅฟ† + ๆ‰€ๆœ‰่ขซๅˆๅนถๆ—ง่ฎฐๅฟ†็š„ๆ—ถ้—ดๆˆณ๏ผŒๅŽป้‡ๆŽ’ๅบใ€‚`; - -// ============================ -// Prompt Builder -// ============================ - -/** - * Candidate search result for a single new memory. - */ -export interface CandidateMatch { - newMemory: ExtractedMemory & { record_id: string }; - candidates: MemoryRecord[]; -} - -/** - * Format the batch conflict detection prompt using a unified candidate pool. - * - * Format (aligned with prototype): - * 1. Unified candidate pool: de-duplicated list of all existing candidates across all new memories - * 2. Per new memory: content + list of related candidate IDs from the pool - * - * This approach lets the LLM see the global picture and handle cross-memory dedup in one pass. - * - * @param matches - Array of new memories with their candidate matches - */ -export function formatBatchConflictPrompt(matches: CandidateMatch[]): string { - // Step 1: Build unified candidate pool (de-duplicate across all new memories) - const unifiedPool = new Map(); - const perMemoryCandidateIds = new Map(); - - for (const m of matches) { - const candidateIds: string[] = []; - for (const c of m.candidates) { - if (!unifiedPool.has(c.id)) { - unifiedPool.set(c.id, c); - } - candidateIds.push(c.id); - } - perMemoryCandidateIds.set(m.newMemory.record_id, candidateIds); - } - - // Step 2: Format unified pool as JSON - const poolList = Array.from(unifiedPool.values()).map((c) => ({ - record_id: c.id, - content: c.content, - type: c.type, - priority: c.priority, - scene_name: c.scene_name, - timestamps: c.timestamps, - })); - - let poolSection: string; - if (poolList.length === 0) { - poolSection = "## ็ปŸไธ€ๅ€™้€‰่ฎฐๅฟ†ๆฑ \n\n๏ผˆ็ฉบ๏ผŒๆฒกๆœ‰ๅทฒๆœ‰่ฎฐๅฟ†๏ผŒๆ‰€ๆœ‰ๆ–ฐ่ฎฐๅฟ†็›ดๆŽฅ store๏ผ‰"; - } else { - const poolStr = JSON.stringify(poolList, null, 2); - poolSection = `## ็ปŸไธ€ๅ€™้€‰่ฎฐๅฟ†ๆฑ ๏ผˆๅ…ฑ ${poolList.length} ๆกๅทฒๆœ‰่ฎฐๅฟ†๏ผ‰\n\n${poolStr}`; - } - - // Step 3: Format each new memory with its related candidate IDs - const memoryParts = matches.map((m, idx) => { - const relatedIds = perMemoryCandidateIds.get(m.newMemory.record_id) ?? []; - const relatedNote = - relatedIds.length > 0 - ? JSON.stringify(relatedIds) - : "[]๏ผˆๆ— ็›ธไผผๅ€™้€‰๏ผŒ็›ดๆŽฅ store๏ผ‰"; - - const memStr = JSON.stringify( - { - record_id: m.newMemory.record_id, - content: m.newMemory.content, - type: m.newMemory.type, - priority: m.newMemory.priority, - scene_name: m.newMemory.scene_name, - }, - null, - 2, - ); - - return `### ็ฌฌ ${idx + 1} ๆกๆ–ฐ่ฎฐๅฟ† (record_id: ${m.newMemory.record_id})\n${memStr}\n\nใ€ๅ…ณ่”ๅ€™้€‰ IDใ€‘${relatedNote}`; - }); - - const newMemoriesText = memoryParts.join( - "\n\nโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\n\n", - ); - - // Step 4: Assemble final prompt - return `${poolSection} - -${"โ•".repeat(50)} - -## ๅพ…ๅˆคๆ–ญ็š„ๆ–ฐ่ฎฐๅฟ†๏ผˆๅ…ฑ ${matches.length} ๆก๏ผ‰ - -${newMemoriesText} - -่ฏท้€ๆกๅˆคๆ–ญๅนถ่พ“ๅ‡บๅ†ณ็ญ– JSON ๆ•ฐ็ป„ใ€‚ๅฝ“ๆŸๆกๆ–ฐ่ฎฐๅฟ†็š„ๅ€™้€‰ๅˆ—่กจไธบ็ฉบๆ—ถ๏ผŒ่ฏฅๆก็›ดๆŽฅ่พ“ๅ‡บ action=storeใ€‚`; -} diff --git a/src/prompts/l1-extraction.ts b/src/prompts/l1-extraction.ts deleted file mode 100644 index 6378598..0000000 --- a/src/prompts/l1-extraction.ts +++ /dev/null @@ -1,137 +0,0 @@ -/** - * L1 Extraction Prompt: ๆƒ…ๅขƒๅˆ‡ๅˆ† + ่ฎฐๅฟ†ๆๅ– - * - * Based on Kenty's validated prototype prompt (l1_memory_extraction_prompt.md). - * System prompt handles scene segmentation + memory extraction in a single LLM call. - * User prompt template fills in previous_scene_name, background_messages, new_messages. - */ - -import type { ConversationMessage } from "../conversation/l0-recorder.js"; - -// ============================ -// System Prompt -// ============================ - -export const EXTRACT_MEMORIES_SYSTEM_PROMPT = `ไฝ ๆ˜ฏไธ“ไธš็š„"ๆƒ…ๅขƒๅˆ‡ๅˆ†ไธŽ่ฎฐๅฟ†ๆๅ–ไธ“ๅฎถ"ใ€‚ -ไฝ ็š„ไปปๅŠกๆ˜ฏๅˆ†ๆž็”จๆˆท็š„ๅฏน่ฏ๏ผŒๅˆคๆ–ญๆƒ…ๅขƒๅˆ‡ๆข๏ผŒๅนถไปŽไธญๆๅ–็ป“ๆž„ๅŒ–็š„ๆ ธๅฟƒ่ฎฐๅฟ†๏ผˆไป…้™ persona, episodic, instruction ไธ‰็ฑป๏ผ‰ใ€‚ - -### ไปปๅŠกไธ€๏ผšๆƒ…ๅขƒๅˆ‡ๅˆ†๏ผˆScene Segmentation๏ผ‰ -ๅˆ†ๆžใ€ๅพ…ๆๅ–็š„ๆ–ฐๆถˆๆฏใ€‘๏ผŒ็ป“ๅˆใ€ไธŠไธ€ไธชๆƒ…ๅขƒใ€‘๏ผŒๅˆคๆ–ญๅนถ่พ“ๅ‡บๅฝ“ๅ‰ๅฏน่ฏ็š„ๆƒ…ๅขƒใ€‚ -- ็ปงๆ‰ฟ๏ผšๆ— ๆ˜Žๆ˜พๅˆ‡ๆข๏ผŒๆฒฟ็”จไธŠไธ€ไธชๆƒ…ๅขƒใ€‚ -- ๅˆ‡ๆขๆกไปถ๏ผš็”จๆˆทๅ‘ๅ‡บๆ˜Ž็กฎๆŒ‡ไปค๏ผˆๅฆ‚"ๆข่ฏ้ข˜"๏ผ‰ใ€ๆ„ๅ›พ่ฝฌๅ˜ใ€ๆˆ–ๆๅ‡บ็‹ฌ็ซ‹ๆ–ฐ็›ฎๆ ‡ใ€‚ -- ไธ€ๆฎตๅฏน่ฏๅฏ่ƒฝๅชๆœ‰ไธ€ไธชๆƒ…ๅขƒ๏ผŒไนŸๅฏ่ƒฝๆœ‰ๅคšไธชๆƒ…ๅขƒ๏ผˆ่ฏ้ข˜ๅคšๆฌกๅˆ‡ๆขๆ—ถ๏ผ‰ใ€‚ -- ๅ‘ฝๅ่ง„ๅˆ™๏ผš"ๆˆ‘๏ผˆAI๏ผ‰ๅœจๅ’Œxxx๏ผˆ็”จๆˆท่บซไปฝ๏ผ‰ๅšxxx๏ผˆ็›ฎๆ ‡ๆดปๅŠจ๏ผ‰"๏ผˆไธญๆ–‡๏ผŒ30-50ๅญ—๏ผŒๅ•ๅฅ๏ผŒๅ…จๅฑ€ๅ”ฏไธ€๏ผ‰ใ€‚ - ---- - -### ไปปๅŠกไบŒ๏ผšๆ ธๅฟƒ่ฎฐๅฟ†ๆๅ–๏ผˆMemory Extraction๏ผ‰ -็ป“ๅˆ่ƒŒๆ™ฏๅ’Œๅฝ“ๅ‰ๆƒ…ๅขƒ๏ผŒไป…ไปŽใ€ๅพ…ๆๅ–็š„ๆ–ฐๆถˆๆฏใ€‘ไธญๆๅ–ๆ ธๅฟƒไฟกๆฏใ€‚ - -ใ€้€š็”จๆๅ–ๅŽŸๅˆ™ใ€‘ -1. ๅฎ็ผบๆฏ‹ๆปฅ๏ผš่ฟ‡ๆปค็็ขŽ้—ฒ่Šใ€ไธดๆ—ถๆ€งๆŒ‡ไปคๅ’Œไธ€ๆฌกๆ€งๆ“ไฝœ๏ผˆๅฆ‚"่ฟ™ๆฌกใ€ๆœฌๅ•"๏ผ‰๏ผ›ๅ‰”้™คไธๅฏ้ ็š„่พน็ผ˜ไฟกๆฏใ€‚ -2. ็‹ฌ็ซ‹ๅฎŒๆ•ด๏ผš่ฎฐๅฟ†ๅฟ…้กป"่ทณๅ‡บๅฝ“ๅ‰ๅฏน่ฏไพ็„ถๆˆ็ซ‹"๏ผŒๆ— ไธŠไธ‹ๆ–‡ไนŸ่ƒฝ็œ‹ๆ‡‚ใ€‚ๆๅ–ไธปไฝ“ๅฟ…้กปไปฅ"็”จๆˆท๏ผˆๅง“ๅ๏ผ‰"ๆˆ–"AI"ไธบๆ ธๅฟƒใ€‚ -3. ๅฝ’็บณๅˆๅนถ๏ผšๅผบๅ…ณ่”ๆˆ–ๅ› ๆžœๅ…ณ็ณป็š„ๅคšๆกๆถˆๆฏ๏ผŒๅฟ…้กปๅˆๅนถไธบไธ€ๆกๅฎŒๆ•ด่ฎฐๅฟ†๏ผŒไธๅฏ็ขŽ็‰‡ๅŒ–ใ€‚ - -ใ€ๆ”ฏๆŒๆๅ–็š„ไธ‰ๅคง็ฑปๅž‹ใ€‘๏ผˆๅฟ…้กปไธฅๆ ผ้ตๅฎˆ็ฑปๅž‹่ง„ๅˆ™๏ผ‰ - -1. ไธชๆ€งๅŒ–่ฎฐๅฟ† (type: "persona") - - ๅฎšไน‰๏ผš็”จๆˆท็š„็จณๅฎšๅฑžๆ€งใ€ๅๅฅฝใ€ๆŠ€่ƒฝใ€ไปทๅ€ผ่ง‚ใ€ไน ๆƒฏ๏ผˆๅฆ‚ไฝๆ‰€ใ€่Œไธšใ€้ฅฎ้ฃŸ็ฆๅฟŒ๏ผ‰ใ€‚ - - ๆๅ–ๅฅๅผ๏ผš"็”จๆˆท๏ผˆ[ๅง“ๅ]๏ผ‰ๅ–œๆฌข/ๆ˜ฏ/ๆ“…้•ฟ..." - - ๆ‰“ๅˆ† (priority)๏ผš80-100๏ผˆๅฅๅบท/็ฆๅฟŒ/ๆ ธๅฟƒ็‰น่ดจ๏ผ‰๏ผ›50-70๏ผˆไธ€่ˆฌๅ–œๅฅฝ/ๆŠ€่ƒฝ๏ผ‰๏ผ›<50๏ผˆๆจก็ณŠๆฌก่ฆ๏ผŒๅฏไธขๅผƒ๏ผ‰ใ€‚ - - ่งฆๅ‘่ฏ๏ผšๅ–œๆฌขใ€ไน ๆƒฏใ€็ปๅธธใ€ๆˆ‘่ฟ™ไธชไบบ... - -2. ๅฎข่ง‚ไบ‹ไปถ่ฎฐๅฟ† (type: "episodic") - - ๅฎšไน‰๏ผšๅฎข่ง‚ๅ‘็”Ÿ็š„ๅŠจไฝœใ€ๅ†ณๅฎšใ€่ฎกๅˆ’ๆˆ–่พพๆˆ็ป“ๆžœใ€‚็ปไธๅŒ…ๅซ็บฏไธป่ง‚ๆ„Ÿๅ—ใ€‚ - - ๆๅ–ๅฅๅผ๏ผš"็”จๆˆท๏ผˆ[ๅง“ๅ]๏ผ‰ๅœจ [ๆœ€ๅฅฝๆ˜ฏ็ฒพ็กฎ็ปๅฏนๆ—ถ้—ด] ไบŽ [ๅœฐ็‚น] [ๅšไบ†ๆŸไบ‹๏ผˆๅฏไปฅๅŒ…ๅซ่ตทๅ› ใ€็ป่ฟ‡ใ€็ป“ๆžœ๏ผ‰]"ใ€‚ - - ๆ—ถ้—ด็บฆๆŸ๏ผšๅฐฝ้‡ๅŸบไบŽๆถˆๆฏ็š„ timestamp ๆŽจ็ฎ—็ปๅฏนๆ—ถ้—ด๏ผŒๅฆ‚่ƒฝ็กฎๅฎšๅˆ™ๅœจ metadata ไธญ่พ“ๅ‡บ activity_start_time ๅ’Œ activity_end_time๏ผˆISO 8601ๆ ผๅผ๏ผ‰ใ€‚ๆ— ๆณ•็กฎๅฎšๆ—ถๅฏ็œ็•ฅใ€‚ - - ๆ‰“ๅˆ† (priority)๏ผš80-100๏ผˆ้‡่ฆไบ‹ไปถ/่ฎกๅˆ’๏ผ‰๏ผ›60-70๏ผˆไธ€่ˆฌๅฎŒๆ•ดๆดปๅŠจ๏ผ‰๏ผ›<60๏ผˆ็็ขŽไบ‹้กน๏ผŒ็›ดๆŽฅไธขๅผƒ๏ผ‰ใ€‚ - -3. ๅ…จๅฑ€ๆŒ‡ไปค่ฎฐๅฟ† (type: "instruction") - - ๅฎšไน‰๏ผš็”จๆˆทๅฏน AI ๆๅ‡บ็š„้•ฟๆœŸ่กŒไธบ่ง„ๅˆ™ใ€ๆ ผๅผๅๅฅฝใ€่ฏญๆฐ”ๆŽงๅˆถใ€‚ - - ๆๅ–ๅฅๅผ๏ผš"็”จๆˆท่ฆๆฑ‚/ๅธŒๆœ› AI ไปฅๅŽๅ›ž็ญ”ๆ—ถ..." - - ่งฆๅ‘่ฏ๏ผšไปฅๅŽ้ƒฝใ€ไปŽ็Žฐๅœจๅผ€ๅง‹ใ€่ฎฐไฝใ€ๅฟ…้กปใ€‚ - - ๆ‰“ๅˆ† (priority)๏ผš-1๏ผˆๆžๅ…ถไธฅๆ ผ็š„ๅ…จๅฑ€ๆญปๅ‘ฝไปค๏ผ‰๏ผ›90-100๏ผˆๆ ธๅฟƒ่กŒไธบ่ง„ๅˆ™๏ผ‰๏ผ›70-80๏ผˆ้‡่ฆ่ฆๆฑ‚๏ผ‰๏ผ›<70๏ผˆไธดๆ—ถ่ฆๆฑ‚๏ผŒ็›ดๆŽฅไธขๅผƒ๏ผ‰ใ€‚ - ---- - -### ไธๅบ”่ฏฅๆๅ–็š„ๅ†…ๅฎน -- ็็ขŽ้—ฒ่Šใ€้—ฎๅ€™๏ผ›ไธดๆ—ถๆ€ง็š„็บฏๅทฅๅ…ทๆ€ง่ฏทๆฑ‚๏ผˆๅฆ‚"่ฟ™ๆฌกๅธฎๆˆ‘็ฟป่ฏ‘ไธ€ไธ‹"๏ผ‰ -- ไธ€ๆฌกๆ€งๆ“ไฝœๆŒ‡ไปค๏ผˆๅฆ‚"่ฟ™ๆฌกใ€ๆœฌๅ•"็›ธๅ…ณ๏ผ‰ -- ้‡ๅค็š„ๅ†…ๅฎน๏ผ›AIๅŠฉๆ‰‹่‡ช่บซ็š„่กŒไธบๆˆ–่พ“ๅ‡บ -- ไธๅฑžไบŽไปฅไธŠ3็ฑป็š„ไฟกๆฏ -- ็บฏไธป่ง‚ๆ„Ÿๅ—๏ผˆไธๅธฆๅฎข่ง‚ไบ‹ไปถ็š„ๆƒ…็ปช่กจ่พพ๏ผ‰ - ---- - -### ไปปๅŠกไธ‰๏ผš่พ“ๅ‡บๆ ผๅผ่ง„่Œƒ๏ผˆJSON๏ผ‰ -่ฟ”ๅ›žไธ”ไป…่ฟ”ๅ›žไธ€ไธชๅˆๆณ•็š„ JSON ๆ•ฐ็ป„ใ€‚ๆ•ฐ็ป„็š„ๆฏไธ€้กนๆ˜ฏไธ€ไธชๆƒ…ๅขƒ๏ผŒๅŒ…ๅซ่ฏฅๆƒ…ๅขƒ็š„ๆถˆๆฏ่Œƒๅ›ดๅ’ŒๆŠฝๅ–ๅˆฐ็š„่ฎฐๅฟ†๏ผš - -[ - { - "scene_name": "ๅฝ“ๅ‰็”Ÿๆˆๆˆ–็ปงๆ‰ฟ็š„ๆƒ…ๅขƒๅ็งฐ", - "message_ids": ["ๅฑžไบŽ่ฏฅๆƒ…ๅขƒ็š„ๆถˆๆฏIDๅˆ—่กจ"], - "memories": [ - { - "content": "ๅฎŒๆ•ดใ€็‹ฌ็ซ‹็š„่ฎฐๅฟ†้™ˆ่ฟฐ๏ผˆๆŒ‰ๅฏนๅบ”็ฑปๅž‹็š„ๅฅๅผ่ฆๆฑ‚๏ผ‰", - "type": "persona|episodic|instruction", - "priority": 80, - "source_message_ids": ["ๆถˆๆฏID_1", "ๆถˆๆฏID_2"], - "metadata": {} - } - ] - } -] - -metadata ๅญ—ๆฎต่ฏดๆ˜Ž๏ผš -- episodic ็ฑปๅž‹๏ผšๅฆ‚่ƒฝ็กฎๅฎšๆดปๅŠจๆ—ถ้—ด๏ผŒๅกซๅ…ฅ {"activity_start_time": "ISO8601", "activity_end_time": "ISO8601"} -- ๅ…ถไป–็ฑปๅž‹ๆˆ–ๆ— ๆณ•็กฎๅฎšๆ—ถ้—ด๏ผš่พ“ๅ‡บ็ฉบๅฏน่ฑก {} - -ๅฆ‚ๆžœๆ•ดๆฎตๅฏน่ฏๆ— ๆœ‰ๆ„ไน‰็š„่ฎฐๅฟ†๏ผŒไนŸ่ฆ่พ“ๅ‡บๆƒ…ๅขƒๅˆ†ๅ‰ฒ็ป“ๆžœ๏ผŒmemories ไธบ็ฉบๆ•ฐ็ป„๏ผš -[ - { - "scene_name": "ๆƒ…ๅขƒๅ็งฐ", - "message_ids": ["id1", "id2"], - "memories": [] - } -] - -่ฏทไธฅๆ ผๆŒ‰ไธŠ่ฟฐ JSON ๆ•ฐ็ป„ๆ ผๅผ่พ“ๅ‡บ๏ผŒไธ่ฆ่พ“ๅ‡บไปปไฝ•้ขๅค–็š„ Markdown ไปฃ็ ๅ—ไฟฎ้ฅฐ็ฌฆ๏ผˆๅฆ‚ \`\`\`json๏ผ‰ๆˆ–่งฃ้‡Šๆ–‡ๆœฌใ€‚`; - -// ============================ -// Prompt Builder -// ============================ - -/** - * Format the user prompt for L1 extraction. - * - * @param newMessages - Messages to extract memories from (with ids and timestamps) - * @param backgroundMessages - Previous messages for context only (not for extraction) - * @param previousSceneName - The last known scene name (for continuity) - */ -export function formatExtractionPrompt(params: { - newMessages: ConversationMessage[]; - backgroundMessages?: ConversationMessage[]; - previousSceneName?: string; -}): string { - const { newMessages, backgroundMessages = [], previousSceneName = "ๆ— " } = params; - - const bgText = backgroundMessages.length > 0 - ? backgroundMessages - .map((m) => `[${m.id}] [${m.role}] [${new Date(m.timestamp).toISOString()}]: ${m.content}`) - .join("\n\n") - : "ๆ— "; - - const newText = newMessages - .map((m) => `[${m.id}] [${m.role}] [${new Date(m.timestamp).toISOString()}]: ${m.content}`) - .join("\n\n"); - - return `ใ€ไธŠไธ€ไธชๆƒ…ๅขƒใ€‘๏ผš${previousSceneName} - -ใ€่ƒŒๆ™ฏๅฏน่ฏใ€‘๏ผˆไป…ไพ›็†่งฃไธŠไธ‹ๆ–‡ๆŽจๆ–ญๅ…ณ็ณป/ๆ—ถ้—ด๏ผŒไธฅ็ฆไปŽไธญๆๅ–่ฎฐๅฟ†๏ผ‰๏ผš -${bgText} - -โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” - -ใ€ๅพ…ๆๅ–็š„ๆ–ฐๆถˆๆฏใ€‘๏ผˆๅŠกๅฟ…็ป“ๅˆ timestamp ๆŽจ็ฎ—ๆ—ถ้—ด๏ผŒๅชไปŽ่ฟ™้‡Œๆๅ–่ฎฐๅฟ†๏ผ๏ผ‰๏ผš -${newText}`; -} diff --git a/src/prompts/persona-generation.ts b/src/prompts/persona-generation.ts deleted file mode 100644 index 4a42b01..0000000 --- a/src/prompts/persona-generation.ts +++ /dev/null @@ -1,172 +0,0 @@ -/** - * Persona Generation Prompt โ€” instructs LLM to generate/update user persona - * using the four-layer deep scan model. - * - * v2: Updated prompt with anti-hallucination guardrails, richer output - * template with per-chapter writing guidance, and streamlined iteration guide. - */ - -export interface PersonaPromptParams { - mode: "first" | "incremental"; - currentTime: string; - totalProcessed: number; - sceneCount: number; - changedSceneCount: number; - changedScenesContent: string; - existingPersona?: string; - triggerInfo?: string; - /** @deprecated Kept for call-site compatibility; no longer used in prompt. */ - personaFilePath: string; - /** @deprecated Kept for call-site compatibility; no longer used in prompt. */ - checkpointPath: string; -} - -export function buildPersonaPrompt(params: PersonaPromptParams): string { - const { - mode, - currentTime, - totalProcessed, - sceneCount, - changedSceneCount, - changedScenesContent, - existingPersona, - triggerInfo, - } = params; - - const modeLabel = mode === "first" ? "๐Ÿ†• ้ฆ–ๆฌก็”Ÿๆˆ" : "๐Ÿ”„ ่ฟญไปฃๆ›ดๆ–ฐ"; - - const triggerSection = triggerInfo - ? `\n### ่งฆๅ‘ไฟกๆฏ\n${triggerInfo}\n` - : ""; - - // Existing persona is now placed AFTER scene blocks (closer to the - // processing instructions) so the LLM sees fresh evidence first. - const existingPersonaSection = existingPersona - ? `\n## ๐Ÿ“„ ๅฝ“ๅ‰ Persona๏ผˆๅทฅ็จ‹ๅทฒ้ข„ๅŠ ่ฝฝ๏ผ‰\n\n` + - `*ไปฅไธ‹ๆ˜ฏ็Žฐๆœ‰ persona.md ็š„ๅฎŒๆ•ดๅ†…ๅฎน๏ผˆ${existingPersona.length} ๅญ—็ฌฆ๏ผ‰๏ผŒๅŸบไบŽๆญคๆ›ดๆ–ฐๅŽ่ฏทๆŽงๅˆถๅœจ2000ๅญ—ๅ†…๏ผš*\n\n` + - `\`\`\`markdown\n${existingPersona}\n\`\`\`\n\n---\n` - : ""; - - const iterationGuide = mode === "incremental" - ? `\n## ๐Ÿ”„ ่ฟญไปฃๅ†ณ็ญ–ๆŒ‡ๅ—\n\n` + - `้ขๅฏนๅ˜ๅŒ–ๅœบๆ™ฏ๏ผŒ่‡ชไธปๅˆคๆ–ญๅค„็†ๆ–นๅผ๏ผšๅผบๅŒ–๏ผˆไฝ่ฏๅทฒๆœ‰ๆดžๅฏŸ๏ผ‰/ ่กฅๅ……๏ผˆๆ–ฐ็ปดๅบฆ๏ผ‰/ ไฟฎๆญฃ๏ผˆ็Ÿ›็›พ๏ผ‰/ ้‡ๆž„๏ผˆ็ป“ๆž„่ฐƒๆ•ด๏ผ‰/ ไธๆ”น๏ผˆๆ— ๆœ‰็”จๆ–ฐๅขžๅ†…ๅฎน๏ผ‰ใ€‚\n` - : ""; - - return `# ๐Ÿงฌ Persona Architect - Incremental Evolution Protocol - -**โฐ ๆ›ดๆ–ฐๆ—ถ้—ด**: ${currentTime} -**ๆจกๅผ**: ${modeLabel} - -่ฏทไฝ ็ป“ๅˆๅทฒๆœ‰็š„ persona.md ๅ’Œๆ–ฐๅขž/ๅ˜ๅŒ–็š„ block ไฟกๆฏๆทฑๅบฆๅˆ†ๆž๏ผŒ็„ถๅŽไฝฟ็”จๆ–‡ไปถๅทฅๅ…ทๅฐ†็ป“ๆžœๅ†™ๅ…ฅ \`persona.md\` ๆ–‡ไปถใ€‚ - -## โ›” ๆ–‡ไปถๆ“ไฝœ็บฆๆŸ๏ผˆๅฟ…้กปไธฅๆ ผ้ตๅฎˆ๏ผ‰ - -1. **ๅฟ…้กปไฝฟ็”จๆ–‡ไปถๅทฅๅ…ทๅฐ†ๆœ€็ปˆ persona ๅ†…ๅฎนๅ†™ๅ…ฅ \`persona.md\`**ใ€‚ๅฝ“ๅ‰ๅทฅไฝœ็›ฎๅฝ•ๅทฒ่ฎพไธบๆ•ฐๆฎ็›ฎๅฝ•๏ผŒ็›ดๆŽฅไฝฟ็”จๆ–‡ไปถๅ \`persona.md\`ใ€‚ - - **้ฆ–ๆฌก็”Ÿๆˆ / ๅคงๅน…้‡ๅ†™**๏ผšไฝฟ็”จ \`write_to_file('persona.md', ๅฎŒๆ•ดๅ†…ๅฎน)\` ๆ•ดไฝ“ๅ†™ๅ…ฅ - - **ๅขž้‡ๆ›ดๆ–ฐ๏ผˆๅฑ€้ƒจไฟฎๆ”น๏ผ‰**๏ผšไฝฟ็”จ \`replace_in_file('persona.md', ๆ—งๅ†…ๅฎน็‰‡ๆฎต, ๆ–ฐๅ†…ๅฎน็‰‡ๆฎต)\` ็ฒพ็กฎๆ›ฟๆข๏ผŒๅฏ่Š‚็œๅผ€้”€ -2. **ๅช่ƒฝๆ“ไฝœ \`persona.md\` ่ฟ™ไธ€ไธชๆ–‡ไปถ**๏ผŒ็ฆๆญข่ฏปๅ–ๆˆ–ๅ†™ๅ…ฅไปปไฝ•ๅ…ถไป–ๆ–‡ไปถ๏ผˆๅŒ…ๆ‹ฌ scene_blocks/ใ€.metadata/ ็ญ‰๏ผ‰ใ€‚ -3. **ๅ†™ๅ…ฅ็š„ๅ†…ๅฎนๅฟ…้กปๅชๅŒ…ๅซๆœ€็ปˆ็š„ persona ๆ–‡ๆกฃ**๏ผŒไธ่ฆๅŒ…ๅซไฝ ็š„ๆ€่€ƒ่ฟ‡็จ‹ใ€ๅˆ†ๆžๆญฅ้ชคๆˆ–ไปปไฝ•้ž persona ๅ†…ๅฎนใ€‚ -4. **ๆ— ้œ€ read_file**๏ผšๅฝ“ๅ‰ persona.md ็š„ๅฎŒๆ•ดๅ†…ๅฎนๅทฒๅœจไธ‹ๆ–นๆไพ›๏ผŒ็›ดๆŽฅๅŸบไบŽๅฎƒ่ฟ›่กŒๆ›ดๆ–ฐๅณๅฏใ€‚ - -### ๐Ÿšซ ไธฅๆ ผ็ฆๆญข -- **็ฆๆญข่ฟ‡้•ฟ**๏ผšpersona.md ๅ†…ๅฎนๆ€ป้•ฟๅบฆไธ่ฆ่ถ…่ฟ‡ 2000 ๅญ—็ฌฆ๏ผŒๅŠๆ—ถๅšๆ€ป็ป“ๅ’Œๅˆ ้™คไธ้‡่ฆ็š„ไฟกๆฏใ€‚ -- **็ฆๆญข่ฟ‡ๅบฆๆŽจๆต‹**๏ผšๆฒกๆๅˆฐ็š„ไฟกๆฏไธ่ฆ่ฟ‡ๅบฆ่‡†ๆƒณๅฏผ่‡ดไบง็”Ÿๅนป่ง‰๏ผŒ็‰นๅˆซๆ˜ฏๅœจๅ†ทๅฏๅŠจ้˜ถๆฎต๏ผŒ่ฆไฟๆŒๅ…‹ๅˆถ๏ผŒๅฆ‚ๆžœๆฒกๆœ‰็›ธๅ…ณไฟกๆฏๅฎŒๅ…จๅฏไปฅไธๅกซ๏ผ -- **็ฆๆญขไฝฟ็”จ้žๅœบๆ™ฏๆฅๆบ็š„ไฟกๆฏ**๏ผšPersona ็š„ๆ‰€ๆœ‰ๅ†…ๅฎนๅฟ…้กปไธ”ๅช่ƒฝๆฅ่‡ชไธ‹ๆ–นๆไพ›็š„ๅœบๆ™ฏๆ•ฐๆฎใ€‚ไธ่ฆไปŽ workspace ็›ฎๅฝ•็ป“ๆž„ใ€ๆ–‡ไปถ่ทฏๅพ„ใ€็ณป็ปŸไฟกๆฏ็ญ‰ๆŠ€ๆœฏๅ…ƒๆ•ฐๆฎไธญๆๅ–ไปปไฝ•ๅ…ณไบŽ็”จๆˆท็š„ไธชไบบไฟกๆฏใ€‚ -- **็ฆๆญขๆ“ไฝœ persona.md ไปฅๅค–็š„ไปปไฝ•ๆ–‡ไปถ**ใ€‚ - ---- -${triggerSection} -## ๐Ÿ“Š ็ปŸ่ฎก -- **ๆ€ป่ฎฐๅฟ†ๆ•ฐ**: ${totalProcessed} ๆก -- **ๅœบๆ™ฏๆ€ปๆ•ฐ**: ${sceneCount} ไธช -- **ๅ˜ๅŒ–ๅœบๆ™ฏ**: ${changedSceneCount} ไธช๏ผˆ่‡ชไธŠๆฌกๆ›ดๆ–ฐๅŽ๏ผ‰ - ---- -${changedScenesContent} - - -${existingPersonaSection} -## โš™๏ธ ๆ ธๅฟƒ่ฟไฝœ้€ป่พ‘ (The Core Logic) - -๐Ÿง  ๆ ธๅฟƒๆ€็ปดๅผ•ๆ“Ž๏ผš่ฟžๆŽฅไธŽ็ปผๅˆ (Connect & Synthesize) -่ฏท้ตๅพช "ๅ™ไบ‹่ฟž่ดฏๆ€ง" ๅŽŸๅˆ™ๅค„็†ไฟกๆฏใ€‚็ฆๆญข็ฎ€ๅ•็š„็ฝ—ๅˆ—๏ผˆNo Bullet-point Spamming๏ผ‰ใ€‚ - -1. ๅฏปๆ‰พ"่ดฏ็ฉฟ็บฟ" (The Connecting Thread) -ไธ่ฆๅญค็ซ‹ๅœฐ็œ‹ไฟกๆฏใ€‚่ฆๅฏปๆ‰พไธๅŒ้ข†ๅŸŸ่กŒไธบ่ƒŒๅŽ็š„ๅ…ฑๅŒ้€ป่พ‘ใ€‚ -** ่ฆไฟๆŒ็ฒพ็ฎ€๏ผŒไธ่ฟ‡ๅบฆ็Œœๆƒณ๏ผŒๅฆ‚ๆžœไธ็กฎๅฎšๅฏไปฅไธๅ†™ ** - -ๆ‰ง่กŒไปฅไธ‹**ๅ››ๅฑ‚ๆทฑๅบฆๆ‰ซๆ**๏ผš - -### ๐ŸŸข Layer 1: ๅŸบ็ก€้”š็‚น (The Base & Facts) -> ใ€ๅปบ็ซ‹่ฟžๆŽฅใ€‘ -* **ๆ‰ซๆ็›ฎๆ ‡**: ็กฎๅ‡ฟ็š„ไบ‹ๅฎžใ€ไบบๅฃ็ปŸ่ฎกๅญฆ็‰นๅพใ€ๅฝ“ๅ‰็Šถๆ€ใ€‚ -* **ๅฎž็”จไปทๅ€ผ**: ไธบ Agent ๆไพ›**็ ดๅ†ฐ่ฏ้ข˜**ๅ’Œ**ไธŠไธ‹ๆ–‡ๆ„Ÿ็Ÿฅ**ใ€‚ - -### ๐Ÿ”ต Layer 2: ๅ…ด่ถฃๅ›พ่ฐฑ (The Interest Graph) -> ใ€ๆไพ›่ฐˆ่ต„ใ€‘ -* **ๆ‰ซๆ็›ฎๆ ‡**: ็”จๆˆทๆŠ•ๅ…ฅๆ—ถ้—ดใ€้‡‘้’ฑๆˆ–ๆณจๆ„ๅŠ›็š„ไบ‹็‰ฉใ€‚ -* **ๆๅ–ๅŽŸๅˆ™**: **ๅŒบๅˆ†ๆดป่ทƒๅบฆ**๏ผˆๆดป่ทƒ็ˆฑๅฅฝ / ่ขซๅŠจๆถˆ่ดน / ไผ‘็œ ๅ…ด่ถฃ๏ผ‰ใ€‚ -* **ๅฎž็”จไปทๅ€ผ**: ่ฎฉ Agent ่ƒฝๅคŸ่ฟ›่กŒ**้ซ˜่ดจ้‡็š„้—ฒ่Š (Chit-chat)** ๅ’Œ **็”ŸๆดปๆŽจ่**ใ€‚ - -### ๐ŸŸก Layer 3: ไบคไบ’ๅ่ฎฎ (The Interface) -> ใ€ๆถˆ้™คๆ‘ฉๆ“ฆใ€‘ -* **ๆ‰ซๆ็›ฎๆ ‡**: ็”จๆˆท็š„ๆฒŸ้€šไน ๆƒฏใ€้›ทๅŒบใ€ๅทฅไฝœๆตๅๅฅฝใ€‚ -* **ๅฎž็”จไปทๅ€ผ**: ๆŒ‡ๅฏผ Agent **ๅฆ‚ไฝ•่ฏด่ฏใ€ๅฆ‚ไฝ•ไบคไป˜็ป“ๆžœ**๏ผŒ้ฟๅ…่ธฉ้›ทใ€‚ - -### ๐Ÿ”ด Layer 4: ่ฎค็Ÿฅๅ†…ๆ ธ (The Core) -> ใ€ๆทฑๅบฆๅ…ฑ้ธฃใ€‘ -* **ๆ‰ซๆ็›ฎๆ ‡**: ๅ†ณ็ญ–้€ป่พ‘ใ€็Ÿ›็›พ็‚นใ€็ปˆๆž้ฉฑๅŠจๅŠ›ใ€‚ -* **ๅฎž็”จไปทๅ€ผ**: ่ฎฉ Agent ๆˆไธบ**่ƒฝๅคŸๆ›ฟ็”จๆˆทๅšๅ†ณ็ญ–**็š„"ๅ‰ฏ้ฉพ้ฉถ"ใ€‚ -${iterationGuide} ---- - -## ๐Ÿ“ ่พ“ๅ‡บๆจกๆฟ (The Persona Template) - -่ฏทๅ‚่€ƒไปฅไธ‹ๆ ผๅผ๏ผŒไฝฟ็”จ \`write_to_file('persona.md', ...)\` ๅ†™ๅ…ฅๆœ€็ปˆๅ†…ๅฎนใ€‚ๅฏไปฅๅš่‡ชไธป่ฐƒๆ•ด๏ผˆไฟกๆฏไธ่ถณๆ—ถๅฏไปฅๅ‡ๅฐ‘ๆˆ–ๆ–ฐๅขž chapter๏ผ‰๏ผˆ**ๅฟ…้กปไฟๆŒ Markdown ๆ ผๅผ**๏ผ‰๏ผš - -\`\`\`\`markdown -# User Narrative Profile - -> **Archetype (ๆ ธๅฟƒๅŽŸๅž‹)**: [ไธ€ๅฅ่ฏๅฎšไน‰ใ€‚ไพ‹ๅฆ‚๏ผšไธ€ไฝๅœจ็Žฐๅฎž้‡ๅŠ›ไธ‹ๆŒฃๆ‰Ž๏ผŒไฝ†่ฏ•ๅ›พ้€š่ฟ‡ๆŠ€ๆœฏๆž„ๅปบ็†ๆƒณๅ›ฝ็š„"ๅŠกๅฎž็†ๆƒณไธปไน‰่€…"ใ€‚] - -> **ๅŸบๆœฌไฟกๆฏ** -๏ผˆ็”จๆˆท็š„ๅŸบๆœฌไฟกๆฏ๏ผŒๅฆ‚ๅนด้พ„ใ€ๆ€งๅˆซใ€่Œไธš็ญ‰๏ผŒๆ›ดๆ–ฐๆ—ถ่‹ฅๆœ‰ๅ†ฒ็ชๅˆ™่ฆ†็›–๏ผŒไธๅ†ฒ็ชๅฐฝ้‡ๅ ๅŠ ๏ผ‰ - - - - - -> **้•ฟๆœŸๅๅฅฝ** -๏ผˆไฝ ่ง‚ๅฏŸๅˆฐ็š„็”จๆˆทๆœ€็จณๅฎšไธ”ๅฏๅค็”จ็š„ๅๅฅฝ๏ผ‰ - - - - - -## ๐Ÿ“– Chapter 1: Context & Current State (ๅ…จๆ™ฏ่ฏญๅขƒ) -*(ๅฐ†ๅŸบ็ก€ไบ‹ๅฎžไธŽๅฝ“ๅ‰็Šถๆ€่žๅˆ๏ผŒๅ†™ๆˆไธ€ๆฎต่ฟž่ดฏ็š„่ƒŒๆ™ฏไป‹็ป)* - -**[่ฟ™้‡Œๅ†™่ฟž่ดฏๆ่ฟฐ๏ผŒๅŒบๅˆซ่พƒๅคง็š„ๆ—ถๅ€™ๅฏไปฅๅˆ†็‚น้˜่ฟฐ]** - -## ๐ŸŽจ Chapter 2: The Texture of Life (็”Ÿๆดป็š„่‚Œ็†) -*(ๅฐ†ๅ…ด่ถฃใ€ๆถˆ่ดนใ€็”Ÿๆดปไน ๆƒฏไธฒ่”่ตทๆฅ๏ผŒๅฑ•็คบ็”Ÿๆดปๅ“ๅ‘ณ)* - -**[่ฟ™้‡Œๅ†™่ฟž่ดฏ็š„ๆ่ฟฐ๏ผŒ้‡็‚นๅœจไบŽ"ๅ…ด่ถฃ/ๅๅฅฝ"ๅ’Œ"ๅ“ๅ‘ณ"็š„็ปŸไธ€ๆ€ง๏ผŒๅŒบๅˆซ่พƒๅคง็š„ๆ—ถๅ€™ๅฏไปฅๅˆ†็‚น้˜่ฟฐ]** - -## ๐Ÿค– Chapter 3: Interaction & Cognitive Protocol (ไบคไบ’ไธŽ่ฎค็Ÿฅๅ่ฎฎ) -*(่ฟ™ๆ˜ฏ Main Agent ็š„่กŒๅŠจๆŒ‡ๅ—ใ€‚ไธบไบ†ๅฎž็”จ๏ผŒ่ฟ™้‡ŒไฟๆŒๅŠ็ป“ๆž„ๅŒ–๏ผŒไฝ†่ฆ่งฃ้‡Š"ไธบไป€ไนˆ")* - -### 3.1 ๆฒŸ้€š็ญ–็•ฅ (How to Speak) -### 3.2 ๅ†ณ็ญ–้€ป่พ‘ (How to Think) - -## ๐Ÿงฉ Chapter 4: Deep Insights & Evolution (ๆทฑๅฑ‚ๆดžๅฏŸไธŽๆผ”ๅ˜) -*(ไบบ็ฑปๅญฆ่ง‚ๅฏŸ็ฌ”่ฎฐ)* - -* **็Ÿ›็›พ็ปŸไธ€ๆ€ง**: [ๆ่ฟฐ็”จๆˆท่บซไธŠ็œ‹ไผผๅ†ฒ็ชไฝ†ๅฎžๅˆ™ๅˆ็†็š„็‰น่ดจ]ใ€‚ -* **ๆผ”ๅ˜่ฝจ่ฟน**: [ๅฏๅŠ ไธŠๆ—ถ้—ด๏ผŒๅˆ†ไธบๅคš็‚น๏ผŒๆ่ฟฐ็”จๆˆทๆœ€่ฟ‘ๅ‘็”Ÿ็š„ๅ˜ๅŒ–]ใ€‚ -* **ๆถŒ็Žฐ็‰นๅพ**: ๆ็‚ผ 3-7 ไธชๆœ€ๆ ธๅฟƒ็š„็‰น่ดจๆ ‡็ญพ๏ผŒๆฏไธชๆ ‡็ญพๅ•็‹ฌไธ€่กŒๅนถ้™„ไธŠ็ฎ€็Ÿญๆณจ้‡Š๏ผˆ10-15ๅญ—๏ผ‰ - - \`TagName\` - ็ฎ€็Ÿญๆณจ้‡Š่ฏดๆ˜Ž -\`\`\`\` - ---- - -### โš ๏ธ ๆˆๅŠŸๆ ‡ๅ‡† -- โœ… **ๅฟ…้กปไฝฟ็”จ \`write_to_file\` ๆˆ– \`replace_in_file\` ๅทฅๅ…ทๅ†™ๅ…ฅๆœ€็ปˆ็ป“ๆžœๅˆฐ \`persona.md\`** -- โœ… ๅŸบไบŽๅœบๆ™ฏ่ฏๆฎ็”ŸๆˆๆทฑๅบฆๆดžๅฏŸ -- โœ… ๅ†…ๅฎนๅˆฐ Chapter 4 ็ป“ๆŸ๏ผˆไธๅŒ…ๅซๅœบๆ™ฏๅฏผ่ˆช๏ผŒๅทฅ็จ‹ไผš่‡ชๅŠจ่ฟฝๅŠ ๏ผ‰ -- โœ… ๅฟ…้กปไธฅๆ ผๆŒ‰็…งไธŠ้ข็š„ๆจกๆฟๆ ผๅผ -- โœ… ไธ่ฆๆทปๅŠ ๅœบๆ™ฏๅฏผ่ˆช๏ผˆๅทฅ็จ‹ไผš่‡ชๅŠจ่ฟฝๅŠ ๏ผ‰ -- โœ… ๅชๆ“ไฝœ persona.md๏ผŒไธ่ฆๆ“ไฝœๅ…ถไป–ๆ–‡ไปถ`; -} diff --git a/src/prompts/scene-extraction.ts b/src/prompts/scene-extraction.ts deleted file mode 100644 index 8648943..0000000 --- a/src/prompts/scene-extraction.ts +++ /dev/null @@ -1,240 +0,0 @@ -/** - * Scene Extraction Prompt โ€” instructs LLM to consolidate memories into scene blocks - * using file tools (read_file, write_to_file, replace_in_file). - * - * Scene files can be updated via: - * - read_file + write_to_file (full rewrite) for large structural changes - * - replace_in_file for targeted partial updates (e.g. updating a single section) - * - * Security: The LLM is sandboxed to scene_blocks/ only (workspaceDir = scene_blocks/). - * It has NO visibility into checkpoint, scene_index, persona.md, or any other system file. - * File deletion is achieved via "soft-delete" โ€” writing the marker `[DELETED]` to the file - * โ€” and the SceneExtractor subsequently removes soft-deleted files with fs.unlink. - * Note: writing an empty/whitespace-only string is rejected by the core write tool's - * parameter validation, so we use a non-empty marker instead. - * - * Persona update requests are communicated via text output signals (out-of-band), - * parsed by the engineering side after LLM execution completes. - */ - -export interface SceneExtractionPromptParams { - memoriesJson: string; - sceneSummaries: string; - currentTimestamp: string; - sceneCountWarning?: string; - /** List of existing scene filenames (relative, e.g. ["work.md", "hobby.md"]) */ - existingSceneFiles?: string[]; - /** Maximum number of scene blocks allowed */ - maxScenes: number; -} - -export function buildSceneExtractionPrompt(params: SceneExtractionPromptParams): string { - const { - memoriesJson, - sceneSummaries, - currentTimestamp, - sceneCountWarning, - existingSceneFiles, - maxScenes, - } = params; - - const warningSection = sceneCountWarning - ? `\nโš ๏ธ **ๅœบๆ™ฏๆ•ฐ้‡่ญฆๅ‘Š**: ${sceneCountWarning}\n` - : ""; - - return `# System Prompt: Memory Consolidation Architect - -## ่ง’่‰ฒๅฎšไน‰ (Role Definition) -ไฝ ๆ˜ฏ่ฎฐๅฟ†ๆ•ดๅˆๆžถๆž„ๅธˆใ€‚ไฝ ็š„็›ฎๆ ‡ๆ˜ฏไธบ็”จๆˆทๆž„ๅปบไธ€ไธช"ๆ•ฐๅญ—็ฌฌไบŒๅคง่„‘"ใ€‚ไฝ ไธไป…ไป…ๆ˜ฏๅœจ่ฎฐๅฝ•ๆ•ฐๆฎ๏ผŒไฝ ๆ›ดๅƒๆ˜ฏไธ€ไฝไบบ็ฑปๅญฆๅฎถๅ’Œๅฟƒ็†ๅญฆๅฎถ๏ผŒ่ดŸ่ดฃๅˆ†ๆžๅŽŸๅง‹่ฎฐๅฟ†๏ผŒไปŽไธญๆๅ–ๆ ธๅฟƒ็‰นๅพใ€ๆ•ๆ‰้šๆ€งไฟกๅท๏ผŒๅนถๆž„ๅปบไธๆ–ญๆผ”ๅ˜็š„ๅ™ไบ‹ใ€‚ - - -## ๆžถๆž„ๆจกๅž‹ - -### Layer 1 (Input): Raw Memories -- **ๆฅๆบ**๏ผšAPI ๅˆ†ๆ‰นๅฌๅ›ž๏ผˆๆฏๆ‰น 20 ๆก๏ผ‰ -- **็Šถๆ€**๏ผš็ขŽ็‰‡ๅŒ–ใ€ๆ— ๅบ - -### Layer 2 (Processing): Scene Diaries -- **ๅฝขๆ€**๏ผš**ไธๆ˜ฏๆธ…ๅ•๏ผŒๆ˜ฏ่ฟž่ดฏ็š„ๅ™ไบ‹ๆ–‡ๆกฃ** -- **้€ป่พ‘**๏ผšๅฐ† L1 ็ขŽ็‰‡่žๅˆ่ฟ›็‰นๅฎšๅœบๆ™ฏๆ–‡ไปถ -- **ๅŠจไฝœ**๏ผšCreate๏ผˆๅˆ›ๅปบ๏ผ‰ใ€Integrate๏ผˆๆ•ดๅˆ๏ผ‰ใ€Rewrite๏ผˆ้‡ๅ†™๏ผ‰ -- **็ฆๆญข**๏ผš็ฎ€ๅ•่ฟฝๅŠ ๅˆ—่กจ - -ไฝ ไธป่ฆ่ดŸ่ดฃL1ๅˆฐL2็š„็”ŸๆˆไปปๅŠก -${warningSection} -## ่พ“ๅ…ฅ็Žฏๅขƒ (Input Context) -ไฝ ๅฐ†ๆŽฅๆ”ถไธ‰ไธช่พ“ๅ…ฅ๏ผš -1. ๆ–ฐๅขž่ฎฐๅฟ† (New Memory): ไธ€ๆฎตๅŽŸๅง‹็š„ใ€้ž็ป“ๆž„ๅŒ–็š„ๆ–ฐ่ฟ‘ๅ›žๅฟ†ไฟกๆฏใ€‚ -2. ็Žฐๆœ‰ Block ๆ˜ ๅฐ„่กจ (Existing Blocks Map): ๅŒ…ๅซๅฝ“ๅ‰ๆ‰€ๆœ‰่ฎฐๅฟ†ๅ—๏ผˆMarkdown ๆ–‡ไปถ๏ผ‰็š„ๆ–‡ไปถๅๅ’Œๆ‘˜่ฆ็š„ๅˆ—่กจใ€‚ -3. ๅฝ“ๅ‰ๆ—ถ้—ด (Current Time): ็”จไบŽ็”Ÿๆˆๅ…ƒๆ•ฐๆฎ็š„ๅ…ทไฝ“ๆ—ถ้—ดๆˆณใ€‚ - -**โš ๏ธ ๅœบๆ™ฏๆ–‡ไปถๆ•ฐ้‡ไธŠ้™๏ผš${maxScenes} ไธชใ€‚ๅค„็†ๅฎŒๆˆๅŽ็›ฎๅฝ•ไธญ็š„ๅœบๆ™ฏๆ–‡ไปถๆ•ฐ้‡ๅฟ…้กปไธฅๆ ผๅฐไบŽๆญคไธŠ้™ใ€‚** - - -### 1๏ธโƒฃ New Memories List -${memoriesJson} - -### 2๏ธโƒฃ Existing Scene Blocks Summary -${sceneSummaries} - -### 3๏ธโƒฃ Current Timestamp -${currentTimestamp} - -${existingSceneFiles && existingSceneFiles.length > 0 - ? `### ๐Ÿ“ ๅทฒๆœ‰ๅœบๆ™ฏๆ–‡ไปถๆธ…ๅ•๏ผˆไป…ไปฅไธ‹ๆ–‡ไปถๅฏ read_file๏ผ‰ -${existingSceneFiles.map((f) => `- \`${f}\``).join("\n")} -` - : `### ๐Ÿ“ ๅทฒๆœ‰ๅœบๆ™ฏๆ–‡ไปถๆธ…ๅ• -๏ผˆๅฝ“ๅ‰ๆ— ๅทฒๆœ‰ๅœบๆ™ฏๆ–‡ไปถ๏ผ‰ -`} -## โ›” ๆ–‡ไปถๆ“ไฝœ็บฆๆŸ๏ผˆๅฟ…้กปไธฅๆ ผ้ตๅฎˆ๏ผ‰ -1. **ๆ‰€ๆœ‰ๆ–‡ไปถๆ“ไฝœไฝฟ็”จ็›ธๅฏนๆ–‡ไปถๅ**๏ผˆๅฆ‚ \`ๆŠ€ๆœฏ็ ”็ฉถ-Rustๅญฆไน .md\`๏ผ‰๏ผŒๅฝ“ๅ‰ๅทฅไฝœ็›ฎๅฝ•ๅทฒ่ฎพไธบๅœบๆ™ฏๆ–‡ไปถ็›ฎๅฝ• -2. **read_file ๅช่ƒฝ่ฏปๅ–ไธŠ้ข"ๅทฒๆœ‰ๅœบๆ™ฏๆ–‡ไปถๆธ…ๅ•"ไธญๅˆ—ๅ‡บ็š„ๆ–‡ไปถ**๏ผŒ็ฆๆญข็Œœๆต‹ๆˆ–็ผ–้€ ไธๅœจๆธ…ๅ•ไธญ็š„ๆ–‡ไปถๅ -3. **ๅˆ›ๅปบๆ–ฐๅœบๆ™ฏๆ–‡ไปถๆ—ถ**๏ผŒ็›ดๆŽฅไฝฟ็”จๆ–‡ไปถๅ๏ผŒๅฆ‚ \`ๆ–ฐๅœบๆ™ฏๅ.md\` -4. **ๅœบๆ™ฏๆ–‡ไปถๆ”ฏๆŒ replace_in_file**ใ€‚ๅฏนไบŽๅฑ€้ƒจๆ›ดๆ–ฐ๏ผˆๅฆ‚ๅชๆ›ดๆ–ฐๆŸไธช็ซ ่Š‚ๆˆ– META ๅญ—ๆฎต๏ผ‰๏ผŒๅฏไปฅไฝฟ็”จ \`replace_in_file\` ่ฟ›่กŒ็ฒพ็กฎๆ›ฟๆขใ€‚ๅฏนไบŽๅคง่Œƒๅ›ด้‡ๅ†™ๆˆ–็ป“ๆž„ๆ€งๅ˜ๆ›ด๏ผŒๅปบ่ฎฎไฝฟ็”จ \`read_file\` + \`write_to_file\` ๆ•ดไฝ“้‡ๅ†™ใ€‚ -5. **ๅœบๆ™ฏ็ดขๅผ•ๅ’Œ็ณป็ปŸ้…็ฝฎ็”ฑๅทฅ็จ‹็ณป็ปŸ่‡ชๅŠจ็ปดๆŠค**๏ผŒไฝ ๅช้œ€ไธ“ๆณจไบŽๆ“ไฝœ \`.md\` ๅœบๆ™ฏๆ–‡ไปถ -6. **ๅˆ ้™คๆ–‡ไปถ็š„ๅ”ฏไธ€ๆ–นๅผ**๏ผšไฝฟ็”จ \`write_to_file(filename, '[DELETED]')\` ๅฐ†ๆ–‡ไปถๅ†…ๅฎนๅ†™ไธบ \`[DELETED]\` ๆ ‡่ฎฐใ€‚็ณป็ปŸไผš่‡ชๅŠจๆธ…็†ๅธฆๆœ‰ๆญคๆ ‡่ฎฐ็š„ๆ–‡ไปถใ€‚**็ฆๆญข**ๅ†™ๅ…ฅ็ฉบๅญ—็ฌฆไธฒ๏ผˆไผš่ขซ็ณป็ปŸๆ‹’็ป๏ผ‰ใ€‚**็ฆๆญข**็”จ \`[ARCHIVE]\`ใ€\`[CONSOLIDATED]\` ็ญ‰ๅ…ถไป–ๆ ‡่ฎฐๆ›ฟไปฃๅˆ ้™คโ€”โ€”ๅชๆœ‰ \`[DELETED]\` ๆ ‡่ฎฐไผš่งฆๅ‘็ณป็ปŸๆธ…็†ใ€‚ -7. **็ฆๆญขๅˆ›ๅปบๆŠฅๅ‘Š/ๆ•ดๅˆ/ๆฑ‡ๆ€ป็ฑปๆ–‡ไปถ**ใ€‚ไฝ ็š„่พ“ๅ‡บๅฟ…้กปๆ˜ฏๆœ‰ๆ„ไน‰็š„ๅœบๆ™ฏๅ™ไบ‹ๆ–‡ไปถ๏ผˆๅฆ‚"ๆŠ€ๆœฏๆžถๆž„ไธŽๅทฅ็จ‹ๅฎž่ทต.md"ใ€"ๆ—ฅๅธธ็”ŸๆดปไธŽๅทฅไฝœ่Š‚ๅฅ.md"๏ผ‰ใ€‚็ฆๆญขๅˆ›ๅปบไปฅ BATCHใ€REPORTใ€CONSOLIDATIONใ€INTEGRATIONใ€ARCHIVEใ€SUMMARY ็ญ‰ไธบๅ‰็ผ€็š„ๆ–‡ไปถใ€‚ - -## ๅทฅไฝœๆตไธŽ้€ป่พ‘ (Workflow & Logic) -ๅœจ็”Ÿๆˆ่พ“ๅ‡บไน‹ๅ‰๏ผŒไฝ ๅฟ…้กปๆ‰ง่กŒไปฅไธ‹"ๆ€็ปด้“พ"่ฟ‡็จ‹๏ผš - -### โš ๏ธ ้˜ถๆฎต 0๏ผšๅผบๅˆถๆฃ€ๆŸฅๅœบๆ™ฏๆ€ปๆ•ฐ๏ผˆๅฟ…้กปๅ…ˆๆ‰ง่กŒ๏ผ‰ - -**ๅœจๅค„็†ไปปไฝ•่ฎฐๅฟ†ไน‹ๅ‰๏ผŒไฝ ๅฟ…้กป๏ผš** - -1. **็ปŸ่ฎกๅฝ“ๅ‰ๅœบๆ™ฏๆ€ปๆ•ฐ**๏ผšๆŸฅ็œ‹ "Existing Scene Blocks Summary" ้กถ้ƒจๆ ‡ๆณจ็š„ๅฝ“ๅ‰ๅœบๆ™ฏๆ€ปๆ•ฐ -2. **ๆœ€็ปˆ็›ฎๆ ‡**๏ผšๅค„็†ๅฎŒๆˆๅŽ๏ผŒ็›ฎๅฝ•ไธญ็š„ๅœบๆ™ฏๆ–‡ไปถๆ•ฐ้‡ๅฟ…้กป **ไธฅๆ ผๅฐไบŽ ${maxScenes}** -3. **้ตๅฎˆๅˆ†็บง้ข„่ญฆ**๏ผš - - ็บข่‰ฒ้ข„่ญฆ๏ผˆโ‰ฅ ${maxScenes}๏ผ‰๏ผš**ๅฟ…้กปๅ…ˆ้€š่ฟ‡ MERGE ๅ‡ๅฐ‘ๆ–‡ไปถๆ•ฐ้‡**๏ผŒๅฐ†ๆœ€็›ธไผผ็š„ 2-4 ไธชๅœบๆ™ฏๅˆๅนถไธบ 1 ไธช๏ผŒ**ๅนถๅˆ ้™ค่ขซๅˆๅนถ็š„ๆ—งๆ–‡ไปถ**๏ผŒ็›ดๅˆฐๆ–‡ไปถๆ•ฐ < ${maxScenes} ๅŽ๏ผŒๅ†ๅค„็†ๆ–ฐ่ฎฐๅฟ† - - ๆฉ™่‰ฒ้ข„่ญฆ๏ผˆ= ${maxScenes - 1}๏ผ‰๏ผš**ๅช่ƒฝ UPDATE ็Žฐๆœ‰ๅœบๆ™ฏ๏ผŒไธ่ƒฝ CREATE ๆ–ฐๅœบๆ™ฏ** - - ้ป„่‰ฒ้ข„่ญฆ๏ผˆๆŽฅ่ฟ‘ ${maxScenes}๏ผ‰๏ผš**ไผ˜ๅ…ˆ UPDATE ๆˆ–ไธปๅŠจ MERGE ็›ธไผผๅœบๆ™ฏ** - -**ๅˆๅนถไผ˜ๅ…ˆ็บง**๏ผˆๅฝ“้œ€่ฆๅˆๅนถๆ—ถ๏ผŒๆŒ‰ไปฅไธ‹้กบๅบ้€‰ๆ‹ฉ๏ผ‰๏ผš -1. **ไธป้ข˜้ซ˜ๅบฆ้‡ๅ **๏ผšๅฆ‚"PythonๅŽ็ซฏๅผ€ๅ‘"ๅ’Œ"GoๅŽ็ซฏๅผ€ๅ‘" โ†’ ๅˆๅนถไธบ"ๅŽ็ซฏๅผ€ๅ‘ๆŠ€ๆœฏๆ ˆ" -2. **ๅ™ไบ‹ๅผง็บฟ็›ธๅŒ**๏ผšๅฆ‚"ๆฑ‚่Œๆๆ–™-JDๅŒน้…"ๅ’Œ"่Œไธšๅ‘ๅฑ•-่ƒฝๅŠ›ๅฏน้ฝ" โ†’ ๅˆๅนถไธบ"่Œไธšๅ‘ๅฑ•ไธŽๆฑ‚่Œ" -3. **็ƒญๅบฆๆœ€ไฝŽ็š„ๅœบๆ™ฏ**๏ผšๅฆ‚ๆžœๆฒกๆœ‰ๆ˜Žๆ˜พ้‡ๅ ๏ผŒๅˆๅนถๆˆ–ๅˆ ้™ค heat ๆœ€ไฝŽ็š„ 2-3 ไธชๅœบๆ™ฏ - -### ้˜ถๆฎต 1๏ผšๅˆ†ๆžไธŽๅˆ†็ฑป -ๅˆ†ๆž ๆ–ฐๅขž่ฎฐๅฟ†ใ€‚ๅฎƒ็š„ๆ ธๅฟƒ้ข†ๅŸŸๆ˜ฏไป€ไนˆ๏ผŸ๏ผˆไพ‹ๅฆ‚๏ผš็ผ–็จ‹้ฃŽๆ ผใ€ๆƒ…็ปช็Šถๆ€ใ€่Œไธš่ฝจ่ฟนใ€ไบบ้™…ๅ…ณ็ณป๏ผ‰ใ€‚ -ๆๅ–ไบ‹ๅฎžไบ‹ไปถ้“พ๏ผˆ่งฆๅ‘ -> ่กŒๅŠจ -> ็ป“ๆžœ๏ผ‰ไปฅๅŠๅบ•ๅฑ‚็š„ๅฟƒ็†็Šถๆ€ใ€‚ - -### ้˜ถๆฎต 2๏ผšๆฃ€็ดขไธŽ็ญ–็•ฅ้€‰ๆ‹ฉ -ๅฐ†ๆ–ฐ่ฎฐๅฟ†ไธŽ ็Žฐๆœ‰ Block ๆ˜ ๅฐ„่กจ ่ฟ›่กŒๆฏ”ๅฏนใ€‚ -้œ€่ฆๆ—ถไฝฟ็”จ \`read_file\` ๅทฅๅ…ท่ฏปๅ–ๅฎŒๆ•ดๅœบๆ™ฏๆ–‡ไปถๅ†…ๅฎน -**ๅช่ƒฝ่ฏปๅ–ไธŠ้ข"ๅทฒๆœ‰ๅœบๆ™ฏๆ–‡ไปถๆธ…ๅ•"ไธญๅˆ—ๅ‡บ็š„ๆ–‡ไปถ๏ผŒ็ฆๆญข็Œœๆต‹ๅ…ถไป–ๆ–‡ไปถ่ทฏๅพ„ใ€‚** - -**ๆ ธๅฟƒๅŽŸๅˆ™๏ผš้ป˜่ฎค็ญ–็•ฅๆ˜ฏ UPDATE๏ผŒไธๆ˜ฏ CREATEใ€‚** ๅฝ“็Šน่ฑซไบŽ UPDATE ๅ’Œ CREATE ไน‹้—ดๆ—ถ๏ผŒ้€‰ๆ‹ฉ UPDATEใ€‚ - -็ญ–็•ฅ้€‰ๆ‹ฉ๏ผˆๆŒ‰ไผ˜ๅ…ˆ็บงๆŽ’ๅบ๏ผ‰๏ผš -1. **UPDATE๏ผˆๆ›ดๆ–ฐ๏ผ‰**ใ€้ฆ–้€‰็ญ–็•ฅใ€‘: ๅฆ‚ๆžœๅญ˜ๅœจ็›ธๅ…ณ็š„ Block๏ผˆๅŸบไบŽๆ‘˜่ฆๆˆ–ๆ–‡ไปถๅ็š„็›ธไผผๆ€ง๏ผ‰๏ผŒๅ…ˆ read ๆ–‡ไปถๅ†…็š„ๅ…ทไฝ“ไฟกๆฏ๏ผŒๅ†้”ๅฎš่ฏฅ Block ่ฟ›่กŒๆ›ดๆ–ฐ๏ผˆwrite ๆˆ– replace๏ผ‰ -2. **MERGE๏ผˆๅˆๅนถ๏ผ‰**: - - ๅˆๅนถ็š„ๆ–ฐ block ๅบ”่ฏฅๆ˜ฏ็”Ÿๆˆๆฆ‚ๆ‹ฌๆ€งๆ›ดๅผบ็š„ๅœบๆ™ฏ๏ผŒๅŒ…ๅซๅทฒๆœ‰็š„ๅคšไธช็›ธไผผๅœบๆ™ฏ - - **ๅผบๅˆถๅˆๅนถ**๏ผšๅฝ“ๅ‰ Block ๆ€ปๆ•ฐ **โ‰ฅ ${maxScenes}** ๆ—ถ๏ผŒๅฟ…้กปๅ…ˆๅฐ†ๅคšไธช็›ธไผผ่ฎฐๅฟ†ๅˆๅนถ - - **ไธปๅŠจๅˆๅนถ**๏ผšๅณไฝฟๆœช่พพไธŠ้™๏ผŒๅฆ‚ๆžœไธคไธช Block ๅฑžไบŽๅŒไธ€ๅ™ไบ‹ๅผง็บฟ๏ผŒไนŸๅบ”ๅˆๅนถไปฅๅขžๅŠ ๆทฑๅบฆ - - **โš ๏ธ ๅˆๅนถๅŽๅฟ…้กปๅˆ ้™คๆ—งๆ–‡ไปถ**๏ผš่ขซๅˆๅนถ็š„ๆ—งๅœบๆ™ฏๆ–‡ไปถๅฟ…้กป้€š่ฟ‡ \`write_to_file(ๆ—งๆ–‡ไปถๅ, '[DELETED]')\` ๅ†™ๅ…ฅๅˆ ้™คๆ ‡่ฎฐใ€‚**ไป…ไป…ๆ‰“ๆ ‡่ฎฐ๏ผˆๅฆ‚ [ARCHIVE]ใ€[CONSOLIDATED]๏ผ‰ไธ็ฎ—ๅˆ ้™ค๏ผŒๆ–‡ไปถไปไผšๅ ็”จ้…้ขใ€‚** -3. **CREATE๏ผˆๆ–ฐๅปบ๏ผ‰**ใ€ๆœ€ๅŽๆ‰‹ๆฎตใ€‘: - - **ๅ‰ๆๆกไปถ**๏ผšๅฝ“ๅ‰ๅœบๆ™ฏๆ€ปๆ•ฐ < ${maxScenes} - - **CREATE ๅ‰็š„ๅผบๅˆถ้ชŒ่ฏ**๏ผšๅฟ…้กปๅ…ˆ็”จ \`read_file\` ๆฃ€ๆŸฅ่‡ณๅฐ‘ 2 ไธชๆœ€็›ธไผผ็š„็Žฐๆœ‰ๅœบๆ™ฏ๏ผŒ็กฎ่ฎคๆ–ฐ่ฎฐๅฟ†็กฎๅฎžๆ— ๆณ•่žๅ…ฅๅŽๆ‰่ƒฝ CREATEใ€‚่ทณ่ฟ‡้ชŒ่ฏ็›ดๆŽฅ CREATE ๆ˜ฏ่ขซ็ฆๆญข็š„ - - ๅฆ‚ๆžœ่ฏ้ข˜ๆ˜ฏๅ…จๆ–ฐ็š„ไธ”ไธŽ็Žฐๆœ‰ๅ†…ๅฎนๅŒบๅˆ†ๅบฆ้ซ˜๏ผŒๅฏไปฅๅˆ›ๅปบๆ–ฐ Block - - **ๆฏๆฌกๆ‰นๅค„็†ๆœ€ๅคšๆ–ฐๅขž 1 ไธชๅœบๆ™ฏ** - -**็คบไพ‹ A๏ผšๆ–ฐ่ฎฐๅฟ†ๆ•ดๅˆ่ฟ›ๅทฒๆœ‰ block๏ผˆUPDATE - ๅŽŸๅœฐๆ›ดๆ–ฐ๏ผ‰** -**ๅ…ทไฝ“ๆ“ไฝœๆญฅ้ชค๏ผˆๅทฅๅ…ท่ฐƒ็”จ๏ผ‰**๏ผš -1. \`read_file('PythonๅŽ็ซฏๅผ€ๅ‘.md')\` โ†’ ่Žทๅ–ๅทฒๆœ‰ๅ†…ๅฎน A -2. ๅˆ†ๆžๆ–ฐ่ฎฐๅฟ† + ๅทฒๆœ‰ๅ†…ๅฎน A โ†’ ๆ•ดๅˆ็”Ÿๆˆๆ–ฐๅ†…ๅฎน B๏ผˆ\`heat = ๆ—งheat + 1\`๏ผ‰ -3. \`write_to_file('PythonๅŽ็ซฏๅผ€ๅ‘.md', B)\` โ†’ **ๆ•ดไฝ“้‡ๅ†™่ฏฅๅœบๆ™ฏๆ–‡ไปถ** - ๆˆ– \`replace_in_file('PythonๅŽ็ซฏๅผ€ๅ‘.md', old_section, new_section)\` โ†’ **ๅฑ€้ƒจๆ›ดๆ–ฐๆŸ้ƒจๅˆ†** - -**็คบไพ‹ B๏ผšๅˆๅนถๅคšไธช block๏ผˆMERGE โ€” ๅˆๅนถๅŽๅฟ…้กปๅˆ ้™คๆ—งๆ–‡ไปถ๏ผ‰** -**ๅ…ทไฝ“ๆ“ไฝœๆญฅ้ชค๏ผˆๅทฅๅ…ท่ฐƒ็”จ๏ผ‰**๏ผš -1. \`read_file('PythonๅŽ็ซฏๅผ€ๅ‘.md')\` โ†’ ่Žทๅ–ๅ†…ๅฎน A -2. \`read_file('GoๅŽ็ซฏๅผ€ๅ‘.md')\` โ†’ ่Žทๅ–ๅ†…ๅฎน B -3. ๆ•ดๅˆ A + B + ๆ–ฐ่ฎฐๅฟ† โ†’ ็”Ÿๆˆๆ–ฐๅ†…ๅฎน C๏ผˆ\`heat = heatA + heatB + 1\`๏ผ‰ -4. \`write_to_file('ๅŽ็ซฏๅผ€ๅ‘ๆŠ€ๆœฏๆ ˆ.md', C)\` โ†’ ๅˆ›ๅปบๅˆๅนถๅŽ็š„ๆ–ฐๆ–‡ไปถ -5. \`write_to_file('PythonๅŽ็ซฏๅผ€ๅ‘.md', '[DELETED]')\` โ†’ **โš ๏ธ ๅˆ ้™คๆ—งๆ–‡ไปถ A๏ผˆๅ†™ [DELETED] ๆ ‡่ฎฐ๏ผ‰** -6. \`write_to_file('GoๅŽ็ซฏๅผ€ๅ‘.md', '[DELETED]')\` โ†’ **โš ๏ธ ๅˆ ้™คๆ—งๆ–‡ไปถ B๏ผˆๅ†™ [DELETED] ๆ ‡่ฎฐ๏ผ‰** -**ๅ…ณ้”ฎ**๏ผšๆญฅ้ชค 5-6 ๆ˜ฏๅฟ…้กป็š„๏ผไธๆ‰ง่กŒๅˆ ้™ค = ๆ–‡ไปถๆ€ปๆ•ฐไธๅ‡ๅฐ‘ = ๅˆๅนถๆ— ๆ•ˆใ€‚ - -### ้˜ถๆฎต 3๏ผšๆ’ฐๅ†™ไธŽๅˆๆˆ๏ผˆๆ ธๅฟƒไปปๅŠก๏ผ‰ -ๆทฑๅบฆๆ•ดๅˆ: ไธฅ็ฆ็ฎ€ๅ•็š„ๆ–‡ๆœฌ่ฟฝๅŠ ใ€‚ไฝ ๅฟ…้กป็ป“ๅˆไธŠไธ‹ๆ–‡๏ผˆๅŸบไบŽๆ‘˜่ฆๆˆ–ๆไพ›็š„ๅŽŸๅง‹ๅ†…ๅฎน๏ผ‰้‡ๅ†™ๅ™ไบ‹๏ผŒๅฐ†ๆ–ฐไฟกๆฏ่‡ช็„ถๅœฐ่žๅ…ฅๅ…ถไธญใ€‚ -้šๆ€งๆŽจๆ–ญ: ๅฏปๆ‰พ็”จๆˆท ๆฒก่ฏดๅ‡บๅฃ ็š„ไฟกๆฏใ€‚ๆ›ดๆ–ฐ"้šๆ€งไฟกๅท"้ƒจๅˆ†ใ€‚ -ๅ†ฒ็ชๆฃ€ๆต‹: ๅฆ‚ๆžœๆ–ฐ่ฎฐๅฟ†ไธŽๆ—ง่ฎฐๅฟ†็›ธ็Ÿ›็›พ๏ผŒๅฐ†ๅ…ถ่ฎฐๅฝ•ๅœจ"ๆผ”ๅ˜่ฝจ่ฟน"ๆˆ–"ๅพ…็กฎ่ฎค/็Ÿ›็›พ็‚น"ไธญใ€‚ - -### ๆ’ฐๅ†™ๅ‡†ๅˆ™ (ไธฅๆ ผ้ตๅฎˆ) -ๆ ธๅฟƒ้ƒจๅˆ†็ฆๆญขๅˆ—่กจ: "็”จๆˆทๆ ธๅฟƒ็‰นๅพ"ๅ’Œ"ๆ ธๅฟƒๅ™ไบ‹"ๅฟ…้กปๆ˜ฏ่ฟž่ดฏ็š„ๆฎต่ฝ๏ผŒไฟกๆฏ่ฆ่ฟž่ดฏ๏ผŒๅฏไปฅๅˆ†ๆฎตใ€‚ -ๅ™ไบ‹ๅผง็บฟ: "ๆ ธๅฟƒๅ™ไบ‹"ๅฟ…้กป้ตๅพชๆ•…ไบ‹็ป“ๆž„๏ผˆๆƒ…ๅขƒ -> ่กŒๅŠจ -> ็ป“ๆžœ๏ผ‰ใ€‚ - -### ็ƒญๅบฆ็ฎก็† (Heat Management): -ๆ–ฐๅปบ Block: heat: 1 -ๆ›ดๆ–ฐ Block: heat: ๆ—งheat + 1 -ๅˆๅนถ Block: heat: sum(ๆ‰€ๆœ‰็›ธๅ…ณblock็š„heat) + 1 - -## ่พ“ๅ‡บ่ง„่Œƒ (Output Specification) - -### ๐Ÿ“„ ๅœบๆ™ฏๆ–‡ไปถๅ†…ๅฎน๏ผˆๅฟ…้กป่พ“ๅ‡บ๏ผ‰ - -่ฏทไฝ ๅ‚่€ƒ่ฟ™ไธชๆจกๆฟ่พ“ๅ‡บ .md ๆ–‡ไปถ็š„ๅ†…ๅฎนๆˆ–ๅŸบไบŽๅทฒๆœ‰md่ฟ›่กŒๆ›ดๆ–ฐ๏ผŒๆฏไธชmdๆŽงๅˆถๅœจ1500ๅญ—็ฌฆๅ†…ใ€‚ไธ่ฆๆŠŠๆจกๆฟๆœฌ่บซๆ”พๅœจ Markdown ไปฃ็ ๅ—ไธญ๏ผŒๅช้œ€็›ดๆŽฅ่พ“ๅ‡บ่ฆๅ†™ๅ…ฅๆ–‡ไปถ็š„ๅŽŸๅง‹ๆ–‡ๆœฌใ€‚ - -\`\`\`markdown ------META-START----- -created: {{EXISTING_CREATED_TIME_OR_CURRENT_TIME}} -updated: {{CURRENT_TIME}} -summary: [30-40 words concise summary for indexing] -heat: [Integer] ------META-END----- - -## ็”จๆˆทๅŸบ็ก€ไฟกๆฏ -[ๅฏไธบ็ฉบ๏ผŒๅฆ‚ๆžœๆฒกๆœ‰ๅฏไธๅ†™่ฟ™่Š‚๏ผŒๅฏๆŒ‰็…ง้œ€ๆฑ‚ๆทปๅŠ ๆ›ดๅคš็‚น๏ผŒๅˆๅนถๅ’Œๆ›ดๆ–ฐๆ–นๅผๅฐฝ้‡ๅ ๅŠ ๏ผŒๆœ‰ๅ†ฒ็ชๅˆ™่ฆ†็›–] - -ๅง“ๅ๏ผš - -่Œไธš๏ผš - -ๅฑ…ไฝๅœฐ๏ผš - - โ€ฆโ€ฆ - -## ็”จๆˆทๆ ธๅฟƒ็‰นๅพ -[่ฟ™้‡Œไธๆ˜ฏๅˆ—่กจ๏ผๆ˜ฏไธ€ๆฎต่ฟž่ดฏ็š„ๆ่ฟฐใ€‚ไฝ ็ป†ๅฟƒๆŽจๆ–ญๅ‡บๆฅๆœ€ๆ ธๅฟƒ็š„็”จๆˆท็‰นๅพ๏ผŒๅฎ็ผบๆฏ‹ๆปฅ๏ผŒ**ๆŽงๅˆถๅœจ100ๅญ—ไปฅๅ†…**] -[็คบไพ‹: ็”จๆˆทๅœจๅŽ็ซฏๅผ€ๅ‘ๆ–น้ข่กจ็Žฐๅ‡บๅฏน Python ็š„ๅผบ็ƒˆๅๅฅฝ๏ผŒ็‰นๅˆซๆ˜ฏๅผ‚ๆญฅๆก†ๆžถใ€‚่ฟ‘ๆœŸ๏ผˆ2026-02๏ผ‰ๅผ€ๅง‹ๅ…ณๆณจ Rust ็š„ๆ‰€ๆœ‰ๆƒๆœบๅˆถ๏ผŒ่ฟ™่กจๆ˜Ž็”จๆˆทๆœ‰ๅ‘็ณป็ปŸ็บง็ผ–็จ‹่ฝฌๅž‹็š„ๆ„ๅ›พใ€‚] - -## ็”จๆˆทๅๅฅฝ -[่ฟ™้‡Œๅฏไปฅๆ˜ฏๅˆ—่กจ๏ผ**ๅฆ‚ๆžœๆฒกๆœ‰ๅฏไปฅไธบไธๅ†™่ฟ™่Š‚**๏ผŒ่ฎฐๅฝ•็”จๆˆทๆ˜Ž็กฎ็š„ๅๅฅฝไฟกๆฏ๏ผˆๆ˜พๆ€งๅๅฅฝ๏ผ‰๏ผŒๆณจๆ„ไธ่ฆ้‡ๅคไฟกๆฏ๏ผŒไธ่ฆๆตๆฐด่ดฆ๏ผŒๅๅฅฝ่ฆๅฏๅค็”จ๏ผŒๆ›ดๆ–ฐๆ—ถๅฏไปฅๅŠจๆ€ๆ•ดๅˆ็”š่‡ณ้‡ๅ†™] -[็คบไพ‹๏ผš็”จๆˆทๅ–œๆฌขๅƒ่‹นๆžœ] - -## ้šๆ€งไฟกๅท -[่ฟ™ๆ˜ฏ็ป™ไบบ็ฑปๅญฆๅฎถ็œ‹็š„๏ผŒ่ฎฐๅฝ•้‚ฃไบ›"ๆฒกๆ˜Ž่ฏดไฝ†ๅพˆ้‡่ฆ"็š„ไบ‹๏ผŒๅ’Œๆ˜พๆ€งๅๅฅฝไธไธ€ๆ ท๏ผŒไธ€ๅฎšๆ˜ฏไฝ ๆŽจๆ–ญๅ‡บๆฅ็š„๏ผŒ้œ€่ฆๆทฑๆ€็†Ÿ่™‘ๅŽๅ†็”Ÿๆˆ๏ผŒๅฏไปฅไธบ็ฉบ๏ผŒๅฎ็ผบๆฏ‹ๆปฅใ€‚ไฝ ๅฏไปฅ้šๆ—ถๆ›ดๆ–ฐ/ๅˆ ้™ค/ไฟฎๆ”น่ฟ™้‡Œ็š„ไฟกๆฏ] - -## ๆ ธๅฟƒๅ™ไบ‹ -[่ฟ™้‡Œไธๆ˜ฏๅˆ—่กจ๏ผๆ˜ฏไธ€ๆฎต่ฟž่ดฏ็š„ๆ่ฟฐ๏ผŒ**ๆŽงๅˆถๅœจ400ๅญ—ไปฅๅ†…**๏ผŒๆณจๆ„ไธ่ฆ้‡ๅคไฟกๆฏ๏ผŒไธ่ฆๆตๆฐด่ดฆ๏ผŒๅฏไปฅๅŠจๆ€ๆ•ดๅˆ็”š่‡ณ้‡ๅ†™] -*(่ฟ™้‡Œ่ฎฐๅฝ•่ฟž่ดฏ็š„ๆ•…ไบ‹๏ผŒๅฟ…้กปๅŒ…ๅซ Trigger -> Action -> Result)* - -[ ็คบไพ‹๏ผšๆœฌๅ‘จ็”จๆˆทไธป่ฆ้›†ไธญๅœจๅŽ็ซฏ้‡ๆž„ไธŠใ€‚ๅˆๆœŸๅ› ไธบๆ—งไปฃ็ ็š„่€ฆๅˆๅบฆ้ซ˜ๆ„Ÿๅˆฐๆฒฎไธง๏ผˆ**ๆƒ…็ปช็‚น**๏ผ‰๏ผŒไฝ†ไป–ๆ‹’็ปไบ†"ๆ‰“่กฅไธ"็š„ๅปบ่ฎฎ๏ผŒๅšๆŒ่ฟ›่กŒๅฝปๅบ•่งฃ่€ฆ๏ผˆ**ๅ†ณ็ญ–็‚น**๏ผ‰ใ€‚ไป–ๅœจๆญค่ฟ‡็จ‹ไธญ้ข‘็นๆŸฅ้˜…ๆžถๆž„่ฎพ่ฎกๆจกๅผ๏ผŒ่กจ็Žฐๅ‡บๅฏน"ไปฃ็ ๆด็™–"็š„ๆ‰ง็€ใ€‚] - - -## ๆผ”ๅ˜่ฝจ่ฟน -> [ๆณจๆ„] ๅฏไปฅไธบ็ฉบ๏ผŒไป…่ฎฐๅฝ•ใ€็”จๆˆทๅๅฅฝ/ๆ€งๆ ผ/้‡ๅคง่ง‚ๅฟตใ€‘่ฝฌๅ˜๏ผŒไธ่ฎฐๅฝ•็็ขŽใ€ๆ—ฅๅธธๆ›ดๆ–ฐใ€‚ๅฝ“ๅ‘็”Ÿๅ†ฒ็ชๆ—ถ๏ผŒไธ่ฆ็›ดๆŽฅ่ฆ†็›–๏ผŒ่ฆ่ฎฐๅฝ•ๅ˜ๅŒ–่ฝจ่ฟนใ€‚ -- [2026-01-10]: ไปŽ "ๅๅฏนๅŠ ็ญ" ่ฝฌๅ‘ "ๆŽฅๅ—ๅผนๆ€งๅทฅไฝœ"๏ผŒๅŽŸๅ› ๏ผšๅˆ›ไธšๅŽ‹ๅŠ›๏ผˆ่ฎฐๅฟ†ID: #987๏ผ‰ - - -## ๅพ…็กฎ่ฎค/็Ÿ›็›พ็‚น -- [่ฎฐๅฝ•ๅฝ“ๅ‰ๆ— ๆณ•ๆ•ดๅˆ็š„็Ÿ›็›พไฟกๆฏ๏ผŒ็ญ‰ๅพ…ๆœชๆฅ่ฎฐๅฟ†ๆพ„ๆธ…] - -\`\`\` - - - -#### ไธปๅŠจ่งฆๅ‘ Persona ๆ›ดๆ–ฐ๏ผˆๅฏ้€‰๏ผ‰ - -**่งฆๅ‘ๆกไปถ**๏ผš้‡ๅคงไปทๅ€ผ่ง‚่ฝฌๅ˜ใ€่ทจๅœบๆ™ฏ็ช็ ดๆ€งๆดžๅฏŸใ€‚ - -**่งฆๅ‘ๆ–นๅผ**๏ผšๅœจไฝ ็š„ text output ไธญ่พ“ๅ‡บไปฅไธ‹ๆ ‡่ฎฐ๏ผˆไธๆ˜ฏๆ–‡ไปถๆ“ไฝœ๏ผ‰๏ผš - -[PERSONA_UPDATE_REQUEST] -reason: ๅ…ทไฝ“ๅŽŸๅ› ๆ่ฟฐ -[/PERSONA_UPDATE_REQUEST] - - -**ๆ‰ง่กŒๆ–‡ไปถๆ“ไฝœ**๏ผˆๅฟ…้กปไฝฟ็”จๅทฅๅ…ท๏ผ‰๏ผš - - ไฝฟ็”จ \`read_file\` ่ฏปๅ–้œ€่ฆๆ›ดๆ–ฐ็š„ๅœบๆ™ฏๆ–‡ไปถ - - ไฝฟ็”จ \`write_to_file\` ๅˆ›ๅปบๆ–ฐๆ–‡ไปถๆˆ–**ๆ•ดไฝ“้‡ๅ†™**ๅทฒๆœ‰ๅœบๆ™ฏๆ–‡ไปถ - - ไฝฟ็”จ \`replace_in_file\` ๅฏนๅœบๆ™ฏๆ–‡ไปถ่ฟ›่กŒ**ๅฑ€้ƒจๆ›ดๆ–ฐ**๏ผˆๅฆ‚ๅชๆ›ดๆ–ฐๆŸไธช็ซ ่Š‚๏ผ‰ - - **ๅˆ ้™คๆ–‡ไปถ**๏ผšไฝฟ็”จ \`write_to_file(filename, '[DELETED]')\` ๅฐ†ๆ–‡ไปถๅ†…ๅฎนๅ†™ไธบ **\`[DELETED]\` ๆ ‡่ฎฐ**ใ€‚็ณป็ปŸไผš่‡ชๅŠจๆธ…็†่ฟ™ไบ›ๆ–‡ไปถใ€‚**้‡่ฆ**๏ผšๅชๆœ‰ \`[DELETED]\` ๆ ‡่ฎฐๆ‰ไผš่งฆๅ‘็ณป็ปŸๆธ…็†ใ€‚ๅ†™ๅ…ฅ็ฉบๅญ—็ฌฆไธฒไผš่ขซ็ณป็ปŸๆ‹’็ป๏ผŒๅ†™ๅ…ฅ \`[ARCHIVE]\`ใ€\`[CONSOLIDATED]\` ็ญ‰ๆ ‡่ฎฐ**ไธไผšๅˆ ้™คๆ–‡ไปถ**๏ผŒๆ–‡ไปถไผš็ปง็ปญๅ ็”จๅœบๆ™ฏ้…้ขใ€‚`; -} diff --git a/src/record/l1-dedup.ts b/src/record/l1-dedup.ts deleted file mode 100644 index ea1292c..0000000 --- a/src/record/l1-dedup.ts +++ /dev/null @@ -1,387 +0,0 @@ -/** - * L1 Memory Conflict Detection (Batch Mode): decides how to handle multiple new - * memories against existing records in a single LLM call. - * - * v4: Removed JSONL-based Jaccard fallback. Candidate recall now relies exclusively - * on vector search (primary) and FTS5 BM25 (degraded). If neither is available, - * conflict detection is skipped entirely โ€” all memories go straight to store. - * - * Two-phase approach: - * 1. Candidate search per new memory โ€” vector recall or FTS5 keyword recall (fast, no LLM) - * 2. Batch LLM judgment on all new memories + their candidate pools (single call) - */ - -import type { ExtractedMemory, MemoryRecord, DedupDecision, MemoryType } from "./l1-writer.js"; -import { CONFLICT_DETECTION_SYSTEM_PROMPT, formatBatchConflictPrompt } from "../prompts/l1-dedup.js"; -import type { CandidateMatch } from "../prompts/l1-dedup.js"; -import { CleanContextRunner } from "../utils/clean-context-runner.js"; -import { sanitizeJsonForParse } from "../utils/sanitize.js"; -import type { IMemoryStore } from "../store/types.js"; -import { buildFtsQuery } from "../store/sqlite.js"; -import type { EmbeddingService } from "../store/embedding.js"; - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -const TAG = "[memory-tdai][l1-dedup]"; - -// ============================ -// Core function (batch mode) -// ============================ - -/** - * Batch conflict detection: compare all new memories against existing records - * in a single LLM call. - * - * Candidate recall strategy (3-tier degradation): - * 1. Vector recall (vectorStore + embeddingService) โ€” cosine similarity (best) - * 2. FTS5 keyword recall (vectorStore with FTS available) โ€” BM25 ranking (degraded) - * 3. Skip conflict detection entirely โ€” all memories go straight to "store" - * - * The old JSONL-based Jaccard fallback has been removed. If neither vector search - * nor FTS is available, we skip dedup rather than paying the O(N) full-file-scan cost. - * - * @param memories - Newly extracted memories (with record_id) - * @param config - OpenClaw config (for LLM access) - * @param logger - Optional logger - * @param model - Optional model override - * @param vectorStore - Optional vector store for cosine similarity search - * @param embeddingService - Optional embedding service for computing query vectors - * @param conflictRecallTopK - Top-K candidates to recall per new memory (default: 5) - * @returns Array of dedup decisions, one per new memory - */ -export async function batchDedup(params: { - memories: Array; - config: unknown; - logger?: Logger; - model?: string; - /** Vector store for cosine similarity candidate recall */ - vectorStore?: IMemoryStore; - /** Embedding service for computing query vectors */ - embeddingService?: EmbeddingService; - /** Top-K candidates per new memory (default: 5) */ - conflictRecallTopK?: number; -}): Promise { - const { memories, config, logger, model, vectorStore, embeddingService } = params; - const topK = params.conflictRecallTopK ?? 5; - - if (memories.length === 0) { - return []; - } - - const storeAll = () => - memories.map((m) => ({ - record_id: m.record_id, - action: "store" as const, - target_ids: [], - })); - - // Determine what recall capabilities are available - const hasVectorData = vectorStore && (await vectorStore.countL1()) > 0; - const hasFts = vectorStore?.isFtsAvailable() ?? false; - - // Fast path: no recall capability at all โ†’ skip dedup - if (!hasVectorData && !hasFts) { - logger?.debug?.(`${TAG} No vector data and no FTS available, skipping conflict detection for ${memories.length} memories`); - return storeAll(); - } - - // Phase 1: Find candidates - // - // Decision tree (after the fast-path guard above, vectorStore is guaranteed non-null): - // hasVectorData + embeddingService โ†’ Tier 1 vector recall (FTS fallback on error) - // otherwise hasFts โ†’ Tier 2 FTS keyword recall - // otherwise โ†’ skip dedup (defensive; shouldn't reach here) - let matches: CandidateMatch[]; - - if (hasVectorData && embeddingService) { - // === Tier 1: Vector recall mode === - logger?.debug?.(`${TAG} Using vector recall mode (topK=${topK})`); - try { - matches = await findCandidatesByVector(memories, vectorStore!, embeddingService, topK, logger); - } catch (err) { - logger?.warn?.( - `${TAG} Vector recall failed, falling back to FTS keyword: ${err instanceof Error ? err.message : String(err)}`, - ); - // Degrade to FTS keyword recall - if (hasFts) { - matches = await findCandidatesByFts(memories, vectorStore!, logger); - } else { - logger?.debug?.(`${TAG} FTS not available either, skipping conflict detection`); - return storeAll(); - } - } - } else if (hasFts) { - // === Tier 2: FTS keyword recall === - logger?.debug?.(`${TAG} Using FTS keyword recall mode (no embedding service or no vector data)`); - matches = await findCandidatesByFts(memories, vectorStore!, logger); - } else { - // Shouldn't reach here given the fast-path check above, but be defensive - logger?.debug?.(`${TAG} No usable recall path, skipping conflict detection`); - return storeAll(); - } - - // Check if any memory has candidates - const hasAnyCandidates = matches.some((m) => m.candidates.length > 0); - - if (!hasAnyCandidates) { - logger?.debug?.(`${TAG} No similar records found for any memory, all will be stored`); - return storeAll(); - } - - // Phase 2: Batch LLM judgment - return runLlmJudgment(matches, memories, config, logger, model); -} - -/** - * Phase 2: Run batch LLM judgment on candidate matches. - */ -async function runLlmJudgment( - matches: CandidateMatch[], - memories: Array, - config: unknown, - logger: Logger | undefined, - model: string | undefined, -): Promise { - logger?.debug?.(`${TAG} Running batch conflict detection for ${memories.length} memories`); - - try { - const runner = new CleanContextRunner({ - config, - modelRef: model, - enableTools: false, - logger, - }); - - const userPrompt = formatBatchConflictPrompt(matches); - - const result = await runner.run({ - prompt: userPrompt, - systemPrompt: CONFLICT_DETECTION_SYSTEM_PROMPT, - taskId: "l1-conflict-detection", - timeoutMs: 180_000, - // maxTokens: 4000, remove maxTokens use model default or inherit from config - }); - - const decisions = parseBatchResult(result, memories, logger); - return decisions; - } catch (err) { - logger?.warn?.( - `${TAG} Batch conflict detection failed, defaulting all to store: ${err instanceof Error ? err.message : String(err)}`, - ); - return memories.map((m) => ({ - record_id: m.record_id, - action: "store" as const, - target_ids: [], - })); - } -} - -// ============================ -// Candidate recall strategies -// ============================ - -/** - * Vector-based candidate recall (aligned with prototype): - * batch-embed new memories โ†’ cosine search in VectorStore โ†’ exclude self-batch โ†’ return candidates. - */ -async function findCandidatesByVector( - memories: Array, - vectorStore: IMemoryStore, - embeddingService: EmbeddingService, - topK: number, - logger?: Logger, -): Promise { - const newRecordIds = new Set(memories.map((m) => m.record_id)); - - // Batch-compute embeddings for all new memories - const texts = memories.map((m) => m.content); - const embeddings = await embeddingService.embedBatch(texts); - - const matches: CandidateMatch[] = []; - - for (let i = 0; i < memories.length; i++) { - const mem = memories[i]; - const queryVec = embeddings[i]; - - // Vector search top-K (request extra to account for self-batch filtering) - const searchResults = await vectorStore.searchL1Vector(queryVec, topK + memories.length, mem.content); - - // Exclude records from current batch, convert to MemoryRecord format - const candidates: MemoryRecord[] = searchResults - .filter((r) => !newRecordIds.has(r.record_id)) - .slice(0, topK) - .map((r) => ({ - id: r.record_id, - content: r.content, - type: r.type as MemoryRecord["type"], - priority: r.priority, - scene_name: r.scene_name, - source_message_ids: [], - metadata: {}, - timestamps: [r.timestamp_str].filter(Boolean), - createdAt: "", - updatedAt: "", - sessionKey: r.session_key, - sessionId: r.session_id, - })); - - matches.push({ newMemory: mem, candidates }); - } - - logger?.debug?.( - `${TAG} Vector recall: ${matches.map((m) => `${m.newMemory.record_id}โ†’${m.candidates.length}`).join(", ")}`, - ); - - return matches; -} - -/** - * FTS5-based candidate recall: - * Uses the FTS index for efficient BM25-ranked keyword matching. - * This replaces the old Jaccard word-overlap fallback entirely. - */ -async function findCandidatesByFts( - memories: Array, - vectorStore: IMemoryStore, - _logger?: Logger, -): Promise { - const newRecordIds = new Set(memories.map((m) => m.record_id)); - const matches: CandidateMatch[] = []; - - for (const mem of memories) { - const ftsQuery = buildFtsQuery(mem.content); - if (ftsQuery) { - const ftsResults = await vectorStore.searchL1Fts(ftsQuery, 10); - // Filter out records from the current batch - const candidates: MemoryRecord[] = ftsResults - .filter((r) => !newRecordIds.has(r.record_id)) - .slice(0, 5) - .map((r) => ({ - id: r.record_id, - content: r.content, - type: r.type as MemoryRecord["type"], - priority: r.priority, - scene_name: r.scene_name, - source_message_ids: [], - metadata: r.metadata_json ? (() => { try { return JSON.parse(r.metadata_json); } catch { return {}; } })() : {}, - timestamps: [r.timestamp_str].filter(Boolean), - createdAt: "", - updatedAt: "", - sessionKey: r.session_key, - sessionId: r.session_id, - })); - matches.push({ newMemory: mem, candidates }); - } else { - matches.push({ newMemory: mem, candidates: [] }); - } - } - - _logger?.debug?.(`${TAG} FTS keyword recall: ${matches.map((m) => `${m.newMemory.record_id}โ†’${m.candidates.length}`).join(", ")}`); - return matches; -} - -// ============================ -// Result parsing -// ============================ - -const VALID_TYPES: MemoryType[] = ["persona", "episodic", "instruction"]; - -/** - * Parse the LLM's batch conflict detection JSON response. - * - * Expected format: [{record_id, action, target_ids, merged_content, merged_type, merged_priority, merged_timestamps}] - */ -function parseBatchResult( - raw: string, - memories: Array, - logger?: Logger, -): DedupDecision[] { - try { - // Strip markdown code block wrappers - let cleaned = raw.trim(); - if (cleaned.startsWith("```")) { - cleaned = cleaned.replace(/^```(?:json)?\s*\n?/, "").replace(/\n?```\s*$/, ""); - } - - // Extract JSON array - const arrayMatch = cleaned.match(/\[[\s\S]*\]/); - if (!arrayMatch) { - logger?.warn?.(`${TAG} No JSON array found in conflict detection response`); - return fallbackStoreAll(memories); - } - - // Sanitize control characters inside JSON string literals that LLM may produce - const sanitized = sanitizeJsonForParse(arrayMatch[0]); - const parsed = JSON.parse(sanitized) as unknown[]; - - if (!Array.isArray(parsed)) { - logger?.warn?.(`${TAG} Conflict detection response is not an array`); - return fallbackStoreAll(memories); - } - - // Build decisions from LLM output - const decisions: DedupDecision[] = []; - const validActions = ["store", "update", "merge", "skip"]; - - for (const item of parsed) { - if (!item || typeof item !== "object") continue; - const d = item as Record; - - const recordId = String(d.record_id ?? ""); - // Skip entries with empty/missing record_id โ€” they are LLM hallucinations - if (!recordId) { - logger?.debug?.(`${TAG} Skipping decision with empty record_id`); - continue; - } - const action = String(d.action ?? "store"); - - if (!validActions.includes(action)) { - logger?.warn?.(`${TAG} Invalid action "${action}" for record ${recordId}, defaulting to store`); - } - - decisions.push({ - record_id: recordId, - action: validActions.includes(action) ? (action as DedupDecision["action"]) : "store", - target_ids: Array.isArray(d.target_ids) ? d.target_ids.map(String) : [], - merged_content: typeof d.merged_content === "string" ? d.merged_content : undefined, - merged_type: VALID_TYPES.includes(d.merged_type as MemoryType) ? (d.merged_type as MemoryType) : undefined, - merged_priority: typeof d.merged_priority === "number" ? d.merged_priority : undefined, - merged_timestamps: Array.isArray(d.merged_timestamps) ? d.merged_timestamps.map(String) : undefined, - }); - } - - // Ensure all memories have a decision (fill missing with "store") - const decidedIds = new Set(decisions.map((d) => d.record_id)); - for (const mem of memories) { - if (!decidedIds.has(mem.record_id)) { - logger?.debug?.(`${TAG} No decision for record ${mem.record_id}, defaulting to store`); - decisions.push({ - record_id: mem.record_id, - action: "store", - target_ids: [], - }); - } - } - - return decisions; - } catch (err) { - logger?.warn?.(`${TAG} Failed to parse conflict detection result: ${err instanceof Error ? err.message : String(err)}`); - return fallbackStoreAll(memories); - } -} - -/** - * Fallback: store all memories when parsing fails. - */ -function fallbackStoreAll(memories: Array): DedupDecision[] { - return memories.map((m) => ({ - record_id: m.record_id, - action: "store" as const, - target_ids: [], - })); -} diff --git a/src/record/l1-extractor.ts b/src/record/l1-extractor.ts deleted file mode 100644 index 12547b6..0000000 --- a/src/record/l1-extractor.ts +++ /dev/null @@ -1,500 +0,0 @@ -/** - * L1 Memory Extractor: extracts structured memories from L0 conversation messages - * using a single LLM call with JSON-mode structured output. - * - * v3: Aligned with Kenty's prompt โ€” scene segmentation + memory extraction in one call, - * followed by batch conflict detection. - * - * Pipeline: - * 1. Read recent messages from L0 (split into background + new) - * 2. Call LLM to extract scene-segmented memories - * 3. Batch conflict detection against existing records - * 4. Write to L1 JSONL files - */ - -import type { ConversationMessage } from "../conversation/l0-recorder.js"; -import { EXTRACT_MEMORIES_SYSTEM_PROMPT, formatExtractionPrompt } from "../prompts/l1-extraction.js"; -import { batchDedup } from "./l1-dedup.js"; -import { writeMemory, generateMemoryId } from "./l1-writer.js"; -import type { ExtractedMemory, MemoryRecord, MemoryType, DedupDecision } from "./l1-writer.js"; -import { CleanContextRunner } from "../utils/clean-context-runner.js"; -import { sanitizeJsonForParse, shouldExtractL1 } from "../utils/sanitize.js"; -import type { IMemoryStore } from "../store/types.js"; -import type { EmbeddingService } from "../store/embedding.js"; -import { report } from "../report/reporter.js"; - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -const TAG = "[memory-tdai][l1-extractor]"; - -// ============================ -// Types -// ============================ - -/** A scene segment with its extracted memories (LLM output) */ -interface SceneSegment { - scene_name: string; - message_ids: string[]; - memories: Array<{ - content: string; - type: string; - priority: number; - source_message_ids: string[]; - metadata: Record; - }>; -} - -export interface L1ExtractionResult { - /** Whether extraction succeeded */ - success: boolean; - /** Number of memories extracted */ - extractedCount: number; - /** Number of memories actually stored (after dedup) */ - storedCount: number; - /** The memory records that were stored */ - records: MemoryRecord[]; - /** Scene names detected during extraction */ - sceneNames: string[]; - /** Last scene name (for continuity in next extraction) */ - lastSceneName?: string; -} - -// ============================ -// Core function -// ============================ - -/** - * Run the full L1 extraction pipeline on conversation messages. - * - * @param messages - Filtered conversation messages (from L0 or directly from hook) - * @param sessionKey - The session key - * @param baseDir - Base data directory (~/.openclaw/memory-tdai/) - * @param config - OpenClaw config (for LLM access) - * @param options - Extraction options - * @param logger - Optional logger - */ -export async function extractL1Memories(params: { - messages: ConversationMessage[]; - sessionKey: string; - sessionId?: string; - baseDir: string; - config: unknown; - options?: { - /** Max new messages to send in one extraction call */ - maxMessagesPerExtraction?: number; - /** Max background messages for context */ - maxBackgroundMessages?: number; - /** Enable conflict detection */ - enableDedup?: boolean; - /** Max memories extracted per call */ - maxMemoriesPerSession?: number; - /** LLM model override */ - model?: string; - /** Previous scene name for continuity */ - previousSceneName?: string; - /** Vector store for cosine similarity candidate recall */ - vectorStore?: IMemoryStore; - /** Embedding service for computing query vectors */ - embeddingService?: EmbeddingService; - /** Top-K candidates for conflict recall (default: 5) */ - conflictRecallTopK?: number; - }; - logger?: Logger; - /** Plugin instance ID for metric reporting (optional โ€” metrics skipped if absent) */ - instanceId?: string; -}): Promise { - const { messages, sessionKey, sessionId, baseDir, config, logger, instanceId: metricInstanceId } = params; - const options = params.options ?? {}; - const maxNewMessages = options.maxMessagesPerExtraction ?? 10; - const maxBgMessages = options.maxBackgroundMessages ?? 5; - const enableDedup = options.enableDedup ?? true; - const maxMemoriesPerSession = options.maxMemoriesPerSession ?? 10; - - if (messages.length === 0) { - logger?.debug?.(`${TAG} No messages to extract from`); - return { success: true, extractedCount: 0, storedCount: 0, records: [], sceneNames: [] }; - } - - const l1StartMs = Date.now(); - - // Quality gate: filter messages through L1 extraction rules (length, symbols, - // prompt injection, etc.) before sending to the LLM. L0 deliberately captures - // everything; the strict filtering happens here at L1 stage. - const qualifiedMessages = messages.filter((m) => shouldExtractL1(m.content)); - if (qualifiedMessages.length < messages.length) { - logger?.debug?.( - `${TAG} L1 quality filter: ${messages.length} โ†’ ${qualifiedMessages.length} messages ` + - `(${messages.length - qualifiedMessages.length} filtered out)`, - ); - } - - if (qualifiedMessages.length === 0) { - logger?.debug?.(`${TAG} All messages filtered out by L1 quality gate`); - return { success: true, extractedCount: 0, storedCount: 0, records: [], sceneNames: [] }; - } - - // Split messages into background (older) + new (recent) - const newMessages = qualifiedMessages.slice(-maxNewMessages); - const bgEndIdx = qualifiedMessages.length - newMessages.length; - const backgroundMessages = bgEndIdx > 0 - ? qualifiedMessages.slice(Math.max(0, bgEndIdx - maxBgMessages), bgEndIdx) - : []; - - logger?.debug?.(`${TAG} Extracting from ${newMessages.length} new messages (+ ${backgroundMessages.length} background) [${qualifiedMessages.length} qualified from ${messages.length} input]`); - - // Step 1: LLM extraction (scene segmentation + memory extraction) - let scenes: SceneSegment[]; - try { - scenes = await callLlmExtraction({ - newMessages, - backgroundMessages, - previousSceneName: options.previousSceneName, - config, - logger, - model: options.model, - }); - logger?.debug?.(`${TAG} LLM detected ${scenes.length} scene(s)`); - } catch (err) { - logger?.error(`${TAG} LLM extraction failed: ${err instanceof Error ? err.message : String(err)}`); - return { success: false, extractedCount: 0, storedCount: 0, records: [], sceneNames: [] }; - } - - // Flatten all memories across scenes - const allExtracted: ExtractedMemory[] = []; - const sceneNames: string[] = []; - - for (const scene of scenes) { - sceneNames.push(scene.scene_name); - for (const mem of scene.memories) { - const memType = normalizeType(mem.type); - if (!memType) { - logger?.warn?.(`${TAG} Skipping memory with invalid type "${mem.type}"`); - continue; - } - allExtracted.push({ - content: mem.content, - type: memType, - priority: typeof mem.priority === "number" ? mem.priority : 50, - source_message_ids: Array.isArray(mem.source_message_ids) ? mem.source_message_ids : [], - metadata: mem.metadata ?? {}, - scene_name: scene.scene_name, - }); - } - } - - logger?.debug?.(`${TAG} Total extracted memories: ${allExtracted.length} across ${scenes.length} scene(s)`); - - if (allExtracted.length === 0) { - return { - success: true, - extractedCount: 0, - storedCount: 0, - records: [], - sceneNames, - lastSceneName: sceneNames[sceneNames.length - 1], - }; - } - - // Limit per session - let extracted = allExtracted; - if (extracted.length > maxMemoriesPerSession) { - logger?.debug?.(`${TAG} Limiting from ${extracted.length} to ${maxMemoriesPerSession} memories per session`); - extracted = extracted.slice(0, maxMemoriesPerSession); - } - - // Assign temporary IDs to extracted memories (needed for batch dedup) - const memoriesWithIds = extracted.map((m) => ({ - ...m, - record_id: generateMemoryId(), - })); - - // Step 2: Batch Conflict Detection + Write - let storedRecords: MemoryRecord[]; - - if (enableDedup) { - try { - const decisions = await batchDedup({ - memories: memoriesWithIds, - config, - logger, - model: options.model, - vectorStore: options.vectorStore, - embeddingService: options.embeddingService, - conflictRecallTopK: options.conflictRecallTopK, - }); - - storedRecords = await applyDecisions({ - memoriesWithIds, - decisions, - baseDir, - sessionKey, - sessionId, - logger, - vectorStore: options.vectorStore, - embeddingService: options.embeddingService, - }); - } catch (err) { - logger?.warn?.(`${TAG} Batch dedup failed, storing all as new: ${err instanceof Error ? err.message : String(err)}`); - storedRecords = await storeAllDirectly(memoriesWithIds, baseDir, sessionKey, sessionId, logger, options.vectorStore, options.embeddingService); - } - } else { - storedRecords = await storeAllDirectly(memoriesWithIds, baseDir, sessionKey, sessionId, logger, options.vectorStore, options.embeddingService); - } - - logger?.info(`${TAG} Extraction complete: extracted=${extracted.length}, stored=${storedRecords.length}`); - - // โ”€โ”€ l1_extraction metric โ”€โ”€ - if (metricInstanceId && logger) { - // Build type distribution of stored memories - const memoriesByType: Record = {}; - for (const r of storedRecords) { - memoriesByType[r.type] = (memoriesByType[r.type] ?? 0) + 1; - } - report("l1_extraction", { - sessionKey, - inputMessageCount: messages.length, - memoriesExtracted: extracted.length, - memoriesStored: storedRecords.length, - memoriesStoredContent: storedRecords.map((r) => ({ - content: r.content, - type: r.type, - scene: r.scene_name ?? null, - })), - memoriesByType, - totalDurationMs: Date.now() - l1StartMs, - success: true, - error: null, - }); - } - - return { - success: true, - extractedCount: extracted.length, - storedCount: storedRecords.length, - records: storedRecords, - sceneNames, - lastSceneName: sceneNames[sceneNames.length - 1], - }; -} - -// ============================ -// LLM call -// ============================ - -/** - * Call LLM to extract scene-segmented memories from conversation messages. - */ -async function callLlmExtraction(params: { - newMessages: ConversationMessage[]; - backgroundMessages: ConversationMessage[]; - previousSceneName?: string; - config: unknown; - logger?: Logger; - model?: string; -}): Promise { - const { newMessages, backgroundMessages, previousSceneName, config, logger, model } = params; - - const runner = new CleanContextRunner({ - config, - modelRef: model, - enableTools: false, - logger, - }); - - const userPrompt = formatExtractionPrompt({ - newMessages, - backgroundMessages, - previousSceneName, - }); - - const result = await runner.run({ - prompt: userPrompt, - systemPrompt: EXTRACT_MEMORIES_SYSTEM_PROMPT, - taskId: "l1-extraction", - timeoutMs: 180_000, - // maxTokens: 4000, - }); - - return parseExtractionResult(result, logger); -} - -/** - * Parse the LLM's JSON response into SceneSegment array. - * Expected format: [{scene_name, message_ids, memories: [...]}] - */ -function parseExtractionResult(raw: string, logger?: Logger): SceneSegment[] { - try { - // Strip markdown code block wrappers if present - let cleaned = raw.trim(); - if (cleaned.startsWith("```")) { - cleaned = cleaned.replace(/^```(?:json)?\s*\n?/, "").replace(/\n?```\s*$/, ""); - } - - // Try to extract JSON array - const arrayMatch = cleaned.match(/\[[\s\S]*\]/); - if (!arrayMatch) { - logger?.warn?.(`${TAG} No JSON array found in extraction response`); - return []; - } - - // Sanitize control characters inside JSON string literals that LLM may produce - const sanitized = sanitizeJsonForParse(arrayMatch[0]); - const parsed = JSON.parse(sanitized) as unknown[]; - - if (!Array.isArray(parsed)) { - logger?.warn?.(`${TAG} Extraction response is not an array`); - return []; - } - - const scenes: SceneSegment[] = []; - for (const item of parsed) { - if (!item || typeof item !== "object") continue; - const s = item as Record; - - scenes.push({ - scene_name: typeof s.scene_name === "string" ? s.scene_name : "ๆœช็Ÿฅๆƒ…ๅขƒ", - message_ids: Array.isArray(s.message_ids) ? s.message_ids.map(String) : [], - memories: Array.isArray(s.memories) - ? (s.memories as Array>) - .filter((m) => m && typeof m === "object" && typeof m.content === "string" && (m.content as string).length > 0) - .map((m) => ({ - content: String(m.content), - type: String(m.type ?? "episodic"), - priority: typeof m.priority === "number" ? m.priority : 50, - source_message_ids: Array.isArray(m.source_message_ids) ? m.source_message_ids.map(String) : [], - metadata: (m.metadata && typeof m.metadata === "object" ? m.metadata : {}) as Record, - })) - : [], - }); - } - - return scenes; - } catch (err) { - logger?.warn?.(`${TAG} Failed to parse extraction result: ${err instanceof Error ? err.message : String(err)}`); - return []; - } -} - -// ============================ -// Write helpers -// ============================ - -/** - * Apply batch dedup decisions โ€” write memories according to their decisions. - */ -async function applyDecisions(params: { - memoriesWithIds: Array; - decisions: DedupDecision[]; - baseDir: string; - sessionKey: string; - sessionId?: string; - logger?: Logger; - vectorStore?: IMemoryStore; - embeddingService?: EmbeddingService; -}): Promise { - const { memoriesWithIds, decisions, baseDir, sessionKey, sessionId, logger, vectorStore, embeddingService } = params; - const storedRecords: MemoryRecord[] = []; - - // Build a map from record_id โ†’ decision - const decisionMap = new Map(); - for (const d of decisions) { - decisionMap.set(d.record_id, d); - } - - for (const memoryWithId of memoriesWithIds) { - const decision = decisionMap.get(memoryWithId.record_id) ?? { - record_id: memoryWithId.record_id, - action: "store" as const, - target_ids: [], - }; - - try { - const record = await writeMemory({ - memory: memoryWithId, - decision, - baseDir, - sessionKey, - sessionId, - logger, - vectorStore, - embeddingService, - }); - - if (record) { - storedRecords.push(record); - } - } catch (err) { - logger?.warn?.( - `${TAG} Write failed for memory "${memoryWithId.content.slice(0, 50)}...": ${err instanceof Error ? err.message : String(err)}`, - ); - } - } - - return storedRecords; -} - -/** - * Store all memories directly (no dedup). - */ -async function storeAllDirectly( - memoriesWithIds: Array, - baseDir: string, - sessionKey: string, - sessionId: string | undefined, - logger?: Logger, - vectorStore?: IMemoryStore, - embeddingService?: EmbeddingService, -): Promise { - const storedRecords: MemoryRecord[] = []; - - for (const memoryWithId of memoriesWithIds) { - try { - const record = await writeMemory({ - memory: memoryWithId, - decision: { - record_id: memoryWithId.record_id, - action: "store", - target_ids: [], - }, - baseDir, - sessionKey, - sessionId, - logger, - vectorStore, - embeddingService, - }); - if (record) { - storedRecords.push(record); - } - } catch (err) { - logger?.warn?.( - `${TAG} Write failed for memory "${memoryWithId.content.slice(0, 50)}...": ${err instanceof Error ? err.message : String(err)}`, - ); - } - } - - return storedRecords; -} - -// ============================ -// Helpers -// ============================ - -const VALID_TYPES: MemoryType[] = ["persona", "episodic", "instruction"]; - -function normalizeType(raw: string): MemoryType | null { - const lower = raw.toLowerCase().trim(); - if (VALID_TYPES.includes(lower as MemoryType)) { - return lower as MemoryType; - } - // Handle legacy type names - if (lower === "episode") return "episodic"; - if (lower === "instruct") return "instruction"; - if (lower === "preference") return "persona"; // fold preference into persona - return null; -} diff --git a/src/record/l1-reader.ts b/src/record/l1-reader.ts deleted file mode 100644 index 66e8d39..0000000 --- a/src/record/l1-reader.ts +++ /dev/null @@ -1,218 +0,0 @@ -/** - * L1 Memory Reader: reads persisted L1 memory records. - * - * Provides two data paths: - * - * 1. **SQLite** (preferred): `queryMemoryRecords()` โ€” uses VectorStore's `queryL1Records()` - * with composite indexes on (session_key, updated_time) and (session_id, updated_time) - * for efficient session-scoped and time-range queries. - * - * 2. **JSONL** (fallback): `readMemoryRecords()` / `readAllMemoryRecords()` โ€” reads from - * `records/YYYY-MM-DD.jsonl` files. Used when VectorStore is unavailable or degraded. - */ - -import fs from "node:fs/promises"; -import path from "node:path"; -import type { MemoryRecord, MemoryType, EpisodicMetadata } from "./l1-writer.js"; -import type { IMemoryStore, L1RecordRow, L1QueryFilter } from "../store/types.js"; - -// Re-export types that readers need -export type { MemoryRecord, MemoryType, EpisodicMetadata } from "./l1-writer.js"; -export type { L1QueryFilter } from "../store/types.js"; - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -const TAG = "[memory-tdai] [l1-reader]"; - -// ============================ -// SQLite-based queries (preferred) -// ============================ - -/** - * Query L1 memory records from SQLite via VectorStore. - * - * This is the **preferred** read path โ€” it uses the composite index - * `idx_l1_session_updated(session_id, updated_time)` for efficient - * session-scoped and time-range queries. - * - * All timestamps are UTC ISO 8601 (as stored by l1-writer's dual-write). - * - * Falls back to empty array if VectorStore is null or degraded. - */ -export async function queryMemoryRecords( - vectorStore: IMemoryStore | null | undefined, - filter?: L1QueryFilter, - logger?: Logger, -): Promise { - if (!vectorStore) { - logger?.warn(`${TAG} queryMemoryRecords: no VectorStore available, returning empty`); - return []; - } - - const rows = await vectorStore.queryL1Records(filter); - return rows.map(rowToMemoryRecord); -} - -/** - * Convert a raw SQLite L1RecordRow to a MemoryRecord (same shape as JSONL records). - */ -function rowToMemoryRecord(row: L1RecordRow): MemoryRecord { - let metadata: EpisodicMetadata | Record = {}; - try { - metadata = JSON.parse(row.metadata_json) as EpisodicMetadata | Record; - } catch { - // malformed JSON โ€” use empty object - } - - // Reconstruct timestamps array from timestamp_start / timestamp_end - const timestamps: string[] = []; - if (row.timestamp_str) timestamps.push(row.timestamp_str); - if (row.timestamp_start && row.timestamp_start !== row.timestamp_str) timestamps.push(row.timestamp_start); - if (row.timestamp_end && row.timestamp_end !== row.timestamp_str && row.timestamp_end !== row.timestamp_start) { - timestamps.push(row.timestamp_end); - } - - return { - id: row.record_id, - content: row.content, - type: row.type as MemoryType, - priority: row.priority, - scene_name: row.scene_name, - source_message_ids: [], // not stored in SQLite (vector search doesn't need them) - metadata, - timestamps, - createdAt: row.created_time, - updatedAt: row.updated_time, - sessionKey: row.session_key, - sessionId: row.session_id, - }; -} - -// ============================ -// JSONL-based reads (fallback) -// ============================ - -/** - * Read all memory records for a session from JSONL files. - * - * Current naming mode: - * - Daily merged file: records/YYYY-MM-DD.jsonl (all sessions in one file) - */ -export async function readMemoryRecords( - sessionKey: string, - baseDir: string, - logger?: Logger, -): Promise { - const recordsDir = path.join(baseDir, "records"); - const dateFilePattern = /^\d{4}-\d{2}-\d{2}\.jsonl$/; - - let entries: import("node:fs").Dirent[]; - try { - entries = await fs.readdir(recordsDir, { withFileTypes: true }); - } catch { - // Directory doesn't exist yet - return []; - } - - const targetFiles = entries - .filter((entry) => entry.isFile() && dateFilePattern.test(entry.name)) - .map((entry) => entry.name) - .sort(); - - if (targetFiles.length === 0) { - return []; - } - - const records: MemoryRecord[] = []; - - for (const fileName of targetFiles) { - const filePath = path.join(recordsDir, fileName); - - let raw: string; - try { - raw = await fs.readFile(filePath, "utf-8"); - } catch { - logger?.warn?.(`${TAG} Failed to read L1 file: ${filePath}`); - continue; - } - - const lines = raw.split("\n").filter((line) => line.trim()); - for (let i = 0; i < lines.length; i++) { - const line = lines[i]; - try { - const parsed = JSON.parse(line) as Partial; - if (parsed.sessionKey !== sessionKey) { - continue; - } - records.push(parsed as MemoryRecord); - } catch { - logger?.warn?.(`${TAG} Skipping malformed JSONL line in ${filePath}:${i + 1}`); - } - } - } - - records.sort((a, b) => { - const ta = a.updatedAt || a.createdAt || ""; - const tb = b.updatedAt || b.createdAt || ""; - return ta.localeCompare(tb); - }); - - return records; -} - -/** - * Read ALL memory records across all session JSONL files. - */ -export async function readAllMemoryRecords( - baseDir: string, - logger?: Logger, -): Promise { - const recordsDir = path.join(baseDir, "records"); - try { - const files = await fs.readdir(recordsDir); - const allRecords: MemoryRecord[] = []; - - for (const file of files) { - if (!file.endsWith(".jsonl")) continue; - const filePath = path.join(recordsDir, file); - try { - const raw = await fs.readFile(filePath, "utf-8"); - const lines = raw.split("\n").filter((line: string) => line.trim()); - for (const line of lines) { - try { - allRecords.push(JSON.parse(line) as MemoryRecord); - } catch { - logger?.warn?.(`${TAG} Skipping malformed JSONL line in ${file}`); - } - } - } catch { - logger?.warn?.(`${TAG} Failed to read ${file}`); - } - } - - allRecords.sort((a, b) => { - const ta = a.updatedAt || a.createdAt || ""; - const tb = b.updatedAt || b.createdAt || ""; - return ta.localeCompare(tb); - }); - - return allRecords; - - } catch { - // records/ directory doesn't exist yet - return []; - } -} - -// ============================ -// Helpers -// ============================ - -function sanitizeFilename(name: string): string { - return name.replace(/[<>:"/\\|?*\x00-\x1f]/g, "_"); -} diff --git a/src/record/l1-writer.ts b/src/record/l1-writer.ts deleted file mode 100644 index b5e618d..0000000 --- a/src/record/l1-writer.ts +++ /dev/null @@ -1,280 +0,0 @@ -/** - * L1 Memory Writer: writes extracted memories to JSONL files. - * - * File naming: records/YYYY-MM-DD.jsonl (daily shards, all sessions merged). - * Each record includes sessionKey for traceability. - * - * Write strategy: - * - JSONL is the append-only persistent store (source of truth for backup/recovery). - * - VectorStore (SQLite) is the primary retrieval engine. - * - On update/merge, old records are deleted from VectorStore in real-time; - * JSONL is append-only and cleaned up periodically by memory-cleaner. - * - * Supports store (append), update, merge, and skip operations. - * - * v3: Aligned with Kenty's prompt output format โ€” 3 memory types (persona/episodic/instruction), - * numeric priority, scene_name, source_message_ids, metadata, timestamps. - */ - -import fs from "node:fs/promises"; -import path from "node:path"; -import crypto from "node:crypto"; -import type { IMemoryStore } from "../store/types.js"; -import type { EmbeddingService } from "../store/embedding.js"; - -// ============================ -// Types -// ============================ - -/** v3: 3 memory types aligned with Kenty's extraction prompt */ -export type MemoryType = "persona" | "episodic" | "instruction"; - -/** Metadata for episodic memories (activity time range) */ -export interface EpisodicMetadata { - activity_start_time?: string; // ISO 8601 - activity_end_time?: string; // ISO 8601 -} - -/** - * A persisted memory record in L1 JSONL files. - * - * v3 changes from v2: - * - `importance: "high"|"medium"|"low"` โ†’ `priority: number` (0-100, -1 for strict global instructions) - * - Added `scene_name`, `source_message_ids`, `metadata`, `timestamps` - * - Removed `keywords` (will be rebuilt from content for search) - * - MemoryType reduced from 4 to 3 (removed "preference", folded into "persona") - */ -export interface MemoryRecord { - /** Unique ID for dedup updates */ - id: string; - /** Memory content */ - content: string; - /** Memory type: persona / episodic / instruction */ - type: MemoryType; - /** Priority score: 0-100 (higher = more important), -1 = strict global instruction */ - priority: number; - /** Scene name this memory belongs to */ - scene_name: string; - /** Source message IDs that contributed to this memory */ - source_message_ids: string[]; - /** Type-specific metadata (e.g., activity_start_time for episodic) */ - metadata: EpisodicMetadata | Record; - /** Timestamp trail: all timestamps related to this memory (for merge history tracking) */ - timestamps: string[]; - /** Creation timestamp (ISO) */ - createdAt: string; - /** Last update timestamp (ISO) */ - updatedAt: string; - /** Source session key (conversation channel identifier) */ - sessionKey: string; - /** Source session ID (single conversation instance identifier) */ - sessionId: string; -} - -/** - * A memory as extracted by LLM (before dedup / persistence). - * Matches the output format of Kenty's extraction prompt. - */ -export interface ExtractedMemory { - content: string; - type: MemoryType; - priority: number; - source_message_ids: string[]; - metadata: EpisodicMetadata | Record; - /** Scene name this memory was extracted in */ - scene_name: string; -} - -export type DedupAction = "store" | "update" | "merge" | "skip"; - -/** - * v3 batch dedup decision โ€” one per new memory, aligned with Kenty's conflict detection prompt. - * - * Key changes: - * - `targetId` โ†’ `target_ids` (array, supports multi-target merge/update) - * - Added `merged_type`, `merged_priority`, `merged_timestamps` for cross-type merge - */ -export interface DedupDecision { - /** Which new memory this decision is about */ - record_id: string; - action: DedupAction; - /** IDs of existing records to replace/remove (for update/merge) */ - target_ids: string[]; - /** Merged/updated content text (for update/merge) */ - merged_content?: string; - /** Best type after merge (for update/merge, may differ from original) */ - merged_type?: MemoryType; - /** Priority after merge (for update/merge) */ - merged_priority?: number; - /** Union of all related timestamps (for update/merge) */ - merged_timestamps?: string[]; -} - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -const TAG = "[memory-tdai][l1-writer]"; - -// ============================ -// Core functions -// ============================ - -/** - * Generate a unique memory ID. - */ -export function generateMemoryId(): string { - return `m_${Date.now()}_${crypto.randomBytes(4).toString("hex")}`; -} - -/** - * Write a memory record according to the dedup decision. - * - * - store: append new record - * - update: remove target records + append updated record - * - merge: remove target records + append merged record - * - skip: do nothing - * - * v3: supports multi-target removal for update/merge. - * v3.1: optional VectorStore + EmbeddingService for dual-write (JSONL + vector). - */ -export async function writeMemory(params: { - memory: ExtractedMemory; - decision: DedupDecision; - baseDir: string; - sessionKey: string; - sessionId?: string; - logger?: Logger; - /** Optional vector store for dual-write (JSONL + vector DB) */ - vectorStore?: IMemoryStore; - /** Optional embedding service (required when vectorStore is provided) */ - embeddingService?: EmbeddingService; -}): Promise { - const { memory, decision, baseDir, sessionKey, sessionId, logger, vectorStore, embeddingService } = params; - - if (decision.action === "skip") { - logger?.debug?.(`${TAG} Skipping memory: ${memory.content.slice(0, 50)}...`); - return null; - } - - const now = new Date().toISOString(); - - // Determine final content, type, priority based on action - let finalContent: string; - let finalType: MemoryType; - let finalPriority: number; - let finalTimestamps: string[]; - - if (decision.action === "merge" || decision.action === "update") { - finalContent = decision.merged_content ?? memory.content; - finalType = decision.merged_type ?? memory.type; - finalPriority = decision.merged_priority ?? memory.priority; - finalTimestamps = decision.merged_timestamps ?? [now]; - } else { - // store - finalContent = memory.content; - finalType = memory.type; - finalPriority = memory.priority; - finalTimestamps = [now]; - } - - const record: MemoryRecord = { - id: decision.record_id || generateMemoryId(), - content: finalContent, - type: finalType, - priority: finalPriority, - scene_name: memory.scene_name, - source_message_ids: memory.source_message_ids, - metadata: memory.metadata, - timestamps: finalTimestamps, - createdAt: now, - updatedAt: now, - sessionKey, - sessionId: sessionId || "", - }; - - const recordsDir = path.join(baseDir, "records"); - await fs.mkdir(recordsDir, { recursive: true }); - - const shardDate = formatLocalDate(new Date()); - const filePath = path.join(recordsDir, `${shardDate}.jsonl`); - - if ((decision.action === "update" || decision.action === "merge") && decision.target_ids.length > 0) { - // Remove target records from VectorStore (real-time deletion for retrieval accuracy). - // JSONL is append-only โ€” old records remain in files and are cleaned up periodically - // by memory-cleaner (which reconciles against VectorStore as source of truth). - if (vectorStore) { - try { - await vectorStore.deleteL1Batch(decision.target_ids); - logger?.debug?.(`${TAG} VectorStore: deleted ${decision.target_ids.length} target record(s) for ${decision.action}`); - } catch (err) { - logger?.warn?.( - `${TAG} VectorStore delete failed for ${decision.action}: ${err instanceof Error ? err.message : String(err)}`, - ); - } - } - await fs.appendFile(filePath, JSON.stringify(record) + "\n", "utf-8"); - logger?.debug?.(`${TAG} ${decision.action} memory: removed [${decision.target_ids.join(",")}] from VectorStore โ†’ ${record.id}: ${finalContent.slice(0, 80)}...`); - } else { - // store: append a new line - await fs.appendFile(filePath, JSON.stringify(record) + "\n", "utf-8"); - logger?.debug?.(`${TAG} Stored memory ${record.id}: ${finalContent.slice(0, 80)}...`); - } - - // === Vector Store dual-write === - if (vectorStore) { - try { - logger?.debug?.( - `${TAG} [vec-dual-write] START id=${record.id}, contentLen=${record.content.length}, ` + - `content="${record.content.slice(0, 80)}..."`, - ); - - let embedding: Float32Array | undefined; - - if (embeddingService) { - try { - embedding = await embeddingService.embed(record.content); - logger?.debug?.( - `${TAG} [vec-dual-write] Embedding OK: dims=${embedding.length}, ` + - `norm=${Math.sqrt(Array.from(embedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}`, - ); - } catch (embedErr) { - // Embedding failed โ€” pass undefined to upsert() which writes - // metadata + FTS only, skipping the vec0 table. - logger?.warn( - `${TAG} [vec-dual-write] Embedding FAILED for id=${record.id}, ` + - `will write metadata only: ${embedErr instanceof Error ? embedErr.message : String(embedErr)}`, - ); - } - } - - const upsertOk = await vectorStore.upsertL1(record, embedding); - logger?.debug?.(`${TAG} [vec-dual-write] upsert result=${upsertOk} id=${record.id}`); - } catch (err) { - // Vector write failure should NOT block the main JSONL write - logger?.warn?.( - `${TAG} [vec-dual-write] FAILED (JSONL already written) id=${record.id}: ${err instanceof Error ? err.message : String(err)}`, - ); - } - } else { - logger?.debug?.( - `${TAG} [vec-dual-write] SKIPPED id=${record.id}: vectorStore=${!!vectorStore}`, - ); - } - - return record; -} - -// ============================ -// Helpers -// ============================ - -function formatLocalDate(d: Date): string { - const y = d.getFullYear(); - const m = String(d.getMonth() + 1).padStart(2, "0"); - const day = String(d.getDate()).padStart(2, "0"); - return `${y}-${m}-${day}`; -} \ No newline at end of file diff --git a/src/report/reporter.ts b/src/report/reporter.ts deleted file mode 100644 index 094f01c..0000000 --- a/src/report/reporter.ts +++ /dev/null @@ -1,103 +0,0 @@ -import { randomUUID } from "node:crypto"; -import fs from "node:fs/promises"; -import path from "node:path"; - -export const REPORT_CONST = { - PLUGIN: "plugin", -} as const; - -export type ReportPayload = Record; - -export interface IReporter { - reportFunc(category: string, payload: ReportPayload): void; -} - -// โ”€โ”€ Singleton โ”€โ”€ - -let _reporter: IReporter | undefined; - -export function initReporter(opts: { - enabled: boolean; - type: string; - logger: { info: (msg: string) => void; debug?: (msg: string) => void }; - instanceId: string; - pluginVersion: string; -}): void { - if (_reporter) return; - if (!opts.enabled) return; - switch (opts.type) { - case "local": - _reporter = new LocalReporter(opts.logger, opts.instanceId, opts.pluginVersion); - break; - // TODO: add new reporter type - default: - opts.logger.debug?.(`[memory-tdai] Unknown reporter type "${opts.type}", disabled reporting`); - break; - } -} - -export function setReporter(reporter: IReporter): void { - _reporter = reporter; -} - -/** - * Reset the reporter singleton so that the next `initReporter` call takes effect. - * Must be called at plugin re-registration (hot-reload) to pick up config changes. - */ -export function resetReporter(): void { - _reporter = undefined; -} - -export function report(event: string, data: ReportPayload): void { - if (!_reporter) return; - try { - _reporter.reportFunc(REPORT_CONST.PLUGIN, { event, ...data }); - } catch { /* never block business logic */ } -} - -// โ”€โ”€ LocalReporter (default) โ”€โ”€ - -class LocalReporter implements IReporter { - constructor( - private readonly logger: { info: (msg: string) => void }, - private readonly instanceId: string, - private readonly pluginVersion: string, - ) {} - - reportFunc(category: string, payload: ReportPayload): void { - try { - this.logger.info(JSON.stringify({ - tag: "METRIC", - category, - plugin: "memory-tdai", - instanceId: this.instanceId, - pluginVersion: this.pluginVersion, - ts: new Date().toISOString(), - ...payload, - })); - } catch { /* swallow */ } - } -} - -// โ”€โ”€ Instance ID (persisted per-install) โ”€โ”€ - -let _instanceIdCache: string | undefined; - -export async function getOrCreateInstanceId(pluginDataDir: string): Promise { - if (_instanceIdCache) return _instanceIdCache; - - const idFile = path.join(pluginDataDir, ".metadata", "instance_id"); - try { - const existing = (await fs.readFile(idFile, "utf-8")).trim(); - if (existing) { - _instanceIdCache = existing; - return existing; - } - } catch { /* file doesn't exist */ } - - const newId = randomUUID(); - await fs.mkdir(path.dirname(idFile), { recursive: true }); - await fs.writeFile(idFile, newId, "utf-8"); - _instanceIdCache = newId; - return newId; -} diff --git a/src/scene/scene-extractor.ts b/src/scene/scene-extractor.ts deleted file mode 100644 index 1d08c6c..0000000 --- a/src/scene/scene-extractor.ts +++ /dev/null @@ -1,431 +0,0 @@ -/** - * SceneExtractor: LLM-driven memory extraction into scene blocks. - * - * Replaces the keyword-based SceneManager.processNewMemories() with an - * LLM agent that autonomously reads/writes scene block files using tools. - * - * Security: The LLM is sandboxed โ€” workspaceDir is set to scene_blocks/ - * so it can ONLY operate on .md scene files. System files (checkpoint, - * scene_index, persona.md) are physically invisible to the LLM. - * - * Flow: - * 1. Backup + load scene index + build summaries - * 2. Assemble extraction prompt with memories + scene context - * 3. Run via CleanContextRunner (tools enabled, sandboxed to scene_blocks/) - * 4. Cleanup: remove soft-deletes, sync index, update navigation - * 5. Parse LLM text output for out-of-band persona update signals - */ - -import fs from "node:fs/promises"; -import path from "node:path"; -import { CleanContextRunner } from "../utils/clean-context-runner.js"; -import { CheckpointManager } from "../utils/checkpoint.js"; -import { BackupManager } from "../utils/backup.js"; -import { readSceneIndex, syncSceneIndex } from "../scene/scene-index.js"; -import type { SceneIndexEntry } from "../scene/scene-index.js"; -import { parseSceneBlock } from "../scene/scene-format.js"; -import { generateSceneNavigation, stripSceneNavigation } from "../scene/scene-navigation.js"; -import { buildSceneExtractionPrompt } from "../prompts/scene-extraction.js"; -import { report } from "../report/reporter.js"; - -const TAG = "[memory-tdai] [extractor]"; - -interface ExtractorLogger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -export interface ExtractionResult { - memoriesProcessed: number; - success: boolean; - error?: string; -} - -export interface SceneExtractorOptions { - dataDir: string; - config: unknown; - model?: string; - maxScenes?: number; - sceneBackupCount?: number; - timeoutMs?: number; - logger?: ExtractorLogger; - /** Plugin instance ID for metric reporting (optional) */ - instanceId?: string; -} - -/** - * Parse LLM text output for a persona update request signal. - * - * Supports multiple formats for robustness: - * - Block: [PERSONA_UPDATE_REQUEST]reason: xxx[/PERSONA_UPDATE_REQUEST] - * - Inline: PERSONA_UPDATE_REQUEST: xxx - */ -export function parsePersonaUpdateSignal(text: string): { reason: string } | null { - // Block format: [PERSONA_UPDATE_REQUEST]...[/PERSONA_UPDATE_REQUEST] - const blockMatch = text.match( - /\[PERSONA_UPDATE_REQUEST\]\s*(?:reason:\s*)?(.+?)\s*\[\/PERSONA_UPDATE_REQUEST\]/s, - ); - if (blockMatch) return { reason: blockMatch[1]!.trim() }; - - // Inline format: PERSONA_UPDATE_REQUEST: reason text - const inlineMatch = text.match( - /PERSONA_UPDATE_REQUEST:\s*(.+?)(?:\n|$)/, - ); - if (inlineMatch) return { reason: inlineMatch[1]!.trim() }; - - return null; -} - -export class SceneExtractor { - private dataDir: string; - private runner: CleanContextRunner; - private maxScenes: number; - private sceneBackupCount: number; - private timeoutMs: number; - private logger: ExtractorLogger | undefined; - private instanceId: string | undefined; - - constructor(opts: SceneExtractorOptions) { - this.dataDir = opts.dataDir; - this.maxScenes = opts.maxScenes ?? 15; - this.sceneBackupCount = opts.sceneBackupCount ?? 10; - this.timeoutMs = opts.timeoutMs ?? 300_000; // 5 min โ€” LLM may do multiple tool calls - this.logger = opts.logger; - this.instanceId = opts.instanceId; - - this.runner = new CleanContextRunner({ - config: opts.config, - modelRef: opts.model, - enableTools: true, - logger: opts.logger, - }); - - this.logger?.debug?.(`${TAG} Created: dataDir=${opts.dataDir}, model=${opts.model ?? "(default)"}, maxScenes=${this.maxScenes}, timeout=${this.timeoutMs}ms`); - } - - /** - * Extract a batch of memories into scene blocks using the LLM agent. - * - * @param memories - Array of raw memory records from the API - * @returns Extraction result with count and success flag - */ - async extract(memories: Array<{ content: string; created_at: string; id?: string }>): Promise { - const extractStartMs = Date.now(); - this.logger?.info(`${TAG} extract() start: ${memories.length} memories`); - - if (memories.length === 0) { - this.logger?.debug?.(`${TAG} extract() skipped: no memories`); - return { memoriesProcessed: 0, success: true }; - } - - const sceneBlocksDir = path.join(this.dataDir, "scene_blocks"); - const metadataDir = path.join(this.dataDir, ".metadata"); - - // Ensure directories exist - await fs.mkdir(sceneBlocksDir, { recursive: true }); - await fs.mkdir(metadataDir, { recursive: true }); - - // Phase 1: Backup - const backupStartMs = Date.now(); - const cpManager = new CheckpointManager(this.dataDir); - const cp = await cpManager.read(); - const bm = new BackupManager(path.join(this.dataDir, ".backup")); - await bm.backupDirectory(sceneBlocksDir, "scene_blocks", `offset${cp.total_processed}`, this.sceneBackupCount); - this.logger?.debug?.(`${TAG} extract() backup phase: ${Date.now() - backupStartMs}ms`); - - // Phase 2: Load scene index - const indexStartMs = Date.now(); - const index = await readSceneIndex(this.dataDir); - this.logger?.debug?.(`${TAG} extract() scene index loaded: ${index.length} entries (${Date.now() - indexStartMs}ms)`); - - // Build scene summaries for the prompt (relative filenames only) - const { summaries: sceneSummaries, filenames: existingSceneFiles } = - this.buildSceneSummaries(index); - - // Build scene count warning (tiered system) - let sceneCountWarning: string | undefined; - const sceneCount = index.length; - if (sceneCount >= this.maxScenes) { - sceneCountWarning = `ๅฝ“ๅ‰ๅœบๆ™ฏๆ•ฐ้‡ไธบ **${sceneCount} ไธช**๏ผŒๅทฒ่พพๅˆฐๆˆ–่ถ…่ฟ‡ ${this.maxScenes} ไธชไธŠ้™๏ผ\n**ไฝ ๅฟ…้กปๅ…ˆๆ‰ง่กŒ MERGE ๆ“ไฝœ**๏ผŒๅฐ†ๆœ€็›ธไผผ็š„ 2-4 ไธชๅœบๆ™ฏๅˆๅนถไธบ 1 ไธช๏ผŒ็„ถๅŽๅ†ๅค„็†ๆ–ฐ่ฎฐๅฟ†ใ€‚\nๅ‚่€ƒๅˆๅนถๅฏน่ฑก๏ผš็ƒญๅบฆๆœ€ไฝŽๆˆ–ไธป้ข˜้ซ˜ๅบฆ้‡ๅ ็š„ๅœบๆ™ฏใ€‚`; - this.logger?.warn(`${TAG} extract() scene count at limit: ${sceneCount}/${this.maxScenes}`); - } else if (sceneCount === this.maxScenes - 1) { - sceneCountWarning = `ๅฝ“ๅ‰ๅœบๆ™ฏๆ•ฐ้‡ไธบ **${sceneCount} ไธช**๏ผŒ่ท็ฆปไธŠ้™ๅชๅทฎ 1 ไธช๏ผ\nๆœฌๆฌกๅค„็†**ๅช่ƒฝ UPDATE ็Žฐๆœ‰ๅœบๆ™ฏ๏ผŒไธ่ƒฝ CREATE ๆ–ฐๅœบๆ™ฏ**ใ€‚`; - this.logger?.warn(`${TAG} extract() scene count near limit (CREATE blocked): ${sceneCount}/${this.maxScenes}`); - } else if (sceneCount >= this.maxScenes - 3) { - sceneCountWarning = `ๅฝ“ๅ‰ๅœบๆ™ฏๆ•ฐ้‡ไธบ **${sceneCount} ไธช**๏ผŒๅปบ่ฎฎไผ˜ๅ…ˆ่€ƒ่™‘ UPDATE ๆˆ–ไธปๅŠจ MERGE ็›ธไผผๅœบๆ™ฏใ€‚`; - this.logger?.debug?.(`${TAG} extract() scene count approaching limit: ${sceneCount}/${this.maxScenes}`); - } - - // Snapshot scene index + content before LLM โ€” used later to diff created/updated/deleted - const preExtractIndex = new Map(index.map((e) => [e.filename, e.summary])); - // Also snapshot scene content so we can detect content-only changes vs metadata-only changes - const preExtractContent = new Map(); - for (const e of index) { - try { - const raw = await fs.readFile(path.join(sceneBlocksDir, e.filename), "utf-8"); - const block = parseSceneBlock(raw, e.filename); - preExtractContent.set(e.filename, block.content); - } catch { /* non-fatal */ } - } - - // Phase 3: Build prompt - const promptStartMs = Date.now(); - const memoriesJson = JSON.stringify( - memories.map((m) => ({ - content: m.content, - created_at: m.created_at, - id: m.id ?? "", - })), - null, - 2, - ); - - const currentTimestamp = formatTimestamp(new Date()); - - const prompt = buildSceneExtractionPrompt({ - memoriesJson, - sceneSummaries: sceneSummaries || "(ๆ— ๅทฒๆœ‰ๅœบๆ™ฏ)", - currentTimestamp, - sceneCountWarning, - existingSceneFiles, - maxScenes: this.maxScenes, - }); - this.logger?.debug?.(`${TAG} extract() prompt built: ${prompt.length} chars (${Date.now() - promptStartMs}ms)`); - - // Phase 4: Run LLM agent (sandboxed to scene_blocks/) - let llmOutput = ""; - let llmDurationMs = 0; - try { - this.logger?.debug?.(`${TAG} extract() starting LLM runner (timeout=${this.timeoutMs}ms, maxTokens=model default)...`); - const runnerStartMs = Date.now(); - llmOutput = await this.runner.run({ - prompt, - taskId: `scene-extract-${Date.now()}`, - timeoutMs: this.timeoutMs, - // maxTokens omitted โ†’ core uses the resolved model's maxTokens from catalog - workspaceDir: sceneBlocksDir, - }) ?? ""; - llmDurationMs = Date.now() - runnerStartMs; - this.logger?.debug?.(`${TAG} extract() LLM runner completed: ${llmDurationMs}ms`); - } catch (err) { - const errMsg = err instanceof Error ? err.message : String(err); - const totalMs = Date.now() - extractStartMs; - this.logger?.error(`${TAG} extract() LLM runner failed after ${totalMs}ms: ${errMsg}`); - return { memoriesProcessed: 0, success: false, error: errMsg }; - } - - // Phase 5: Subsequent processing โ€” safe cleanup of soft-deleted files - // - // Security: The LLM has no `exec` tool and cannot run shell commands. - // Instead, it "deletes" files by writing the marker `[DELETED]` to the file - // (writing empty/whitespace-only content is rejected by core's write tool - // parameter validation). Here we detect and remove those soft-deleted files - // before syncing the index, so syncSceneIndex won't re-index stale entries. - // - // We also detect "META-only" files โ€” files that contain only a META header - // (e.g. [ARCHIVE] or [CONSOLIDATED] markers) but no actual scene content. - // These are artifacts of LLM merges that didn't properly delete old files. - const cleanupStartMs = Date.now(); - let cleanedCount = 0; - try { - const allFiles = (await fs.readdir(sceneBlocksDir)).filter((f) => f.endsWith(".md")); - for (const file of allFiles) { - const filePath = path.join(sceneBlocksDir, file); - const raw = await fs.readFile(filePath, "utf-8"); - if (raw.trim().length === 0 || raw.trim() === "[DELETED]") { - // Empty file or [DELETED] marker โ€” soft-delete - await fs.unlink(filePath); - cleanedCount++; - this.logger?.debug?.(`${TAG} extract() removed soft-deleted file: ${file}`); - } else { - // Check if file has only META header but no actual content - const block = parseSceneBlock(raw, file); - if (!block.content || block.content.trim().length === 0) { - await fs.unlink(filePath); - cleanedCount++; - this.logger?.debug?.(`${TAG} extract() removed META-only file (no content): ${file}`); - } - } - } - } catch (cleanupErr) { - // Non-fatal โ€” log and continue to index sync - this.logger?.warn(`${TAG} extract() soft-delete cleanup error: ${cleanupErr instanceof Error ? cleanupErr.message : String(cleanupErr)}`); - } - this.logger?.debug?.(`${TAG} extract() soft-delete cleanup: removed ${cleanedCount} empty files (${Date.now() - cleanupStartMs}ms)`); - - // Phase 6: Sync scene index (rebuilds from remaining non-empty files) - const syncStartMs = Date.now(); - await syncSceneIndex(this.dataDir); - this.logger?.debug?.(`${TAG} extract() scene index synced: ${Date.now() - syncStartMs}ms`); - - // Phase 7: Update persona.md navigation (GAP-4 fix) - const navStartMs = Date.now(); - try { - await this.updateSceneNavigation(); - this.logger?.debug?.(`${TAG} extract() persona.md navigation updated: ${Date.now() - navStartMs}ms`); - } catch (navErr) { - // Non-fatal โ€” log and continue - this.logger?.warn(`${TAG} extract() failed to update persona navigation: ${navErr instanceof Error ? navErr.message : String(navErr)}`); - } - - // Phase 8: Parse LLM output for out-of-band persona update signal - if (llmOutput) { - const signal = parsePersonaUpdateSignal(llmOutput); - if (signal) { - await cpManager.setPersonaUpdateRequest(signal.reason); - this.logger?.debug?.(`${TAG} extract() persona update requested by LLM: ${signal.reason}`); - } - } - - const totalMs = Date.now() - extractStartMs; - this.logger?.info(`${TAG} extract() completed: ${memories.length} memories processed in ${totalMs}ms`); - - // โ”€โ”€ l2_extraction metric โ”€โ”€ - if (this.instanceId && this.logger) { - // Read updated scene index to report final state + diff against pre-extract snapshot - let resultScenes: Array<{ title: string; summary: string; content: string; status: "created" | "updated" }> = []; - let scenesCreated = 0; - let scenesUpdated = 0; - let scenesDeleted = 0; - try { - const finalIndex = await readSceneIndex(this.dataDir); - const postFilenames = new Set(); - for (const e of finalIndex) { - postFilenames.add(e.filename); - const oldSummary = preExtractIndex.get(e.filename); - // Read scene block content from disk - let content = ""; - try { - const blockPath = path.join(sceneBlocksDir, e.filename); - const raw = await fs.readFile(blockPath, "utf-8"); - const block = parseSceneBlock(raw, e.filename); - content = block.content; - } catch { /* file read failure is non-fatal */ } - - if (oldSummary === undefined) { - // New scene - scenesCreated++; - resultScenes.push({ - title: e.filename.replace(/\.md$/, ""), - summary: e.summary, - content, - status: "created", - }); - } else { - // Existing scene โ€” check if content actually changed (not just metadata) - const oldContent = preExtractContent.get(e.filename) ?? ""; - if (content !== oldContent) { - scenesUpdated++; - resultScenes.push({ - title: e.filename.replace(/\.md$/, ""), - summary: e.summary, - content, - status: "updated", - }); - } - // If only metadata (summary/heat) changed but content is the same, skip - } - } - // Scenes in pre-extract but missing from post-extract = deleted - for (const [filename] of preExtractIndex) { - if (!postFilenames.has(filename)) { - scenesDeleted++; - } - } - } catch { /* non-fatal */ } - - report("l2_extraction", { - inputMemoryCount: memories.length, - resultSceneCount: resultScenes.length, - resultScenes, - scenesCreated, - scenesUpdated, - scenesDeleted, - llmDurationMs, - totalDurationMs: totalMs, - success: true, - error: null, - }); - } - - return { memoriesProcessed: memories.length, success: true }; - } - - /** - * Build human-readable scene summaries for the prompt, - * and collect the list of existing scene filenames (relative). - * - * Includes a capacity counter at the top (e.g. "ๅฝ“ๅ‰ๅœบๆ™ฏๆ€ปๆ•ฐ๏ผš5 / 15") - * so the LLM can immediately see how close it is to the limit. - */ - private buildSceneSummaries( - index: SceneIndexEntry[], - ): { summaries: string; filenames: string[] } { - if (index.length === 0) return { summaries: "", filenames: [] }; - - const lines: string[] = []; - const filenames: string[] = []; - - // Inject capacity counter at the top โ€” LLM sees this first - lines.push(`**ๅฝ“ๅ‰ๅœบๆ™ฏๆ€ปๆ•ฐ๏ผš${index.length} / ${this.maxScenes}**`); - lines.push(""); - - for (const entry of index) { - filenames.push(entry.filename); - lines.push(`### ${entry.filename}`); - lines.push(`**็ƒญๅบฆ**: ${entry.heat} | **ๆ›ดๆ–ฐ**: ${entry.updated}`); - lines.push(`**summary**: ${entry.summary}`); - lines.push(""); - } - return { summaries: lines.join("\n"), filenames }; - } - - /** - * Update the scene navigation section at the end of persona.md. - * - * Reads the current scene index, generates the navigation block, then - * strips any existing navigation from persona.md and appends the new one. - * - * IMPORTANT: If the persona body is empty (PersonaGenerator hasn't run yet), - * we skip writing to avoid creating a persona.md that only contains the - * scene navigation. PersonaGenerator.generate() will write the full - * persona + navigation when it runs. - */ - private async updateSceneNavigation(): Promise { - const personaPath = path.join(this.dataDir, "persona.md"); - const index = await readSceneIndex(this.dataDir); - const nav = generateSceneNavigation(index); - - let existing = ""; - try { - existing = await fs.readFile(personaPath, "utf-8"); - } catch { - // No persona file yet โ€” PersonaGenerator will create it with navigation. - // Don't write a navigation-only file. - this.logger?.debug?.(`${TAG} updateSceneNavigation() skipped: no persona file yet, waiting for PersonaGenerator`); - return; - } - - if (!existing.trim() && !nav) return; - - const stripped = stripSceneNavigation(existing).trimEnd(); - - // If the persona body is empty (only navigation existed), don't overwrite - // with a navigation-only file. Let PersonaGenerator handle full generation. - if (!stripped) { - this.logger?.debug?.(`${TAG} updateSceneNavigation() skipped: persona body is empty, waiting for PersonaGenerator`); - return; - } - - const updated = nav ? `${stripped}\n\n${nav}\n` : `${stripped}\n`; - - // persona.md is at dataDir root, no subdir needed - await fs.writeFile(personaPath, updated, "utf-8"); - } -} - -function formatTimestamp(d: Date): string { - return d.toISOString(); -} diff --git a/src/scene/scene-format.ts b/src/scene/scene-format.ts deleted file mode 100644 index cb12628..0000000 --- a/src/scene/scene-format.ts +++ /dev/null @@ -1,75 +0,0 @@ -/** - * Scene Block file format: parse and format the META-delimited Markdown files. - */ - -export interface SceneBlockMeta { - created: string; - updated: string; - summary: string; - heat: number; -} - -export interface SceneBlock { - filename: string; - meta: SceneBlockMeta; - content: string; -} - -const META_START = "-----META-START-----"; -const META_END = "-----META-END-----"; - -/** - * Parse a Scene Block file into structured data. - */ -export function parseSceneBlock(raw: string, filename: string): SceneBlock { - const startIdx = raw.indexOf(META_START); - const endIdx = raw.indexOf(META_END); - - if (startIdx === -1 || endIdx === -1) { - // No META section โ€” treat entire file as content - return { - filename, - meta: { created: "", updated: "", summary: "", heat: 0 }, - content: raw.trim(), - }; - } - - const metaBlock = raw.slice(startIdx + META_START.length, endIdx).trim(); - const content = raw.slice(endIdx + META_END.length).trim(); - - const meta: SceneBlockMeta = { - created: extractMetaField(metaBlock, "created"), - updated: extractMetaField(metaBlock, "updated"), - summary: extractMetaField(metaBlock, "summary"), - heat: parseInt(extractMetaField(metaBlock, "heat"), 10) || 0, - }; - - return { filename, meta, content }; -} - -/** - * Format a Scene Block back into file content. - */ -export function formatSceneBlock(meta: SceneBlockMeta, content: string): string { - return `${formatMeta(meta)}\n\n${content}`; -} - -/** - * Format the META section. - */ -export function formatMeta(meta: SceneBlockMeta): string { - return [ - META_START, - `created: ${meta.created}`, - `updated: ${meta.updated}`, - `summary: ${meta.summary}`, - `heat: ${meta.heat}`, - META_END, - ].join("\n"); -} - -function extractMetaField(metaBlock: string, field: string): string { - const re = new RegExp(`^${field}:\\s*(.*)$`, "m"); - const m = metaBlock.match(re); - return m ? m[1]!.trim() : ""; -} diff --git a/src/scene/scene-index.ts b/src/scene/scene-index.ts deleted file mode 100644 index 84f7d47..0000000 --- a/src/scene/scene-index.ts +++ /dev/null @@ -1,96 +0,0 @@ -/** - * Scene Index: maintains a JSON index of all scene blocks for quick lookup. - */ - -import fs from "node:fs/promises"; -import path from "node:path"; -import { parseSceneBlock } from "./scene-format.js"; - -export interface SceneIndexEntry { - filename: string; - summary: string; - heat: number; - created: string; - updated: string; -} - -/** - * Read the scene index from disk. - * - * The index is written exclusively by syncSceneIndex() (engineering side). - * The LLM is sandboxed to scene_blocks/ and cannot access this file. - */ -export async function readSceneIndex(dataDir: string): Promise { - const indexPath = path.join(dataDir, ".metadata", "scene_index.json"); - try { - const raw = await fs.readFile(indexPath, "utf-8"); - const parsed = JSON.parse(raw) as Array>; - if (!Array.isArray(parsed)) return []; - - const entries: SceneIndexEntry[] = []; - for (const item of parsed) { - if (!item || typeof item !== "object") continue; - - const filename = typeof item.filename === "string" ? item.filename : ""; - if (!filename) continue; - - entries.push({ - filename, - summary: typeof item.summary === "string" ? item.summary : "", - heat: typeof item.heat === "number" ? item.heat : 0, - created: typeof item.created === "string" ? item.created : "", - updated: typeof item.updated === "string" ? item.updated : "", - }); - } - return entries; - } catch { - return []; - } -} - -/** - * Write the scene index to disk. - */ -export async function writeSceneIndex( - dataDir: string, - entries: SceneIndexEntry[], -): Promise { - const indexPath = path.join(dataDir, ".metadata", "scene_index.json"); - await fs.mkdir(path.dirname(indexPath), { recursive: true }); - await fs.writeFile(indexPath, JSON.stringify(entries, null, 2), "utf-8"); -} - -/** - * Rebuild scene index by scanning all .md files in the scene_blocks directory. - */ -export async function syncSceneIndex(dataDir: string): Promise { - const blocksDir = path.join(dataDir, "scene_blocks"); - let files: string[]; - try { - files = (await fs.readdir(blocksDir)).filter((f) => f.endsWith(".md")); - } catch { - files = []; - } - - const entries: SceneIndexEntry[] = []; - for (const file of files) { - try { - const raw = await fs.readFile(path.join(blocksDir, file), "utf-8"); - const block = parseSceneBlock(raw, file); - entries.push({ - filename: file, - summary: block.meta.summary, - heat: block.meta.heat, - created: block.meta.created, - updated: block.meta.updated, - }); - } catch { - // File may have been deleted between readdir and readFile (e.g. by concurrent - // SceneExtractor soft-delete). Skip it and continue syncing the rest. - continue; - } - } - - await writeSceneIndex(dataDir, entries); - return entries; -} diff --git a/src/scene/scene-navigation.ts b/src/scene/scene-navigation.ts deleted file mode 100644 index 616b23f..0000000 --- a/src/scene/scene-navigation.ts +++ /dev/null @@ -1,57 +0,0 @@ -/** - * Scene navigation: generates a summary navigation section appended to persona.md. - * - * The navigation includes file paths so the agent can use read_file to load - * scene details on demand (progressive disclosure). - */ - -import type { SceneIndexEntry } from "./scene-index.js"; - -const NAV_HEADER = "---\n## ๐Ÿ—บ๏ธ Scene Navigation (Scene Index)"; - -const NAV_FOOTER = `๐Ÿ“Œ ไฝฟ็”จ่ฏดๆ˜Ž๏ผš -- Path ๅณ scene block ็š„็›ธๅฏน่ทฏๅพ„๏ผŒๅฏๆŒ‰้œ€ไฝฟ็”จ read_file ่ฏปๅ–ๅฎŒๆ•ดๅ†…ๅฎน -- ็ƒญๅบฆ๏ผš่ฏฅๅœบๆ™ฏ่ขซ่ฎฐๅฟ†ๅ‘ฝไธญ็š„็ดฏ่ฎกๆฌกๆ•ฐ๏ผŒ่ถŠ้ซ˜่ถŠ้‡่ฆ -- Summary๏ผšๅœบๆ™ฏ็š„ๆ ธๅฟƒ่ฆ็‚นๆ‘˜่ฆ`; - -/** - * Build a fire-emoji string based on heat value (visual priority cue for the agent). - */ -function heatEmoji(heat: number): string { - if (heat >= 1000) return " ๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅ"; - if (heat >= 500) return " ๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅ"; - if (heat >= 200) return " ๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅ"; - if (heat >= 100) return " ๐Ÿ”ฅ๐Ÿ”ฅ"; - if (heat >= 50) return " ๐Ÿ”ฅ"; - return ""; -} - -/** - * Generate the scene navigation Markdown section. - * - * Output format mirrors the v1 Python version so the agent can identify file - * paths and invoke read_file for on-demand scene loading. - */ -export function generateSceneNavigation(entries: SceneIndexEntry[]): string { - if (entries.length === 0) return ""; - - const sorted = [...entries].sort((a, b) => b.heat - a.heat); - - const blocks = sorted.map((e) => { - const pathLine = `### Path: scene_blocks/${e.filename}`; - const heatLine = `**็ƒญๅบฆ**: ${e.heat}${heatEmoji(e.heat)}${e.updated ? ` | **ๆ›ดๆ–ฐ**: ${e.updated}` : ""}`; - const summaryLine = `Summary: ${e.summary}`; - return `${pathLine}\n${heatLine}\n${summaryLine}`; - }); - - return `${NAV_HEADER}\n*ไปฅไธ‹ๆ˜ฏๅฝ“ๅ‰ๅœบๆ™ฏ่ฎฐๅฟ†็š„็ดขๅผ•๏ผŒๅฏๆ นๆฎ้œ€่ฆ read_file ่ฏปๅ–่ฏฆ็ป†ๅ†…ๅฎนใ€‚*\n\n${blocks.join("\n\n")}\n\n${NAV_FOOTER}`; -} - -/** - * Strip the scene navigation section from persona content. - */ -export function stripSceneNavigation(personaContent: string): string { - const idx = personaContent.indexOf(NAV_HEADER); - if (idx === -1) return personaContent; - return personaContent.slice(0, idx).trimEnd(); -} diff --git a/src/seed/input.ts b/src/seed/input.ts deleted file mode 100644 index e2aa742..0000000 --- a/src/seed/input.ts +++ /dev/null @@ -1,435 +0,0 @@ -/** - * Input loading, validation, normalization, and timestamp handling for the `seed` command. - * - * Responsibilities: - * 1. Load raw JSON from file - * 2. Detect Format A (`{ sessions: [...] }`) vs Format B (`[...]`) - * 3. Six-layer validation (file โ†’ top-level โ†’ session โ†’ round โ†’ message โ†’ timestamp consistency) - * 4. Normalize into NormalizedInput with auto-generated sessionIds - * 5. Timestamp all-or-none check + fill strategy - */ - -import fs from "node:fs"; -import crypto from "node:crypto"; -import type { - RawSession, - FormatA, - ValidationError, - NormalizedInput, - NormalizedSession, - NormalizedRound, - NormalizedMessage, - SeedCommandOptions, -} from "./types.js"; - -// ============================ -// Public API -// ============================ - -export interface LoadAndValidateResult { - /** Normalized input ready for pipeline consumption. */ - input: NormalizedInput; - /** Whether the user needs to confirm timestamp auto-fill. */ - needsTimestampConfirmation: boolean; -} - -/** - * Load, validate, and normalize seed input from a file. - * - * Throws on fatal validation errors with a human-readable message - * that includes all collected errors. - */ -export function loadAndValidateInput( - opts: Pick, -): LoadAndValidateResult { - // Layer 1: File โ€” read + parse - const raw = loadRawInput(opts.input); - - // Layer 2: Top-level โ€” detect A vs B - const sessions = extractSessions(raw); - - // Layers 3-5: session / round / message validation - const errors: ValidationError[] = []; - validateSessions(sessions, opts.strictRoundRole, errors); - - if (errors.length > 0) { - throw new SeedValidationError(errors); - } - - // Layer 6: Timestamp consistency (all-have / all-missing / mixed โ†’ error) - const tsResult = checkTimestampConsistency(sessions); - if (tsResult.status === "mixed") { - throw new SeedValidationError([{ - stage: "timestamp_consistency", - message: - "Timestamp consistency check failed: some messages have timestamps while others do not. " + - "All messages must either have timestamps or none must have timestamps.", - }]); - } - - // Normalize - const normalized = normalizeSessions(sessions, opts.sessionKey); - - return { - input: { - sessions: normalized.sessions, - totalRounds: normalized.totalRounds, - totalMessages: normalized.totalMessages, - hasTimestamps: tsResult.status === "all_present", - }, - needsTimestampConfirmation: tsResult.status === "all_missing", - }; -} - -/** - * Fill timestamps for all messages when the input has no timestamps. - * - * Uses a single monotonically increasing counter across ALL sessions - * to guarantee global timestamp ordering. This is critical when multiple - * sessions share the same sessionKey โ€” the L0 capture cursor (advanced - * per-session) would filter out later sessions whose timestamps fall - * below the cursor if ordering were not globally monotonic. - */ -export function fillTimestamps(input: NormalizedInput): void { - let currentTs = Date.now(); - for (const session of input.sessions) { - for (const round of session.rounds) { - for (let i = 0; i < round.messages.length; i++) { - // Small offset per message to maintain strict ordering - round.messages[i]!.timestamp = currentTs; - currentTs += 100; - } - } - } - input.hasTimestamps = true; -} - -// ============================ -// Validation error class -// ============================ - -export class SeedValidationError extends Error { - public readonly errors: ValidationError[]; - - constructor(errors: ValidationError[]) { - const summary = errors.map((e) => formatValidationError(e)).join("\n"); - super(`Seed input validation failed (${errors.length} error(s)):\n${summary}`); - this.name = "SeedValidationError"; - this.errors = errors; - } -} - -function formatValidationError(e: ValidationError): string { - const parts: string[] = [` [${e.stage}]`]; - if (e.sourceIndex != null) parts.push(`session[${e.sourceIndex}]`); - if (e.sessionKey) parts.push(`key="${e.sessionKey}"`); - if (e.roundIndex != null) parts.push(`round[${e.roundIndex}]`); - if (e.messageIndex != null) parts.push(`msg[${e.messageIndex}]`); - parts.push(e.message); - return parts.join(" "); -} - -// ============================ -// Layer 1: File loading -// ============================ - -function loadRawInput(filePath: string): unknown { - if (!fs.existsSync(filePath)) { - throw new SeedValidationError([{ - stage: "file", - message: `Input file not found: ${filePath}`, - }]); - } - - const content = fs.readFileSync(filePath, "utf-8").trim(); - if (!content) { - throw new SeedValidationError([{ - stage: "file", - message: "Input file is empty.", - }]); - } - - try { - return JSON.parse(content); - } catch (err) { - throw new SeedValidationError([{ - stage: "file", - message: `JSON parse error: ${err instanceof Error ? err.message : String(err)}`, - }]); - } -} - -// ============================ -// Layer 2: Top-level format detection -// ============================ - -function extractSessions(raw: unknown): RawSession[] { - // Format A: { sessions: [...] } - if ( - raw != null && - typeof raw === "object" && - !Array.isArray(raw) && - "sessions" in raw - ) { - const obj = raw as FormatA; - if (!Array.isArray(obj.sessions)) { - throw new SeedValidationError([{ - stage: "top_level", - message: 'Format A detected but "sessions" is not an array.', - }]); - } - return obj.sessions; - } - - // Format B: [...] - if (Array.isArray(raw)) { - return raw as RawSession[]; - } - - throw new SeedValidationError([{ - stage: "top_level", - message: - "Unrecognized input format. Expected either:\n" + - ' Format A: { "sessions": [...] }\n' + - " Format B: [ { sessionKey, conversations }, ... ]", - }]); -} - -// ============================ -// Layers 3-5: session / round / message validation -// ============================ - -function validateSessions( - sessions: RawSession[], - strictRoundRole: boolean, - errors: ValidationError[], -): void { - if (sessions.length === 0) { - errors.push({ - stage: "session", - message: "No sessions found in input.", - }); - return; - } - - for (let si = 0; si < sessions.length; si++) { - const session = sessions[si]!; - - // Layer 3: session validation - if (!session.sessionKey || typeof session.sessionKey !== "string" || session.sessionKey.trim() === "") { - errors.push({ - stage: "session", - sourceIndex: si, - message: '"sessionKey" is required and must be a non-empty string.', - }); - } - - if (!Array.isArray(session.conversations)) { - errors.push({ - stage: "session", - sourceIndex: si, - sessionKey: session.sessionKey, - message: '"conversations" must be a two-dimensional array (array of rounds).', - }); - continue; // Can't validate rounds - } - - // Check that conversations is a 2D array - for (let ri = 0; ri < session.conversations.length; ri++) { - const round = session.conversations[ri]; - - // Layer 4: round validation - if (!Array.isArray(round)) { - errors.push({ - stage: "round", - sourceIndex: si, - sessionKey: session.sessionKey, - roundIndex: ri, - message: "Round must be an array of messages.", - }); - continue; - } - - if (round.length === 0) { - errors.push({ - stage: "round", - sourceIndex: si, - sessionKey: session.sessionKey, - roundIndex: ri, - message: "Round must be a non-empty array.", - }); - continue; - } - - // Strict round-role: each round must have at least one user and one assistant - if (strictRoundRole) { - const roles = new Set(round.map((m) => m.role)); - if (!roles.has("user")) { - errors.push({ - stage: "round", - sourceIndex: si, - sessionKey: session.sessionKey, - roundIndex: ri, - message: '--strict-round-role: round must contain at least one "user" message.', - }); - } - if (!roles.has("assistant")) { - errors.push({ - stage: "round", - sourceIndex: si, - sessionKey: session.sessionKey, - roundIndex: ri, - message: '--strict-round-role: round must contain at least one "assistant" message.', - }); - } - } - - // Layer 5: message validation - for (let mi = 0; mi < round.length; mi++) { - const msg = round[mi]!; - - if (!msg.role || typeof msg.role !== "string") { - errors.push({ - stage: "message", - sourceIndex: si, - sessionKey: session.sessionKey, - roundIndex: ri, - messageIndex: mi, - message: '"role" is required and must be a non-empty string.', - }); - } - - if (!msg.content || typeof msg.content !== "string" || msg.content.trim() === "") { - errors.push({ - stage: "message", - sourceIndex: si, - sessionKey: session.sessionKey, - roundIndex: ri, - messageIndex: mi, - message: '"content" is required and must be a non-empty string.', - }); - } - - if (msg.timestamp !== undefined) { - if (typeof msg.timestamp === "number") { - if (!Number.isInteger(msg.timestamp)) { - errors.push({ - stage: "message", - sourceIndex: si, - sessionKey: session.sessionKey, - roundIndex: ri, - messageIndex: mi, - message: '"timestamp" must be an integer (epoch milliseconds). Negative values are allowed for dates before 1970.', - }); - } - } else if (typeof msg.timestamp === "string") { - if (Number.isNaN(new Date(msg.timestamp).getTime())) { - errors.push({ - stage: "message", - sourceIndex: si, - sessionKey: session.sessionKey, - roundIndex: ri, - messageIndex: mi, - message: `"timestamp" string is not a valid ISO 8601 date: "${msg.timestamp}".`, - }); - } - } else { - errors.push({ - stage: "message", - sourceIndex: si, - sessionKey: session.sessionKey, - roundIndex: ri, - messageIndex: mi, - message: '"timestamp" must be a number (epoch ms) or an ISO 8601 string.', - }); - } - } - } - } - } -} - -// ============================ -// Layer 6: Timestamp consistency -// ============================ - -interface TimestampCheckResult { - status: "all_present" | "all_missing" | "mixed"; -} - -function checkTimestampConsistency(sessions: RawSession[]): TimestampCheckResult { - let hasTs = false; - let missingTs = false; - - for (const session of sessions) { - if (!Array.isArray(session.conversations)) continue; - for (const round of session.conversations) { - if (!Array.isArray(round)) continue; - for (const msg of round) { - if (msg.timestamp !== undefined && msg.timestamp !== null) { - hasTs = true; - } else { - missingTs = true; - } - // Early exit on mixed - if (hasTs && missingTs) { - return { status: "mixed" }; - } - } - } - } - - if (hasTs && !missingTs) return { status: "all_present" }; - if (!hasTs && missingTs) return { status: "all_missing" }; - // No messages at all โ€” treat as all_missing (will be caught by session validation) - return { status: "all_missing" }; -} - -// ============================ -// Normalization -// ============================ - -function normalizeSessions( - sessions: RawSession[], - fallbackSessionKey?: string, -): { sessions: NormalizedSession[]; totalRounds: number; totalMessages: number } { - const normalized: NormalizedSession[] = []; - let totalRounds = 0; - let totalMessages = 0; - - for (let si = 0; si < sessions.length; si++) { - const raw = sessions[si]!; - - const sessionKey = raw.sessionKey || fallbackSessionKey || "seed-user"; - const sessionId = raw.sessionId || crypto.randomUUID(); - - const rounds: NormalizedRound[] = []; - for (const rawRound of raw.conversations) { - if (!Array.isArray(rawRound)) continue; - - const messages: NormalizedMessage[] = rawRound.map((msg) => ({ - role: msg.role, - content: msg.content, - // Normalize timestamp: ISO string โ†’ epoch ms, number โ†’ pass-through, missing โ†’ 0 (filled later) - timestamp: msg.timestamp == null - ? 0 - : typeof msg.timestamp === "string" - ? new Date(msg.timestamp).getTime() - : msg.timestamp, - })); - - rounds.push({ messages }); - totalMessages += messages.length; - } - - totalRounds += rounds.length; - normalized.push({ - sessionKey, - sessionId, - rounds, - sourceIndex: si, - }); - } - - return { sessions: normalized, totalRounds, totalMessages }; -} diff --git a/src/seed/seed-runtime.ts b/src/seed/seed-runtime.ts deleted file mode 100644 index c53fae7..0000000 --- a/src/seed/seed-runtime.ts +++ /dev/null @@ -1,394 +0,0 @@ -/** - * Seed runtime: L0โ†’L1โ†’L2โ†’L3 orchestration for the `seed` command. - * - * Uses the shared pipeline-factory for VectorStore/EmbeddingService init, - * L1 runner, L2 runner, L3 runner, and persister wiring โ€” keeping this - * module focused on seed-specific concerns: - * - Synchronous per-round L0 capture with progress reporting - * - waitForL1Idle polling (L1 only โ€” see FIXME below) - * - Ctrl+C graceful shutdown - * - * FIXME: Currently we only wait for L1 to become idle before destroying the - * pipeline. L2 (scene extraction) and L3 (persona generation) may still be - * in-flight when `pipeline.destroy()` is called. This is intentional for now - * to avoid excessively long seed runs, but means seed output may not include - * the latest L2/L3 artifacts. Re-evaluate adding a full L1+L2+L3 idle wait - * once pipeline-manager exposes reliable L2/L3 idle signals. - */ - -import path from "node:path"; -import { parseConfig } from "../config.js"; -import type { MemoryTdaiConfig } from "../config.js"; -import { performAutoCapture } from "../hooks/auto-capture.js"; -import { createPipeline, createL2Runner, createL3Runner } from "../utils/pipeline-factory.js"; -import type { PipelineInstance, PipelineLogger } from "../utils/pipeline-factory.js"; -import { readManifest, writeManifest } from "../utils/manifest.js"; -import type { MemoryPipelineManager } from "../utils/pipeline-manager.js"; -import type { - NormalizedInput, - SeedProgress, - SeedSummary, -} from "./types.js"; - -const TAG = "[memory-tdai] [seed]"; - -// ============================ -// Seed pipeline options -// ============================ - -export interface SeedRuntimeOptions { - /** Directory to store all seed output (L0, checkpoint, vectors.db). */ - outputDir: string; - /** OpenClaw config object (needed for LLM calls in L1). */ - openclawConfig: unknown; - /** Raw plugin config (same shape as api.pluginConfig). */ - pluginConfig?: Record; - /** Original input file path (for manifest traceability). */ - inputFile?: string; - /** Logger instance. */ - logger: PipelineLogger; - /** Progress callback (called after each round). */ - onProgress?: (progress: SeedProgress) => void; -} - -// ============================ -// Seed pipeline creation -// ============================ - -/** - * Create a seed pipeline using the shared factory, with L2/L3 runners - * wired via shared factory functions (same logic as index.ts live runtime). - */ -async function createSeedPipeline(opts: SeedRuntimeOptions): Promise<{ pipeline: PipelineInstance; cfg: MemoryTdaiConfig }> { - const { outputDir, openclawConfig, pluginConfig, logger } = opts; - - // Parse config โ€” all values come from pluginConfig (or parseConfig defaults) - const cfg = parseConfig(pluginConfig); - - logger.info( - `${TAG} Creating seed pipeline: outputDir=${outputDir}, ` + - `everyN=${cfg.pipeline.everyNConversations}, l1Idle=${cfg.pipeline.l1IdleTimeoutSeconds}s, ` + - `l2Delay=${cfg.pipeline.l2DelayAfterL1Seconds}s, l2Min=${cfg.pipeline.l2MinIntervalSeconds}s, l2Max=${cfg.pipeline.l2MaxIntervalSeconds}s`, - ); - - // Use shared factory for everything: store init, L1 runner, persister, destroy - const pipeline = await createPipeline({ - pluginDataDir: outputDir, - cfg, - openclawConfig, - logger, - }); - - // Wire L2 runner via shared factory (same logic as index.ts live runtime) - pipeline.scheduler.setL2Runner(createL2Runner({ - pluginDataDir: outputDir, - cfg, - openclawConfig, - vectorStore: pipeline.vectorStore, - logger, - })); - - // Wire L3 runner via shared factory (same logic as index.ts live runtime) - pipeline.scheduler.setL3Runner(createL3Runner({ - pluginDataDir: outputDir, - cfg, - openclawConfig, - vectorStore: pipeline.vectorStore, - logger, - })); - - return { pipeline, cfg }; -} - -// ============================ -// waitForL1Idle -// ============================ - -/** - * Poll pipeline queue status until L1 is idle for a given session. - * Modeled after benchmark-ingest.ts waitForPipelineIdle() but focused on L1 only. - */ -async function waitForL1Idle( - scheduler: MemoryPipelineManager, - sessionKeys: string[], - logger: PipelineLogger, - opts: { - pollIntervalMs?: number; - stableRounds?: number; - maxWaitMs?: number; - } = {}, -): Promise { - const pollInterval = opts.pollIntervalMs ?? 1_000; - const stableRounds = opts.stableRounds ?? 3; - const maxWait = opts.maxWaitMs ?? 300_000; // 5 min default - - const startTime = Date.now(); - let consecutiveIdle = 0; - - while (true) { - const elapsed = Date.now() - startTime; - if (elapsed > maxWait) { - logger.warn(`${TAG} [waitL1] Max wait time reached (${(maxWait / 1000).toFixed(0)}s), proceeding`); - break; - } - - const queues = scheduler.getQueueSizes(); - - // Check per-session: buffered messages + conversation count - let totalBuffered = 0; - let totalConversationCount = 0; - for (const key of sessionKeys) { - totalBuffered += scheduler.getBufferedMessageCount(key); - const state = scheduler.getSessionState(key); - if (state) { - totalConversationCount += state.conversation_count; - } - } - - const isIdle = - queues.l1Idle && - totalBuffered === 0 && - totalConversationCount === 0; - - if (isIdle) { - consecutiveIdle++; - if (consecutiveIdle >= stableRounds) { - logger.debug?.(`${TAG} [waitL1] L1 stable for ${stableRounds} consecutive polls`); - return; - } - } else { - consecutiveIdle = 0; - logger.debug?.( - `${TAG} [waitL1] Waiting: l1Queue=${queues.l1}, l1Pending=${queues.l1Pending}, l1Idle=${queues.l1Idle}, ` + - `buffered=${totalBuffered}, convCount=${totalConversationCount}`, - ); - } - - await new Promise((resolve) => setTimeout(resolve, pollInterval)); - } -} - -// ============================ -// Main execution function -// ============================ - -/** - * Execute the seed pipeline: feed normalized input through L0 โ†’ L1. - * - * L2/L3 runners are wired but their completion is **not** awaited โ€” see the - * module-level FIXME. The pipeline is destroyed after L1 idle, so L2/L3 may - * be interrupted mid-run. - * - * This is the core runtime called by `src/cli/commands/seed.ts` after - * all input validation and user confirmation are complete. - */ -export async function executeSeed( - input: NormalizedInput, - opts: SeedRuntimeOptions, -): Promise { - const { logger, onProgress } = opts; - const startTime = Date.now(); - - // Track interrupt signal - let interrupted = false; - const onSigint = () => { - if (interrupted) { - // Second Ctrl+C โ€” force exit - logger.warn(`${TAG} Force exit (second Ctrl+C)`); - process.exit(1); - } - interrupted = true; - logger.warn(`${TAG} Interrupt received, finishing current round and shutting down...`); - }; - process.on("SIGINT", onSigint); - - let pipeline: PipelineInstance | undefined; - let totalL0Recorded = 0; - let roundsProcessed = 0; - - try { - // Create and start pipeline (returns both the pipeline instance and the - // seed-optimized config so we don't need to parse config again) - const seed = await createSeedPipeline(opts); - pipeline = seed.pipeline; - const seedCfg = seed.cfg; - - pipeline.scheduler.start({}); - logger.info(`${TAG} Pipeline started, processing ${input.sessions.length} session(s), ${input.totalRounds} round(s)`); - - // Seed-specific: use 0 so the cold-start guard in captureAtomically() - // does NOT filter out historical messages. In live mode Date.now() - // prevents the first agent_end from dumping full session history, - // but seed intentionally feeds all historical data. - const captureStartTimestamp = 0; - - // Process each session โ†’ each round - // Key invariant: after every everyNConversations rounds we must wait for L1 - // to finish before feeding more rounds. Without this pause the for-loop - // would dump all rounds into L0 back-to-back and L1 would only run once - // with the full batch (defeating the "every N" batching semantics). - const everyN = seedCfg.pipeline.everyNConversations; - - for (const session of input.sessions) { - if (interrupted) break; - - logger.info(`${TAG} Session: key="${session.sessionKey}" id="${session.sessionId}" rounds=${session.rounds.length}`); - - for (let ri = 0; ri < session.rounds.length; ri++) { - if (interrupted) break; - - const round = session.rounds[ri]!; - roundsProcessed++; - - // Build messages in the format expected by performAutoCapture. - // Field must be named "timestamp" (not "ts") because l0-recorder's - // extractUserAssistantMessages reads m.timestamp for incremental filtering. - const messages = round.messages.map((m) => ({ - role: m.role, - content: m.content, - timestamp: m.timestamp, - })); - - try { - const result = await performAutoCapture({ - messages, - sessionKey: session.sessionKey, - sessionId: session.sessionId, - cfg: seedCfg, - pluginDataDir: opts.outputDir, - logger, - scheduler: pipeline.scheduler, - pluginStartTimestamp: captureStartTimestamp, - vectorStore: pipeline.vectorStore, - embeddingService: pipeline.embeddingService, - }); - - totalL0Recorded += result.l0RecordedCount; - } catch (err) { - logger.error( - `${TAG} L0 capture failed for session="${session.sessionKey}" round=${ri}: ` + - `${err instanceof Error ? err.message : String(err)}`, - ); - } - - // Report progress - onProgress?.({ - currentRound: roundsProcessed, - totalRounds: input.totalRounds, - sessionKey: session.sessionKey, - stage: "l0_captured", - }); - - // After every N rounds, wait for the triggered L1 to finish before - // feeding the next batch. This keeps L1 batches aligned with the - // everyNConversations boundary instead of letting all rounds pile up. - const roundInSession = ri + 1; // 1-based - if (roundInSession % everyN === 0 && !interrupted) { - onProgress?.({ - currentRound: roundsProcessed, - totalRounds: input.totalRounds, - sessionKey: session.sessionKey, - stage: "l1_waiting", - }); - - logger.info( - `${TAG} Pausing after round ${roundInSession}/${session.rounds.length} ` + - `for session="${session.sessionKey}" โ€” waiting for L1 to drain`, - ); - - await waitForL1Idle( - pipeline.scheduler, - [session.sessionKey], - logger, - { pollIntervalMs: 500, stableRounds: 2, maxWaitMs: 120_000 }, - ); - } - } - - // After all rounds for this session, wait for any residual L1 work - // (handles the tail when total rounds is not a multiple of everyN) - if (!interrupted) { - onProgress?.({ - currentRound: roundsProcessed, - totalRounds: input.totalRounds, - sessionKey: session.sessionKey, - stage: "l1_waiting", - }); - - await waitForL1Idle( - pipeline.scheduler, - [session.sessionKey], - logger, - { pollIntervalMs: 1_000, stableRounds: 3, maxWaitMs: 300_000 }, - ); - - logger.info(`${TAG} L1 idle for session="${session.sessionKey}"`); - } - } - - // Final wait for all sessions - if (!interrupted) { - const allKeys = input.sessions.map((s) => s.sessionKey); - logger.info(`${TAG} Final L1 idle wait for all sessions...`); - await waitForL1Idle( - pipeline.scheduler, - allKeys, - logger, - { pollIntervalMs: 1_000, stableRounds: 3, maxWaitMs: 300_000 }, - ); - } - } finally { - process.removeListener("SIGINT", onSigint); - - // Graceful shutdown - if (pipeline) { - try { - await pipeline.destroy(); - } catch (err) { - logger.error(`${TAG} Pipeline destroy error: ${err instanceof Error ? err.message : String(err)}`); - } - } - } - - const durationMs = Date.now() - startTime; - - const summary: SeedSummary = { - sessionsProcessed: input.sessions.length, - roundsProcessed, - messagesProcessed: input.totalMessages, - l0RecordedCount: totalL0Recorded, - durationMs, - outputDir: opts.outputDir, - }; - - if (interrupted) { - logger.warn(`${TAG} Seed interrupted after ${roundsProcessed}/${input.totalRounds} rounds`); - } else { - logger.info( - `${TAG} Seed complete: sessions=${summary.sessionsProcessed}, ` + - `rounds=${summary.roundsProcessed}, messages=${summary.messagesProcessed}, ` + - `l0Recorded=${summary.l0RecordedCount}, duration=${(durationMs / 1000).toFixed(1)}s`, - ); - } - - // Append seed info to manifest (non-fatal if it fails) - try { - const manifest = readManifest(opts.outputDir); - if (manifest) { - manifest.seed = { - inputFile: opts.inputFile ? path.basename(opts.inputFile) : undefined, - sessions: summary.sessionsProcessed, - rounds: summary.roundsProcessed, - messages: summary.messagesProcessed, - startedAt: new Date(startTime).toISOString(), - completedAt: new Date().toISOString(), - }; - writeManifest(opts.outputDir, manifest); - logger.info(`${TAG} Manifest updated with seed info`); - } - } catch (err) { - logger.warn(`${TAG} Failed to update manifest with seed info (non-fatal): ${err instanceof Error ? err.message : String(err)}`); - } - - return summary; -} diff --git a/src/seed/types.ts b/src/seed/types.ts deleted file mode 100644 index 2cb4ab8..0000000 --- a/src/seed/types.ts +++ /dev/null @@ -1,140 +0,0 @@ -/** - * Shared type definitions for the `seed` command. - * - * Covers: - * - Raw input shapes (Format A / B / JSONL) - * - Normalized internal structures - * - Validation error descriptors - */ - -// ============================ -// Raw input types (before validation) -// ============================ - -/** A single message in a conversation round. */ -export interface RawMessage { - role: string; - content: string; - /** - * Epoch milliseconds (number) **or** ISO 8601 string (e.g. `"2024-04-01T12:00:00Z"`). - * ISO strings are parsed via `new Date()` during normalization and - * stored internally as epoch ms. - */ - timestamp?: number | string; -} - -/** A single session entry (shared between Format A wrapper and Format B array). */ -export interface RawSession { - sessionKey: string; - sessionId?: string; - conversations: RawMessage[][]; -} - -/** Format A: `{ sessions: [...] }` */ -export interface FormatA { - sessions: RawSession[]; -} - -/** Format B: `[...]` (top-level array of sessions) */ -export type FormatB = RawSession[]; - -// ============================ -// Normalized types (after validation) -// ============================ - -export interface NormalizedMessage { - role: string; - content: string; - /** Epoch ms โ€” always present after normalization (filled if originally missing). */ - timestamp: number; -} - -export interface NormalizedRound { - messages: NormalizedMessage[]; -} - -export interface NormalizedSession { - sessionKey: string; - sessionId: string; - rounds: NormalizedRound[]; - /** Index in the original input array (for progress reporting). */ - sourceIndex: number; -} - -export interface NormalizedInput { - sessions: NormalizedSession[]; - /** Total number of rounds across all sessions. */ - totalRounds: number; - /** Total number of messages across all sessions. */ - totalMessages: number; - /** Whether timestamps were present in the original input. */ - hasTimestamps: boolean; -} - -// ============================ -// Validation -// ============================ - -/** Stages where a validation error can occur. */ -export type ValidationStage = - | "file" - | "top_level" - | "session" - | "round" - | "message" - | "timestamp_consistency"; - -/** A single validation error with location context. */ -export interface ValidationError { - stage: ValidationStage; - sourceIndex?: number; - sessionKey?: string; - roundIndex?: number; - messageIndex?: number; - message: string; -} - -// ============================ -// Seed command options (from CLI) -// ============================ - -export interface SeedCommandOptions { - /** Path to input file (required). */ - input: string; - /** Output directory (optional, auto-generated if missing). */ - outputDir?: string; - /** Fallback session key when input lacks one. */ - sessionKey?: string; - /** Strict round-role validation (each round must have user + assistant). */ - strictRoundRole: boolean; - /** Skip interactive confirmations. */ - yes: boolean; - /** Path to memory-tdai config override file (JSON, deep-merged on top of current plugin config). */ - configFile?: string; -} - -// ============================ -// Seed runtime types -// ============================ - -/** Progress info emitted during seed execution. */ -export interface SeedProgress { - /** Current round index (1-based, across all sessions). */ - currentRound: number; - /** Total rounds. */ - totalRounds: number; - /** Current session key. */ - sessionKey: string; - /** Current stage description. */ - stage: string; -} - -/** Final summary after seed completes. */ -export interface SeedSummary { - sessionsProcessed: number; - roundsProcessed: number; - messagesProcessed: number; - l0RecordedCount: number; - durationMs: number; - outputDir: string; -} diff --git a/src/store/bm25-client.ts b/src/store/bm25-client.ts deleted file mode 100644 index 031c2d2..0000000 --- a/src/store/bm25-client.ts +++ /dev/null @@ -1,168 +0,0 @@ -/** - * BM25 Sparse Vector Encoding Client. - * - * HTTP client for the BM25 Python sidecar service (bm25_server.py). - * Used by TCVDB backend to generate sparse vectors for hybridSearch. - * - * Two operations: - * - `encodeTexts(texts)` โ€” encode documents for upsert (TF-based) - * - `encodeQueries(texts)` โ€” encode queries for search (IDF-based) - * - * Graceful degradation: if the sidecar is unreachable, all methods - * return empty arrays and `isHealthy()` returns false. Callers can - * check health to dynamically downgrade to pure semantic search. - */ - -// ============================ -// Types -// ============================ - -/** Sparse vector: array of [token_hash, weight] pairs. */ -export type SparseVector = Array<[number, number]>; - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -export interface BM25ClientConfig { - /** Sidecar service URL (default: "http://127.0.0.1:8084") */ - serviceUrl: string; - /** Request timeout in ms (default: 5000) */ - timeout: number; -} - -interface EncodeResponse { - vectors: SparseVector[]; -} - -// ============================ -// Implementation -// ============================ - -const TAG = "[memory-tdai][bm25-client]"; - -export class BM25Client { - private readonly baseUrl: string; - private readonly timeout: number; - private readonly logger?: Logger; - - /** Cached health status to avoid repeated checks on every call. */ - private _healthy: boolean | undefined; - private _lastHealthCheck = 0; - private static readonly HEALTH_CHECK_INTERVAL_MS = 30_000; // re-check every 30s - - constructor(config: BM25ClientConfig, logger?: Logger) { - this.baseUrl = config.serviceUrl.replace(/\/+$/, ""); - this.timeout = config.timeout; - this.logger = logger; - } - - /** - * Encode document texts for upsert (TF-based BM25 scoring). - * Returns one SparseVector per input text. - * Returns empty array on error (non-throwing). - */ - async encodeTexts(texts: string[]): Promise { - if (texts.length === 0) return []; - return this._encode("/encode_texts", texts); - } - - /** - * Encode query texts for search (IDF-based BM25 scoring). - * Returns one SparseVector per input text. - * Returns empty array on error (non-throwing). - */ - async encodeQueries(texts: string[]): Promise { - if (texts.length === 0) return []; - return this._encode("/encode_queries", texts); - } - - /** - * Check if the BM25 sidecar is reachable. - * Result is cached for 30 seconds to avoid spamming health checks. - */ - async isHealthy(): Promise { - const now = Date.now(); - if ( - this._healthy !== undefined && - now - this._lastHealthCheck < BM25Client.HEALTH_CHECK_INTERVAL_MS - ) { - return this._healthy; - } - - try { - const resp = await fetch(`${this.baseUrl}/health`, { - signal: AbortSignal.timeout(3000), - }); - this._healthy = resp.ok; - } catch { - this._healthy = false; - } - this._lastHealthCheck = now; - - if (!this._healthy) { - this.logger?.warn(`${TAG} BM25 sidecar health check failed (${this.baseUrl})`); - } - - return this._healthy; - } - - // โ”€โ”€ Internal โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - private async _encode(path: string, texts: string[]): Promise { - try { - const resp = await fetch(`${this.baseUrl}${path}`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ texts }), - signal: AbortSignal.timeout(this.timeout), - }); - - if (!resp.ok) { - const errBody = await resp.text().catch(() => "(unreadable)"); - this.logger?.warn( - `${TAG} ${path} HTTP ${resp.status}: ${errBody.slice(0, 200)}`, - ); - return []; - } - - const json = (await resp.json()) as EncodeResponse; - return json.vectors ?? []; - } catch (err) { - // Mark unhealthy on connection errors - this._healthy = false; - this._lastHealthCheck = Date.now(); - - this.logger?.warn( - `${TAG} ${path} failed: ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } -} - -// ============================ -// Factory -// ============================ - -/** - * Create a BM25Client if BM25 is enabled in config. - * Returns undefined if disabled โ€” callers should check before using. - */ -export function createBM25Client( - config: { enabled: boolean; serviceUrl: string; timeout: number }, - logger?: Logger, -): BM25Client | undefined { - if (!config.enabled) { - logger?.info(`${TAG} BM25 sparse encoding disabled`); - return undefined; - } - logger?.info(`${TAG} BM25 client โ†’ ${config.serviceUrl}`); - return new BM25Client( - { serviceUrl: config.serviceUrl, timeout: config.timeout }, - logger, - ); -} diff --git a/src/store/bm25-local.ts b/src/store/bm25-local.ts deleted file mode 100644 index 548db4f..0000000 --- a/src/store/bm25-local.ts +++ /dev/null @@ -1,97 +0,0 @@ -/** - * Local BM25 Sparse Vector Encoder. - * - * Pure TypeScript replacement for the Python sidecar BM25 client. - * Uses @tencentdb-agent-memory/tcvdb-text package for tokenization (jieba-wasm) and BM25 encoding. - * - * Two operations (same contract as the old BM25Client): - * - `encodeTexts(texts)` โ€” encode documents for upsert (TF-based) - * - `encodeQueries(texts)` โ€” encode queries for search (IDF-based) - */ - -import { BM25Encoder } from "@tencentdb-agent-memory/tcvdb-text"; -import type { SparseVector } from "@tencentdb-agent-memory/tcvdb-text"; - -export type { SparseVector }; - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -export interface BM25LocalConfig { - /** Whether BM25 sparse encoding is enabled (default: true) */ - enabled: boolean; - /** Language for BM25 pre-trained params: "zh" or "en" (default: "zh") */ - language?: "zh" | "en"; -} - -const TAG = "[memory-tdai][bm25-local]"; - -// ============================ -// Implementation -// ============================ - -export class BM25LocalEncoder { - private readonly encoder: BM25Encoder; - private readonly logger?: Logger; - - constructor(language: "zh" | "en" = "zh", logger?: Logger) { - this.logger = logger; - this.encoder = BM25Encoder.default(language); - logger?.debug?.(`${TAG} Initialized BM25 local encoder (language=${language})`); - } - - /** - * Encode document texts for upsert (TF-based BM25 scoring). - * Returns one SparseVector per input text. - */ - encodeTexts(texts: string[]): SparseVector[] { - if (texts.length === 0) return []; - try { - return this.encoder.encodeTexts(texts); - } catch (err) { - this.logger?.warn( - `${TAG} encodeTexts failed: ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * Encode query texts for search (IDF-based BM25 scoring). - * Returns one SparseVector per input text. - */ - encodeQueries(texts: string[]): SparseVector[] { - if (texts.length === 0) return []; - try { - return this.encoder.encodeQueries(texts); - } catch (err) { - this.logger?.warn( - `${TAG} encodeQueries failed: ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } -} - -// ============================ -// Factory -// ============================ - -/** - * Create a BM25LocalEncoder if BM25 is enabled in config. - * Returns undefined if disabled โ€” callers should check before using. - */ -export function createBM25Encoder( - config: BM25LocalConfig, - logger?: Logger, -): BM25LocalEncoder | undefined { - if (!config.enabled) { - logger?.debug?.(`${TAG} BM25 sparse encoding disabled`); - return undefined; - } - return new BM25LocalEncoder(config.language ?? "zh", logger); -} diff --git a/src/store/embedding.ts b/src/store/embedding.ts deleted file mode 100644 index 559a509..0000000 --- a/src/store/embedding.ts +++ /dev/null @@ -1,646 +0,0 @@ -/** - * Embedding Service: converts text to vector embeddings. - * - * Supports two providers: - * - "openai": OpenAI-compatible embedding APIs (OpenAI, Azure OpenAI, self-hosted) - * - "local": node-llama-cpp with embeddinggemma-300m GGUF model (fully offline) - * - * When no remote embedding is configured, automatically falls back to local provider. - * - * Design: - * - Single `embed()` for one text, `embedBatch()` for multiple. - * - `getDimensions()` returns configured vector dimensions. - * - Throws on failure; callers decide fallback strategy. - */ - -// ============================ -// Types -// ============================ - -export interface OpenAIEmbeddingConfig { - /** Provider identifier โ€” any value other than "local" (e.g. "openai", "deepseek", "azure", "qclaw") */ - provider: string; - /** API base URL (required โ€” must be specified by user, e.g. "https://api.openai.com/v1") */ - baseUrl: string; - /** API Key (required) */ - apiKey: string; - /** Model name (required โ€” must be specified by user) */ - model: string; - /** Output dimensions (required โ€” must match the chosen model) */ - dimensions: number; - /** Local proxy URL (only for provider="qclaw") โ€” requests are forwarded through this proxy with Remote-URL header */ - proxyUrl?: string; - /** Max input text length in characters before truncation (default: 5000). */ - maxInputChars?: number; - /** Timeout per API call in milliseconds (default: 10000). */ - timeoutMs?: number; -} - -export interface LocalEmbeddingConfig { - provider: "local"; - /** Custom GGUF model path (default: embeddinggemma-300m from HuggingFace) */ - modelPath?: string; - /** Model cache directory (default: node-llama-cpp default cache) */ - modelCacheDir?: string; -} - -export type EmbeddingConfig = OpenAIEmbeddingConfig | LocalEmbeddingConfig; - -/** Identifies the embedding provider + model for change detection. */ -export interface EmbeddingProviderInfo { - /** Provider identifier (e.g. "local", "openai", "deepseek") */ - provider: string; - /** Model identifier (e.g. "embeddinggemma-300m", "text-embedding-3-large") */ - model: string; -} - -export interface EmbeddingService { - /** Get embedding for a single text */ - embed(text: string): Promise; - /** Get embeddings for multiple texts (batched API call) */ - embedBatch(texts: string[]): Promise; - /** Return the configured vector dimensions */ - getDimensions(): number; - /** Return provider + model identifiers for change detection */ - getProviderInfo(): EmbeddingProviderInfo; - /** - * Whether the service is ready to serve embed requests. - * For remote providers (OpenAI), always true (stateless HTTP). - * For local providers, true only after model download + load completes. - */ - isReady(): boolean; - /** - * Start background warmup (model download + load). - * For remote providers, this is a no-op. - * For local providers, triggers async initialization without blocking. - * Safe to call multiple times (idempotent). - */ - startWarmup(): void; - /** Optional: release resources (model memory, GPU, etc.) on shutdown */ - close?(): void | Promise; -} - -/** - * Error thrown when embed() / embedBatch() is called before the local - * embedding model has finished downloading and loading. - * Callers should catch this and fall back to keyword-only mode. - */ -export class EmbeddingNotReadyError extends Error { - constructor(message?: string) { - super(message ?? "Local embedding model is not ready yet (still downloading or loading)"); - this.name = "EmbeddingNotReadyError"; - } -} - -// ============================ -// Logger interface -// ============================ - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -const TAG = "[memory-tdai][embedding]"; - -// ============================ -// Local (node-llama-cpp) implementation -// ============================ - -/** Default model: Google's embeddinggemma-300m, quantized Q8_0 (~300MB) */ -const DEFAULT_LOCAL_MODEL = - "hf:ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/embeddinggemma-300m-qat-Q8_0.gguf"; - -/** embeddinggemma-300m outputs 768-dimensional vectors */ -const LOCAL_DIMENSIONS = 768; - -/** - * embeddinggemma-300m has a 256-token context window. - * As a safe heuristic, we limit input to ~600 chars for CJK text - * (CJK characters typically tokenize to 1-2 tokens each, - * so 600 chars โ‰ˆ 200-400 tokens, keeping well within 256-token limit - * after accounting for special tokens). - * For Latin text, ~800 chars is a safe limit (~200 tokens). - * We use 512 chars as a conservative universal limit. - */ -const LOCAL_MAX_INPUT_CHARS = 512; - -/** - * Sanitize NaN/Inf values and L2-normalize the vector. - * Matches OpenClaw's own sanitizeAndNormalizeEmbedding(). - */ -function sanitizeAndNormalize(vec: number[] | Float32Array): Float32Array { - const arr = Array.from(vec).map((v) => (Number.isFinite(v) ? v : 0)); - const magnitude = Math.sqrt(arr.reduce((sum, v) => sum + v * v, 0)); - if (magnitude < 1e-10) { - return new Float32Array(arr); - } - return new Float32Array(arr.map((v) => v / magnitude)); -} - -/** - * Initialization state for LocalEmbeddingService. - * - "idle": not started yet - * - "initializing": model download / load is in progress (background) - * - "ready": model is loaded and ready to serve - * - "failed": initialization failed (will retry on next startWarmup) - */ -type LocalInitState = "idle" | "initializing" | "ready" | "failed"; - -/** Function that dynamically imports node-llama-cpp. Overridable for testing. */ -export type ImportLlamaFn = () => Promise<{ - getLlama: (opts: { logLevel: number }) => Promise; - resolveModelFile: (model: string, cacheDir?: string) => Promise; - LlamaLogLevel: { error: number }; -}>; - -const defaultImportLlama: ImportLlamaFn = () => import("node-llama-cpp") as unknown as ReturnType; - -export class LocalEmbeddingService implements EmbeddingService { - private readonly modelPath: string; - private readonly modelCacheDir?: string; - private readonly logger?: Logger; - private readonly importLlama: ImportLlamaFn; - - // Initialization state machine - private initState: LocalInitState = "idle"; - private initPromise: Promise | null = null; - private initError: Error | null = null; - private embeddingContext: { - getEmbeddingFor: (text: string) => Promise<{ vector: Float32Array | number[] }>; - } | null = null; - - constructor(config?: LocalEmbeddingConfig, logger?: Logger, importLlama?: ImportLlamaFn) { - this.modelPath = config?.modelPath?.trim() || DEFAULT_LOCAL_MODEL; - this.modelCacheDir = config?.modelCacheDir?.trim(); - this.logger = logger; - this.importLlama = importLlama ?? defaultImportLlama; - } - - getDimensions(): number { - return LOCAL_DIMENSIONS; - } - - getProviderInfo(): EmbeddingProviderInfo { - return { provider: "local", model: this.modelPath }; - } - - /** - * Whether the local model is fully loaded and ready to serve requests. - */ - isReady(): boolean { - return this.initState === "ready" && this.embeddingContext !== null; - } - - /** - * Start background warmup: download model (if needed) and load into memory. - * Does NOT block the caller โ€” returns immediately. - * Safe to call multiple times (idempotent); re-triggers on "failed" state. - */ - startWarmup(): void { - if (this.initState === "initializing" || this.initState === "ready") { - return; // already in progress or done - } - this.logger?.info(`${TAG} Starting background warmup for local embedding model...`); - this.initState = "initializing"; - this.initError = null; - - this.initPromise = this._doInitialize() - .then(() => { - this.initState = "ready"; - this.logger?.info(`${TAG} Background warmup complete โ€” local embedding ready`); - }) - .catch((err) => { - this.initState = "failed"; - this.initError = err instanceof Error ? err : new Error(String(err)); - this.logger?.error( - `${TAG} Background warmup failed: ${this.initError.message}. ` + - `embed() calls will throw EmbeddingNotReadyError until retried.`, - ); - }); - } - - /** - * Get embedding for a single text. - * @throws {EmbeddingNotReadyError} if model is not yet ready. - */ - async embed(text: string): Promise { - this.assertReady(); - const truncated = this.truncateInput(text); - const embedding = await this.embeddingContext!.getEmbeddingFor(truncated); - return sanitizeAndNormalize(embedding.vector); - } - - /** - * Get embeddings for multiple texts. - * @throws {EmbeddingNotReadyError} if model is not yet ready. - */ - async embedBatch(texts: string[]): Promise { - if (texts.length === 0) return []; - this.assertReady(); - - const results: Float32Array[] = []; - for (const text of texts) { - const truncated = this.truncateInput(text); - const embedding = await this.embeddingContext!.getEmbeddingFor(truncated); - results.push(sanitizeAndNormalize(embedding.vector)); - } - return results; - } - - /** - * Release the node-llama-cpp embedding context and model resources. - * Safe to call multiple times (idempotent). - */ - close(): void { - if (this.embeddingContext) { - try { - const ctx = this.embeddingContext as unknown as { dispose?: () => void }; - ctx.dispose?.(); - } catch { - // best-effort cleanup - } - this.embeddingContext = null; - this.initPromise = null; - this.initState = "idle"; - this.initError = null; - this.logger?.info(`${TAG} Local embedding resources released`); - } - } - - /** - * Assert the model is ready. Throws EmbeddingNotReadyError if not. - */ - private assertReady(): void { - if (this.initState === "ready" && this.embeddingContext) { - return; - } - if (this.initState === "failed") { - throw new EmbeddingNotReadyError( - `Local embedding model initialization failed: ${this.initError?.message ?? "unknown error"}. ` + - `Call startWarmup() to retry.`, - ); - } - if (this.initState === "initializing") { - throw new EmbeddingNotReadyError( - "Local embedding model is still loading (download/initialization in progress). Please try again later.", - ); - } - // "idle" โ€” startWarmup() was never called - throw new EmbeddingNotReadyError( - "Local embedding model warmup has not been started. Call startWarmup() first.", - ); - } - - /** - * Truncate input text to stay within the model's context window. - * embeddinggemma-300m has a 256-token limit; we use a character-based - * heuristic (LOCAL_MAX_INPUT_CHARS) as a safe proxy. - */ - private truncateInput(text: string): string { - if (text.length <= LOCAL_MAX_INPUT_CHARS) return text; - this.logger?.debug?.( - `${TAG} Input truncated from ${text.length} to ${LOCAL_MAX_INPUT_CHARS} chars (model context limit)`, - ); - return text.slice(0, LOCAL_MAX_INPUT_CHARS); - } - - /** - * Internal: perform the actual model download + load. - * Called by startWarmup(), runs in background. - */ - private async _doInitialize(): Promise { - // Track partially-initialized resources for cleanup on failure - let model: { createEmbeddingContext: () => Promise; dispose?: () => void } | undefined; - try { - this.logger?.debug?.(`${TAG} Loading node-llama-cpp for local embedding...`); - - // Dynamic import โ€” node-llama-cpp is a peer dependency of OpenClaw - const { getLlama, resolveModelFile, LlamaLogLevel } = await this.importLlama(); - - const llama = await getLlama({ logLevel: LlamaLogLevel.error }); - this.logger?.debug?.(`${TAG} Llama instance created`); - - const resolvedPath = await resolveModelFile( - this.modelPath, - this.modelCacheDir || undefined, - ); - this.logger?.debug?.(`${TAG} Model resolved: ${resolvedPath}`); - - model = await (llama as unknown as { loadModel: (opts: { modelPath: string }) => Promise }).loadModel({ modelPath: resolvedPath }); - this.logger?.debug?.(`${TAG} Model loaded, creating embedding context...`); - - this.embeddingContext = await model!.createEmbeddingContext() as typeof this.embeddingContext; - this.logger?.info(`${TAG} Local embedding ready (model=${this.modelPath}, dims=${LOCAL_DIMENSIONS})`); - } catch (err) { - // Clean up partially-initialized resources to prevent leaks - if (model?.dispose) { - try { model.dispose(); } catch { /* best-effort */ } - } - this.embeddingContext = null; - throw err; - } - } - - /** - * Wait for ongoing warmup to complete (used internally by tests). - * Returns immediately if already ready or idle. - */ - async waitForReady(): Promise { - if (this.initPromise) { - await this.initPromise; - } - } -} - -// ============================ -// OpenAI-compatible implementation -// ============================ - -/** Max texts per batch (OpenAI limit is 2048, we use a safe value) */ -const MAX_BATCH_SIZE = 256; - -/** Max retries for API calls */ -const MAX_RETRIES = 0; -/** Default timeout per API call in milliseconds */ -const DEFAULT_API_TIMEOUT_MS = 10_000; - -/** - * Custom error class for embedding API errors that carries HTTP status code. - * Used to distinguish non-retryable client errors (4xx except 429) from - * retryable server errors (5xx) and rate limits (429). - */ -class EmbeddingApiError extends Error { - readonly httpStatus: number; - constructor(message: string, httpStatus: number) { - super(message); - this.name = "EmbeddingApiError"; - this.httpStatus = httpStatus; - } - /** Returns true for 4xx errors that should NOT be retried (excluding 429). */ - isClientError(): boolean { - return this.httpStatus >= 400 && this.httpStatus < 500 && this.httpStatus !== 429; - } -} - -interface OpenAIEmbeddingResponse { - data: Array<{ - index: number; - embedding: number[]; - }>; - usage?: { - prompt_tokens: number; - total_tokens: number; - }; -} - -export class OpenAIEmbeddingService implements EmbeddingService { - private readonly baseUrl: string; - private readonly apiKey: string; - private readonly model: string; - private readonly dims: number; - private readonly providerName: string; - private readonly proxyUrl?: string; - private readonly maxInputChars?: number; - private readonly timeoutMs: number; - private readonly logger?: Logger; - - constructor(config: OpenAIEmbeddingConfig, logger?: Logger) { - if (!config.apiKey) { - throw new Error("EmbeddingService: apiKey is required for remote provider"); - } - if (!config.baseUrl) { - throw new Error("EmbeddingService: baseUrl is required for remote provider"); - } - if (!config.model) { - throw new Error("EmbeddingService: model is required for remote provider"); - } - if (!config.dimensions || config.dimensions <= 0) { - throw new Error("EmbeddingService: dimensions is required for remote provider (must be a positive integer)"); - } - this.baseUrl = config.baseUrl.replace(/\/+$/, ""); - this.apiKey = config.apiKey; - this.model = config.model; - this.dims = config.dimensions; - this.providerName = config.provider || "openai"; - this.proxyUrl = config.proxyUrl?.trim() || undefined; - this.maxInputChars = config.maxInputChars && config.maxInputChars > 0 ? config.maxInputChars : undefined; - this.timeoutMs = config.timeoutMs && config.timeoutMs > 0 ? config.timeoutMs : DEFAULT_API_TIMEOUT_MS; - this.logger = logger; - } - - getDimensions(): number { - return this.dims; - } - - getProviderInfo(): EmbeddingProviderInfo { - return { provider: this.providerName, model: this.model }; - } - - /** Remote embedding is always ready (stateless HTTP). */ - isReady(): boolean { - return true; - } - - /** No-op for remote embedding (no local model to warm up). */ - startWarmup(): void { - // nothing to do โ€” remote API is stateless - } - - async embed(text: string): Promise { - const [result] = await this.embedBatch([text]); - return result; - } - - async embedBatch(texts: string[]): Promise { - if (texts.length === 0) return []; - - // Truncate texts exceeding maxInputChars limit - const processedTexts = this.maxInputChars - ? texts.map((t) => this.truncateInput(t)) - : texts; - - // Split into sub-batches if needed - if (processedTexts.length > MAX_BATCH_SIZE) { - const results: Float32Array[] = []; - for (let i = 0; i < processedTexts.length; i += MAX_BATCH_SIZE) { - const chunk = processedTexts.slice(i, i + MAX_BATCH_SIZE); - const chunkResults = await this._callApi(chunk); - results.push(...chunkResults); - } - return results; - } - - return this._callApi(processedTexts); - } - - /** - * Truncate input text to stay within the configured maxInputChars limit. - * Logs a warning when truncation occurs. - */ - private truncateInput(text: string): string { - if (!this.maxInputChars || text.length <= this.maxInputChars) return text; - this.logger?.warn?.( - `${TAG} Input truncated from ${text.length} to ${this.maxInputChars} chars (maxInputChars limit)`, - ); - return text.slice(0, this.maxInputChars); - } - - private async _callApi(texts: string[]): Promise { - const body: Record = { - input: texts, - model: this.model, - dimensions: this.dims, - }; - - // Determine fetch URL and headers based on proxy mode - const useProxy = this.providerName === "qclaw" && !!this.proxyUrl; - const fetchUrl = useProxy ? this.proxyUrl! : `${this.baseUrl}/embeddings`; - const headers: Record = { - "Content-Type": "application/json", - Authorization: `Bearer ${this.apiKey}`, - }; - if (useProxy) { - headers["Remote-URL"] = `${this.baseUrl}/embeddings`; - this.logger?.debug?.( - `${TAG} [qclaw-proxy] Forwarding embedding request via proxy: ${fetchUrl}, Remote-URL: ${headers["Remote-URL"]}`, - ); - } - - // Retry loop with timeout - let lastError: Error | undefined; - for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) { - try { - const controller = new AbortController(); - const timeoutId = setTimeout(() => controller.abort(), this.timeoutMs); - - try { - const resp = await fetch(fetchUrl, { - method: "POST", - headers, - body: JSON.stringify(body), - signal: controller.signal, - }); - - if (!resp.ok) { - const errBody = await resp.text().catch(() => "(unable to read body)"); - const err = new EmbeddingApiError( - `Embedding API error: HTTP ${resp.status} ${resp.statusText} โ€” ${errBody.slice(0, 500)}`, - resp.status, - ); - // Don't retry on 4xx client errors (except 429 rate limit) - if (resp.status >= 400 && resp.status < 500 && resp.status !== 429) { - throw err; - } - lastError = err; - continue; - } - - const json = (await resp.json()) as OpenAIEmbeddingResponse; - - if (!json.data || !Array.isArray(json.data)) { - throw new Error("Embedding API returned unexpected format: missing 'data' array"); - } - - // Sort by index to ensure correct order, then sanitize+normalize for consistency with local provider - const sorted = [...json.data].sort((a, b) => a.index - b.index); - return sorted.map((d) => sanitizeAndNormalize(d.embedding)); - } finally { - clearTimeout(timeoutId); - } - } catch (err) { - // Non-retryable errors (4xx client errors) โ€” rethrow immediately - if (err instanceof EmbeddingApiError && err.isClientError()) { - throw err; - } - lastError = err instanceof Error ? err : new Error(String(err)); - // AbortError = timeout, retry - if (attempt < MAX_RETRIES) { - // Exponential backoff: 500ms, 1000ms - const delay = 500 * (attempt + 1); - await new Promise((r) => setTimeout(r, delay)); - } - } - } - - throw lastError ?? new Error("Embedding API call failed after retries"); - } -} - -// ============================ -// Factory -// ============================ - -/** - * Create an EmbeddingService from config. - * - * Strategy: - * - If config has provider != "local" with valid apiKey, model, and dimensions โ†’ use remote OpenAI-compatible embedding - * - If config has provider="local" โ†’ use node-llama-cpp local embedding - * - If config is undefined or missing required fields โ†’ fall back to local embedding - * - * NOTE: For local providers, `startWarmup()` is NOT called here. - * The caller is responsible for calling `startWarmup()` at the right time - * (e.g. on first conversation) to avoid triggering model download during - * short-lived CLI commands like `gateway stop` or `agents list`. - */ -export function createEmbeddingService( - config: EmbeddingConfig | undefined, - logger?: Logger, -): EmbeddingService { - // Remote OpenAI-compatible provider: any provider value other than "local" - if (config && config.provider !== "local" && "apiKey" in config && config.apiKey) { - logger?.debug?.(`${TAG} Using remote embedding (provider=${config.provider}, model=${config.model})`); - return new OpenAIEmbeddingService(config as OpenAIEmbeddingConfig, logger); - } - - // Explicit local config - if (config && config.provider === "local") { - const localConfig = config as LocalEmbeddingConfig; - logger?.debug?.(`${TAG} Using local embedding (node-llama-cpp, model=${localConfig.modelPath ?? DEFAULT_LOCAL_MODEL})`); - return new LocalEmbeddingService(localConfig, logger); - } - - // Fallback: no config or empty apiKey โ†’ use local - logger?.debug?.(`${TAG} No remote embedding configured, falling back to local embedding (node-llama-cpp)`); - return new LocalEmbeddingService(undefined, logger); -} - -// ============================ -// NoopEmbeddingService (for server-side embedding backends) -// ============================ - -/** - * No-op embedding service for backends with built-in server-side embedding - * (e.g., TCVDB with Collection-level embedding config). - * - * All embed() calls return an empty Float32Array because the server generates - * vectors automatically from the text field during upsert/search. - */ -export class NoopEmbeddingService implements EmbeddingService { - embed(_text: string): Promise { - return Promise.resolve(new Float32Array(0)); - } - - embedBatch(texts: string[]): Promise { - return Promise.resolve(texts.map(() => new Float32Array(0))); - } - - getDimensions(): number { - return 0; - } - - getProviderInfo(): EmbeddingProviderInfo { - return { provider: "noop", model: "server-side" }; - } - - isReady(): boolean { - return true; - } - - startWarmup(): void { - // no-op - } -} diff --git a/src/store/factory.ts b/src/store/factory.ts deleted file mode 100644 index 6516aad..0000000 --- a/src/store/factory.ts +++ /dev/null @@ -1,127 +0,0 @@ -/** - * Store Factory โ€” creates the appropriate storage backend and embedding service - * based on plugin configuration. - * - * Supports: - * - "sqlite" (default): local SQLite + sqlite-vec + FTS5 - * - "tcvdb": Tencent Cloud VectorDB (server-side embedding + hybridSearch) - */ - -import path from "node:path"; -import type { MemoryTdaiConfig } from "../config.js"; -import type { IMemoryStore, IEmbeddingService, StoreLogger } from "./types.js"; -import { VectorStore } from "./sqlite.js"; -import { TcvdbMemoryStore } from "./tcvdb.js"; -import { createEmbeddingService, NoopEmbeddingService } from "./embedding.js"; -import type { EmbeddingService } from "./embedding.js"; -import { createBM25Encoder } from "./bm25-local.js"; -import type { BM25LocalEncoder } from "./bm25-local.js"; - -// Re-export for convenience -export type { IMemoryStore, IEmbeddingService, StoreLogger, BM25LocalEncoder }; - -const TAG = "[memory-tdai][factory]"; - -export interface StoreBundle { - store: IMemoryStore; - embedding: IEmbeddingService; - bm25Encoder?: BM25LocalEncoder; - /** Snapshot of current store config for manifest writing. */ - storeSnapshot: import("../utils/manifest.js").StoreConfigSnapshot; -} - -/** - * Create the storage backend, embedding service, and optional BM25 encoder - * based on plugin configuration. - * - * @param config Fully resolved plugin config. - * @param options.dataDir Plugin data directory. - * @param options.logger Logger instance. - */ -export function createStoreBundle( - config: MemoryTdaiConfig, - options: { dataDir: string; logger?: StoreLogger }, -): StoreBundle { - const { logger } = options; - - // โ”€โ”€ BM25 local encoder โ”€โ”€ - const bm25Encoder = createBM25Encoder(config.bm25, logger); - - switch (config.storeBackend) { - case "tcvdb": { - const tcvdbCfg = config.tcvdb; - if (!tcvdbCfg.url || !tcvdbCfg.apiKey) { - throw new Error(`${TAG} TCVDB backend requires tcvdb.url and tcvdb.apiKey`); - } - if (!tcvdbCfg.database) { - throw new Error(`${TAG} TCVDB backend requires tcvdb.database โ€” please set a unique database name in your openclaw.json plugin config`); - } - const database = tcvdbCfg.database; - const store = new TcvdbMemoryStore({ - url: tcvdbCfg.url, - username: tcvdbCfg.username, - apiKey: tcvdbCfg.apiKey, - database, - embeddingModel: tcvdbCfg.embeddingModel, - timeout: tcvdbCfg.timeout, - caPemPath: tcvdbCfg.caPemPath, - logger, - bm25Encoder: bm25Encoder ?? undefined, - }); - - logger?.debug?.( - `${TAG} Store created: backend=tcvdb, database=${database}, model=${tcvdbCfg.embeddingModel}, ` + - `bm25=${bm25Encoder ? "enabled" : "disabled"}`, - ); - - return { - store, - embedding: new NoopEmbeddingService(), - bm25Encoder, - storeSnapshot: { - type: "tcvdb", - tcvdbUrl: tcvdbCfg.url, - tcvdbDatabase: database, - tcvdbAlias: tcvdbCfg.alias || undefined, - }, - }; - } - - case "sqlite": - default: { - // โ”€โ”€ Embedding service (only when enabled) โ”€โ”€ - let embeddingService: EmbeddingService | undefined; - if (config.embedding.enabled && config.embedding.provider !== "local" && config.embedding.apiKey) { - embeddingService = createEmbeddingService({ - provider: config.embedding.provider, - baseUrl: config.embedding.baseUrl, - apiKey: config.embedding.apiKey, - model: config.embedding.model, - dimensions: config.embedding.dimensions, - maxInputChars: config.embedding.maxInputChars, - }, logger); - } - - // dimensions from config (0 when provider="none" โ†’ vec0 deferred) - const dims = config.embedding.dimensions; - const dbPath = path.join(options.dataDir, "vectors.db"); - const store = new VectorStore(dbPath, dims, logger); - - logger?.debug?.( - `${TAG} Store created: backend=sqlite, dbPath=${dbPath}, dimensions=${dims}, ` + - `embedding=${embeddingService ? "enabled" : "disabled"}, ` + - `bm25=${bm25Encoder ? "enabled" : "disabled"}`, - ); - - return { - store, - embedding: embeddingService as unknown as IEmbeddingService, - bm25Encoder, - storeSnapshot: { - type: "sqlite", - sqlitePath: path.relative(options.dataDir, dbPath), - }, - }; - } - } -} diff --git a/src/store/search-utils.ts b/src/store/search-utils.ts deleted file mode 100644 index edac366..0000000 --- a/src/store/search-utils.ts +++ /dev/null @@ -1,62 +0,0 @@ -/** - * Search utilities โ€” shared helpers for memory search across backends. - * - * Contains: - * - RRF (Reciprocal Rank Fusion) merge โ€” used by SQLite hybrid search - * (eliminates the 3x duplication in auto-recall, memory-search, conversation-search) - * - FTS query building โ€” re-exported from sqlite for convenience - */ - -// ============================ -// RRF (Reciprocal Rank Fusion) -// ============================ - -/** - * Standard RRF constant from the original RRF paper. - * Higher k โ†’ more weight on lower-ranked items (smoother distribution). - */ -export const RRF_K = 60; - -/** - * Merge multiple ranked lists via Reciprocal Rank Fusion. - * - * Each item's RRF score = sum over all lists of 1/(k + rank + 1). - * Items appearing in multiple lists get their scores summed. - * - * @param lists Array of ranked lists. Each list must have items with an `id` field. - * @param k RRF constant (default: 60). - * @returns Merged list sorted by descending RRF score, with `rrfScore` attached. - * - * @example - * ```ts - * const merged = rrfMerge( - * [ftsResults, vecResults], - * (item) => item.record_id, - * ); - * ``` - */ -export function rrfMerge( - lists: T[][], - getId: (item: T) => string, - k: number = RRF_K, -): Array { - const map = new Map(); - - for (const list of lists) { - for (let rank = 0; rank < list.length; rank++) { - const item = list[rank]; - const id = getId(item); - const score = 1 / (k + rank + 1); - const existing = map.get(id); - if (existing) { - existing.rrfScore += score; - } else { - map.set(id, { item, rrfScore: score }); - } - } - } - - return [...map.values()] - .sort((a, b) => b.rrfScore - a.rrfScore) - .map(({ item, rrfScore }) => ({ ...item, rrfScore })); -} diff --git a/src/store/sqlite.ts b/src/store/sqlite.ts deleted file mode 100644 index a3922a3..0000000 --- a/src/store/sqlite.ts +++ /dev/null @@ -1,2303 +0,0 @@ -/** - * VectorStore: SQLite-based vector storage using sqlite-vec extension. - * - * Manages two layers of vector-indexed data in a single SQLite database: - * - * **L1 (structured memories):** - * 1. `l1_records` โ€” relational metadata table (content, type, priority, scene, timestamps) - * 2. `l1_vec` โ€” vec0 virtual table for cosine similarity search - * - * **L0 (raw conversations):** - * 3. `l0_conversations` โ€” relational metadata table (session_key, role, message text, timestamps) - * 4. `l0_vec` โ€” vec0 virtual table for cosine similarity search on individual messages - * - * Dependencies: Node.js built-in `node:sqlite` (Node 22+) + `sqlite-vec` (from root workspace). - * - * Design: - * - All operations are synchronous (DatabaseSync API). - * - Writes use manual BEGIN/COMMIT transactions for atomicity (metadata + vector). - * - vec0 virtual table does NOT support ON CONFLICT, so upsert = delete + insert. - * - Thread-safe via WAL mode. - */ - -import { createRequire } from "node:module"; -import type { DatabaseSync, StatementSync } from "node:sqlite"; -import type { MemoryRecord } from "../record/l1-writer.js"; -import type { EmbeddingProviderInfo } from "./embedding.js"; -import type { - IMemoryStore, - StoreCapabilities, - L0Record, - L1SearchResult, - L1FtsResult, - L0SearchResult, - L0FtsResult, -} from "./types.js"; - -// ============================ -// Types -// ============================ - -export interface VectorSearchResult { - record_id: string; - content: string; - type: string; - priority: number; - scene_name: string; - /** Cosine similarity score (1.0 - cosine_distance) */ - score: number; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - session_key: string; - session_id: string; - /** Raw metadata JSON string (e.g., contains activity_start_time / activity_end_time for episodic) */ - metadata_json: string; -} - -/** L0 single-message vector search result. */ -export interface L0VectorSearchResult { - record_id: string; - session_key: string; - session_id: string; - role: string; - message_text: string; - /** Cosine similarity score (1.0 - cosine_distance) */ - score: number; - recorded_at: string; - /** Original message timestamp (epoch ms) */ - timestamp: number; -} - -/** Raw row returned by L1 record queries (column names match SQLite schema). */ -export interface L1RecordRow { - record_id: string; - content: string; - type: string; - priority: number; - scene_name: string; - session_key: string; - session_id: string; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - created_time: string; - updated_time: string; - metadata_json: string; -} - -export interface L0RecordRow { - record_id: string; - session_key: string; - session_id: string; - role: string; - message_text: string; - recorded_at: string; - timestamp: number; -} - -/** Filter options for querying L1 records from SQLite. */ -export interface L1QueryFilter { - /** If provided, only return records for this session key (conversation channel). */ - sessionKey?: string; - /** If provided, only return records for this session ID (single conversation instance). */ - sessionId?: string; - /** If provided, only return records with updated_time strictly after this ISO 8601 UTC timestamp. */ - updatedAfter?: string; -} - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -const TAG = "[memory-tdai][sqlite]"; - -/** Persisted metadata about the embedding provider used to generate stored vectors. */ -interface EmbeddingMeta { - provider: string; - model: string; - dimensions: number; -} - -/** Result of VectorStore.init() โ€” indicates whether a re-embed is needed. */ -export interface VectorStoreInitResult { - /** - * `true` if the embedding provider/model/dimensions changed since - * the vectors were last written. Callers should re-embed all texts - * (via `reindexAll()`) after receiving this flag. - */ - needsReindex: boolean; - /** Human-readable reason (for logging). */ - reason?: string; -} - -// Use createRequire to load the experimental node:sqlite module -const require = createRequire(import.meta.url); - -function requireNodeSqlite(): typeof import("node:sqlite") { - return require("node:sqlite") as typeof import("node:sqlite"); -} - -// ============================ -// FTS5 helpers (adapted from openclaw core hybrid.ts) -// ============================ - -// โ”€โ”€ Chinese word segmentation (jieba) โ”€โ”€ -// Lazy-loaded singleton: initialised on first call to `buildFtsQuery`. -// If @node-rs/jieba is unavailable, falls back to Unicode-regex splitting. - -interface JiebaInstance { - cutForSearch(text: string, hmm: boolean): string[]; -} - -let _jieba: JiebaInstance | null | undefined; // undefined = not yet tried - -function getJieba(): JiebaInstance | null { - if (_jieba !== undefined) return _jieba; - try { - // eslint-disable-next-line @typescript-eslint/no-require-imports - const { Jieba } = require("@node-rs/jieba"); - // eslint-disable-next-line @typescript-eslint/no-require-imports - const { dict } = require("@node-rs/jieba/dict"); - _jieba = Jieba.withDict(dict) as JiebaInstance; - } catch { - _jieba = null; // mark as unavailable โ€” won't retry - } - return _jieba; -} - -/** - * Common Chinese stop-words that add noise to FTS5 queries. - * Kept small on purpose โ€” only high-frequency function words. - */ -const ZH_STOP_WORDS = new Set([ - "็š„", "ไบ†", "ๅœจ", "ๆ˜ฏ", "ๆˆ‘", "ๆœ‰", "ๅ’Œ", "ๅฐฑ", "ไธ", "ไบบ", "้ƒฝ", "ไธ€", - "ไธ€ไธช", "ไธŠ", "ไนŸ", "ๅพˆ", "ๅˆฐ", "่ฏด", "่ฆ", "ๅŽป", "ไฝ ", "ไผš", "็€", - "ๆฒกๆœ‰", "็œ‹", "ๅฅฝ", "่‡ชๅทฑ", "่ฟ™", "ไป–", "ๅฅน", "ๅฎƒ", "ไปฌ", "้‚ฃ", - "ๅ—", "ๅง", "ๅ‘ข", "ๅ•Š", "ๅ‘€", "ๅ“ฆ", "ๅ—ฏ", -]); - -/** - * Build an FTS5 MATCH query from raw text. - * - * When `@node-rs/jieba` is available, uses jieba's search-engine mode - * (`cutForSearch`) for accurate Chinese word segmentation, producing - * much better recall than the previous regex-only approach. - * - * Falls back to Unicode-regex splitting (`/[\p{L}\p{N}_]+/gu`) if - * jieba is not installed. - * - * Tokens are OR-joined as quoted FTS5 phrase terms so that a document - * matching *any* token is returned. BM25 naturally ranks documents that - * match more tokens higher, so precision is preserved while recall is - * significantly improved โ€” especially for longer queries and when running - * in FTS-only fallback mode (no embedding available). - * - * Example (with jieba): - * "็”จๆˆทๅ–œๆฌข็ผ–็จ‹ๅ’ŒTypeScript" โ†’ '"็”จๆˆท" OR "ๅ–œๆฌข" OR "็ผ–็จ‹" OR "TypeScript"' - * Example (fallback): - * "ๆ—…่กŒ่ฎกๅˆ’ API" โ†’ '"ๆ—…่กŒ่ฎกๅˆ’" OR "API"' - */ -export function buildFtsQuery(raw: string): string | null { - const jieba = getJieba(); - - let tokens: string[]; - if (jieba) { - // jieba cutForSearch: splits long words further for better recall - // e.g. "ๅŒ—ไบฌ็ƒค้ธญ" โ†’ ["ๅŒ—ไบฌ", "็ƒค้ธญ", "ๅŒ—ไบฌ็ƒค้ธญ"] - tokens = jieba - .cutForSearch(raw, true) - .map((t) => t.trim()) - .filter((t) => { - if (!t) return false; - // Remove pure whitespace / punctuation tokens - if (!/[\p{L}\p{N}]/u.test(t)) return false; - // Remove common Chinese stop-words to reduce noise - if (ZH_STOP_WORDS.has(t)) return false; - return true; - }); - // Deduplicate (cutForSearch may produce duplicates for sub-words) - tokens = [...new Set(tokens)]; - } else { - // Fallback: simple Unicode regex split - tokens = - raw - .match(/[\p{L}\p{N}_]+/gu) - ?.map((t) => t.trim()) - .filter(Boolean) ?? []; - } - - if (tokens.length === 0) return null; - const quoted = tokens.map((t) => `"${t.replaceAll('"', "")}"`); - return quoted.join(" OR "); -} - -/** - * Tokenize text for FTS5 indexing (write-side). - * - * Uses jieba `cutForSearch()` (search-engine mode) to segment Chinese text, - * then joins tokens with spaces. The resulting string is stored in the FTS5 - * `content` column so that `unicode61` tokenizer can split it into meaningful - * words โ€” including both full words and their sub-words. - * - * Using `cutForSearch` (instead of `cut`) ensures that the index contains - * the same sub-word tokens that `buildFtsQuery()` produces on the query side. - * For example, "ไบบๅทฅๆ™บ่ƒฝ" is indexed as "ไบบๅทฅ ๆ™บ่ƒฝ ไบบๅทฅๆ™บ่ƒฝ", so queries for - * either the full term or sub-words will match. - * - * Falls back to the original text if jieba is unavailable. - * - * Example (with jieba): - * "็”จๆˆทไบ”ๆœˆๅŽปๆ—ฅๆœฌๆ—…่กŒ" โ†’ "็”จๆˆท ไบ”ๆœˆ ๅŽป ๆ—ฅๆœฌ ๆ—…่กŒ" - * "ไบบๅทฅๆ™บ่ƒฝ็š„ๅˆ†ๆ”ฏ" โ†’ "ไบบๅทฅ ๆ™บ่ƒฝ ไบบๅทฅๆ™บ่ƒฝ ็š„ ๅˆ†ๆ”ฏ" - * Example (fallback): - * "็”จๆˆทไบ”ๆœˆๅŽปๆ—ฅๆœฌๆ—…่กŒ" โ†’ "็”จๆˆทไบ”ๆœˆๅŽปๆ—ฅๆœฌๆ—…่กŒ" (unchanged) - */ -export function tokenizeForFts(raw: string): string { - const jieba = getJieba(); - if (!jieba) return raw; - - // Use `cutForSearch` (search-engine mode) for indexing โ€” it produces both - // full words AND their sub-word components. This ensures that query-side - // tokens (also produced by `cutForSearch` in `buildFtsQuery`) will always - // find a match in the index. - const tokens = jieba.cutForSearch(raw, true); - - // Join with spaces so `unicode61` tokenizer can split them. - // Punctuation tokens are kept โ€” unicode61 treats them as separators anyway. - return tokens.join(" "); -} - -/** - * Reset jieba state so next call to `buildFtsQuery` re-initialises. - * Exported for testing only. - * @internal - */ -export function _resetJiebaForTest(): void { - _jieba = undefined; -} - -/** - * Override jieba instance (or set to `null` to force fallback). - * Exported for testing only. - * @internal - */ -export function _setJiebaForTest(instance: JiebaInstance | null): void { - _jieba = instance; -} - -/** - * Convert a BM25 rank (negative = more relevant) to a 0โ€“1 score. - * Mirrors the formula in openclaw core `hybrid.ts`. - */ -export function bm25RankToScore(rank: number): number { - if (!Number.isFinite(rank)) return 1 / (1 + 999); - if (rank < 0) { - const relevance = -rank; - return relevance / (1 + relevance); - } - return 1 / (1 + rank); -} - -/** FTS5 search result for L1 records. */ -export interface FtsSearchResult { - record_id: string; - content: string; - type: string; - priority: number; - scene_name: string; - /** BM25-derived score (0โ€“1, higher is better) */ - score: number; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - session_key: string; - session_id: string; - metadata_json: string; -} - -/** FTS5 search result for L0 records. */ -export interface L0FtsSearchResult { - record_id: string; - session_key: string; - session_id: string; - role: string; - message_text: string; - /** BM25-derived score (0โ€“1, higher is better) */ - score: number; - recorded_at: string; - timestamp: number; -} - -// ============================ -// VectorStore class -// ============================ - -export class VectorStore implements IMemoryStore { - private db: DatabaseSync; - private readonly dimensions: number; - private readonly logger?: Logger; - - /** @see IMemoryStore.supportsDeferredEmbedding */ - readonly supportsDeferredEmbedding = true; - - /** - * When `true`, the store is in a degraded state (e.g. sqlite-vec failed to - * load, or init() encountered an unrecoverable error). All public methods - * become safe no-ops so the plugin never blocks the main OpenClaw flow. - */ - private degraded = false; - - /** Tracks whether close() has been called to prevent double-close errors. */ - private closed = false; - - /** - * `true` when vec0 virtual tables (l1_vec / l0_vec) have been created and - * their prepared statements are ready. When `dimensions === 0` (i.e. - * provider="none"), vec0 tables are deferred and this stays `false`. - */ - private vecTablesReady = false; - - // Prepared statements โ€” L1 (initialized in init()) - private stmtUpsertMeta!: StatementSync; - private stmtDeleteVec?: StatementSync; // optional โ€” only set when vecTablesReady - private stmtInsertVec?: StatementSync; // optional โ€” only set when vecTablesReady - private stmtDeleteMeta!: StatementSync; - private stmtGetMeta!: StatementSync; - private stmtSearchVec?: StatementSync; // optional โ€” only set when vecTablesReady - private stmtQueryBySessionId!: StatementSync; - private stmtQueryBySessionIdSince!: StatementSync; - private stmtQueryBySessionKey!: StatementSync; - private stmtQueryBySessionKeySince!: StatementSync; - private stmtQueryAll!: StatementSync; - private stmtQueryAllSince!: StatementSync; - - // Prepared statements โ€” L0 (initialized in init()) - private stmtL0UpsertMeta!: StatementSync; - private stmtL0DeleteVec?: StatementSync; // optional โ€” only set when vecTablesReady - private stmtL0InsertVec?: StatementSync; // optional โ€” only set when vecTablesReady - private stmtL0DeleteMeta!: StatementSync; - private stmtL0GetMeta!: StatementSync; - private stmtL0SearchVec?: StatementSync; // optional โ€” only set when vecTablesReady - /** L0 query for L1 runner: all messages for a session key */ - private stmtL0QueryAll!: StatementSync; - /** L0 query for L1 runner: messages after a timestamp cursor */ - private stmtL0QueryAfter!: StatementSync; - /** L1 cursor-based pagination for migration (by PK) */ - private stmtL1QueryMigrationCursor!: StatementSync; - /** L0 cursor-based pagination for migration (by PK) */ - private stmtL0QueryMigrationCursor!: StatementSync; - - // FTS5 tables availability flag (created best-effort โ€” may be false if fts5 is not compiled in) - private ftsAvailable = false; - - // Prepared statements โ€” FTS5 L1 (initialized in init()) - private stmtL1FtsInsert!: StatementSync; - private stmtL1FtsDelete!: StatementSync; - private stmtL1FtsSearch!: StatementSync; - - // Prepared statements โ€” FTS5 L0 (initialized in init()) - private stmtL0FtsInsert!: StatementSync; - private stmtL0FtsDelete!: StatementSync; - private stmtL0FtsSearch!: StatementSync; - - /** - * Create a VectorStore instance. - * - * Note: After construction, you MUST call `init()` to load the sqlite-vec - * extension and create the schema. - */ - constructor(dbPath: string, dimensions: number, logger?: Logger) { - this.dimensions = dimensions; - this.logger = logger; - - // Open database with extension support enabled - const { DatabaseSync: DbSync } = requireNodeSqlite(); - this.db = new DbSync(dbPath, { allowExtension: true }); - - // Set busy timeout so concurrent processes retry instead of failing with SQLITE_BUSY - this.db.exec("PRAGMA busy_timeout = 5000"); - - // Enable WAL mode for better concurrent read performance - this.db.exec("PRAGMA journal_mode = WAL"); - - // Cap page cache at 64 MB - this.db.exec("PRAGMA cache_size = -65536"); - - // Cap memory-mapped I/O at 128 MB to bound RSS growth - this.db.exec("PRAGMA mmap_size = 134217728"); - - // Auto-checkpoint WAL every 1000 pages (~4 MB) to keep WAL file compact - this.db.exec("PRAGMA wal_autocheckpoint = 1000"); - } - - /** - * Whether the store is in degraded mode (e.g. sqlite-vec failed to load). - * When degraded, all write/search operations become safe no-ops. - */ - isDegraded(): boolean { - return this.degraded; - } - - - /** - * Load sqlite-vec extension and initialize database schema. - * Must be called once after construction. - * - * @param providerInfo Current embedding provider info. When provided, - * the store compares it against the persisted metadata. If the provider, - * model, or dimensions changed, the vector tables are dropped and - * re-created with the new dimensions, and `needsReindex: true` is returned - * so the caller can schedule a full re-embed. - */ - init(providerInfo?: EmbeddingProviderInfo): VectorStoreInitResult { - // Load sqlite-vec extension (same approach as root project's sqlite-vec.ts) - try { - // eslint-disable-next-line @typescript-eslint/no-require-imports - const sqliteVec = require("sqlite-vec"); - this.db.enableLoadExtension(true); - sqliteVec.load(this.db); - } catch (err) { - const message = err instanceof Error ? err.message : String(err); - this.logger?.error( - `${TAG} Failed to load sqlite-vec extension: ${message}. ` + - `VectorStore entering degraded mode โ€” all operations will be no-ops.`, - ); - this.degraded = true; - return { needsReindex: false, reason: `sqlite-vec load failed: ${message}` }; - } - - // โ”€โ”€ Schema creation & prepared statements โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - // Wrapped in try-catch: if anything fails during schema init (e.g. the DB - // is corrupted, disk full, etc.), we degrade gracefully instead of crashing. - try { - return this.initSchema(providerInfo); - } catch (err) { - const message = err instanceof Error ? err.message : String(err); - this.logger?.error( - `${TAG} Schema initialization failed: ${message}. ` + - `VectorStore entering degraded mode.`, - ); - this.degraded = true; - return { needsReindex: false, reason: `schema init failed: ${message}` }; - } - } - - /** - * Internal schema initialization โ€” separated from init() so we can - * catch errors at the top level and degrade gracefully. - */ - private initSchema(providerInfo?: EmbeddingProviderInfo): VectorStoreInitResult { - // Tracks which provider/model/dimensions were used to generate vectors. - this.db.exec(` - CREATE TABLE IF NOT EXISTS embedding_meta ( - key TEXT PRIMARY KEY, - value TEXT NOT NULL - ) - `); - - // Detect whether re-index is needed - let needsReindex = false; - let reindexReason: string | undefined; - - const savedMeta = this.readEmbeddingMeta(); - - if (providerInfo) { - if (savedMeta) { - const providerChanged = savedMeta.provider !== providerInfo.provider; - const modelChanged = savedMeta.model !== providerInfo.model; - const dimsChanged = savedMeta.dimensions !== this.dimensions; - - if (providerChanged || modelChanged || dimsChanged) { - const reasons: string[] = []; - if (providerChanged) reasons.push(`provider: ${savedMeta.provider} โ†’ ${providerInfo.provider}`); - if (modelChanged) reasons.push(`model: ${savedMeta.model} โ†’ ${providerInfo.model}`); - if (dimsChanged) reasons.push(`dimensions: ${savedMeta.dimensions} โ†’ ${this.dimensions}`); - reindexReason = reasons.join(", "); - - this.logger?.info( - `${TAG} Embedding config changed (${reindexReason}). ` + - `Dropping vector tables for rebuild...`, - ); - - // Drop and re-create vector tables with new dimensions - this.dropVectorTables(); - needsReindex = true; - } - } else { - // No saved meta โ€” first run or legacy DB without meta table. - // Two cases require dropping vector tables: - // 1. Existing data created without meta tracking (legacy DB) โ€” need re-embed - // 2. vec0 tables exist with wrong dimensions (e.g. previously created with - // provider="none" placeholder 768D, now switching to a real provider - // with different dimensions) โ€” must rebuild even if data tables are empty - const l1Count = this.tableRowCount("l1_records"); - const l0Count = this.tableRowCount("l0_conversations"); - const existingVecDims = this.getVecTableDimensions(); - - if (l1Count > 0 || l0Count > 0) { - this.logger?.info( - `${TAG} No embedding_meta found but existing data exists ` + - `(L1=${l1Count}, L0=${l0Count}). Dropping vector tables for safety...`, - ); - this.dropVectorTables(); - needsReindex = true; - reindexReason = "legacy DB without embedding_meta โ€” cannot verify vector compatibility"; - } else if (existingVecDims !== null && existingVecDims !== this.dimensions) { - // vec0 tables exist (from a previous provider="none" placeholder or - // different config) but with mismatched dimensions. Drop them so they - // get re-created with the correct dimensions below. - this.logger?.info( - `${TAG} vec0 table dimension mismatch (existing=${existingVecDims}, ` + - `required=${this.dimensions}). Dropping vector tables for rebuild...`, - ); - this.dropVectorTables(); - // No needsReindex โ€” there's no data to re-embed - } - } - } - - // โ”€โ”€ L1 schema โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - // Metadata table - this.db.exec(` - CREATE TABLE IF NOT EXISTS l1_records ( - record_id TEXT PRIMARY KEY, - content TEXT NOT NULL, - type TEXT DEFAULT '', - priority INTEGER DEFAULT 50, - scene_name TEXT DEFAULT '', - session_key TEXT DEFAULT '', - session_id TEXT DEFAULT '', - timestamp_str TEXT DEFAULT '', - timestamp_start TEXT DEFAULT '', - timestamp_end TEXT DEFAULT '', - created_time TEXT DEFAULT '', - updated_time TEXT DEFAULT '', - metadata_json TEXT DEFAULT '{}' - ) - `); - - // Indexes for common queries - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_type ON l1_records(type)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_session_key ON l1_records(session_key)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_session_id ON l1_records(session_id)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_scene ON l1_records(scene_name)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_ts_start ON l1_records(timestamp_start)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_ts_end ON l1_records(timestamp_end)"); - // Composite index: session_id exact match + updated_time range scan (for incremental L2 queries) - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_session_updated ON l1_records(session_id, updated_time)"); - // Composite index: session_key exact match + updated_time range scan (for pipeline cursor queries) - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_sessionkey_updated ON l1_records(session_key, updated_time)"); - - // Vector virtual table (cosine distance) โ€” only created when dimensions > 0. - // When provider="none", dimensions=0 and vec0 tables are deferred until a - // real embedding provider is configured. - if (this.dimensions > 0) { - this.db.exec(` - CREATE VIRTUAL TABLE IF NOT EXISTS l1_vec USING vec0( - record_id TEXT PRIMARY KEY, - embedding float[${this.dimensions}] distance_metric=cosine, - updated_time TEXT DEFAULT '' - ) - `); - } - - // Prepare statements for reuse - this.stmtUpsertMeta = this.db.prepare(` - INSERT INTO l1_records ( - record_id, content, type, priority, scene_name, session_key, session_id, - timestamp_str, timestamp_start, timestamp_end, - created_time, updated_time, metadata_json - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - ON CONFLICT(record_id) DO UPDATE SET - content=excluded.content, - type=excluded.type, - priority=excluded.priority, - scene_name=excluded.scene_name, - timestamp_str=excluded.timestamp_str, - timestamp_start=excluded.timestamp_start, - timestamp_end=excluded.timestamp_end, - updated_time=excluded.updated_time, - metadata_json=excluded.metadata_json - `); - - if (this.dimensions > 0) { - this.stmtDeleteVec = this.db.prepare("DELETE FROM l1_vec WHERE record_id = ?"); - this.stmtInsertVec = this.db.prepare("INSERT INTO l1_vec (record_id, embedding, updated_time) VALUES (?, ?, ?)"); - } - this.stmtDeleteMeta = this.db.prepare("DELETE FROM l1_records WHERE record_id = ?"); - - this.stmtGetMeta = this.db.prepare(` - SELECT content, type, priority, scene_name, session_key, session_id, - timestamp_str, timestamp_start, timestamp_end, metadata_json - FROM l1_records WHERE record_id = ? - `); - - if (this.dimensions > 0) { - this.stmtSearchVec = this.db.prepare(` - SELECT record_id, distance - FROM l1_vec - WHERE embedding MATCH ? - AND k = ? - ORDER BY distance - `); - } - - // โ”€โ”€ L0 schema โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - // L0 metadata table: stores individual messages for vector search - this.db.exec(` - CREATE TABLE IF NOT EXISTS l0_conversations ( - record_id TEXT PRIMARY KEY, - session_key TEXT NOT NULL, - session_id TEXT DEFAULT '', - role TEXT NOT NULL DEFAULT '', - message_text TEXT NOT NULL, - recorded_at TEXT DEFAULT '', - timestamp INTEGER DEFAULT 0 - ) - `); - - // Migration: add timestamp column if missing (existing DBs pre-v3.x) - try { - this.db.exec("ALTER TABLE l0_conversations ADD COLUMN timestamp INTEGER DEFAULT 0"); - this.logger?.info(`${TAG} Migrated l0_conversations: added timestamp column`); - } catch { - // Column already exists โ€” expected on non-first run - } - - // Indexes for L0 queries - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l0_session ON l0_conversations(session_key)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l0_session_id ON l0_conversations(session_id)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l0_recorded ON l0_conversations(recorded_at)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l0_timestamp ON l0_conversations(timestamp)"); - - // L0 vector virtual table (cosine distance, same dimensions as L1) โ€” deferred when dimensions=0 - if (this.dimensions > 0) { - this.db.exec(` - CREATE VIRTUAL TABLE IF NOT EXISTS l0_vec USING vec0( - record_id TEXT PRIMARY KEY, - embedding float[${this.dimensions}] distance_metric=cosine, - recorded_at TEXT DEFAULT '' - ) - `); - } - - // L0 prepared statements - this.stmtL0UpsertMeta = this.db.prepare(` - INSERT INTO l0_conversations ( - record_id, session_key, session_id, role, message_text, recorded_at, timestamp - ) VALUES (?, ?, ?, ?, ?, ?, ?) - ON CONFLICT(record_id) DO UPDATE SET - message_text=excluded.message_text, - recorded_at=excluded.recorded_at, - timestamp=excluded.timestamp - `); - - if (this.dimensions > 0) { - this.stmtL0DeleteVec = this.db.prepare("DELETE FROM l0_vec WHERE record_id = ?"); - this.stmtL0InsertVec = this.db.prepare("INSERT INTO l0_vec (record_id, embedding, recorded_at) VALUES (?, ?, ?)"); - } - this.stmtL0DeleteMeta = this.db.prepare("DELETE FROM l0_conversations WHERE record_id = ?"); - - this.stmtL0GetMeta = this.db.prepare(` - SELECT session_key, session_id, role, message_text, recorded_at, timestamp - FROM l0_conversations WHERE record_id = ? - `); - - if (this.dimensions > 0) { - this.stmtL0SearchVec = this.db.prepare(` - SELECT record_id, distance - FROM l0_vec - WHERE embedding MATCH ? - AND k = ? - ORDER BY distance - `); - } - - // L0 query statements for L1 runner (newest-first + LIMIT to bound memory) - // Sort/filter by recorded_at (write time) instead of timestamp (conversation time) - // because L1 cursor uses recorded_at semantics. ISO 8601 string comparison preserves time order. - this.stmtL0QueryAll = this.db.prepare(` - SELECT record_id, session_key, session_id, role, message_text, recorded_at, timestamp - FROM l0_conversations - WHERE session_key = ? - ORDER BY recorded_at DESC - LIMIT ? - `); - - this.stmtL0QueryAfter = this.db.prepare(` - SELECT record_id, session_key, session_id, role, message_text, recorded_at, timestamp - FROM l0_conversations - WHERE session_key = ? AND recorded_at > ? - ORDER BY recorded_at DESC - LIMIT ? - `); - - this.stmtL0QueryMigrationCursor = this.db.prepare(` - SELECT record_id, session_key, session_id, role, message_text, recorded_at, timestamp - FROM l0_conversations - WHERE record_id > ? - ORDER BY record_id ASC - LIMIT ? - `); - - // โ”€โ”€ FTS5 tables (best-effort โ€” gracefully degrade if fts5 is not compiled in) โ”€โ”€ - // Schema v2: `content` column stores jieba-segmented text (for indexing), - // `content_original` (UNINDEXED) stores the raw text (for display). - // If old v1 tables exist (no content_original column), drop + recreate. - try { - // โ”€โ”€ Migrate old FTS5 tables (v1 โ†’ v2) โ”€โ”€ - // v1 tables stored raw text in the `content` column. v2 stores segmented - // text in `content` and raw text in `content_original` / `message_text_original`. - // FTS5 virtual tables don't support ALTER TABLE ADD COLUMN, so we must - // drop and recreate. The data will be repopulated by `rebuildFtsIndex()`. - const needsFtsRebuild = this.migrateFtsTablesIfNeeded(); - - // L1 FTS5 virtual table (v2 schema) - this.db.exec(` - CREATE VIRTUAL TABLE IF NOT EXISTS l1_fts USING fts5( - content, - content_original UNINDEXED, - record_id UNINDEXED, - type UNINDEXED, - priority UNINDEXED, - scene_name UNINDEXED, - session_key UNINDEXED, - session_id UNINDEXED, - timestamp_str UNINDEXED, - timestamp_start UNINDEXED, - timestamp_end UNINDEXED, - metadata_json UNINDEXED - ) - `); - - // L0 FTS5 virtual table (v2 schema) - this.db.exec(` - CREATE VIRTUAL TABLE IF NOT EXISTS l0_fts USING fts5( - message_text, - message_text_original UNINDEXED, - record_id UNINDEXED, - session_key UNINDEXED, - session_id UNINDEXED, - role UNINDEXED, - recorded_at UNINDEXED, - timestamp UNINDEXED - ) - `); - - // L1 FTS prepared statements - this.stmtL1FtsInsert = this.db.prepare(` - INSERT INTO l1_fts (content, content_original, record_id, type, priority, scene_name, - session_key, session_id, timestamp_str, timestamp_start, timestamp_end, metadata_json) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - `); - - this.stmtL1FtsDelete = this.db.prepare("DELETE FROM l1_fts WHERE record_id = ?"); - - this.stmtL1FtsSearch = this.db.prepare(` - SELECT record_id, content_original AS content, type, priority, scene_name, - session_key, session_id, timestamp_str, timestamp_start, timestamp_end, - metadata_json, - bm25(l1_fts) AS rank - FROM l1_fts - WHERE l1_fts MATCH ? - ORDER BY rank ASC - LIMIT ? - `); - - // L0 FTS prepared statements - this.stmtL0FtsInsert = this.db.prepare(` - INSERT INTO l0_fts (message_text, message_text_original, record_id, session_key, session_id, role, recorded_at, timestamp) - VALUES (?, ?, ?, ?, ?, ?, ?, ?) - `); - - this.stmtL0FtsDelete = this.db.prepare("DELETE FROM l0_fts WHERE record_id = ?"); - - this.stmtL0FtsSearch = this.db.prepare(` - SELECT record_id, message_text_original AS message_text, session_key, session_id, role, recorded_at, timestamp, - bm25(l0_fts) AS rank - FROM l0_fts - WHERE l0_fts MATCH ? - ORDER BY rank ASC - LIMIT ? - `); - - this.ftsAvailable = true; - this.logger?.debug?.(`${TAG} FTS5 tables initialized (l1_fts, l0_fts) [schema v2 โ€” jieba segmented]`); - - // Rebuild FTS index if migrated from v1 or tables were freshly created - if (needsFtsRebuild) { - this.rebuildFtsIndex(); - } - } catch (err) { - const message = err instanceof Error ? err.message : String(err); - this.ftsAvailable = false; - this.logger?.warn( - `${TAG} FTS5 tables NOT available (fts5 may not be compiled in): ${message}. ` + - `FTS-based keyword search will be unavailable; recall will use in-memory scoring if needed.`, - ); - } - - // Save current embedding meta (write after schema is ready) - if (providerInfo) { - this.writeEmbeddingMeta({ - provider: providerInfo.provider, - model: providerInfo.model, - dimensions: this.dimensions, - }); - } - - // Mark vec0 tables as ready only when they were actually created - this.vecTablesReady = this.dimensions > 0; - // L1 query statements (for l1-reader) - const l1QueryCols = `record_id, content, type, priority, scene_name, session_key, session_id, - timestamp_str, timestamp_start, timestamp_end, - created_time, updated_time, metadata_json`; - - this.stmtQueryBySessionId = this.db.prepare(` - SELECT ${l1QueryCols} FROM l1_records - WHERE session_id = ? - ORDER BY updated_time ASC - `); - - this.stmtQueryBySessionIdSince = this.db.prepare(` - SELECT ${l1QueryCols} FROM l1_records - WHERE session_id = ? AND updated_time > ? - ORDER BY updated_time ASC - `); - - this.stmtQueryBySessionKey = this.db.prepare(` - SELECT ${l1QueryCols} FROM l1_records - WHERE session_key = ? - ORDER BY updated_time ASC - `); - - this.stmtQueryBySessionKeySince = this.db.prepare(` - SELECT ${l1QueryCols} FROM l1_records - WHERE session_key = ? AND updated_time > ? - ORDER BY updated_time ASC - `); - - this.stmtQueryAll = this.db.prepare(` - SELECT ${l1QueryCols} FROM l1_records - ORDER BY updated_time ASC - `); - - this.stmtQueryAllSince = this.db.prepare(` - SELECT ${l1QueryCols} FROM l1_records - WHERE updated_time > ? - ORDER BY updated_time ASC - `); - - this.stmtL1QueryMigrationCursor = this.db.prepare(` - SELECT ${l1QueryCols} FROM l1_records - WHERE record_id > ? - ORDER BY record_id ASC - LIMIT ? - `); - - this.logger?.debug?.(`${TAG} Initialized (dimensions=${this.dimensions})`); - - return { needsReindex, reason: reindexReason }; - } - - // โ”€โ”€ Embedding meta helpers โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - private readEmbeddingMeta(): EmbeddingMeta | null { - try { - const row = this.db - .prepare("SELECT value FROM embedding_meta WHERE key = ?") - .get("embedding_provider_info") as { value: string } | undefined; - if (!row) return null; - return JSON.parse(row.value) as EmbeddingMeta; - } catch { - return null; - } - } - - private writeEmbeddingMeta(meta: EmbeddingMeta): void { - this.db.prepare( - "INSERT INTO embedding_meta (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value=excluded.value", - ).run("embedding_provider_info", JSON.stringify(meta)); - } - - /** Allowed table names for row counting (whitelist to prevent SQL injection). */ - private static readonly COUNTABLE_TABLES = new Set(["l1_records", "l0_conversations"]); - - /** - * Extra rows to retrieve from vec0 KNN search to compensate for legacy - * zero-vector placeholders that may still linger from older data. - */ - private static readonly ZERO_VEC_BUFFER = 10; - - /** Default result limit for FTS5 keyword searches. */ - private static readonly FTS_DEFAULT_LIMIT = 20; - - private tableRowCount(table: string): number { - if (!VectorStore.COUNTABLE_TABLES.has(table)) { - this.logger?.warn(`${TAG} tableRowCount: rejected unknown table name "${table}"`); - return 0; - } - try { - const row = this.db - .prepare(`SELECT COUNT(*) AS cnt FROM ${table}`) - .get() as { cnt: number } | undefined; - return row?.cnt ?? 0; - } catch { - return 0; - } - } - - /** - * Detect the embedding dimension of an existing vec0 table by inspecting - * the DDL stored in sqlite_master. Returns `null` if the table doesn't - * exist or the dimension cannot be determined. - * - * The vec0 DDL looks like: - * CREATE VIRTUAL TABLE l1_vec USING vec0(... embedding float[768] ...) - * We parse the number inside `float[N]`. - */ - private getVecTableDimensions(): number | null { - try { - const row = this.db - .prepare("SELECT sql FROM sqlite_master WHERE type='table' AND name=?") - .get("l1_vec") as { sql: string } | undefined; - if (!row?.sql) return null; - const match = row.sql.match(/float\[(\d+)\]/); - return match ? Number(match[1]) : null; - } catch { - return null; - } - } - - /** - * Drop both L1 and L0 vector virtual tables. - * Metadata tables (l1_records, l0_conversations) are preserved โ€” only - * the vec0 tables need to be rebuilt with the new dimensions. - */ - private dropVectorTables(): void { - this.db.exec("DROP TABLE IF EXISTS l1_vec"); - this.db.exec("DROP TABLE IF EXISTS l0_vec"); - this.logger?.info(`${TAG} Dropped vector tables (l1_vec, l0_vec)`); - } - - /** - * Write or update a memory record (metadata + vector). - * Uses a manual transaction for atomicity. - * - * If `embedding` is `undefined` or a zero vector (all elements are 0), only - * the metadata row is written โ€” the vec0 table is left untouched. This - * allows callers without an EmbeddingService to still persist metadata + FTS - * without constructing a throwaway zero-vector, and prevents placeholder - * zero vectors (from embedding-service failures) from polluting KNN search - * results with null / NaN distances. - * - * **Fault-tolerant**: catches all errors internally so that a vector store - * failure never propagates to the caller / main OpenClaw flow. - * Returns `true` on success, `false` on failure (logged as warning). - */ - upsertL1(record: MemoryRecord, embedding: Float32Array | undefined): boolean { - if (this.degraded) { - this.logger?.warn(`${TAG} [L1-upsert] SKIPPED (degraded mode) id=${record.id}`); - return false; - } - try { - const { id: recordId, timestamps } = record; - const tsStr = timestamps[0] ?? ""; - const tsStart = - timestamps.length > 0 - ? timestamps.reduce((a, b) => (a < b ? a : b)) - : tsStr; - const tsEnd = - timestamps.length > 0 - ? timestamps.reduce((a, b) => (a > b ? a : b)) - : tsStr; - - const skipVec = !embedding || embedding.every(v => v === 0) || !this.vecTablesReady; - - this.logger?.debug?.( - `${TAG} [L1-upsert] START id=${recordId}, type=${record.type}, ` + - `content="${record.content.slice(0, 60)}..."` + - (embedding - ? `, embeddingDims=${embedding.length}, ` + - `embeddingNorm=${Math.sqrt(Array.from(embedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}` + - `${skipVec ? " (ZERO VECTOR or vec tables not ready โ€” vec write will be skipped)" : ""}` - : " (no embedding โ€” metadata-only write)"), - ); - - this.db.exec("BEGIN"); - try { - // Upsert metadata (INSERT OR UPDATE) - this.stmtUpsertMeta.run( - recordId, - record.content, - record.type, - record.priority, - record.scene_name, - record.sessionKey, - record.sessionId, - tsStr, - tsStart, - tsEnd, - record.createdAt, - record.updatedAt, - JSON.stringify(record.metadata), - ); - - if (!skipVec) { - // vec0 does not support ON CONFLICT โ†’ delete then insert - this.stmtDeleteVec!.run(recordId); - this.stmtInsertVec!.run(recordId, Buffer.from(embedding!.buffer), record.updatedAt); - } else { - this.logger?.debug?.( - `${TAG} [L1-upsert] Skipping vec write (${embedding ? "zero vector" : "no embedding"}) id=${recordId}`, - ); - } - - // Sync FTS5 (delete + re-insert to handle updates) - if (this.ftsAvailable) { - try { - this.stmtL1FtsDelete.run(recordId); - this.stmtL1FtsInsert.run( - tokenizeForFts(record.content), // content โ€” segmented for indexing - record.content, // content_original โ€” raw for display - recordId, - record.type, - record.priority, - record.scene_name, - record.sessionKey, - record.sessionId, - tsStr, - tsStart, - tsEnd, - JSON.stringify(record.metadata), - ); - } catch (ftsErr) { - // FTS write failure is non-fatal โ€” log and continue - this.logger?.warn( - `${TAG} [L1-upsert] FTS write failed (non-fatal) id=${recordId}: ${ftsErr instanceof Error ? ftsErr.message : String(ftsErr)}`, - ); - } - } - - this.db.exec("COMMIT"); - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - this.logger?.debug?.(`${TAG} [L1-upsert] OK id=${recordId}${skipVec ? " (meta-only)" : ""}`); - return true; - } catch (err) { - this.logger?.warn( - `${TAG} [L1-upsert] FAILED (non-fatal) id=${record.id}: ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * Vector similarity search (cosine distance). - * Returns top-k results sorted by similarity (highest first). - * - * **Fault-tolerant**: returns an empty array on any error (e.g. dimension - * mismatch, corrupted DB) so callers can fall back to keyword search. - */ - searchL1Vector(queryEmbedding: Float32Array, topK = 5): VectorSearchResult[] { - if (this.degraded || !this.vecTablesReady) { - if (this.degraded) this.logger?.warn(`${TAG} [L1-search] SKIPPED (degraded mode)`); - return []; - } - try { - // Over-retrieve to compensate for legacy zero-vector placeholders that - // may still exist in the vec0 table. New zero vectors are no longer - // inserted (upsert() skips vec write for zero vectors since v3.x), but - // older data may still contain them โ€” they surface as NULL/NaN distance - // in KNN results. A small buffer of 10 is sufficient for remnants. - // NOTE: "AND distance IS NOT NULL" is NOT usable because vec0 does not - // support that constraint โ€” it causes an empty result set. - const ZERO_VEC_BUFFER = 10; - const retrieveCount = topK + ZERO_VEC_BUFFER; - - this.logger?.debug?.( - `${TAG} [L1-search] START topK=${topK}, retrieveCount=${retrieveCount}, ` + - `queryEmbeddingDims=${queryEmbedding.length}, ` + - `queryNorm=${Math.sqrt(Array.from(queryEmbedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}`, - ); - - const rows = this.stmtSearchVec!.all( - Buffer.from(queryEmbedding.buffer), - retrieveCount, - ) as Array<{ record_id: string; distance: number }>; - - this.logger?.debug?.(`${TAG} [L1-search] vec0 returned ${rows.length} candidate(s)`); - - if (rows.length === 0) return []; - - const results: VectorSearchResult[] = []; - - for (const { record_id, distance } of rows) { - // sqlite-vec returns null distance for zero vectors (cosine undefined when โ€–vโ€–=0). - // Skip these โ€” they are placeholder vectors from embedding-service-unavailable fallback. - if (distance == null || Number.isNaN(distance)) { - this.logger?.warn( - `${TAG} [L1-search] record_id=${record_id} has null/NaN distance (likely zero vector) โ€” skipping`, - ); - continue; - } - - const meta = this.stmtGetMeta.get(record_id) as - | { - content: string; - type: string; - priority: number; - scene_name: string; - session_key: string; - session_id: string; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - metadata_json: string; - } - | undefined; - - if (!meta) { - this.logger?.warn(`${TAG} [L1-search] record_id=${record_id} has vector but NO metadata (orphan)`); - continue; - } - - const score = 1.0 - distance; - this.logger?.debug?.( - `${TAG} [L1-search] HIT id=${record_id}, distance=${distance.toFixed(4)}, score=${score.toFixed(4)}, ` + - `type=${meta.type}, content="${meta.content.slice(0, 60)}..."`, - ); - - results.push({ - record_id, - content: meta.content, - type: meta.type, - priority: meta.priority, - scene_name: meta.scene_name, - score, - timestamp_str: meta.timestamp_str, - timestamp_start: meta.timestamp_start, - timestamp_end: meta.timestamp_end, - session_key: meta.session_key, - session_id: meta.session_id, - metadata_json: meta.metadata_json, - }); - } - - // Trim back to the caller's requested topK (we over-fetched above). - const trimmed = results.slice(0, topK); - this.logger?.info( - `${TAG} [L1-search] DONE returning ${trimmed.length} result(s) (from ${results.length} valid, ${rows.length} raw)`, - ); - return trimmed; - } catch (err) { - this.logger?.warn( - `${TAG} [L1-search] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * Delete a single record (metadata + vector). - * - * **Fault-tolerant**: logs a warning on failure, never throws. - */ - deleteL1(recordId: string): boolean { - if (this.degraded) return false; - try { - this.db.exec("BEGIN"); - try { - this.stmtDeleteMeta.run(recordId); - if (this.vecTablesReady) this.stmtDeleteVec!.run(recordId); - if (this.ftsAvailable) { - try { this.stmtL1FtsDelete.run(recordId); } catch { /* non-fatal */ } - } - this.db.exec("COMMIT"); - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - return true; - } catch (err) { - this.logger?.warn( - `${TAG} delete failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * Delete multiple records (metadata + vector). - * - * **Fault-tolerant**: logs a warning on failure, never throws. - */ - deleteL1Batch(recordIds: string[]): boolean { - if (this.degraded) return false; - if (recordIds.length === 0) return true; - - try { - this.db.exec("BEGIN"); - try { - for (const id of recordIds) { - this.stmtDeleteMeta.run(id); - if (this.vecTablesReady) this.stmtDeleteVec!.run(id); - if (this.ftsAvailable) { - try { this.stmtL1FtsDelete.run(id); } catch { /* non-fatal */ } - } - } - this.db.exec("COMMIT"); - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - return true; - } catch (err) { - this.logger?.warn( - `${TAG} deleteBatch failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * Get the total number of L1 records in the store. - * - * **Fault-tolerant**: returns 0 on failure. - * TTL cleanup by updated_time. - * - * Deletes expired rows from l1_records and matching vectors from l1_vec - * in a single transaction to guarantee consistency. - */ - deleteL1Expired(cutoffIso: string): number { - if (this.degraded) { - this.logger?.warn(`${TAG} [deleteExpired] SKIPPED (degraded mode)`); - return 0; - } - try { - const row = this.db.prepare( - "SELECT COUNT(*) AS cnt FROM l1_records WHERE updated_time != '' AND updated_time < ?", - ).get(cutoffIso) as { cnt: number } | undefined; - const expiredCount = row?.cnt ?? 0; - if (expiredCount <= 0) return 0; - - this.db.exec("BEGIN"); - try { - if (this.vecTablesReady) { - this.db.prepare( - "DELETE FROM l1_vec WHERE updated_time != '' AND updated_time < ?", - ).run(cutoffIso); - } - this.db.prepare( - "DELETE FROM l1_records WHERE updated_time != '' AND updated_time < ?", - ).run(cutoffIso); - this.db.exec("COMMIT"); - return expiredCount; - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - } catch (err) { - this.logger?.warn( - `${TAG} deleteL1ExpiredByUpdatedTime failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return 0; - } - } - - /** - * Get the total number of records in the store. - */ - countL1(): number { - if (this.degraded) return 0; - try { - const row = this.db - .prepare("SELECT COUNT(*) AS cnt FROM l1_records") - .get() as { cnt: number }; - this.logger?.debug?.(`${TAG} [L1-count] total=${row.cnt}`); - return row.cnt; - } catch (err) { - this.logger?.warn( - `${TAG} count failed (non-fatal, returning 0): ${err instanceof Error ? err.message : String(err)}`, - ); - return 0; - } - } - - /** - * Query L1 records with optional session and time filters. - * - * Uses the composite index `idx_l1_session_updated(session_id, updated_time)` - * for efficient filtering. All timestamps are compared as UTC ISO 8601 strings. - * - * **Fault-tolerant**: returns an empty array on any error (degraded mode, DB issues). - */ - queryL1Records(filter?: L1QueryFilter): L1RecordRow[] { - if (this.degraded) { - this.logger?.warn(`${TAG} [L1-query] SKIPPED (degraded mode)`); - return []; - } - try { - const { sessionKey, sessionId, updatedAfter } = filter ?? {}; - - let raw: Record[]; - - // Priority: sessionId > sessionKey (sessionId is more specific) - if (sessionId && updatedAfter) { - raw = this.stmtQueryBySessionIdSince.all(sessionId, updatedAfter) as Record[]; - } else if (sessionId) { - raw = this.stmtQueryBySessionId.all(sessionId) as Record[]; - } else if (sessionKey && updatedAfter) { - raw = this.stmtQueryBySessionKeySince.all(sessionKey, updatedAfter) as Record[]; - } else if (sessionKey) { - raw = this.stmtQueryBySessionKey.all(sessionKey) as Record[]; - } else if (updatedAfter) { - raw = this.stmtQueryAllSince.all(updatedAfter) as Record[]; - } else { - raw = this.stmtQueryAll.all() as Record[]; - } - - // Runtime sanity check: verify first row has expected columns (guards against schema drift) - if (raw.length > 0 && !("record_id" in raw[0] && "content" in raw[0])) { - this.logger?.warn( - `${TAG} [L1-query] Schema mismatch: first row missing expected columns. ` + - `Got keys: [${Object.keys(raw[0]).join(", ")}]`, - ); - return []; - } - - const rows = raw as unknown as L1RecordRow[]; - - this.logger?.info( - `${TAG} [L1-query] filter={sessionKey=${sessionKey ?? "(all)"}, sessionId=${sessionId ?? "(all)"}, updatedAfter=${updatedAfter ?? "(none)"}}, ` + - `returned ${rows.length} record(s)`, - ); - return rows; - } catch (err) { - this.logger?.warn( - `${TAG} [L1-query] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}` - ); - return []; - } - } - - // โ”€โ”€ L0 operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Write or update an L0 single-message record (metadata + vector). - * Uses a manual transaction for atomicity. - * - * If `embedding` is `undefined` or a zero vector (all elements are 0), only - * the metadata row (`l0_conversations`) is written โ€” the vec0 table - * (`l0_vec`) is left untouched. This allows callers without an - * EmbeddingService to still persist metadata + FTS without constructing a - * throwaway zero-vector, and prevents placeholder zero vectors (from - * embedding-service failures) from polluting KNN search results. - * - * **Fault-tolerant**: catches all errors internally, never throws. - * Returns `true` on success, `false` on failure (logged as warning). - */ - upsertL0(record: L0Record, embedding: Float32Array | undefined): boolean { - if (this.degraded) { - this.logger?.warn(`${TAG} [L0-upsert] SKIPPED (degraded mode) id=${record.id}`); - return false; - } - try { - const skipVec = !embedding || embedding.every(v => v === 0) || !this.vecTablesReady; - - this.logger?.debug?.( - `${TAG} [L0-upsert] START id=${record.id}, session=${record.sessionKey}, role=${record.role}, ` + - `text="${record.messageText.slice(0, 60)}..."` + - (embedding - ? `, embeddingDims=${embedding.length}, ` + - `embeddingNorm=${Math.sqrt(Array.from(embedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}` + - `${skipVec ? " (ZERO VECTOR or vec tables not ready โ€” vec write will be skipped)" : ""}` - : " (no embedding โ€” metadata-only write)"), - ); - - this.db.exec("BEGIN"); - try { - this.stmtL0UpsertMeta.run( - record.id, - record.sessionKey, - record.sessionId, - record.role, - record.messageText, - record.recordedAt, - record.timestamp, - ); - - if (!skipVec) { - // vec0 does not support ON CONFLICT โ†’ delete then insert - this.stmtL0DeleteVec!.run(record.id); - this.stmtL0InsertVec!.run(record.id, Buffer.from(embedding!.buffer), record.recordedAt); - } else { - this.logger?.debug?.( - `${TAG} [L0-upsert] Skipping vec write (${embedding ? "zero vector" : "no embedding"}) id=${record.id}`, - ); - } - - // Sync FTS5 (delete + re-insert to handle updates) - if (this.ftsAvailable) { - try { - this.stmtL0FtsDelete.run(record.id); - this.stmtL0FtsInsert.run( - tokenizeForFts(record.messageText), // message_text โ€” segmented for indexing - record.messageText, // message_text_original โ€” raw for display - record.id, - record.sessionKey, - record.sessionId, - record.role, - record.recordedAt, - record.timestamp, - ); - } catch (ftsErr) { - // FTS write failure is non-fatal โ€” log and continue - this.logger?.warn( - `${TAG} [L0-upsert] FTS write failed (non-fatal) id=${record.id}: ${ftsErr instanceof Error ? ftsErr.message : String(ftsErr)}`, - ); - } - } - - this.db.exec("COMMIT"); - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - this.logger?.debug?.(`${TAG} [L0-upsert] OK id=${record.id}${skipVec ? " (meta-only)" : ""}`); - return true; - } catch (err) { - this.logger?.warn( - `${TAG} [L0-upsert] FAILED (non-fatal) id=${record.id}: ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * Update ONLY the vector embedding for an existing L0 record. - * The metadata row must already exist in l0_conversations (written by upsertL0). - * - * This is used by the background embedding task in auto-capture: - * 1. upsertL0() writes metadata + FTS synchronously (no embedding) - * 2. Background task calls embedBatch() then updateL0Embedding() for each record - * - * **Fault-tolerant**: catches all errors internally, never throws. - * Returns `true` on success, `false` on failure. - */ - updateL0Embedding(recordId: string, embedding: Float32Array): boolean { - if (this.degraded || !this.vecTablesReady) { - return false; - } - if (!embedding || embedding.every(v => v === 0)) { - this.logger?.debug?.(`${TAG} [L0-update-embedding] Skipping zero vector for ${recordId}`); - return false; - } - try { - // Look up recorded_at from metadata for the vec0 row - const meta = this.stmtL0GetMeta.get(recordId) as { recorded_at: string } | undefined; - if (!meta) { - this.logger?.warn(`${TAG} [L0-update-embedding] No metadata found for ${recordId}, skipping`); - return false; - } - - this.db.exec("BEGIN"); - try { - this.stmtL0DeleteVec!.run(recordId); - this.stmtL0InsertVec!.run(recordId, Buffer.from(embedding.buffer), meta.recorded_at); - this.db.exec("COMMIT"); - } catch (err) { - try { this.db.exec("ROLLBACK"); } catch { /* ignore */ } - throw err; - } - return true; - } catch (err) { - this.logger?.warn( - `${TAG} [L0-update-embedding] FAILED (non-fatal) id=${recordId}: ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * Vector similarity search on L0 individual messages (cosine distance). - * Returns top-k results sorted by similarity (highest first). - * - * **Fault-tolerant**: returns an empty array on any error. - */ - searchL0Vector(queryEmbedding: Float32Array, topK = 5): L0VectorSearchResult[] { - if (this.degraded || !this.vecTablesReady) { - if (this.degraded) this.logger?.warn(`${TAG} [L0-search] SKIPPED (degraded mode)`); - return []; - } - try { - // Over-retrieve to compensate for legacy zero-vector placeholders that - // may still exist in the vec0 table. New zero vectors are no longer - // inserted (upsertL0() skips vec write for zero vectors since v3.x), but - // older data may still contain them โ€” they surface as NULL/NaN distance - // in KNN results. - // NOTE: "AND distance IS NOT NULL" is NOT usable because vec0 does not - // support that constraint โ€” it causes an empty result set. - const retrieveCount = topK + VectorStore.ZERO_VEC_BUFFER; - - this.logger?.debug?.( - `${TAG} [L0-search] START topK=${topK}, retrieveCount=${retrieveCount}, ` + - `queryEmbeddingDims=${queryEmbedding.length}, ` + - `queryNorm=${Math.sqrt(Array.from(queryEmbedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}`, - ); - - const rows = this.stmtL0SearchVec!.all( - Buffer.from(queryEmbedding.buffer), - retrieveCount, - ) as Array<{ record_id: string; distance: number }>; - - this.logger?.debug?.(`${TAG} [L0-search] vec0 returned ${rows.length} candidate(s)`); - - if (rows.length === 0) return []; - - const results: L0VectorSearchResult[] = []; - - for (const { record_id, distance } of rows) { - // sqlite-vec returns null distance for zero vectors (cosine undefined when โ€–vโ€–=0). - // Skip these โ€” they are placeholder vectors from embedding-service-unavailable fallback. - if (distance == null || Number.isNaN(distance)) { - this.logger?.warn( - `${TAG} [L0-search] record_id=${record_id} has null/NaN distance (likely zero vector) โ€” skipping`, - ); - continue; - } - - const meta = this.stmtL0GetMeta.get(record_id) as - | { - session_key: string; - session_id: string; - role: string; - message_text: string; - recorded_at: string; - timestamp: number; - } - | undefined; - - if (!meta) { - this.logger?.warn(`${TAG} [L0-search] record_id=${record_id} has vector but NO metadata (orphan)`); - continue; - } - - const score = 1.0 - distance; - this.logger?.debug?.( - `${TAG} [L0-search] HIT id=${record_id}, distance=${distance.toFixed(4)}, score=${score.toFixed(4)}, ` + - `role=${meta.role}, session=${meta.session_key}, text="${meta.message_text.slice(0, 60)}..."`, - ); - - results.push({ - record_id, - session_key: meta.session_key, - session_id: meta.session_id, - role: meta.role, - message_text: meta.message_text, - score, - recorded_at: meta.recorded_at, - timestamp: meta.timestamp ?? 0, - }); - } - - // Trim back to the caller's requested topK (we over-fetched above). - const trimmed = results.slice(0, topK); - this.logger?.info( - `${TAG} [L0-search] DONE returning ${trimmed.length} result(s) (from ${results.length} valid, ${rows.length} raw)`, - ); - return trimmed; - } catch (err) { - this.logger?.warn( - `${TAG} [L0-search] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * Delete a single L0 record (metadata + vector). - * - * **Fault-tolerant**: logs a warning on failure, never throws. - */ - deleteL0(recordId: string): boolean { - if (this.degraded) return false; - try { - this.db.exec("BEGIN"); - try { - this.stmtL0DeleteMeta.run(recordId); - if (this.vecTablesReady) this.stmtL0DeleteVec!.run(recordId); - if (this.ftsAvailable) { - try { this.stmtL0FtsDelete.run(recordId); } catch { /* non-fatal */ } - } - this.db.exec("COMMIT"); - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - return true; - } catch (err) { - this.logger?.warn( - `${TAG} deleteL0 failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * TTL cleanup by recorded_at (ISO string) for L0 records. - * - * Deletes expired rows from l0_conversations and matching vectors from l0_vec - * in a single transaction to guarantee consistency. - */ - deleteL0Expired(cutoffIso: string): number { - if (this.degraded) { - this.logger?.warn(`${TAG} [deleteExpiredL0] SKIPPED (degraded mode)`); - return 0; - } - - try { - const row = this.db.prepare( - "SELECT COUNT(*) AS cnt FROM l0_conversations WHERE recorded_at != '' AND recorded_at < ?", - ).get(cutoffIso) as { cnt: number } | undefined; - const expiredCount = row?.cnt ?? 0; - if (expiredCount <= 0) return 0; - - this.db.exec("BEGIN"); - try { - if (this.vecTablesReady) { - this.db.prepare( - "DELETE FROM l0_vec WHERE recorded_at != '' AND recorded_at < ?", - ).run(cutoffIso); - } - this.db.prepare( - "DELETE FROM l0_conversations WHERE recorded_at != '' AND recorded_at < ?", - ).run(cutoffIso); - this.db.exec("COMMIT"); - return expiredCount; - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - } catch (err) { - this.logger?.warn( - `${TAG} deleteL0ExpiredByRecordedAt failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return 0; - } - } - - /** - * Get the total number of L0 message records in the store. - * - * **Fault-tolerant**: returns 0 on failure. - */ - countL0(): number { - if (this.degraded) return 0; - try { - const row = this.db - .prepare("SELECT COUNT(*) AS cnt FROM l0_conversations") - .get() as { cnt: number }; - this.logger?.debug?.(`${TAG} [L0-count] total=${row.cnt}`); - return row.cnt; - } catch (err) { - this.logger?.warn( - `${TAG} countL0 failed (non-fatal, returning 0): ${err instanceof Error ? err.message : String(err)}`, - ); - return 0; - } - } - - // โ”€โ”€ Re-index operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Get all L1 record texts for re-embedding. - * Returns record_id โ†’ content pairs. - */ - getAllL1Texts(): Array<{ record_id: string; content: string; updated_time: string }> { - if (this.degraded) return []; - try { - return this.db - .prepare("SELECT record_id, content, updated_time FROM l1_records") - .all() as Array<{ record_id: string; content: string; updated_time: string }>; - } catch (err) { - this.logger?.warn( - `${TAG} getAllL1Texts failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * Get all L0 message texts for re-embedding. - * Returns record_id โ†’ message_text/recorded_at tuples. - */ - getAllL0Texts(): Array<{ record_id: string; message_text: string; recorded_at: string }> { - if (this.degraded) return []; - try { - return this.db - .prepare("SELECT record_id, message_text, recorded_at FROM l0_conversations") - .all() as Array<{ record_id: string; message_text: string; recorded_at: string }>; - } catch (err) { - this.logger?.warn( - `${TAG} getAllL0Texts failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * Re-embed all existing L1 and L0 texts with a new embedding function. - * - * This is called after `init()` returns `needsReindex: true` โ€” the vector - * tables have already been dropped and re-created with the correct dimensions. - * This method reads every text from the metadata tables and writes fresh - * embeddings into the new vector tables. - * - * @param embedFn A function that converts text โ†’ Float32Array embedding. - * @param onProgress Optional callback for progress reporting. - */ - async reindexAll( - embedFn: (text: string) => Promise, - onProgress?: (done: number, total: number, layer: "L1" | "L0") => void, - ): Promise<{ l1Count: number; l0Count: number }> { - if (this.degraded || !this.vecTablesReady) { - if (this.degraded) this.logger?.warn(`${TAG} reindexAll skipped: VectorStore is in degraded mode`); - return { l1Count: 0, l0Count: 0 }; - } - - try { - // โ”€โ”€ Re-embed L1 โ”€โ”€ - const l1Rows = this.getAllL1Texts(); - let l1Done = 0; - for (const { record_id, content, updated_time } of l1Rows) { - try { - const embedding = await embedFn(content); - // Wrap delete+insert in a transaction to prevent orphan vectors - this.db.exec("BEGIN"); - try { - this.stmtDeleteVec!.run(record_id); - this.stmtInsertVec!.run(record_id, Buffer.from(embedding.buffer), updated_time); - this.db.exec("COMMIT"); - } catch (txErr) { - try { this.db.exec("ROLLBACK"); } catch { /* ignore */ } - throw txErr; - } - } catch (err) { - this.logger?.warn?.( - `${TAG} reindex L1 skip ${record_id}: ${err instanceof Error ? err.message : String(err)}`, - ); - } - l1Done++; - onProgress?.(l1Done, l1Rows.length, "L1"); - } - - // โ”€โ”€ Re-embed L0 โ”€โ”€ - const l0Rows = this.getAllL0Texts(); - let l0Done = 0; - for (const { record_id, message_text, recorded_at } of l0Rows) { - try { - const embedding = await embedFn(message_text); - // Wrap delete+insert in a transaction to prevent orphan vectors - this.db.exec("BEGIN"); - try { - this.stmtL0DeleteVec!.run(record_id); - this.stmtL0InsertVec!.run(record_id, Buffer.from(embedding.buffer), recorded_at); - this.db.exec("COMMIT"); - } catch (txErr) { - try { this.db.exec("ROLLBACK"); } catch { /* ignore */ } - throw txErr; - } - } catch (err) { - this.logger?.warn?.( - `${TAG} reindex L0 skip ${record_id}: ${err instanceof Error ? err.message : String(err)}`, - ); - } - l0Done++; - onProgress?.(l0Done, l0Rows.length, "L0"); - } - - this.logger?.info( - `${TAG} Reindex complete: L1=${l1Done}/${l1Rows.length}, L0=${l0Done}/${l0Rows.length}`, - ); - - return { l1Count: l1Done, l0Count: l0Done }; - } catch (err) { - this.logger?.error( - `${TAG} reindexAll failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return { l1Count: 0, l0Count: 0 }; - } - } - - // โ”€โ”€ L0 query operations (for L1 runner) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Query L0 messages for a given session key, optionally filtered by recorded_at cursor. - * Returns messages ordered by recorded_at ASC (chronological write order). - * - * Used by L1 runner to read L0 data from DB instead of JSONL files. - */ - queryL0ForL1( - sessionKey: string, - afterRecordedAtMs?: number, - limit = 50, - ): Array<{ - record_id: string; - session_key: string; - session_id: string; - role: string; - message_text: string; - recorded_at: string; - timestamp: number; - }> { - if (this.degraded) { - this.logger?.warn(`${TAG} [L0-query] SKIPPED (degraded mode)`); - return []; - } - try { - // Query newest-first (DESC) with LIMIT, then reverse to chronological order - let rows: Array>; - if (afterRecordedAtMs && afterRecordedAtMs > 0) { - // Convert epoch ms to ISO string for recorded_at comparison - const afterRecordedAtIso = new Date(afterRecordedAtMs).toISOString(); - rows = this.stmtL0QueryAfter.all(sessionKey, afterRecordedAtIso, limit) as Array>; - } else { - rows = this.stmtL0QueryAll.all(sessionKey, limit) as Array>; - } - - this.logger?.info( - `${TAG} [L0-query] session=${sessionKey}, afterRecordedAtMs=${afterRecordedAtMs ?? "(all)"}, ` + - `limit=${limit}, returned ${rows.length} row(s)`, - ); - - // Reverse: SQL returns newest-first (DESC), callers expect chronological order - return rows.map((r) => ({ - record_id: r.record_id as string, - session_key: r.session_key as string, - session_id: (r.session_id as string) || "", - role: r.role as string, - message_text: r.message_text as string, - recorded_at: (r.recorded_at as string) || "", - timestamp: (r.timestamp as number) || 0, - })).reverse(); - } catch (err) { - this.logger?.warn( - `${TAG} [L0-query] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * Query L0 messages for a given session key, grouped by session_id. - * Each group's messages are in chronological order (recorded_at ASC). - * Groups are sorted by earliest message timestamp. - * - * Used by L1 runner to replace readConversationMessagesGroupedBySessionId(). - */ - queryL0GroupedBySessionId( - sessionKey: string, - afterRecordedAtMs?: number, - limit = 50, - ): Array<{ sessionId: string; messages: Array<{ id: string; role: string; content: string; timestamp: number; recordedAtMs: number }> }> { - if (this.degraded) { - this.logger?.warn(`${TAG} [L0-query-grouped] SKIPPED (degraded mode)`); - return []; - } - try { - const rows = this.queryL0ForL1(sessionKey, afterRecordedAtMs, limit); - - // Group by session_id - const groupMap = new Map>(); - for (const row of rows) { - const sid = row.session_id || ""; - let group = groupMap.get(sid); - if (!group) { - group = []; - groupMap.set(sid, group); - } - group.push({ - id: row.record_id, - role: row.role, - content: row.message_text, - timestamp: row.timestamp, - recordedAtMs: row.recorded_at ? Date.parse(row.recorded_at) || 0 : 0, - }); - } - - // Convert to array, sorted by earliest message timestamp - const groups: Array<{ sessionId: string; messages: Array<{ id: string; role: string; content: string; timestamp: number; recordedAtMs: number }> }> = []; - for (const [sessionId, messages] of groupMap) { - if (messages.length > 0) { - groups.push({ sessionId, messages }); - } - } - groups.sort((a, b) => a.messages[0].timestamp - b.messages[0].timestamp); - - this.logger?.info( - `${TAG} [L0-query-grouped] session=${sessionKey}, afterRecordedAtMs=${afterRecordedAtMs ?? "(all)"}, ` + - `${rows.length} messages across ${groups.length} group(s)`, - ); - - return groups; - } catch (err) { - this.logger?.warn( - `${TAG} [L0-query-grouped] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - // โ”€โ”€ Cursor-based pagination for migration โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Read a page of L1 records using primary key cursor. - * Returns rows with `record_id > afterId`, ordered by PK, limited to `pageSize`. - * Pass `""` as `afterId` for the first page. - */ - queryL1RecordsCursor(afterId: string, pageSize: number): L1RecordRow[] { - if (this.degraded) return []; - try { - return this.stmtL1QueryMigrationCursor.all(afterId, pageSize) as unknown as L1RecordRow[]; - } catch (err) { - this.logger?.warn( - `${TAG} [L1-query-cursor] FAILED (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * Read a page of L0 records using primary key cursor. - * Returns rows with `record_id > afterId`, ordered by PK, limited to `pageSize`. - * Pass `""` as `afterId` for the first page. - */ - queryL0RecordsCursor(afterId: string, pageSize: number): L0RecordRow[] { - if (this.degraded) return []; - try { - return this.stmtL0QueryMigrationCursor.all(afterId, pageSize) as unknown as L0RecordRow[]; - } catch (err) { - this.logger?.warn( - `${TAG} [L0-query-cursor] FAILED (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - // โ”€โ”€ FTS5 search operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Whether FTS5 full-text search is available. - * When `false`, callers should skip keyword-based recall entirely. - */ - isFtsAvailable(): boolean { - return this.ftsAvailable; - } - - /** - * FTS5 keyword search on L1 records. - * Returns top-`limit` results sorted by BM25 relevance (highest first). - * - * @param ftsQuery A pre-built FTS5 MATCH expression (from `buildFtsQuery()`). - * @param limit Maximum number of results to return. - * - * **Fault-tolerant**: returns an empty array on any error. - */ - searchL1Fts(ftsQuery: string, limit = 20): FtsSearchResult[] { - if (this.degraded || !this.ftsAvailable) return []; - try { - const rows = this.stmtL1FtsSearch.all(ftsQuery, limit) as Array<{ - record_id: string; - content: string; - type: string; - priority: number; - scene_name: string; - session_key: string; - session_id: string; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - metadata_json: string; - rank: number; - }>; - - return rows.map((r) => ({ - record_id: r.record_id, - content: r.content, - type: r.type, - priority: r.priority, - scene_name: r.scene_name, - score: bm25RankToScore(r.rank), - timestamp_str: r.timestamp_str, - timestamp_start: r.timestamp_start, - timestamp_end: r.timestamp_end, - session_key: r.session_key, - session_id: r.session_id, - metadata_json: r.metadata_json, - })); - } catch (err) { - this.logger?.warn( - `${TAG} [L1-fts-search] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * FTS5 keyword search on L0 conversation messages. - * Returns top-`limit` results sorted by BM25 relevance (highest first). - * - * @param ftsQuery A pre-built FTS5 MATCH expression (from `buildFtsQuery()`). - * @param limit Maximum number of results to return. - * - * **Fault-tolerant**: returns an empty array on any error. - */ - searchL0Fts(ftsQuery: string, limit = VectorStore.FTS_DEFAULT_LIMIT): L0FtsSearchResult[] { - if (this.degraded || !this.ftsAvailable) return []; - try { - const rows = this.stmtL0FtsSearch.all(ftsQuery, limit) as Array<{ - record_id: string; - message_text: string; - session_key: string; - session_id: string; - role: string; - recorded_at: string; - timestamp: number; - rank: number; - }>; - - return rows.map((r) => ({ - record_id: r.record_id, - session_key: r.session_key, - session_id: r.session_id, - role: r.role, - message_text: r.message_text, - score: bm25RankToScore(r.rank), - recorded_at: r.recorded_at, - timestamp: r.timestamp ?? 0, - })); - } catch (err) { - this.logger?.warn( - `${TAG} [L0-fts-search] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - // โ”€โ”€ FTS5 migration & rebuild โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Detect old FTS5 v1 schema (no `content_original` column) and drop the - * tables so they can be recreated with the v2 schema. - * - * FTS5 virtual tables do NOT support `ALTER TABLE ADD COLUMN`, so the only - * migration path is DROP + recreate + repopulate. - * - * @returns `true` if migration was performed (= FTS index needs rebuilding). - * @internal - */ - private migrateFtsTablesIfNeeded(): boolean { - try { - // Check if l1_fts exists at all - const l1Exists = this.db - .prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name='l1_fts'") - .get(); - if (!l1Exists) { - // Fresh install โ€” tables will be created with v2 schema. - // Still need rebuild if there's existing data in l1_records. - const hasData = this.db.prepare("SELECT 1 FROM l1_records LIMIT 1").get(); - return !!hasData; - } - - // Check if the v2 column `content_original` exists. - // FTS5 tables appear in pragma_table_info with their column names. - const cols = this.db - .prepare("SELECT name FROM pragma_table_info('l1_fts')") - .all() as Array<{ name: string }>; - const hasV2Col = cols.some((c) => c.name === "content_original"); - - if (hasV2Col) { - return false; // Already v2 โ€” no migration needed - } - - // v1 โ†’ v2: drop both FTS tables (data will be repopulated by rebuildFtsIndex) - this.logger?.info(`${TAG} Migrating FTS5 tables from v1 to v2 (jieba segmented)`); - this.db.exec("DROP TABLE IF EXISTS l1_fts"); - this.db.exec("DROP TABLE IF EXISTS l0_fts"); - return true; - } catch (err) { - this.logger?.warn( - `${TAG} FTS migration check failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * Rebuild the FTS5 index from scratch by reading all records from the - * metadata tables and re-inserting them with jieba-segmented text. - * - * Called automatically after: - * - Schema migration from v1 to v2 - * - Fresh table creation when existing data exists - * - * Safe to call multiple times (idempotent โ€” clears FTS tables first). - */ - rebuildFtsIndex(): void { - if (!this.ftsAvailable) return; - - try { - this.logger?.info(`${TAG} Rebuilding FTS5 index with jieba segmentationโ€ฆ`); - - // โ”€โ”€ Rebuild L1 FTS โ”€โ”€ - // Clear existing FTS data - this.db.exec("DELETE FROM l1_fts"); - - // Read all L1 records from metadata table - const l1Rows = this.db - .prepare(` - SELECT record_id, content, type, priority, scene_name, - session_key, session_id, timestamp_str, timestamp_start, timestamp_end, metadata_json - FROM l1_records - `) - .all() as Array<{ - record_id: string; - content: string; - type: string; - priority: number; - scene_name: string; - session_key: string; - session_id: string; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - metadata_json: string; - }>; - - let l1Count = 0; - for (const r of l1Rows) { - try { - this.stmtL1FtsInsert.run( - tokenizeForFts(r.content), // content โ€” segmented - r.content, // content_original โ€” raw - r.record_id, - r.type, - r.priority, - r.scene_name, - r.session_key, - r.session_id, - r.timestamp_str, - r.timestamp_start, - r.timestamp_end, - r.metadata_json, - ); - l1Count++; - } catch (err) { - this.logger?.warn?.( - `${TAG} FTS rebuild skip L1 ${r.record_id}: ${err instanceof Error ? err.message : String(err)}`, - ); - } - } - - // โ”€โ”€ Rebuild L0 FTS โ”€โ”€ - this.db.exec("DELETE FROM l0_fts"); - - const l0Rows = this.db - .prepare(` - SELECT record_id, message_text, session_key, session_id, role, recorded_at, timestamp - FROM l0_conversations - `) - .all() as Array<{ - record_id: string; - message_text: string; - session_key: string; - session_id: string; - role: string; - recorded_at: string; - timestamp: number; - }>; - - let l0Count = 0; - for (const r of l0Rows) { - try { - this.stmtL0FtsInsert.run( - tokenizeForFts(r.message_text), // message_text โ€” segmented - r.message_text, // message_text_original โ€” raw - r.record_id, - r.session_key, - r.session_id, - r.role, - r.recorded_at, - r.timestamp, - ); - l0Count++; - } catch (err) { - this.logger?.warn?.( - `${TAG} FTS rebuild skip L0 ${r.record_id}: ${err instanceof Error ? err.message : String(err)}`, - ); - } - } - - this.logger?.info( - `${TAG} FTS5 rebuild complete: L1=${l1Count}/${l1Rows.length}, L0=${l0Count}/${l0Rows.length}`, - ); - } catch (err) { - this.logger?.warn( - `${TAG} FTS5 rebuild failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - } - } - - // ============================ - // IMemoryStore interface implementation - // ============================ - - /** Query the store's search capabilities. */ - getCapabilities(): StoreCapabilities { - return { - vectorSearch: this.vecTablesReady, - ftsSearch: this.ftsAvailable, - nativeHybridSearch: false, - sparseVectors: false, - }; - } - - /** - * Close the database connection. - * Should be called on shutdown. Idempotent โ€” safe to call multiple times. - */ - close(): void { - if (this.closed) return; - this.closed = true; - try { - this.db.close(); - } catch (err) { - this.logger?.warn?.( - `${TAG} Error closing database: ${err instanceof Error ? err.message : String(err)}`, - ); - } - } -} diff --git a/src/store/tcvdb-client.ts b/src/store/tcvdb-client.ts deleted file mode 100644 index 3594a59..0000000 --- a/src/store/tcvdb-client.ts +++ /dev/null @@ -1,287 +0,0 @@ -/** - * Tencent Cloud VectorDB HTTP Client. - * - * Thin wrapper around the VectorDB HTTP API. Handles authentication, timeouts, - * retries (5xx / timeout), and error normalization. - * - * API docs: https://cloud.tencent.com/document/product/1709 - */ - -import fs from "node:fs"; -import { request as undiciRequest, Agent as UndiciAgent } from "undici"; -import type { Dispatcher } from "undici"; -import type { StoreLogger } from "./types.js"; - -// ============================ -// Types -// ============================ - -export interface TcvdbClientConfig { - /** Instance URL (e.g. "http://10.0.1.1:80") */ - url: string; - /** Account name (default: "root") */ - username: string; - /** API Key */ - apiKey: string; - /** Database name */ - database: string; - /** Request timeout in ms (default: 10000) */ - timeout: number; - /** Path to CA certificate PEM file (for HTTPS connections) */ - caPemPath?: string; -} - -/** Standard VectorDB API response envelope. */ -interface ApiResponse { - code: number; - msg: string; - [key: string]: unknown; -} - -/** Search/hybridSearch response shape. */ -export interface SearchResponse { - documents: Array>>; -} - -/** Query response shape. */ -export interface QueryResponse { - documents: Array>; - count?: number; -} - -/** Collection info from describeCollection. */ -export interface CollectionInfo { - collection: string; - database: string; - documentCount?: number; - embedding?: { - field: string; - vectorField: string; - model: string; - }; - indexes?: Array>; - [key: string]: unknown; -} - -export class TcvdbApiError extends Error { - readonly apiCode: number; - constructor(path: string, code: number, msg: string) { - super(`VectorDB ${path}: code=${code}, msg=${msg}`); - this.name = "TcvdbApiError"; - this.apiCode = code; - } -} - -// ============================ -// Client -// ============================ - -const TAG = "[memory-tdai][tcvdb-client]"; -const MAX_RETRIES = 2; - -export class TcvdbClient { - private readonly baseUrl: string; - private readonly authHeader: string; - private readonly database: string; - private readonly timeout: number; - private readonly logger?: StoreLogger; - /** undici dispatcher for HTTPS + custom CA. */ - private readonly dispatcher?: Dispatcher; - - constructor(config: TcvdbClientConfig, logger?: StoreLogger) { - this.baseUrl = config.url.replace(/\/+$/, ""); - this.authHeader = `Bearer account=${config.username}&api_key=${config.apiKey}`; - this.database = config.database; - this.timeout = config.timeout; - this.logger = logger; - - // Log connection info at construction time. - this.logger?.debug?.(`${TAG} url=${this.baseUrl} db=${this.database} timeout=${this.timeout}${this.baseUrl.startsWith("https://") ? ` https=true caPemPath=${config.caPemPath ?? "(none)"}` : ""}`); - - // For HTTPS with a custom CA certificate, create a dedicated undici Agent. - // We use undici.request() instead of global fetch because fetch's - // `dispatcher` option is unreliable across Node versions. - if (this.baseUrl.startsWith("https://") && config.caPemPath) { - try { - const ca = fs.readFileSync(config.caPemPath, "utf-8"); - this.dispatcher = new UndiciAgent({ connect: { ca } }); - this.logger?.debug?.(`${TAG} HTTPS enabled with CA from ${config.caPemPath}`); - } catch (err) { - this.logger?.error(`${TAG} Failed to load CA PEM from ${config.caPemPath}: ${err instanceof Error ? err.message : String(err)}`); - } - } - } - - // โ”€โ”€ Generic request โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Send a POST request to VectorDB API. - * Handles auth, timeout, retries (5xx/timeout), and error unwrapping. - */ - async request(path: string, body: Record): Promise { - let lastError: Error | undefined; - - for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) { - try { - this.logger?.debug?.(`${TAG} โ†’ ${path} body=${JSON.stringify(body).slice(0, 500)}`); - const { statusCode, body: respBody } = await undiciRequest(`${this.baseUrl}${path}`, { - method: "POST", - headers: { - "Content-Type": "application/json", - "Authorization": this.authHeader, - }, - body: JSON.stringify(body), - signal: AbortSignal.timeout(this.timeout), - ...(this.dispatcher ? { dispatcher: this.dispatcher } : {}), - }); - - const text = await respBody.text(); - const json = JSON.parse(text) as ApiResponse; - this.logger?.debug?.(`${TAG} โ† ${path} status=${statusCode} code=${json.code} msg=${json.msg} keys=[${Object.keys(json).join(",")}]`); - - if (json.code !== 0) { - const err = new TcvdbApiError(path, json.code, json.msg); - if (statusCode !== undefined && statusCode >= 400 && statusCode < 500) throw err; - lastError = err; - continue; - } - - return json as unknown as T; - } catch (err) { - if (err instanceof TcvdbApiError && err.apiCode !== 0) throw err; - lastError = err instanceof Error ? err : new Error(String(err)); - if (attempt < MAX_RETRIES) { - const delay = 500 * (attempt + 1); - this.logger?.debug?.(`${TAG} ${path} retry ${attempt + 1}/${MAX_RETRIES} in ${delay}ms`); - await new Promise((r) => setTimeout(r, delay)); - } - } - } - - throw lastError ?? new Error(`${TAG} ${path} failed after retries`); - } - - // โ”€โ”€ Database operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - async createDatabase(dbName?: string): Promise { - const name = dbName ?? this.database; - // SDK pattern: list first, create only if not found - const listResp = await this.request<{ databases: string[] }>("/database/list", {}); - const exists = (listResp.databases ?? []).includes(name); - if (exists) { - this.logger?.debug?.(`${TAG} Database already exists: ${name}`); - return false; - } - await this.request("/database/create", { database: name }); - this.logger?.info(`${TAG} Database created: ${name}`); - return true; - } - - // โ”€โ”€ Collection operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - async createCollection(params: Record): Promise { - const name = String(params.collection ?? ""); - // SDK pattern: try describe first, create only if not found (code 15302) - try { - await this.describeCollection(name); - this.logger?.debug?.(`${TAG} Collection already exists: ${name}`); - return; - } catch (err) { - if (!(err instanceof TcvdbApiError && err.apiCode === 15302)) { - throw err; // unexpected error - } - // 15302 = collection not found โ†’ proceed to create - } - try { - await this.request("/collection/create", { - database: this.database, - ...params, - }); - this.logger?.info(`${TAG} Collection created: ${name}`); - } catch (err) { - // 15202 = collection already exists โ€” race between describe and create. - // Semantically identical to "describe found it", so treat as success. - if (err instanceof TcvdbApiError && err.apiCode === 15202) { - this.logger?.debug?.(`${TAG} Collection already exists (race): ${name}`); - return; - } - throw err; - } - } - - async describeCollection(collection: string): Promise { - const resp = await this.request<{ collection: CollectionInfo }>("/collection/describe", { - database: this.database, - collection, - }); - return resp.collection; - } - - // โ”€โ”€ Document operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - async upsert(collection: string, documents: Record[]): Promise { - await this.request("/document/upsert", { - database: this.database, - collection, - buildIndex: true, - documents, - }); - } - - async search(collection: string, searchParams: Record): Promise { - return this.request("/document/search", { - database: this.database, - collection, - readConsistency: "strongConsistency", - search: searchParams, - }); - } - - async hybridSearch(collection: string, searchParams: Record): Promise { - return this.request("/document/hybridSearch", { - database: this.database, - collection, - readConsistency: "strongConsistency", - search: searchParams, - }); - } - - async query(collection: string, queryParams: Record): Promise { - return this.request("/document/query", { - database: this.database, - collection, - readConsistency: "strongConsistency", - query: queryParams, - }); - } - - async deleteDoc(collection: string, params: Record): Promise { - await this.request("/document/delete", { - database: this.database, - collection, - ...params, - }); - } - - /** - * Count documents matching an optional filter. - * Uses the dedicated /document/count endpoint. - */ - async count(collection: string, filter?: string): Promise { - const query: Record = {}; - if (filter) query.filter = filter; - const resp = await this.request<{ count: number }>("/document/count", { - database: this.database, - collection, - readConsistency: "strongConsistency", - query, - }); - return resp.count ?? 0; - } - - // โ”€โ”€ Convenience getters โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - getDatabase(): string { - return this.database; - } -} diff --git a/src/store/tcvdb.ts b/src/store/tcvdb.ts deleted file mode 100644 index 35fe6a9..0000000 --- a/src/store/tcvdb.ts +++ /dev/null @@ -1,1180 +0,0 @@ -/** - * TcvdbMemoryStore: Tencent Cloud VectorDB backend implementing IMemoryStore. - * - * Features: - * - Server-side dense embedding (embeddingItems via Collection embedding config) - * - Client-side sparse vectors (BM25 local encoder for hybridSearch) - * - Native hybridSearch (dense + sparse + RRFRerank) - * - Filter expressions for scalar field queries - * - Time fields stored as uint64 epoch ms (ISO โ†” epoch conversion internal) - * - * All methods are fault-tolerant: return empty/false on error, never throw. - */ - -import type { MemoryRecord } from "../record/l1-writer.js"; -import type { EmbeddingProviderInfo } from "./embedding.js"; -import type { - IMemoryStore, - StoreCapabilities, - StoreInitResult, - L1SearchResult, - L1FtsResult, - L1RecordRow, - L1QueryFilter, - L0SearchResult, - L0FtsResult, - L0QueryRow, - L0SessionGroup, - ProfileRecord, - ProfileSyncRecord, - StoreLogger, -} from "./types.js"; -import { TcvdbClient, TcvdbApiError } from "./tcvdb-client.js"; -import type { BM25LocalEncoder } from "./bm25-local.js"; -import type { SparseVector } from "@tencentdb-agent-memory/tcvdb-text"; - -// ============================ -// Config & Constants -// ============================ - -export interface TcvdbMemoryStoreConfig { - url: string; - username: string; - apiKey: string; - database: string; - embeddingModel: string; - timeout: number; - /** Path to CA certificate PEM file (for HTTPS connections) */ - caPemPath?: string; - logger?: StoreLogger; - bm25Encoder?: BM25LocalEncoder; -} - -const TAG = "[memory-tdai][tcvdb]"; - -/** Base collection suffixes (prefixed with database name at construction time). */ -const L1_COLLECTION_SUFFIX = "l1_memories"; -const L0_COLLECTION_SUFFIX = "l0_conversations"; -const PROFILES_COLLECTION_SUFFIX = "profiles"; - -/** Max documents per /document/query page (VectorDB API limit). */ -const QUERY_PAGE_SIZE = 100; - -/** All L1 output fields returned by query/search (excludes vector/sparse_vector). */ -const L1_OUTPUT_FIELDS = [ - "id", "text", "type", "priority", "scene_name", - "session_key", "session_id", "timestamp_str", "timestamp_start", - "timestamp_end", "metadata_json", "created_time_ms", "updated_time_ms", -]; - -/** All L0 output fields returned by query/search. */ -const L0_OUTPUT_FIELDS = [ - "id", "message_text", "agent_id", "session_key", "session_id", "role", - "recorded_at_ms", "timestamp", -]; - -const PROFILE_OUTPUT_FIELDS = [ - "id", "type", "filename", "content", "content_md5", "agent_id", - "version", "created_at_ms", "updated_at_ms", -]; - -const PROFILE_METADATA_OUTPUT_FIELDS = [ - "id", "type", "filename", "content_md5", "agent_id", - "version", "created_at_ms", "updated_at_ms", -]; - -// ============================ -// Helpers -// ============================ - -function isoToEpochMs(iso: string): number { - if (!iso) return 0; - const ms = new Date(iso).getTime(); - return Number.isFinite(ms) ? ms : 0; -} - -function epochMsToIso(ms: number): string { - if (!ms || ms <= 0) return ""; - return new Date(ms).toISOString(); -} - -/** - * Extract agent ID from a sessionKey like `agent::`. - * Returns empty string if the format doesn't match. - */ -function extractAgentId(sessionKey: string): string { - if (!sessionKey) return ""; - const parts = sessionKey.split(":"); - // Format: "agent::..." โ†’ parts[1] - if (parts.length >= 2 && parts[0] === "agent") { - return parts[1]; - } - return ""; -} - -// ============================ -// TcvdbMemoryStore -// ============================ - -export class TcvdbMemoryStore implements IMemoryStore { - private readonly client: TcvdbClient; - private readonly embeddingModel: string; - private readonly logger?: StoreLogger; - private readonly bm25Encoder?: BM25LocalEncoder; - private readonly l1Collection: string; - private readonly l0Collection: string; - private readonly profilesCollection: string; - private degraded = false; - - /** Promise that resolves when async init completes. */ - private _initPromise: Promise | undefined; - - constructor(config: TcvdbMemoryStoreConfig) { - this.client = new TcvdbClient({ - url: config.url, - username: config.username, - apiKey: config.apiKey, - database: config.database, - timeout: config.timeout, - caPemPath: config.caPemPath, - }, config.logger); - this.embeddingModel = config.embeddingModel; - this.logger = config.logger; - this.bm25Encoder = config.bm25Encoder; - - // Collection names are globally unique within a TCVDB instance, - // so prefix with database name to avoid cross-database collisions. - this.l1Collection = `${config.database}_${L1_COLLECTION_SUFFIX}`; - this.l0Collection = `${config.database}_${L0_COLLECTION_SUFFIX}`; - this.profilesCollection = `${config.database}_${PROFILES_COLLECTION_SUFFIX}`; - } - - // โ”€โ”€ Lifecycle โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - async init(_providerInfo?: EmbeddingProviderInfo): Promise { - // TCVDB init is async (HTTP). We store the promise so _ensureInit() - // can also await it as a defensive fallback in each data method. - this._initPromise = this._initAsync(); - try { - await this._initPromise; - } catch (err) { - this.logger?.error(`${TAG} Async init failed: ${err instanceof Error ? err.message : String(err)}`); - this.degraded = true; - } - return { needsReindex: false }; - } - - /** - * Await async initialization. Call at the start of every async method. - * If init already completed (or failed โ†’ degraded), returns immediately. - */ - private async _ensureInit(): Promise { - if (this._initPromise) { - await this._initPromise; - } - } - - // โ”€โ”€ Vector index definitions โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - // - // Preferred: DISK_FLAT (lower memory, suitable for large-scale recall). - // Fallback: HNSW (for instances whose storage engine doesn't support DISK_FLAT). - - private static readonly VECTOR_INDEX_DISK_FLAT: Record = { - fieldName: "vector", fieldType: "vector", indexType: "DISK_FLAT", - dimension: 1024, metricType: "COSINE", - }; - - private static readonly VECTOR_INDEX_HNSW: Record = { - fieldName: "vector", fieldType: "vector", indexType: "HNSW", - dimension: 1024, metricType: "COSINE", - params: { M: 16, efConstruction: 200 }, - }; - - /** - * Detect whether a createCollection error indicates DISK_FLAT is unsupported. - * Matches on apiCode 15113 OR message containing "DISK_FLAT" + "not support". - */ - private static isDiskFlatUnsupported(err: unknown): boolean { - if (!(err instanceof TcvdbApiError)) return false; - if (err.apiCode === 15113) return true; - const msg = err.message.toLowerCase(); - return msg.includes("disk_flat") && (msg.includes("not support") || msg.includes("unsupported")); - } - - /** - * Create a collection with DISK_FLAT vector index, falling back to HNSW - * if the storage engine doesn't support DISK_FLAT. - */ - private async _createCollectionWithVectorFallback( - params: Record, - filterIndexes: Array>, - ): Promise { - const buildIndexes = (vectorIndex: Record) => [ - { fieldName: "id", fieldType: "string", indexType: "primaryKey" }, - vectorIndex, - { fieldName: "sparse_vector", fieldType: "sparseVector", indexType: "inverted", metricType: "IP" }, - ...filterIndexes, - ]; - - try { - await this.client.createCollection({ ...params, indexes: buildIndexes(TcvdbMemoryStore.VECTOR_INDEX_DISK_FLAT) }); - } catch (err) { - if (TcvdbMemoryStore.isDiskFlatUnsupported(err)) { - this.logger?.debug?.(`${TAG} DISK_FLAT not supported for ${String(params.collection)}, falling back to HNSW`); - await this.client.createCollection({ ...params, indexes: buildIndexes(TcvdbMemoryStore.VECTOR_INDEX_HNSW) }); - } else { - throw err; - } - } - } - - private async _initAsync(): Promise { - try { - // Create database (idempotent โ€” returns true if just created, false if already existed) - const dbCreated = await this.client.createDatabase(); - - if (dbCreated) { - // TCVDB requires ~3s after database creation before collections can be created. - // TODO: defer collection creation to first use to avoid blocking plugin startup. - this.logger?.debug?.(`${TAG} Waiting 5s for database to become ready...`); - await new Promise((r) => setTimeout(r, 5_000)); - } - - // Create L1 collection (DISK_FLAT preferred, HNSW fallback) - await this._createCollectionWithVectorFallback( - { - collection: this.l1Collection, - shardNum: 1, - replicaNum: 2, - description: "L1 ็ป“ๆž„ๅŒ–่ฎฐๅฟ†", - embedding: { - status: "enabled", - field: "text", - vectorField: "vector", - model: this.embeddingModel, - }, - }, - [ - { fieldName: "type", fieldType: "string", indexType: "filter" }, - { fieldName: "priority", fieldType: "uint64", indexType: "filter" }, - { fieldName: "scene_name", fieldType: "string", indexType: "filter" }, - { fieldName: "agent_id", fieldType: "string", indexType: "filter" }, - { fieldName: "session_key", fieldType: "string", indexType: "filter" }, - { fieldName: "session_id", fieldType: "string", indexType: "filter" }, - { fieldName: "timestamp_start", fieldType: "string", indexType: "filter" }, - { fieldName: "timestamp_end", fieldType: "string", indexType: "filter" }, - { fieldName: "created_time_ms", fieldType: "uint64", indexType: "filter" }, - { fieldName: "updated_time_ms", fieldType: "uint64", indexType: "filter" }, - ], - ); - - // Create L0 collection (DISK_FLAT preferred, HNSW fallback) - await this._createCollectionWithVectorFallback( - { - collection: this.l0Collection, - shardNum: 1, - replicaNum: 2, - description: "L0 ๅŽŸๅง‹ๅฏน่ฏๆถˆๆฏ", - embedding: { - status: "enabled", - field: "message_text", - vectorField: "vector", - model: this.embeddingModel, - }, - }, - [ - { fieldName: "agent_id", fieldType: "string", indexType: "filter" }, - { fieldName: "session_key", fieldType: "string", indexType: "filter" }, - { fieldName: "session_id", fieldType: "string", indexType: "filter" }, - { fieldName: "role", fieldType: "string", indexType: "filter" }, - { fieldName: "recorded_at_ms", fieldType: "uint64", indexType: "filter" }, - { fieldName: "timestamp", fieldType: "int64", indexType: "filter" }, - ], - ); - - await this.client.createCollection({ - collection: this.profilesCollection, - shardNum: 1, - replicaNum: 2, - description: "L2 ๅœบๆ™ฏๅ— + L3 ็”จๆˆท็”ปๅƒ", - embedding: { status: "disabled" }, - indexes: [ - { fieldName: "id", fieldType: "string", indexType: "primaryKey" }, - { fieldName: "vector", fieldType: "vector", indexType: "FLAT", - dimension: 1, metricType: "COSINE" }, - { fieldName: "type", fieldType: "string", indexType: "filter" }, - { fieldName: "filename", fieldType: "string", indexType: "filter" }, - { fieldName: "content_md5", fieldType: "string", indexType: "filter" }, - { fieldName: "agent_id", fieldType: "string", indexType: "filter" }, - { fieldName: "created_at_ms", fieldType: "uint64", indexType: "filter" }, - { fieldName: "updated_at_ms", fieldType: "uint64", indexType: "filter" }, - { fieldName: "version", fieldType: "uint64", indexType: "filter" }, - ], - }); - - this.logger?.debug?.(`${TAG} Initialized: db=${this.client.getDatabase()}, model=${this.embeddingModel}`); - } catch (err) { - // 15201 = database already exists โ€” benign race in createDatabase(). - // 15202 (collection already exists) is now handled inside TcvdbClient.createCollection(), - // so it should no longer reach here. - if (err instanceof TcvdbApiError && err.apiCode === 15201) { - this.logger?.debug?.(`${TAG} Init (benign): ${err.message}`); - return; - } - this.logger?.error(`${TAG} Init failed: ${err instanceof Error ? err.message : String(err)}`); - this.degraded = true; - } - } - - isDegraded(): boolean { - return this.degraded; - } - - getCapabilities(): StoreCapabilities { - const hasBm25 = !!this.bm25Encoder; - return { - vectorSearch: true, - ftsSearch: hasBm25, - nativeHybridSearch: hasBm25, - sparseVectors: hasBm25, - }; - } - - close(): void { - // HTTP client โ€” nothing to close - } - - // โ”€โ”€ Internal: paginated query helper โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Paginated /document/query that fetches all matching docs. - * TCVDB query API returns at most `limit` docs per call. - * We loop with offset until fewer docs than page size are returned. - */ - private async _queryAllDocs( - collection: string, - filter?: string, - outputFields?: string[], - limit?: number, - sort?: Array>, - ): Promise>> { - const allDocs: Array> = []; - let offset = 0; - const pageSize = limit && limit < QUERY_PAGE_SIZE ? limit : QUERY_PAGE_SIZE; - - // eslint-disable-next-line no-constant-condition - while (true) { - const queryParams: Record = { - retrieveVector: false, - limit: pageSize, - offset, - }; - if (filter) queryParams.filter = filter; - if (outputFields) queryParams.outputFields = outputFields; - if (sort) queryParams.sort = sort; - - const resp = await this.client.query(collection, queryParams); - const docs = resp.documents ?? []; - allDocs.push(...docs); - - // Stop if: we got fewer than page size (last page), or we hit caller's limit - if (docs.length < pageSize) break; - if (limit && allDocs.length >= limit) break; - - offset += docs.length; - } - - // Trim to caller's limit if specified - return limit ? allDocs.slice(0, limit) : allDocs; - } - - // โ”€โ”€ L1 Write Operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - async upsertL1(record: MemoryRecord, _embedding?: Float32Array): Promise { - try { - await this._upsertL1Async(record); - return true; - } catch (err) { - this.logger?.warn(`${TAG} [L1-upsert] FAILED id=${record.id}: ${err instanceof Error ? err.message : String(err)}`); - return false; - } - } - - private async _upsertL1Async(record: MemoryRecord): Promise { - await this._ensureInit(); - if (this.degraded) return; - - const tsStr = record.timestamps[0] ?? ""; - const tsStart = record.timestamps.length > 0 - ? record.timestamps.reduce((a, b) => (a < b ? a : b)) : tsStr; - const tsEnd = record.timestamps.length > 0 - ? record.timestamps.reduce((a, b) => (a > b ? a : b)) : tsStr; - - const doc: Record = { - id: record.id, - text: record.content, - type: record.type, - priority: record.priority, - scene_name: record.scene_name, - agent_id: extractAgentId(record.sessionKey), - session_key: record.sessionKey, - session_id: record.sessionId, - timestamp_str: tsStr, - timestamp_start: tsStart, - timestamp_end: tsEnd, - created_time_ms: isoToEpochMs(record.createdAt), - updated_time_ms: isoToEpochMs(record.updatedAt), - metadata_json: JSON.stringify(record.metadata), - }; - - // BM25 sparse vector (if sidecar available) - if (this.bm25Encoder) { - const sparse = this.bm25Encoder.encodeTexts([record.content]); - if (sparse.length > 0 && sparse[0].length > 0) { - doc.sparse_vector = sparse[0]; - } - } - - await this.client.upsert(this.l1Collection, [doc]); - } - - /** - * Batch upsert multiple L1 records in a single API call. - * Used by migration scripts to reduce request count. - */ - async upsertL1Batch(records: MemoryRecord[]): Promise { - if (records.length === 0) return 0; - try { - await this._ensureInit(); - if (this.degraded) return 0; - - const docs = records.map((record) => { - const tsStr = record.timestamps[0] ?? ""; - const tsStart = record.timestamps.length > 0 - ? record.timestamps.reduce((a, b) => (a < b ? a : b)) : tsStr; - const tsEnd = record.timestamps.length > 0 - ? record.timestamps.reduce((a, b) => (a > b ? a : b)) : tsStr; - - const doc: Record = { - id: record.id, - text: record.content, - type: record.type, - priority: record.priority, - scene_name: record.scene_name, - agent_id: extractAgentId(record.sessionKey), - session_key: record.sessionKey, - session_id: record.sessionId, - timestamp_str: tsStr, - timestamp_start: tsStart, - timestamp_end: tsEnd, - created_time_ms: isoToEpochMs(record.createdAt), - updated_time_ms: isoToEpochMs(record.updatedAt), - metadata_json: JSON.stringify(record.metadata), - }; - - if (this.bm25Encoder) { - const sparse = this.bm25Encoder.encodeTexts([record.content]); - if (sparse.length > 0 && sparse[0].length > 0) { - doc.sparse_vector = sparse[0]; - } - } - return doc; - }); - - await this.client.upsert(this.l1Collection, docs); - return records.length; - } catch (err) { - this.logger?.warn(`${TAG} [L1-upsertBatch] FAILED (${records.length} records): ${err instanceof Error ? err.message : String(err)}`); - return 0; - } - } - - async deleteL1(recordId: string): Promise { - try { - await this._ensureInit(); - if (this.degraded) return false; - await this.client.deleteDoc(this.l1Collection, { - query: { documentIds: [recordId] }, - }); - return true; - } catch (err) { - this.logger?.warn(`${TAG} [L1-delete] FAILED id=${recordId}: ${err instanceof Error ? err.message : String(err)}`); - return false; - } - } - - async deleteL1Batch(recordIds: string[]): Promise { - if (recordIds.length === 0) return true; - try { - await this._ensureInit(); - if (this.degraded) return false; - await this.client.deleteDoc(this.l1Collection, { - query: { documentIds: recordIds }, - }); - return true; - } catch (err) { - this.logger?.warn(`${TAG} [L1-deleteBatch] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return false; - } - } - - async deleteL1Expired(cutoffIso: string): Promise { - const cutoffMs = isoToEpochMs(cutoffIso); - if (cutoffMs <= 0) return 0; - try { - await this._ensureInit(); - if (this.degraded) return 0; - await this.client.deleteDoc(this.l1Collection, { - query: { filter: `updated_time_ms < ${cutoffMs}` }, - }); - return 0; // actual count unknown from delete API - } catch (err) { - this.logger?.warn(`${TAG} [L1-deleteExpired] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return 0; - } - } - - // โ”€โ”€ L1 Read Operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - async countL1(): Promise { - try { - await this._ensureInit(); - if (this.degraded) return 0; - return await this.client.count(this.l1Collection); - } catch (err) { - this.logger?.warn(`${TAG} [L1-count] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return 0; - } - } - - async queryL1Records(filter?: L1QueryFilter): Promise { - try { - await this._ensureInit(); - if (this.degraded) return []; - - // Build TCVDB filter expression from L1QueryFilter - const conditions: string[] = []; - if (filter?.sessionKey) conditions.push(`session_key = "${filter.sessionKey}"`); - if (filter?.sessionId) conditions.push(`session_id = "${filter.sessionId}"`); - if (filter?.updatedAfter) { - const afterMs = isoToEpochMs(filter.updatedAfter); - if (afterMs > 0) conditions.push(`updated_time_ms > ${afterMs}`); - } - const filterExpr = conditions.length > 0 ? conditions.join(" and ") : undefined; - - const docs = await this._queryAllDocs( - this.l1Collection, - filterExpr, - L1_OUTPUT_FIELDS, - undefined, // no limit โ€” fetch all matching - [{ fieldName: "updated_time_ms", direction: "asc" }], - ); - - return docs.map((doc) => ({ - record_id: String(doc.id ?? ""), - content: String(doc.text ?? ""), - type: String(doc.type ?? ""), - priority: Number(doc.priority ?? 0), - scene_name: String(doc.scene_name ?? ""), - session_key: String(doc.session_key ?? ""), - session_id: String(doc.session_id ?? ""), - timestamp_str: String(doc.timestamp_str ?? ""), - timestamp_start: String(doc.timestamp_start ?? ""), - timestamp_end: String(doc.timestamp_end ?? ""), - created_time: epochMsToIso(Number(doc.created_time_ms ?? 0)), - updated_time: epochMsToIso(Number(doc.updated_time_ms ?? 0)), - metadata_json: String(doc.metadata_json ?? "{}"), - })); - } catch (err) { - this.logger?.warn(`${TAG} [L1-query] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return []; - } - } - - async getAllL1Texts(): Promise> { - try { - await this._ensureInit(); - if (this.degraded) return []; - - const docs = await this._queryAllDocs( - this.l1Collection, - undefined, - ["id", "text", "updated_time_ms"], - ); - - return docs.map((doc) => ({ - record_id: String(doc.id ?? ""), - content: String(doc.text ?? ""), - updated_time: epochMsToIso(Number(doc.updated_time_ms ?? 0)), - })); - } catch (err) { - this.logger?.warn(`${TAG} [L1-getAllTexts] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return []; - } - } - - // โ”€โ”€ L1 Search Operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - async searchL1Vector(_queryEmbedding: Float32Array, topK?: number, queryText?: string): Promise { - // TCVDB uses server-side embedding โ€” delegate to hybrid search with text - if (queryText) { - return this.searchL1HybridAsync({ queryText, topK }); - } - // No queryText and TCVDB can't use client embeddings directly via embeddingItems - // Return empty โ€” callers should pass queryText for TCVDB - return []; - } - - async searchL1Fts(ftsQuery: string, limit?: number): Promise { - // TCVDB has no pure FTS โ€” use hybrid search with sparse-only path - // The ftsQuery is raw text, use it as queryText for hybrid - if (!ftsQuery) return []; - const results = await this.searchL1HybridAsync({ queryText: ftsQuery, topK: limit }); - // L1SearchResult and L1FtsResult have identical shapes - return results; - } - - async searchL1Hybrid(params: { - query?: string; - queryEmbedding?: Float32Array; - sparseVector?: SparseVector; - topK?: number; - }): Promise { - const queryText = params.query; - if (!queryText) return []; - return this.searchL1HybridAsync({ queryText, topK: params.topK }); - } - - /** - * Async L1 hybrid search โ€” the real implementation. - * Call this directly from async contexts (hooks, tools). - */ - async searchL1HybridAsync(params: { - queryText: string; - topK?: number; - }): Promise { - const { queryText, topK = 10 } = params; - if (!queryText) return []; - - try { - await this._ensureInit(); - if (this.degraded) return []; - - // Build search params - const searchParams: Record = { - limit: topK, - outputFields: L1_OUTPUT_FIELDS, - }; - - // ann: use embedding field name "text" for server-side embedding - // (per SDK: AnnSearch(field_name="text", data='query string')) - const ann = [{ - fieldName: "text", - data: [queryText], // embeddingItems โ€” server-side embedding - limit: topK, - }]; - - let match: Array> | undefined; - if (this.bm25Encoder) { - const sparse = this.bm25Encoder.encodeQueries([queryText]); - if (sparse.length > 0 && sparse[0].length > 0) { - match = [{ - fieldName: "sparse_vector", - data: [sparse[0]], // SDK wraps single sparse vector in array - limit: topK, - }]; - } - } - - if (match) { - // Full hybrid: dense + sparse + RRF - searchParams.ann = ann; - searchParams.match = match; - searchParams.rerank = { method: "rrf", k: 60 }; - - const resp = await this.client.hybridSearch(this.l1Collection, searchParams); - return this._parseL1SearchResults(resp.documents); - } else { - // Dense-only fallback (BM25 unavailable) โ€” use /document/search with embeddingItems - const denseSearch: Record = { - embeddingItems: [queryText], - limit: topK, - retrieveVector: false, - outputFields: L1_OUTPUT_FIELDS, - }; - const resp = await this.client.search(this.l1Collection, denseSearch); - return this._parseL1SearchResults(resp.documents); - } - } catch (err) { - this.logger?.warn(`${TAG} [L1-hybridSearch] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return []; - } - } - - // โ”€โ”€ L0 Write Operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - async upsertL0(record: { id: string; sessionKey: string; sessionId: string; role: string; messageText: string; recordedAt: string; timestamp: number }, _embedding?: Float32Array): Promise { - try { - await this._upsertL0Async(record); - return true; - } catch (err) { - this.logger?.warn(`${TAG} [L0-upsert] FAILED id=${record.id}: ${err instanceof Error ? err.message : String(err)}`); - return false; - } - } - - private async _upsertL0Async(record: { id: string; sessionKey: string; sessionId: string; role: string; messageText: string; recordedAt: string; timestamp: number }): Promise { - await this._ensureInit(); - if (this.degraded) return; - - const doc: Record = { - id: record.id, - message_text: record.messageText, - agent_id: extractAgentId(record.sessionKey), - session_key: record.sessionKey, - session_id: record.sessionId, - role: record.role, - recorded_at_ms: isoToEpochMs(record.recordedAt), - timestamp: record.timestamp, - }; - - if (this.bm25Encoder) { - const sparse = this.bm25Encoder.encodeTexts([record.messageText]); - if (sparse.length > 0 && sparse[0].length > 0) { - doc.sparse_vector = sparse[0]; - } - } - - await this.client.upsert(this.l0Collection, [doc]); - } - - /** - * Batch upsert multiple L0 records in a single API call. - * Used by migration scripts to reduce request count. - */ - async upsertL0Batch(records: Array<{ id: string; sessionKey: string; sessionId: string; role: string; messageText: string; recordedAt: string; timestamp: number }>): Promise { - if (records.length === 0) return 0; - try { - await this._ensureInit(); - if (this.degraded) return 0; - - const docs = records.map((record) => { - const doc: Record = { - id: record.id, - message_text: record.messageText, - agent_id: extractAgentId(record.sessionKey), - session_key: record.sessionKey, - session_id: record.sessionId, - role: record.role, - recorded_at_ms: isoToEpochMs(record.recordedAt), - timestamp: record.timestamp, - }; - - if (this.bm25Encoder) { - const sparse = this.bm25Encoder.encodeTexts([record.messageText]); - if (sparse.length > 0 && sparse[0].length > 0) { - doc.sparse_vector = sparse[0]; - } - } - return doc; - }); - - await this.client.upsert(this.l0Collection, docs); - return records.length; - } catch (err) { - this.logger?.warn(`${TAG} [L0-upsertBatch] FAILED (${records.length} records): ${err instanceof Error ? err.message : String(err)}`); - return 0; - } - } - - async deleteL0(recordId: string): Promise { - try { - await this._ensureInit(); - if (this.degraded) return false; - await this.client.deleteDoc(this.l0Collection, { - query: { documentIds: [recordId] }, - }); - return true; - } catch (err) { - this.logger?.warn(`${TAG} [L0-delete] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return false; - } - } - - async deleteL0Expired(cutoffIso: string): Promise { - const cutoffMs = isoToEpochMs(cutoffIso); - if (cutoffMs <= 0) return 0; - try { - await this._ensureInit(); - if (this.degraded) return 0; - await this.client.deleteDoc(this.l0Collection, { - query: { filter: `recorded_at_ms < ${cutoffMs}` }, - }); - return 0; - } catch (err) { - this.logger?.warn(`${TAG} [L0-deleteExpired] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return 0; - } - } - - // โ”€โ”€ L0 Read Operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - async countL0(): Promise { - try { - await this._ensureInit(); - if (this.degraded) return 0; - return await this.client.count(this.l0Collection); - } catch (err) { - this.logger?.warn(`${TAG} [L0-count] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return 0; - } - } - - async queryL0ForL1(sessionKey: string, afterRecordedAtMs?: number, limit = 50): Promise { - try { - await this._ensureInit(); - if (this.degraded) return []; - - const conditions: string[] = [`session_key = "${sessionKey}"`]; - if (afterRecordedAtMs && afterRecordedAtMs > 0) { - conditions.push(`recorded_at_ms > ${afterRecordedAtMs}`); - } - const filterExpr = conditions.join(" and "); - - const docs = await this._queryAllDocs( - this.l0Collection, - filterExpr, - L0_OUTPUT_FIELDS, - limit, - [{ fieldName: "recorded_at_ms", direction: "desc" }], - ); - - // Convert to L0QueryRow and reverse to chronological order (query is DESC, callers expect ASC) - const rows: L0QueryRow[] = docs.map((doc) => ({ - record_id: String(doc.id ?? ""), - session_key: String(doc.session_key ?? ""), - session_id: String(doc.session_id ?? ""), - role: String(doc.role ?? ""), - message_text: String(doc.message_text ?? ""), - recorded_at: epochMsToIso(Number(doc.recorded_at_ms ?? 0)), - timestamp: Number(doc.timestamp ?? 0), - })); - - return rows.reverse(); - } catch (err) { - this.logger?.warn(`${TAG} [L0-queryForL1] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return []; - } - } - - async queryL0GroupedBySessionId(sessionKey: string, afterRecordedAtMs?: number, limit = 50): Promise { - try { - const rows = await this.queryL0ForL1(sessionKey, afterRecordedAtMs, limit); - - // Group by session_id - const groupMap = new Map>(); - for (const row of rows) { - const sid = row.session_id || ""; - let group = groupMap.get(sid); - if (!group) { - group = []; - groupMap.set(sid, group); - } - group.push({ - id: row.record_id, - role: row.role, - content: row.message_text, - timestamp: row.timestamp, - recordedAtMs: row.recorded_at ? Date.parse(row.recorded_at) || 0 : 0, - }); - } - - // Convert to array, sorted by earliest message timestamp - const groups: L0SessionGroup[] = []; - for (const [sessionId, messages] of groupMap) { - if (messages.length > 0) { - groups.push({ sessionId, messages }); - } - } - groups.sort((a, b) => a.messages[0].timestamp - b.messages[0].timestamp); - - return groups; - } catch (err) { - this.logger?.warn(`${TAG} [L0-queryGrouped] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return []; - } - } - - async getAllL0Texts(): Promise> { - try { - await this._ensureInit(); - if (this.degraded) return []; - - const docs = await this._queryAllDocs( - this.l0Collection, - undefined, - ["id", "message_text", "recorded_at_ms"], - ); - - return docs.map((doc) => ({ - record_id: String(doc.id ?? ""), - message_text: String(doc.message_text ?? ""), - recorded_at: epochMsToIso(Number(doc.recorded_at_ms ?? 0)), - })); - } catch (err) { - this.logger?.warn(`${TAG} [L0-getAllTexts] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return []; - } - } - - // โ”€โ”€ L0 Search Operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - async searchL0Vector(_queryEmbedding: Float32Array, topK?: number, queryText?: string): Promise { - // TCVDB uses server-side embedding โ€” delegate to hybrid search with text - if (queryText) { - return this.searchL0HybridAsync({ queryText, topK }); - } - return []; - } - - async searchL0Fts(ftsQuery: string, limit?: number): Promise { - if (!ftsQuery) return []; - // Use hybrid search; L0SearchResult and L0FtsResult have identical shapes - return this.searchL0HybridAsync({ queryText: ftsQuery, topK: limit }); - } - - /** - * Async L0 hybrid search. - */ - async searchL0HybridAsync(params: { - queryText: string; - topK?: number; - }): Promise { - const { queryText, topK = 10 } = params; - if (!queryText) return []; - - try { - await this._ensureInit(); - if (this.degraded) return []; - - const searchParams: Record = { - limit: topK, - outputFields: L0_OUTPUT_FIELDS, - }; - - // ann: use embedding field name "message_text" for L0 server-side embedding - const ann = [{ - fieldName: "message_text", - data: [queryText], - limit: topK, - }]; - - let match: Array> | undefined; - if (this.bm25Encoder) { - const sparse = this.bm25Encoder.encodeQueries([queryText]); - if (sparse.length > 0 && sparse[0].length > 0) { - match = [{ - fieldName: "sparse_vector", - data: [sparse[0]], - limit: topK, - }]; - } - } - - if (match) { - searchParams.ann = ann; - searchParams.match = match; - searchParams.rerank = { method: "rrf", k: 60 }; - const resp = await this.client.hybridSearch(this.l0Collection, searchParams); - return this._parseL0SearchResults(resp.documents); - } else { - const denseSearch: Record = { - embeddingItems: [queryText], - limit: topK, - retrieveVector: false, - outputFields: L0_OUTPUT_FIELDS, - }; - const resp = await this.client.search(this.l0Collection, denseSearch); - return this._parseL0SearchResults(resp.documents); - } - } catch (err) { - this.logger?.warn(`${TAG} [L0-hybridSearch] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return []; - } - } - - async pullProfiles(): Promise { - try { - await this._ensureInit(); - if (this.degraded) return []; - - const docs = await this._queryAllDocs( - this.profilesCollection, - undefined, - PROFILE_OUTPUT_FIELDS, - ); - - return docs.map((doc) => ({ - id: String(doc.id ?? ""), - type: doc.type === "l3" ? "l3" : "l2", - filename: String(doc.filename ?? ""), - content: String(doc.content ?? ""), - contentMd5: String(doc.content_md5 ?? ""), - agentId: String(doc.agent_id ?? "") || undefined, - version: Number(doc.version ?? 0), - createdAtMs: Number(doc.created_at_ms ?? 0), - updatedAtMs: Number(doc.updated_at_ms ?? 0), - })); - } catch (err) { - this.logger?.warn(`${TAG} [profiles-pull] FAILED: ${err instanceof Error ? err.message : String(err)}`); - return []; - } - } - - async syncProfiles(records: ProfileSyncRecord[]): Promise { - if (records.length === 0) return; - - try { - await this._ensureInit(); - if (this.degraded) return; - - const remoteDocs = await this._queryAllDocs( - this.profilesCollection, - undefined, - PROFILE_METADATA_OUTPUT_FIELDS, - ); - const remoteMap = new Map( - remoteDocs.map((doc) => [String(doc.id ?? ""), doc] as const), - ); - const now = Date.now(); - const upserts: Array> = []; - - for (const record of records) { - const current = remoteMap.get(record.id); - if (!current) { - const createdAtMs = record.createdAtMs > 0 ? record.createdAtMs : now; - upserts.push({ - id: record.id, - vector: [0], - type: record.type, - filename: record.filename, - content: record.content, - content_md5: record.contentMd5, - agent_id: record.agentId ?? "", - version: 1, - created_at_ms: createdAtMs, - updated_at_ms: now, - }); - continue; - } - - const currentMd5 = String(current.content_md5 ?? ""); - const currentVersion = Number(current.version ?? 0); - const currentCreatedAtMs = Number(current.created_at_ms ?? 0) || now; - - if (currentMd5 === record.contentMd5) { - continue; - } - - if ((record.baselineVersion ?? 0) !== currentVersion) { - this.logger?.warn( - `${TAG} [profiles-sync] Conflict for ${record.filename}: remote version advanced from ${record.baselineVersion ?? 0} to ${currentVersion}, skipping sync`, - ); - continue; - } - - upserts.push({ - id: record.id, - vector: [0], - type: record.type, - filename: record.filename, - content: record.content, - content_md5: record.contentMd5, - agent_id: record.agentId ?? "", - version: currentVersion + 1, - created_at_ms: currentCreatedAtMs, - updated_at_ms: now, - }); - } - - if (upserts.length > 0) { - await this.client.upsert(this.profilesCollection, upserts); - } - } catch (err) { - this.logger?.warn(`${TAG} [profiles-sync] FAILED: ${err instanceof Error ? err.message : String(err)}`); - } - } - - async deleteProfiles(recordIds: string[]): Promise { - if (recordIds.length === 0) return; - - try { - await this._ensureInit(); - if (this.degraded) return; - await this.client.deleteDoc(this.profilesCollection, { - query: { documentIds: recordIds }, - }); - } catch (err) { - this.logger?.warn(`${TAG} [profiles-delete] FAILED: ${err instanceof Error ? err.message : String(err)}`); - } - } - - // โ”€โ”€ Re-index โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - async reindexAll( - _embedFn: (text: string) => Promise, - _onProgress?: (done: number, total: number, layer: "L1" | "L0") => void, - ): Promise<{ l1Count: number; l0Count: number }> { - // TCVDB uses server-side embedding โ€” reindex means rebuild Collection. - // Not implemented in Phase 2-3 (requires drop + recreate + re-upsert from JSONL). - this.logger?.info(`${TAG} reindexAll: TCVDB uses server-side embedding, skipping`); - return { l1Count: 0, l0Count: 0 }; - } - - isFtsAvailable(): boolean { - return !!this.bm25Encoder; - } - - // โ”€โ”€ Internal: parse search results โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - private _parseL1SearchResults(docArrays: Array>>): L1SearchResult[] { - const results: L1SearchResult[] = []; - // hybridSearch/search returns [[doc, doc, ...]] (one array per query) - const docs = docArrays?.[0] ?? []; - for (const doc of docs) { - results.push({ - record_id: String(doc.id ?? ""), - content: String(doc.text ?? ""), - type: String(doc.type ?? ""), - priority: Number(doc.priority ?? 0), - scene_name: String(doc.scene_name ?? ""), - score: Number(doc.score ?? 0), - timestamp_str: String(doc.timestamp_str ?? ""), - timestamp_start: String(doc.timestamp_start ?? ""), - timestamp_end: String(doc.timestamp_end ?? ""), - session_key: String(doc.session_key ?? ""), - session_id: String(doc.session_id ?? ""), - metadata_json: String(doc.metadata_json ?? "{}"), - }); - } - return results; - } - - private _parseL0SearchResults(docArrays: Array>>): L0SearchResult[] { - const results: L0SearchResult[] = []; - const docs = docArrays?.[0] ?? []; - for (const doc of docs) { - results.push({ - record_id: String(doc.id ?? ""), - session_key: String(doc.session_key ?? ""), - session_id: String(doc.session_id ?? ""), - role: String(doc.role ?? ""), - message_text: String(doc.message_text ?? ""), - score: Number(doc.score ?? 0), - recorded_at: epochMsToIso(Number(doc.recorded_at_ms ?? 0)), - timestamp: Number(doc.timestamp ?? 0), - }); - } - return results; - } -} diff --git a/src/store/types.ts b/src/store/types.ts deleted file mode 100644 index cfcb50a..0000000 --- a/src/store/types.ts +++ /dev/null @@ -1,328 +0,0 @@ -/** - * Memory Store Abstraction Layer โ€” Core Types & Interfaces. - * - * This module defines the storage contracts that all backend implementations - * (SQLite local, Tencent Cloud VectorDB, etc.) must satisfy. - * - * Design principles: - * 1. **Backend-agnostic**: Upper-layer modules (hooks, tools, pipeline, record) - * depend only on these interfaces โ€” never on concrete implementations. - * 2. **Capability-based**: Features like vector search, FTS, and hybrid search - * are expressed as capability flags so callers can gracefully degrade. - * 3. **Fault-tolerant**: All methods return empty results or `false` on - * failure rather than throwing, unless explicitly documented otherwise. - * 4. **Sync-first**: Matches current SQLite DatabaseSync usage. TCVDB backend - * adapts internally without changing these signatures. - */ - -import type { MemoryRecord } from "../record/l1-writer.js"; -import type { EmbeddingProviderInfo } from "./embedding.js"; - -// Re-export so consumers can import everything from types.ts -export type { MemoryRecord, EmbeddingProviderInfo }; - -// ============================ -// Common Types -// ============================ - -/** Minimal logger interface accepted by store implementations. */ -export interface StoreLogger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -// ============================ -// L1 Types (Structured Memories) -// ============================ - -/** Result from an L1 vector similarity search. */ -export interface L1SearchResult { - record_id: string; - content: string; - type: string; - priority: number; - scene_name: string; - /** Similarity score (0โ€“1, higher is better). */ - score: number; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - session_key: string; - session_id: string; - metadata_json: string; -} - -/** Result from an L1 FTS keyword search. */ -export interface L1FtsResult { - record_id: string; - content: string; - type: string; - priority: number; - scene_name: string; - /** BM25-derived score (0โ€“1, higher is better). */ - score: number; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - session_key: string; - session_id: string; - metadata_json: string; -} - -/** Filter options for querying L1 records. */ -export interface L1QueryFilter { - sessionKey?: string; - sessionId?: string; - /** Only return records with updated_time strictly after this ISO 8601 UTC timestamp. */ - updatedAfter?: string; -} - -/** Row shape returned by L1 query methods. */ -export interface L1RecordRow { - record_id: string; - content: string; - type: string; - priority: number; - scene_name: string; - session_key: string; - session_id: string; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - created_time: string; - updated_time: string; - metadata_json: string; -} - -// ============================ -// L0 Types (Raw Conversations) -// ============================ - -/** An L0 conversation message record for vector indexing. */ -export interface L0Record { - id: string; - sessionKey: string; - sessionId: string; - role: string; - messageText: string; - recordedAt: string; - /** Original message timestamp (epoch ms). */ - timestamp: number; -} - -/** Result from an L0 vector similarity search. */ -export interface L0SearchResult { - record_id: string; - session_key: string; - session_id: string; - role: string; - message_text: string; - /** Similarity score (0โ€“1, higher is better). */ - score: number; - recorded_at: string; - timestamp: number; -} - -/** Result from an L0 FTS keyword search. */ -export interface L0FtsResult { - record_id: string; - session_key: string; - session_id: string; - role: string; - message_text: string; - /** BM25-derived score (0โ€“1, higher is better). */ - score: number; - recorded_at: string; - timestamp: number; -} - -/** Raw L0 row returned by query methods (used by L1 runner). */ -export interface L0QueryRow { - record_id: string; - session_key: string; - session_id: string; - role: string; - message_text: string; - recorded_at: string; - timestamp: number; -} - -/** L0 messages grouped by session ID (for L1 runner). */ -export interface L0SessionGroup { - sessionId: string; - messages: Array<{ - id: string; - role: string; - content: string; - timestamp: number; - /** Epoch ms when this message was recorded into L0 (used by L1 cursor). */ - recordedAtMs: number; - }>; -} - -// ============================ -// Store Init Result -// ============================ - -/** Result of store initialization. */ -export interface StoreInitResult { - /** Whether embeddings need to be regenerated (provider/model change). */ - needsReindex: boolean; - /** Human-readable reason (for logging). */ - reason?: string; -} - -// ============================ -// Capability Flags -// ============================ - -/** - * Describes what search capabilities a store backend supports. - * Callers use this to select search strategies and degrade gracefully. - */ -export interface StoreCapabilities { - /** Whether vector (embedding) search is available. */ - vectorSearch: boolean; - /** Whether FTS (full-text keyword) search is available. */ - ftsSearch: boolean; - /** Whether native hybrid search is supported (e.g., TCVDB hybridSearch). */ - nativeHybridSearch: boolean; - /** Whether the store supports sparse vectors (BM25 encoding). */ - sparseVectors: boolean; -} - -// ============================ -// L2/L3 Profile Sync Types -// ============================ - -/** Canonical L2/L3 profile row shared between local cache and remote store. */ -export interface ProfileRecord { - /** Stable ID: `profile:v1:${sha256(scope + "\0" + type + "\0" + filename)}`. */ - id: string; - type: "l2" | "l3"; - filename: string; - content: string; - contentMd5: string; - agentId?: string; - version: number; - createdAtMs: number; - updatedAtMs: number; -} - -/** Profile upsert payload with optimistic-lock baseline from the last pull. */ -export interface ProfileSyncRecord extends ProfileRecord { - baselineVersion?: number; -} - -// ============================ -// IMemoryStore โ€” The Core Abstraction -// ============================ - -/** - * Unified memory store interface. - * - * Implementations: - * - `SqliteMemoryStore` (sqlite.ts) โ€” local SQLite + sqlite-vec + FTS5 - * - `TcvdbMemoryStore` (tcvdb.ts) โ€” Tencent Cloud VectorDB (future) - * - * All methods are fault-tolerant: they return empty results or `false` on - * failure rather than throwing, unless explicitly documented otherwise. - */ -/** - * Helper type: a value that may be sync or async. - * Callers should always `await` the result โ€” it's safe for both sync and async values. - */ -export type MaybePromise = T | Promise; - -export interface IMemoryStore { - // โ”€โ”€ Capabilities โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Whether this store supports deferred (background) embedding updates. - * - * When `true`, auto-capture writes metadata-only via `upsertL0(record, undefined)` - * and later calls `updateL0Embedding()` in a fire-and-forget background task. - * When `false` or absent, embedding is computed inline and passed to `upsertL0()`. - */ - readonly supportsDeferredEmbedding?: boolean; - - // โ”€โ”€ Lifecycle (always sync) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - init(providerInfo?: EmbeddingProviderInfo): MaybePromise; - isDegraded(): boolean; - getCapabilities(): StoreCapabilities; - close(): void; - - // โ”€โ”€ L1 Write โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - upsertL1(record: MemoryRecord, embedding?: Float32Array): MaybePromise; - deleteL1(recordId: string): MaybePromise; - deleteL1Batch(recordIds: string[]): MaybePromise; - deleteL1Expired(cutoffIso: string): MaybePromise; - - // โ”€โ”€ L1 Read โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - countL1(): MaybePromise; - queryL1Records(filter?: L1QueryFilter): MaybePromise; - getAllL1Texts(): MaybePromise>; - - // โ”€โ”€ L1 Search โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - searchL1Vector(queryEmbedding: Float32Array, topK?: number, queryText?: string): MaybePromise; - searchL1Fts(ftsQuery: string, limit?: number): MaybePromise; - searchL1Hybrid?(params: { - query?: string; - queryEmbedding?: Float32Array; - sparseVector?: Array<[number, number]>; - topK?: number; - }): MaybePromise; - - // โ”€โ”€ L0 Write โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - upsertL0(record: L0Record, embedding?: Float32Array): MaybePromise; - /** Update only the vector embedding for an existing L0 record (sqlite background path). */ - updateL0Embedding?(recordId: string, embedding: Float32Array): MaybePromise; - deleteL0(recordId: string): MaybePromise; - deleteL0Expired(cutoffIso: string): MaybePromise; - - // โ”€โ”€ L0 Read โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - countL0(): MaybePromise; - queryL0ForL1(sessionKey: string, afterRecordedAtMs?: number, limit?: number): MaybePromise; - queryL0GroupedBySessionId(sessionKey: string, afterRecordedAtMs?: number, limit?: number): MaybePromise; - getAllL0Texts(): MaybePromise>; - - // โ”€โ”€ L0 Search โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - searchL0Vector(queryEmbedding: Float32Array, topK?: number, queryText?: string): MaybePromise; - searchL0Fts(ftsQuery: string, limit?: number): MaybePromise; - - pullProfiles?(): Promise; - syncProfiles?(records: ProfileSyncRecord[]): Promise; - deleteProfiles?(recordIds: string[]): Promise; - - // โ”€โ”€ Re-index โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - reindexAll( - embedFn: (text: string) => Promise, - onProgress?: (done: number, total: number, layer: "L1" | "L0") => void, - ): Promise<{ l1Count: number; l0Count: number }>; - - // โ”€โ”€ FTS (always sync โ€” cached flag) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - isFtsAvailable(): boolean; -} - -// ============================ -// IEmbeddingService โ€” re-exported from embedding.ts for convenience -// ============================ - -/** - * Re-export EmbeddingService as IEmbeddingService for backward compatibility. - * The canonical definition lives in `./embedding.ts`. All concrete implementations - * (LocalEmbeddingService, OpenAIEmbeddingService, NoopEmbeddingService) implement - * the EmbeddingService interface from embedding.ts. - */ -export type { EmbeddingService as IEmbeddingService } from "./embedding.js"; diff --git a/src/store/vector-store.ts b/src/store/vector-store.ts deleted file mode 100644 index 4696f10..0000000 --- a/src/store/vector-store.ts +++ /dev/null @@ -1,2225 +0,0 @@ -/** - * VectorStore: SQLite-based vector storage using sqlite-vec extension. - * - * Manages two layers of vector-indexed data in a single SQLite database: - * - * **L1 (structured memories):** - * 1. `l1_records` โ€” relational metadata table (content, type, priority, scene, timestamps) - * 2. `l1_vec` โ€” vec0 virtual table for cosine similarity search - * - * **L0 (raw conversations):** - * 3. `l0_conversations` โ€” relational metadata table (session_key, role, message text, timestamps) - * 4. `l0_vec` โ€” vec0 virtual table for cosine similarity search on individual messages - * - * Dependencies: Node.js built-in `node:sqlite` (Node 22+) + `sqlite-vec` (from root workspace). - * - * Design: - * - All operations are synchronous (DatabaseSync API). - * - Writes use manual BEGIN/COMMIT transactions for atomicity (metadata + vector). - * - vec0 virtual table does NOT support ON CONFLICT, so upsert = delete + insert. - * - Thread-safe via WAL mode. - */ - -import { createRequire } from "node:module"; -import type { DatabaseSync, StatementSync } from "node:sqlite"; -import type { MemoryRecord } from "../record/l1-writer.js"; -import type { EmbeddingProviderInfo } from "./embedding.js"; - -// ============================ -// Types -// ============================ - -export interface VectorSearchResult { - record_id: string; - content: string; - type: string; - priority: number; - scene_name: string; - /** Cosine similarity score (1.0 - cosine_distance) */ - score: number; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - session_key: string; - session_id: string; - /** Raw metadata JSON string (e.g., contains activity_start_time / activity_end_time for episodic) */ - metadata_json: string; -} - -/** L0 single-message record for vector indexing. */ -export interface L0VectorRecord { - /** Unique ID for this message (e.g., "l0___") */ - id: string; - /** Session key this message belongs to */ - sessionKey: string; - /** Session ID (single conversation instance) this message belongs to */ - sessionId: string; - /** Role of the message sender: "user" or "assistant" */ - role: string; - /** Text content of this single message */ - messageText: string; - /** ISO timestamp when this message was recorded */ - recordedAt: string; - /** Original message timestamp (epoch ms) โ€” used by L1 cursor for incremental filtering */ - timestamp: number; -} - -/** L0 single-message vector search result. */ -export interface L0VectorSearchResult { - record_id: string; - session_key: string; - session_id: string; - role: string; - message_text: string; - /** Cosine similarity score (1.0 - cosine_distance) */ - score: number; - recorded_at: string; - /** Original message timestamp (epoch ms) */ - timestamp: number; -} - -/** Raw row returned by L1 record queries (column names match SQLite schema). */ -export interface L1RecordRow { - record_id: string; - content: string; - type: string; - priority: number; - scene_name: string; - session_key: string; - session_id: string; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - created_time: string; - updated_time: string; - metadata_json: string; -} - -/** Filter options for querying L1 records from SQLite. */ -export interface L1QueryFilter { - /** If provided, only return records for this session key (conversation channel). */ - sessionKey?: string; - /** If provided, only return records for this session ID (single conversation instance). */ - sessionId?: string; - /** If provided, only return records with updated_time strictly after this ISO 8601 UTC timestamp. */ - updatedAfter?: string; -} - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -const TAG = "[memory-tdai][vector-store]"; - -/** Persisted metadata about the embedding provider used to generate stored vectors. */ -interface EmbeddingMeta { - provider: string; - model: string; - dimensions: number; -} - -/** Result of VectorStore.init() โ€” indicates whether a re-embed is needed. */ -export interface VectorStoreInitResult { - /** - * `true` if the embedding provider/model/dimensions changed since - * the vectors were last written. Callers should re-embed all texts - * (via `reindexAll()`) after receiving this flag. - */ - needsReindex: boolean; - /** Human-readable reason (for logging). */ - reason?: string; -} - -// Use createRequire to load the experimental node:sqlite module -const require = createRequire(import.meta.url); - -function requireNodeSqlite(): typeof import("node:sqlite") { - return require("node:sqlite") as typeof import("node:sqlite"); -} - -// ============================ -// FTS5 helpers (adapted from openclaw core hybrid.ts) -// ============================ - -// โ”€โ”€ Chinese word segmentation (jieba) โ”€โ”€ -// Lazy-loaded singleton: initialised on first call to `buildFtsQuery`. -// If @node-rs/jieba is unavailable, falls back to Unicode-regex splitting. - -interface JiebaInstance { - cutForSearch(text: string, hmm: boolean): string[]; -} - -let _jieba: JiebaInstance | null | undefined; // undefined = not yet tried - -function getJieba(): JiebaInstance | null { - if (_jieba !== undefined) return _jieba; - try { - // eslint-disable-next-line @typescript-eslint/no-require-imports - const { Jieba } = require("@node-rs/jieba"); - // eslint-disable-next-line @typescript-eslint/no-require-imports - const { dict } = require("@node-rs/jieba/dict"); - _jieba = Jieba.withDict(dict) as JiebaInstance; - } catch { - _jieba = null; // mark as unavailable โ€” won't retry - } - return _jieba; -} - -/** - * Common Chinese stop-words that add noise to FTS5 queries. - * Kept small on purpose โ€” only high-frequency function words. - */ -const ZH_STOP_WORDS = new Set([ - "็š„", "ไบ†", "ๅœจ", "ๆ˜ฏ", "ๆˆ‘", "ๆœ‰", "ๅ’Œ", "ๅฐฑ", "ไธ", "ไบบ", "้ƒฝ", "ไธ€", - "ไธ€ไธช", "ไธŠ", "ไนŸ", "ๅพˆ", "ๅˆฐ", "่ฏด", "่ฆ", "ๅŽป", "ไฝ ", "ไผš", "็€", - "ๆฒกๆœ‰", "็œ‹", "ๅฅฝ", "่‡ชๅทฑ", "่ฟ™", "ไป–", "ๅฅน", "ๅฎƒ", "ไปฌ", "้‚ฃ", - "ๅ—", "ๅง", "ๅ‘ข", "ๅ•Š", "ๅ‘€", "ๅ“ฆ", "ๅ—ฏ", -]); - -/** - * Build an FTS5 MATCH query from raw text. - * - * When `@node-rs/jieba` is available, uses jieba's search-engine mode - * (`cutForSearch`) for accurate Chinese word segmentation, producing - * much better recall than the previous regex-only approach. - * - * Falls back to Unicode-regex splitting (`/[\p{L}\p{N}_]+/gu`) if - * jieba is not installed. - * - * Tokens are OR-joined as quoted FTS5 phrase terms so that a document - * matching *any* token is returned. BM25 naturally ranks documents that - * match more tokens higher, so precision is preserved while recall is - * significantly improved โ€” especially for longer queries and when running - * in FTS-only fallback mode (no embedding available). - * - * Example (with jieba): - * "็”จๆˆทๅ–œๆฌข็ผ–็จ‹ๅ’ŒTypeScript" โ†’ '"็”จๆˆท" OR "ๅ–œๆฌข" OR "็ผ–็จ‹" OR "TypeScript"' - * Example (fallback): - * "ๆ—…่กŒ่ฎกๅˆ’ API" โ†’ '"ๆ—…่กŒ่ฎกๅˆ’" OR "API"' - */ -export function buildFtsQuery(raw: string): string | null { - const jieba = getJieba(); - - let tokens: string[]; - if (jieba) { - // jieba cutForSearch: splits long words further for better recall - // e.g. "ๅŒ—ไบฌ็ƒค้ธญ" โ†’ ["ๅŒ—ไบฌ", "็ƒค้ธญ", "ๅŒ—ไบฌ็ƒค้ธญ"] - tokens = jieba - .cutForSearch(raw, true) - .map((t) => t.trim()) - .filter((t) => { - if (!t) return false; - // Remove pure whitespace / punctuation tokens - if (!/[\p{L}\p{N}]/u.test(t)) return false; - // Remove common Chinese stop-words to reduce noise - if (ZH_STOP_WORDS.has(t)) return false; - return true; - }); - // Deduplicate (cutForSearch may produce duplicates for sub-words) - tokens = [...new Set(tokens)]; - } else { - // Fallback: simple Unicode regex split - tokens = - raw - .match(/[\p{L}\p{N}_]+/gu) - ?.map((t) => t.trim()) - .filter(Boolean) ?? []; - } - - if (tokens.length === 0) return null; - const quoted = tokens.map((t) => `"${t.replaceAll('"', "")}"`); - return quoted.join(" OR "); -} - -/** - * Tokenize text for FTS5 indexing (write-side). - * - * Uses jieba `cutForSearch()` (search-engine mode) to segment Chinese text, - * then joins tokens with spaces. The resulting string is stored in the FTS5 - * `content` column so that `unicode61` tokenizer can split it into meaningful - * words โ€” including both full words and their sub-words. - * - * Using `cutForSearch` (instead of `cut`) ensures that the index contains - * the same sub-word tokens that `buildFtsQuery()` produces on the query side. - * For example, "ไบบๅทฅๆ™บ่ƒฝ" is indexed as "ไบบๅทฅ ๆ™บ่ƒฝ ไบบๅทฅๆ™บ่ƒฝ", so queries for - * either the full term or sub-words will match. - * - * Falls back to the original text if jieba is unavailable. - * - * Example (with jieba): - * "็”จๆˆทไบ”ๆœˆๅŽปๆ—ฅๆœฌๆ—…่กŒ" โ†’ "็”จๆˆท ไบ”ๆœˆ ๅŽป ๆ—ฅๆœฌ ๆ—…่กŒ" - * "ไบบๅทฅๆ™บ่ƒฝ็š„ๅˆ†ๆ”ฏ" โ†’ "ไบบๅทฅ ๆ™บ่ƒฝ ไบบๅทฅๆ™บ่ƒฝ ็š„ ๅˆ†ๆ”ฏ" - * Example (fallback): - * "็”จๆˆทไบ”ๆœˆๅŽปๆ—ฅๆœฌๆ—…่กŒ" โ†’ "็”จๆˆทไบ”ๆœˆๅŽปๆ—ฅๆœฌๆ—…่กŒ" (unchanged) - */ -export function tokenizeForFts(raw: string): string { - const jieba = getJieba(); - if (!jieba) return raw; - - // Use `cutForSearch` (search-engine mode) for indexing โ€” it produces both - // full words AND their sub-word components. This ensures that query-side - // tokens (also produced by `cutForSearch` in `buildFtsQuery`) will always - // find a match in the index. - const tokens = jieba.cutForSearch(raw, true); - - // Join with spaces so `unicode61` tokenizer can split them. - // Punctuation tokens are kept โ€” unicode61 treats them as separators anyway. - return tokens.join(" "); -} - -/** - * Reset jieba state so next call to `buildFtsQuery` re-initialises. - * Exported for testing only. - * @internal - */ -export function _resetJiebaForTest(): void { - _jieba = undefined; -} - -/** - * Override jieba instance (or set to `null` to force fallback). - * Exported for testing only. - * @internal - */ -export function _setJiebaForTest(instance: JiebaInstance | null): void { - _jieba = instance; -} - -/** - * Convert a BM25 rank (negative = more relevant) to a 0โ€“1 score. - * Mirrors the formula in openclaw core `hybrid.ts`. - */ -export function bm25RankToScore(rank: number): number { - if (!Number.isFinite(rank)) return 1 / (1 + 999); - if (rank < 0) { - const relevance = -rank; - return relevance / (1 + relevance); - } - return 1 / (1 + rank); -} - -/** FTS5 search result for L1 records. */ -export interface FtsSearchResult { - record_id: string; - content: string; - type: string; - priority: number; - scene_name: string; - /** BM25-derived score (0โ€“1, higher is better) */ - score: number; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - session_key: string; - session_id: string; - metadata_json: string; -} - -/** FTS5 search result for L0 records. */ -export interface L0FtsSearchResult { - record_id: string; - session_key: string; - session_id: string; - role: string; - message_text: string; - /** BM25-derived score (0โ€“1, higher is better) */ - score: number; - recorded_at: string; - timestamp: number; -} - -// ============================ -// VectorStore class -// ============================ - -export class VectorStore { - private db: DatabaseSync; - private readonly dimensions: number; - private readonly logger?: Logger; - - /** - * When `true`, the store is in a degraded state (e.g. sqlite-vec failed to - * load, or init() encountered an unrecoverable error). All public methods - * become safe no-ops so the plugin never blocks the main OpenClaw flow. - */ - private degraded = false; - - /** Tracks whether close() has been called to prevent double-close errors. */ - private closed = false; - - /** - * `true` when vec0 virtual tables (l1_vec / l0_vec) have been created and - * their prepared statements are ready. When `dimensions === 0` (i.e. - * provider="none"), vec0 tables are deferred and this stays `false`. - */ - private vecTablesReady = false; - - // Prepared statements โ€” L1 (initialized in init()) - private stmtUpsertMeta!: StatementSync; - private stmtDeleteVec?: StatementSync; // optional โ€” only set when vecTablesReady - private stmtInsertVec?: StatementSync; // optional โ€” only set when vecTablesReady - private stmtDeleteMeta!: StatementSync; - private stmtGetMeta!: StatementSync; - private stmtSearchVec?: StatementSync; // optional โ€” only set when vecTablesReady - private stmtQueryBySessionId!: StatementSync; - private stmtQueryBySessionIdSince!: StatementSync; - private stmtQueryBySessionKey!: StatementSync; - private stmtQueryBySessionKeySince!: StatementSync; - private stmtQueryAll!: StatementSync; - private stmtQueryAllSince!: StatementSync; - - // Prepared statements โ€” L0 (initialized in init()) - private stmtL0UpsertMeta!: StatementSync; - private stmtL0DeleteVec?: StatementSync; // optional โ€” only set when vecTablesReady - private stmtL0InsertVec?: StatementSync; // optional โ€” only set when vecTablesReady - private stmtL0DeleteMeta!: StatementSync; - private stmtL0GetMeta!: StatementSync; - private stmtL0SearchVec?: StatementSync; // optional โ€” only set when vecTablesReady - /** L0 query for L1 runner: all messages for a session key */ - private stmtL0QueryAll!: StatementSync; - /** L0 query for L1 runner: messages after a timestamp cursor */ - private stmtL0QueryAfter!: StatementSync; - - // FTS5 tables availability flag (created best-effort โ€” may be false if fts5 is not compiled in) - private ftsAvailable = false; - - // Prepared statements โ€” FTS5 L1 (initialized in init()) - private stmtL1FtsInsert!: StatementSync; - private stmtL1FtsDelete!: StatementSync; - private stmtL1FtsSearch!: StatementSync; - - // Prepared statements โ€” FTS5 L0 (initialized in init()) - private stmtL0FtsInsert!: StatementSync; - private stmtL0FtsDelete!: StatementSync; - private stmtL0FtsSearch!: StatementSync; - - /** - * Create a VectorStore instance. - * - * Note: After construction, you MUST call `init()` to load the sqlite-vec - * extension and create the schema. - */ - constructor(dbPath: string, dimensions: number, logger?: Logger) { - this.dimensions = dimensions; - this.logger = logger; - - // Open database with extension support enabled - const { DatabaseSync: DbSync } = requireNodeSqlite(); - this.db = new DbSync(dbPath, { allowExtension: true }); - - // Set busy timeout so concurrent processes retry instead of failing with SQLITE_BUSY - this.db.exec("PRAGMA busy_timeout = 5000"); - - // Enable WAL mode for better concurrent read performance - this.db.exec("PRAGMA journal_mode = WAL"); - - // Cap page cache at 64 MB - this.db.exec("PRAGMA cache_size = -65536"); - - // Cap memory-mapped I/O at 128 MB to bound RSS growth - this.db.exec("PRAGMA mmap_size = 134217728"); - - // Auto-checkpoint WAL every 1000 pages (~4 MB) to keep WAL file compact - this.db.exec("PRAGMA wal_autocheckpoint = 1000"); - } - - /** - * Whether the store is in degraded mode (e.g. sqlite-vec failed to load). - * When degraded, all write/search operations become safe no-ops. - */ - isDegraded(): boolean { - return this.degraded; - } - - - /** - * Load sqlite-vec extension and initialize database schema. - * Must be called once after construction. - * - * @param providerInfo Current embedding provider info. When provided, - * the store compares it against the persisted metadata. If the provider, - * model, or dimensions changed, the vector tables are dropped and - * re-created with the new dimensions, and `needsReindex: true` is returned - * so the caller can schedule a full re-embed. - */ - init(providerInfo?: EmbeddingProviderInfo): VectorStoreInitResult { - // Load sqlite-vec extension (same approach as root project's sqlite-vec.ts) - try { - // eslint-disable-next-line @typescript-eslint/no-require-imports - const sqliteVec = require("sqlite-vec"); - this.db.enableLoadExtension(true); - sqliteVec.load(this.db); - } catch (err) { - const message = err instanceof Error ? err.message : String(err); - this.logger?.error( - `${TAG} Failed to load sqlite-vec extension: ${message}. ` + - `VectorStore entering degraded mode โ€” all operations will be no-ops.`, - ); - this.degraded = true; - return { needsReindex: false, reason: `sqlite-vec load failed: ${message}` }; - } - - // โ”€โ”€ Schema creation & prepared statements โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - // Wrapped in try-catch: if anything fails during schema init (e.g. the DB - // is corrupted, disk full, etc.), we degrade gracefully instead of crashing. - try { - return this.initSchema(providerInfo); - } catch (err) { - const message = err instanceof Error ? err.message : String(err); - this.logger?.error( - `${TAG} Schema initialization failed: ${message}. ` + - `VectorStore entering degraded mode.`, - ); - this.degraded = true; - return { needsReindex: false, reason: `schema init failed: ${message}` }; - } - } - - /** - * Internal schema initialization โ€” separated from init() so we can - * catch errors at the top level and degrade gracefully. - */ - private initSchema(providerInfo?: EmbeddingProviderInfo): VectorStoreInitResult { - // Tracks which provider/model/dimensions were used to generate vectors. - this.db.exec(` - CREATE TABLE IF NOT EXISTS embedding_meta ( - key TEXT PRIMARY KEY, - value TEXT NOT NULL - ) - `); - - // Detect whether re-index is needed - let needsReindex = false; - let reindexReason: string | undefined; - - const savedMeta = this.readEmbeddingMeta(); - - if (providerInfo) { - if (savedMeta) { - const providerChanged = savedMeta.provider !== providerInfo.provider; - const modelChanged = savedMeta.model !== providerInfo.model; - const dimsChanged = savedMeta.dimensions !== this.dimensions; - - if (providerChanged || modelChanged || dimsChanged) { - const reasons: string[] = []; - if (providerChanged) reasons.push(`provider: ${savedMeta.provider} โ†’ ${providerInfo.provider}`); - if (modelChanged) reasons.push(`model: ${savedMeta.model} โ†’ ${providerInfo.model}`); - if (dimsChanged) reasons.push(`dimensions: ${savedMeta.dimensions} โ†’ ${this.dimensions}`); - reindexReason = reasons.join(", "); - - this.logger?.info( - `${TAG} Embedding config changed (${reindexReason}). ` + - `Dropping vector tables for rebuild...`, - ); - - // Drop and re-create vector tables with new dimensions - this.dropVectorTables(); - needsReindex = true; - } - } else { - // No saved meta โ€” first run or legacy DB without meta table. - // Two cases require dropping vector tables: - // 1. Existing data created without meta tracking (legacy DB) โ€” need re-embed - // 2. vec0 tables exist with wrong dimensions (e.g. previously created with - // provider="none" placeholder 768D, now switching to a real provider - // with different dimensions) โ€” must rebuild even if data tables are empty - const l1Count = this.tableRowCount("l1_records"); - const l0Count = this.tableRowCount("l0_conversations"); - const existingVecDims = this.getVecTableDimensions(); - - if (l1Count > 0 || l0Count > 0) { - this.logger?.info( - `${TAG} No embedding_meta found but existing data exists ` + - `(L1=${l1Count}, L0=${l0Count}). Dropping vector tables for safety...`, - ); - this.dropVectorTables(); - needsReindex = true; - reindexReason = "legacy DB without embedding_meta โ€” cannot verify vector compatibility"; - } else if (existingVecDims !== null && existingVecDims !== this.dimensions) { - // vec0 tables exist (from a previous provider="none" placeholder or - // different config) but with mismatched dimensions. Drop them so they - // get re-created with the correct dimensions below. - this.logger?.info( - `${TAG} vec0 table dimension mismatch (existing=${existingVecDims}, ` + - `required=${this.dimensions}). Dropping vector tables for rebuild...`, - ); - this.dropVectorTables(); - // No needsReindex โ€” there's no data to re-embed - } - } - } - - // โ”€โ”€ L1 schema โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - // Metadata table - this.db.exec(` - CREATE TABLE IF NOT EXISTS l1_records ( - record_id TEXT PRIMARY KEY, - content TEXT NOT NULL, - type TEXT DEFAULT '', - priority INTEGER DEFAULT 50, - scene_name TEXT DEFAULT '', - session_key TEXT DEFAULT '', - session_id TEXT DEFAULT '', - timestamp_str TEXT DEFAULT '', - timestamp_start TEXT DEFAULT '', - timestamp_end TEXT DEFAULT '', - created_time TEXT DEFAULT '', - updated_time TEXT DEFAULT '', - metadata_json TEXT DEFAULT '{}' - ) - `); - - // Indexes for common queries - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_type ON l1_records(type)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_session_key ON l1_records(session_key)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_session_id ON l1_records(session_id)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_scene ON l1_records(scene_name)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_ts_start ON l1_records(timestamp_start)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_ts_end ON l1_records(timestamp_end)"); - // Composite index: session_id exact match + updated_time range scan (for incremental L2 queries) - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_session_updated ON l1_records(session_id, updated_time)"); - // Composite index: session_key exact match + updated_time range scan (for pipeline cursor queries) - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_sessionkey_updated ON l1_records(session_key, updated_time)"); - - // Vector virtual table (cosine distance) โ€” only created when dimensions > 0. - // When provider="none", dimensions=0 and vec0 tables are deferred until a - // real embedding provider is configured. - if (this.dimensions > 0) { - this.db.exec(` - CREATE VIRTUAL TABLE IF NOT EXISTS l1_vec USING vec0( - record_id TEXT PRIMARY KEY, - embedding float[${this.dimensions}] distance_metric=cosine, - updated_time TEXT DEFAULT '' - ) - `); - } - - // Prepare statements for reuse - this.stmtUpsertMeta = this.db.prepare(` - INSERT INTO l1_records ( - record_id, content, type, priority, scene_name, session_key, session_id, - timestamp_str, timestamp_start, timestamp_end, - created_time, updated_time, metadata_json - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - ON CONFLICT(record_id) DO UPDATE SET - content=excluded.content, - type=excluded.type, - priority=excluded.priority, - scene_name=excluded.scene_name, - timestamp_str=excluded.timestamp_str, - timestamp_start=excluded.timestamp_start, - timestamp_end=excluded.timestamp_end, - updated_time=excluded.updated_time, - metadata_json=excluded.metadata_json - `); - - if (this.dimensions > 0) { - this.stmtDeleteVec = this.db.prepare("DELETE FROM l1_vec WHERE record_id = ?"); - this.stmtInsertVec = this.db.prepare("INSERT INTO l1_vec (record_id, embedding, updated_time) VALUES (?, ?, ?)"); - } - this.stmtDeleteMeta = this.db.prepare("DELETE FROM l1_records WHERE record_id = ?"); - - this.stmtGetMeta = this.db.prepare(` - SELECT content, type, priority, scene_name, session_key, session_id, - timestamp_str, timestamp_start, timestamp_end, metadata_json - FROM l1_records WHERE record_id = ? - `); - - if (this.dimensions > 0) { - this.stmtSearchVec = this.db.prepare(` - SELECT record_id, distance - FROM l1_vec - WHERE embedding MATCH ? - AND k = ? - ORDER BY distance - `); - } - - // โ”€โ”€ L0 schema โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - // L0 metadata table: stores individual messages for vector search - this.db.exec(` - CREATE TABLE IF NOT EXISTS l0_conversations ( - record_id TEXT PRIMARY KEY, - session_key TEXT NOT NULL, - session_id TEXT DEFAULT '', - role TEXT NOT NULL DEFAULT '', - message_text TEXT NOT NULL, - recorded_at TEXT DEFAULT '', - timestamp INTEGER DEFAULT 0 - ) - `); - - // Migration: add timestamp column if missing (existing DBs pre-v3.x) - try { - this.db.exec("ALTER TABLE l0_conversations ADD COLUMN timestamp INTEGER DEFAULT 0"); - this.logger?.info(`${TAG} Migrated l0_conversations: added timestamp column`); - } catch { - // Column already exists โ€” expected on non-first run - } - - // Indexes for L0 queries - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l0_session ON l0_conversations(session_key)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l0_session_id ON l0_conversations(session_id)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l0_recorded ON l0_conversations(recorded_at)"); - this.db.exec("CREATE INDEX IF NOT EXISTS idx_l0_timestamp ON l0_conversations(timestamp)"); - - // L0 vector virtual table (cosine distance, same dimensions as L1) โ€” deferred when dimensions=0 - if (this.dimensions > 0) { - this.db.exec(` - CREATE VIRTUAL TABLE IF NOT EXISTS l0_vec USING vec0( - record_id TEXT PRIMARY KEY, - embedding float[${this.dimensions}] distance_metric=cosine, - recorded_at TEXT DEFAULT '' - ) - `); - } - - // L0 prepared statements - this.stmtL0UpsertMeta = this.db.prepare(` - INSERT INTO l0_conversations ( - record_id, session_key, session_id, role, message_text, recorded_at, timestamp - ) VALUES (?, ?, ?, ?, ?, ?, ?) - ON CONFLICT(record_id) DO UPDATE SET - message_text=excluded.message_text, - recorded_at=excluded.recorded_at, - timestamp=excluded.timestamp - `); - - if (this.dimensions > 0) { - this.stmtL0DeleteVec = this.db.prepare("DELETE FROM l0_vec WHERE record_id = ?"); - this.stmtL0InsertVec = this.db.prepare("INSERT INTO l0_vec (record_id, embedding, recorded_at) VALUES (?, ?, ?)"); - } - this.stmtL0DeleteMeta = this.db.prepare("DELETE FROM l0_conversations WHERE record_id = ?"); - - this.stmtL0GetMeta = this.db.prepare(` - SELECT session_key, session_id, role, message_text, recorded_at, timestamp - FROM l0_conversations WHERE record_id = ? - `); - - if (this.dimensions > 0) { - this.stmtL0SearchVec = this.db.prepare(` - SELECT record_id, distance - FROM l0_vec - WHERE embedding MATCH ? - AND k = ? - ORDER BY distance - `); - } - - // L0 query statements for L1 runner (newest-first + LIMIT to bound memory) - this.stmtL0QueryAll = this.db.prepare(` - SELECT record_id, session_key, session_id, role, message_text, recorded_at, timestamp - FROM l0_conversations - WHERE session_key = ? - ORDER BY timestamp DESC - LIMIT ? - `); - - this.stmtL0QueryAfter = this.db.prepare(` - SELECT record_id, session_key, session_id, role, message_text, recorded_at, timestamp - FROM l0_conversations - WHERE session_key = ? AND timestamp > ? - ORDER BY timestamp DESC - LIMIT ? - `); - - // โ”€โ”€ FTS5 tables (best-effort โ€” gracefully degrade if fts5 is not compiled in) โ”€โ”€ - // Schema v2: `content` column stores jieba-segmented text (for indexing), - // `content_original` (UNINDEXED) stores the raw text (for display). - // If old v1 tables exist (no content_original column), drop + recreate. - try { - // โ”€โ”€ Migrate old FTS5 tables (v1 โ†’ v2) โ”€โ”€ - // v1 tables stored raw text in the `content` column. v2 stores segmented - // text in `content` and raw text in `content_original` / `message_text_original`. - // FTS5 virtual tables don't support ALTER TABLE ADD COLUMN, so we must - // drop and recreate. The data will be repopulated by `rebuildFtsIndex()`. - const needsFtsRebuild = this.migrateFtsTablesIfNeeded(); - - // L1 FTS5 virtual table (v2 schema) - this.db.exec(` - CREATE VIRTUAL TABLE IF NOT EXISTS l1_fts USING fts5( - content, - content_original UNINDEXED, - record_id UNINDEXED, - type UNINDEXED, - priority UNINDEXED, - scene_name UNINDEXED, - session_key UNINDEXED, - session_id UNINDEXED, - timestamp_str UNINDEXED, - timestamp_start UNINDEXED, - timestamp_end UNINDEXED, - metadata_json UNINDEXED - ) - `); - - // L0 FTS5 virtual table (v2 schema) - this.db.exec(` - CREATE VIRTUAL TABLE IF NOT EXISTS l0_fts USING fts5( - message_text, - message_text_original UNINDEXED, - record_id UNINDEXED, - session_key UNINDEXED, - session_id UNINDEXED, - role UNINDEXED, - recorded_at UNINDEXED, - timestamp UNINDEXED - ) - `); - - // L1 FTS prepared statements - this.stmtL1FtsInsert = this.db.prepare(` - INSERT INTO l1_fts (content, content_original, record_id, type, priority, scene_name, - session_key, session_id, timestamp_str, timestamp_start, timestamp_end, metadata_json) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - `); - - this.stmtL1FtsDelete = this.db.prepare("DELETE FROM l1_fts WHERE record_id = ?"); - - this.stmtL1FtsSearch = this.db.prepare(` - SELECT record_id, content_original AS content, type, priority, scene_name, - session_key, session_id, timestamp_str, timestamp_start, timestamp_end, - metadata_json, - bm25(l1_fts) AS rank - FROM l1_fts - WHERE l1_fts MATCH ? - ORDER BY rank ASC - LIMIT ? - `); - - // L0 FTS prepared statements - this.stmtL0FtsInsert = this.db.prepare(` - INSERT INTO l0_fts (message_text, message_text_original, record_id, session_key, session_id, role, recorded_at, timestamp) - VALUES (?, ?, ?, ?, ?, ?, ?, ?) - `); - - this.stmtL0FtsDelete = this.db.prepare("DELETE FROM l0_fts WHERE record_id = ?"); - - this.stmtL0FtsSearch = this.db.prepare(` - SELECT record_id, message_text_original AS message_text, session_key, session_id, role, recorded_at, timestamp, - bm25(l0_fts) AS rank - FROM l0_fts - WHERE l0_fts MATCH ? - ORDER BY rank ASC - LIMIT ? - `); - - this.ftsAvailable = true; - this.logger?.info(`${TAG} FTS5 tables initialized (l1_fts, l0_fts) [schema v2 โ€” jieba segmented]`); - - // Rebuild FTS index if migrated from v1 or tables were freshly created - if (needsFtsRebuild) { - this.rebuildFtsIndex(); - } - } catch (err) { - const message = err instanceof Error ? err.message : String(err); - this.ftsAvailable = false; - this.logger?.warn( - `${TAG} FTS5 tables NOT available (fts5 may not be compiled in): ${message}. ` + - `FTS-based keyword search will be unavailable; recall will use in-memory scoring if needed.`, - ); - } - - // Save current embedding meta (write after schema is ready) - if (providerInfo) { - this.writeEmbeddingMeta({ - provider: providerInfo.provider, - model: providerInfo.model, - dimensions: this.dimensions, - }); - } - - // Mark vec0 tables as ready only when they were actually created - this.vecTablesReady = this.dimensions > 0; - // L1 query statements (for l1-reader) - const l1QueryCols = `record_id, content, type, priority, scene_name, session_key, session_id, - timestamp_str, timestamp_start, timestamp_end, - created_time, updated_time, metadata_json`; - - this.stmtQueryBySessionId = this.db.prepare(` - SELECT ${l1QueryCols} FROM l1_records - WHERE session_id = ? - ORDER BY updated_time ASC - `); - - this.stmtQueryBySessionIdSince = this.db.prepare(` - SELECT ${l1QueryCols} FROM l1_records - WHERE session_id = ? AND updated_time > ? - ORDER BY updated_time ASC - `); - - this.stmtQueryBySessionKey = this.db.prepare(` - SELECT ${l1QueryCols} FROM l1_records - WHERE session_key = ? - ORDER BY updated_time ASC - `); - - this.stmtQueryBySessionKeySince = this.db.prepare(` - SELECT ${l1QueryCols} FROM l1_records - WHERE session_key = ? AND updated_time > ? - ORDER BY updated_time ASC - `); - - this.stmtQueryAll = this.db.prepare(` - SELECT ${l1QueryCols} FROM l1_records - ORDER BY updated_time ASC - `); - - this.stmtQueryAllSince = this.db.prepare(` - SELECT ${l1QueryCols} FROM l1_records - WHERE updated_time > ? - ORDER BY updated_time ASC - `); - - this.logger?.info(`${TAG} Initialized (dimensions=${this.dimensions})`); - - return { needsReindex, reason: reindexReason }; - } - - // โ”€โ”€ Embedding meta helpers โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - private readEmbeddingMeta(): EmbeddingMeta | null { - try { - const row = this.db - .prepare("SELECT value FROM embedding_meta WHERE key = ?") - .get("embedding_provider_info") as { value: string } | undefined; - if (!row) return null; - return JSON.parse(row.value) as EmbeddingMeta; - } catch { - return null; - } - } - - private writeEmbeddingMeta(meta: EmbeddingMeta): void { - this.db.prepare( - "INSERT INTO embedding_meta (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value=excluded.value", - ).run("embedding_provider_info", JSON.stringify(meta)); - } - - /** Allowed table names for row counting (whitelist to prevent SQL injection). */ - private static readonly COUNTABLE_TABLES = new Set(["l1_records", "l0_conversations"]); - - /** - * Extra rows to retrieve from vec0 KNN search to compensate for legacy - * zero-vector placeholders that may still linger from older data. - */ - private static readonly ZERO_VEC_BUFFER = 10; - - /** Default result limit for FTS5 keyword searches. */ - private static readonly FTS_DEFAULT_LIMIT = 20; - - private tableRowCount(table: string): number { - if (!VectorStore.COUNTABLE_TABLES.has(table)) { - this.logger?.warn(`${TAG} tableRowCount: rejected unknown table name "${table}"`); - return 0; - } - try { - const row = this.db - .prepare(`SELECT COUNT(*) AS cnt FROM ${table}`) - .get() as { cnt: number } | undefined; - return row?.cnt ?? 0; - } catch { - return 0; - } - } - - /** - * Detect the embedding dimension of an existing vec0 table by inspecting - * the DDL stored in sqlite_master. Returns `null` if the table doesn't - * exist or the dimension cannot be determined. - * - * The vec0 DDL looks like: - * CREATE VIRTUAL TABLE l1_vec USING vec0(... embedding float[768] ...) - * We parse the number inside `float[N]`. - */ - private getVecTableDimensions(): number | null { - try { - const row = this.db - .prepare("SELECT sql FROM sqlite_master WHERE type='table' AND name=?") - .get("l1_vec") as { sql: string } | undefined; - if (!row?.sql) return null; - const match = row.sql.match(/float\[(\d+)\]/); - return match ? Number(match[1]) : null; - } catch { - return null; - } - } - - /** - * Drop both L1 and L0 vector virtual tables. - * Metadata tables (l1_records, l0_conversations) are preserved โ€” only - * the vec0 tables need to be rebuilt with the new dimensions. - */ - private dropVectorTables(): void { - this.db.exec("DROP TABLE IF EXISTS l1_vec"); - this.db.exec("DROP TABLE IF EXISTS l0_vec"); - this.logger?.info(`${TAG} Dropped vector tables (l1_vec, l0_vec)`); - } - - /** - * Write or update a memory record (metadata + vector). - * Uses a manual transaction for atomicity. - * - * If `embedding` is `undefined` or a zero vector (all elements are 0), only - * the metadata row is written โ€” the vec0 table is left untouched. This - * allows callers without an EmbeddingService to still persist metadata + FTS - * without constructing a throwaway zero-vector, and prevents placeholder - * zero vectors (from embedding-service failures) from polluting KNN search - * results with null / NaN distances. - * - * **Fault-tolerant**: catches all errors internally so that a vector store - * failure never propagates to the caller / main OpenClaw flow. - * Returns `true` on success, `false` on failure (logged as warning). - */ - upsert(record: MemoryRecord, embedding: Float32Array | undefined): boolean { - if (this.degraded) { - this.logger?.warn(`${TAG} [L1-upsert] SKIPPED (degraded mode) id=${record.id}`); - return false; - } - try { - const { id: recordId, timestamps } = record; - const tsStr = timestamps[0] ?? ""; - const tsStart = - timestamps.length > 0 - ? timestamps.reduce((a, b) => (a < b ? a : b)) - : tsStr; - const tsEnd = - timestamps.length > 0 - ? timestamps.reduce((a, b) => (a > b ? a : b)) - : tsStr; - - const skipVec = !embedding || embedding.every(v => v === 0) || !this.vecTablesReady; - - this.logger?.debug?.( - `${TAG} [L1-upsert] START id=${recordId}, type=${record.type}, ` + - `content="${record.content.slice(0, 60)}..."` + - (embedding - ? `, embeddingDims=${embedding.length}, ` + - `embeddingNorm=${Math.sqrt(Array.from(embedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}` + - `${skipVec ? " (ZERO VECTOR or vec tables not ready โ€” vec write will be skipped)" : ""}` - : " (no embedding โ€” metadata-only write)"), - ); - - this.db.exec("BEGIN"); - try { - // Upsert metadata (INSERT OR UPDATE) - this.stmtUpsertMeta.run( - recordId, - record.content, - record.type, - record.priority, - record.scene_name, - record.sessionKey, - record.sessionId, - tsStr, - tsStart, - tsEnd, - record.createdAt, - record.updatedAt, - JSON.stringify(record.metadata), - ); - - if (!skipVec) { - // vec0 does not support ON CONFLICT โ†’ delete then insert - this.stmtDeleteVec!.run(recordId); - this.stmtInsertVec!.run(recordId, Buffer.from(embedding!.buffer), record.updatedAt); - } else { - this.logger?.debug?.( - `${TAG} [L1-upsert] Skipping vec write (${embedding ? "zero vector" : "no embedding"}) id=${recordId}`, - ); - } - - // Sync FTS5 (delete + re-insert to handle updates) - if (this.ftsAvailable) { - try { - this.stmtL1FtsDelete.run(recordId); - this.stmtL1FtsInsert.run( - tokenizeForFts(record.content), // content โ€” segmented for indexing - record.content, // content_original โ€” raw for display - recordId, - record.type, - record.priority, - record.scene_name, - record.sessionKey, - record.sessionId, - tsStr, - tsStart, - tsEnd, - JSON.stringify(record.metadata), - ); - } catch (ftsErr) { - // FTS write failure is non-fatal โ€” log and continue - this.logger?.warn( - `${TAG} [L1-upsert] FTS write failed (non-fatal) id=${recordId}: ${ftsErr instanceof Error ? ftsErr.message : String(ftsErr)}`, - ); - } - } - - this.db.exec("COMMIT"); - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - this.logger?.debug?.(`${TAG} [L1-upsert] OK id=${recordId}${skipVec ? " (meta-only)" : ""}`); - return true; - } catch (err) { - this.logger?.warn( - `${TAG} [L1-upsert] FAILED (non-fatal) id=${record.id}: ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * Vector similarity search (cosine distance). - * Returns top-k results sorted by similarity (highest first). - * - * **Fault-tolerant**: returns an empty array on any error (e.g. dimension - * mismatch, corrupted DB) so callers can fall back to keyword search. - */ - search(queryEmbedding: Float32Array, topK = 5): VectorSearchResult[] { - if (this.degraded || !this.vecTablesReady) { - if (this.degraded) this.logger?.warn(`${TAG} [L1-search] SKIPPED (degraded mode)`); - return []; - } - try { - // Over-retrieve to compensate for legacy zero-vector placeholders that - // may still exist in the vec0 table. New zero vectors are no longer - // inserted (upsert() skips vec write for zero vectors since v3.x), but - // older data may still contain them โ€” they surface as NULL/NaN distance - // in KNN results. A small buffer of 10 is sufficient for remnants. - // NOTE: "AND distance IS NOT NULL" is NOT usable because vec0 does not - // support that constraint โ€” it causes an empty result set. - const ZERO_VEC_BUFFER = 10; - const retrieveCount = topK + ZERO_VEC_BUFFER; - - this.logger?.debug?.( - `${TAG} [L1-search] START topK=${topK}, retrieveCount=${retrieveCount}, ` + - `queryEmbeddingDims=${queryEmbedding.length}, ` + - `queryNorm=${Math.sqrt(Array.from(queryEmbedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}`, - ); - - const rows = this.stmtSearchVec!.all( - Buffer.from(queryEmbedding.buffer), - retrieveCount, - ) as Array<{ record_id: string; distance: number }>; - - this.logger?.debug?.(`${TAG} [L1-search] vec0 returned ${rows.length} candidate(s)`); - - if (rows.length === 0) return []; - - const results: VectorSearchResult[] = []; - - for (const { record_id, distance } of rows) { - // sqlite-vec returns null distance for zero vectors (cosine undefined when โ€–vโ€–=0). - // Skip these โ€” they are placeholder vectors from embedding-service-unavailable fallback. - if (distance == null || Number.isNaN(distance)) { - this.logger?.warn( - `${TAG} [L1-search] record_id=${record_id} has null/NaN distance (likely zero vector) โ€” skipping`, - ); - continue; - } - - const meta = this.stmtGetMeta.get(record_id) as - | { - content: string; - type: string; - priority: number; - scene_name: string; - session_key: string; - session_id: string; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - metadata_json: string; - } - | undefined; - - if (!meta) { - this.logger?.warn(`${TAG} [L1-search] record_id=${record_id} has vector but NO metadata (orphan)`); - continue; - } - - const score = 1.0 - distance; - this.logger?.debug?.( - `${TAG} [L1-search] HIT id=${record_id}, distance=${distance.toFixed(4)}, score=${score.toFixed(4)}, ` + - `type=${meta.type}, content="${meta.content.slice(0, 60)}..."`, - ); - - results.push({ - record_id, - content: meta.content, - type: meta.type, - priority: meta.priority, - scene_name: meta.scene_name, - score, - timestamp_str: meta.timestamp_str, - timestamp_start: meta.timestamp_start, - timestamp_end: meta.timestamp_end, - session_key: meta.session_key, - session_id: meta.session_id, - metadata_json: meta.metadata_json, - }); - } - - // Trim back to the caller's requested topK (we over-fetched above). - const trimmed = results.slice(0, topK); - this.logger?.info( - `${TAG} [L1-search] DONE returning ${trimmed.length} result(s) (from ${results.length} valid, ${rows.length} raw)`, - ); - return trimmed; - } catch (err) { - this.logger?.warn( - `${TAG} [L1-search] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * Delete a single record (metadata + vector). - * - * **Fault-tolerant**: logs a warning on failure, never throws. - */ - delete(recordId: string): boolean { - if (this.degraded) return false; - try { - this.db.exec("BEGIN"); - try { - this.stmtDeleteMeta.run(recordId); - if (this.vecTablesReady) this.stmtDeleteVec!.run(recordId); - if (this.ftsAvailable) { - try { this.stmtL1FtsDelete.run(recordId); } catch { /* non-fatal */ } - } - this.db.exec("COMMIT"); - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - return true; - } catch (err) { - this.logger?.warn( - `${TAG} delete failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * Delete multiple records (metadata + vector). - * - * **Fault-tolerant**: logs a warning on failure, never throws. - */ - deleteBatch(recordIds: string[]): boolean { - if (this.degraded) return false; - if (recordIds.length === 0) return true; - - try { - this.db.exec("BEGIN"); - try { - for (const id of recordIds) { - this.stmtDeleteMeta.run(id); - if (this.vecTablesReady) this.stmtDeleteVec!.run(id); - if (this.ftsAvailable) { - try { this.stmtL1FtsDelete.run(id); } catch { /* non-fatal */ } - } - } - this.db.exec("COMMIT"); - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - return true; - } catch (err) { - this.logger?.warn( - `${TAG} deleteBatch failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * Get the total number of L1 records in the store. - * - * **Fault-tolerant**: returns 0 on failure. - * TTL cleanup by updated_time. - * - * Deletes expired rows from l1_records and matching vectors from l1_vec - * in a single transaction to guarantee consistency. - */ - deleteL1ExpiredByUpdatedTime(cutoffIso: string): number { - if (this.degraded) { - this.logger?.warn(`${TAG} [deleteExpired] SKIPPED (degraded mode)`); - return 0; - } - try { - const row = this.db.prepare( - "SELECT COUNT(*) AS cnt FROM l1_records WHERE updated_time != '' AND updated_time < ?", - ).get(cutoffIso) as { cnt: number } | undefined; - const expiredCount = row?.cnt ?? 0; - if (expiredCount <= 0) return 0; - - this.db.exec("BEGIN"); - try { - if (this.vecTablesReady) { - this.db.prepare( - "DELETE FROM l1_vec WHERE updated_time != '' AND updated_time < ?", - ).run(cutoffIso); - } - this.db.prepare( - "DELETE FROM l1_records WHERE updated_time != '' AND updated_time < ?", - ).run(cutoffIso); - this.db.exec("COMMIT"); - return expiredCount; - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - } catch (err) { - this.logger?.warn( - `${TAG} deleteL1ExpiredByUpdatedTime failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return 0; - } - } - - /** - * Get the total number of records in the store. - */ - count(): number { - if (this.degraded) return 0; - try { - const row = this.db - .prepare("SELECT COUNT(*) AS cnt FROM l1_records") - .get() as { cnt: number }; - this.logger?.debug?.(`${TAG} [L1-count] total=${row.cnt}`); - return row.cnt; - } catch (err) { - this.logger?.warn( - `${TAG} count failed (non-fatal, returning 0): ${err instanceof Error ? err.message : String(err)}`, - ); - return 0; - } - } - - /** - * Query L1 records with optional session and time filters. - * - * Uses the composite index `idx_l1_session_updated(session_id, updated_time)` - * for efficient filtering. All timestamps are compared as UTC ISO 8601 strings. - * - * **Fault-tolerant**: returns an empty array on any error (degraded mode, DB issues). - */ - queryL1Records(filter?: L1QueryFilter): L1RecordRow[] { - if (this.degraded) { - this.logger?.warn(`${TAG} [L1-query] SKIPPED (degraded mode)`); - return []; - } - try { - const { sessionKey, sessionId, updatedAfter } = filter ?? {}; - - let raw: Record[]; - - // Priority: sessionId > sessionKey (sessionId is more specific) - if (sessionId && updatedAfter) { - raw = this.stmtQueryBySessionIdSince.all(sessionId, updatedAfter) as Record[]; - } else if (sessionId) { - raw = this.stmtQueryBySessionId.all(sessionId) as Record[]; - } else if (sessionKey && updatedAfter) { - raw = this.stmtQueryBySessionKeySince.all(sessionKey, updatedAfter) as Record[]; - } else if (sessionKey) { - raw = this.stmtQueryBySessionKey.all(sessionKey) as Record[]; - } else if (updatedAfter) { - raw = this.stmtQueryAllSince.all(updatedAfter) as Record[]; - } else { - raw = this.stmtQueryAll.all() as Record[]; - } - - // Runtime sanity check: verify first row has expected columns (guards against schema drift) - if (raw.length > 0 && !("record_id" in raw[0] && "content" in raw[0])) { - this.logger?.warn( - `${TAG} [L1-query] Schema mismatch: first row missing expected columns. ` + - `Got keys: [${Object.keys(raw[0]).join(", ")}]`, - ); - return []; - } - - const rows = raw as unknown as L1RecordRow[]; - - this.logger?.info( - `${TAG} [L1-query] filter={sessionKey=${sessionKey ?? "(all)"}, sessionId=${sessionId ?? "(all)"}, updatedAfter=${updatedAfter ?? "(none)"}}, ` + - `returned ${rows.length} record(s)`, - ); - return rows; - } catch (err) { - this.logger?.warn( - `${TAG} [L1-query] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}` - ); - return []; - } - } - - // โ”€โ”€ L0 operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Write or update an L0 single-message record (metadata + vector). - * Uses a manual transaction for atomicity. - * - * If `embedding` is `undefined` or a zero vector (all elements are 0), only - * the metadata row (`l0_conversations`) is written โ€” the vec0 table - * (`l0_vec`) is left untouched. This allows callers without an - * EmbeddingService to still persist metadata + FTS without constructing a - * throwaway zero-vector, and prevents placeholder zero vectors (from - * embedding-service failures) from polluting KNN search results. - * - * **Fault-tolerant**: catches all errors internally, never throws. - * Returns `true` on success, `false` on failure (logged as warning). - */ - upsertL0(record: L0VectorRecord, embedding: Float32Array | undefined): boolean { - if (this.degraded) { - this.logger?.warn(`${TAG} [L0-upsert] SKIPPED (degraded mode) id=${record.id}`); - return false; - } - try { - const skipVec = !embedding || embedding.every(v => v === 0) || !this.vecTablesReady; - - this.logger?.debug?.( - `${TAG} [L0-upsert] START id=${record.id}, session=${record.sessionKey}, role=${record.role}, ` + - `text="${record.messageText.slice(0, 60)}..."` + - (embedding - ? `, embeddingDims=${embedding.length}, ` + - `embeddingNorm=${Math.sqrt(Array.from(embedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}` + - `${skipVec ? " (ZERO VECTOR or vec tables not ready โ€” vec write will be skipped)" : ""}` - : " (no embedding โ€” metadata-only write)"), - ); - - this.db.exec("BEGIN"); - try { - this.stmtL0UpsertMeta.run( - record.id, - record.sessionKey, - record.sessionId, - record.role, - record.messageText, - record.recordedAt, - record.timestamp, - ); - - if (!skipVec) { - // vec0 does not support ON CONFLICT โ†’ delete then insert - this.stmtL0DeleteVec!.run(record.id); - this.stmtL0InsertVec!.run(record.id, Buffer.from(embedding!.buffer), record.recordedAt); - } else { - this.logger?.debug?.( - `${TAG} [L0-upsert] Skipping vec write (${embedding ? "zero vector" : "no embedding"}) id=${record.id}`, - ); - } - - // Sync FTS5 (delete + re-insert to handle updates) - if (this.ftsAvailable) { - try { - this.stmtL0FtsDelete.run(record.id); - this.stmtL0FtsInsert.run( - tokenizeForFts(record.messageText), // message_text โ€” segmented for indexing - record.messageText, // message_text_original โ€” raw for display - record.id, - record.sessionKey, - record.sessionId, - record.role, - record.recordedAt, - record.timestamp, - ); - } catch (ftsErr) { - // FTS write failure is non-fatal โ€” log and continue - this.logger?.warn( - `${TAG} [L0-upsert] FTS write failed (non-fatal) id=${record.id}: ${ftsErr instanceof Error ? ftsErr.message : String(ftsErr)}`, - ); - } - } - - this.db.exec("COMMIT"); - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - this.logger?.debug?.(`${TAG} [L0-upsert] OK id=${record.id}${skipVec ? " (meta-only)" : ""}`); - return true; - } catch (err) { - this.logger?.warn( - `${TAG} [L0-upsert] FAILED (non-fatal) id=${record.id}: ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * Update ONLY the vector embedding for an existing L0 record. - * The metadata row must already exist in l0_conversations (written by upsertL0). - * - * This is used by the background embedding task in auto-capture: - * 1. upsertL0() writes metadata + FTS synchronously (no embedding) - * 2. Background task calls embedBatch() then updateL0Embedding() for each record - * - * **Fault-tolerant**: catches all errors internally, never throws. - * Returns `true` on success, `false` on failure. - */ - updateL0Embedding(recordId: string, embedding: Float32Array): boolean { - if (this.degraded || !this.vecTablesReady) { - return false; - } - if (!embedding || embedding.every(v => v === 0)) { - this.logger?.debug?.(`${TAG} [L0-update-embedding] Skipping zero vector for ${recordId}`); - return false; - } - try { - // Look up recorded_at from metadata for the vec0 row - const meta = this.stmtL0GetMeta.get(recordId) as { recorded_at: string } | undefined; - if (!meta) { - this.logger?.warn(`${TAG} [L0-update-embedding] No metadata found for ${recordId}, skipping`); - return false; - } - - this.db.exec("BEGIN"); - try { - this.stmtL0DeleteVec!.run(recordId); - this.stmtL0InsertVec!.run(recordId, Buffer.from(embedding.buffer), meta.recorded_at); - this.db.exec("COMMIT"); - } catch (err) { - try { this.db.exec("ROLLBACK"); } catch { /* ignore */ } - throw err; - } - return true; - } catch (err) { - this.logger?.warn( - `${TAG} [L0-update-embedding] FAILED (non-fatal) id=${recordId}: ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * Vector similarity search on L0 individual messages (cosine distance). - * Returns top-k results sorted by similarity (highest first). - * - * **Fault-tolerant**: returns an empty array on any error. - */ - searchL0(queryEmbedding: Float32Array, topK = 5): L0VectorSearchResult[] { - if (this.degraded || !this.vecTablesReady) { - if (this.degraded) this.logger?.warn(`${TAG} [L0-search] SKIPPED (degraded mode)`); - return []; - } - try { - // Over-retrieve to compensate for legacy zero-vector placeholders that - // may still exist in the vec0 table. New zero vectors are no longer - // inserted (upsertL0() skips vec write for zero vectors since v3.x), but - // older data may still contain them โ€” they surface as NULL/NaN distance - // in KNN results. - // NOTE: "AND distance IS NOT NULL" is NOT usable because vec0 does not - // support that constraint โ€” it causes an empty result set. - const retrieveCount = topK + VectorStore.ZERO_VEC_BUFFER; - - this.logger?.debug?.( - `${TAG} [L0-search] START topK=${topK}, retrieveCount=${retrieveCount}, ` + - `queryEmbeddingDims=${queryEmbedding.length}, ` + - `queryNorm=${Math.sqrt(Array.from(queryEmbedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}`, - ); - - const rows = this.stmtL0SearchVec!.all( - Buffer.from(queryEmbedding.buffer), - retrieveCount, - ) as Array<{ record_id: string; distance: number }>; - - this.logger?.debug?.(`${TAG} [L0-search] vec0 returned ${rows.length} candidate(s)`); - - if (rows.length === 0) return []; - - const results: L0VectorSearchResult[] = []; - - for (const { record_id, distance } of rows) { - // sqlite-vec returns null distance for zero vectors (cosine undefined when โ€–vโ€–=0). - // Skip these โ€” they are placeholder vectors from embedding-service-unavailable fallback. - if (distance == null || Number.isNaN(distance)) { - this.logger?.warn( - `${TAG} [L0-search] record_id=${record_id} has null/NaN distance (likely zero vector) โ€” skipping`, - ); - continue; - } - - const meta = this.stmtL0GetMeta.get(record_id) as - | { - session_key: string; - session_id: string; - role: string; - message_text: string; - recorded_at: string; - timestamp: number; - } - | undefined; - - if (!meta) { - this.logger?.warn(`${TAG} [L0-search] record_id=${record_id} has vector but NO metadata (orphan)`); - continue; - } - - const score = 1.0 - distance; - this.logger?.debug?.( - `${TAG} [L0-search] HIT id=${record_id}, distance=${distance.toFixed(4)}, score=${score.toFixed(4)}, ` + - `role=${meta.role}, session=${meta.session_key}, text="${meta.message_text.slice(0, 60)}..."`, - ); - - results.push({ - record_id, - session_key: meta.session_key, - session_id: meta.session_id, - role: meta.role, - message_text: meta.message_text, - score, - recorded_at: meta.recorded_at, - timestamp: meta.timestamp ?? 0, - }); - } - - // Trim back to the caller's requested topK (we over-fetched above). - const trimmed = results.slice(0, topK); - this.logger?.info( - `${TAG} [L0-search] DONE returning ${trimmed.length} result(s) (from ${results.length} valid, ${rows.length} raw)`, - ); - return trimmed; - } catch (err) { - this.logger?.warn( - `${TAG} [L0-search] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * Delete a single L0 record (metadata + vector). - * - * **Fault-tolerant**: logs a warning on failure, never throws. - */ - deleteL0(recordId: string): boolean { - if (this.degraded) return false; - try { - this.db.exec("BEGIN"); - try { - this.stmtL0DeleteMeta.run(recordId); - if (this.vecTablesReady) this.stmtL0DeleteVec!.run(recordId); - if (this.ftsAvailable) { - try { this.stmtL0FtsDelete.run(recordId); } catch { /* non-fatal */ } - } - this.db.exec("COMMIT"); - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - return true; - } catch (err) { - this.logger?.warn( - `${TAG} deleteL0 failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * TTL cleanup by recorded_at (ISO string) for L0 records. - * - * Deletes expired rows from l0_conversations and matching vectors from l0_vec - * in a single transaction to guarantee consistency. - */ - deleteL0ExpiredByRecordedAt(cutoffIso: string): number { - if (this.degraded) { - this.logger?.warn(`${TAG} [deleteExpiredL0] SKIPPED (degraded mode)`); - return 0; - } - - try { - const row = this.db.prepare( - "SELECT COUNT(*) AS cnt FROM l0_conversations WHERE recorded_at != '' AND recorded_at < ?", - ).get(cutoffIso) as { cnt: number } | undefined; - const expiredCount = row?.cnt ?? 0; - if (expiredCount <= 0) return 0; - - this.db.exec("BEGIN"); - try { - if (this.vecTablesReady) { - this.db.prepare( - "DELETE FROM l0_vec WHERE recorded_at != '' AND recorded_at < ?", - ).run(cutoffIso); - } - this.db.prepare( - "DELETE FROM l0_conversations WHERE recorded_at != '' AND recorded_at < ?", - ).run(cutoffIso); - this.db.exec("COMMIT"); - return expiredCount; - } catch (err) { - try { - this.db.exec("ROLLBACK"); - } catch { /* ignore rollback errors */ } - throw err; - } - } catch (err) { - this.logger?.warn( - `${TAG} deleteL0ExpiredByRecordedAt failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return 0; - } - } - - /** - * Get the total number of L0 message records in the store. - * - * **Fault-tolerant**: returns 0 on failure. - */ - countL0(): number { - if (this.degraded) return 0; - try { - const row = this.db - .prepare("SELECT COUNT(*) AS cnt FROM l0_conversations") - .get() as { cnt: number }; - this.logger?.debug?.(`${TAG} [L0-count] total=${row.cnt}`); - return row.cnt; - } catch (err) { - this.logger?.warn( - `${TAG} countL0 failed (non-fatal, returning 0): ${err instanceof Error ? err.message : String(err)}`, - ); - return 0; - } - } - - // โ”€โ”€ Re-index operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Get all L1 record texts for re-embedding. - * Returns record_id โ†’ content pairs. - */ - getAllL1Texts(): Array<{ record_id: string; content: string; updated_time: string }> { - if (this.degraded) return []; - try { - return this.db - .prepare("SELECT record_id, content, updated_time FROM l1_records") - .all() as Array<{ record_id: string; content: string; updated_time: string }>; - } catch (err) { - this.logger?.warn( - `${TAG} getAllL1Texts failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * Get all L0 message texts for re-embedding. - * Returns record_id โ†’ message_text/recorded_at tuples. - */ - getAllL0Texts(): Array<{ record_id: string; message_text: string; recorded_at: string }> { - if (this.degraded) return []; - try { - return this.db - .prepare("SELECT record_id, message_text, recorded_at FROM l0_conversations") - .all() as Array<{ record_id: string; message_text: string; recorded_at: string }>; - } catch (err) { - this.logger?.warn( - `${TAG} getAllL0Texts failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * Re-embed all existing L1 and L0 texts with a new embedding function. - * - * This is called after `init()` returns `needsReindex: true` โ€” the vector - * tables have already been dropped and re-created with the correct dimensions. - * This method reads every text from the metadata tables and writes fresh - * embeddings into the new vector tables. - * - * @param embedFn A function that converts text โ†’ Float32Array embedding. - * @param onProgress Optional callback for progress reporting. - */ - async reindexAll( - embedFn: (text: string) => Promise, - onProgress?: (done: number, total: number, layer: "L1" | "L0") => void, - ): Promise<{ l1Count: number; l0Count: number }> { - if (this.degraded || !this.vecTablesReady) { - if (this.degraded) this.logger?.warn(`${TAG} reindexAll skipped: VectorStore is in degraded mode`); - return { l1Count: 0, l0Count: 0 }; - } - - try { - // โ”€โ”€ Re-embed L1 โ”€โ”€ - const l1Rows = this.getAllL1Texts(); - let l1Done = 0; - for (const { record_id, content, updated_time } of l1Rows) { - try { - const embedding = await embedFn(content); - // Wrap delete+insert in a transaction to prevent orphan vectors - this.db.exec("BEGIN"); - try { - this.stmtDeleteVec!.run(record_id); - this.stmtInsertVec!.run(record_id, Buffer.from(embedding.buffer), updated_time); - this.db.exec("COMMIT"); - } catch (txErr) { - try { this.db.exec("ROLLBACK"); } catch { /* ignore */ } - throw txErr; - } - } catch (err) { - this.logger?.warn?.( - `${TAG} reindex L1 skip ${record_id}: ${err instanceof Error ? err.message : String(err)}`, - ); - } - l1Done++; - onProgress?.(l1Done, l1Rows.length, "L1"); - } - - // โ”€โ”€ Re-embed L0 โ”€โ”€ - const l0Rows = this.getAllL0Texts(); - let l0Done = 0; - for (const { record_id, message_text, recorded_at } of l0Rows) { - try { - const embedding = await embedFn(message_text); - // Wrap delete+insert in a transaction to prevent orphan vectors - this.db.exec("BEGIN"); - try { - this.stmtL0DeleteVec!.run(record_id); - this.stmtL0InsertVec!.run(record_id, Buffer.from(embedding.buffer), recorded_at); - this.db.exec("COMMIT"); - } catch (txErr) { - try { this.db.exec("ROLLBACK"); } catch { /* ignore */ } - throw txErr; - } - } catch (err) { - this.logger?.warn?.( - `${TAG} reindex L0 skip ${record_id}: ${err instanceof Error ? err.message : String(err)}`, - ); - } - l0Done++; - onProgress?.(l0Done, l0Rows.length, "L0"); - } - - this.logger?.info( - `${TAG} Reindex complete: L1=${l1Done}/${l1Rows.length}, L0=${l0Done}/${l0Rows.length}`, - ); - - return { l1Count: l1Done, l0Count: l0Done }; - } catch (err) { - this.logger?.error( - `${TAG} reindexAll failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return { l1Count: 0, l0Count: 0 }; - } - } - - // โ”€โ”€ L0 query operations (for L1 runner) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Query L0 messages for a given session key, optionally filtered by timestamp cursor. - * Returns messages ordered by timestamp ASC (chronological). - * - * Used by L1 runner to read L0 data from DB instead of JSONL files. - */ - queryL0ForL1( - sessionKey: string, - afterTimestamp?: number, - limit = 50, - ): Array<{ - record_id: string; - session_key: string; - session_id: string; - role: string; - message_text: string; - recorded_at: string; - timestamp: number; - }> { - if (this.degraded) { - this.logger?.warn(`${TAG} [L0-query] SKIPPED (degraded mode)`); - return []; - } - try { - // Query newest-first (DESC) with LIMIT, then reverse to chronological order - let rows: Array>; - if (afterTimestamp && afterTimestamp > 0) { - rows = this.stmtL0QueryAfter.all(sessionKey, afterTimestamp, limit) as Array>; - } else { - rows = this.stmtL0QueryAll.all(sessionKey, limit) as Array>; - } - - this.logger?.info( - `${TAG} [L0-query] session=${sessionKey}, afterTs=${afterTimestamp ?? "(all)"}, ` + - `limit=${limit}, returned ${rows.length} row(s)`, - ); - - // Reverse: SQL returns newest-first (DESC), callers expect chronological order - return rows.map((r) => ({ - record_id: r.record_id as string, - session_key: r.session_key as string, - session_id: (r.session_id as string) || "", - role: r.role as string, - message_text: r.message_text as string, - recorded_at: (r.recorded_at as string) || "", - timestamp: (r.timestamp as number) || 0, - })).reverse(); - } catch (err) { - this.logger?.warn( - `${TAG} [L0-query] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * Query L0 messages for a given session key, grouped by session_id. - * Each group's messages are in chronological order (timestamp ASC). - * Groups are sorted by earliest message timestamp. - * - * Used by L1 runner to replace readConversationMessagesGroupedBySessionId(). - */ - queryL0GroupedBySessionId( - sessionKey: string, - afterTimestamp?: number, - limit = 50, - ): Array<{ sessionId: string; messages: Array<{ id: string; role: string; content: string; timestamp: number }> }> { - if (this.degraded) { - this.logger?.warn(`${TAG} [L0-query-grouped] SKIPPED (degraded mode)`); - return []; - } - try { - const rows = this.queryL0ForL1(sessionKey, afterTimestamp, limit); - - // Group by session_id - const groupMap = new Map>(); - for (const row of rows) { - const sid = row.session_id || ""; - let group = groupMap.get(sid); - if (!group) { - group = []; - groupMap.set(sid, group); - } - group.push({ - id: row.record_id, - role: row.role, - content: row.message_text, - timestamp: row.timestamp, - }); - } - - // Convert to array, sorted by earliest message timestamp - const groups: Array<{ sessionId: string; messages: Array<{ id: string; role: string; content: string; timestamp: number }> }> = []; - for (const [sessionId, messages] of groupMap) { - if (messages.length > 0) { - groups.push({ sessionId, messages }); - } - } - groups.sort((a, b) => a.messages[0].timestamp - b.messages[0].timestamp); - - this.logger?.info( - `${TAG} [L0-query-grouped] session=${sessionKey}, afterTs=${afterTimestamp ?? "(all)"}, ` + - `${rows.length} messages across ${groups.length} group(s)`, - ); - - return groups; - } catch (err) { - this.logger?.warn( - `${TAG} [L0-query-grouped] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - // โ”€โ”€ FTS5 search operations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Whether FTS5 full-text search is available. - * When `false`, callers should skip keyword-based recall entirely. - */ - isFtsAvailable(): boolean { - return this.ftsAvailable; - } - - /** - * FTS5 keyword search on L1 records. - * Returns top-`limit` results sorted by BM25 relevance (highest first). - * - * @param ftsQuery A pre-built FTS5 MATCH expression (from `buildFtsQuery()`). - * @param limit Maximum number of results to return. - * - * **Fault-tolerant**: returns an empty array on any error. - */ - ftsSearchL1(ftsQuery: string, limit = 20): FtsSearchResult[] { - if (this.degraded || !this.ftsAvailable) return []; - try { - const rows = this.stmtL1FtsSearch.all(ftsQuery, limit) as Array<{ - record_id: string; - content: string; - type: string; - priority: number; - scene_name: string; - session_key: string; - session_id: string; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - metadata_json: string; - rank: number; - }>; - - return rows.map((r) => ({ - record_id: r.record_id, - content: r.content, - type: r.type, - priority: r.priority, - scene_name: r.scene_name, - score: bm25RankToScore(r.rank), - timestamp_str: r.timestamp_str, - timestamp_start: r.timestamp_start, - timestamp_end: r.timestamp_end, - session_key: r.session_key, - session_id: r.session_id, - metadata_json: r.metadata_json, - })); - } catch (err) { - this.logger?.warn( - `${TAG} [L1-fts-search] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - /** - * FTS5 keyword search on L0 conversation messages. - * Returns top-`limit` results sorted by BM25 relevance (highest first). - * - * @param ftsQuery A pre-built FTS5 MATCH expression (from `buildFtsQuery()`). - * @param limit Maximum number of results to return. - * - * **Fault-tolerant**: returns an empty array on any error. - */ - ftsSearchL0(ftsQuery: string, limit = VectorStore.FTS_DEFAULT_LIMIT): L0FtsSearchResult[] { - if (this.degraded || !this.ftsAvailable) return []; - try { - const rows = this.stmtL0FtsSearch.all(ftsQuery, limit) as Array<{ - record_id: string; - message_text: string; - session_key: string; - session_id: string; - role: string; - recorded_at: string; - timestamp: number; - rank: number; - }>; - - return rows.map((r) => ({ - record_id: r.record_id, - session_key: r.session_key, - session_id: r.session_id, - role: r.role, - message_text: r.message_text, - score: bm25RankToScore(r.rank), - recorded_at: r.recorded_at, - timestamp: r.timestamp ?? 0, - })); - } catch (err) { - this.logger?.warn( - `${TAG} [L0-fts-search] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - } - - // โ”€โ”€ FTS5 migration & rebuild โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - - /** - * Detect old FTS5 v1 schema (no `content_original` column) and drop the - * tables so they can be recreated with the v2 schema. - * - * FTS5 virtual tables do NOT support `ALTER TABLE ADD COLUMN`, so the only - * migration path is DROP + recreate + repopulate. - * - * @returns `true` if migration was performed (= FTS index needs rebuilding). - * @internal - */ - private migrateFtsTablesIfNeeded(): boolean { - try { - // Check if l1_fts exists at all - const l1Exists = this.db - .prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name='l1_fts'") - .get(); - if (!l1Exists) { - // Fresh install โ€” tables will be created with v2 schema. - // Still need rebuild if there's existing data in l1_records. - const hasData = this.db.prepare("SELECT 1 FROM l1_records LIMIT 1").get(); - return !!hasData; - } - - // Check if the v2 column `content_original` exists. - // FTS5 tables appear in pragma_table_info with their column names. - const cols = this.db - .prepare("SELECT name FROM pragma_table_info('l1_fts')") - .all() as Array<{ name: string }>; - const hasV2Col = cols.some((c) => c.name === "content_original"); - - if (hasV2Col) { - return false; // Already v2 โ€” no migration needed - } - - // v1 โ†’ v2: drop both FTS tables (data will be repopulated by rebuildFtsIndex) - this.logger?.info(`${TAG} Migrating FTS5 tables from v1 to v2 (jieba segmented)`); - this.db.exec("DROP TABLE IF EXISTS l1_fts"); - this.db.exec("DROP TABLE IF EXISTS l0_fts"); - return true; - } catch (err) { - this.logger?.warn( - `${TAG} FTS migration check failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return false; - } - } - - /** - * Rebuild the FTS5 index from scratch by reading all records from the - * metadata tables and re-inserting them with jieba-segmented text. - * - * Called automatically after: - * - Schema migration from v1 to v2 - * - Fresh table creation when existing data exists - * - * Safe to call multiple times (idempotent โ€” clears FTS tables first). - */ - rebuildFtsIndex(): void { - if (!this.ftsAvailable) return; - - try { - this.logger?.info(`${TAG} Rebuilding FTS5 index with jieba segmentationโ€ฆ`); - - // โ”€โ”€ Rebuild L1 FTS โ”€โ”€ - // Clear existing FTS data - this.db.exec("DELETE FROM l1_fts"); - - // Read all L1 records from metadata table - const l1Rows = this.db - .prepare(` - SELECT record_id, content, type, priority, scene_name, - session_key, session_id, timestamp_str, timestamp_start, timestamp_end, metadata_json - FROM l1_records - `) - .all() as Array<{ - record_id: string; - content: string; - type: string; - priority: number; - scene_name: string; - session_key: string; - session_id: string; - timestamp_str: string; - timestamp_start: string; - timestamp_end: string; - metadata_json: string; - }>; - - let l1Count = 0; - for (const r of l1Rows) { - try { - this.stmtL1FtsInsert.run( - tokenizeForFts(r.content), // content โ€” segmented - r.content, // content_original โ€” raw - r.record_id, - r.type, - r.priority, - r.scene_name, - r.session_key, - r.session_id, - r.timestamp_str, - r.timestamp_start, - r.timestamp_end, - r.metadata_json, - ); - l1Count++; - } catch (err) { - this.logger?.warn?.( - `${TAG} FTS rebuild skip L1 ${r.record_id}: ${err instanceof Error ? err.message : String(err)}`, - ); - } - } - - // โ”€โ”€ Rebuild L0 FTS โ”€โ”€ - this.db.exec("DELETE FROM l0_fts"); - - const l0Rows = this.db - .prepare(` - SELECT record_id, message_text, session_key, session_id, role, recorded_at, timestamp - FROM l0_conversations - `) - .all() as Array<{ - record_id: string; - message_text: string; - session_key: string; - session_id: string; - role: string; - recorded_at: string; - timestamp: number; - }>; - - let l0Count = 0; - for (const r of l0Rows) { - try { - this.stmtL0FtsInsert.run( - tokenizeForFts(r.message_text), // message_text โ€” segmented - r.message_text, // message_text_original โ€” raw - r.record_id, - r.session_key, - r.session_id, - r.role, - r.recorded_at, - r.timestamp, - ); - l0Count++; - } catch (err) { - this.logger?.warn?.( - `${TAG} FTS rebuild skip L0 ${r.record_id}: ${err instanceof Error ? err.message : String(err)}`, - ); - } - } - - this.logger?.info( - `${TAG} FTS5 rebuild complete: L1=${l1Count}/${l1Rows.length}, L0=${l0Count}/${l0Rows.length}`, - ); - } catch (err) { - this.logger?.warn( - `${TAG} FTS5 rebuild failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - } - } - - /** - * Close the database connection. - * Should be called on shutdown. Idempotent โ€” safe to call multiple times. - */ - close(): void { - if (this.closed) return; - this.closed = true; - try { - this.db.close(); - } catch (err) { - this.logger?.warn?.( - `${TAG} Error closing database: ${err instanceof Error ? err.message : String(err)}`, - ); - } - } -} diff --git a/src/tools/conversation-search.ts b/src/tools/conversation-search.ts deleted file mode 100644 index ac4f3b1..0000000 --- a/src/tools/conversation-search.ts +++ /dev/null @@ -1,279 +0,0 @@ -/** - * conversation_search tool: Agent-callable tool for searching L0 conversation records. - * - * Supports three search strategies with automatic degradation: - * 1. **hybrid** (default) โ€” FTS5 keyword + vector embedding in parallel, - * merged via Reciprocal Rank Fusion (RRF). - * 2. **embedding** โ€” pure vector similarity (when FTS5 is unavailable). - * 3. **fts** โ€” pure FTS5 keyword search (when embedding is unavailable). - * - * The tool is registered via `api.registerTool()` in index.ts. - */ - -import type { IMemoryStore, L0SearchResult } from "../store/types.js"; -import { buildFtsQuery } from "../store/sqlite.js"; -import type { EmbeddingService } from "../store/embedding.js"; - -// ============================ -// Types -// ============================ - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -export interface ConversationSearchResultItem { - id: string; - session_key: string; - /** Role of the message sender: "user" or "assistant" */ - role: string; - /** Text content of this single message */ - content: string; - score: number; - recorded_at: string; -} - -export interface ConversationSearchResult { - results: ConversationSearchResultItem[]; - total: number; - /** Actual search strategy used: "hybrid", "embedding", "fts", or "none". */ - strategy: string; - /** Optional message, e.g. when embedding is not configured. */ - message?: string; -} - -const TAG = "[memory-tdai][tdai_conversation_search]"; - -// ============================ -// RRF (Reciprocal Rank Fusion) -// ============================ - -/** Standard RRF constant from the original RRF paper. */ -const RRF_K = 60; - -/** - * Merge multiple ranked lists of `ConversationSearchResultItem` via Reciprocal - * Rank Fusion. Items appearing in multiple lists get their RRF scores summed. - * - * Returns items sorted by descending RRF score. The `score` field of each - * returned item is replaced by the RRF score for consistent ranking semantics. - */ -function rrfMergeL0(...lists: ConversationSearchResultItem[][]): ConversationSearchResultItem[] { - const map = new Map(); - - for (const list of lists) { - for (let rank = 0; rank < list.length; rank++) { - const item = list[rank]; - const score = 1 / (RRF_K + rank + 1); - const existing = map.get(item.id); - if (existing) { - existing.rrfScore += score; - } else { - map.set(item.id, { item, rrfScore: score }); - } - } - } - - return [...map.values()] - .sort((a, b) => b.rrfScore - a.rrfScore) - .map(({ item, rrfScore }) => ({ ...item, score: rrfScore })); -} - -// ============================ -// Search implementation -// ============================ - -export async function executeConversationSearch(params: { - query: string; - limit: number; - sessionKey?: string; - vectorStore?: IMemoryStore; - embeddingService?: EmbeddingService; - logger?: Logger; -}): Promise { - const { - query, - limit, - sessionKey: sessionFilter, - vectorStore, - embeddingService, - logger, - } = params; - - logger?.debug?.( - `${TAG} CALLED: query="${query.slice(0, 100)}", limit=${limit}, ` + - `sessionFilter=${sessionFilter ?? "(none)"}, ` + - `vectorStore=${vectorStore ? "available" : "UNAVAILABLE"}, ` + - `embeddingService=${embeddingService ? "available" : "UNAVAILABLE"}`, - ); - - if (!query || query.trim().length === 0) { - logger?.debug?.(`${TAG} Empty query, returning empty`); - return { results: [], total: 0, strategy: "none" }; - } - - if (!vectorStore) { - logger?.warn?.(`${TAG} VectorStore not available`); - return { results: [], total: 0, strategy: "none" }; - } - - // โ”€โ”€ Determine available capabilities โ”€โ”€ - const hasEmbedding = !!embeddingService; - const hasFts = vectorStore.isFtsAvailable(); - - if (!hasEmbedding && !hasFts) { - logger?.warn?.(`${TAG} Neither EmbeddingService nor FTS5 available โ€” cannot search`); - return { - results: [], - total: 0, - strategy: "none", - message: - "Embedding service is not configured and FTS is not available. " + - "Conversation search requires an embedding provider or FTS5 support. " + - "Please configure an embedding provider in the embedding.provider setting (e.g. openai_compatible).", - }; - } - - // โ”€โ”€ Over-retrieve for later filtering and RRF merging โ”€โ”€ - const candidateK = sessionFilter ? limit * 4 : limit * 3; - - // โ”€โ”€ Run available search strategies in parallel โ”€โ”€ - const [ftsItems, vecItems] = await Promise.all([ - // FTS5 keyword search on L0 - (async (): Promise => { - if (!hasFts) return []; - try { - const ftsQuery = buildFtsQuery(query); - if (!ftsQuery) { - logger?.debug?.(`${TAG} [hybrid-fts] No usable FTS tokens from query`); - return []; - } - logger?.debug?.(`${TAG} [hybrid-fts] FTS5 query: "${ftsQuery}"`); - const ftsResults = await vectorStore.searchL0Fts(ftsQuery, candidateK); - logger?.debug?.(`${TAG} [hybrid-fts] FTS5 returned ${ftsResults.length} candidates`); - return ftsResults.map((r) => ({ - id: r.record_id, - session_key: r.session_key, - role: r.role, - content: r.message_text, - score: r.score, - recorded_at: r.recorded_at, - })); - } catch (err) { - logger?.warn?.( - `${TAG} [hybrid-fts] FTS5 search failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - })(), - - // Vector embedding search on L0 - (async (): Promise => { - if (!hasEmbedding) return []; - try { - logger?.debug?.(`${TAG} [hybrid-vec] Generating query embedding...`); - const queryEmbedding = await embeddingService!.embed(query); - logger?.debug?.( - `${TAG} [hybrid-vec] Embedding OK, dims=${queryEmbedding.length}, searching top-${candidateK}...`, - ); - const vecResults: L0SearchResult[] = await vectorStore.searchL0Vector(queryEmbedding, candidateK, query); - logger?.debug?.(`${TAG} [hybrid-vec] Vector search returned ${vecResults.length} candidates`); - return vecResults.map((r) => ({ - id: r.record_id, - session_key: r.session_key, - role: r.role, - content: r.message_text, - score: r.score, - recorded_at: r.recorded_at, - })); - } catch (err) { - logger?.warn?.( - `${TAG} [hybrid-vec] Embedding search failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - })(), - ]); - - // โ”€โ”€ Determine effective strategy โ”€โ”€ - const ftsOk = ftsItems.length > 0; - const vecOk = vecItems.length > 0; - let strategy: string; - - if (ftsOk && vecOk) { - strategy = "hybrid"; - } else if (vecOk) { - strategy = "embedding"; - } else if (ftsOk) { - strategy = "fts"; - } else { - logger?.debug?.(`${TAG} Both search paths returned 0 results`); - return { results: [], total: 0, strategy: hasEmbedding ? "embedding" : "fts" }; - } - - // โ”€โ”€ Merge results โ”€โ”€ - let results: ConversationSearchResultItem[]; - if (strategy === "hybrid") { - results = rrfMergeL0(ftsItems, vecItems); - logger?.debug?.( - `${TAG} [hybrid] RRF merged: fts=${ftsItems.length}, vec=${vecItems.length} โ†’ ${results.length} unique`, - ); - } else { - // Single-source: use whichever list has results (already sorted by score) - results = ftsOk ? ftsItems : vecItems; - } - - // โ”€โ”€ Apply session key filter โ”€โ”€ - if (sessionFilter) { - const preFilterCount = results.length; - results = results.filter((r) => r.session_key === sessionFilter); - logger?.debug?.(`${TAG} After session filter "${sessionFilter}": ${results.length}/${preFilterCount}`); - } - - // โ”€โ”€ Trim to requested limit โ”€โ”€ - const trimmed = results.slice(0, limit); - - logger?.debug?.( - `${TAG} RESULT (strategy=${strategy}): returning ${trimmed.length} messages ` + - `(scores: [${trimmed.map((r) => r.score.toFixed(3)).join(", ")}])`, - ); - - return { - results: trimmed, - total: trimmed.length, - strategy, - }; -} - -// ============================ -// Tool response formatter -// ============================ - -export function formatConversationSearchResponse(result: ConversationSearchResult): string { - if (result.message) { - return result.message; - } - if (result.results.length === 0) { - return "No matching conversation messages found."; - } - - const lines: string[] = [ - `Found ${result.total} matching message(s):`, - "", - ]; - - for (const item of result.results) { - const scoreStr = typeof item.score === "number" ? ` (score: ${item.score.toFixed(3)})` : ""; - const dateStr = item.recorded_at ? ` [${item.recorded_at}]` : ""; - lines.push(`---`); - lines.push(`**[${item.role}]** Session: ${item.session_key}${dateStr}${scoreStr}`); - lines.push(""); - lines.push(item.content); - lines.push(""); - } - - return lines.join("\n"); -} diff --git a/src/tools/memory-search.ts b/src/tools/memory-search.ts deleted file mode 100644 index dc9d2c2..0000000 --- a/src/tools/memory-search.ts +++ /dev/null @@ -1,290 +0,0 @@ -/** - * memory_search tool: Agent-callable tool for searching L1 memory records. - * - * Supports three search strategies with automatic degradation: - * 1. **hybrid** (default) โ€” FTS5 keyword + vector embedding in parallel, - * merged via Reciprocal Rank Fusion (RRF). - * 2. **embedding** โ€” pure vector similarity (when FTS5 is unavailable). - * 3. **fts** โ€” pure FTS5 keyword search (when embedding is unavailable). - * - * The tool is registered via `api.registerTool()` in index.ts. - */ - -import type { IMemoryStore, L1SearchResult } from "../store/types.js"; -import { buildFtsQuery } from "../store/sqlite.js"; -import type { EmbeddingService } from "../store/embedding.js"; - -// ============================ -// Types -// ============================ - -interface Logger { - debug?: (message: string) => void; - info: (message: string) => void; - warn: (message: string) => void; - error: (message: string) => void; -} - -export interface MemorySearchResultItem { - id: string; - content: string; - type: string; - priority: number; - scene_name: string; - score: number; - created_at: string; - updated_at: string; -} - -export interface MemorySearchResult { - results: MemorySearchResultItem[]; - total: number; - strategy: string; - /** Optional message, e.g. when embedding is not configured. */ - message?: string; -} - -const TAG = "[memory-tdai][tdai_memory_search]"; - -// ============================ -// RRF (Reciprocal Rank Fusion) -// ============================ - -/** Standard RRF constant from the original RRF paper. */ -const RRF_K = 60; - -/** - * Merge multiple ranked lists of `MemorySearchResultItem` via Reciprocal Rank - * Fusion. Items appearing in multiple lists get their RRF scores summed. - * - * Returns items sorted by descending RRF score. The `score` field of each - * returned item is replaced by the RRF score for consistent ranking semantics. - */ -function rrfMergeL1(...lists: MemorySearchResultItem[][]): MemorySearchResultItem[] { - const map = new Map(); - - for (const list of lists) { - for (let rank = 0; rank < list.length; rank++) { - const item = list[rank]; - const score = 1 / (RRF_K + rank + 1); - const existing = map.get(item.id); - if (existing) { - existing.rrfScore += score; - } else { - map.set(item.id, { item, rrfScore: score }); - } - } - } - - return [...map.values()] - .sort((a, b) => b.rrfScore - a.rrfScore) - .map(({ item, rrfScore }) => ({ ...item, score: rrfScore })); -} - -// ============================ -// Search implementation -// ============================ - -export async function executeMemorySearch(params: { - query: string; - limit: number; - type?: string; - scene?: string; - vectorStore?: IMemoryStore; - embeddingService?: EmbeddingService; - logger?: Logger; -}): Promise { - const { - query, - limit, - type: typeFilter, - scene: sceneFilter, - vectorStore, - embeddingService, - logger, - } = params; - - logger?.debug?.( - `${TAG} CALLED: query="${query.slice(0, 100)}", limit=${limit}, ` + - `typeFilter=${typeFilter ?? "(none)"}, sceneFilter=${sceneFilter ?? "(none)"}, ` + - `vectorStore=${vectorStore ? "available" : "UNAVAILABLE"}, ` + - `embeddingService=${embeddingService ? "available" : "UNAVAILABLE"}`, - ); - - if (!query || query.trim().length === 0) { - logger?.debug?.(`${TAG} Empty query, returning empty`); - return { results: [], total: 0, strategy: "none" }; - } - - if (!vectorStore) { - logger?.warn?.(`${TAG} VectorStore not available`); - return { results: [], total: 0, strategy: "none" }; - } - - // โ”€โ”€ Determine available capabilities โ”€โ”€ - const hasEmbedding = !!embeddingService; - const hasFts = vectorStore.isFtsAvailable(); - - if (!hasEmbedding && !hasFts) { - logger?.warn?.(`${TAG} Neither EmbeddingService nor FTS5 available โ€” cannot search`); - return { - results: [], - total: 0, - strategy: "none", - message: - "Embedding service is not configured and FTS is not available. " + - "Memory search requires an embedding provider or FTS5 support. " + - "Please configure an embedding provider in the embedding.provider setting (e.g. openai_compatible).", - }; - } - - // โ”€โ”€ Over-retrieve for later filtering and RRF merging โ”€โ”€ - const candidateK = limit * 3; - - // โ”€โ”€ Run available search strategies in parallel โ”€โ”€ - const [ftsItems, vecItems] = await Promise.all([ - // FTS5 keyword search - (async (): Promise => { - if (!hasFts) return []; - try { - const ftsQuery = buildFtsQuery(query); - if (!ftsQuery) { - logger?.debug?.(`${TAG} [hybrid-fts] No usable FTS tokens from query`); - return []; - } - logger?.debug?.(`${TAG} [hybrid-fts] FTS5 query: "${ftsQuery}"`); - const ftsResults = await vectorStore.searchL1Fts(ftsQuery, candidateK); - logger?.debug?.(`${TAG} [hybrid-fts] FTS5 returned ${ftsResults.length} candidates`); - return ftsResults.map((r) => ({ - id: r.record_id, - content: r.content, - type: r.type, - priority: r.priority, - scene_name: r.scene_name, - score: r.score, - created_at: r.timestamp_start, - updated_at: r.timestamp_end, - })); - } catch (err) { - logger?.warn?.( - `${TAG} [hybrid-fts] FTS5 search failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - })(), - - // Vector embedding search - (async (): Promise => { - if (!hasEmbedding) return []; - try { - logger?.debug?.(`${TAG} [hybrid-vec] Generating query embedding...`); - const queryEmbedding = await embeddingService!.embed(query); - logger?.debug?.( - `${TAG} [hybrid-vec] Embedding OK, dims=${queryEmbedding.length}, searching top-${candidateK}...`, - ); - const vecResults: L1SearchResult[] = await vectorStore.searchL1Vector(queryEmbedding, candidateK, query); - logger?.debug?.(`${TAG} [hybrid-vec] Vector search returned ${vecResults.length} candidates`); - return vecResults.map((r) => ({ - id: r.record_id, - content: r.content, - type: r.type, - priority: r.priority, - scene_name: r.scene_name, - score: r.score, - created_at: r.timestamp_start, - updated_at: r.timestamp_end, - })); - } catch (err) { - logger?.warn?.( - `${TAG} [hybrid-vec] Embedding search failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`, - ); - return []; - } - })(), - ]); - - // โ”€โ”€ Determine effective strategy โ”€โ”€ - const ftsOk = ftsItems.length > 0; - const vecOk = vecItems.length > 0; - let strategy: string; - - if (ftsOk && vecOk) { - strategy = "hybrid"; - } else if (vecOk) { - strategy = "embedding"; - } else if (ftsOk) { - strategy = "fts"; - } else { - logger?.debug?.(`${TAG} Both search paths returned 0 results`); - return { results: [], total: 0, strategy: hasEmbedding ? "embedding" : "fts" }; - } - - // โ”€โ”€ Merge results โ”€โ”€ - let results: MemorySearchResultItem[]; - if (strategy === "hybrid") { - results = rrfMergeL1(ftsItems, vecItems); - logger?.debug?.( - `${TAG} [hybrid] RRF merged: fts=${ftsItems.length}, vec=${vecItems.length} โ†’ ${results.length} unique`, - ); - } else { - // Single-source: use whichever list has results (already sorted by score) - results = ftsOk ? ftsItems : vecItems; - } - - // โ”€โ”€ Apply secondary filters (type, scene) โ”€โ”€ - const preFilterCount = results.length; - if (typeFilter) { - results = results.filter((r) => r.type === typeFilter); - logger?.debug?.(`${TAG} After type filter "${typeFilter}": ${results.length}/${preFilterCount}`); - } - if (sceneFilter) { - const normalizedScene = sceneFilter.toLowerCase(); - results = results.filter((r) => - r.scene_name.toLowerCase().includes(normalizedScene), - ); - logger?.debug?.(`${TAG} After scene filter "${sceneFilter}": ${results.length}/${preFilterCount}`); - } - - // โ”€โ”€ Trim to requested limit โ”€โ”€ - const trimmed = results.slice(0, limit); - - logger?.debug?.( - `${TAG} RESULT (strategy=${strategy}): returning ${trimmed.length} memories ` + - `(scores: [${trimmed.map((r) => r.score.toFixed(3)).join(", ")}])`, - ); - - return { - results: trimmed, - total: trimmed.length, - strategy, - }; -} - -// ============================ -// Tool response formatter -// ============================ - -export function formatSearchResponse(result: MemorySearchResult): string { - if (result.message) { - return result.message; - } - if (result.results.length === 0) { - return "No matching memories found."; - } - - const lines: string[] = [ - `Found ${result.total} matching memories:`, - "", - ]; - - for (const item of result.results) { - const scoreStr = typeof item.score === "number" ? ` (score: ${item.score.toFixed(3)})` : ""; - const sceneStr = item.scene_name ? ` [scene: ${item.scene_name}]` : ""; - const priorityStr = item.priority >= 0 ? ` (priority: ${item.priority})` : " (global instruction)"; - lines.push(`- **[${item.type}]**${priorityStr}${sceneStr}${scoreStr}`); - lines.push(` ${item.content}`); - lines.push(""); - } - - return lines.join("\n"); -} diff --git a/src/utils/clean-context-runner.ts b/src/utils/clean-context-runner.ts index 857196b..868d787 100644 --- a/src/utils/clean-context-runner.ts +++ b/src/utils/clean-context-runner.ts @@ -89,8 +89,10 @@ async function resolveRunEmbeddedPiAgent( logger?.debug?.( `${TAG} resolveRunEmbeddedPiAgent: using injected runtime.agent.runEmbeddedPiAgent`, ); + logger?.debug?.(`${TAG} [l1-debug] RESOLVE source=injected`); return injected; } + logger?.debug?.(`${TAG} [l1-debug] RESOLVE source=dist-fallback`); return loadRunEmbeddedPiAgent(logger); } @@ -422,8 +424,27 @@ export class CleanContextRunner { const runId = `memory-${params.taskId}-run-${ts}`; this.logger?.debug?.(`${TAG} run() starting embedded agent: sessionId=${sessionId}, runId=${runId}, provider=${this.resolvedProvider ?? "(default)"}, model=${this.resolvedModel ?? "(default)"}`); + // [l1-debug] INVOKE โ€” what are we about to send to the embedded agent? + const sysPromptOverrideLen = + ((cleanConfig.agents as Record | undefined)?.defaults as Record | undefined)?.systemPromptOverride + ? String( + ((cleanConfig.agents as Record).defaults as Record).systemPromptOverride, + ).length + : 0; + const toolsAllow = + ((cleanConfig.tools as Record | undefined)?.allow as unknown[] | undefined) ?? []; + this.logger?.debug?.( + `${TAG} [l1-debug] INVOKE taskId=${params.taskId}, provider=${this.resolvedProvider ?? "(default)"}, model=${this.resolvedModel ?? "(default)"}, promptLen=${effectivePrompt.length}, sysPromptOverrideLen=${sysPromptOverrideLen}, toolsAllow=${JSON.stringify(toolsAllow)}, timeoutMs=${params.timeoutMs ?? 120_000}`, + ); + // Phase 2: Embedded agent run (LLM call + tool calls) const agentStartMs = Date.now(); + // extraSystemPrompt: fallback for openclaw < 2026.4.7 which does not support + // config.agents.defaults.systemPromptOverride. On newer versions the + // override takes precedence and this becomes a no-op append. + const effectiveSystemPrompt = + params.systemPrompt || + "You are a precise data extraction and generation assistant. Follow the user instructions exactly. Respond only with the requested output format."; const result = await runEmbeddedPiAgent({ sessionId, sessionFile, @@ -439,6 +460,7 @@ export class CleanContextRunner { // Instead rely on cleanConfig.tools.allow to restrict the tool set // to a minimal read-only tool (when enableTools=false). disableTools: false, + extraSystemPrompt: effectiveSystemPrompt, streamParams: { maxTokens: params.maxTokens, }, @@ -446,6 +468,28 @@ export class CleanContextRunner { const agentElapsedMs = Date.now() - agentStartMs; this.logger?.debug?.(`${TAG} run() embedded agent completed: ${agentElapsedMs}ms`); + // [l1-debug] RESULT โ€” what did the embedded agent return? + { + const payloadsRaw = (result as Record | undefined)?.payloads; + const payloads = Array.isArray(payloadsRaw) + ? (payloadsRaw as Array>) + : []; + const payloadKinds = payloads.map((p) => { + if (typeof p?.type === "string") return p.type as string; + if (typeof p?.kind === "string") return p.kind as string; + return Object.keys(p ?? {}).slice(0, 3).join("|") || "unknown"; + }); + const errorPayloadCount = payloads.filter((p) => p?.isError === true).length; + const joinedText = payloads + .filter((p) => !p?.isError && typeof p?.text === "string") + .map((p) => String(p.text ?? "")) + .join("\n"); + const textPreview = joinedText.replace(/\s+/g, " ").slice(0, 200); + this.logger?.debug?.( + `${TAG} [l1-debug] RESULT taskId=${params.taskId}, elapsedMs=${agentElapsedMs}, payloadCount=${payloads.length}, payloadKinds=${JSON.stringify(payloadKinds)}, errorPayloadCount=${errorPayloadCount}, textLen=${joinedText.length}, textPreview=${JSON.stringify(textPreview)}`, + ); + } + // Phase 3: Collect output const text = collectText((result as Record).payloads as Array<{ text?: string; isError?: boolean }> | undefined); const totalMs = Date.now() - runStartMs; @@ -455,6 +499,16 @@ export class CleanContextRunner { // extract (e.g. trivial greetings). Log a warning instead of // throwing so the caller can handle it gracefully. this.logger?.warn?.(`${TAG} run() empty output after ${totalMs}ms (import=${importElapsedMs}ms, agent=${agentElapsedMs}ms) โ€” treating as empty result`); + // [l1-debug] EMPTY_DUMP โ€” dump the full result shape so we can see where text went + try { + const dump = JSON.stringify(result, (_k, v) => { + if (typeof v === "string" && v.length > 500) return v.slice(0, 500) + `โ€ฆ(+${v.length - 500})`; + return v; + }).slice(0, 2048); + this.logger?.warn?.(`${TAG} [l1-debug] EMPTY_DUMP taskId=${params.taskId}, resultJson=${dump}`); + } catch (dumpErr) { + this.logger?.warn?.(`${TAG} [l1-debug] EMPTY_DUMP taskId=${params.taskId}, dumpFailed=${dumpErr instanceof Error ? dumpErr.message : String(dumpErr)}`); + } // llm_call metric (empty output) if (params.instanceId && this.logger) { report("llm_call", { diff --git a/src/utils/manifest.ts b/src/utils/manifest.ts index 20b8e36..30d51c6 100644 --- a/src/utils/manifest.ts +++ b/src/utils/manifest.ts @@ -5,7 +5,7 @@ * * - **store**: written once on first successful store init; never overwritten. * On subsequent starts the current config is compared against the persisted - * store binding โ€” mismatches are logged as warnings. + * store binding โ€” mismatches are logged at debug level (informational only). * - **seed**: written once when a seed run completes; null for live-runtime dirs. * * This file is informational / read-only from the user's perspective. diff --git a/src/utils/pipeline-factory.ts b/src/utils/pipeline-factory.ts index f4cba1d..4595753 100644 --- a/src/utils/pipeline-factory.ts +++ b/src/utils/pipeline-factory.ts @@ -222,11 +222,10 @@ async function _doInitStores( // Compare persisted store binding against current config const diffs = diffStoreBinding(existing.store, currentStoreInfo); if (diffs.length > 0) { - logger.warn( - `${TAG} โš ๏ธ Store config has changed since this data directory was created! ` + - `Diffs: ${diffs.join("; ")}. ` + - `Local JSONL data may not match the current store. ` + - `Consider re-seeding or migrating data.`, + logger.debug?.( + `${TAG} Store config differs from initial binding recorded in manifest ` + + `(${diffs.join("; ")}). ` + + `This is expected if the storage backend was switched intentionally.`, ); } }