diff --git a/.gitignore b/.gitignore
index 9cde883..2a57051 100644
--- a/.gitignore
+++ b/.gitignore
@@ -10,5 +10,10 @@ workspace/
# Test caches
__tests__/soak/.model-cache/
+# Migration build output
+scripts/export-tencent-vdb/dist/
+scripts/migrate-sqlite-to-tcvdb/dist/
+scripts/read-local-memory/dist/
+
node_modules/
-benchmark-runs/
\ No newline at end of file
+benchmark-runs/
diff --git a/.npmignore b/.npmignore
new file mode 100644
index 0000000..6a9a7e9
--- /dev/null
+++ b/.npmignore
@@ -0,0 +1,22 @@
+# 测试文件
+*.test.ts
+*.test.js
+*.spec.ts
+*.spec.js
+__tests__/
+
+# 开发文档与辅助
+docs/
+benchmark-runs/
+workspace/
+
+# 环境与配置
+.env
+.env.*
+.gitignore
+.codebuddy/
+.coding-ci.yaml
+
+# 运行时产物
+node_modules/
+*.tgz
\ No newline at end of file
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 74cbbfb..dc79017 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,101 @@
---
+## [0.2.2] - 2026-04-17
+
+### 🐛 修复
+
+- 修复因未声明 `undici` 依赖导致 TCVDB 客户端加载失败的问题（开发环境之前依赖 monorepo 根 `node_modules` 的传递解析）
+- 将插件注册阶段的大量 INFO 日志降级为 DEBUG，避免 CLI 模式下输出过多无关日志
+
+## [0.2.1] - 2026-04-16 (deprecated)
+
+> NOTE: 此版本由于存在 undici 依赖导致插件启动失败的问题，已废弃
+> 相关问题在 0.2.2 及以后版本中已修复
+
+### 🚀 新功能
+
+- TCVDB 新增 HTTPS 连接支持，可通过插件配置 `caPemPath` 或迁移脚本参数 `--tcvdb-ca-pem` 指定自定义 CA 证书 PEM 文件
+- `read-local-memory` 脚本新增 L2 单文件查询，并将 L0 / L1 查询切换为直接从 `vectors.db` 读取，支持 SQL 层过滤、排序与分页
+
+### ✨ 改进
+
+- TCVDB 的 L0 / L1 向量索引默认调整为 `DISK_FLAT`，并在不支持该索引类型的实例上自动回退到 `HNSW`
+- 默认服务端 embedding 模型调整为 `bge-large-zh`
+- TCVDB 所有读接口统一启用 `readConsistency: "strongConsistency"`，消除 read-after-write 不一致
+- 健康检测脚本 VDB 连接支持 HTTPS 自签证书
+
+### 🐛 修复
+
+- 修复 L3 persona sync 因未拉取远端 baseline 导致版本冲突跳过写入的问题
+- 修复 `memories_since_last_persona` 被 L0 和 L1 双重计数导致 persona 触发阈值膨胀的问题
+- 移除 `CheckpointManager` 中已被 `captureAtomically()` 替代的废弃方法
+
+---
+
+## [0.2.0] - 2026-04-15
+
+### 🚀 新功能
+
+**腾讯云向量数据库（TCVDB）存储后端**
+
+- 新增腾讯云向量数据库存储后端，支持向量 + BM25 混合召回
+- 支持 SQLite 与 TCVDB 之间的索引结构同步
+- L2 场景 / L3 画像支持在本地缓存与向量数据库之间双向同步
+- 插件配置（manifest）暴露 `storeBackend`、`tcvdb`、`bm25`、`embedding.timeoutMs` 等配置项
+
+**本地 BM25 关键字检索**
+
+- 使用本地 tcvdb-text 编码器替代原有的 BM25 HTTP sidecar 服务，消除外部依赖
+
+**Seed 数据导入工具**
+
+- 新增 CLI `seed` 命令，支持从外部数据批量导入记忆
+- 提取共享的 pipeline-factory，供 seed 和正常运行时复用
+- 支持 ISO 8601 时间戳格式（移除 JSONL 支持）
+
+**数据迁移与运维工具**
+
+- 新增 SQLite → 腾讯云向量数据库迁移脚本，支持 `--help` / `-h` 展示完整参数说明和使用示例
+- 新增 VDB 数据导出脚本（含预编译 JS 和 CLI 启动器）
+- 新增本地 Memory 数据查询脚本
+- 注册全部 CLI bin 入口：`migrate-sqlite-to-tcvdb`、`export-tencent-vdb`、`read-local-memory`
+
+**记忆搜索工具调用限制**
+
+- `tdai_memory_search` + `tdai_conversation_search` 增加每轮合计最多 3 次的调用次数限制，通过 tool description 和召回引导提示词约束模型行为，防止陷入无效重复搜索
+
+### 🐛 修复
+
+- 修复 L2 场景合并（MERGE）无法删除旧文件的问题：OpenClaw 4.1+ 的 write 工具拒绝空白内容，改用 `[DELETED]` 标记实现软删除，SceneExtractor cleanup 阶段同步识别并清理
+- 修复 L2 抽取产生孤立 BATCH/ARCHIVE 文件的问题，统一 maxScenes 上限为 15
+- 修复 L3 启动时重复拉取 profile 的问题
+- 过滤 skill wrapper 噪声标记（`¥¥[...]¥¥`）
+- 处理 `createCollection` 并发竞态（错误码 15202）
+
+### ♻️ 重构
+
+- Pipeline checkpoint 游标语义从 timestamp 改为 update_at
+- Runner 改用 `api.runtime.agent.runEmbeddedPiAgent`，避免跨环境导入失败
+- 统一脚本构建流程：新增 `build:scripts` 一键编译命令，`prepack` 钩子确保 `npm pack` 前自动编译全部脚本产物
+
+### 📚 文档
+
+- 新增 AI Agent 长期记忆插件设计与实现技术文档
+- 新增项目指南、研发系统分层架构文档
+- 新增 VDB 存储设计文档及迁移指南
+
+---
+
+
+预发布版本
+
+## [0.2.0-beta.1] - 2026-04-14
+
+*此版本的内容已合并至 [0.2.0] 正式版。*
+
+
+
## [0.1.4] - 2026-04-10
### 🚀 Features
diff --git a/SKILL-DIAGNOSTIC-EXPORT.md b/SKILL-DIAGNOSTIC-EXPORT.md
new file mode 100644
index 0000000..09976ec
--- /dev/null
+++ b/SKILL-DIAGNOSTIC-EXPORT.md
@@ -0,0 +1,152 @@
+---
+name: openclaw-diagnostic-export
+description: 帮助用户导出 OpenClaw + memory-tencentdb（原 memory-tdai）记忆插件的现场诊断数据，用于排查问题。当用户提到"导出诊断数据""export diagnostic""现场数据""排查问题""导出日志""收集现场""打包现场数据"时应触发。
+version: 1.0.0
+---
+
+## 目的
+
+将 OpenClaw 日志、记忆插件数据（L0~L3）、脱敏后的配置打包为本地压缩包，由用户确认后手动发送给研发团队排查问题。
+
+> **名称说明**：插件已从 `@tdai/memory-tdai` 更名为 `@tencentdb-agent-memory/memory-tencentdb`，但数据目录始终为 `~/.openclaw/memory-tdai/`（代码中硬编码）。本 skill 中所有对 `memory-tdai` 目录的引用均指实际数据目录路径，与插件 ID 无关。
+
+## 导出工作流
+
+### Step 1: 确认环境
+
+在导出前，先确认 OpenClaw 工作目录存在且可访问：
+
+```bash
+# 探测工作目录（优先级：环境变量 > ~/.openclaw > ~/.clawdbot）
+OPENCLAW_DIR="${OPENCLAW_STATE_DIR:-$HOME/.openclaw}"
+[ -d "$OPENCLAW_DIR" ] || OPENCLAW_DIR="$HOME/.clawdbot"
+ls -la "$OPENCLAW_DIR/" 2>/dev/null && echo "✅ 找到: $OPENCLAW_DIR" || echo "❌ 未找到 OpenClaw 工作目录"
+```
+
+确认 memory-tdai 子目录存在：
+
+```bash
+ls -la "$OPENCLAW_DIR/memory-tdai/" 2>/dev/null
+```
+
+### Step 2: 执行导出脚本
+
+运行项目 `scripts/` 目录下的导出脚本：
+
+```bash
+bash scripts/export-diagnostic.sh
+```
+
+> 脚本位于本项目的 `scripts/export-diagnostic.sh`，如果通过 `pnpm` 或其他方式运行，需确保工作目录在项目根目录下。
+
+脚本默认将压缩包输出到 `~/Downloads/openclaw-diagnostic-.tar.gz`。
+
+如需指定其他输出目录：
+
+```bash
+bash scripts/export-diagnostic.sh /tmp
+```
+
+### Step 3: 确认导出结果
+
+脚本执行完成后，检查输出：
+
+1. **确认压缩包已生成** — 脚本末尾会打印压缩包路径和大小
+2. **向用户说明包含内容**：
+
+| 文件/目录 | 内容 | 隐私风险 |
+|-----------|------|---------|
+| `env-info.txt` | 系统版本、OpenClaw 版本、目录结构、磁盘占用 | 低 |
+| `logs/` | OpenClaw 网关日志 + 滚动日志（最近 3 天，每文件最多 5000 行） | 低 |
+| `memory-tdai/` | 记忆插件全量数据：L0 对话、L1 记忆、L2 场景、L3 画像、SQLite 数据库、checkpoint | **高** — 包含用户对话原文 |
+| `openclaw-config-redacted.json` | 脱敏后的配置（已移除 API Key/Token/Password/Secret，models/channels/env 整体替换） | 低 |
+| `plugins-info.txt` | 已安装插件列表和版本 | 低 |
+
+3. **提醒用户**：
+ - 配置文件已自动脱敏，API Key、Token 等敏感信息已被替换为 `***REDACTED***`
+ - **记忆数据（memory-tdai/）包含用户对话原文**，请确认可以分享后再发送
+ - 压缩包存放在本地，**不会自动上传**，需要用户手动发送给研发团队
+
+### Step 4: 告知用户后续操作
+
+导出完成后，告知用户：
+
+1. 压缩包已保存在本地（打印具体路径）
+2. 请检查内容后，通过企微/邮件等方式手动发送给研发团队
+3. 如只需部分数据（如仅日志或仅配置），可解压后选择性发送
+
+## 导出内容详解
+
+### OpenClaw 日志位置
+
+| 日志类型 | 路径 | 说明 |
+|---------|------|------|
+| 网关 stdout | `~/.openclaw/logs/gateway.log` | 网关守护进程标准输出 |
+| 网关 stderr | `~/.openclaw/logs/gateway.err.log` | 网关守护进程错误输出 |
+| 滚动日志 | `/tmp/openclaw/openclaw-YYYY-MM-DD.log` | 按日期滚动，JSON Lines 格式，24h 自动清理 |
+| 配置审计 | `~/.openclaw/logs/config-audit.jsonl` | 配置写入审计记录 |
+| 命令日志 | `~/.openclaw/logs/commands.log` | 命令事件日志（hook 可选） |
+
+### 记忆插件数据结构
+
+```
+~/.openclaw/memory-tdai/
+├── conversations/ — L0 原始对话（每日 JSONL 分片）
+├── records/ — L1 结构化记忆（每日 JSONL 分片）
+├── scene_blocks/ — L2 场景 Markdown 文件
+├── persona.md — L3 用户画像
+├── vectors.db — SQLite 数据库（向量 + 全文索引）
+├── .metadata/ — checkpoint、scene_index.json
+└── .backup/ — 滚动备份
+```
+
+### 配置脱敏规则
+
+导出脚本对 `openclaw.json` 执行以下脱敏：
+
+| 规则 | 处理方式 |
+|------|---------|
+| 字段名匹配 `apiKey/token/password/secret/credential` 且值为字符串 | 替换为 `***REDACTED(Nchars)***` |
+| SecretRef 对象（含 source/provider/id） | id 替换为 `***REDACTED***` |
+| 顶层 `models`、`secrets`、`channels`、`env` 块 | 整体替换为 `***REDACTED_SECTION***` |
+| `gateway.auth` 下的 token/password | 替换为 `***REDACTED***` |
+| 其余字段（含 `plugins` 完整配置） | **保留原样**（插件配置是排查重点） |
+
+## 手动导出（脚本不可用时的备选方案）
+
+如果导出脚本无法执行（如 Node.js 不可用），按以下步骤手动收集：
+
+```bash
+# 1. 创建导出目录
+EXPORT_DIR=~/Downloads/openclaw-diagnostic-$(date +%Y%m%d-%H%M%S)
+mkdir -p "$EXPORT_DIR"
+
+# 2. 复制日志
+cp -r ~/.openclaw/logs/ "$EXPORT_DIR/logs/" 2>/dev/null
+cp /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log "$EXPORT_DIR/" 2>/dev/null
+
+# 3. 复制记忆插件数据
+cp -r ~/.openclaw/memory-tdai/ "$EXPORT_DIR/memory-tdai/" 2>/dev/null
+
+# 4. 手动脱敏配置（⚠️ 必须手动删除敏感字段！）
+# 复制配置并用编辑器删除 models/secrets/channels 块和所有 apiKey/token 值
+cp ~/.openclaw/openclaw.json "$EXPORT_DIR/openclaw-config-NEEDS-MANUAL-REDACTION.json"
+
+# 5. 打包
+cd ~/Downloads && tar -czf "$EXPORT_DIR.tar.gz" "$(basename $EXPORT_DIR)"
+
+echo "⚠️ 请务必在发送前手动检查并删除配置中的敏感信息！"
+```
+
+## 常见问题排查线索
+
+导出数据后，研发团队通常关注以下方面：
+
+| 排查方向 | 查看文件 | 关键信息 |
+|---------|---------|---------|
+| 插件是否加载 | `logs/` 中搜索 `[memory-tdai]` | 插件注册、配置解析日志（注：日志标签仍为 `[memory-tdai]`，与插件 ID 无关） |
+| 记忆召回是否工作 | `logs/` 中搜索 `[recall]` | 搜索策略、耗时、命中数 |
+| L1 提取是否触发 | `logs/` 中搜索 `[pipeline]` | 调度触发、L1/L2/L3 执行状态 |
+| 向量搜索是否可用 | `openclaw-config-redacted.json` 的 `plugins.entries` | embedding 配置是否正确 |
+| 数据量/磁盘占用 | `env-info.txt` | du 输出、文件数量 |
+| checkpoint 状态 | `memory-tdai/.metadata/recall_checkpoint.json` | 进度、游标、计数器 |
diff --git a/SKILL-MIGRATION.md b/SKILL-MIGRATION.md
new file mode 100644
index 0000000..0498f64
--- /dev/null
+++ b/SKILL-MIGRATION.md
@@ -0,0 +1,239 @@
+---
+name: openclaw-memory-tencentdb-migration
+description: 帮助存量用户将 OpenClaw 记忆插件从旧包 @tdai/memory-tdai 迁移到新包 @tencentdb-agent-memory/memory-tencentdb。当用户提到"插件迁移""更换记忆插件包名""memory-tdai 升级""包名变更"或出现旧包相关安装报错时应触发。
+version: 1.0.0
+---
+
+## 目的
+
+帮助已安装 `@tdai/memory-tdai`（旧包名）的存量用户，平滑迁移到 `@tencentdb-agent-memory/memory-tencentdb`（新包名），确保已有记忆数据不丢失、配置完整还原。
+
+## 背景
+
+- **旧包名**：`@tdai/memory-tdai`（插件 ID：`memory-tdai`）
+- **新包名**：`@tencentdb-agent-memory/memory-tencentdb`（插件 ID：`memory-tencentdb`）
+- 新旧插件共用相同的数据目录（`~/.openclaw/memory-tdai/`），卸载旧插件**不会删除数据目录**，已有记忆数据不受影响
+- 卸载旧插件**会删除** `openclaw.json` 中该插件的配置段，需提前备份
+
+## 适用场景
+
+- 用户已安装 `@tdai/memory-tdai`，需迁移到新包名
+- 用户执行 `openclaw plugins install @tdai/memory-tdai` 报 404 / not found
+- 用户被告知旧包已废弃，需要迁移
+
+## 不适用场景
+
+- 用户从未安装过记忆插件（应使用 `openclaw-memory-tencentdb-setup` skill）
+- 用户使用的是其他记忆插件（如 `openclaw-mem0`）
+
+## 标准工作流
+
+### 1) 确认当前状态
+
+确认旧插件是否已安装：
+
+```bash
+openclaw plugins list | grep -i memory
+```
+
+预期看到 `memory-tdai` 或 `@tdai/memory-tdai` 处于 loaded 状态。
+
+如果未看到旧插件，跳过迁移流程，直接使用 `openclaw-memory-tencentdb-setup` skill 进行全新安装。
+
+### 2) 备份现有配置（关键步骤）
+
+卸载旧插件会删除 `openclaw.json` 中的配置段。**必须先备份**。
+
+执行以下命令提取旧插件配置：
+
+```bash
+cat ~/.openclaw/openclaw.json | python3 -c "
+import sys, json
+cfg = json.load(sys.stdin)
+plugins = cfg.get('plugins', {}).get('entries', {})
+old_cfg = plugins.get('memory-tdai', {})
+if old_cfg:
+ print(json.dumps(old_cfg, indent=2, ensure_ascii=False))
+ with open('/tmp/memory-tdai-config-backup.json', 'w') as f:
+ json.dump(old_cfg, f, indent=2, ensure_ascii=False)
+ print('\n✅ 配置已备份到 /tmp/memory-tdai-config-backup.json')
+else:
+ print('⚠️ 未找到 memory-tdai 配置段（可能使用默认配置）')
+"
+```
+
+**特别关注以下配置是否存在（如有则必须记录）**：
+
+- `embedding` 配置（`provider`、`baseUrl`、`apiKey`、`model`、`dimensions`、`proxyUrl`）
+- `extraction.model`（提取使用的模型）
+- `persona.model`（画像使用的模型）
+- `capture.excludeAgents`（排除的 agent 列表）
+- `capture.l0l1RetentionDays`（数据保留天数）
+
+### 3) 确认数据目录存在
+
+```bash
+ls -la ~/.openclaw/memory-tdai/
+```
+
+预期看到：`conversations/`、`records/`、`scene_blocks/`、`vectors.db`、`persona.md` 等文件。
+
+记录当前数据量作为迁移后验证依据：
+
+```bash
+echo "=== 迁移前数据统计 ==="
+wc -l ~/.openclaw/memory-tdai/conversations/*.jsonl 2>/dev/null || echo "无对话数据"
+wc -l ~/.openclaw/memory-tdai/records/*.jsonl 2>/dev/null || echo "无记录数据"
+ls ~/.openclaw/memory-tdai/scene_blocks/*.md 2>/dev/null | wc -l | xargs -I{} echo "场景块: {} 个"
+wc -c ~/.openclaw/memory-tdai/persona.md 2>/dev/null || echo "无 persona"
+```
+
+### 4) 卸载旧插件
+
+```bash
+openclaw plugins uninstall memory-tdai
+```
+
+执行后确认：
+
+- `openclaw.json` 中 `memory-tdai` 配置段已被删除（预期行为）
+- `~/.openclaw/memory-tdai/` 数据目录**仍然存在**（不会被删除）
+
+```bash
+# 验证数据目录仍在
+ls ~/.openclaw/memory-tdai/ && echo "✅ 数据目录完好" || echo "❌ 数据目录丢失！"
+```
+
+### 5) 安装新插件
+
+```bash
+openclaw plugins install @tencentdb-agent-memory/memory-tencentdb
+```
+
+### 6) 还原配置
+
+将步骤 2 备份的配置写回 `openclaw.json`，注意新插件的配置 key 是 `memory-tencentdb`：
+
+```bash
+python3 -c "
+import json, os
+
+# 读取备份配置
+backup_path = '/tmp/memory-tdai-config-backup.json'
+if os.path.exists(backup_path):
+ with open(backup_path) as f:
+ old_cfg = json.load(f)
+ print('📋 备份配置内容：')
+ print(json.dumps(old_cfg, indent=2, ensure_ascii=False))
+else:
+ old_cfg = {'enabled': True}
+ print('⚠️ 未找到备份，使用最小配置')
+
+# 读取当前 openclaw.json
+config_path = os.path.expanduser('~/.openclaw/openclaw.json')
+with open(config_path) as f:
+ cfg = json.load(f)
+
+# 写入新插件配置
+cfg.setdefault('plugins', {}).setdefault('entries', {})['memory-tencentdb'] = old_cfg
+
+with open(config_path, 'w') as f:
+ json.dump(cfg, f, indent=2, ensure_ascii=False)
+
+print('\n✅ 配置已写入 memory-tencentdb')
+"
+```
+
+如果备份丢失或用户需要手动恢复，至少确保写入最小配置：
+
+```json
+{
+ "memory-tencentdb": {
+ "enabled": true
+ }
+}
+```
+
+### 7) 重启 Gateway 并验证
+
+```bash
+openclaw gateway restart
+```
+
+检查项：
+
+- Gateway 日志中出现 `[memory-tdai]` 前缀（注：日志标签仍为 memory-tdai，这是正常的）
+- 数据目录内容未变化
+
+```bash
+echo "=== 迁移后验证 ==="
+# 确认新插件已加载
+openclaw plugins list | grep -i memory
+
+# 确认数据量与迁移前一致
+wc -l ~/.openclaw/memory-tdai/conversations/*.jsonl 2>/dev/null
+wc -l ~/.openclaw/memory-tdai/records/*.jsonl 2>/dev/null
+```
+
+### 8) 功能冒烟验证
+
+执行一次对话确认记忆链路正常：
+
+1. 发送一条包含个人信息的消息（如偏好、习惯）
+2. 确认日志中有 `[before_prompt_build]` 和 `[agent_end]` 相关输出
+3. 如有 embedding 配置，确认向量检索正常（日志无 embedding 报错）
+
+## 回滚方案
+
+如迁移后出现问题，可快速回滚：
+
+```bash
+# 1. 卸载新插件
+openclaw plugins uninstall memory-tencentdb
+
+# 2. 重新安装旧插件（如 npm 源仍可用）
+openclaw plugins install @tdai/memory-tdai
+
+# 3. 手动还原配置（从备份）
+# 将 /tmp/memory-tdai-config-backup.json 内容写回 openclaw.json 的 memory-tdai 段
+
+# 4. 重启
+openclaw gateway restart
+```
+
+## 故障排查
+
+| 现象 | 可能原因 | 解决方案 |
+|------|----------|----------|
+| 新插件无日志输出 | 配置中 `enabled` 未设为 `true` | 检查 `openclaw.json` 中 `memory-tencentdb.enabled` |
+| 安装新插件报错 | npm 源不可用 | 检查网络 / npm registry 配置 |
+| 迁移后无历史记忆 | 配置还原不完整 | 对比 `/tmp/memory-tdai-config-backup.json` 与当前配置 |
+| embedding 报错 | `apiKey` 等配置丢失 | 从备份中还原 `embedding` 配置段 |
+| 数据目录为空 | 卸载时异常删除（极少见） | 检查 `~/.openclaw/memory-tdai/` 是否存在 |
+
+## 安全与合规约束
+
+- 备份文件 `/tmp/memory-tdai-config-backup.json` 可能包含 `apiKey`，迁移完成后建议删除：`rm /tmp/memory-tdai-config-backup.json`
+- 不在聊天、日志中明文展示 `apiKey`
+- 仅修改 `memory-tencentdb` 配置段，不影响用户其它插件
+
+## 完成定义（Definition of Done）
+
+迁移完成需同时满足：
+
+- [x] 旧插件 `@tdai/memory-tdai` 已卸载
+- [x] 新插件 `@tencentdb-agent-memory/memory-tencentdb` 已安装并加载
+- [x] `openclaw.json` 中存在完整的 `memory-tencentdb` 配置（含用户自定义的 embedding 等配置）
+- [x] Gateway 已重启
+- [x] 日志中出现 `[memory-tdai]` 前缀
+- [x] 数据目录完好，数据量与迁移前一致
+- [x] 至少 1 次对话验证记忆链路正常
+- [x] 已清理备份文件中的敏感信息
+
+## 交付话术模板
+
+> 已完成记忆插件迁移：
+> - 旧插件 `@tdai/memory-tdai` → 新插件 `@tencentdb-agent-memory/memory-tencentdb`
+> - 已有记忆数据完整保留（对话/记录/场景块/向量库均未受影响）
+> - 配置已从旧插件完整还原（含 embedding / extraction / persona 等自定义配置）
+> - Gateway 已重启，记忆链路验证正常
diff --git a/SKILL.md b/SKILL.md
new file mode 100644
index 0000000..74c3d7a
--- /dev/null
+++ b/SKILL.md
@@ -0,0 +1,201 @@
+---
+name: openclaw-memory-tencentdb-setup
+description: 用于在 OpenClaw 环境中安装、配置并验证 @tencentdb-agent-memory/memory-tencentdb 插件。当用户提到"安装记忆插件""配置 memory-tencentdb""开启长期记忆/召回"或出现相关报错时应触发。
+version: 1.0.0
+---
+
+## 目的
+
+在不依赖外部托管记忆服务的前提下，为 OpenClaw 提供可持续的本地长期记忆能力（L0→L1→L2→L3），并完成从安装、配置到验收的一次性闭环。
+
+## 适用场景
+
+- 用户要求在 OpenClaw 中安装或启用 `memory-tencentdb`
+- 用户需要配置召回、提取、画像、清理等参数
+- 用户反馈"插件已装但无记忆 / 无召回 / 无向量检索"
+
+## 不适用场景
+
+- 用户只需要解释 memory 理念，不要求实际落地
+- 用户要接入非 OpenClaw 宿主（先确认目标框架）
+
+## 标准工作流
+
+### 1) 环境预检
+
+先确认基础版本满足要求：
+
+- OpenClaw: `>= 2026.3.13`
+- Node.js: `>= 22.16.0`
+
+执行：
+
+```bash
+openclaw --version
+node -v
+```
+
+若版本不满足，先升级再继续。
+
+### 2) 安装插件
+
+执行安装命令：
+
+```bash
+openclaw plugins install @tencentdb-agent-memory/memory-tencentdb
+```
+
+如已安装则执行更新：
+
+```bash
+openclaw plugins update memory-tencentdb
+```
+
+### 3) 写入最小配置
+
+编辑 `~/.openclaw/openclaw.json`，确保存在：
+
+```json
+{
+ "memory-tencentdb": {
+ "enabled": true
+ }
+}
+```
+
+说明：该插件支持零配置启动；不补充其它字段也能运行基础能力。
+
+### 4) 按需追加推荐配置（生产常用）
+
+根据用户需求补充如下分组：
+
+- `capture`: 对话捕获与保留策略
+- `extraction`: L1 提取与去重
+- `pipeline`: L1→L2→L3 调度
+- `recall`: 召回数量、阈值、策略
+- `persona`: 场景与画像触发参数
+- `embedding`: 向量检索配置（远端 OpenAI 兼容）
+
+推荐模板：
+
+```json
+{
+ "memory-tencentdb": {
+ "capture": {
+ "enabled": true,
+ "excludeAgents": [],
+ "l0l1RetentionDays": 90,
+ "cleanTime": "03:00"
+ },
+ "extraction": {
+ "enabled": true,
+ "enableDedup": true,
+ "maxMemoriesPerSession": 10,
+ "model": "provider/model"
+ },
+ "pipeline": {
+ "everyNConversations": 5,
+ "enableWarmup": true,
+ "l1IdleTimeoutSeconds": 60,
+ "l2DelayAfterL1Seconds": 90,
+ "l2MinIntervalSeconds": 300,
+ "l2MaxIntervalSeconds": 1800,
+ "sessionActiveWindowHours": 24
+ },
+ "recall": {
+ "enabled": true,
+ "maxResults": 5,
+ "scoreThreshold": 0.3,
+ "strategy": "hybrid"
+ },
+ "persona": {
+ "triggerEveryN": 50,
+ "maxScenes": 15,
+ "backupCount": 3,
+ "sceneBackupCount": 10,
+ "model": "provider/model"
+ },
+ "embedding": {
+ "enabled": true,
+ "provider": "openai",
+ "baseUrl": "https://api.openai.com/v1",
+ "apiKey": "${EMBEDDING_API_KEY}",
+ "model": "text-embedding-3-small",
+ "dimensions": 1536,
+ "conflictRecallTopK": 5
+ }
+ }
+}
+```
+
+### 5) 关键配置规则（避免隐性失败）
+
+- `embedding.provider = "none"` 时，向量能力会禁用，仅保留关键词路径。
+- 若配置远端 `provider`（如 `openai` / `deepseek`），必须同时提供：
+ - `apiKey`
+ - `baseUrl`
+ - `model`
+ - `dimensions`
+- 上述任一缺失时，插件会继续运行，但自动降级为非向量模式。
+- `l0l1RetentionDays`：
+ - `0` 表示不清理
+ - 非 `0` 时建议 `>=3`
+ - 若设为 `1~2`，需显式开启 `allowAggressiveCleanup`
+
+### 6) 重启并验证生效
+
+执行：
+
+```bash
+openclaw gateway restart
+```
+
+检查项：
+
+- Gateway 日志中出现 `[memory-tdai]` 前缀
+- 数据目录已创建：`~/.openclaw/state/memory-tdai/`
+- 至少包含：`conversations/`、`records/`、`scene_blocks/`、`vectors.db`
+
+### 7) 功能冒烟测试
+
+执行一次最小对话回路并验证：
+
+1. 连续对话 2~3 轮，提供可记忆信息（偏好、约束、背景）。
+2. 发起新一轮对话，观察是否出现召回上下文注入。
+3. 在 Agent 中调用：
+ - `tdai_memory_search`
+ - `tdai_conversation_search`
+4. 确认能检索到刚刚产生的内容。
+
+## 故障排查速查
+
+- 插件无日志：检查 `openclaw.json` 中 `memory-tencentdb.enabled` 是否为 `true`，并确认已重启 Gateway。
+- 有记录无召回：检查 `recall.enabled`、`scoreThreshold` 是否过高。
+- 无向量结果：检查 `embedding` 四元组（`apiKey/baseUrl/model/dimensions`）是否齐全。
+- 清理过猛导致历史过少：检查 `l0l1RetentionDays` 与 `allowAggressiveCleanup`。
+- 配置已改但行为不变：确认修改的是 `~/.openclaw/openclaw.json`，并再次重启 Gateway。
+
+## 安全与合规约束
+
+- 将 `apiKey` 视为敏感信息；不在聊天、日志、截图中明文扩散。
+- 优先使用环境变量注入密钥；配置示例中仅保留占位符。
+- 仅修改 `memory-tencentdb` 对应配置段，避免覆盖用户其它插件配置。
+
+## 完成定义（Definition of Done）
+
+在结束任务前，必须同时满足：
+
+- 插件安装/更新命令执行成功
+- `openclaw.json` 已存在有效 `memory-tencentdb` 配置
+- Gateway 已重启
+- `[memory-tdai]` 日志可见
+- 数据目录与关键文件已生成
+- 至少 1 次检索工具调用成功返回结果
+
+## 交付话术模板
+
+可在完成后向用户输出：
+
+- 已完成 `memory-tencentdb` 安装与配置，并重启 Gateway。
+- 已验证日志与数据目录生效，记忆链路可用。
+- 如需下一步优化，可继续调优 `recall.scoreThreshold`、`pipeline.everyNConversations`、`persona.triggerEveryN` 与 `embedding` 模型参数。
\ No newline at end of file
diff --git a/index.ts b/index.ts
index 594fb8f..eb7431a 100644
--- a/index.ts
+++ b/index.ts
@@ -10,7 +10,6 @@
* All processing is local, zero external API dependencies.
*/
-import fs from "node:fs";
import path from "node:path";
import { createRequire } from "node:module";
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/core";
@@ -19,54 +18,33 @@ import type { MemoryTdaiConfig } from "./src/config.js";
import { performAutoRecall } from "./src/hooks/auto-recall.js";
import { performAutoCapture } from "./src/hooks/auto-capture.js";
import { MemoryPipelineManager } from "./src/utils/pipeline-manager.js";
-import { SceneExtractor } from "./src/scene/scene-extractor.js";
import { CheckpointManager } from "./src/utils/checkpoint.js";
-import { PersonaTrigger } from "./src/persona/persona-trigger.js";
-import { PersonaGenerator } from "./src/persona/persona-generator.js";
-import { prewarmEmbeddedAgent } from "./src/utils/clean-context-runner.js";
+import {
+ prewarmEmbeddedAgent,
+ setPreferredEmbeddedAgentRuntime,
+} from "./src/utils/clean-context-runner.js";
import { SessionFilter } from "./src/utils/session-filter.js";
-import { extractL1Memories } from "./src/record/l1-extractor.js";
-import { readConversationMessagesGroupedBySessionId } from "./src/conversation/l0-recorder.js";
-import type { ConversationMessage } from "./src/conversation/l0-recorder.js";
-import { VectorStore } from "./src/store/vector-store.js";
-import { createEmbeddingService } from "./src/store/embedding.js";
+import type { IMemoryStore } from "./src/store/types.js";
import type { EmbeddingService } from "./src/store/embedding.js";
import { executeMemorySearch, formatSearchResponse } from "./src/tools/memory-search.js";
import { executeConversationSearch, formatConversationSearchResponse } from "./src/tools/conversation-search.js";
import { LocalMemoryCleaner } from "./src/utils/memory-cleaner.js";
-import { getOrCreateInstanceId, initReporter, report } from "./src/report/reporter.js";
+import { registerMemoryTdaiCli } from "./src/cli/index.js";
+import {
+ initDataDirectories,
+ initStores,
+ resetStores,
+ createPipelineManager,
+ createL1Runner,
+ createPersister,
+ createL2Runner,
+ createL3Runner,
+} from "./src/utils/pipeline-factory.js";
+import { getOrCreateInstanceId, initReporter, report, resetReporter } from "./src/report/reporter.js";
+import { ensureL2L3Local } from "./src/profile/profile-sync.js";
const TAG = "[memory-tdai]";
-/**
- * Initialize all required data directories under the plugin data root.
- *
- * Called once at plugin registration time so downstream modules
- * (L0 recorder, L1 writer, scene extractor, persona generator, etc.)
- * don't need to lazily mkdir on every write — the directories are
- * guaranteed to exist from startup.
- *
- * Directory layout:
- * /
- * ├── conversations/ — L0 daily JSONL shards (one message per line)
- * ├── records/ — L1 daily JSONL shards (extracted memories)
- * ├── scene_blocks/ — L2 scene block .md files (LLM-managed)
- * ├── .metadata/ — checkpoint, scene_index.json
- * └── .backup/ — rotating backups (persona, scene_blocks)
- */
-function initDataDirectories(dataDir: string): void {
- const dirs = [
- "conversations",
- "records",
- "scene_blocks",
- ".metadata",
- ".backup",
- ];
- for (const sub of dirs) {
- fs.mkdirSync(path.join(dataDir, sub), { recursive: true });
- }
-}
-
/**
* Epoch ms when the plugin was registered (cold-start timestamp).
* Used as a fallback cursor in performAutoCapture when no checkpoint
@@ -151,20 +129,20 @@ function sweepStaleCaches(): void {
export default function register(api: OpenClawPluginApi) {
pluginStartTimestamp = Date.now();
+ setPreferredEmbeddedAgentRuntime(api.runtime.agent);
+ // Reset reporter singleton so config changes take effect on hot-reload.
+ resetReporter();
const _require = createRequire(import.meta.url);
const pluginVersion = (() => { try { return (_require("./package.json") as { version?: string }).version ?? "unknown"; } catch { return "unknown"; } })();
- api.logger.info(
+ api.logger.debug?.(
`${TAG} Registering plugin ... ` +
`startTimestamp=${pluginStartTimestamp} (${new Date(pluginStartTimestamp).toISOString()})`,
);
- // Persistent instance ID for metric reporting (populated async below)
- let instanceId: string | undefined;
-
let cfg: MemoryTdaiConfig;
try {
cfg = parseConfig(api.pluginConfig as Record | undefined);
- api.logger.info(
+ api.logger.debug?.(
`${TAG} Config parsed: ` +
`capture=${cfg.capture.enabled}, ` +
`recall=${cfg.recall.enabled}(maxResults=${cfg.recall.maxResults}), ` +
@@ -186,12 +164,26 @@ export default function register(api: OpenClawPluginApi) {
// Resolve plugin data directory via runtime API (avoid importing internal paths directly)
const pluginDataDir = path.join(api.runtime.state.resolveStateDir(), "memory-tdai");
initDataDirectories(pluginDataDir);
- api.logger.info(`${TAG} Data dir: ${pluginDataDir} (all subdirectories initialized)`);
+ api.logger.debug?.(`${TAG} Data dir: ${pluginDataDir} (all subdirectories initialized)`);
+
+ // Kick off instanceId resolution immediately after data dir is ready.
+ // getOrCreateInstanceId only reads/writes a small UUID file and caches the
+ // result — starting it here means it will almost certainly be settled before
+ // the first L1 runner fires, avoiding the need to defer metric reporting.
+ let instanceId: string | undefined;
+ getOrCreateInstanceId(pluginDataDir).then((id) => {
+ instanceId = id;
+ // initReporter is guarded by a "already initialised" check, so calling it
+ // here is safe even if the registration-complete call below fires first.
+ initReporter({ enabled: cfg.report.enabled, type: cfg.report.type, logger: api.logger, instanceId: id, pluginVersion });
+ }).catch((err) => {
+ api.logger.warn(`${TAG} Failed to initialize instanceId for metrics: ${err instanceof Error ? err.message : String(err)}`);
+ });
// Unified session/agent filter: combines internal-session detection + user-configured excludeAgents
const sessionFilter = new SessionFilter(cfg.capture.excludeAgents);
if (cfg.capture.excludeAgents.length > 0) {
- api.logger.info(`${TAG} Agent exclude patterns: ${cfg.capture.excludeAgents.join(", ")}`);
+ api.logger.debug?.(`${TAG} Agent exclude patterns: ${cfg.capture.excludeAgents.join(", ")}`);
}
// Daily local JSONL cleaner (L0/L1), enabled only when retentionDays is configured.
@@ -205,13 +197,13 @@ export default function register(api: OpenClawPluginApi) {
logger: api.logger,
});
sharedMemoryCleaner.start();
- api.logger.info(`${TAG} Memory cleaner started (singleton)`);
+ api.logger.debug?.(`${TAG} Memory cleaner started (singleton)`);
} else {
- api.logger.info(`${TAG} Memory cleaner already started in this process, reusing existing instance`);
+ api.logger.debug?.(`${TAG} Memory cleaner already started in this process, reusing existing instance`);
}
memoryCleaner = sharedMemoryCleaner;
} else {
- api.logger.info(`${TAG} Memory cleaner disabled (retentionDays not configured)`);
+ api.logger.debug?.(`${TAG} Memory cleaner disabled (retentionDays not configured)`);
}
// Hardcoded actor ID (legacy, to be removed)
@@ -228,7 +220,7 @@ export default function register(api: OpenClawPluginApi) {
// ============================
// Shared references for tools (populated when extraction scheduler creates them)
- let sharedVectorStore: VectorStore | undefined;
+ let sharedVectorStore: IMemoryStore | undefined;
let sharedEmbeddingService: EmbeddingService | undefined;
/**
@@ -272,12 +264,14 @@ export default function register(api: OpenClawPluginApi) {
};
// tdai_memory_search — Agent-callable L1 memory search tool
+ // TODO: implement hard per-turn call limit via before_tool_call hook + execute early-return (方案 D)
api.registerTool(
{
name: "tdai_memory_search",
label: "Memory Search",
description:
- "Search through the user's long-term memories. Use this when you need to recall specific information about the user's preferences, past events, instructions, or context from previous conversations. Returns relevant memory records ranked by relevance.",
+ "Search through the user's long-term memories. Use this when you need to recall specific information about the user's preferences, past events, instructions, or context from previous conversations. Returns relevant memory records ranked by relevance. " +
+ "Limit: tdai_memory_search and tdai_conversation_search share a combined limit of 3 calls per turn. Stop searching after 3 total attempts.",
parameters: {
type: "object",
properties: {
@@ -367,6 +361,7 @@ export default function register(api: OpenClawPluginApi) {
);
// tdai_conversation_search — Agent-callable L0 conversation search tool
+ // TODO: implement hard per-turn call limit via before_tool_call hook + execute early-return (方案 D)
api.registerTool(
{
name: "tdai_conversation_search",
@@ -375,7 +370,8 @@ export default function register(api: OpenClawPluginApi) {
"Search through past conversation history (raw dialogue records). " +
"Use this when tdai_memory_search (structured memories) doesn't have the information you need, " +
"or when you want to find specific past conversations, dialogue context, or exact words " +
- "the user said before. Returns relevant individual messages ranked by relevance.",
+ "the user said before. Returns relevant individual messages ranked by relevance. " +
+ "Limit: tdai_memory_search and tdai_conversation_search share a combined limit of 3 calls per turn. Stop searching after 3 total attempts.",
parameters: {
type: "object",
properties: {
@@ -464,7 +460,7 @@ export default function register(api: OpenClawPluginApi) {
// (migrated from legacy before_agent_start to before_prompt_build so that
// event.messages is guaranteed to be available — session is already loaded)
if (cfg.recall.enabled) {
- api.logger.info(`${TAG} Registering before_prompt_build hook (auto-recall)`);
+ api.logger.debug?.(`${TAG} Registering before_prompt_build hook (auto-recall)`);
api.on("before_prompt_build", async (event, ctx) => {
const startMs = Date.now();
api.logger.debug?.(`${TAG} [before_prompt_build] Hook triggered`);
@@ -619,343 +615,128 @@ export default function register(api: OpenClawPluginApi) {
scheduler.start({});
}
- // Pre-warm the embedded agent import so the first extraction run doesn't
- // pay the cold-start cost (~35s jiti compile → <50ms with dist/ path).
- prewarmEmbeddedAgent(api.logger);
+ // Pre-warm the embedded agent entrypoint. When runtime already exposes
+ // runEmbeddedPiAgent this becomes a no-op; otherwise it still preloads
+ // the legacy dist bridge to reduce first-run cold start.
+ prewarmEmbeddedAgent(api.logger, api.runtime.agent);
};
if (cfg.extraction.enabled) {
- // === Initialize VectorStore (always) + EmbeddingService (only when embedding enabled) ===
- let vectorStore: VectorStore | undefined;
+ // === Store + scheduler initialization (async, runs eagerly) ===
+ // Wrapped in an async IIFE because register() is synchronous.
+ // initStores() is once-async: the first call creates the store,
+ // subsequent calls (e.g. from seed CLI) reuse the cached result.
+ let vectorStore: IMemoryStore | undefined;
let embeddingService: EmbeddingService | undefined;
- // VectorStore is always created as the metadata store for L0/L1 records.
- // It works as a pure SQLite store even without embedding — keyword search,
- // L0/L1 reads, and pipeline queries all use structured SQL, not vectors.
- try {
- const dims = cfg.embedding.dimensions; // 0 when provider="none" → vec0 tables deferred
- const dbPath = path.join(pluginDataDir, "vectors.db");
- vectorStore = new VectorStore(dbPath, dims, api.logger);
+ const storeReady = (async () => {
+ const stores = await initStores(cfg, pluginDataDir, api.logger);
+ vectorStore = stores.vectorStore;
+ embeddingService = stores.embeddingService;
- // Create EmbeddingService only when embedding is enabled (remote provider configured)
- if (cfg.embedding.enabled) {
+ // Share with tools immediately
+ sharedVectorStore = vectorStore;
+ sharedEmbeddingService = embeddingService;
+
+ // Keep cleaner's SQLite handle updated (singleton cleaner may start earlier).
+ memoryCleaner?.setVectorStore(vectorStore);
+
+ if (vectorStore?.pullProfiles) {
try {
- if (cfg.embedding.provider !== "local" && cfg.embedding.apiKey) {
- // Remote embedding provider (OpenAI-compatible API: OpenAI, Azure, self-hosted, etc.)
- embeddingService = createEmbeddingService({
- provider: cfg.embedding.provider,
- baseUrl: cfg.embedding.baseUrl,
- apiKey: cfg.embedding.apiKey,
- model: cfg.embedding.model,
- dimensions: cfg.embedding.dimensions,
- proxyUrl: cfg.embedding.proxyUrl,
- maxInputChars: cfg.embedding.maxInputChars,
- timeoutMs: cfg.embedding.timeoutMs,
- }, api.logger);
- } else {
- // Local provider (node-llama-cpp) — preserved internally but not reachable from user config
- embeddingService = createEmbeddingService({
- provider: "local",
- modelPath: cfg.embedding.model || undefined,
- modelCacheDir: cfg.embedding.modelCacheDir,
- }, api.logger);
- }
+ await ensureL2L3Local(pluginDataDir, vectorStore, api.logger);
} catch (err) {
- api.logger.warn(
- `${TAG} EmbeddingService init failed, continuing with keyword-only mode: ${err instanceof Error ? err.message : String(err)}`,
- );
- embeddingService = undefined;
+ api.logger.warn(`${TAG} Startup L2/L3 pull failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`);
}
- } else {
- api.logger.info(`${TAG} Embedding disabled by config, VectorStore will serve as metadata-only store`);
}
- // Init VectorStore with provider info (undefined when no embedding → skips provider change detection)
- const providerInfo = embeddingService?.getProviderInfo();
- const initResult = vectorStore.init(providerInfo);
-
- // If VectorStore entered degraded mode (e.g. sqlite-vec load failed),
- // treat it as unavailable and fall back to keyword-only mode.
- if (vectorStore.isDegraded()) {
- api.logger.warn(
- `${TAG} VectorStore is in degraded mode, falling back to keyword dedup`,
- );
- vectorStore = undefined;
- embeddingService = undefined;
- } else {
+ // If embedding provider/model/dimensions changed, re-embed all existing texts
+ if (stores.needsReindex && embeddingService && vectorStore) {
+ const svc = embeddingService;
+ const vs = vectorStore;
api.logger.info(
- `${TAG} VectorStore initialized: ${dbPath} (${dims}D, provider=${cfg.embedding.provider})`,
+ `${TAG} Embedding config changed (${stores.reindexReason}). ` +
+ `Starting background re-embed of all stored texts...`,
);
-
- // If embedding provider/model/dimensions changed, re-embed all existing texts
- if (initResult.needsReindex && embeddingService) {
- const svc = embeddingService; // capture for async closure
- const vs = vectorStore; // capture for async closure
+ vs.reindexAll(
+ (text) => svc.embed(text),
+ (done, total, layer) => {
+ if (done === total || done % 50 === 0) {
+ api.logger.debug?.(`${TAG} Re-embed progress: ${layer} ${done}/${total}`);
+ }
+ },
+ ).then(({ l1Count, l0Count }) => {
api.logger.info(
- `${TAG} Embedding config changed (${initResult.reason}). ` +
- `Starting background re-embed of all stored texts...`,
+ `${TAG} Re-embed complete: L1=${l1Count} records, L0=${l0Count} messages`,
);
- // Run re-embed asynchronously so it doesn't block plugin startup
- vs.reindexAll(
- (text) => svc.embed(text),
- (done, total, layer) => {
- if (done === total || done % 50 === 0) {
- api.logger.debug?.(`${TAG} Re-embed progress: ${layer} ${done}/${total}`);
- }
- },
- ).then(({ l1Count, l0Count }) => {
- api.logger.info(
- `${TAG} Re-embed complete: L1=${l1Count} records, L0=${l0Count} messages`,
- );
- }).catch((err) => {
- api.logger.error(
- `${TAG} Re-embed failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
- );
- });
- }
+ }).catch((err) => {
+ api.logger.error(
+ `${TAG} Re-embed failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ });
}
- } catch (err) {
- api.logger.warn(
- `${TAG} VectorStore init failed; vector/FTS recall and dedup conflict detection will be unavailable: ${err instanceof Error ? err.message : String(err)}`,
- );
- vectorStore = undefined;
- embeddingService = undefined;
- }
+ })();
- // Share vectorStore/embeddingService with tdai_memory_search tool
- sharedVectorStore = vectorStore;
- sharedEmbeddingService = embeddingService;
+ // === Create pipeline manager (sync — does not need store) ===
+ scheduler = createPipelineManager(cfg, api.logger, sessionFilter);
- // Keep cleaner's SQLite handle updated (singleton cleaner may start earlier).
- memoryCleaner?.setVectorStore(vectorStore);
+ // Wire runners after store is ready
+ storeReady.then(() => {
+ // L1 runner via shared factory
+ scheduler!.setL1Runner(createL1Runner({
+ pluginDataDir,
+ cfg,
+ openclawConfig: api.config,
+ vectorStore,
+ embeddingService,
+ logger: api.logger,
+ getInstanceId: () => instanceId,
+ }));
- // === Create pipeline manager ===
- scheduler = new MemoryPipelineManager(
- {
- everyNConversations: cfg.pipeline.everyNConversations,
- enableWarmup: cfg.pipeline.enableWarmup,
- l1: { idleTimeoutSeconds: cfg.pipeline.l1IdleTimeoutSeconds },
- l2: {
- delayAfterL1Seconds: cfg.pipeline.l2DelayAfterL1Seconds,
- minIntervalSeconds: cfg.pipeline.l2MinIntervalSeconds,
- maxIntervalSeconds: cfg.pipeline.l2MaxIntervalSeconds,
- sessionActiveWindowHours: cfg.pipeline.sessionActiveWindowHours,
- },
- },
- api.logger,
- sessionFilter,
- );
+ // Persister via shared factory
+ scheduler!.setPersister(createPersister(pluginDataDir, api.logger));
- // L1 runner: read L0 from DB (primary) or JSONL (fallback) → local LLM extraction → L1 JSONL + VectorStore
- scheduler.setL1Runner(async ({ sessionKey }) => {
- // L1 reads L0 data from VectorStore DB (primary, indexed query).
- // Fallback: read from L0 JSONL files when VectorStore is unavailable.
- if (!api.config) {
- api.logger.debug?.(`${TAG} [pipeline-l1] No OpenClaw config, skipping L1 extraction`);
- return { processedCount: 0 };
- }
-
- const checkpoint = new CheckpointManager(pluginDataDir, api.logger);
- const cp = await checkpoint.read();
- const runnerState = checkpoint.getRunnerState(cp, sessionKey);
-
- api.logger.info(
- `${TAG} [pipeline-l1] Session ${sessionKey}: ` +
- `l1_cursor=${runnerState.last_l1_cursor || "(start)"}`,
- );
-
- try {
- // Read L0 messages since last L1 cursor, grouped by sessionId.
- // Within the same sessionKey, different sessionIds represent different
- // conversation instances (e.g. after /reset). Each group is extracted
- // independently so its sessionId is correctly associated with L1 records.
- //
- // Primary path: read from VectorStore DB (indexed query, fast).
- // Fallback: read from L0 JSONL files (scan + parse, slower).
- let groups: Array<{ sessionId: string; messages: ConversationMessage[] }>;
-
- if (vectorStore && !vectorStore.isDegraded()) {
- // DB path: fast indexed query
- // NOTE: When last_l1_cursor is 0 (first L1 run), we pass undefined
- // to query all messages — but only those captured AFTER plugin start
- // (L0 capture uses pluginStartTimestamp as floor, so DB won't contain
- // pre-existing messages). This is safe because auto-capture already
- // filters out messages older than pluginStartTimestamp.
- const l1Cursor = runnerState.last_l1_cursor > 0
- ? runnerState.last_l1_cursor
- : undefined;
- const dbGroups = vectorStore.queryL0GroupedBySessionId(
- sessionKey,
- l1Cursor,
- );
- // Cast role from string to "user" | "assistant" (DB stores as string)
- groups = dbGroups.map((g) => ({
- sessionId: g.sessionId,
- messages: g.messages.map((m) => ({
- id: m.id,
- role: m.role as "user" | "assistant",
- content: m.content,
- timestamp: m.timestamp,
- })),
- }));
- api.logger.debug?.(
- `${TAG} [pipeline-l1] L0 data source: VectorStore DB`,
- );
- } else {
- // Fallback: JSONL files
- api.logger.debug?.(
- `${TAG} [pipeline-l1] L0 data source: JSONL files (VectorStore unavailable)`,
- );
- const jsonlGroups = await readConversationMessagesGroupedBySessionId(
- sessionKey,
+ // L2 runner: read L1 records (incremental) → SceneExtractor
+ scheduler!.setL2Runner(async (sessionKey: string, cursor?: string) => {
+ try {
+ const l2Runner = createL2Runner({
pluginDataDir,
- runnerState.last_l1_cursor || undefined,
- api.logger,
- 50, // Match DB path limit (queryL0ForL1 default)
- );
- // Convert SessionIdMessageGroup[] to the same shape
- groups = jsonlGroups.map((g) => ({
- sessionId: g.sessionId,
- messages: g.messages,
- }));
- }
-
- if (groups.length === 0) {
- api.logger.debug?.(`${TAG} [pipeline-l1] No new L0 messages for session ${sessionKey}`);
- return { processedCount: 0 };
- }
-
- const totalMessages = groups.reduce((sum, g) => sum + g.messages.length, 0);
- api.logger.info(
- `${TAG} [pipeline-l1] Processing ${totalMessages} L0 messages across ${groups.length} sessionId group(s) for session ${sessionKey}`,
- );
-
- let totalExtracted = 0;
- let totalStored = 0;
- let lastSceneName: string | undefined;
- let maxTimestamp = 0;
-
- for (const group of groups) {
- api.logger.debug?.(
- `${TAG} [pipeline-l1] Group sessionId=${group.sessionId || "(empty)"}: ${group.messages.length} messages`,
- );
-
- const l1Result = await extractL1Memories({
- messages: group.messages,
- sessionKey,
- sessionId: group.sessionId,
- baseDir: pluginDataDir,
- config: api.config,
- options: {
- enableDedup: cfg.extraction.enableDedup,
- maxMemoriesPerSession: cfg.extraction.maxMemoriesPerSession,
- model: cfg.extraction.model,
- previousSceneName: lastSceneName ?? (runnerState.last_scene_name || undefined),
- vectorStore,
- embeddingService,
- conflictRecallTopK: cfg.embedding.conflictRecallTopK,
- },
+ cfg,
+ openclawConfig: api.config,
+ vectorStore,
logger: api.logger,
instanceId,
});
-
- totalExtracted += l1Result.extractedCount;
- totalStored += l1Result.storedCount;
- if (l1Result.lastSceneName) {
- lastSceneName = l1Result.lastSceneName;
- }
-
- const groupMaxTs = Math.max(...group.messages.map((m) => m.timestamp));
- maxTimestamp = Math.max(maxTimestamp, groupMaxTs);
+ return await l2Runner(sessionKey, cursor);
+ } catch (err) {
+ api.logger.error(`${TAG} [pipeline-l2] L2 failed: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
+ throw err;
}
+ });
- // Update checkpoint on disk — cursor is the global max timestamp across all groups
- await checkpoint.markL1ExtractionComplete(
- sessionKey,
- totalStored,
- maxTimestamp,
- lastSceneName,
- );
-
- api.logger.info(
- `${TAG} [pipeline-l1] L1 complete: extracted=${totalExtracted}, stored=${totalStored} (${groups.length} group(s))`,
- );
-
- return { processedCount: totalMessages };
- } catch (err) {
- api.logger.error(`${TAG} [pipeline-l1] L1 failed: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
- // ── error_degradation metric ──
- if (instanceId) {
- report("error_degradation", {
- module: "l1-extraction",
- action: "extractL1Memories",
- errorType: "exception",
- errorMessage: err instanceof Error ? err.message : String(err),
- degradedTo: null,
- impact: "blocking",
+ // L3 runner: persona trigger + generation
+ scheduler!.setL3Runner(async () => {
+ try {
+ const l3Runner = createL3Runner({
+ pluginDataDir,
+ cfg,
+ openclawConfig: api.config,
+ vectorStore,
+ logger: api.logger,
+ instanceId,
});
+ await l3Runner();
+ } catch (err) {
+ api.logger.error(`${TAG} [pipeline-l3] Failed: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
}
- throw err; // rethrow so pipeline-manager can retry
- }
+ });
+ }).catch((err) => {
+ api.logger.error(
+ `${TAG} Store init failed; vector/FTS recall and dedup will be unavailable: ${err instanceof Error ? err.message : String(err)}`,
+ );
});
- // Persister: saves pipeline session states to checkpoint
- scheduler.setPersister(async (states) => {
- const checkpoint = new CheckpointManager(pluginDataDir, api.logger);
- await checkpoint.mergePipelineStates(states);
- });
-
- // L2 runner: read L1 records (incremental) → SceneExtractor
- scheduler.setL2Runner(async (sessionKey: string, cursor?: string) => {
- try {
- return await runLocalL2Extraction(api, cfg, pluginDataDir, sessionKey, vectorStore, cursor, instanceId);
- } catch (err) {
- api.logger.error(`${TAG} [pipeline-l2] L2 failed: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
- throw err; // rethrow so pipeline-manager can handle retry/fallback (consistent with L1 runner)
- }
- });
-
- // L3 runner: persona trigger + generation
- scheduler.setL3Runner(async () => {
- try {
- const trigger = new PersonaTrigger({
- dataDir: pluginDataDir,
- interval: cfg.persona.triggerEveryN,
- logger: api.logger,
- });
-
- const { should, reason } = await trigger.shouldGenerate();
- if (!should) {
- api.logger.debug?.(`${TAG} [pipeline-l3] Persona generation not needed`);
- return;
- }
-
- if (!api.config) {
- api.logger.warn(`${TAG} [pipeline-l3] No OpenClaw config, skipping persona generation`);
- return;
- }
-
- api.logger.info(`${TAG} [pipeline-l3] Starting persona generation: ${reason}`);
- const generator = new PersonaGenerator({
- dataDir: pluginDataDir,
- config: api.config,
- model: cfg.persona.model,
- backupCount: cfg.persona.backupCount,
- logger: api.logger,
- instanceId,
- });
- const genResult = await generator.generate(reason);
- api.logger.info(`${TAG} [pipeline-l3] Persona generation ${genResult ? "succeeded" : "skipped (no changes)"}`);
- } catch (err) {
- api.logger.error(`${TAG} [pipeline-l3] Failed: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
- }
- });
-
- // Capture vectorStore reference for cleanup
- const vectorStoreRef = vectorStore;
-
// Register a SINGLE gateway_stop hook for ordered shutdown.
- // Order: memoryCleaner → scheduler → vectorStore → embeddingService
+ // Order: memoryCleaner → scheduler → vectorStore → embeddingService → resetStores
// (memoryCleaner may use VectorStore during cleanup, so it must stop first)
//
// The entire hook is wrapped with a 3 s timeout to guarantee we never
@@ -965,8 +746,11 @@ export default function register(api: OpenClawPluginApi) {
const GATEWAY_STOP_TIMEOUT_MS = 3_000;
const hookStartMs = Date.now();
+ // Ensure store init has completed before tearing down
+ await storeReady.catch(() => {});
+
const doCleanup = async (): Promise => {
- // 1. Stop the memory cleaner
+ // 1. Stop the memory cleaner first (it may be running deleteL1ExpiredByUpdatedTime)
if (memoryCleaner) {
try {
memoryCleaner.destroy();
@@ -983,16 +767,20 @@ export default function register(api: OpenClawPluginApi) {
const t = Date.now();
await scheduler.destroy();
api.logger.info(`${TAG} [gateway_stop] Scheduler destroyed (${Date.now() - t}ms)`);
+ } else {
+ api.logger.info(`${TAG} [gateway_stop] Scheduler was never started, skipping destroy`);
}
- // 3. Close VectorStore
- if (vectorStoreRef) {
- vectorStoreRef.close();
+ // 3. Close VectorStore last (after all consumers are done)
+ if (vectorStore) {
+ api.logger.info(`${TAG} [gateway_stop] Closing VectorStore`);
+ vectorStore.close();
}
- // 4. Release embedding service resources
+ // 4. Release embedding service resources (model memory, GPU, etc.)
if (embeddingService?.close) {
try {
+ api.logger.info(`${TAG} [gateway_stop] Closing EmbeddingService`);
await embeddingService.close();
} catch (err) {
api.logger.warn(`${TAG} [gateway_stop] EmbeddingService close error: ${err instanceof Error ? err.message : String(err)}`);
@@ -1021,11 +809,14 @@ export default function register(api: OpenClawPluginApi) {
if (timeoutId !== undefined) clearTimeout(timeoutId);
}
+ // 5. Reset store singleton cache so hot-restart can re-initialize
+ resetStores();
+
api.logger.info(`${TAG} [gateway_stop] Cleanup finished, all resources released (${Date.now() - hookStartMs}ms)`);
});
}
- api.logger.info(`${TAG} Registering agent_end hook (auto-capture)`);
+ api.logger.debug?.(`${TAG} Registering agent_end hook (auto-capture)`);
api.on("agent_end", async (event, ctx) => {
const startMs = Date.now();
api.logger.debug?.(`${TAG} [agent_end] Hook triggered`);
@@ -1139,7 +930,7 @@ export default function register(api: OpenClawPluginApi) {
}
});
} else {
- api.logger.info(`${TAG} Auto-capture disabled`);
+ api.logger.debug?.(`${TAG} Auto-capture disabled`);
}
// memoryCleaner gateway_stop is handled in the unified handler above (inside extraction.enabled block).
@@ -1159,177 +950,30 @@ export default function register(api: OpenClawPluginApi) {
});
}
- api.logger.info(
+ // ============================
+ // CLI registration
+ // ============================
+
+ api.registerCli(
+ ({ program, config, logger: cliLogger }) => {
+ const memoryTdai = program
+ .command("memory-tdai")
+ .description("memory-tdai plugin commands (seed, query, stats)");
+
+ registerMemoryTdaiCli(memoryTdai, {
+ config,
+ pluginConfig: api.pluginConfig,
+ stateDir: api.runtime.state.resolveStateDir(),
+ logger: cliLogger,
+ });
+ },
+ { commands: ["memory-tdai"] },
+ );
+
+ api.logger.debug?.(
`${TAG} Plugin registration complete (v3). ` +
`startTimestamp=${pluginStartTimestamp} (${new Date(pluginStartTimestamp).toISOString()})`,
);
-
- // Resolve persistent instance ID for metric reporting (used by agent_turn, error_degradation, etc.)
- getOrCreateInstanceId(pluginDataDir).then((id) => {
- instanceId = id;
- initReporter({ enabled: cfg.report.enabled, type: cfg.report.type, logger: api.logger, instanceId: id, pluginVersion });
- }).catch((err) => {
- api.logger.warn(`${TAG} Failed to initialize instanceId for metrics: ${err instanceof Error ? err.message : String(err)}`);
- });
-}
-
-// ============================
-// L2 extraction implementations
-// ============================
-
-/**
- * Local L2 extraction: read L1 records → SceneExtractor.
- *
- * Uses **incremental** reads when VectorStore is available:
- * 1. Receive the pipeline cursor (`last_extraction_updated_time`) from pipeline-manager
- * 2. Query only L1 records updated AFTER that cursor via `queryMemoryRecords`
- * 3. Return the latest `updatedAt` from the batch so pipeline-manager can advance the cursor
- *
- * Falls back to JSONL read (with client-side time filtering) when VectorStore is unavailable.
- */
-async function runLocalL2Extraction(
- api: OpenClawPluginApi,
- cfg: MemoryTdaiConfig,
- pluginDataDir: string,
- sessionKey: string,
- vectorStore?: VectorStore,
- updatedAfter?: string,
- instanceId?: string,
-): Promise<{ latestCursor?: string } | void> {
- api.logger.debug?.(
- `${TAG} [L2-local] session=${sessionKey}, updatedAfter=${updatedAfter ?? "(full)"}`,
- );
-
- let records: Array<{ content: string; created_at: string; id: string; updatedAt: string }>;
-
- // Prefer incremental SQLite query when VectorStore is available
- if (vectorStore && !vectorStore.isDegraded()) {
- const { queryMemoryRecords } = await import("./src/record/l1-reader.js");
- const memRecords = queryMemoryRecords(vectorStore, {
- sessionKey,
- updatedAfter,
- }, api.logger);
-
- if (memRecords.length === 0) {
- api.logger.debug?.(
- `${TAG} [L2-local] No new L1 records since cursor (session=${sessionKey}, updatedAfter=${updatedAfter ?? "(full)"}), skipping scene extraction`,
- );
- return;
- }
-
- api.logger.debug?.(
- `${TAG} [L2-local] Incremental query returned ${memRecords.length} record(s) (session=${sessionKey})`,
- );
-
- records = memRecords.map((r) => ({
- content: r.content,
- created_at: r.createdAt,
- id: r.id,
- updatedAt: r.updatedAt,
- }));
- } else {
- // Fallback: read JSONL files with client-side time filtering
- api.logger.debug?.(`${TAG} [L2-local] VectorStore unavailable, falling back to JSONL read`);
- const { readAllMemoryRecords } = await import("./src/record/l1-reader.js");
- let allRecords = await readAllMemoryRecords(pluginDataDir, api.logger);
-
- // Apply updatedAfter filter on JSONL records (same semantics as SQLite path)
- if (updatedAfter) {
- const beforeCount = allRecords.length;
- allRecords = allRecords.filter((r) => {
- const t = r.updatedAt || r.createdAt || "";
- return t > updatedAfter;
- });
- api.logger.debug?.(
- `${TAG} [L2-local] JSONL time filter: ${beforeCount} → ${allRecords.length} record(s) (updatedAfter=${updatedAfter})`,
- );
- }
-
- if (allRecords.length === 0) {
- api.logger.debug?.(`${TAG} [L2-local] No new L1 records found (JSONL fallback), skipping scene extraction`);
- return;
- }
-
- records = allRecords.map((r) => ({
- content: r.content,
- created_at: r.createdAt,
- id: r.id,
- updatedAt: r.updatedAt,
- }));
- }
-
- const extractor = new SceneExtractor({
- dataDir: pluginDataDir,
- config: api.config!,
- model: cfg.persona.model,
- maxScenes: cfg.persona.maxScenes,
- sceneBackupCount: cfg.persona.sceneBackupCount,
- logger: api.logger,
- instanceId,
- });
-
- const memories = records.map((r) => ({
- content: r.content,
- created_at: r.created_at,
- id: r.id,
- }));
-
- // ── Checkpoint guard ──────────────────────────────────────────────
- // Snapshot critical counters BEFORE the LLM agent runs.
- // The LLM operates on checkpoint via raw file tools (write_to_file /
- // replace_in_file) which bypass CheckpointManager's file lock.
- // If the LLM accidentally overwrites the entire checkpoint (e.g. via
- // write_to_file), system-managed counters like scenes_processed and
- // memories_since_last_persona can be reset to stale values.
- // After extraction we detect and repair such corruption.
- const preCheckpoint = new CheckpointManager(pluginDataDir, api.logger);
- const preState = await preCheckpoint.read();
- const preScenesProcessed = preState.scenes_processed;
- const preMemoriesSince = preState.memories_since_last_persona;
- const preTotalProcessed = preState.total_processed;
-
- const extractResult = await extractor.extract(memories);
- if (extractResult.success && extractResult.memoriesProcessed > 0) {
- const checkpoint = new CheckpointManager(pluginDataDir, api.logger);
-
- // Detect and repair LLM-caused checkpoint corruption.
- // If the LLM wrote the entire checkpoint file (instead of using
- // replace_in_file on specific fields), system-managed counters may
- // have been overwritten with stale/zero values.
- const postState = await checkpoint.read();
- if (
- postState.scenes_processed < preScenesProcessed ||
- postState.total_processed < preTotalProcessed
- ) {
- api.logger.warn(
- `${TAG} [L2-local] ⚠️ Checkpoint corruption detected! ` +
- `scenes_processed: ${preScenesProcessed} → ${postState.scenes_processed}, ` +
- `total_processed: ${preTotalProcessed} → ${postState.total_processed}, ` +
- `memories_since: ${preMemoriesSince} → ${postState.memories_since_last_persona}. ` +
- `Repairing...`,
- );
- await checkpoint.write({
- ...postState,
- scenes_processed: Math.max(postState.scenes_processed, preScenesProcessed),
- total_processed: Math.max(postState.total_processed, preTotalProcessed),
- memories_since_last_persona: Math.max(postState.memories_since_last_persona, preMemoriesSince),
- });
- api.logger.info(`${TAG} [L2-local] Checkpoint repaired`);
- }
-
- await checkpoint.incrementScenesProcessed();
-
- // Return the max updatedAt from this batch as the new cursor
- const latestCursor = records.reduce((latest, r) => {
- return r.updatedAt > latest ? r.updatedAt : latest;
- }, "");
-
- api.logger.debug?.(
- `${TAG} [L2-local] Extraction complete: processed=${extractResult.memoriesProcessed}, latestCursor=${latestCursor}`,
- );
-
- return { latestCursor: latestCursor || undefined };
- }
}
// ============================
diff --git a/openclaw.plugin.json b/openclaw.plugin.json
index 200f714..e3fcc53 100644
--- a/openclaw.plugin.json
+++ b/openclaw.plugin.json
@@ -6,6 +6,12 @@
"type": "object",
"additionalProperties": true,
"properties": {
+ "storeBackend": {
+ "type": "string",
+ "enum": ["sqlite", "tcvdb"],
+ "default": "sqlite",
+ "description": "存储后端：sqlite（本地 SQLite + sqlite-vec）或 tcvdb（腾讯云向量数据库）"
+ },
"capture": {
"type": "object",
"description": "对话自动捕获设置 (L0)",
@@ -32,7 +38,7 @@
"description": "场景归纳与用户画像设置 (L2/L3)",
"properties": {
"triggerEveryN": { "type": "number", "default": 50, "description": "每 N 条新记忆触发画像生成" },
- "maxScenes": { "type": "number", "default": 20, "description": "最大场景数" },
+ "maxScenes": { "type": "number", "default": 15, "description": "最大场景数" },
"backupCount": { "type": "number", "default": 3, "description": "画像备份保留数量" },
"sceneBackupCount": { "type": "number", "default": 10, "description": "场景块备份保留数量" },
"model": { "type": "string", "description": "画像生成模型 (格式: provider/model)，未填写时使用openclaw默认模型" }
@@ -66,7 +72,7 @@
"type": "object",
"description": "向量搜索 (Embedding) 配置",
"properties": {
- "enabled": { "type": "boolean", "default": true, "description": "是否启用向量搜索" },
+ "enabled": { "type": "boolean", "default": true, "description": "是否启用向量搜索（若 provider=none，则实际会被禁用）" },
"provider": { "type": "string", "default": "none", "description": "Embedding 服务提供者：填写兼容 OpenAI API 的远端服务名称（如 openai、deepseek 等）；不填或填 none 则禁用向量搜索" },
"proxyUrl": { "type": "string", "description": "本地代理地址（仅 provider=qclaw 时必填）。配置后 embedding 请求将通过该代理转发，原始 baseUrl 作为 Remote-URL 头传递" },
"baseUrl": { "type": "string", "description": "API Base URL（必填）：填写对应 provider 的 API 地址" },
@@ -74,7 +80,30 @@
"model": { "type": "string", "description": "模型名称（必填）" },
"dimensions": { "type": "number", "description": "向量维度（必填，需与所选模型匹配）" },
"conflictRecallTopK": { "type": "number", "default": 5, "description": "冲突检测时召回 Top-K 数" },
- "maxInputChars": { "type": "number", "default": 5000, "description": "Embedding 输入文本最大字符数，超出时截断并打印警告日志（默认 5000，适合大多数模型的 token 上限）" }
+ "maxInputChars": { "type": "number", "default": 5000, "description": "Embedding 输入文本最大字符数，超出时截断并打印警告日志（默认 5000，适合大多数模型的 token 上限）" },
+ "timeoutMs": { "type": "number", "default": 10000, "description": "单次 embedding API 调用超时（毫秒），超时后该次请求中止且不重试" }
+ }
+ },
+ "tcvdb": {
+ "type": "object",
+ "description": "腾讯云向量数据库配置（仅 storeBackend=tcvdb 时生效）",
+ "properties": {
+ "url": { "type": "string", "description": "实例 URL（必填，如 http://10.0.1.1:8100）" },
+ "username": { "type": "string", "default": "root", "description": "账户名" },
+ "apiKey": { "type": "string", "description": "API Key（必填）" },
+ "database": { "type": "string", "description": "数据库名（未填写时默认根据实例 ID 自动生成）" },
+ "alias": { "type": "string", "description": "用户友好别名（可选，用于 database.json 中识别）" },
+ "embeddingModel": { "type": "string", "default": "bge-large-zh", "description": "服务端 embedding 模型" },
+ "timeout": { "type": "number", "default": 10000, "description": "请求超时（毫秒）" },
+ "caPemPath": { "type": "string", "description": "CA 证书 PEM 文件路径（HTTPS 连接时使用）" }
+ }
+ },
+ "bm25": {
+ "type": "object",
+ "description": "BM25 稀疏向量编码设置（主要用于 tcvdb 后端）",
+ "properties": {
+ "enabled": { "type": "boolean", "default": true, "description": "是否启用 BM25 稀疏向量编码" },
+ "language": { "type": "string", "enum": ["zh", "en"], "default": "zh", "description": "分词语言：zh（中文）或 en（英文）" }
}
},
"report": {
diff --git a/package.json b/package.json
index ba22a31..e3596de 100644
--- a/package.json
+++ b/package.json
@@ -1,14 +1,33 @@
{
"name": "@tencentdb-agent-memory/memory-tencentdb",
- "version": "0.1.4",
+ "version": "0.2.2",
"description": "Four-layer local memory system plugin for OpenClaw — auto-captures, structures, and profiles conversational knowledge using local LLM + SQLite vector search (L0→L1→L2→L3 pipeline)",
"type": "module",
"main": "index.ts",
+ "bin": {
+ "migrate-sqlite-to-tcvdb": "./bin/migrate-sqlite-to-tcvdb.mjs",
+ "export-tencent-vdb": "./bin/export-tencent-vdb.mjs",
+ "read-local-memory": "./bin/read-local-memory.mjs"
+ },
"exports": {
".": "./index.ts"
},
+ "scripts": {
+ "build:scripts": "npm run build:migrate-sqlite-to-vdb && npm run build:export-tencent-vdb && npm run build:read-local-memory",
+ "prepack": "npm run build:scripts",
+ "build:migrate-sqlite-to-vdb": "tsc -p scripts/migrate-sqlite-to-tcvdb/tsconfig.json --noEmitOnError false",
+ "migrate-sqlite-to-tcvdb": "node ./bin/migrate-sqlite-to-tcvdb.mjs",
+ "build:export-tencent-vdb": "tsc --project scripts/export-tencent-vdb/tsconfig.json",
+ "export-tencent-vdb": "node ./bin/export-tencent-vdb.mjs",
+ "build:read-local-memory": "tsc --project scripts/read-local-memory/tsconfig.json",
+ "read-local-memory": "node ./bin/read-local-memory.mjs"
+ },
"files": [
+ "bin/",
"index.ts",
+ "scripts/migrate-sqlite-to-tcvdb/dist/",
+ "scripts/export-tencent-vdb/dist/",
+ "scripts/read-local-memory/dist/",
"src/",
"openclaw.plugin.json",
"README.md",
@@ -39,11 +58,14 @@
},
"dependencies": {
"@node-rs/jieba": "^2.0.1",
- "sqlite-vec": "0.1.7-alpha.2"
+ "@tencentdb-agent-memory/tcvdb-text": "^0.1.1",
+ "json5": "^2.2.3",
+ "sqlite-vec": "0.1.7-alpha.2",
+ "undici": "8.0.2"
},
"peerDependencies": {
- "openclaw": ">=2026.3.7",
- "node-llama-cpp": "^3.16.2"
+ "node-llama-cpp": "^3.16.2",
+ "openclaw": ">=2026.3.7"
},
"peerDependenciesMeta": {
"openclaw": {
@@ -56,6 +78,17 @@
"openclaw": {
"extensions": [
"./index.ts"
- ]
+ ],
+ "compat": {
+ "pluginApi": ">=2026.3.13",
+ "minGatewayVersion": ">=2026.3.13"
+ },
+ "build": {
+ "openclawVersion": "2026.3.13",
+ "pluginSdkVersion": "2026.3.13"
+ },
+ "bundle": {
+ "stageRuntimeDependencies": true
+ }
}
}
diff --git a/scripts/export-diagnostic.sh b/scripts/export-diagnostic.sh
new file mode 100755
index 0000000..9103e00
--- /dev/null
+++ b/scripts/export-diagnostic.sh
@@ -0,0 +1,227 @@
+#!/usr/bin/env bash
+# OpenClaw + memory-tencentdb（原 memory-tdai）诊断数据导出脚本
+# 注：插件已更名为 memory-tencentdb，但数据目录始终为 memory-tdai（代码硬编码）
+# 用法: bash export-diagnostic.sh [输出目录]
+# 默认输出到 ~/Downloads/openclaw-diagnostic-/
+
+set -euo pipefail
+
+# ── 参数 ──
+OUTPUT_BASE="${1:-$HOME/Downloads}"
+TIMESTAMP=$(date +%Y%m%d-%H%M%S)
+EXPORT_DIR="${OUTPUT_BASE}/openclaw-diagnostic-${TIMESTAMP}"
+ARCHIVE_PATH="${EXPORT_DIR}.tar.gz"
+
+# ── OpenClaw 工作目录探测 ──
+if [ -n "${OPENCLAW_STATE_DIR:-}" ]; then
+ STATE_DIR="$OPENCLAW_STATE_DIR"
+elif [ -d "$HOME/.openclaw" ]; then
+ STATE_DIR="$HOME/.openclaw"
+elif [ -d "$HOME/.clawdbot" ]; then
+ STATE_DIR="$HOME/.clawdbot"
+else
+ echo "❌ 未找到 OpenClaw 工作目录 (~/.openclaw 或 ~/.clawdbot)"
+ exit 1
+fi
+
+echo "📂 OpenClaw 工作目录: $STATE_DIR"
+echo "📦 导出目录: $EXPORT_DIR"
+
+mkdir -p "$EXPORT_DIR"
+
+# ── 1. 收集环境信息 ──
+echo "🔍 收集环境信息..."
+{
+ echo "=== 导出时间 ==="
+ date -Iseconds 2>/dev/null || date
+ echo ""
+ echo "=== 系统信息 ==="
+ echo "OS: $(uname -a)"
+ echo "Node: $(node --version 2>/dev/null || echo 'not found')"
+ echo "pnpm: $(pnpm --version 2>/dev/null || echo 'not found')"
+ echo ""
+ echo "=== OpenClaw 版本 ==="
+ openclaw --version 2>/dev/null || pnpm openclaw --version 2>/dev/null || echo "(unknown)"
+ echo ""
+ echo "=== 工作目录 ==="
+ echo "STATE_DIR: $STATE_DIR"
+ echo ""
+ echo "=== 目录结构 ==="
+ ls -la "$STATE_DIR/" 2>/dev/null || echo "(empty)"
+ echo ""
+ echo "=== memory-tdai 目录结构 ==="
+ ls -laR "$STATE_DIR/memory-tdai/" 2>/dev/null || echo "(not found)"
+ echo ""
+ echo "=== 磁盘占用 ==="
+ du -sh "$STATE_DIR/memory-tdai/"* 2>/dev/null || echo "(not found)"
+} > "$EXPORT_DIR/env-info.txt" 2>&1
+
+# ── 2. 收集 OpenClaw 日志 ──
+echo "📋 收集 OpenClaw 日志..."
+mkdir -p "$EXPORT_DIR/logs"
+
+# 网关日志 (~/.openclaw/logs/)
+if [ -d "$STATE_DIR/logs" ]; then
+ cp -r "$STATE_DIR/logs/" "$EXPORT_DIR/logs/gateway-logs/" 2>/dev/null || true
+fi
+
+# 滚动日志 (/tmp/openclaw/)
+TMP_LOG_DIR="/tmp/openclaw"
+if [ -d "$TMP_LOG_DIR" ]; then
+ mkdir -p "$EXPORT_DIR/logs/rolling-logs"
+ # 只取最近 3 个日志文件
+ ls -t "$TMP_LOG_DIR"/openclaw-*.log 2>/dev/null | head -3 | while read -r f; do
+ # 每个文件只取最后 5000 行，避免过大
+ tail -5000 "$f" > "$EXPORT_DIR/logs/rolling-logs/$(basename "$f")" 2>/dev/null || true
+ done
+fi
+
+# ── 3. 收集记忆插件数据 ──
+# 注：数据目录名为 memory-tdai（历史原因，插件更名为 memory-tencentdb 后未改目录名）
+echo "🧠 收集记忆插件数据..."
+MEMORY_DIR="$STATE_DIR/memory-tdai"
+if [ -d "$MEMORY_DIR" ]; then
+ mkdir -p "$EXPORT_DIR/memory-tdai"
+
+ # L0 对话记录 (JSONL)
+ if [ -d "$MEMORY_DIR/conversations" ]; then
+ cp -r "$MEMORY_DIR/conversations/" "$EXPORT_DIR/memory-tdai/conversations/" 2>/dev/null || true
+ fi
+
+ # L1 结构化记忆 (JSONL)
+ if [ -d "$MEMORY_DIR/records" ]; then
+ cp -r "$MEMORY_DIR/records/" "$EXPORT_DIR/memory-tdai/records/" 2>/dev/null || true
+ fi
+
+ # L2 场景文件 (Markdown)
+ if [ -d "$MEMORY_DIR/scene_blocks" ]; then
+ cp -r "$MEMORY_DIR/scene_blocks/" "$EXPORT_DIR/memory-tdai/scene_blocks/" 2>/dev/null || true
+ fi
+
+ # L3 用户画像
+ [ -f "$MEMORY_DIR/persona.md" ] && cp "$MEMORY_DIR/persona.md" "$EXPORT_DIR/memory-tdai/" 2>/dev/null || true
+
+ # checkpoint + scene_index
+ if [ -d "$MEMORY_DIR/.metadata" ]; then
+ cp -r "$MEMORY_DIR/.metadata/" "$EXPORT_DIR/memory-tdai/.metadata/" 2>/dev/null || true
+ fi
+
+ # SQLite 数据库（用于检查向量/FTS索引状态）
+ [ -f "$MEMORY_DIR/vectors.db" ] && cp "$MEMORY_DIR/vectors.db" "$EXPORT_DIR/memory-tdai/" 2>/dev/null || true
+
+ # 备份目录（可选，可能较大）
+ if [ -d "$MEMORY_DIR/.backup" ]; then
+ cp -r "$MEMORY_DIR/.backup/" "$EXPORT_DIR/memory-tdai/.backup/" 2>/dev/null || true
+ fi
+else
+ echo " ⚠️ 未找到 memory-tdai 数据目录（memory-tencentdb 插件的数据也存储在此目录）"
+fi
+
+# ── 4. 收集 OpenClaw 配置（脱敏） ──
+echo "🔧 收集 OpenClaw 配置（已脱敏）..."
+CONFIG_FILE="$STATE_DIR/openclaw.json"
+if [ -f "$CONFIG_FILE" ]; then
+ # 使用 node 脱敏处理配置
+ node -e "
+ const fs = require('fs');
+ const JSON5 = (() => { try { return require('json5'); } catch { return JSON; } })();
+ const raw = fs.readFileSync('$CONFIG_FILE', 'utf-8');
+ let cfg;
+ try { cfg = JSON5.parse(raw); } catch { cfg = JSON.parse(raw); }
+
+ // 递归脱敏函数
+ function redact(obj, path) {
+ if (!obj || typeof obj !== 'object') return obj;
+ if (Array.isArray(obj)) return obj.map((v, i) => redact(v, path + '[' + i + ']'));
+ const result = {};
+ for (const [k, v] of Object.entries(obj)) {
+ const fullPath = path ? path + '.' + k : k;
+ // 脱敏规则：API key、token、password、secret 类字段
+ if (/api_?key|token|password|secret|credential/i.test(k) && typeof v === 'string') {
+ result[k] = v.length > 0 ? '***REDACTED(' + v.length + 'chars)***' : '';
+ }
+ // 脱敏 SecretRef 对象
+ else if (v && typeof v === 'object' && v.source && v.id && v.provider) {
+ result[k] = { source: v.source, provider: v.provider, id: '***REDACTED***' };
+ }
+ // 整体跳过的顶层敏感块
+ else if (['models', 'secrets', 'channels', 'env'].includes(k) && !path) {
+ result[k] = '***REDACTED_SECTION(use openclaw config get ' + k + ' to inspect)***';
+ }
+ // gateway.auth 内的 token/password
+ else if (path === 'gateway.auth' && /token|password/i.test(k)) {
+ result[k] = typeof v === 'string' ? '***REDACTED***' : v;
+ }
+ else {
+ result[k] = redact(v, fullPath);
+ }
+ }
+ return result;
+ }
+
+ const redacted = redact(cfg, '');
+ // plugins 已经过递归 redact()，其中 apiKey/token/password/secret 等字段
+ // 会被自动脱敏，同时保留 provider/model/enabled 等排查所需的非敏感配置
+
+ fs.writeFileSync('$EXPORT_DIR/openclaw-config-redacted.json', JSON.stringify(redacted, null, 2));
+ console.log(' ✅ 配置已脱敏导出');
+ " 2>&1 || {
+ echo " ⚠️ Node 脱敏失败，使用 grep 粗略脱敏"
+ # 粗略脱敏：删除包含敏感关键字的行
+ grep -v -iE '(api.?key|token|password|secret|credential).*:.*"[^"]{8,}"' "$CONFIG_FILE" \
+ | sed -E 's/"(models|secrets|channels|env)"\s*:\s*\{[^}]*\}/"__REDACTED_SECTION__"/g' \
+ > "$EXPORT_DIR/openclaw-config-redacted.json" 2>/dev/null || true
+ }
+else
+ echo " ⚠️ 未找到配置文件"
+fi
+
+# ── 5. 收集插件安装信息 ──
+echo "🔌 收集插件安装信息..."
+if [ -d "$STATE_DIR/extensions" ]; then
+ {
+ echo "=== 已安装插件 ==="
+ ls -la "$STATE_DIR/extensions/" 2>/dev/null
+ echo ""
+ for ext_dir in "$STATE_DIR/extensions"/*/; do
+ [ -d "$ext_dir" ] || continue
+ pkg="$ext_dir/node_modules/openclaw/package.json"
+ plugin_pkg="$ext_dir/package.json"
+ echo "--- $(basename "$ext_dir") ---"
+ if [ -f "$plugin_pkg" ]; then
+ node -e "const p=require('$plugin_pkg'); console.log('name:', p.name, 'version:', p.version)" 2>/dev/null || true
+ fi
+ done
+ } > "$EXPORT_DIR/plugins-info.txt" 2>&1
+fi
+
+# ── 6. 打包 ──
+echo "📦 打包中..."
+cd "$(dirname "$EXPORT_DIR")"
+tar -czf "$ARCHIVE_PATH" "$(basename "$EXPORT_DIR")"
+
+# 计算大小
+ARCHIVE_SIZE=$(du -sh "$ARCHIVE_PATH" | cut -f1)
+
+echo ""
+echo "═══════════════════════════════════════════════════"
+echo " ✅ 诊断数据导出完成"
+echo "═══════════════════════════════════════════════════"
+echo ""
+echo " 📦 压缩包: $ARCHIVE_PATH"
+echo " 📏 大小: $ARCHIVE_SIZE"
+echo ""
+echo " 包含内容:"
+echo " - env-info.txt — 环境信息、目录结构"
+echo " - logs/ — OpenClaw 网关日志 + 滚动日志"
+echo " - memory-tdai/ — 记忆插件全量数据 (L0~L3 + SQLite)"
+echo " - openclaw-config-redacted.json — 脱敏后的配置文件"
+echo " - plugins-info.txt — 插件安装信息"
+echo ""
+echo " ⚠️ 安全提示:"
+echo " - 配置文件已自动脱敏（API Key、Token、Password 等已移除）"
+echo " - models/secrets/channels/env 等敏感配置块已整体替换"
+echo " - 记忆数据中可能包含用户对话内容，请确认后再发送"
+echo ""
+echo " 📤 请手动检查后发送给研发团队"
+echo "═══════════════════════════════════════════════════"
diff --git a/src/config.ts b/src/config.ts
index 532abf2..7d9c473 100644
--- a/src/config.ts
+++ b/src/config.ts
@@ -127,6 +127,37 @@ export interface MemoryCleanupConfig {
cleanTime: string;
}
+/** BM25 sparse vector encoding configuration (local @tencentdb-agent-memory/tcvdb-text). */
+export interface BM25Config {
+ /** Whether BM25 sparse encoding is enabled (default: true) */
+ enabled: boolean;
+ /** Language for BM25 pre-trained params: "zh" or "en" (default: "zh") */
+ language: "zh" | "en";
+}
+
+/** Tencent Cloud VectorDB configuration. */
+export interface TcvdbConfig {
+ /** Instance URL (e.g. "http://10.0.1.1:80" or external domain) */
+ url: string;
+ /** Account name (default: "root") */
+ username: string;
+ /** API Key */
+ apiKey: string;
+ /** Database name (auto-generated from instance_id if empty) */
+ database: string;
+ /** User-friendly alias for this database (optional, for identification in database.json) */
+ alias: string;
+ /** Built-in embedding model (default: "bge-large-zh") */
+ embeddingModel: string;
+ /** Request timeout in ms (default: 10000) */
+ timeout: number;
+ /** Path to CA certificate PEM file (for HTTPS connections) */
+ caPemPath?: string;
+}
+
+/** Storage backend type. */
+export type StoreBackend = "sqlite" | "tcvdb";
+
/** Report settings — controls metric/event reporting. */
export interface ReportConfig {
/** Enable reporting (default: true) */
@@ -143,6 +174,13 @@ export interface MemoryTdaiConfig {
pipeline: PipelineTriggerConfig;
recall: RecallConfig;
embedding: EmbeddingConfig;
+ /** Storage backend: "sqlite" (default) or "tcvdb" */
+ storeBackend: StoreBackend;
+ /** Tencent Cloud VectorDB configuration (required when storeBackend = "tcvdb") */
+ tcvdb: TcvdbConfig;
+ /** BM25 sparse vector encoding (local @tencentdb-agent-memory/tcvdb-text) */
+ bm25: BM25Config;
+ /** Local JSONL cleanup settings */
memoryCleanup: MemoryCleanupConfig;
report: ReportConfig;
}
@@ -272,6 +310,16 @@ export function parseConfig(raw: Record | undefined): MemoryTda
const cleanTime = normalizeCleanTime(str(captureGroup, "cleanTime")) ?? "03:00";
+ // --- BM25 (local @tencentdb-agent-memory/tcvdb-text encoder) ---
+ const bm25Group = obj(c, "bm25");
+
+ // --- Store backend ---
+ const storeBackendRaw = str(c, "storeBackend") ?? "sqlite";
+ const storeBackend: StoreBackend = storeBackendRaw === "tcvdb" ? "tcvdb" : "sqlite";
+
+ // --- TCVDB config ---
+ const tcvdbGroup = obj(c, "tcvdb");
+
const memoryCleanup: MemoryCleanupConfig = {
retentionDays,
enabled: retentionDays != null,
@@ -293,7 +341,7 @@ export function parseConfig(raw: Record | undefined): MemoryTda
},
persona: {
triggerEveryN: num(personaGroup, "triggerEveryN") ?? 50,
- maxScenes: num(personaGroup, "maxScenes") ?? 20,
+ maxScenes: num(personaGroup, "maxScenes") ?? 15,
backupCount: num(personaGroup, "backupCount") ?? 3,
sceneBackupCount: num(personaGroup, "sceneBackupCount") ?? 10,
model: optStr(personaGroup, "model"),
@@ -328,6 +376,21 @@ export function parseConfig(raw: Record | undefined): MemoryTda
modelCacheDir: optStr(embeddingGroup, "modelCacheDir"),
configError: embeddingConfigError,
},
+ storeBackend,
+ tcvdb: {
+ url: str(tcvdbGroup, "url") ?? "",
+ username: str(tcvdbGroup, "username") ?? "root",
+ apiKey: str(tcvdbGroup, "apiKey") ?? "",
+ database: str(tcvdbGroup, "database") ?? "",
+ alias: str(tcvdbGroup, "alias") ?? "",
+ embeddingModel: str(tcvdbGroup, "embeddingModel") ?? "bge-large-zh",
+ timeout: num(tcvdbGroup, "timeout") ?? 10000,
+ caPemPath: str(tcvdbGroup, "caPemPath") || undefined,
+ },
+ bm25: {
+ enabled: bool(bm25Group, "enabled") ?? true,
+ language: (str(bm25Group, "language") === "en" ? "en" : "zh") as "zh" | "en",
+ },
memoryCleanup,
report: {
enabled: bool(obj(c, "report"), "enabled") ?? false,
diff --git a/src/conversation/l0-recorder.ts b/src/conversation/l0-recorder.ts
index f0dd7dd..6eb9274 100644
--- a/src/conversation/l0-recorder.ts
+++ b/src/conversation/l0-recorder.ts
@@ -160,7 +160,7 @@ export async function recordConversation(params: {
// - If a message lacks a timestamp field, extractUserAssistantMessages()
// assigns Date.now() at extraction time, which is always > previous cursor.
const cursor = afterTimestamp ?? 0;
- const extracted = cursor > 0
+ const extracted = cursor !== 0
? allExtracted.filter((m) => m.timestamp > cursor)
: allExtracted;
@@ -440,7 +440,7 @@ export async function readConversationMessages(
*/
export interface SessionIdMessageGroup {
sessionId: string;
- messages: ConversationMessage[];
+ messages: Array;
}
/**
@@ -451,29 +451,32 @@ export interface SessionIdMessageGroup {
* so that each group's sessionId is correctly associated with its extracted memories.
*
* When `limit` is provided, only the **newest** `limit` messages (across all groups)
- * are retained — matching the DB path's `ORDER BY timestamp DESC LIMIT ?` behavior.
+ * are retained — matching the DB path's `ORDER BY recorded_at DESC LIMIT ?` behavior.
* Groups that become empty after truncation are dropped.
*
* Groups are returned in chronological order (by earliest message timestamp).
* Messages within each group are also in chronological order.
+ *
+ * @param afterRecordedAtMs - Epoch ms cursor: only messages with recordedAt > this are included.
*/
export async function readConversationMessagesGroupedBySessionId(
sessionKey: string,
baseDir: string,
- afterTimestamp?: number,
+ afterRecordedAtMs?: number,
logger?: Logger,
limit?: number,
): Promise {
const records = await readConversationRecords(sessionKey, baseDir, logger);
- // Collect all messages with their sessionId, respecting afterTimestamp filter
- const allMessages: Array<{ sessionId: string; msg: ConversationMessage }> = [];
+ // Collect all messages with their sessionId, filtering by recorded_at cursor
+ const allMessages: Array<{ sessionId: string; msg: ConversationMessage & { recordedAtMs: number } }> = [];
for (const record of records) {
const sid = record.sessionId || "";
+ const recMs = Date.parse(record.recordedAt) || 0;
+ if (afterRecordedAtMs && recMs <= afterRecordedAtMs) continue;
for (const msg of record.messages) {
- if (afterTimestamp && msg.timestamp <= afterTimestamp) continue;
- allMessages.push({ sessionId: sid, msg });
+ allMessages.push({ sessionId: sid, msg: { ...msg, recordedAtMs: recMs } });
}
}
@@ -491,7 +494,7 @@ export async function readConversationMessagesGroupedBySessionId(
}
// Re-group by sessionId
- const groupMap = new Map();
+ const groupMap = new Map>();
for (const { sessionId, msg } of selected) {
let group = groupMap.get(sessionId);
if (!group) {
diff --git a/src/hooks/auto-capture.ts b/src/hooks/auto-capture.ts
index 906a2ab..6fd50d7 100644
--- a/src/hooks/auto-capture.ts
+++ b/src/hooks/auto-capture.ts
@@ -16,7 +16,7 @@ import { CheckpointManager } from "../utils/checkpoint.js";
import type { MemoryPipelineManager } from "../utils/pipeline-manager.js";
import { recordConversation } from "../conversation/l0-recorder.js";
import type { ConversationMessage } from "../conversation/l0-recorder.js";
-import type { VectorStore, L0VectorRecord } from "../store/vector-store.js";
+import type { IMemoryStore, L0Record } from "../store/types.js";
import type { EmbeddingService } from "../store/embedding.js";
const TAG = "[memory-tdai] [capture]";
@@ -68,7 +68,7 @@ export async function performAutoCapture(params: {
* prevents the first agent_end from dumping all session history into L0. */
pluginStartTimestamp?: number;
/** VectorStore for L0 vector indexing (optional). */
- vectorStore?: VectorStore;
+ vectorStore?: IMemoryStore;
/** EmbeddingService for L0 vector indexing (optional). */
embeddingService?: EmbeddingService;
}): Promise {
@@ -131,32 +131,42 @@ export async function performAutoCapture(params: {
const tL0RecordEnd = performance.now();
// ============================
- // Step 1.5: L0 vector indexing — metadata written synchronously,
- // embedding done in background (non-blocking)
+ // Step 1.5: L0 vector indexing
// ============================
- // PERF FIX: Remote embedding API calls (2-3s each) were blocking
- // the agent_end hook, adding 5-9s latency per conversation round.
- // Now we:
- // 1. Write L0 metadata + FTS immediately (no embedding) — ~10ms
- // 2. Fire off background task to embed + update vectors (non-blocking)
- // This way the user gets their response immediately.
+ // Two paths depending on store capabilities:
+ //
+ // A) Store supports updateL0Embedding (sqlite):
+ // - Write metadata + FTS immediately WITHOUT embedding (~ms)
+ // - Fire-and-forget background task: embedBatch + updateL0Embedding
+ // - PERF: avoids blocking agent_end with 2-3s embedding calls
+ //
+ // B) Store does NOT support updateL0Embedding (VDB / remote):
+ // - Embed synchronously, then upsertL0 with embedding in one call
+ // - VDB backends handle embedding server-side or need it upfront
const tL0VecStart = performance.now();
let l0VectorsWritten = 0;
+ let l0EmbedTotalMs = 0;
+ let l0UpsertTotalMs = 0;
logger?.debug?.(
`${TAG} [L0-vec-index] Check: filteredMessages=${filteredMessages.length}, ` +
`vectorStore=${vectorStore ? "available" : "UNAVAILABLE"}, ` +
`embeddingService=${embeddingService ? "available" : "UNAVAILABLE"}`,
);
- // Pre-generate L0 records and write metadata synchronously (fast path)
- const l0Records: Array<{ record: L0VectorRecord; content: string }> = [];
+ const supportsBgEmbed = vectorStore?.supportsDeferredEmbedding === true;
+
if (filteredMessages.length > 0 && vectorStore) {
const now = new Date().toISOString();
- logger?.debug?.(`${TAG} [L0-vec-index] START indexing ${filteredMessages.length} message(s) for session ${sessionKey}`);
+ const bgRecords: Array<{ recordId: string; content: string }> = [];
+ logger?.debug?.(
+ `${TAG} [L0-vec-index] START indexing ${filteredMessages.length} message(s) for session ${sessionKey} ` +
+ `(mode=${supportsBgEmbed ? "async-bg" : "sync"})`,
+ );
+
for (let i = 0; i < filteredMessages.length; i++) {
const msg = filteredMessages[i];
try {
- const l0Record: L0VectorRecord = {
+ const l0Record: L0Record = {
id: generateL0RecordId(sessionKey, i),
sessionKey,
sessionId: sessionId || "",
@@ -166,11 +176,45 @@ export async function performAutoCapture(params: {
timestamp: msg.timestamp,
};
- // Write metadata + FTS immediately WITHOUT embedding (fast, ~ms)
- const upsertOk = vectorStore.upsertL0(l0Record, undefined);
+ let embedding: Float32Array | undefined;
+
+ if (!supportsBgEmbed && embeddingService) {
+ // Path B (VDB): embed synchronously — needed for upsertL0
+ // Skip local embed when using server-side embedding (NoopEmbeddingService, dims=0)
+ if (embeddingService.getDimensions() === 0) {
+ logger?.debug?.(
+ `${TAG} [L0-vec-index] Server-side embedding (dims=0), skipping local embed for message ${i}`,
+ );
+ } else {
+ const tEmbedStart = performance.now();
+ try {
+ embedding = await embeddingService.embed(msg.content);
+ l0EmbedTotalMs += performance.now() - tEmbedStart;
+ logger?.debug?.(
+ `${TAG} [L0-vec-index] Embedding OK: dims=${embedding.length}, ` +
+ `norm=${Math.sqrt(Array.from(embedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}`,
+ );
+ } catch (embedErr) {
+ l0EmbedTotalMs += performance.now() - tEmbedStart;
+ logger?.warn(
+ `${TAG} [L0-vec-index] Embedding FAILED for message ${i}, ` +
+ `will write metadata only: ${embedErr instanceof Error ? embedErr.message : String(embedErr)}`,
+ );
+ }
+ }
+ }
+
+ // Path A (sqlite): pass undefined embedding — metadata + FTS only
+ // Path B (VDB): pass embedding (may be undefined on failure)
+ const tUpsertStart = performance.now();
+ const upsertOk = await vectorStore.upsertL0(l0Record, supportsBgEmbed ? undefined : embedding);
+ l0UpsertTotalMs += performance.now() - tUpsertStart;
+
if (upsertOk) {
l0VectorsWritten++;
- l0Records.push({ record: l0Record, content: msg.content });
+ if (supportsBgEmbed) {
+ bgRecords.push({ recordId: l0Record.id, content: msg.content });
+ }
} else {
logger?.warn(`${TAG} [L0-vec-index] upsertL0 returned false for message ${i}`);
}
@@ -178,38 +222,39 @@ export async function performAutoCapture(params: {
logger?.warn?.(`${TAG} [L0-vec-index] FAILED for message ${i} (non-blocking): ${err instanceof Error ? err.message : String(err)}`);
}
}
- logger?.debug?.(`${TAG} [L0-vec-index] DONE: ${l0VectorsWritten}/${filteredMessages.length} metadata records written (sync)`);
- // Fire-and-forget: batch embed + update vectors in background
- if (l0Records.length > 0 && embeddingService) {
- const bgVectorStore = vectorStore; // capture for closure
+ const modeLabel = supportsBgEmbed ? "metadata-only, embed=background" : `embed=${l0EmbedTotalMs.toFixed(0)}ms, upsert=${l0UpsertTotalMs.toFixed(0)}ms`;
+ logger?.debug?.(`${TAG} [L0-vec-index] DONE: ${l0VectorsWritten}/${filteredMessages.length} records written (${modeLabel})`);
+
+ // Path A only: fire-and-forget background embedding for sqlite stores
+ if (supportsBgEmbed && bgRecords.length > 0 && embeddingService) {
+ const bgVectorStore = vectorStore;
const bgEmbeddingService = embeddingService;
- const bgRecords = [...l0Records]; // snapshot
+ const bgSnapshot = [...bgRecords];
const bgLogger = logger;
- // Do NOT await — this runs in background after response is sent
+ // Do NOT await — runs in background after response is sent
void (async () => {
const tBgStart = performance.now();
try {
- // Use embedBatch for a single API call instead of N sequential calls
- const texts = bgRecords.map((r) => r.content);
+ const texts = bgSnapshot.map((r) => r.content);
const embeddings = await bgEmbeddingService.embedBatch(texts);
let bgUpdated = 0;
- for (let i = 0; i < bgRecords.length; i++) {
+ for (let i = 0; i < bgSnapshot.length; i++) {
try {
- const ok = bgVectorStore.updateL0Embedding(bgRecords[i].record.id, embeddings[i]);
+ const ok = await bgVectorStore.updateL0Embedding!(bgSnapshot[i].recordId, embeddings[i]);
if (ok) bgUpdated++;
} catch (err) {
bgLogger?.warn?.(
- `${TAG} [L0-vec-index-bg] Failed to update embedding for ${bgRecords[i].record.id}: ` +
+ `${TAG} [L0-vec-index-bg] Failed to update embedding for ${bgSnapshot[i].recordId}: ` +
`${err instanceof Error ? err.message : String(err)}`,
);
}
}
const bgMs = performance.now() - tBgStart;
bgLogger?.debug?.(
- `${TAG} [L0-vec-index-bg] Background embedding complete: ${bgUpdated}/${bgRecords.length} vectors updated (${bgMs.toFixed(0)}ms)`,
+ `${TAG} [L0-vec-index-bg] Background embedding complete: ${bgUpdated}/${bgSnapshot.length} vectors updated (${bgMs.toFixed(0)}ms)`,
);
} catch (err) {
const bgMs = performance.now() - tBgStart;
@@ -235,11 +280,13 @@ export async function performAutoCapture(params: {
logger?.debug?.(`${TAG} Scheduler notified of conversation round (sessionKey=${sessionKey})`);
const totalMs = performance.now() - tCaptureStart;
+ const vecDetail = supportsBgEmbed
+ ? `metadata-only, embed=background, msgs=${filteredMessages.length}`
+ : `embed=${l0EmbedTotalMs.toFixed(0)}ms, upsert=${l0UpsertTotalMs.toFixed(0)}ms, msgs=${filteredMessages.length}`;
logger?.info(
`${TAG} ⏱ Capture timing: total=${totalMs.toFixed(0)}ms, ` +
`l0Record+checkpoint=${(tL0RecordEnd - tL0RecordStart).toFixed(0)}ms, ` +
- `l0VecIndex=${(tL0VecEnd - tL0VecStart).toFixed(0)}ms ` +
- `(metadata-only, embed=background, msgs=${filteredMessages.length}), ` +
+ `l0VecIndex=${(tL0VecEnd - tL0VecStart).toFixed(0)}ms (${vecDetail}), ` +
`notify=${(performance.now() - tNotifyStart).toFixed(0)}ms`,
);
@@ -252,11 +299,13 @@ export async function performAutoCapture(params: {
}
const totalMs = performance.now() - tCaptureStart;
+ const vecDetail = supportsBgEmbed
+ ? `metadata-only, embed=background, msgs=${filteredMessages.length}`
+ : `embed=${l0EmbedTotalMs.toFixed(0)}ms, upsert=${l0UpsertTotalMs.toFixed(0)}ms, msgs=${filteredMessages.length}`;
logger?.info(
`${TAG} ⏱ Capture timing: total=${totalMs.toFixed(0)}ms, ` +
`l0Record+checkpoint=${(tL0RecordEnd - tL0RecordStart).toFixed(0)}ms, ` +
- `l0VecIndex=${(tL0VecEnd - tL0VecStart).toFixed(0)}ms ` +
- `(metadata-only, embed=background, msgs=${filteredMessages.length}), ` +
+ `l0VecIndex=${(tL0VecEnd - tL0VecStart).toFixed(0)}ms (${vecDetail}), ` +
`notify=${(performance.now() - tNotifyStart).toFixed(0)}ms`,
);
diff --git a/src/hooks/auto-recall.ts b/src/hooks/auto-recall.ts
index 045c8d1..d94230a 100644
--- a/src/hooks/auto-recall.ts
+++ b/src/hooks/auto-recall.ts
@@ -16,8 +16,8 @@ import type { MemoryTdaiConfig } from "../config.js";
import { readSceneIndex } from "../scene/scene-index.js";
import { generateSceneNavigation, stripSceneNavigation } from "../scene/scene-navigation.js";
import type { MemoryRecord } from "../record/l1-reader.js";
-import type { VectorStore, VectorSearchResult, FtsSearchResult } from "../store/vector-store.js";
-import { buildFtsQuery } from "../store/vector-store.js";
+import type { IMemoryStore, L1SearchResult, L1FtsResult } from "../store/types.js";
+import { buildFtsQuery } from "../store/sqlite.js";
import type { EmbeddingService } from "../store/embedding.js";
import { sanitizeText } from "../utils/sanitize.js";
@@ -35,6 +35,11 @@ const MEMORY_TOOLS_GUIDE = `
- **tdai_memory_search**：搜索结构化记忆（L1），适用于回忆用户偏好、历史事件节点、规则等关键信息。
- **tdai_conversation_search**：搜索原始对话（L0），适用于查找具体消息原文、时间线、上下文细节；也可用于补充或校验 memory_search 的结果。
- **read_file**（Scene Navigation 中的路径）：当已定位到相关情境，且需要该场景的完整画像、事件经过或阶段结论时使用。
+
+### ⚠️ 调用次数限制
+每轮对话中，tdai_memory_search 和 tdai_conversation_search **合计最多调用 3 次**。
+- 首次搜索无结果时，可换关键词或换工具重试，但总调用次数不要超过 3 次。
+- 若 3 次搜索后仍无结果，说明该信息不在记忆中，请直接根据已有信息回复用户，不要继续搜索。
`
/**
@@ -82,7 +87,7 @@ export async function performAutoRecall(params: {
cfg: MemoryTdaiConfig;
pluginDataDir: string;
logger?: Logger;
- vectorStore?: VectorStore;
+ vectorStore?: IMemoryStore;
embeddingService?: EmbeddingService;
}): Promise {
const { cfg, logger } = params;
@@ -112,7 +117,7 @@ async function performAutoRecallInner(params: {
cfg: MemoryTdaiConfig;
pluginDataDir: string;
logger?: Logger;
- vectorStore?: VectorStore;
+ vectorStore?: IMemoryStore;
embeddingService?: EmbeddingService;
}): Promise {
const { userText, cfg, pluginDataDir, logger, vectorStore, embeddingService } = params;
@@ -269,7 +274,7 @@ async function searchMemoriesWithDetails(
cfg: MemoryTdaiConfig,
logger: Logger | undefined,
strategy: "keyword" | "embedding" | "hybrid",
- vectorStore?: VectorStore,
+ vectorStore?: IMemoryStore,
embeddingService?: EmbeddingService,
): Promise<{ lines: string[]; memories: RecalledMemory[]; timing: SearchTiming }> {
const result = await searchMemories(userText, pluginDataDir, cfg, logger, strategy, vectorStore, embeddingService);
@@ -305,7 +310,7 @@ async function searchMemories(
cfg: MemoryTdaiConfig,
logger: Logger | undefined,
strategy: "keyword" | "embedding" | "hybrid",
- vectorStore?: VectorStore,
+ vectorStore?: IMemoryStore,
embeddingService?: EmbeddingService,
): Promise {
const emptyResult: SearchResult = { lines: [], timing: { ftsMs: 0, embeddingMs: 0, ftsHits: 0, embeddingHits: 0 } };
@@ -378,14 +383,14 @@ async function searchByKeyword(
maxResults: number,
threshold: number,
logger?: Logger,
- vectorStore?: VectorStore,
+ vectorStore?: IMemoryStore,
): Promise {
// Prefer FTS5 if available
if (vectorStore?.isFtsAvailable()) {
const ftsQuery = buildFtsQuery(userText);
if (ftsQuery) {
logger?.debug?.(`${TAG} [keyword-fts] Using FTS5 BM25 search: query="${ftsQuery}"`);
- const ftsResults = vectorStore.ftsSearchL1(ftsQuery, maxResults * 2);
+ const ftsResults = await vectorStore.searchL1Fts(ftsQuery, maxResults * 2);
if (ftsResults.length > 0) {
logger?.debug?.(
`${TAG} [keyword-fts] FTS5 raw results (${ftsResults.length}): ` +
@@ -428,7 +433,7 @@ async function searchByEmbedding(
userText: string,
maxResults: number,
threshold: number,
- vectorStore: VectorStore,
+ vectorStore: IMemoryStore,
embeddingService: EmbeddingService,
logger?: Logger,
): Promise {
@@ -442,7 +447,7 @@ async function searchByEmbedding(
`searching top-${maxResults * 2}...`,
);
// Retrieve more candidates for subsequent filtering
- const vecResults: VectorSearchResult[] = vectorStore.search(queryEmbedding, maxResults * 2);
+ const vecResults: L1SearchResult[] = await vectorStore.searchL1Vector(queryEmbedding, maxResults * 2);
if (vecResults.length === 0) {
logger?.debug?.(`${TAG} [embedding-search] Returned 0 results`);
@@ -489,7 +494,7 @@ async function searchHybrid(
_pluginDataDir: string,
maxResults: number,
_threshold: number,
- vectorStore: VectorStore,
+ vectorStore: IMemoryStore,
embeddingService: EmbeddingService,
logger?: Logger,
): Promise {
@@ -505,7 +510,7 @@ async function searchHybrid(
if (vectorStore.isFtsAvailable()) {
const ftsQuery = buildFtsQuery(userText);
if (ftsQuery) {
- const ftsResults = vectorStore.ftsSearchL1(ftsQuery, candidateK);
+ const ftsResults = await vectorStore.searchL1Fts(ftsQuery, candidateK);
if (ftsResults.length > 0) {
logger?.debug?.(`${TAG} [hybrid-keyword-fts] FTS5 found ${ftsResults.length} candidates`);
// Convert FtsSearchResult to ScoredRecord for RRF merge
@@ -547,12 +552,12 @@ async function searchHybrid(
logger?.debug?.(
`${TAG} [hybrid-embedding] Embedding OK, dims=${queryEmbedding.length}, searching top-${candidateK}...`,
);
- const results = vectorStore.search(queryEmbedding, candidateK);
+ const results = await vectorStore.searchL1Vector(queryEmbedding, candidateK, userText);
logger?.debug?.(`${TAG} [hybrid-embedding] Got ${results.length} candidates`);
return { results, ms: performance.now() - tStart };
} catch (err) {
logger?.warn?.(`${TAG} Hybrid: embedding part failed: ${err instanceof Error ? err.message : String(err)}`);
- return { results: [] as VectorSearchResult[], ms: performance.now() - tStart };
+ return { results: [] as L1SearchResult[], ms: performance.now() - tStart };
}
})(),
]);
@@ -719,7 +724,7 @@ function recordToFormatable(record: MemoryRecord): FormatableMemory {
* Build a FormatableMemory from a VectorSearchResult (embedding search path).
* Handles empty/invalid metadata_json, empty timestamp_str gracefully.
*/
-function vectorResultToFormatable(r: VectorSearchResult): FormatableMemory {
+function vectorResultToFormatable(r: L1SearchResult): FormatableMemory {
let activityStart: string | undefined;
let activityEnd: string | undefined;
if (r.metadata_json && r.metadata_json !== "{}") {
@@ -743,7 +748,7 @@ function vectorResultToFormatable(r: VectorSearchResult): FormatableMemory {
* Build a FormatableMemory from an FtsSearchResult (FTS5 keyword search path).
* Handles empty/invalid metadata_json, empty timestamp_str gracefully.
*/
-function ftsResultToFormatable(r: FtsSearchResult): FormatableMemory {
+function ftsResultToFormatable(r: L1FtsResult): FormatableMemory {
let activityStart: string | undefined;
let activityEnd: string | undefined;
if (r.metadata_json && r.metadata_json !== "{}") {
diff --git a/src/persona/persona-generator.ts b/src/persona/persona-generator.ts
index 6f6481b..36a753a 100644
--- a/src/persona/persona-generator.ts
+++ b/src/persona/persona-generator.ts
@@ -53,9 +53,9 @@ export class PersonaGenerator {
}
/**
- * Execute persona generation.
+ * Execute local persona generation without advancing checkpoint.
*/
- async generate(triggerReason?: string): Promise {
+ async generateLocalPersona(triggerReason?: string): Promise {
const startMs = Date.now();
this.logger?.debug?.(`${TAG} Starting generation: reason="${triggerReason ?? "none"}"`);
@@ -181,9 +181,6 @@ export class PersonaGenerator {
const finalContent = nav ? `${personaText}\n\n${nav}\n` : personaText;
await fs.writeFile(personaPath, finalContent, "utf-8");
- // 12. Update checkpoint
- await cpManager.markPersonaGenerated(cp.total_processed);
-
const elapsedMs = Date.now() - startMs;
this.logger?.info(`${TAG} Persona written (${finalContent.length} chars) in ${elapsedMs}ms`);
@@ -202,4 +199,17 @@ export class PersonaGenerator {
return true;
}
+
+ /**
+ * Backward-compatible wrapper: local generation + checkpoint advance.
+ */
+ async generate(triggerReason?: string): Promise {
+ const updated = await this.generateLocalPersona(triggerReason);
+ if (!updated) return false;
+
+ const cpManager = new CheckpointManager(this.dataDir);
+ const cp = await cpManager.read();
+ await cpManager.markPersonaGenerated(cp.total_processed);
+ return true;
+ }
}
diff --git a/src/profile/profile-sync.ts b/src/profile/profile-sync.ts
new file mode 100644
index 0000000..c32ee34
--- /dev/null
+++ b/src/profile/profile-sync.ts
@@ -0,0 +1,213 @@
+import { createHash } from "node:crypto";
+import fs from "node:fs/promises";
+import path from "node:path";
+import type { IMemoryStore, ProfileRecord, ProfileSyncRecord } from "../store/types.js";
+import { readSceneIndex, syncSceneIndex } from "../scene/scene-index.js";
+import { generateSceneNavigation, stripSceneNavigation } from "../scene/scene-navigation.js";
+
+const PROFILE_SCOPE = "global";
+
+interface Logger {
+ debug?: (message: string) => void;
+ info: (message: string) => void;
+ warn: (message: string) => void;
+ error: (message: string) => void;
+}
+
+export interface ProfileBaseline {
+ version: number;
+ contentMd5: string;
+ createdAtMs: number;
+}
+
+export function buildProfileStableId(scope: string, type: "l2" | "l3", filename: string): string {
+ const hash = createHash("sha256")
+ .update(`${scope}\u0000${type}\u0000${filename}`)
+ .digest("hex");
+ return `profile:v1:${hash}`;
+}
+
+function md5(text: string): string {
+ return createHash("md5").update(text).digest("hex");
+}
+
+async function statTimes(filePath: string): Promise<{ createdAtMs: number; updatedAtMs: number }> {
+ try {
+ const stat = await fs.stat(filePath);
+ return {
+ createdAtMs: Math.floor(stat.birthtimeMs || stat.ctimeMs || Date.now()),
+ updatedAtMs: Math.floor(stat.mtimeMs || Date.now()),
+ };
+ } catch {
+ const now = Date.now();
+ return { createdAtMs: now, updatedAtMs: now };
+ }
+}
+
+async function refreshPersonaNavigation(dataDir: string): Promise {
+ const personaPath = path.join(dataDir, "persona.md");
+ let body: string;
+ try {
+ body = stripSceneNavigation(await fs.readFile(personaPath, "utf-8")).trim();
+ } catch {
+ return;
+ }
+
+ if (!body) return;
+
+ const index = await readSceneIndex(dataDir);
+ const nav = generateSceneNavigation(index);
+ const finalContent = nav ? `${body}\n\n${nav}\n` : `${body}\n`;
+ await fs.writeFile(personaPath, finalContent, "utf-8");
+}
+
+export async function listLocalProfiles(dataDir: string): Promise {
+ const profiles: ProfileRecord[] = [];
+ const blocksDir = path.join(dataDir, "scene_blocks");
+
+ try {
+ const files = (await fs.readdir(blocksDir)).filter((file) => file.endsWith(".md")).sort();
+ for (const filename of files) {
+ const filePath = path.join(blocksDir, filename);
+ const content = await fs.readFile(filePath, "utf-8");
+ const { createdAtMs, updatedAtMs } = await statTimes(filePath);
+ profiles.push({
+ id: buildProfileStableId(PROFILE_SCOPE, "l2", filename),
+ type: "l2",
+ filename,
+ content,
+ contentMd5: md5(content),
+ version: 0,
+ createdAtMs,
+ updatedAtMs,
+ });
+ }
+ } catch {
+ // ignore missing scene_blocks directory
+ }
+
+ const personaPath = path.join(dataDir, "persona.md");
+ try {
+ const rawPersona = await fs.readFile(personaPath, "utf-8");
+ const body = stripSceneNavigation(rawPersona).trim();
+ if (body) {
+ const { createdAtMs, updatedAtMs } = await statTimes(personaPath);
+ profiles.push({
+ id: buildProfileStableId(PROFILE_SCOPE, "l3", "persona.md"),
+ type: "l3",
+ filename: "persona.md",
+ content: body,
+ contentMd5: md5(body),
+ version: 0,
+ createdAtMs,
+ updatedAtMs,
+ });
+ }
+ } catch {
+ // ignore missing persona file
+ }
+
+ return profiles;
+}
+
+export async function pullProfilesToLocal(
+ dataDir: string,
+ store: IMemoryStore,
+ logger: Logger,
+): Promise> {
+ if (!store.pullProfiles) return new Map();
+
+ const records = await store.pullProfiles();
+ const baseline = new Map();
+ const tempDir = await fs.mkdtemp(path.join(dataDir, ".profiles-pull-"));
+ const tempBlocksDir = path.join(tempDir, "scene_blocks");
+ await fs.mkdir(tempBlocksDir, { recursive: true });
+
+ try {
+ for (const record of records) {
+ baseline.set(record.id, {
+ version: record.version,
+ contentMd5: record.contentMd5,
+ createdAtMs: record.createdAtMs,
+ });
+
+ if (record.type === "l2") {
+ const target = path.join(tempBlocksDir, record.filename);
+ await fs.writeFile(target, record.content, "utf-8");
+ if (md5(record.content) !== record.contentMd5) {
+ await fs.rm(target, { force: true });
+ logger.warn(`[memory-tdai][profile-sync] MD5 mismatch for ${record.filename}`);
+ }
+ continue;
+ }
+
+ if (record.type === "l3") {
+ const body = stripSceneNavigation(record.content).trim();
+ await fs.writeFile(path.join(tempDir, "persona.md"), body, "utf-8");
+ if (md5(body) !== record.contentMd5) {
+ await fs.rm(path.join(tempDir, "persona.md"), { force: true });
+ logger.warn(`[memory-tdai][profile-sync] MD5 mismatch for ${record.filename}`);
+ }
+ }
+ }
+
+ const localBlocksDir = path.join(dataDir, "scene_blocks");
+ await fs.rm(localBlocksDir, { recursive: true, force: true });
+ await fs.mkdir(path.dirname(localBlocksDir), { recursive: true });
+ await fs.rename(tempBlocksDir, localBlocksDir);
+
+ const tempPersonaPath = path.join(tempDir, "persona.md");
+ const localPersonaPath = path.join(dataDir, "persona.md");
+ try {
+ await fs.access(tempPersonaPath);
+ await fs.rm(localPersonaPath, { force: true });
+ await fs.rename(tempPersonaPath, localPersonaPath);
+ } catch {
+ await fs.rm(localPersonaPath, { force: true });
+ }
+
+ await syncSceneIndex(dataDir);
+ await refreshPersonaNavigation(dataDir);
+ logger.debug?.(`[memory-tdai][profile-sync] Pulled ${records.length} profile(s) to local cache`);
+ return baseline;
+ } finally {
+ await fs.rm(tempDir, { recursive: true, force: true });
+ }
+}
+
+export async function syncLocalProfilesToStore(
+ dataDir: string,
+ store: IMemoryStore,
+ baselineMap: Map,
+ logger: Logger,
+): Promise {
+ const localProfiles = await listLocalProfiles(dataDir);
+ const localIds = new Set(localProfiles.map((profile) => profile.id));
+
+ const syncRecords: ProfileSyncRecord[] = localProfiles
+ .filter((profile) => baselineMap.get(profile.id)?.contentMd5 !== profile.contentMd5 || !baselineMap.has(profile.id))
+ .map((profile) => ({
+ ...profile,
+ baselineVersion: baselineMap.get(profile.id)?.version,
+ }));
+
+ if (syncRecords.length > 0 && store.syncProfiles) {
+ await store.syncProfiles(syncRecords);
+ logger.info(`[memory-tdai][profile-sync] Synced ${syncRecords.length} changed profile(s)`);
+ }
+
+ const deletedIds = [...baselineMap.keys()].filter((id) => !localIds.has(id));
+ if (deletedIds.length > 0 && store.deleteProfiles) {
+ await store.deleteProfiles(deletedIds);
+ logger.info(`[memory-tdai][profile-sync] Deleted ${deletedIds.length} stale profile(s)`);
+ }
+}
+
+export async function ensureL2L3Local(
+ dataDir: string,
+ store: IMemoryStore,
+ logger: Logger,
+): Promise> {
+ if (!store.pullProfiles) return new Map();
+ return pullProfilesToLocal(dataDir, store, logger);
+}
diff --git a/src/prompts/scene-extraction.ts b/src/prompts/scene-extraction.ts
index 19e8f06..8648943 100644
--- a/src/prompts/scene-extraction.ts
+++ b/src/prompts/scene-extraction.ts
@@ -8,8 +8,10 @@
*
* Security: The LLM is sandboxed to scene_blocks/ only (workspaceDir = scene_blocks/).
* It has NO visibility into checkpoint, scene_index, persona.md, or any other system file.
- * File deletion is achieved via "soft-delete" — writing an empty string to the file
- * — and the SceneExtractor subsequently removes empty files with fs.unlink.
+ * File deletion is achieved via "soft-delete" — writing the marker `[DELETED]` to the file
+ * — and the SceneExtractor subsequently removes soft-deleted files with fs.unlink.
+ * Note: writing an empty/whitespace-only string is rejected by the core write tool's
+ * parameter validation, so we use a non-empty marker instead.
*
* Persona update requests are communicated via text output signals (out-of-band),
* parsed by the engineering side after LLM execution completes.
@@ -22,6 +24,8 @@ export interface SceneExtractionPromptParams {
sceneCountWarning?: string;
/** List of existing scene filenames (relative, e.g. ["work.md", "hobby.md"]) */
existingSceneFiles?: string[];
+ /** Maximum number of scene blocks allowed */
+ maxScenes: number;
}
export function buildSceneExtractionPrompt(params: SceneExtractionPromptParams): string {
@@ -31,6 +35,7 @@ export function buildSceneExtractionPrompt(params: SceneExtractionPromptParams):
currentTimestamp,
sceneCountWarning,
existingSceneFiles,
+ maxScenes,
} = params;
const warningSection = sceneCountWarning
@@ -51,7 +56,7 @@ export function buildSceneExtractionPrompt(params: SceneExtractionPromptParams):
### Layer 2 (Processing): Scene Diaries
- **形态**：**不是清单，是连贯的叙事文档**
-- **逻辑**：将 L1 碎片融合进特定场景文件（强制限制在15个以内）
+- **逻辑**：将 L1 碎片融合进特定场景文件
- **动作**：Create（创建）、Integrate（整合）、Rewrite（重写）
- **禁止**：简单追加列表
@@ -63,6 +68,8 @@ ${warningSection}
2. 现有 Block 映射表 (Existing Blocks Map): 包含当前所有记忆块（Markdown 文件）的文件名和摘要的列表。
3. 当前时间 (Current Time): 用于生成元数据的具体时间戳。
+**⚠️ 场景文件数量上限：${maxScenes} 个。处理完成后目录中的场景文件数量必须严格小于此上限。**
+
### 1️⃣ New Memories List
${memoriesJson}
@@ -86,6 +93,8 @@ ${existingSceneFiles.map((f) => `- \`${f}\``).join("\n")}
3. **创建新场景文件时**，直接使用文件名，如 \`新场景名.md\`
4. **场景文件支持 replace_in_file**。对于局部更新（如只更新某个章节或 META 字段），可以使用 \`replace_in_file\` 进行精确替换。对于大范围重写或结构性变更，建议使用 \`read_file\` + \`write_to_file\` 整体重写。
5. **场景索引和系统配置由工程系统自动维护**，你只需专注于操作 \`.md\` 场景文件
+6. **删除文件的唯一方式**：使用 \`write_to_file(filename, '[DELETED]')\` 将文件内容写为 \`[DELETED]\` 标记。系统会自动清理带有此标记的文件。**禁止**写入空字符串（会被系统拒绝）。**禁止**用 \`[ARCHIVE]\`、\`[CONSOLIDATED]\` 等其他标记替代删除——只有 \`[DELETED]\` 标记会触发系统清理。
+7. **禁止创建报告/整合/汇总类文件**。你的输出必须是有意义的场景叙事文件（如"技术架构与工程实践.md"、"日常生活与工作节奏.md"）。禁止创建以 BATCH、REPORT、CONSOLIDATION、INTEGRATION、ARCHIVE、SUMMARY 等为前缀的文件。
## 工作流与逻辑 (Workflow & Logic)
在生成输出之前，你必须执行以下"思维链"过程：
@@ -94,11 +103,12 @@ ${existingSceneFiles.map((f) => `- \`${f}\``).join("\n")}
**在处理任何记忆之前，你必须：**
-1. **统计当前场景总数**：检查 "Existing Scene Blocks Summary" 中的场景数量
-2. **遵守分级预警，上限为15个block**：
- - 红色预警（≥ 15）：**必须先合并**，将最相似的 2-4 个场景合并为 1 个，然后再处理新记忆
- - 橙色预警（= 15-1）：**只能 UPDATE 现有场景，不能 CREATE 新场景**
- - 黄色预警（接近15）：**优先 UPDATE 或主动 MERGE 相似场景**
+1. **统计当前场景总数**：查看 "Existing Scene Blocks Summary" 顶部标注的当前场景总数
+2. **最终目标**：处理完成后，目录中的场景文件数量必须 **严格小于 ${maxScenes}**
+3. **遵守分级预警**：
+ - 红色预警（≥ ${maxScenes}）：**必须先通过 MERGE 减少文件数量**，将最相似的 2-4 个场景合并为 1 个，**并删除被合并的旧文件**，直到文件数 < ${maxScenes} 后，再处理新记忆
+ - 橙色预警（= ${maxScenes - 1}）：**只能 UPDATE 现有场景，不能 CREATE 新场景**
+ - 黄色预警（接近 ${maxScenes}）：**优先 UPDATE 或主动 MERGE 相似场景**
**合并优先级**（当需要合并时，按以下顺序选择）：
1. **主题高度重叠**：如"Python后端开发"和"Go后端开发" → 合并为"后端开发技术栈"
@@ -120,10 +130,11 @@ ${existingSceneFiles.map((f) => `- \`${f}\``).join("\n")}
1. **UPDATE（更新）**【首选策略】: 如果存在相关的 Block（基于摘要或文件名的相似性），先 read 文件内的具体信息，再锁定该 Block 进行更新（write 或 replace）
2. **MERGE（合并）**:
- 合并的新 block 应该是生成概括性更强的场景，包含已有的多个相似场景
- - **强制合并**：当前 Block 总数 **≥ 上限**时，必须先将多个相似记忆合并，或者删除最旧或者最不重要的 block
+ - **强制合并**：当前 Block 总数 **≥ ${maxScenes}** 时，必须先将多个相似记忆合并
- **主动合并**：即使未达上限，如果两个 Block 属于同一叙事弧线，也应合并以增加深度
+ - **⚠️ 合并后必须删除旧文件**：被合并的旧场景文件必须通过 \`write_to_file(旧文件名, '[DELETED]')\` 写入删除标记。**仅仅打标记（如 [ARCHIVE]、[CONSOLIDATED]）不算删除，文件仍会占用配额。**
3. **CREATE（新建）**【最后手段】:
- - **前提条件**：当前场景总数未达上限
+ - **前提条件**：当前场景总数 < ${maxScenes}
- **CREATE 前的强制验证**：必须先用 \`read_file\` 检查至少 2 个最相似的现有场景，确认新记忆确实无法融入后才能 CREATE。跳过验证直接 CREATE 是被禁止的
- 如果话题是全新的且与现有内容区分度高，可以创建新 Block
- **每次批处理最多新增 1 个场景**
@@ -135,14 +146,15 @@ ${existingSceneFiles.map((f) => `- \`${f}\``).join("\n")}
3. \`write_to_file('Python后端开发.md', B)\` → **整体重写该场景文件**
或 \`replace_in_file('Python后端开发.md', old_section, new_section)\` → **局部更新某部分**
-合并多个 block 的逻辑：
+**示例 B：合并多个 block（MERGE — 合并后必须删除旧文件）**
**具体操作步骤（工具调用）**：
1. \`read_file('Python后端开发.md')\` → 获取内容 A
2. \`read_file('Go后端开发.md')\` → 获取内容 B
3. 整合 A + B + 新记忆 → 生成新内容 C（\`heat = heatA + heatB + 1\`）
-4. \`write_to_file('后端开发技术栈.md', C)\` → 创建新文件，写入合并后的完整内容
-5. \`write_to_file('Python后端开发.md', '')\` → **清空旧文件 A（标记删除）**
-6. \`write_to_file('Go后端开发.md', '')\` → **清空旧文件 B（标记删除）**
+4. \`write_to_file('后端开发技术栈.md', C)\` → 创建合并后的新文件
+5. \`write_to_file('Python后端开发.md', '[DELETED]')\` → **⚠️ 删除旧文件 A（写 [DELETED] 标记）**
+6. \`write_to_file('Go后端开发.md', '[DELETED]')\` → **⚠️ 删除旧文件 B（写 [DELETED] 标记）**
+**关键**：步骤 5-6 是必须的！不执行删除 = 文件总数不减少 = 合并无效。
### 阶段 3：撰写与合成（核心任务）
深度整合: 严禁简单的文本追加。你必须结合上下文（基于摘要或提供的原始内容）重写叙事，将新信息自然地融入其中。
@@ -224,5 +236,5 @@ reason: 具体原因描述
- 使用 \`read_file\` 读取需要更新的场景文件
- 使用 \`write_to_file\` 创建新文件或**整体重写**已有场景文件
- 使用 \`replace_in_file\` 对场景文件进行**局部更新**（如只更新某个章节）
- - **删除文件**：使用 \`write_to_file(filename, '')\` 将文件内容清空（系统会自动清理空文件，禁止使用其他删除方式）`;
+ - **删除文件**：使用 \`write_to_file(filename, '[DELETED]')\` 将文件内容写为 **\`[DELETED]\` 标记**。系统会自动清理这些文件。**重要**：只有 \`[DELETED]\` 标记才会触发系统清理。写入空字符串会被系统拒绝，写入 \`[ARCHIVE]\`、\`[CONSOLIDATED]\` 等标记**不会删除文件**，文件会继续占用场景配额。`;
}
diff --git a/src/record/l1-dedup.ts b/src/record/l1-dedup.ts
index a130c91..ea1292c 100644
--- a/src/record/l1-dedup.ts
+++ b/src/record/l1-dedup.ts
@@ -16,8 +16,8 @@ import { CONFLICT_DETECTION_SYSTEM_PROMPT, formatBatchConflictPrompt } from "../
import type { CandidateMatch } from "../prompts/l1-dedup.js";
import { CleanContextRunner } from "../utils/clean-context-runner.js";
import { sanitizeJsonForParse } from "../utils/sanitize.js";
-import type { VectorStore } from "../store/vector-store.js";
-import { buildFtsQuery } from "../store/vector-store.js";
+import type { IMemoryStore } from "../store/types.js";
+import { buildFtsQuery } from "../store/sqlite.js";
import type { EmbeddingService } from "../store/embedding.js";
interface Logger {
@@ -60,7 +60,7 @@ export async function batchDedup(params: {
logger?: Logger;
model?: string;
/** Vector store for cosine similarity candidate recall */
- vectorStore?: VectorStore;
+ vectorStore?: IMemoryStore;
/** Embedding service for computing query vectors */
embeddingService?: EmbeddingService;
/** Top-K candidates per new memory (default: 5) */
@@ -81,7 +81,7 @@ export async function batchDedup(params: {
}));
// Determine what recall capabilities are available
- const hasVectorData = vectorStore && vectorStore.count() > 0;
+ const hasVectorData = vectorStore && (await vectorStore.countL1()) > 0;
const hasFts = vectorStore?.isFtsAvailable() ?? false;
// Fast path: no recall capability at all → skip dedup
@@ -109,7 +109,7 @@ export async function batchDedup(params: {
);
// Degrade to FTS keyword recall
if (hasFts) {
- matches = findCandidatesByFts(memories, vectorStore!, logger);
+ matches = await findCandidatesByFts(memories, vectorStore!, logger);
} else {
logger?.debug?.(`${TAG} FTS not available either, skipping conflict detection`);
return storeAll();
@@ -118,7 +118,7 @@ export async function batchDedup(params: {
} else if (hasFts) {
// === Tier 2: FTS keyword recall ===
logger?.debug?.(`${TAG} Using FTS keyword recall mode (no embedding service or no vector data)`);
- matches = findCandidatesByFts(memories, vectorStore!, logger);
+ matches = await findCandidatesByFts(memories, vectorStore!, logger);
} else {
// Shouldn't reach here given the fast-path check above, but be defensive
logger?.debug?.(`${TAG} No usable recall path, skipping conflict detection`);
@@ -191,7 +191,7 @@ async function runLlmJudgment(
*/
async function findCandidatesByVector(
memories: Array,
- vectorStore: VectorStore,
+ vectorStore: IMemoryStore,
embeddingService: EmbeddingService,
topK: number,
logger?: Logger,
@@ -209,7 +209,7 @@ async function findCandidatesByVector(
const queryVec = embeddings[i];
// Vector search top-K (request extra to account for self-batch filtering)
- const searchResults = vectorStore.search(queryVec, topK + memories.length);
+ const searchResults = await vectorStore.searchL1Vector(queryVec, topK + memories.length, mem.content);
// Exclude records from current batch, convert to MemoryRecord format
const candidates: MemoryRecord[] = searchResults
@@ -245,18 +245,18 @@ async function findCandidatesByVector(
* Uses the FTS index for efficient BM25-ranked keyword matching.
* This replaces the old Jaccard word-overlap fallback entirely.
*/
-function findCandidatesByFts(
+async function findCandidatesByFts(
memories: Array,
- vectorStore: VectorStore,
+ vectorStore: IMemoryStore,
_logger?: Logger,
-): CandidateMatch[] {
+): Promise {
const newRecordIds = new Set(memories.map((m) => m.record_id));
const matches: CandidateMatch[] = [];
for (const mem of memories) {
const ftsQuery = buildFtsQuery(mem.content);
if (ftsQuery) {
- const ftsResults = vectorStore.ftsSearchL1(ftsQuery, 10);
+ const ftsResults = await vectorStore.searchL1Fts(ftsQuery, 10);
// Filter out records from the current batch
const candidates: MemoryRecord[] = ftsResults
.filter((r) => !newRecordIds.has(r.record_id))
@@ -333,6 +333,11 @@ function parseBatchResult(
const d = item as Record;
const recordId = String(d.record_id ?? "");
+ // Skip entries with empty/missing record_id — they are LLM hallucinations
+ if (!recordId) {
+ logger?.debug?.(`${TAG} Skipping decision with empty record_id`);
+ continue;
+ }
const action = String(d.action ?? "store");
if (!validActions.includes(action)) {
diff --git a/src/record/l1-extractor.ts b/src/record/l1-extractor.ts
index 58f18de..12547b6 100644
--- a/src/record/l1-extractor.ts
+++ b/src/record/l1-extractor.ts
@@ -19,7 +19,7 @@ import { writeMemory, generateMemoryId } from "./l1-writer.js";
import type { ExtractedMemory, MemoryRecord, MemoryType, DedupDecision } from "./l1-writer.js";
import { CleanContextRunner } from "../utils/clean-context-runner.js";
import { sanitizeJsonForParse, shouldExtractL1 } from "../utils/sanitize.js";
-import type { VectorStore } from "../store/vector-store.js";
+import type { IMemoryStore } from "../store/types.js";
import type { EmbeddingService } from "../store/embedding.js";
import { report } from "../report/reporter.js";
@@ -98,7 +98,7 @@ export async function extractL1Memories(params: {
/** Previous scene name for continuity */
previousSceneName?: string;
/** Vector store for cosine similarity candidate recall */
- vectorStore?: VectorStore;
+ vectorStore?: IMemoryStore;
/** Embedding service for computing query vectors */
embeddingService?: EmbeddingService;
/** Top-K candidates for conflict recall (default: 5) */
@@ -394,7 +394,7 @@ async function applyDecisions(params: {
sessionKey: string;
sessionId?: string;
logger?: Logger;
- vectorStore?: VectorStore;
+ vectorStore?: IMemoryStore;
embeddingService?: EmbeddingService;
}): Promise {
const { memoriesWithIds, decisions, baseDir, sessionKey, sessionId, logger, vectorStore, embeddingService } = params;
@@ -447,7 +447,7 @@ async function storeAllDirectly(
sessionKey: string,
sessionId: string | undefined,
logger?: Logger,
- vectorStore?: VectorStore,
+ vectorStore?: IMemoryStore,
embeddingService?: EmbeddingService,
): Promise {
const storedRecords: MemoryRecord[] = [];
diff --git a/src/record/l1-reader.ts b/src/record/l1-reader.ts
index 0b34554..66e8d39 100644
--- a/src/record/l1-reader.ts
+++ b/src/record/l1-reader.ts
@@ -14,11 +14,11 @@
import fs from "node:fs/promises";
import path from "node:path";
import type { MemoryRecord, MemoryType, EpisodicMetadata } from "./l1-writer.js";
-import type { VectorStore, L1RecordRow, L1QueryFilter } from "../store/vector-store.js";
+import type { IMemoryStore, L1RecordRow, L1QueryFilter } from "../store/types.js";
// Re-export types that readers need
export type { MemoryRecord, MemoryType, EpisodicMetadata } from "./l1-writer.js";
-export type { L1QueryFilter } from "../store/vector-store.js";
+export type { L1QueryFilter } from "../store/types.js";
interface Logger {
debug?: (message: string) => void;
@@ -44,17 +44,17 @@ const TAG = "[memory-tdai] [l1-reader]";
*
* Falls back to empty array if VectorStore is null or degraded.
*/
-export function queryMemoryRecords(
- vectorStore: VectorStore | null | undefined,
+export async function queryMemoryRecords(
+ vectorStore: IMemoryStore | null | undefined,
filter?: L1QueryFilter,
logger?: Logger,
-): MemoryRecord[] {
+): Promise {
if (!vectorStore) {
logger?.warn(`${TAG} queryMemoryRecords: no VectorStore available, returning empty`);
return [];
}
- const rows = vectorStore.queryL1Records(filter);
+ const rows = await vectorStore.queryL1Records(filter);
return rows.map(rowToMemoryRecord);
}
diff --git a/src/record/l1-writer.ts b/src/record/l1-writer.ts
index d9510ef..b5e618d 100644
--- a/src/record/l1-writer.ts
+++ b/src/record/l1-writer.ts
@@ -19,7 +19,7 @@
import fs from "node:fs/promises";
import path from "node:path";
import crypto from "node:crypto";
-import type { VectorStore } from "../store/vector-store.js";
+import type { IMemoryStore } from "../store/types.js";
import type { EmbeddingService } from "../store/embedding.js";
// ============================
@@ -149,7 +149,7 @@ export async function writeMemory(params: {
sessionId?: string;
logger?: Logger;
/** Optional vector store for dual-write (JSONL + vector DB) */
- vectorStore?: VectorStore;
+ vectorStore?: IMemoryStore;
/** Optional embedding service (required when vectorStore is provided) */
embeddingService?: EmbeddingService;
}): Promise {
@@ -208,7 +208,7 @@ export async function writeMemory(params: {
// by memory-cleaner (which reconciles against VectorStore as source of truth).
if (vectorStore) {
try {
- vectorStore.deleteBatch(decision.target_ids);
+ await vectorStore.deleteL1Batch(decision.target_ids);
logger?.debug?.(`${TAG} VectorStore: deleted ${decision.target_ids.length} target record(s) for ${decision.action}`);
} catch (err) {
logger?.warn?.(
@@ -251,7 +251,7 @@ export async function writeMemory(params: {
}
}
- const upsertOk = vectorStore.upsert(record, embedding);
+ const upsertOk = await vectorStore.upsertL1(record, embedding);
logger?.debug?.(`${TAG} [vec-dual-write] upsert result=${upsertOk} id=${record.id}`);
} catch (err) {
// Vector write failure should NOT block the main JSONL write
diff --git a/src/report/reporter.ts b/src/report/reporter.ts
index b0091a4..094f01c 100644
--- a/src/report/reporter.ts
+++ b/src/report/reporter.ts
@@ -19,7 +19,7 @@ let _reporter: IReporter | undefined;
export function initReporter(opts: {
enabled: boolean;
type: string;
- logger: { info: (msg: string) => void };
+ logger: { info: (msg: string) => void; debug?: (msg: string) => void };
instanceId: string;
pluginVersion: string;
}): void {
@@ -31,7 +31,7 @@ export function initReporter(opts: {
break;
// TODO: add new reporter type
default:
- opts.logger.info(`[memory-tdai] Unknown reporter type "${opts.type}", disabled reporting`);
+ opts.logger.debug?.(`[memory-tdai] Unknown reporter type "${opts.type}", disabled reporting`);
break;
}
}
@@ -40,6 +40,14 @@ export function setReporter(reporter: IReporter): void {
_reporter = reporter;
}
+/**
+ * Reset the reporter singleton so that the next `initReporter` call takes effect.
+ * Must be called at plugin re-registration (hot-reload) to pick up config changes.
+ */
+export function resetReporter(): void {
+ _reporter = undefined;
+}
+
export function report(event: string, data: ReportPayload): void {
if (!_reporter) return;
try {
diff --git a/src/scene/scene-extractor.ts b/src/scene/scene-extractor.ts
index f900472..1d08c6c 100644
--- a/src/scene/scene-extractor.ts
+++ b/src/scene/scene-extractor.ts
@@ -89,7 +89,7 @@ export class SceneExtractor {
constructor(opts: SceneExtractorOptions) {
this.dataDir = opts.dataDir;
- this.maxScenes = opts.maxScenes ?? 20;
+ this.maxScenes = opts.maxScenes ?? 15;
this.sceneBackupCount = opts.sceneBackupCount ?? 10;
this.timeoutMs = opts.timeoutMs ?? 300_000; // 5 min — LLM may do multiple tool calls
this.logger = opts.logger;
@@ -190,6 +190,7 @@ export class SceneExtractor {
currentTimestamp,
sceneCountWarning,
existingSceneFiles,
+ maxScenes: this.maxScenes,
});
this.logger?.debug?.(`${TAG} extract() prompt built: ${prompt.length} chars (${Date.now() - promptStartMs}ms)`);
@@ -218,20 +219,34 @@ export class SceneExtractor {
// Phase 5: Subsequent processing — safe cleanup of soft-deleted files
//
// Security: The LLM has no `exec` tool and cannot run shell commands.
- // Instead, it "deletes" files by writing empty content (soft-delete).
- // Here we detect and remove those empty files before syncing the index,
- // so syncSceneIndex won't re-index stale empty entries.
+ // Instead, it "deletes" files by writing the marker `[DELETED]` to the file
+ // (writing empty/whitespace-only content is rejected by core's write tool
+ // parameter validation). Here we detect and remove those soft-deleted files
+ // before syncing the index, so syncSceneIndex won't re-index stale entries.
+ //
+ // We also detect "META-only" files — files that contain only a META header
+ // (e.g. [ARCHIVE] or [CONSOLIDATED] markers) but no actual scene content.
+ // These are artifacts of LLM merges that didn't properly delete old files.
const cleanupStartMs = Date.now();
let cleanedCount = 0;
try {
const allFiles = (await fs.readdir(sceneBlocksDir)).filter((f) => f.endsWith(".md"));
for (const file of allFiles) {
const filePath = path.join(sceneBlocksDir, file);
- const content = await fs.readFile(filePath, "utf-8");
- if (content.trim().length === 0) {
+ const raw = await fs.readFile(filePath, "utf-8");
+ if (raw.trim().length === 0 || raw.trim() === "[DELETED]") {
+ // Empty file or [DELETED] marker — soft-delete
await fs.unlink(filePath);
cleanedCount++;
this.logger?.debug?.(`${TAG} extract() removed soft-deleted file: ${file}`);
+ } else {
+ // Check if file has only META header but no actual content
+ const block = parseSceneBlock(raw, file);
+ if (!block.content || block.content.trim().length === 0) {
+ await fs.unlink(filePath);
+ cleanedCount++;
+ this.logger?.debug?.(`${TAG} extract() removed META-only file (no content): ${file}`);
+ }
}
}
} catch (cleanupErr) {
diff --git a/src/seed/input.ts b/src/seed/input.ts
new file mode 100644
index 0000000..e2aa742
--- /dev/null
+++ b/src/seed/input.ts
@@ -0,0 +1,435 @@
+/**
+ * Input loading, validation, normalization, and timestamp handling for the `seed` command.
+ *
+ * Responsibilities:
+ * 1. Load raw JSON from file
+ * 2. Detect Format A (`{ sessions: [...] }`) vs Format B (`[...]`)
+ * 3. Six-layer validation (file → top-level → session → round → message → timestamp consistency)
+ * 4. Normalize into NormalizedInput with auto-generated sessionIds
+ * 5. Timestamp all-or-none check + fill strategy
+ */
+
+import fs from "node:fs";
+import crypto from "node:crypto";
+import type {
+ RawSession,
+ FormatA,
+ ValidationError,
+ NormalizedInput,
+ NormalizedSession,
+ NormalizedRound,
+ NormalizedMessage,
+ SeedCommandOptions,
+} from "./types.js";
+
+// ============================
+// Public API
+// ============================
+
+export interface LoadAndValidateResult {
+ /** Normalized input ready for pipeline consumption. */
+ input: NormalizedInput;
+ /** Whether the user needs to confirm timestamp auto-fill. */
+ needsTimestampConfirmation: boolean;
+}
+
+/**
+ * Load, validate, and normalize seed input from a file.
+ *
+ * Throws on fatal validation errors with a human-readable message
+ * that includes all collected errors.
+ */
+export function loadAndValidateInput(
+ opts: Pick,
+): LoadAndValidateResult {
+ // Layer 1: File — read + parse
+ const raw = loadRawInput(opts.input);
+
+ // Layer 2: Top-level — detect A vs B
+ const sessions = extractSessions(raw);
+
+ // Layers 3-5: session / round / message validation
+ const errors: ValidationError[] = [];
+ validateSessions(sessions, opts.strictRoundRole, errors);
+
+ if (errors.length > 0) {
+ throw new SeedValidationError(errors);
+ }
+
+ // Layer 6: Timestamp consistency (all-have / all-missing / mixed → error)
+ const tsResult = checkTimestampConsistency(sessions);
+ if (tsResult.status === "mixed") {
+ throw new SeedValidationError([{
+ stage: "timestamp_consistency",
+ message:
+ "Timestamp consistency check failed: some messages have timestamps while others do not. " +
+ "All messages must either have timestamps or none must have timestamps.",
+ }]);
+ }
+
+ // Normalize
+ const normalized = normalizeSessions(sessions, opts.sessionKey);
+
+ return {
+ input: {
+ sessions: normalized.sessions,
+ totalRounds: normalized.totalRounds,
+ totalMessages: normalized.totalMessages,
+ hasTimestamps: tsResult.status === "all_present",
+ },
+ needsTimestampConfirmation: tsResult.status === "all_missing",
+ };
+}
+
+/**
+ * Fill timestamps for all messages when the input has no timestamps.
+ *
+ * Uses a single monotonically increasing counter across ALL sessions
+ * to guarantee global timestamp ordering. This is critical when multiple
+ * sessions share the same sessionKey — the L0 capture cursor (advanced
+ * per-session) would filter out later sessions whose timestamps fall
+ * below the cursor if ordering were not globally monotonic.
+ */
+export function fillTimestamps(input: NormalizedInput): void {
+ let currentTs = Date.now();
+ for (const session of input.sessions) {
+ for (const round of session.rounds) {
+ for (let i = 0; i < round.messages.length; i++) {
+ // Small offset per message to maintain strict ordering
+ round.messages[i]!.timestamp = currentTs;
+ currentTs += 100;
+ }
+ }
+ }
+ input.hasTimestamps = true;
+}
+
+// ============================
+// Validation error class
+// ============================
+
+export class SeedValidationError extends Error {
+ public readonly errors: ValidationError[];
+
+ constructor(errors: ValidationError[]) {
+ const summary = errors.map((e) => formatValidationError(e)).join("\n");
+ super(`Seed input validation failed (${errors.length} error(s)):\n${summary}`);
+ this.name = "SeedValidationError";
+ this.errors = errors;
+ }
+}
+
+function formatValidationError(e: ValidationError): string {
+ const parts: string[] = [` [${e.stage}]`];
+ if (e.sourceIndex != null) parts.push(`session[${e.sourceIndex}]`);
+ if (e.sessionKey) parts.push(`key="${e.sessionKey}"`);
+ if (e.roundIndex != null) parts.push(`round[${e.roundIndex}]`);
+ if (e.messageIndex != null) parts.push(`msg[${e.messageIndex}]`);
+ parts.push(e.message);
+ return parts.join(" ");
+}
+
+// ============================
+// Layer 1: File loading
+// ============================
+
+function loadRawInput(filePath: string): unknown {
+ if (!fs.existsSync(filePath)) {
+ throw new SeedValidationError([{
+ stage: "file",
+ message: `Input file not found: ${filePath}`,
+ }]);
+ }
+
+ const content = fs.readFileSync(filePath, "utf-8").trim();
+ if (!content) {
+ throw new SeedValidationError([{
+ stage: "file",
+ message: "Input file is empty.",
+ }]);
+ }
+
+ try {
+ return JSON.parse(content);
+ } catch (err) {
+ throw new SeedValidationError([{
+ stage: "file",
+ message: `JSON parse error: ${err instanceof Error ? err.message : String(err)}`,
+ }]);
+ }
+}
+
+// ============================
+// Layer 2: Top-level format detection
+// ============================
+
+function extractSessions(raw: unknown): RawSession[] {
+ // Format A: { sessions: [...] }
+ if (
+ raw != null &&
+ typeof raw === "object" &&
+ !Array.isArray(raw) &&
+ "sessions" in raw
+ ) {
+ const obj = raw as FormatA;
+ if (!Array.isArray(obj.sessions)) {
+ throw new SeedValidationError([{
+ stage: "top_level",
+ message: 'Format A detected but "sessions" is not an array.',
+ }]);
+ }
+ return obj.sessions;
+ }
+
+ // Format B: [...]
+ if (Array.isArray(raw)) {
+ return raw as RawSession[];
+ }
+
+ throw new SeedValidationError([{
+ stage: "top_level",
+ message:
+ "Unrecognized input format. Expected either:\n" +
+ ' Format A: { "sessions": [...] }\n' +
+ " Format B: [ { sessionKey, conversations }, ... ]",
+ }]);
+}
+
+// ============================
+// Layers 3-5: session / round / message validation
+// ============================
+
+function validateSessions(
+ sessions: RawSession[],
+ strictRoundRole: boolean,
+ errors: ValidationError[],
+): void {
+ if (sessions.length === 0) {
+ errors.push({
+ stage: "session",
+ message: "No sessions found in input.",
+ });
+ return;
+ }
+
+ for (let si = 0; si < sessions.length; si++) {
+ const session = sessions[si]!;
+
+ // Layer 3: session validation
+ if (!session.sessionKey || typeof session.sessionKey !== "string" || session.sessionKey.trim() === "") {
+ errors.push({
+ stage: "session",
+ sourceIndex: si,
+ message: '"sessionKey" is required and must be a non-empty string.',
+ });
+ }
+
+ if (!Array.isArray(session.conversations)) {
+ errors.push({
+ stage: "session",
+ sourceIndex: si,
+ sessionKey: session.sessionKey,
+ message: '"conversations" must be a two-dimensional array (array of rounds).',
+ });
+ continue; // Can't validate rounds
+ }
+
+ // Check that conversations is a 2D array
+ for (let ri = 0; ri < session.conversations.length; ri++) {
+ const round = session.conversations[ri];
+
+ // Layer 4: round validation
+ if (!Array.isArray(round)) {
+ errors.push({
+ stage: "round",
+ sourceIndex: si,
+ sessionKey: session.sessionKey,
+ roundIndex: ri,
+ message: "Round must be an array of messages.",
+ });
+ continue;
+ }
+
+ if (round.length === 0) {
+ errors.push({
+ stage: "round",
+ sourceIndex: si,
+ sessionKey: session.sessionKey,
+ roundIndex: ri,
+ message: "Round must be a non-empty array.",
+ });
+ continue;
+ }
+
+ // Strict round-role: each round must have at least one user and one assistant
+ if (strictRoundRole) {
+ const roles = new Set(round.map((m) => m.role));
+ if (!roles.has("user")) {
+ errors.push({
+ stage: "round",
+ sourceIndex: si,
+ sessionKey: session.sessionKey,
+ roundIndex: ri,
+ message: '--strict-round-role: round must contain at least one "user" message.',
+ });
+ }
+ if (!roles.has("assistant")) {
+ errors.push({
+ stage: "round",
+ sourceIndex: si,
+ sessionKey: session.sessionKey,
+ roundIndex: ri,
+ message: '--strict-round-role: round must contain at least one "assistant" message.',
+ });
+ }
+ }
+
+ // Layer 5: message validation
+ for (let mi = 0; mi < round.length; mi++) {
+ const msg = round[mi]!;
+
+ if (!msg.role || typeof msg.role !== "string") {
+ errors.push({
+ stage: "message",
+ sourceIndex: si,
+ sessionKey: session.sessionKey,
+ roundIndex: ri,
+ messageIndex: mi,
+ message: '"role" is required and must be a non-empty string.',
+ });
+ }
+
+ if (!msg.content || typeof msg.content !== "string" || msg.content.trim() === "") {
+ errors.push({
+ stage: "message",
+ sourceIndex: si,
+ sessionKey: session.sessionKey,
+ roundIndex: ri,
+ messageIndex: mi,
+ message: '"content" is required and must be a non-empty string.',
+ });
+ }
+
+ if (msg.timestamp !== undefined) {
+ if (typeof msg.timestamp === "number") {
+ if (!Number.isInteger(msg.timestamp)) {
+ errors.push({
+ stage: "message",
+ sourceIndex: si,
+ sessionKey: session.sessionKey,
+ roundIndex: ri,
+ messageIndex: mi,
+ message: '"timestamp" must be an integer (epoch milliseconds). Negative values are allowed for dates before 1970.',
+ });
+ }
+ } else if (typeof msg.timestamp === "string") {
+ if (Number.isNaN(new Date(msg.timestamp).getTime())) {
+ errors.push({
+ stage: "message",
+ sourceIndex: si,
+ sessionKey: session.sessionKey,
+ roundIndex: ri,
+ messageIndex: mi,
+ message: `"timestamp" string is not a valid ISO 8601 date: "${msg.timestamp}".`,
+ });
+ }
+ } else {
+ errors.push({
+ stage: "message",
+ sourceIndex: si,
+ sessionKey: session.sessionKey,
+ roundIndex: ri,
+ messageIndex: mi,
+ message: '"timestamp" must be a number (epoch ms) or an ISO 8601 string.',
+ });
+ }
+ }
+ }
+ }
+ }
+}
+
+// ============================
+// Layer 6: Timestamp consistency
+// ============================
+
+interface TimestampCheckResult {
+ status: "all_present" | "all_missing" | "mixed";
+}
+
+function checkTimestampConsistency(sessions: RawSession[]): TimestampCheckResult {
+ let hasTs = false;
+ let missingTs = false;
+
+ for (const session of sessions) {
+ if (!Array.isArray(session.conversations)) continue;
+ for (const round of session.conversations) {
+ if (!Array.isArray(round)) continue;
+ for (const msg of round) {
+ if (msg.timestamp !== undefined && msg.timestamp !== null) {
+ hasTs = true;
+ } else {
+ missingTs = true;
+ }
+ // Early exit on mixed
+ if (hasTs && missingTs) {
+ return { status: "mixed" };
+ }
+ }
+ }
+ }
+
+ if (hasTs && !missingTs) return { status: "all_present" };
+ if (!hasTs && missingTs) return { status: "all_missing" };
+ // No messages at all — treat as all_missing (will be caught by session validation)
+ return { status: "all_missing" };
+}
+
+// ============================
+// Normalization
+// ============================
+
+function normalizeSessions(
+ sessions: RawSession[],
+ fallbackSessionKey?: string,
+): { sessions: NormalizedSession[]; totalRounds: number; totalMessages: number } {
+ const normalized: NormalizedSession[] = [];
+ let totalRounds = 0;
+ let totalMessages = 0;
+
+ for (let si = 0; si < sessions.length; si++) {
+ const raw = sessions[si]!;
+
+ const sessionKey = raw.sessionKey || fallbackSessionKey || "seed-user";
+ const sessionId = raw.sessionId || crypto.randomUUID();
+
+ const rounds: NormalizedRound[] = [];
+ for (const rawRound of raw.conversations) {
+ if (!Array.isArray(rawRound)) continue;
+
+ const messages: NormalizedMessage[] = rawRound.map((msg) => ({
+ role: msg.role,
+ content: msg.content,
+ // Normalize timestamp: ISO string → epoch ms, number → pass-through, missing → 0 (filled later)
+ timestamp: msg.timestamp == null
+ ? 0
+ : typeof msg.timestamp === "string"
+ ? new Date(msg.timestamp).getTime()
+ : msg.timestamp,
+ }));
+
+ rounds.push({ messages });
+ totalMessages += messages.length;
+ }
+
+ totalRounds += rounds.length;
+ normalized.push({
+ sessionKey,
+ sessionId,
+ rounds,
+ sourceIndex: si,
+ });
+ }
+
+ return { sessions: normalized, totalRounds, totalMessages };
+}
diff --git a/src/seed/seed-runtime.ts b/src/seed/seed-runtime.ts
new file mode 100644
index 0000000..c53fae7
--- /dev/null
+++ b/src/seed/seed-runtime.ts
@@ -0,0 +1,394 @@
+/**
+ * Seed runtime: L0→L1→L2→L3 orchestration for the `seed` command.
+ *
+ * Uses the shared pipeline-factory for VectorStore/EmbeddingService init,
+ * L1 runner, L2 runner, L3 runner, and persister wiring — keeping this
+ * module focused on seed-specific concerns:
+ * - Synchronous per-round L0 capture with progress reporting
+ * - waitForL1Idle polling (L1 only — see FIXME below)
+ * - Ctrl+C graceful shutdown
+ *
+ * FIXME: Currently we only wait for L1 to become idle before destroying the
+ * pipeline. L2 (scene extraction) and L3 (persona generation) may still be
+ * in-flight when `pipeline.destroy()` is called. This is intentional for now
+ * to avoid excessively long seed runs, but means seed output may not include
+ * the latest L2/L3 artifacts. Re-evaluate adding a full L1+L2+L3 idle wait
+ * once pipeline-manager exposes reliable L2/L3 idle signals.
+ */
+
+import path from "node:path";
+import { parseConfig } from "../config.js";
+import type { MemoryTdaiConfig } from "../config.js";
+import { performAutoCapture } from "../hooks/auto-capture.js";
+import { createPipeline, createL2Runner, createL3Runner } from "../utils/pipeline-factory.js";
+import type { PipelineInstance, PipelineLogger } from "../utils/pipeline-factory.js";
+import { readManifest, writeManifest } from "../utils/manifest.js";
+import type { MemoryPipelineManager } from "../utils/pipeline-manager.js";
+import type {
+ NormalizedInput,
+ SeedProgress,
+ SeedSummary,
+} from "./types.js";
+
+const TAG = "[memory-tdai] [seed]";
+
+// ============================
+// Seed pipeline options
+// ============================
+
+export interface SeedRuntimeOptions {
+ /** Directory to store all seed output (L0, checkpoint, vectors.db). */
+ outputDir: string;
+ /** OpenClaw config object (needed for LLM calls in L1). */
+ openclawConfig: unknown;
+ /** Raw plugin config (same shape as api.pluginConfig). */
+ pluginConfig?: Record;
+ /** Original input file path (for manifest traceability). */
+ inputFile?: string;
+ /** Logger instance. */
+ logger: PipelineLogger;
+ /** Progress callback (called after each round). */
+ onProgress?: (progress: SeedProgress) => void;
+}
+
+// ============================
+// Seed pipeline creation
+// ============================
+
+/**
+ * Create a seed pipeline using the shared factory, with L2/L3 runners
+ * wired via shared factory functions (same logic as index.ts live runtime).
+ */
+async function createSeedPipeline(opts: SeedRuntimeOptions): Promise<{ pipeline: PipelineInstance; cfg: MemoryTdaiConfig }> {
+ const { outputDir, openclawConfig, pluginConfig, logger } = opts;
+
+ // Parse config — all values come from pluginConfig (or parseConfig defaults)
+ const cfg = parseConfig(pluginConfig);
+
+ logger.info(
+ `${TAG} Creating seed pipeline: outputDir=${outputDir}, ` +
+ `everyN=${cfg.pipeline.everyNConversations}, l1Idle=${cfg.pipeline.l1IdleTimeoutSeconds}s, ` +
+ `l2Delay=${cfg.pipeline.l2DelayAfterL1Seconds}s, l2Min=${cfg.pipeline.l2MinIntervalSeconds}s, l2Max=${cfg.pipeline.l2MaxIntervalSeconds}s`,
+ );
+
+ // Use shared factory for everything: store init, L1 runner, persister, destroy
+ const pipeline = await createPipeline({
+ pluginDataDir: outputDir,
+ cfg,
+ openclawConfig,
+ logger,
+ });
+
+ // Wire L2 runner via shared factory (same logic as index.ts live runtime)
+ pipeline.scheduler.setL2Runner(createL2Runner({
+ pluginDataDir: outputDir,
+ cfg,
+ openclawConfig,
+ vectorStore: pipeline.vectorStore,
+ logger,
+ }));
+
+ // Wire L3 runner via shared factory (same logic as index.ts live runtime)
+ pipeline.scheduler.setL3Runner(createL3Runner({
+ pluginDataDir: outputDir,
+ cfg,
+ openclawConfig,
+ vectorStore: pipeline.vectorStore,
+ logger,
+ }));
+
+ return { pipeline, cfg };
+}
+
+// ============================
+// waitForL1Idle
+// ============================
+
+/**
+ * Poll pipeline queue status until L1 is idle for a given session.
+ * Modeled after benchmark-ingest.ts waitForPipelineIdle() but focused on L1 only.
+ */
+async function waitForL1Idle(
+ scheduler: MemoryPipelineManager,
+ sessionKeys: string[],
+ logger: PipelineLogger,
+ opts: {
+ pollIntervalMs?: number;
+ stableRounds?: number;
+ maxWaitMs?: number;
+ } = {},
+): Promise {
+ const pollInterval = opts.pollIntervalMs ?? 1_000;
+ const stableRounds = opts.stableRounds ?? 3;
+ const maxWait = opts.maxWaitMs ?? 300_000; // 5 min default
+
+ const startTime = Date.now();
+ let consecutiveIdle = 0;
+
+ while (true) {
+ const elapsed = Date.now() - startTime;
+ if (elapsed > maxWait) {
+ logger.warn(`${TAG} [waitL1] Max wait time reached (${(maxWait / 1000).toFixed(0)}s), proceeding`);
+ break;
+ }
+
+ const queues = scheduler.getQueueSizes();
+
+ // Check per-session: buffered messages + conversation count
+ let totalBuffered = 0;
+ let totalConversationCount = 0;
+ for (const key of sessionKeys) {
+ totalBuffered += scheduler.getBufferedMessageCount(key);
+ const state = scheduler.getSessionState(key);
+ if (state) {
+ totalConversationCount += state.conversation_count;
+ }
+ }
+
+ const isIdle =
+ queues.l1Idle &&
+ totalBuffered === 0 &&
+ totalConversationCount === 0;
+
+ if (isIdle) {
+ consecutiveIdle++;
+ if (consecutiveIdle >= stableRounds) {
+ logger.debug?.(`${TAG} [waitL1] L1 stable for ${stableRounds} consecutive polls`);
+ return;
+ }
+ } else {
+ consecutiveIdle = 0;
+ logger.debug?.(
+ `${TAG} [waitL1] Waiting: l1Queue=${queues.l1}, l1Pending=${queues.l1Pending}, l1Idle=${queues.l1Idle}, ` +
+ `buffered=${totalBuffered}, convCount=${totalConversationCount}`,
+ );
+ }
+
+ await new Promise((resolve) => setTimeout(resolve, pollInterval));
+ }
+}
+
+// ============================
+// Main execution function
+// ============================
+
+/**
+ * Execute the seed pipeline: feed normalized input through L0 → L1.
+ *
+ * L2/L3 runners are wired but their completion is **not** awaited — see the
+ * module-level FIXME. The pipeline is destroyed after L1 idle, so L2/L3 may
+ * be interrupted mid-run.
+ *
+ * This is the core runtime called by `src/cli/commands/seed.ts` after
+ * all input validation and user confirmation are complete.
+ */
+export async function executeSeed(
+ input: NormalizedInput,
+ opts: SeedRuntimeOptions,
+): Promise {
+ const { logger, onProgress } = opts;
+ const startTime = Date.now();
+
+ // Track interrupt signal
+ let interrupted = false;
+ const onSigint = () => {
+ if (interrupted) {
+ // Second Ctrl+C — force exit
+ logger.warn(`${TAG} Force exit (second Ctrl+C)`);
+ process.exit(1);
+ }
+ interrupted = true;
+ logger.warn(`${TAG} Interrupt received, finishing current round and shutting down...`);
+ };
+ process.on("SIGINT", onSigint);
+
+ let pipeline: PipelineInstance | undefined;
+ let totalL0Recorded = 0;
+ let roundsProcessed = 0;
+
+ try {
+ // Create and start pipeline (returns both the pipeline instance and the
+ // seed-optimized config so we don't need to parse config again)
+ const seed = await createSeedPipeline(opts);
+ pipeline = seed.pipeline;
+ const seedCfg = seed.cfg;
+
+ pipeline.scheduler.start({});
+ logger.info(`${TAG} Pipeline started, processing ${input.sessions.length} session(s), ${input.totalRounds} round(s)`);
+
+ // Seed-specific: use 0 so the cold-start guard in captureAtomically()
+ // does NOT filter out historical messages. In live mode Date.now()
+ // prevents the first agent_end from dumping full session history,
+ // but seed intentionally feeds all historical data.
+ const captureStartTimestamp = 0;
+
+ // Process each session → each round
+ // Key invariant: after every everyNConversations rounds we must wait for L1
+ // to finish before feeding more rounds. Without this pause the for-loop
+ // would dump all rounds into L0 back-to-back and L1 would only run once
+ // with the full batch (defeating the "every N" batching semantics).
+ const everyN = seedCfg.pipeline.everyNConversations;
+
+ for (const session of input.sessions) {
+ if (interrupted) break;
+
+ logger.info(`${TAG} Session: key="${session.sessionKey}" id="${session.sessionId}" rounds=${session.rounds.length}`);
+
+ for (let ri = 0; ri < session.rounds.length; ri++) {
+ if (interrupted) break;
+
+ const round = session.rounds[ri]!;
+ roundsProcessed++;
+
+ // Build messages in the format expected by performAutoCapture.
+ // Field must be named "timestamp" (not "ts") because l0-recorder's
+ // extractUserAssistantMessages reads m.timestamp for incremental filtering.
+ const messages = round.messages.map((m) => ({
+ role: m.role,
+ content: m.content,
+ timestamp: m.timestamp,
+ }));
+
+ try {
+ const result = await performAutoCapture({
+ messages,
+ sessionKey: session.sessionKey,
+ sessionId: session.sessionId,
+ cfg: seedCfg,
+ pluginDataDir: opts.outputDir,
+ logger,
+ scheduler: pipeline.scheduler,
+ pluginStartTimestamp: captureStartTimestamp,
+ vectorStore: pipeline.vectorStore,
+ embeddingService: pipeline.embeddingService,
+ });
+
+ totalL0Recorded += result.l0RecordedCount;
+ } catch (err) {
+ logger.error(
+ `${TAG} L0 capture failed for session="${session.sessionKey}" round=${ri}: ` +
+ `${err instanceof Error ? err.message : String(err)}`,
+ );
+ }
+
+ // Report progress
+ onProgress?.({
+ currentRound: roundsProcessed,
+ totalRounds: input.totalRounds,
+ sessionKey: session.sessionKey,
+ stage: "l0_captured",
+ });
+
+ // After every N rounds, wait for the triggered L1 to finish before
+ // feeding the next batch. This keeps L1 batches aligned with the
+ // everyNConversations boundary instead of letting all rounds pile up.
+ const roundInSession = ri + 1; // 1-based
+ if (roundInSession % everyN === 0 && !interrupted) {
+ onProgress?.({
+ currentRound: roundsProcessed,
+ totalRounds: input.totalRounds,
+ sessionKey: session.sessionKey,
+ stage: "l1_waiting",
+ });
+
+ logger.info(
+ `${TAG} Pausing after round ${roundInSession}/${session.rounds.length} ` +
+ `for session="${session.sessionKey}" — waiting for L1 to drain`,
+ );
+
+ await waitForL1Idle(
+ pipeline.scheduler,
+ [session.sessionKey],
+ logger,
+ { pollIntervalMs: 500, stableRounds: 2, maxWaitMs: 120_000 },
+ );
+ }
+ }
+
+ // After all rounds for this session, wait for any residual L1 work
+ // (handles the tail when total rounds is not a multiple of everyN)
+ if (!interrupted) {
+ onProgress?.({
+ currentRound: roundsProcessed,
+ totalRounds: input.totalRounds,
+ sessionKey: session.sessionKey,
+ stage: "l1_waiting",
+ });
+
+ await waitForL1Idle(
+ pipeline.scheduler,
+ [session.sessionKey],
+ logger,
+ { pollIntervalMs: 1_000, stableRounds: 3, maxWaitMs: 300_000 },
+ );
+
+ logger.info(`${TAG} L1 idle for session="${session.sessionKey}"`);
+ }
+ }
+
+ // Final wait for all sessions
+ if (!interrupted) {
+ const allKeys = input.sessions.map((s) => s.sessionKey);
+ logger.info(`${TAG} Final L1 idle wait for all sessions...`);
+ await waitForL1Idle(
+ pipeline.scheduler,
+ allKeys,
+ logger,
+ { pollIntervalMs: 1_000, stableRounds: 3, maxWaitMs: 300_000 },
+ );
+ }
+ } finally {
+ process.removeListener("SIGINT", onSigint);
+
+ // Graceful shutdown
+ if (pipeline) {
+ try {
+ await pipeline.destroy();
+ } catch (err) {
+ logger.error(`${TAG} Pipeline destroy error: ${err instanceof Error ? err.message : String(err)}`);
+ }
+ }
+ }
+
+ const durationMs = Date.now() - startTime;
+
+ const summary: SeedSummary = {
+ sessionsProcessed: input.sessions.length,
+ roundsProcessed,
+ messagesProcessed: input.totalMessages,
+ l0RecordedCount: totalL0Recorded,
+ durationMs,
+ outputDir: opts.outputDir,
+ };
+
+ if (interrupted) {
+ logger.warn(`${TAG} Seed interrupted after ${roundsProcessed}/${input.totalRounds} rounds`);
+ } else {
+ logger.info(
+ `${TAG} Seed complete: sessions=${summary.sessionsProcessed}, ` +
+ `rounds=${summary.roundsProcessed}, messages=${summary.messagesProcessed}, ` +
+ `l0Recorded=${summary.l0RecordedCount}, duration=${(durationMs / 1000).toFixed(1)}s`,
+ );
+ }
+
+ // Append seed info to manifest (non-fatal if it fails)
+ try {
+ const manifest = readManifest(opts.outputDir);
+ if (manifest) {
+ manifest.seed = {
+ inputFile: opts.inputFile ? path.basename(opts.inputFile) : undefined,
+ sessions: summary.sessionsProcessed,
+ rounds: summary.roundsProcessed,
+ messages: summary.messagesProcessed,
+ startedAt: new Date(startTime).toISOString(),
+ completedAt: new Date().toISOString(),
+ };
+ writeManifest(opts.outputDir, manifest);
+ logger.info(`${TAG} Manifest updated with seed info`);
+ }
+ } catch (err) {
+ logger.warn(`${TAG} Failed to update manifest with seed info (non-fatal): ${err instanceof Error ? err.message : String(err)}`);
+ }
+
+ return summary;
+}
diff --git a/src/seed/types.ts b/src/seed/types.ts
new file mode 100644
index 0000000..2cb4ab8
--- /dev/null
+++ b/src/seed/types.ts
@@ -0,0 +1,140 @@
+/**
+ * Shared type definitions for the `seed` command.
+ *
+ * Covers:
+ * - Raw input shapes (Format A / B / JSONL)
+ * - Normalized internal structures
+ * - Validation error descriptors
+ */
+
+// ============================
+// Raw input types (before validation)
+// ============================
+
+/** A single message in a conversation round. */
+export interface RawMessage {
+ role: string;
+ content: string;
+ /**
+ * Epoch milliseconds (number) **or** ISO 8601 string (e.g. `"2024-04-01T12:00:00Z"`).
+ * ISO strings are parsed via `new Date()` during normalization and
+ * stored internally as epoch ms.
+ */
+ timestamp?: number | string;
+}
+
+/** A single session entry (shared between Format A wrapper and Format B array). */
+export interface RawSession {
+ sessionKey: string;
+ sessionId?: string;
+ conversations: RawMessage[][];
+}
+
+/** Format A: `{ sessions: [...] }` */
+export interface FormatA {
+ sessions: RawSession[];
+}
+
+/** Format B: `[...]` (top-level array of sessions) */
+export type FormatB = RawSession[];
+
+// ============================
+// Normalized types (after validation)
+// ============================
+
+export interface NormalizedMessage {
+ role: string;
+ content: string;
+ /** Epoch ms — always present after normalization (filled if originally missing). */
+ timestamp: number;
+}
+
+export interface NormalizedRound {
+ messages: NormalizedMessage[];
+}
+
+export interface NormalizedSession {
+ sessionKey: string;
+ sessionId: string;
+ rounds: NormalizedRound[];
+ /** Index in the original input array (for progress reporting). */
+ sourceIndex: number;
+}
+
+export interface NormalizedInput {
+ sessions: NormalizedSession[];
+ /** Total number of rounds across all sessions. */
+ totalRounds: number;
+ /** Total number of messages across all sessions. */
+ totalMessages: number;
+ /** Whether timestamps were present in the original input. */
+ hasTimestamps: boolean;
+}
+
+// ============================
+// Validation
+// ============================
+
+/** Stages where a validation error can occur. */
+export type ValidationStage =
+ | "file"
+ | "top_level"
+ | "session"
+ | "round"
+ | "message"
+ | "timestamp_consistency";
+
+/** A single validation error with location context. */
+export interface ValidationError {
+ stage: ValidationStage;
+ sourceIndex?: number;
+ sessionKey?: string;
+ roundIndex?: number;
+ messageIndex?: number;
+ message: string;
+}
+
+// ============================
+// Seed command options (from CLI)
+// ============================
+
+export interface SeedCommandOptions {
+ /** Path to input file (required). */
+ input: string;
+ /** Output directory (optional, auto-generated if missing). */
+ outputDir?: string;
+ /** Fallback session key when input lacks one. */
+ sessionKey?: string;
+ /** Strict round-role validation (each round must have user + assistant). */
+ strictRoundRole: boolean;
+ /** Skip interactive confirmations. */
+ yes: boolean;
+ /** Path to memory-tdai config override file (JSON, deep-merged on top of current plugin config). */
+ configFile?: string;
+}
+
+// ============================
+// Seed runtime types
+// ============================
+
+/** Progress info emitted during seed execution. */
+export interface SeedProgress {
+ /** Current round index (1-based, across all sessions). */
+ currentRound: number;
+ /** Total rounds. */
+ totalRounds: number;
+ /** Current session key. */
+ sessionKey: string;
+ /** Current stage description. */
+ stage: string;
+}
+
+/** Final summary after seed completes. */
+export interface SeedSummary {
+ sessionsProcessed: number;
+ roundsProcessed: number;
+ messagesProcessed: number;
+ l0RecordedCount: number;
+ durationMs: number;
+ outputDir: string;
+}
diff --git a/src/store/bm25-client.ts b/src/store/bm25-client.ts
new file mode 100644
index 0000000..031c2d2
--- /dev/null
+++ b/src/store/bm25-client.ts
@@ -0,0 +1,168 @@
+/**
+ * BM25 Sparse Vector Encoding Client.
+ *
+ * HTTP client for the BM25 Python sidecar service (bm25_server.py).
+ * Used by TCVDB backend to generate sparse vectors for hybridSearch.
+ *
+ * Two operations:
+ * - `encodeTexts(texts)` — encode documents for upsert (TF-based)
+ * - `encodeQueries(texts)` — encode queries for search (IDF-based)
+ *
+ * Graceful degradation: if the sidecar is unreachable, all methods
+ * return empty arrays and `isHealthy()` returns false. Callers can
+ * check health to dynamically downgrade to pure semantic search.
+ */
+
+// ============================
+// Types
+// ============================
+
+/** Sparse vector: array of [token_hash, weight] pairs. */
+export type SparseVector = Array<[number, number]>;
+
+interface Logger {
+ debug?: (message: string) => void;
+ info: (message: string) => void;
+ warn: (message: string) => void;
+ error: (message: string) => void;
+}
+
+export interface BM25ClientConfig {
+ /** Sidecar service URL (default: "http://127.0.0.1:8084") */
+ serviceUrl: string;
+ /** Request timeout in ms (default: 5000) */
+ timeout: number;
+}
+
+interface EncodeResponse {
+ vectors: SparseVector[];
+}
+
+// ============================
+// Implementation
+// ============================
+
+const TAG = "[memory-tdai][bm25-client]";
+
+export class BM25Client {
+ private readonly baseUrl: string;
+ private readonly timeout: number;
+ private readonly logger?: Logger;
+
+ /** Cached health status to avoid repeated checks on every call. */
+ private _healthy: boolean | undefined;
+ private _lastHealthCheck = 0;
+ private static readonly HEALTH_CHECK_INTERVAL_MS = 30_000; // re-check every 30s
+
+ constructor(config: BM25ClientConfig, logger?: Logger) {
+ this.baseUrl = config.serviceUrl.replace(/\/+$/, "");
+ this.timeout = config.timeout;
+ this.logger = logger;
+ }
+
+ /**
+ * Encode document texts for upsert (TF-based BM25 scoring).
+ * Returns one SparseVector per input text.
+ * Returns empty array on error (non-throwing).
+ */
+ async encodeTexts(texts: string[]): Promise {
+ if (texts.length === 0) return [];
+ return this._encode("/encode_texts", texts);
+ }
+
+ /**
+ * Encode query texts for search (IDF-based BM25 scoring).
+ * Returns one SparseVector per input text.
+ * Returns empty array on error (non-throwing).
+ */
+ async encodeQueries(texts: string[]): Promise {
+ if (texts.length === 0) return [];
+ return this._encode("/encode_queries", texts);
+ }
+
+ /**
+ * Check if the BM25 sidecar is reachable.
+ * Result is cached for 30 seconds to avoid spamming health checks.
+ */
+ async isHealthy(): Promise {
+ const now = Date.now();
+ if (
+ this._healthy !== undefined &&
+ now - this._lastHealthCheck < BM25Client.HEALTH_CHECK_INTERVAL_MS
+ ) {
+ return this._healthy;
+ }
+
+ try {
+ const resp = await fetch(`${this.baseUrl}/health`, {
+ signal: AbortSignal.timeout(3000),
+ });
+ this._healthy = resp.ok;
+ } catch {
+ this._healthy = false;
+ }
+ this._lastHealthCheck = now;
+
+ if (!this._healthy) {
+ this.logger?.warn(`${TAG} BM25 sidecar health check failed (${this.baseUrl})`);
+ }
+
+ return this._healthy;
+ }
+
+ // ── Internal ──────────────────────────────────────────────────
+
+ private async _encode(path: string, texts: string[]): Promise {
+ try {
+ const resp = await fetch(`${this.baseUrl}${path}`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ texts }),
+ signal: AbortSignal.timeout(this.timeout),
+ });
+
+ if (!resp.ok) {
+ const errBody = await resp.text().catch(() => "(unreadable)");
+ this.logger?.warn(
+ `${TAG} ${path} HTTP ${resp.status}: ${errBody.slice(0, 200)}`,
+ );
+ return [];
+ }
+
+ const json = (await resp.json()) as EncodeResponse;
+ return json.vectors ?? [];
+ } catch (err) {
+ // Mark unhealthy on connection errors
+ this._healthy = false;
+ this._lastHealthCheck = Date.now();
+
+ this.logger?.warn(
+ `${TAG} ${path} failed: ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return [];
+ }
+ }
+}
+
+// ============================
+// Factory
+// ============================
+
+/**
+ * Create a BM25Client if BM25 is enabled in config.
+ * Returns undefined if disabled — callers should check before using.
+ */
+export function createBM25Client(
+ config: { enabled: boolean; serviceUrl: string; timeout: number },
+ logger?: Logger,
+): BM25Client | undefined {
+ if (!config.enabled) {
+ logger?.info(`${TAG} BM25 sparse encoding disabled`);
+ return undefined;
+ }
+ logger?.info(`${TAG} BM25 client → ${config.serviceUrl}`);
+ return new BM25Client(
+ { serviceUrl: config.serviceUrl, timeout: config.timeout },
+ logger,
+ );
+}
diff --git a/src/store/bm25-local.ts b/src/store/bm25-local.ts
new file mode 100644
index 0000000..548db4f
--- /dev/null
+++ b/src/store/bm25-local.ts
@@ -0,0 +1,97 @@
+/**
+ * Local BM25 Sparse Vector Encoder.
+ *
+ * Pure TypeScript replacement for the Python sidecar BM25 client.
+ * Uses @tencentdb-agent-memory/tcvdb-text package for tokenization (jieba-wasm) and BM25 encoding.
+ *
+ * Two operations (same contract as the old BM25Client):
+ * - `encodeTexts(texts)` — encode documents for upsert (TF-based)
+ * - `encodeQueries(texts)` — encode queries for search (IDF-based)
+ */
+
+import { BM25Encoder } from "@tencentdb-agent-memory/tcvdb-text";
+import type { SparseVector } from "@tencentdb-agent-memory/tcvdb-text";
+
+export type { SparseVector };
+
+interface Logger {
+ debug?: (message: string) => void;
+ info: (message: string) => void;
+ warn: (message: string) => void;
+ error: (message: string) => void;
+}
+
+export interface BM25LocalConfig {
+ /** Whether BM25 sparse encoding is enabled (default: true) */
+ enabled: boolean;
+ /** Language for BM25 pre-trained params: "zh" or "en" (default: "zh") */
+ language?: "zh" | "en";
+}
+
+const TAG = "[memory-tdai][bm25-local]";
+
+// ============================
+// Implementation
+// ============================
+
+export class BM25LocalEncoder {
+ private readonly encoder: BM25Encoder;
+ private readonly logger?: Logger;
+
+ constructor(language: "zh" | "en" = "zh", logger?: Logger) {
+ this.logger = logger;
+ this.encoder = BM25Encoder.default(language);
+ logger?.debug?.(`${TAG} Initialized BM25 local encoder (language=${language})`);
+ }
+
+ /**
+ * Encode document texts for upsert (TF-based BM25 scoring).
+ * Returns one SparseVector per input text.
+ */
+ encodeTexts(texts: string[]): SparseVector[] {
+ if (texts.length === 0) return [];
+ try {
+ return this.encoder.encodeTexts(texts);
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} encodeTexts failed: ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return [];
+ }
+ }
+
+ /**
+ * Encode query texts for search (IDF-based BM25 scoring).
+ * Returns one SparseVector per input text.
+ */
+ encodeQueries(texts: string[]): SparseVector[] {
+ if (texts.length === 0) return [];
+ try {
+ return this.encoder.encodeQueries(texts);
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} encodeQueries failed: ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return [];
+ }
+ }
+}
+
+// ============================
+// Factory
+// ============================
+
+/**
+ * Create a BM25LocalEncoder if BM25 is enabled in config.
+ * Returns undefined if disabled — callers should check before using.
+ */
+export function createBM25Encoder(
+ config: BM25LocalConfig,
+ logger?: Logger,
+): BM25LocalEncoder | undefined {
+ if (!config.enabled) {
+ logger?.debug?.(`${TAG} BM25 sparse encoding disabled`);
+ return undefined;
+ }
+ return new BM25LocalEncoder(config.language ?? "zh", logger);
+}
diff --git a/src/store/embedding.ts b/src/store/embedding.ts
index 531efb3..559a509 100644
--- a/src/store/embedding.ts
+++ b/src/store/embedding.ts
@@ -149,10 +149,20 @@ function sanitizeAndNormalize(vec: number[] | Float32Array): Float32Array {
*/
type LocalInitState = "idle" | "initializing" | "ready" | "failed";
+/** Function that dynamically imports node-llama-cpp. Overridable for testing. */
+export type ImportLlamaFn = () => Promise<{
+ getLlama: (opts: { logLevel: number }) => Promise;
+ resolveModelFile: (model: string, cacheDir?: string) => Promise;
+ LlamaLogLevel: { error: number };
+}>;
+
+const defaultImportLlama: ImportLlamaFn = () => import("node-llama-cpp") as unknown as ReturnType;
+
export class LocalEmbeddingService implements EmbeddingService {
private readonly modelPath: string;
private readonly modelCacheDir?: string;
private readonly logger?: Logger;
+ private readonly importLlama: ImportLlamaFn;
// Initialization state machine
private initState: LocalInitState = "idle";
@@ -162,10 +172,11 @@ export class LocalEmbeddingService implements EmbeddingService {
getEmbeddingFor: (text: string) => Promise<{ vector: Float32Array | number[] }>;
} | null = null;
- constructor(config?: LocalEmbeddingConfig, logger?: Logger) {
+ constructor(config?: LocalEmbeddingConfig, logger?: Logger, importLlama?: ImportLlamaFn) {
this.modelPath = config?.modelPath?.trim() || DEFAULT_LOCAL_MODEL;
this.modelCacheDir = config?.modelCacheDir?.trim();
this.logger = logger;
+ this.importLlama = importLlama ?? defaultImportLlama;
}
getDimensions(): number {
@@ -307,7 +318,7 @@ export class LocalEmbeddingService implements EmbeddingService {
this.logger?.debug?.(`${TAG} Loading node-llama-cpp for local embedding...`);
// Dynamic import — node-llama-cpp is a peer dependency of OpenClaw
- const { getLlama, resolveModelFile, LlamaLogLevel } = await import("node-llama-cpp");
+ const { getLlama, resolveModelFile, LlamaLogLevel } = await this.importLlama();
const llama = await getLlama({ logLevel: LlamaLogLevel.error });
this.logger?.debug?.(`${TAG} Llama instance created`);
@@ -581,18 +592,55 @@ export function createEmbeddingService(
): EmbeddingService {
// Remote OpenAI-compatible provider: any provider value other than "local"
if (config && config.provider !== "local" && "apiKey" in config && config.apiKey) {
- logger?.info(`${TAG} Using remote embedding (provider=${config.provider}, model=${config.model})`);
+ logger?.debug?.(`${TAG} Using remote embedding (provider=${config.provider}, model=${config.model})`);
return new OpenAIEmbeddingService(config as OpenAIEmbeddingConfig, logger);
}
// Explicit local config
if (config && config.provider === "local") {
const localConfig = config as LocalEmbeddingConfig;
- logger?.info(`${TAG} Using local embedding (node-llama-cpp, model=${localConfig.modelPath ?? DEFAULT_LOCAL_MODEL})`);
+ logger?.debug?.(`${TAG} Using local embedding (node-llama-cpp, model=${localConfig.modelPath ?? DEFAULT_LOCAL_MODEL})`);
return new LocalEmbeddingService(localConfig, logger);
}
// Fallback: no config or empty apiKey → use local
- logger?.info(`${TAG} No remote embedding configured, falling back to local embedding (node-llama-cpp)`);
+ logger?.debug?.(`${TAG} No remote embedding configured, falling back to local embedding (node-llama-cpp)`);
return new LocalEmbeddingService(undefined, logger);
}
+
+// ============================
+// NoopEmbeddingService (for server-side embedding backends)
+// ============================
+
+/**
+ * No-op embedding service for backends with built-in server-side embedding
+ * (e.g., TCVDB with Collection-level embedding config).
+ *
+ * All embed() calls return an empty Float32Array because the server generates
+ * vectors automatically from the text field during upsert/search.
+ */
+export class NoopEmbeddingService implements EmbeddingService {
+ embed(_text: string): Promise {
+ return Promise.resolve(new Float32Array(0));
+ }
+
+ embedBatch(texts: string[]): Promise {
+ return Promise.resolve(texts.map(() => new Float32Array(0)));
+ }
+
+ getDimensions(): number {
+ return 0;
+ }
+
+ getProviderInfo(): EmbeddingProviderInfo {
+ return { provider: "noop", model: "server-side" };
+ }
+
+ isReady(): boolean {
+ return true;
+ }
+
+ startWarmup(): void {
+ // no-op
+ }
+}
diff --git a/src/store/factory.ts b/src/store/factory.ts
new file mode 100644
index 0000000..6516aad
--- /dev/null
+++ b/src/store/factory.ts
@@ -0,0 +1,127 @@
+/**
+ * Store Factory — creates the appropriate storage backend and embedding service
+ * based on plugin configuration.
+ *
+ * Supports:
+ * - "sqlite" (default): local SQLite + sqlite-vec + FTS5
+ * - "tcvdb": Tencent Cloud VectorDB (server-side embedding + hybridSearch)
+ */
+
+import path from "node:path";
+import type { MemoryTdaiConfig } from "../config.js";
+import type { IMemoryStore, IEmbeddingService, StoreLogger } from "./types.js";
+import { VectorStore } from "./sqlite.js";
+import { TcvdbMemoryStore } from "./tcvdb.js";
+import { createEmbeddingService, NoopEmbeddingService } from "./embedding.js";
+import type { EmbeddingService } from "./embedding.js";
+import { createBM25Encoder } from "./bm25-local.js";
+import type { BM25LocalEncoder } from "./bm25-local.js";
+
+// Re-export for convenience
+export type { IMemoryStore, IEmbeddingService, StoreLogger, BM25LocalEncoder };
+
+const TAG = "[memory-tdai][factory]";
+
+export interface StoreBundle {
+ store: IMemoryStore;
+ embedding: IEmbeddingService;
+ bm25Encoder?: BM25LocalEncoder;
+ /** Snapshot of current store config for manifest writing. */
+ storeSnapshot: import("../utils/manifest.js").StoreConfigSnapshot;
+}
+
+/**
+ * Create the storage backend, embedding service, and optional BM25 encoder
+ * based on plugin configuration.
+ *
+ * @param config Fully resolved plugin config.
+ * @param options.dataDir Plugin data directory.
+ * @param options.logger Logger instance.
+ */
+export function createStoreBundle(
+ config: MemoryTdaiConfig,
+ options: { dataDir: string; logger?: StoreLogger },
+): StoreBundle {
+ const { logger } = options;
+
+ // ── BM25 local encoder ──
+ const bm25Encoder = createBM25Encoder(config.bm25, logger);
+
+ switch (config.storeBackend) {
+ case "tcvdb": {
+ const tcvdbCfg = config.tcvdb;
+ if (!tcvdbCfg.url || !tcvdbCfg.apiKey) {
+ throw new Error(`${TAG} TCVDB backend requires tcvdb.url and tcvdb.apiKey`);
+ }
+ if (!tcvdbCfg.database) {
+ throw new Error(`${TAG} TCVDB backend requires tcvdb.database — please set a unique database name in your openclaw.json plugin config`);
+ }
+ const database = tcvdbCfg.database;
+ const store = new TcvdbMemoryStore({
+ url: tcvdbCfg.url,
+ username: tcvdbCfg.username,
+ apiKey: tcvdbCfg.apiKey,
+ database,
+ embeddingModel: tcvdbCfg.embeddingModel,
+ timeout: tcvdbCfg.timeout,
+ caPemPath: tcvdbCfg.caPemPath,
+ logger,
+ bm25Encoder: bm25Encoder ?? undefined,
+ });
+
+ logger?.debug?.(
+ `${TAG} Store created: backend=tcvdb, database=${database}, model=${tcvdbCfg.embeddingModel}, ` +
+ `bm25=${bm25Encoder ? "enabled" : "disabled"}`,
+ );
+
+ return {
+ store,
+ embedding: new NoopEmbeddingService(),
+ bm25Encoder,
+ storeSnapshot: {
+ type: "tcvdb",
+ tcvdbUrl: tcvdbCfg.url,
+ tcvdbDatabase: database,
+ tcvdbAlias: tcvdbCfg.alias || undefined,
+ },
+ };
+ }
+
+ case "sqlite":
+ default: {
+ // ── Embedding service (only when enabled) ──
+ let embeddingService: EmbeddingService | undefined;
+ if (config.embedding.enabled && config.embedding.provider !== "local" && config.embedding.apiKey) {
+ embeddingService = createEmbeddingService({
+ provider: config.embedding.provider,
+ baseUrl: config.embedding.baseUrl,
+ apiKey: config.embedding.apiKey,
+ model: config.embedding.model,
+ dimensions: config.embedding.dimensions,
+ maxInputChars: config.embedding.maxInputChars,
+ }, logger);
+ }
+
+ // dimensions from config (0 when provider="none" → vec0 deferred)
+ const dims = config.embedding.dimensions;
+ const dbPath = path.join(options.dataDir, "vectors.db");
+ const store = new VectorStore(dbPath, dims, logger);
+
+ logger?.debug?.(
+ `${TAG} Store created: backend=sqlite, dbPath=${dbPath}, dimensions=${dims}, ` +
+ `embedding=${embeddingService ? "enabled" : "disabled"}, ` +
+ `bm25=${bm25Encoder ? "enabled" : "disabled"}`,
+ );
+
+ return {
+ store,
+ embedding: embeddingService as unknown as IEmbeddingService,
+ bm25Encoder,
+ storeSnapshot: {
+ type: "sqlite",
+ sqlitePath: path.relative(options.dataDir, dbPath),
+ },
+ };
+ }
+ }
+}
diff --git a/src/store/search-utils.ts b/src/store/search-utils.ts
new file mode 100644
index 0000000..edac366
--- /dev/null
+++ b/src/store/search-utils.ts
@@ -0,0 +1,62 @@
+/**
+ * Search utilities — shared helpers for memory search across backends.
+ *
+ * Contains:
+ * - RRF (Reciprocal Rank Fusion) merge — used by SQLite hybrid search
+ * (eliminates the 3x duplication in auto-recall, memory-search, conversation-search)
+ * - FTS query building — re-exported from sqlite for convenience
+ */
+
+// ============================
+// RRF (Reciprocal Rank Fusion)
+// ============================
+
+/**
+ * Standard RRF constant from the original RRF paper.
+ * Higher k → more weight on lower-ranked items (smoother distribution).
+ */
+export const RRF_K = 60;
+
+/**
+ * Merge multiple ranked lists via Reciprocal Rank Fusion.
+ *
+ * Each item's RRF score = sum over all lists of 1/(k + rank + 1).
+ * Items appearing in multiple lists get their scores summed.
+ *
+ * @param lists Array of ranked lists. Each list must have items with an `id` field.
+ * @param k RRF constant (default: 60).
+ * @returns Merged list sorted by descending RRF score, with `rrfScore` attached.
+ *
+ * @example
+ * ```ts
+ * const merged = rrfMerge(
+ * [ftsResults, vecResults],
+ * (item) => item.record_id,
+ * );
+ * ```
+ */
+export function rrfMerge(
+ lists: T[][],
+ getId: (item: T) => string,
+ k: number = RRF_K,
+): Array {
+ const map = new Map();
+
+ for (const list of lists) {
+ for (let rank = 0; rank < list.length; rank++) {
+ const item = list[rank];
+ const id = getId(item);
+ const score = 1 / (k + rank + 1);
+ const existing = map.get(id);
+ if (existing) {
+ existing.rrfScore += score;
+ } else {
+ map.set(id, { item, rrfScore: score });
+ }
+ }
+ }
+
+ return [...map.values()]
+ .sort((a, b) => b.rrfScore - a.rrfScore)
+ .map(({ item, rrfScore }) => ({ ...item, rrfScore }));
+}
diff --git a/src/store/sqlite.ts b/src/store/sqlite.ts
new file mode 100644
index 0000000..a3922a3
--- /dev/null
+++ b/src/store/sqlite.ts
@@ -0,0 +1,2303 @@
+/**
+ * VectorStore: SQLite-based vector storage using sqlite-vec extension.
+ *
+ * Manages two layers of vector-indexed data in a single SQLite database:
+ *
+ * **L1 (structured memories):**
+ * 1. `l1_records` — relational metadata table (content, type, priority, scene, timestamps)
+ * 2. `l1_vec` — vec0 virtual table for cosine similarity search
+ *
+ * **L0 (raw conversations):**
+ * 3. `l0_conversations` — relational metadata table (session_key, role, message text, timestamps)
+ * 4. `l0_vec` — vec0 virtual table for cosine similarity search on individual messages
+ *
+ * Dependencies: Node.js built-in `node:sqlite` (Node 22+) + `sqlite-vec` (from root workspace).
+ *
+ * Design:
+ * - All operations are synchronous (DatabaseSync API).
+ * - Writes use manual BEGIN/COMMIT transactions for atomicity (metadata + vector).
+ * - vec0 virtual table does NOT support ON CONFLICT, so upsert = delete + insert.
+ * - Thread-safe via WAL mode.
+ */
+
+import { createRequire } from "node:module";
+import type { DatabaseSync, StatementSync } from "node:sqlite";
+import type { MemoryRecord } from "../record/l1-writer.js";
+import type { EmbeddingProviderInfo } from "./embedding.js";
+import type {
+ IMemoryStore,
+ StoreCapabilities,
+ L0Record,
+ L1SearchResult,
+ L1FtsResult,
+ L0SearchResult,
+ L0FtsResult,
+} from "./types.js";
+
+// ============================
+// Types
+// ============================
+
+export interface VectorSearchResult {
+ record_id: string;
+ content: string;
+ type: string;
+ priority: number;
+ scene_name: string;
+ /** Cosine similarity score (1.0 - cosine_distance) */
+ score: number;
+ timestamp_str: string;
+ timestamp_start: string;
+ timestamp_end: string;
+ session_key: string;
+ session_id: string;
+ /** Raw metadata JSON string (e.g., contains activity_start_time / activity_end_time for episodic) */
+ metadata_json: string;
+}
+
+/** L0 single-message vector search result. */
+export interface L0VectorSearchResult {
+ record_id: string;
+ session_key: string;
+ session_id: string;
+ role: string;
+ message_text: string;
+ /** Cosine similarity score (1.0 - cosine_distance) */
+ score: number;
+ recorded_at: string;
+ /** Original message timestamp (epoch ms) */
+ timestamp: number;
+}
+
+/** Raw row returned by L1 record queries (column names match SQLite schema). */
+export interface L1RecordRow {
+ record_id: string;
+ content: string;
+ type: string;
+ priority: number;
+ scene_name: string;
+ session_key: string;
+ session_id: string;
+ timestamp_str: string;
+ timestamp_start: string;
+ timestamp_end: string;
+ created_time: string;
+ updated_time: string;
+ metadata_json: string;
+}
+
+export interface L0RecordRow {
+ record_id: string;
+ session_key: string;
+ session_id: string;
+ role: string;
+ message_text: string;
+ recorded_at: string;
+ timestamp: number;
+}
+
+/** Filter options for querying L1 records from SQLite. */
+export interface L1QueryFilter {
+ /** If provided, only return records for this session key (conversation channel). */
+ sessionKey?: string;
+ /** If provided, only return records for this session ID (single conversation instance). */
+ sessionId?: string;
+ /** If provided, only return records with updated_time strictly after this ISO 8601 UTC timestamp. */
+ updatedAfter?: string;
+}
+
+interface Logger {
+ debug?: (message: string) => void;
+ info: (message: string) => void;
+ warn: (message: string) => void;
+ error: (message: string) => void;
+}
+
+const TAG = "[memory-tdai][sqlite]";
+
+/** Persisted metadata about the embedding provider used to generate stored vectors. */
+interface EmbeddingMeta {
+ provider: string;
+ model: string;
+ dimensions: number;
+}
+
+/** Result of VectorStore.init() — indicates whether a re-embed is needed. */
+export interface VectorStoreInitResult {
+ /**
+ * `true` if the embedding provider/model/dimensions changed since
+ * the vectors were last written. Callers should re-embed all texts
+ * (via `reindexAll()`) after receiving this flag.
+ */
+ needsReindex: boolean;
+ /** Human-readable reason (for logging). */
+ reason?: string;
+}
+
+// Use createRequire to load the experimental node:sqlite module
+const require = createRequire(import.meta.url);
+
+function requireNodeSqlite(): typeof import("node:sqlite") {
+ return require("node:sqlite") as typeof import("node:sqlite");
+}
+
+// ============================
+// FTS5 helpers (adapted from openclaw core hybrid.ts)
+// ============================
+
+// ── Chinese word segmentation (jieba) ──
+// Lazy-loaded singleton: initialised on first call to `buildFtsQuery`.
+// If @node-rs/jieba is unavailable, falls back to Unicode-regex splitting.
+
+interface JiebaInstance {
+ cutForSearch(text: string, hmm: boolean): string[];
+}
+
+let _jieba: JiebaInstance | null | undefined; // undefined = not yet tried
+
+function getJieba(): JiebaInstance | null {
+ if (_jieba !== undefined) return _jieba;
+ try {
+ // eslint-disable-next-line @typescript-eslint/no-require-imports
+ const { Jieba } = require("@node-rs/jieba");
+ // eslint-disable-next-line @typescript-eslint/no-require-imports
+ const { dict } = require("@node-rs/jieba/dict");
+ _jieba = Jieba.withDict(dict) as JiebaInstance;
+ } catch {
+ _jieba = null; // mark as unavailable — won't retry
+ }
+ return _jieba;
+}
+
+/**
+ * Common Chinese stop-words that add noise to FTS5 queries.
+ * Kept small on purpose — only high-frequency function words.
+ */
+const ZH_STOP_WORDS = new Set([
+ "的", "了", "在", "是", "我", "有", "和", "就", "不", "人", "都", "一",
+ "一个", "上", "也", "很", "到", "说", "要", "去", "你", "会", "着",
+ "没有", "看", "好", "自己", "这", "他", "她", "它", "们", "那",
+ "吗", "吧", "呢", "啊", "呀", "哦", "嗯",
+]);
+
+/**
+ * Build an FTS5 MATCH query from raw text.
+ *
+ * When `@node-rs/jieba` is available, uses jieba's search-engine mode
+ * (`cutForSearch`) for accurate Chinese word segmentation, producing
+ * much better recall than the previous regex-only approach.
+ *
+ * Falls back to Unicode-regex splitting (`/[\p{L}\p{N}_]+/gu`) if
+ * jieba is not installed.
+ *
+ * Tokens are OR-joined as quoted FTS5 phrase terms so that a document
+ * matching *any* token is returned. BM25 naturally ranks documents that
+ * match more tokens higher, so precision is preserved while recall is
+ * significantly improved — especially for longer queries and when running
+ * in FTS-only fallback mode (no embedding available).
+ *
+ * Example (with jieba):
+ * "用户喜欢编程和TypeScript" → '"用户" OR "喜欢" OR "编程" OR "TypeScript"'
+ * Example (fallback):
+ * "旅行计划 API" → '"旅行计划" OR "API"'
+ */
+export function buildFtsQuery(raw: string): string | null {
+ const jieba = getJieba();
+
+ let tokens: string[];
+ if (jieba) {
+ // jieba cutForSearch: splits long words further for better recall
+ // e.g. "北京烤鸭" → ["北京", "烤鸭", "北京烤鸭"]
+ tokens = jieba
+ .cutForSearch(raw, true)
+ .map((t) => t.trim())
+ .filter((t) => {
+ if (!t) return false;
+ // Remove pure whitespace / punctuation tokens
+ if (!/[\p{L}\p{N}]/u.test(t)) return false;
+ // Remove common Chinese stop-words to reduce noise
+ if (ZH_STOP_WORDS.has(t)) return false;
+ return true;
+ });
+ // Deduplicate (cutForSearch may produce duplicates for sub-words)
+ tokens = [...new Set(tokens)];
+ } else {
+ // Fallback: simple Unicode regex split
+ tokens =
+ raw
+ .match(/[\p{L}\p{N}_]+/gu)
+ ?.map((t) => t.trim())
+ .filter(Boolean) ?? [];
+ }
+
+ if (tokens.length === 0) return null;
+ const quoted = tokens.map((t) => `"${t.replaceAll('"', "")}"`);
+ return quoted.join(" OR ");
+}
+
+/**
+ * Tokenize text for FTS5 indexing (write-side).
+ *
+ * Uses jieba `cutForSearch()` (search-engine mode) to segment Chinese text,
+ * then joins tokens with spaces. The resulting string is stored in the FTS5
+ * `content` column so that `unicode61` tokenizer can split it into meaningful
+ * words — including both full words and their sub-words.
+ *
+ * Using `cutForSearch` (instead of `cut`) ensures that the index contains
+ * the same sub-word tokens that `buildFtsQuery()` produces on the query side.
+ * For example, "人工智能" is indexed as "人工 智能 人工智能", so queries for
+ * either the full term or sub-words will match.
+ *
+ * Falls back to the original text if jieba is unavailable.
+ *
+ * Example (with jieba):
+ * "用户五月去日本旅行" → "用户 五月 去 日本 旅行"
+ * "人工智能的分支" → "人工 智能 人工智能 的 分支"
+ * Example (fallback):
+ * "用户五月去日本旅行" → "用户五月去日本旅行" (unchanged)
+ */
+export function tokenizeForFts(raw: string): string {
+ const jieba = getJieba();
+ if (!jieba) return raw;
+
+ // Use `cutForSearch` (search-engine mode) for indexing — it produces both
+ // full words AND their sub-word components. This ensures that query-side
+ // tokens (also produced by `cutForSearch` in `buildFtsQuery`) will always
+ // find a match in the index.
+ const tokens = jieba.cutForSearch(raw, true);
+
+ // Join with spaces so `unicode61` tokenizer can split them.
+ // Punctuation tokens are kept — unicode61 treats them as separators anyway.
+ return tokens.join(" ");
+}
+
+/**
+ * Reset jieba state so next call to `buildFtsQuery` re-initialises.
+ * Exported for testing only.
+ * @internal
+ */
+export function _resetJiebaForTest(): void {
+ _jieba = undefined;
+}
+
+/**
+ * Override jieba instance (or set to `null` to force fallback).
+ * Exported for testing only.
+ * @internal
+ */
+export function _setJiebaForTest(instance: JiebaInstance | null): void {
+ _jieba = instance;
+}
+
+/**
+ * Convert a BM25 rank (negative = more relevant) to a 0–1 score.
+ * Mirrors the formula in openclaw core `hybrid.ts`.
+ */
+export function bm25RankToScore(rank: number): number {
+ if (!Number.isFinite(rank)) return 1 / (1 + 999);
+ if (rank < 0) {
+ const relevance = -rank;
+ return relevance / (1 + relevance);
+ }
+ return 1 / (1 + rank);
+}
+
+/** FTS5 search result for L1 records. */
+export interface FtsSearchResult {
+ record_id: string;
+ content: string;
+ type: string;
+ priority: number;
+ scene_name: string;
+ /** BM25-derived score (0–1, higher is better) */
+ score: number;
+ timestamp_str: string;
+ timestamp_start: string;
+ timestamp_end: string;
+ session_key: string;
+ session_id: string;
+ metadata_json: string;
+}
+
+/** FTS5 search result for L0 records. */
+export interface L0FtsSearchResult {
+ record_id: string;
+ session_key: string;
+ session_id: string;
+ role: string;
+ message_text: string;
+ /** BM25-derived score (0–1, higher is better) */
+ score: number;
+ recorded_at: string;
+ timestamp: number;
+}
+
+// ============================
+// VectorStore class
+// ============================
+
+export class VectorStore implements IMemoryStore {
+ private db: DatabaseSync;
+ private readonly dimensions: number;
+ private readonly logger?: Logger;
+
+ /** @see IMemoryStore.supportsDeferredEmbedding */
+ readonly supportsDeferredEmbedding = true;
+
+ /**
+ * When `true`, the store is in a degraded state (e.g. sqlite-vec failed to
+ * load, or init() encountered an unrecoverable error). All public methods
+ * become safe no-ops so the plugin never blocks the main OpenClaw flow.
+ */
+ private degraded = false;
+
+ /** Tracks whether close() has been called to prevent double-close errors. */
+ private closed = false;
+
+ /**
+ * `true` when vec0 virtual tables (l1_vec / l0_vec) have been created and
+ * their prepared statements are ready. When `dimensions === 0` (i.e.
+ * provider="none"), vec0 tables are deferred and this stays `false`.
+ */
+ private vecTablesReady = false;
+
+ // Prepared statements — L1 (initialized in init())
+ private stmtUpsertMeta!: StatementSync;
+ private stmtDeleteVec?: StatementSync; // optional — only set when vecTablesReady
+ private stmtInsertVec?: StatementSync; // optional — only set when vecTablesReady
+ private stmtDeleteMeta!: StatementSync;
+ private stmtGetMeta!: StatementSync;
+ private stmtSearchVec?: StatementSync; // optional — only set when vecTablesReady
+ private stmtQueryBySessionId!: StatementSync;
+ private stmtQueryBySessionIdSince!: StatementSync;
+ private stmtQueryBySessionKey!: StatementSync;
+ private stmtQueryBySessionKeySince!: StatementSync;
+ private stmtQueryAll!: StatementSync;
+ private stmtQueryAllSince!: StatementSync;
+
+ // Prepared statements — L0 (initialized in init())
+ private stmtL0UpsertMeta!: StatementSync;
+ private stmtL0DeleteVec?: StatementSync; // optional — only set when vecTablesReady
+ private stmtL0InsertVec?: StatementSync; // optional — only set when vecTablesReady
+ private stmtL0DeleteMeta!: StatementSync;
+ private stmtL0GetMeta!: StatementSync;
+ private stmtL0SearchVec?: StatementSync; // optional — only set when vecTablesReady
+ /** L0 query for L1 runner: all messages for a session key */
+ private stmtL0QueryAll!: StatementSync;
+ /** L0 query for L1 runner: messages after a timestamp cursor */
+ private stmtL0QueryAfter!: StatementSync;
+ /** L1 cursor-based pagination for migration (by PK) */
+ private stmtL1QueryMigrationCursor!: StatementSync;
+ /** L0 cursor-based pagination for migration (by PK) */
+ private stmtL0QueryMigrationCursor!: StatementSync;
+
+ // FTS5 tables availability flag (created best-effort — may be false if fts5 is not compiled in)
+ private ftsAvailable = false;
+
+ // Prepared statements — FTS5 L1 (initialized in init())
+ private stmtL1FtsInsert!: StatementSync;
+ private stmtL1FtsDelete!: StatementSync;
+ private stmtL1FtsSearch!: StatementSync;
+
+ // Prepared statements — FTS5 L0 (initialized in init())
+ private stmtL0FtsInsert!: StatementSync;
+ private stmtL0FtsDelete!: StatementSync;
+ private stmtL0FtsSearch!: StatementSync;
+
+ /**
+ * Create a VectorStore instance.
+ *
+ * Note: After construction, you MUST call `init()` to load the sqlite-vec
+ * extension and create the schema.
+ */
+ constructor(dbPath: string, dimensions: number, logger?: Logger) {
+ this.dimensions = dimensions;
+ this.logger = logger;
+
+ // Open database with extension support enabled
+ const { DatabaseSync: DbSync } = requireNodeSqlite();
+ this.db = new DbSync(dbPath, { allowExtension: true });
+
+ // Set busy timeout so concurrent processes retry instead of failing with SQLITE_BUSY
+ this.db.exec("PRAGMA busy_timeout = 5000");
+
+ // Enable WAL mode for better concurrent read performance
+ this.db.exec("PRAGMA journal_mode = WAL");
+
+ // Cap page cache at 64 MB
+ this.db.exec("PRAGMA cache_size = -65536");
+
+ // Cap memory-mapped I/O at 128 MB to bound RSS growth
+ this.db.exec("PRAGMA mmap_size = 134217728");
+
+ // Auto-checkpoint WAL every 1000 pages (~4 MB) to keep WAL file compact
+ this.db.exec("PRAGMA wal_autocheckpoint = 1000");
+ }
+
+ /**
+ * Whether the store is in degraded mode (e.g. sqlite-vec failed to load).
+ * When degraded, all write/search operations become safe no-ops.
+ */
+ isDegraded(): boolean {
+ return this.degraded;
+ }
+
+
+ /**
+ * Load sqlite-vec extension and initialize database schema.
+ * Must be called once after construction.
+ *
+ * @param providerInfo Current embedding provider info. When provided,
+ * the store compares it against the persisted metadata. If the provider,
+ * model, or dimensions changed, the vector tables are dropped and
+ * re-created with the new dimensions, and `needsReindex: true` is returned
+ * so the caller can schedule a full re-embed.
+ */
+ init(providerInfo?: EmbeddingProviderInfo): VectorStoreInitResult {
+ // Load sqlite-vec extension (same approach as root project's sqlite-vec.ts)
+ try {
+ // eslint-disable-next-line @typescript-eslint/no-require-imports
+ const sqliteVec = require("sqlite-vec");
+ this.db.enableLoadExtension(true);
+ sqliteVec.load(this.db);
+ } catch (err) {
+ const message = err instanceof Error ? err.message : String(err);
+ this.logger?.error(
+ `${TAG} Failed to load sqlite-vec extension: ${message}. ` +
+ `VectorStore entering degraded mode — all operations will be no-ops.`,
+ );
+ this.degraded = true;
+ return { needsReindex: false, reason: `sqlite-vec load failed: ${message}` };
+ }
+
+ // ── Schema creation & prepared statements ──────────────────────────────
+ // Wrapped in try-catch: if anything fails during schema init (e.g. the DB
+ // is corrupted, disk full, etc.), we degrade gracefully instead of crashing.
+ try {
+ return this.initSchema(providerInfo);
+ } catch (err) {
+ const message = err instanceof Error ? err.message : String(err);
+ this.logger?.error(
+ `${TAG} Schema initialization failed: ${message}. ` +
+ `VectorStore entering degraded mode.`,
+ );
+ this.degraded = true;
+ return { needsReindex: false, reason: `schema init failed: ${message}` };
+ }
+ }
+
+ /**
+ * Internal schema initialization — separated from init() so we can
+ * catch errors at the top level and degrade gracefully.
+ */
+ private initSchema(providerInfo?: EmbeddingProviderInfo): VectorStoreInitResult {
+ // Tracks which provider/model/dimensions were used to generate vectors.
+ this.db.exec(`
+ CREATE TABLE IF NOT EXISTS embedding_meta (
+ key TEXT PRIMARY KEY,
+ value TEXT NOT NULL
+ )
+ `);
+
+ // Detect whether re-index is needed
+ let needsReindex = false;
+ let reindexReason: string | undefined;
+
+ const savedMeta = this.readEmbeddingMeta();
+
+ if (providerInfo) {
+ if (savedMeta) {
+ const providerChanged = savedMeta.provider !== providerInfo.provider;
+ const modelChanged = savedMeta.model !== providerInfo.model;
+ const dimsChanged = savedMeta.dimensions !== this.dimensions;
+
+ if (providerChanged || modelChanged || dimsChanged) {
+ const reasons: string[] = [];
+ if (providerChanged) reasons.push(`provider: ${savedMeta.provider} → ${providerInfo.provider}`);
+ if (modelChanged) reasons.push(`model: ${savedMeta.model} → ${providerInfo.model}`);
+ if (dimsChanged) reasons.push(`dimensions: ${savedMeta.dimensions} → ${this.dimensions}`);
+ reindexReason = reasons.join(", ");
+
+ this.logger?.info(
+ `${TAG} Embedding config changed (${reindexReason}). ` +
+ `Dropping vector tables for rebuild...`,
+ );
+
+ // Drop and re-create vector tables with new dimensions
+ this.dropVectorTables();
+ needsReindex = true;
+ }
+ } else {
+ // No saved meta — first run or legacy DB without meta table.
+ // Two cases require dropping vector tables:
+ // 1. Existing data created without meta tracking (legacy DB) — need re-embed
+ // 2. vec0 tables exist with wrong dimensions (e.g. previously created with
+ // provider="none" placeholder 768D, now switching to a real provider
+ // with different dimensions) — must rebuild even if data tables are empty
+ const l1Count = this.tableRowCount("l1_records");
+ const l0Count = this.tableRowCount("l0_conversations");
+ const existingVecDims = this.getVecTableDimensions();
+
+ if (l1Count > 0 || l0Count > 0) {
+ this.logger?.info(
+ `${TAG} No embedding_meta found but existing data exists ` +
+ `(L1=${l1Count}, L0=${l0Count}). Dropping vector tables for safety...`,
+ );
+ this.dropVectorTables();
+ needsReindex = true;
+ reindexReason = "legacy DB without embedding_meta — cannot verify vector compatibility";
+ } else if (existingVecDims !== null && existingVecDims !== this.dimensions) {
+ // vec0 tables exist (from a previous provider="none" placeholder or
+ // different config) but with mismatched dimensions. Drop them so they
+ // get re-created with the correct dimensions below.
+ this.logger?.info(
+ `${TAG} vec0 table dimension mismatch (existing=${existingVecDims}, ` +
+ `required=${this.dimensions}). Dropping vector tables for rebuild...`,
+ );
+ this.dropVectorTables();
+ // No needsReindex — there's no data to re-embed
+ }
+ }
+ }
+
+ // ── L1 schema ──────────────────────────────────
+
+ // Metadata table
+ this.db.exec(`
+ CREATE TABLE IF NOT EXISTS l1_records (
+ record_id TEXT PRIMARY KEY,
+ content TEXT NOT NULL,
+ type TEXT DEFAULT '',
+ priority INTEGER DEFAULT 50,
+ scene_name TEXT DEFAULT '',
+ session_key TEXT DEFAULT '',
+ session_id TEXT DEFAULT '',
+ timestamp_str TEXT DEFAULT '',
+ timestamp_start TEXT DEFAULT '',
+ timestamp_end TEXT DEFAULT '',
+ created_time TEXT DEFAULT '',
+ updated_time TEXT DEFAULT '',
+ metadata_json TEXT DEFAULT '{}'
+ )
+ `);
+
+ // Indexes for common queries
+ this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_type ON l1_records(type)");
+ this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_session_key ON l1_records(session_key)");
+ this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_session_id ON l1_records(session_id)");
+ this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_scene ON l1_records(scene_name)");
+ this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_ts_start ON l1_records(timestamp_start)");
+ this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_ts_end ON l1_records(timestamp_end)");
+ // Composite index: session_id exact match + updated_time range scan (for incremental L2 queries)
+ this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_session_updated ON l1_records(session_id, updated_time)");
+ // Composite index: session_key exact match + updated_time range scan (for pipeline cursor queries)
+ this.db.exec("CREATE INDEX IF NOT EXISTS idx_l1_sessionkey_updated ON l1_records(session_key, updated_time)");
+
+ // Vector virtual table (cosine distance) — only created when dimensions > 0.
+ // When provider="none", dimensions=0 and vec0 tables are deferred until a
+ // real embedding provider is configured.
+ if (this.dimensions > 0) {
+ this.db.exec(`
+ CREATE VIRTUAL TABLE IF NOT EXISTS l1_vec USING vec0(
+ record_id TEXT PRIMARY KEY,
+ embedding float[${this.dimensions}] distance_metric=cosine,
+ updated_time TEXT DEFAULT ''
+ )
+ `);
+ }
+
+ // Prepare statements for reuse
+ this.stmtUpsertMeta = this.db.prepare(`
+ INSERT INTO l1_records (
+ record_id, content, type, priority, scene_name, session_key, session_id,
+ timestamp_str, timestamp_start, timestamp_end,
+ created_time, updated_time, metadata_json
+ ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+ ON CONFLICT(record_id) DO UPDATE SET
+ content=excluded.content,
+ type=excluded.type,
+ priority=excluded.priority,
+ scene_name=excluded.scene_name,
+ timestamp_str=excluded.timestamp_str,
+ timestamp_start=excluded.timestamp_start,
+ timestamp_end=excluded.timestamp_end,
+ updated_time=excluded.updated_time,
+ metadata_json=excluded.metadata_json
+ `);
+
+ if (this.dimensions > 0) {
+ this.stmtDeleteVec = this.db.prepare("DELETE FROM l1_vec WHERE record_id = ?");
+ this.stmtInsertVec = this.db.prepare("INSERT INTO l1_vec (record_id, embedding, updated_time) VALUES (?, ?, ?)");
+ }
+ this.stmtDeleteMeta = this.db.prepare("DELETE FROM l1_records WHERE record_id = ?");
+
+ this.stmtGetMeta = this.db.prepare(`
+ SELECT content, type, priority, scene_name, session_key, session_id,
+ timestamp_str, timestamp_start, timestamp_end, metadata_json
+ FROM l1_records WHERE record_id = ?
+ `);
+
+ if (this.dimensions > 0) {
+ this.stmtSearchVec = this.db.prepare(`
+ SELECT record_id, distance
+ FROM l1_vec
+ WHERE embedding MATCH ?
+ AND k = ?
+ ORDER BY distance
+ `);
+ }
+
+ // ── L0 schema ──────────────────────────────────
+
+ // L0 metadata table: stores individual messages for vector search
+ this.db.exec(`
+ CREATE TABLE IF NOT EXISTS l0_conversations (
+ record_id TEXT PRIMARY KEY,
+ session_key TEXT NOT NULL,
+ session_id TEXT DEFAULT '',
+ role TEXT NOT NULL DEFAULT '',
+ message_text TEXT NOT NULL,
+ recorded_at TEXT DEFAULT '',
+ timestamp INTEGER DEFAULT 0
+ )
+ `);
+
+ // Migration: add timestamp column if missing (existing DBs pre-v3.x)
+ try {
+ this.db.exec("ALTER TABLE l0_conversations ADD COLUMN timestamp INTEGER DEFAULT 0");
+ this.logger?.info(`${TAG} Migrated l0_conversations: added timestamp column`);
+ } catch {
+ // Column already exists — expected on non-first run
+ }
+
+ // Indexes for L0 queries
+ this.db.exec("CREATE INDEX IF NOT EXISTS idx_l0_session ON l0_conversations(session_key)");
+ this.db.exec("CREATE INDEX IF NOT EXISTS idx_l0_session_id ON l0_conversations(session_id)");
+ this.db.exec("CREATE INDEX IF NOT EXISTS idx_l0_recorded ON l0_conversations(recorded_at)");
+ this.db.exec("CREATE INDEX IF NOT EXISTS idx_l0_timestamp ON l0_conversations(timestamp)");
+
+ // L0 vector virtual table (cosine distance, same dimensions as L1) — deferred when dimensions=0
+ if (this.dimensions > 0) {
+ this.db.exec(`
+ CREATE VIRTUAL TABLE IF NOT EXISTS l0_vec USING vec0(
+ record_id TEXT PRIMARY KEY,
+ embedding float[${this.dimensions}] distance_metric=cosine,
+ recorded_at TEXT DEFAULT ''
+ )
+ `);
+ }
+
+ // L0 prepared statements
+ this.stmtL0UpsertMeta = this.db.prepare(`
+ INSERT INTO l0_conversations (
+ record_id, session_key, session_id, role, message_text, recorded_at, timestamp
+ ) VALUES (?, ?, ?, ?, ?, ?, ?)
+ ON CONFLICT(record_id) DO UPDATE SET
+ message_text=excluded.message_text,
+ recorded_at=excluded.recorded_at,
+ timestamp=excluded.timestamp
+ `);
+
+ if (this.dimensions > 0) {
+ this.stmtL0DeleteVec = this.db.prepare("DELETE FROM l0_vec WHERE record_id = ?");
+ this.stmtL0InsertVec = this.db.prepare("INSERT INTO l0_vec (record_id, embedding, recorded_at) VALUES (?, ?, ?)");
+ }
+ this.stmtL0DeleteMeta = this.db.prepare("DELETE FROM l0_conversations WHERE record_id = ?");
+
+ this.stmtL0GetMeta = this.db.prepare(`
+ SELECT session_key, session_id, role, message_text, recorded_at, timestamp
+ FROM l0_conversations WHERE record_id = ?
+ `);
+
+ if (this.dimensions > 0) {
+ this.stmtL0SearchVec = this.db.prepare(`
+ SELECT record_id, distance
+ FROM l0_vec
+ WHERE embedding MATCH ?
+ AND k = ?
+ ORDER BY distance
+ `);
+ }
+
+ // L0 query statements for L1 runner (newest-first + LIMIT to bound memory)
+ // Sort/filter by recorded_at (write time) instead of timestamp (conversation time)
+ // because L1 cursor uses recorded_at semantics. ISO 8601 string comparison preserves time order.
+ this.stmtL0QueryAll = this.db.prepare(`
+ SELECT record_id, session_key, session_id, role, message_text, recorded_at, timestamp
+ FROM l0_conversations
+ WHERE session_key = ?
+ ORDER BY recorded_at DESC
+ LIMIT ?
+ `);
+
+ this.stmtL0QueryAfter = this.db.prepare(`
+ SELECT record_id, session_key, session_id, role, message_text, recorded_at, timestamp
+ FROM l0_conversations
+ WHERE session_key = ? AND recorded_at > ?
+ ORDER BY recorded_at DESC
+ LIMIT ?
+ `);
+
+ this.stmtL0QueryMigrationCursor = this.db.prepare(`
+ SELECT record_id, session_key, session_id, role, message_text, recorded_at, timestamp
+ FROM l0_conversations
+ WHERE record_id > ?
+ ORDER BY record_id ASC
+ LIMIT ?
+ `);
+
+ // ── FTS5 tables (best-effort — gracefully degrade if fts5 is not compiled in) ──
+ // Schema v2: `content` column stores jieba-segmented text (for indexing),
+ // `content_original` (UNINDEXED) stores the raw text (for display).
+ // If old v1 tables exist (no content_original column), drop + recreate.
+ try {
+ // ── Migrate old FTS5 tables (v1 → v2) ──
+ // v1 tables stored raw text in the `content` column. v2 stores segmented
+ // text in `content` and raw text in `content_original` / `message_text_original`.
+ // FTS5 virtual tables don't support ALTER TABLE ADD COLUMN, so we must
+ // drop and recreate. The data will be repopulated by `rebuildFtsIndex()`.
+ const needsFtsRebuild = this.migrateFtsTablesIfNeeded();
+
+ // L1 FTS5 virtual table (v2 schema)
+ this.db.exec(`
+ CREATE VIRTUAL TABLE IF NOT EXISTS l1_fts USING fts5(
+ content,
+ content_original UNINDEXED,
+ record_id UNINDEXED,
+ type UNINDEXED,
+ priority UNINDEXED,
+ scene_name UNINDEXED,
+ session_key UNINDEXED,
+ session_id UNINDEXED,
+ timestamp_str UNINDEXED,
+ timestamp_start UNINDEXED,
+ timestamp_end UNINDEXED,
+ metadata_json UNINDEXED
+ )
+ `);
+
+ // L0 FTS5 virtual table (v2 schema)
+ this.db.exec(`
+ CREATE VIRTUAL TABLE IF NOT EXISTS l0_fts USING fts5(
+ message_text,
+ message_text_original UNINDEXED,
+ record_id UNINDEXED,
+ session_key UNINDEXED,
+ session_id UNINDEXED,
+ role UNINDEXED,
+ recorded_at UNINDEXED,
+ timestamp UNINDEXED
+ )
+ `);
+
+ // L1 FTS prepared statements
+ this.stmtL1FtsInsert = this.db.prepare(`
+ INSERT INTO l1_fts (content, content_original, record_id, type, priority, scene_name,
+ session_key, session_id, timestamp_str, timestamp_start, timestamp_end, metadata_json)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+ `);
+
+ this.stmtL1FtsDelete = this.db.prepare("DELETE FROM l1_fts WHERE record_id = ?");
+
+ this.stmtL1FtsSearch = this.db.prepare(`
+ SELECT record_id, content_original AS content, type, priority, scene_name,
+ session_key, session_id, timestamp_str, timestamp_start, timestamp_end,
+ metadata_json,
+ bm25(l1_fts) AS rank
+ FROM l1_fts
+ WHERE l1_fts MATCH ?
+ ORDER BY rank ASC
+ LIMIT ?
+ `);
+
+ // L0 FTS prepared statements
+ this.stmtL0FtsInsert = this.db.prepare(`
+ INSERT INTO l0_fts (message_text, message_text_original, record_id, session_key, session_id, role, recorded_at, timestamp)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?)
+ `);
+
+ this.stmtL0FtsDelete = this.db.prepare("DELETE FROM l0_fts WHERE record_id = ?");
+
+ this.stmtL0FtsSearch = this.db.prepare(`
+ SELECT record_id, message_text_original AS message_text, session_key, session_id, role, recorded_at, timestamp,
+ bm25(l0_fts) AS rank
+ FROM l0_fts
+ WHERE l0_fts MATCH ?
+ ORDER BY rank ASC
+ LIMIT ?
+ `);
+
+ this.ftsAvailable = true;
+ this.logger?.debug?.(`${TAG} FTS5 tables initialized (l1_fts, l0_fts) [schema v2 — jieba segmented]`);
+
+ // Rebuild FTS index if migrated from v1 or tables were freshly created
+ if (needsFtsRebuild) {
+ this.rebuildFtsIndex();
+ }
+ } catch (err) {
+ const message = err instanceof Error ? err.message : String(err);
+ this.ftsAvailable = false;
+ this.logger?.warn(
+ `${TAG} FTS5 tables NOT available (fts5 may not be compiled in): ${message}. ` +
+ `FTS-based keyword search will be unavailable; recall will use in-memory scoring if needed.`,
+ );
+ }
+
+ // Save current embedding meta (write after schema is ready)
+ if (providerInfo) {
+ this.writeEmbeddingMeta({
+ provider: providerInfo.provider,
+ model: providerInfo.model,
+ dimensions: this.dimensions,
+ });
+ }
+
+ // Mark vec0 tables as ready only when they were actually created
+ this.vecTablesReady = this.dimensions > 0;
+ // L1 query statements (for l1-reader)
+ const l1QueryCols = `record_id, content, type, priority, scene_name, session_key, session_id,
+ timestamp_str, timestamp_start, timestamp_end,
+ created_time, updated_time, metadata_json`;
+
+ this.stmtQueryBySessionId = this.db.prepare(`
+ SELECT ${l1QueryCols} FROM l1_records
+ WHERE session_id = ?
+ ORDER BY updated_time ASC
+ `);
+
+ this.stmtQueryBySessionIdSince = this.db.prepare(`
+ SELECT ${l1QueryCols} FROM l1_records
+ WHERE session_id = ? AND updated_time > ?
+ ORDER BY updated_time ASC
+ `);
+
+ this.stmtQueryBySessionKey = this.db.prepare(`
+ SELECT ${l1QueryCols} FROM l1_records
+ WHERE session_key = ?
+ ORDER BY updated_time ASC
+ `);
+
+ this.stmtQueryBySessionKeySince = this.db.prepare(`
+ SELECT ${l1QueryCols} FROM l1_records
+ WHERE session_key = ? AND updated_time > ?
+ ORDER BY updated_time ASC
+ `);
+
+ this.stmtQueryAll = this.db.prepare(`
+ SELECT ${l1QueryCols} FROM l1_records
+ ORDER BY updated_time ASC
+ `);
+
+ this.stmtQueryAllSince = this.db.prepare(`
+ SELECT ${l1QueryCols} FROM l1_records
+ WHERE updated_time > ?
+ ORDER BY updated_time ASC
+ `);
+
+ this.stmtL1QueryMigrationCursor = this.db.prepare(`
+ SELECT ${l1QueryCols} FROM l1_records
+ WHERE record_id > ?
+ ORDER BY record_id ASC
+ LIMIT ?
+ `);
+
+ this.logger?.debug?.(`${TAG} Initialized (dimensions=${this.dimensions})`);
+
+ return { needsReindex, reason: reindexReason };
+ }
+
+ // ── Embedding meta helpers ──────────────────────────────
+
+ private readEmbeddingMeta(): EmbeddingMeta | null {
+ try {
+ const row = this.db
+ .prepare("SELECT value FROM embedding_meta WHERE key = ?")
+ .get("embedding_provider_info") as { value: string } | undefined;
+ if (!row) return null;
+ return JSON.parse(row.value) as EmbeddingMeta;
+ } catch {
+ return null;
+ }
+ }
+
+ private writeEmbeddingMeta(meta: EmbeddingMeta): void {
+ this.db.prepare(
+ "INSERT INTO embedding_meta (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value=excluded.value",
+ ).run("embedding_provider_info", JSON.stringify(meta));
+ }
+
+ /** Allowed table names for row counting (whitelist to prevent SQL injection). */
+ private static readonly COUNTABLE_TABLES = new Set(["l1_records", "l0_conversations"]);
+
+ /**
+ * Extra rows to retrieve from vec0 KNN search to compensate for legacy
+ * zero-vector placeholders that may still linger from older data.
+ */
+ private static readonly ZERO_VEC_BUFFER = 10;
+
+ /** Default result limit for FTS5 keyword searches. */
+ private static readonly FTS_DEFAULT_LIMIT = 20;
+
+ private tableRowCount(table: string): number {
+ if (!VectorStore.COUNTABLE_TABLES.has(table)) {
+ this.logger?.warn(`${TAG} tableRowCount: rejected unknown table name "${table}"`);
+ return 0;
+ }
+ try {
+ const row = this.db
+ .prepare(`SELECT COUNT(*) AS cnt FROM ${table}`)
+ .get() as { cnt: number } | undefined;
+ return row?.cnt ?? 0;
+ } catch {
+ return 0;
+ }
+ }
+
+ /**
+ * Detect the embedding dimension of an existing vec0 table by inspecting
+ * the DDL stored in sqlite_master. Returns `null` if the table doesn't
+ * exist or the dimension cannot be determined.
+ *
+ * The vec0 DDL looks like:
+ * CREATE VIRTUAL TABLE l1_vec USING vec0(... embedding float[768] ...)
+ * We parse the number inside `float[N]`.
+ */
+ private getVecTableDimensions(): number | null {
+ try {
+ const row = this.db
+ .prepare("SELECT sql FROM sqlite_master WHERE type='table' AND name=?")
+ .get("l1_vec") as { sql: string } | undefined;
+ if (!row?.sql) return null;
+ const match = row.sql.match(/float\[(\d+)\]/);
+ return match ? Number(match[1]) : null;
+ } catch {
+ return null;
+ }
+ }
+
+ /**
+ * Drop both L1 and L0 vector virtual tables.
+ * Metadata tables (l1_records, l0_conversations) are preserved — only
+ * the vec0 tables need to be rebuilt with the new dimensions.
+ */
+ private dropVectorTables(): void {
+ this.db.exec("DROP TABLE IF EXISTS l1_vec");
+ this.db.exec("DROP TABLE IF EXISTS l0_vec");
+ this.logger?.info(`${TAG} Dropped vector tables (l1_vec, l0_vec)`);
+ }
+
+ /**
+ * Write or update a memory record (metadata + vector).
+ * Uses a manual transaction for atomicity.
+ *
+ * If `embedding` is `undefined` or a zero vector (all elements are 0), only
+ * the metadata row is written — the vec0 table is left untouched. This
+ * allows callers without an EmbeddingService to still persist metadata + FTS
+ * without constructing a throwaway zero-vector, and prevents placeholder
+ * zero vectors (from embedding-service failures) from polluting KNN search
+ * results with null / NaN distances.
+ *
+ * **Fault-tolerant**: catches all errors internally so that a vector store
+ * failure never propagates to the caller / main OpenClaw flow.
+ * Returns `true` on success, `false` on failure (logged as warning).
+ */
+ upsertL1(record: MemoryRecord, embedding: Float32Array | undefined): boolean {
+ if (this.degraded) {
+ this.logger?.warn(`${TAG} [L1-upsert] SKIPPED (degraded mode) id=${record.id}`);
+ return false;
+ }
+ try {
+ const { id: recordId, timestamps } = record;
+ const tsStr = timestamps[0] ?? "";
+ const tsStart =
+ timestamps.length > 0
+ ? timestamps.reduce((a, b) => (a < b ? a : b))
+ : tsStr;
+ const tsEnd =
+ timestamps.length > 0
+ ? timestamps.reduce((a, b) => (a > b ? a : b))
+ : tsStr;
+
+ const skipVec = !embedding || embedding.every(v => v === 0) || !this.vecTablesReady;
+
+ this.logger?.debug?.(
+ `${TAG} [L1-upsert] START id=${recordId}, type=${record.type}, ` +
+ `content="${record.content.slice(0, 60)}..."` +
+ (embedding
+ ? `, embeddingDims=${embedding.length}, ` +
+ `embeddingNorm=${Math.sqrt(Array.from(embedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}` +
+ `${skipVec ? " (ZERO VECTOR or vec tables not ready — vec write will be skipped)" : ""}`
+ : " (no embedding — metadata-only write)"),
+ );
+
+ this.db.exec("BEGIN");
+ try {
+ // Upsert metadata (INSERT OR UPDATE)
+ this.stmtUpsertMeta.run(
+ recordId,
+ record.content,
+ record.type,
+ record.priority,
+ record.scene_name,
+ record.sessionKey,
+ record.sessionId,
+ tsStr,
+ tsStart,
+ tsEnd,
+ record.createdAt,
+ record.updatedAt,
+ JSON.stringify(record.metadata),
+ );
+
+ if (!skipVec) {
+ // vec0 does not support ON CONFLICT → delete then insert
+ this.stmtDeleteVec!.run(recordId);
+ this.stmtInsertVec!.run(recordId, Buffer.from(embedding!.buffer), record.updatedAt);
+ } else {
+ this.logger?.debug?.(
+ `${TAG} [L1-upsert] Skipping vec write (${embedding ? "zero vector" : "no embedding"}) id=${recordId}`,
+ );
+ }
+
+ // Sync FTS5 (delete + re-insert to handle updates)
+ if (this.ftsAvailable) {
+ try {
+ this.stmtL1FtsDelete.run(recordId);
+ this.stmtL1FtsInsert.run(
+ tokenizeForFts(record.content), // content — segmented for indexing
+ record.content, // content_original — raw for display
+ recordId,
+ record.type,
+ record.priority,
+ record.scene_name,
+ record.sessionKey,
+ record.sessionId,
+ tsStr,
+ tsStart,
+ tsEnd,
+ JSON.stringify(record.metadata),
+ );
+ } catch (ftsErr) {
+ // FTS write failure is non-fatal — log and continue
+ this.logger?.warn(
+ `${TAG} [L1-upsert] FTS write failed (non-fatal) id=${recordId}: ${ftsErr instanceof Error ? ftsErr.message : String(ftsErr)}`,
+ );
+ }
+ }
+
+ this.db.exec("COMMIT");
+ } catch (err) {
+ try {
+ this.db.exec("ROLLBACK");
+ } catch { /* ignore rollback errors */ }
+ throw err;
+ }
+ this.logger?.debug?.(`${TAG} [L1-upsert] OK id=${recordId}${skipVec ? " (meta-only)" : ""}`);
+ return true;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} [L1-upsert] FAILED (non-fatal) id=${record.id}: ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return false;
+ }
+ }
+
+ /**
+ * Vector similarity search (cosine distance).
+ * Returns top-k results sorted by similarity (highest first).
+ *
+ * **Fault-tolerant**: returns an empty array on any error (e.g. dimension
+ * mismatch, corrupted DB) so callers can fall back to keyword search.
+ */
+ searchL1Vector(queryEmbedding: Float32Array, topK = 5): VectorSearchResult[] {
+ if (this.degraded || !this.vecTablesReady) {
+ if (this.degraded) this.logger?.warn(`${TAG} [L1-search] SKIPPED (degraded mode)`);
+ return [];
+ }
+ try {
+ // Over-retrieve to compensate for legacy zero-vector placeholders that
+ // may still exist in the vec0 table. New zero vectors are no longer
+ // inserted (upsert() skips vec write for zero vectors since v3.x), but
+ // older data may still contain them — they surface as NULL/NaN distance
+ // in KNN results. A small buffer of 10 is sufficient for remnants.
+ // NOTE: "AND distance IS NOT NULL" is NOT usable because vec0 does not
+ // support that constraint — it causes an empty result set.
+ const ZERO_VEC_BUFFER = 10;
+ const retrieveCount = topK + ZERO_VEC_BUFFER;
+
+ this.logger?.debug?.(
+ `${TAG} [L1-search] START topK=${topK}, retrieveCount=${retrieveCount}, ` +
+ `queryEmbeddingDims=${queryEmbedding.length}, ` +
+ `queryNorm=${Math.sqrt(Array.from(queryEmbedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}`,
+ );
+
+ const rows = this.stmtSearchVec!.all(
+ Buffer.from(queryEmbedding.buffer),
+ retrieveCount,
+ ) as Array<{ record_id: string; distance: number }>;
+
+ this.logger?.debug?.(`${TAG} [L1-search] vec0 returned ${rows.length} candidate(s)`);
+
+ if (rows.length === 0) return [];
+
+ const results: VectorSearchResult[] = [];
+
+ for (const { record_id, distance } of rows) {
+ // sqlite-vec returns null distance for zero vectors (cosine undefined when ‖v‖=0).
+ // Skip these — they are placeholder vectors from embedding-service-unavailable fallback.
+ if (distance == null || Number.isNaN(distance)) {
+ this.logger?.warn(
+ `${TAG} [L1-search] record_id=${record_id} has null/NaN distance (likely zero vector) — skipping`,
+ );
+ continue;
+ }
+
+ const meta = this.stmtGetMeta.get(record_id) as
+ | {
+ content: string;
+ type: string;
+ priority: number;
+ scene_name: string;
+ session_key: string;
+ session_id: string;
+ timestamp_str: string;
+ timestamp_start: string;
+ timestamp_end: string;
+ metadata_json: string;
+ }
+ | undefined;
+
+ if (!meta) {
+ this.logger?.warn(`${TAG} [L1-search] record_id=${record_id} has vector but NO metadata (orphan)`);
+ continue;
+ }
+
+ const score = 1.0 - distance;
+ this.logger?.debug?.(
+ `${TAG} [L1-search] HIT id=${record_id}, distance=${distance.toFixed(4)}, score=${score.toFixed(4)}, ` +
+ `type=${meta.type}, content="${meta.content.slice(0, 60)}..."`,
+ );
+
+ results.push({
+ record_id,
+ content: meta.content,
+ type: meta.type,
+ priority: meta.priority,
+ scene_name: meta.scene_name,
+ score,
+ timestamp_str: meta.timestamp_str,
+ timestamp_start: meta.timestamp_start,
+ timestamp_end: meta.timestamp_end,
+ session_key: meta.session_key,
+ session_id: meta.session_id,
+ metadata_json: meta.metadata_json,
+ });
+ }
+
+ // Trim back to the caller's requested topK (we over-fetched above).
+ const trimmed = results.slice(0, topK);
+ this.logger?.info(
+ `${TAG} [L1-search] DONE returning ${trimmed.length} result(s) (from ${results.length} valid, ${rows.length} raw)`,
+ );
+ return trimmed;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} [L1-search] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return [];
+ }
+ }
+
+ /**
+ * Delete a single record (metadata + vector).
+ *
+ * **Fault-tolerant**: logs a warning on failure, never throws.
+ */
+ deleteL1(recordId: string): boolean {
+ if (this.degraded) return false;
+ try {
+ this.db.exec("BEGIN");
+ try {
+ this.stmtDeleteMeta.run(recordId);
+ if (this.vecTablesReady) this.stmtDeleteVec!.run(recordId);
+ if (this.ftsAvailable) {
+ try { this.stmtL1FtsDelete.run(recordId); } catch { /* non-fatal */ }
+ }
+ this.db.exec("COMMIT");
+ } catch (err) {
+ try {
+ this.db.exec("ROLLBACK");
+ } catch { /* ignore rollback errors */ }
+ throw err;
+ }
+ return true;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} delete failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return false;
+ }
+ }
+
+ /**
+ * Delete multiple records (metadata + vector).
+ *
+ * **Fault-tolerant**: logs a warning on failure, never throws.
+ */
+ deleteL1Batch(recordIds: string[]): boolean {
+ if (this.degraded) return false;
+ if (recordIds.length === 0) return true;
+
+ try {
+ this.db.exec("BEGIN");
+ try {
+ for (const id of recordIds) {
+ this.stmtDeleteMeta.run(id);
+ if (this.vecTablesReady) this.stmtDeleteVec!.run(id);
+ if (this.ftsAvailable) {
+ try { this.stmtL1FtsDelete.run(id); } catch { /* non-fatal */ }
+ }
+ }
+ this.db.exec("COMMIT");
+ } catch (err) {
+ try {
+ this.db.exec("ROLLBACK");
+ } catch { /* ignore rollback errors */ }
+ throw err;
+ }
+ return true;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} deleteBatch failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return false;
+ }
+ }
+
+ /**
+ * Get the total number of L1 records in the store.
+ *
+ * **Fault-tolerant**: returns 0 on failure.
+ * TTL cleanup by updated_time.
+ *
+ * Deletes expired rows from l1_records and matching vectors from l1_vec
+ * in a single transaction to guarantee consistency.
+ */
+ deleteL1Expired(cutoffIso: string): number {
+ if (this.degraded) {
+ this.logger?.warn(`${TAG} [deleteExpired] SKIPPED (degraded mode)`);
+ return 0;
+ }
+ try {
+ const row = this.db.prepare(
+ "SELECT COUNT(*) AS cnt FROM l1_records WHERE updated_time != '' AND updated_time < ?",
+ ).get(cutoffIso) as { cnt: number } | undefined;
+ const expiredCount = row?.cnt ?? 0;
+ if (expiredCount <= 0) return 0;
+
+ this.db.exec("BEGIN");
+ try {
+ if (this.vecTablesReady) {
+ this.db.prepare(
+ "DELETE FROM l1_vec WHERE updated_time != '' AND updated_time < ?",
+ ).run(cutoffIso);
+ }
+ this.db.prepare(
+ "DELETE FROM l1_records WHERE updated_time != '' AND updated_time < ?",
+ ).run(cutoffIso);
+ this.db.exec("COMMIT");
+ return expiredCount;
+ } catch (err) {
+ try {
+ this.db.exec("ROLLBACK");
+ } catch { /* ignore rollback errors */ }
+ throw err;
+ }
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} deleteL1ExpiredByUpdatedTime failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return 0;
+ }
+ }
+
+ /**
+ * Get the total number of records in the store.
+ */
+ countL1(): number {
+ if (this.degraded) return 0;
+ try {
+ const row = this.db
+ .prepare("SELECT COUNT(*) AS cnt FROM l1_records")
+ .get() as { cnt: number };
+ this.logger?.debug?.(`${TAG} [L1-count] total=${row.cnt}`);
+ return row.cnt;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} count failed (non-fatal, returning 0): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return 0;
+ }
+ }
+
+ /**
+ * Query L1 records with optional session and time filters.
+ *
+ * Uses the composite index `idx_l1_session_updated(session_id, updated_time)`
+ * for efficient filtering. All timestamps are compared as UTC ISO 8601 strings.
+ *
+ * **Fault-tolerant**: returns an empty array on any error (degraded mode, DB issues).
+ */
+ queryL1Records(filter?: L1QueryFilter): L1RecordRow[] {
+ if (this.degraded) {
+ this.logger?.warn(`${TAG} [L1-query] SKIPPED (degraded mode)`);
+ return [];
+ }
+ try {
+ const { sessionKey, sessionId, updatedAfter } = filter ?? {};
+
+ let raw: Record[];
+
+ // Priority: sessionId > sessionKey (sessionId is more specific)
+ if (sessionId && updatedAfter) {
+ raw = this.stmtQueryBySessionIdSince.all(sessionId, updatedAfter) as Record[];
+ } else if (sessionId) {
+ raw = this.stmtQueryBySessionId.all(sessionId) as Record[];
+ } else if (sessionKey && updatedAfter) {
+ raw = this.stmtQueryBySessionKeySince.all(sessionKey, updatedAfter) as Record[];
+ } else if (sessionKey) {
+ raw = this.stmtQueryBySessionKey.all(sessionKey) as Record[];
+ } else if (updatedAfter) {
+ raw = this.stmtQueryAllSince.all(updatedAfter) as Record[];
+ } else {
+ raw = this.stmtQueryAll.all() as Record[];
+ }
+
+ // Runtime sanity check: verify first row has expected columns (guards against schema drift)
+ if (raw.length > 0 && !("record_id" in raw[0] && "content" in raw[0])) {
+ this.logger?.warn(
+ `${TAG} [L1-query] Schema mismatch: first row missing expected columns. ` +
+ `Got keys: [${Object.keys(raw[0]).join(", ")}]`,
+ );
+ return [];
+ }
+
+ const rows = raw as unknown as L1RecordRow[];
+
+ this.logger?.info(
+ `${TAG} [L1-query] filter={sessionKey=${sessionKey ?? "(all)"}, sessionId=${sessionId ?? "(all)"}, updatedAfter=${updatedAfter ?? "(none)"}}, ` +
+ `returned ${rows.length} record(s)`,
+ );
+ return rows;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} [L1-query] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`
+ );
+ return [];
+ }
+ }
+
+ // ── L0 operations ──────────────────────────────────
+
+ /**
+ * Write or update an L0 single-message record (metadata + vector).
+ * Uses a manual transaction for atomicity.
+ *
+ * If `embedding` is `undefined` or a zero vector (all elements are 0), only
+ * the metadata row (`l0_conversations`) is written — the vec0 table
+ * (`l0_vec`) is left untouched. This allows callers without an
+ * EmbeddingService to still persist metadata + FTS without constructing a
+ * throwaway zero-vector, and prevents placeholder zero vectors (from
+ * embedding-service failures) from polluting KNN search results.
+ *
+ * **Fault-tolerant**: catches all errors internally, never throws.
+ * Returns `true` on success, `false` on failure (logged as warning).
+ */
+ upsertL0(record: L0Record, embedding: Float32Array | undefined): boolean {
+ if (this.degraded) {
+ this.logger?.warn(`${TAG} [L0-upsert] SKIPPED (degraded mode) id=${record.id}`);
+ return false;
+ }
+ try {
+ const skipVec = !embedding || embedding.every(v => v === 0) || !this.vecTablesReady;
+
+ this.logger?.debug?.(
+ `${TAG} [L0-upsert] START id=${record.id}, session=${record.sessionKey}, role=${record.role}, ` +
+ `text="${record.messageText.slice(0, 60)}..."` +
+ (embedding
+ ? `, embeddingDims=${embedding.length}, ` +
+ `embeddingNorm=${Math.sqrt(Array.from(embedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}` +
+ `${skipVec ? " (ZERO VECTOR or vec tables not ready — vec write will be skipped)" : ""}`
+ : " (no embedding — metadata-only write)"),
+ );
+
+ this.db.exec("BEGIN");
+ try {
+ this.stmtL0UpsertMeta.run(
+ record.id,
+ record.sessionKey,
+ record.sessionId,
+ record.role,
+ record.messageText,
+ record.recordedAt,
+ record.timestamp,
+ );
+
+ if (!skipVec) {
+ // vec0 does not support ON CONFLICT → delete then insert
+ this.stmtL0DeleteVec!.run(record.id);
+ this.stmtL0InsertVec!.run(record.id, Buffer.from(embedding!.buffer), record.recordedAt);
+ } else {
+ this.logger?.debug?.(
+ `${TAG} [L0-upsert] Skipping vec write (${embedding ? "zero vector" : "no embedding"}) id=${record.id}`,
+ );
+ }
+
+ // Sync FTS5 (delete + re-insert to handle updates)
+ if (this.ftsAvailable) {
+ try {
+ this.stmtL0FtsDelete.run(record.id);
+ this.stmtL0FtsInsert.run(
+ tokenizeForFts(record.messageText), // message_text — segmented for indexing
+ record.messageText, // message_text_original — raw for display
+ record.id,
+ record.sessionKey,
+ record.sessionId,
+ record.role,
+ record.recordedAt,
+ record.timestamp,
+ );
+ } catch (ftsErr) {
+ // FTS write failure is non-fatal — log and continue
+ this.logger?.warn(
+ `${TAG} [L0-upsert] FTS write failed (non-fatal) id=${record.id}: ${ftsErr instanceof Error ? ftsErr.message : String(ftsErr)}`,
+ );
+ }
+ }
+
+ this.db.exec("COMMIT");
+ } catch (err) {
+ try {
+ this.db.exec("ROLLBACK");
+ } catch { /* ignore rollback errors */ }
+ throw err;
+ }
+ this.logger?.debug?.(`${TAG} [L0-upsert] OK id=${record.id}${skipVec ? " (meta-only)" : ""}`);
+ return true;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} [L0-upsert] FAILED (non-fatal) id=${record.id}: ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return false;
+ }
+ }
+
+ /**
+ * Update ONLY the vector embedding for an existing L0 record.
+ * The metadata row must already exist in l0_conversations (written by upsertL0).
+ *
+ * This is used by the background embedding task in auto-capture:
+ * 1. upsertL0() writes metadata + FTS synchronously (no embedding)
+ * 2. Background task calls embedBatch() then updateL0Embedding() for each record
+ *
+ * **Fault-tolerant**: catches all errors internally, never throws.
+ * Returns `true` on success, `false` on failure.
+ */
+ updateL0Embedding(recordId: string, embedding: Float32Array): boolean {
+ if (this.degraded || !this.vecTablesReady) {
+ return false;
+ }
+ if (!embedding || embedding.every(v => v === 0)) {
+ this.logger?.debug?.(`${TAG} [L0-update-embedding] Skipping zero vector for ${recordId}`);
+ return false;
+ }
+ try {
+ // Look up recorded_at from metadata for the vec0 row
+ const meta = this.stmtL0GetMeta.get(recordId) as { recorded_at: string } | undefined;
+ if (!meta) {
+ this.logger?.warn(`${TAG} [L0-update-embedding] No metadata found for ${recordId}, skipping`);
+ return false;
+ }
+
+ this.db.exec("BEGIN");
+ try {
+ this.stmtL0DeleteVec!.run(recordId);
+ this.stmtL0InsertVec!.run(recordId, Buffer.from(embedding.buffer), meta.recorded_at);
+ this.db.exec("COMMIT");
+ } catch (err) {
+ try { this.db.exec("ROLLBACK"); } catch { /* ignore */ }
+ throw err;
+ }
+ return true;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} [L0-update-embedding] FAILED (non-fatal) id=${recordId}: ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return false;
+ }
+ }
+
+ /**
+ * Vector similarity search on L0 individual messages (cosine distance).
+ * Returns top-k results sorted by similarity (highest first).
+ *
+ * **Fault-tolerant**: returns an empty array on any error.
+ */
+ searchL0Vector(queryEmbedding: Float32Array, topK = 5): L0VectorSearchResult[] {
+ if (this.degraded || !this.vecTablesReady) {
+ if (this.degraded) this.logger?.warn(`${TAG} [L0-search] SKIPPED (degraded mode)`);
+ return [];
+ }
+ try {
+ // Over-retrieve to compensate for legacy zero-vector placeholders that
+ // may still exist in the vec0 table. New zero vectors are no longer
+ // inserted (upsertL0() skips vec write for zero vectors since v3.x), but
+ // older data may still contain them — they surface as NULL/NaN distance
+ // in KNN results.
+ // NOTE: "AND distance IS NOT NULL" is NOT usable because vec0 does not
+ // support that constraint — it causes an empty result set.
+ const retrieveCount = topK + VectorStore.ZERO_VEC_BUFFER;
+
+ this.logger?.debug?.(
+ `${TAG} [L0-search] START topK=${topK}, retrieveCount=${retrieveCount}, ` +
+ `queryEmbeddingDims=${queryEmbedding.length}, ` +
+ `queryNorm=${Math.sqrt(Array.from(queryEmbedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}`,
+ );
+
+ const rows = this.stmtL0SearchVec!.all(
+ Buffer.from(queryEmbedding.buffer),
+ retrieveCount,
+ ) as Array<{ record_id: string; distance: number }>;
+
+ this.logger?.debug?.(`${TAG} [L0-search] vec0 returned ${rows.length} candidate(s)`);
+
+ if (rows.length === 0) return [];
+
+ const results: L0VectorSearchResult[] = [];
+
+ for (const { record_id, distance } of rows) {
+ // sqlite-vec returns null distance for zero vectors (cosine undefined when ‖v‖=0).
+ // Skip these — they are placeholder vectors from embedding-service-unavailable fallback.
+ if (distance == null || Number.isNaN(distance)) {
+ this.logger?.warn(
+ `${TAG} [L0-search] record_id=${record_id} has null/NaN distance (likely zero vector) — skipping`,
+ );
+ continue;
+ }
+
+ const meta = this.stmtL0GetMeta.get(record_id) as
+ | {
+ session_key: string;
+ session_id: string;
+ role: string;
+ message_text: string;
+ recorded_at: string;
+ timestamp: number;
+ }
+ | undefined;
+
+ if (!meta) {
+ this.logger?.warn(`${TAG} [L0-search] record_id=${record_id} has vector but NO metadata (orphan)`);
+ continue;
+ }
+
+ const score = 1.0 - distance;
+ this.logger?.debug?.(
+ `${TAG} [L0-search] HIT id=${record_id}, distance=${distance.toFixed(4)}, score=${score.toFixed(4)}, ` +
+ `role=${meta.role}, session=${meta.session_key}, text="${meta.message_text.slice(0, 60)}..."`,
+ );
+
+ results.push({
+ record_id,
+ session_key: meta.session_key,
+ session_id: meta.session_id,
+ role: meta.role,
+ message_text: meta.message_text,
+ score,
+ recorded_at: meta.recorded_at,
+ timestamp: meta.timestamp ?? 0,
+ });
+ }
+
+ // Trim back to the caller's requested topK (we over-fetched above).
+ const trimmed = results.slice(0, topK);
+ this.logger?.info(
+ `${TAG} [L0-search] DONE returning ${trimmed.length} result(s) (from ${results.length} valid, ${rows.length} raw)`,
+ );
+ return trimmed;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} [L0-search] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return [];
+ }
+ }
+
+ /**
+ * Delete a single L0 record (metadata + vector).
+ *
+ * **Fault-tolerant**: logs a warning on failure, never throws.
+ */
+ deleteL0(recordId: string): boolean {
+ if (this.degraded) return false;
+ try {
+ this.db.exec("BEGIN");
+ try {
+ this.stmtL0DeleteMeta.run(recordId);
+ if (this.vecTablesReady) this.stmtL0DeleteVec!.run(recordId);
+ if (this.ftsAvailable) {
+ try { this.stmtL0FtsDelete.run(recordId); } catch { /* non-fatal */ }
+ }
+ this.db.exec("COMMIT");
+ } catch (err) {
+ try {
+ this.db.exec("ROLLBACK");
+ } catch { /* ignore rollback errors */ }
+ throw err;
+ }
+ return true;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} deleteL0 failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return false;
+ }
+ }
+
+ /**
+ * TTL cleanup by recorded_at (ISO string) for L0 records.
+ *
+ * Deletes expired rows from l0_conversations and matching vectors from l0_vec
+ * in a single transaction to guarantee consistency.
+ */
+ deleteL0Expired(cutoffIso: string): number {
+ if (this.degraded) {
+ this.logger?.warn(`${TAG} [deleteExpiredL0] SKIPPED (degraded mode)`);
+ return 0;
+ }
+
+ try {
+ const row = this.db.prepare(
+ "SELECT COUNT(*) AS cnt FROM l0_conversations WHERE recorded_at != '' AND recorded_at < ?",
+ ).get(cutoffIso) as { cnt: number } | undefined;
+ const expiredCount = row?.cnt ?? 0;
+ if (expiredCount <= 0) return 0;
+
+ this.db.exec("BEGIN");
+ try {
+ if (this.vecTablesReady) {
+ this.db.prepare(
+ "DELETE FROM l0_vec WHERE recorded_at != '' AND recorded_at < ?",
+ ).run(cutoffIso);
+ }
+ this.db.prepare(
+ "DELETE FROM l0_conversations WHERE recorded_at != '' AND recorded_at < ?",
+ ).run(cutoffIso);
+ this.db.exec("COMMIT");
+ return expiredCount;
+ } catch (err) {
+ try {
+ this.db.exec("ROLLBACK");
+ } catch { /* ignore rollback errors */ }
+ throw err;
+ }
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} deleteL0ExpiredByRecordedAt failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return 0;
+ }
+ }
+
+ /**
+ * Get the total number of L0 message records in the store.
+ *
+ * **Fault-tolerant**: returns 0 on failure.
+ */
+ countL0(): number {
+ if (this.degraded) return 0;
+ try {
+ const row = this.db
+ .prepare("SELECT COUNT(*) AS cnt FROM l0_conversations")
+ .get() as { cnt: number };
+ this.logger?.debug?.(`${TAG} [L0-count] total=${row.cnt}`);
+ return row.cnt;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} countL0 failed (non-fatal, returning 0): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return 0;
+ }
+ }
+
+ // ── Re-index operations ──────────────────────────────────
+
+ /**
+ * Get all L1 record texts for re-embedding.
+ * Returns record_id → content pairs.
+ */
+ getAllL1Texts(): Array<{ record_id: string; content: string; updated_time: string }> {
+ if (this.degraded) return [];
+ try {
+ return this.db
+ .prepare("SELECT record_id, content, updated_time FROM l1_records")
+ .all() as Array<{ record_id: string; content: string; updated_time: string }>;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} getAllL1Texts failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return [];
+ }
+ }
+
+ /**
+ * Get all L0 message texts for re-embedding.
+ * Returns record_id → message_text/recorded_at tuples.
+ */
+ getAllL0Texts(): Array<{ record_id: string; message_text: string; recorded_at: string }> {
+ if (this.degraded) return [];
+ try {
+ return this.db
+ .prepare("SELECT record_id, message_text, recorded_at FROM l0_conversations")
+ .all() as Array<{ record_id: string; message_text: string; recorded_at: string }>;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} getAllL0Texts failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return [];
+ }
+ }
+
+ /**
+ * Re-embed all existing L1 and L0 texts with a new embedding function.
+ *
+ * This is called after `init()` returns `needsReindex: true` — the vector
+ * tables have already been dropped and re-created with the correct dimensions.
+ * This method reads every text from the metadata tables and writes fresh
+ * embeddings into the new vector tables.
+ *
+ * @param embedFn A function that converts text → Float32Array embedding.
+ * @param onProgress Optional callback for progress reporting.
+ */
+ async reindexAll(
+ embedFn: (text: string) => Promise,
+ onProgress?: (done: number, total: number, layer: "L1" | "L0") => void,
+ ): Promise<{ l1Count: number; l0Count: number }> {
+ if (this.degraded || !this.vecTablesReady) {
+ if (this.degraded) this.logger?.warn(`${TAG} reindexAll skipped: VectorStore is in degraded mode`);
+ return { l1Count: 0, l0Count: 0 };
+ }
+
+ try {
+ // ── Re-embed L1 ──
+ const l1Rows = this.getAllL1Texts();
+ let l1Done = 0;
+ for (const { record_id, content, updated_time } of l1Rows) {
+ try {
+ const embedding = await embedFn(content);
+ // Wrap delete+insert in a transaction to prevent orphan vectors
+ this.db.exec("BEGIN");
+ try {
+ this.stmtDeleteVec!.run(record_id);
+ this.stmtInsertVec!.run(record_id, Buffer.from(embedding.buffer), updated_time);
+ this.db.exec("COMMIT");
+ } catch (txErr) {
+ try { this.db.exec("ROLLBACK"); } catch { /* ignore */ }
+ throw txErr;
+ }
+ } catch (err) {
+ this.logger?.warn?.(
+ `${TAG} reindex L1 skip ${record_id}: ${err instanceof Error ? err.message : String(err)}`,
+ );
+ }
+ l1Done++;
+ onProgress?.(l1Done, l1Rows.length, "L1");
+ }
+
+ // ── Re-embed L0 ──
+ const l0Rows = this.getAllL0Texts();
+ let l0Done = 0;
+ for (const { record_id, message_text, recorded_at } of l0Rows) {
+ try {
+ const embedding = await embedFn(message_text);
+ // Wrap delete+insert in a transaction to prevent orphan vectors
+ this.db.exec("BEGIN");
+ try {
+ this.stmtL0DeleteVec!.run(record_id);
+ this.stmtL0InsertVec!.run(record_id, Buffer.from(embedding.buffer), recorded_at);
+ this.db.exec("COMMIT");
+ } catch (txErr) {
+ try { this.db.exec("ROLLBACK"); } catch { /* ignore */ }
+ throw txErr;
+ }
+ } catch (err) {
+ this.logger?.warn?.(
+ `${TAG} reindex L0 skip ${record_id}: ${err instanceof Error ? err.message : String(err)}`,
+ );
+ }
+ l0Done++;
+ onProgress?.(l0Done, l0Rows.length, "L0");
+ }
+
+ this.logger?.info(
+ `${TAG} Reindex complete: L1=${l1Done}/${l1Rows.length}, L0=${l0Done}/${l0Rows.length}`,
+ );
+
+ return { l1Count: l1Done, l0Count: l0Done };
+ } catch (err) {
+ this.logger?.error(
+ `${TAG} reindexAll failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return { l1Count: 0, l0Count: 0 };
+ }
+ }
+
+ // ── L0 query operations (for L1 runner) ──────────────────────────────────
+
+ /**
+ * Query L0 messages for a given session key, optionally filtered by recorded_at cursor.
+ * Returns messages ordered by recorded_at ASC (chronological write order).
+ *
+ * Used by L1 runner to read L0 data from DB instead of JSONL files.
+ */
+ queryL0ForL1(
+ sessionKey: string,
+ afterRecordedAtMs?: number,
+ limit = 50,
+ ): Array<{
+ record_id: string;
+ session_key: string;
+ session_id: string;
+ role: string;
+ message_text: string;
+ recorded_at: string;
+ timestamp: number;
+ }> {
+ if (this.degraded) {
+ this.logger?.warn(`${TAG} [L0-query] SKIPPED (degraded mode)`);
+ return [];
+ }
+ try {
+ // Query newest-first (DESC) with LIMIT, then reverse to chronological order
+ let rows: Array>;
+ if (afterRecordedAtMs && afterRecordedAtMs > 0) {
+ // Convert epoch ms to ISO string for recorded_at comparison
+ const afterRecordedAtIso = new Date(afterRecordedAtMs).toISOString();
+ rows = this.stmtL0QueryAfter.all(sessionKey, afterRecordedAtIso, limit) as Array>;
+ } else {
+ rows = this.stmtL0QueryAll.all(sessionKey, limit) as Array>;
+ }
+
+ this.logger?.info(
+ `${TAG} [L0-query] session=${sessionKey}, afterRecordedAtMs=${afterRecordedAtMs ?? "(all)"}, ` +
+ `limit=${limit}, returned ${rows.length} row(s)`,
+ );
+
+ // Reverse: SQL returns newest-first (DESC), callers expect chronological order
+ return rows.map((r) => ({
+ record_id: r.record_id as string,
+ session_key: r.session_key as string,
+ session_id: (r.session_id as string) || "",
+ role: r.role as string,
+ message_text: r.message_text as string,
+ recorded_at: (r.recorded_at as string) || "",
+ timestamp: (r.timestamp as number) || 0,
+ })).reverse();
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} [L0-query] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return [];
+ }
+ }
+
+ /**
+ * Query L0 messages for a given session key, grouped by session_id.
+ * Each group's messages are in chronological order (recorded_at ASC).
+ * Groups are sorted by earliest message timestamp.
+ *
+ * Used by L1 runner to replace readConversationMessagesGroupedBySessionId().
+ */
+ queryL0GroupedBySessionId(
+ sessionKey: string,
+ afterRecordedAtMs?: number,
+ limit = 50,
+ ): Array<{ sessionId: string; messages: Array<{ id: string; role: string; content: string; timestamp: number; recordedAtMs: number }> }> {
+ if (this.degraded) {
+ this.logger?.warn(`${TAG} [L0-query-grouped] SKIPPED (degraded mode)`);
+ return [];
+ }
+ try {
+ const rows = this.queryL0ForL1(sessionKey, afterRecordedAtMs, limit);
+
+ // Group by session_id
+ const groupMap = new Map>();
+ for (const row of rows) {
+ const sid = row.session_id || "";
+ let group = groupMap.get(sid);
+ if (!group) {
+ group = [];
+ groupMap.set(sid, group);
+ }
+ group.push({
+ id: row.record_id,
+ role: row.role,
+ content: row.message_text,
+ timestamp: row.timestamp,
+ recordedAtMs: row.recorded_at ? Date.parse(row.recorded_at) || 0 : 0,
+ });
+ }
+
+ // Convert to array, sorted by earliest message timestamp
+ const groups: Array<{ sessionId: string; messages: Array<{ id: string; role: string; content: string; timestamp: number; recordedAtMs: number }> }> = [];
+ for (const [sessionId, messages] of groupMap) {
+ if (messages.length > 0) {
+ groups.push({ sessionId, messages });
+ }
+ }
+ groups.sort((a, b) => a.messages[0].timestamp - b.messages[0].timestamp);
+
+ this.logger?.info(
+ `${TAG} [L0-query-grouped] session=${sessionKey}, afterRecordedAtMs=${afterRecordedAtMs ?? "(all)"}, ` +
+ `${rows.length} messages across ${groups.length} group(s)`,
+ );
+
+ return groups;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} [L0-query-grouped] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return [];
+ }
+ }
+
+ // ── Cursor-based pagination for migration ──────────────────
+
+ /**
+ * Read a page of L1 records using primary key cursor.
+ * Returns rows with `record_id > afterId`, ordered by PK, limited to `pageSize`.
+ * Pass `""` as `afterId` for the first page.
+ */
+ queryL1RecordsCursor(afterId: string, pageSize: number): L1RecordRow[] {
+ if (this.degraded) return [];
+ try {
+ return this.stmtL1QueryMigrationCursor.all(afterId, pageSize) as unknown as L1RecordRow[];
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} [L1-query-cursor] FAILED (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return [];
+ }
+ }
+
+ /**
+ * Read a page of L0 records using primary key cursor.
+ * Returns rows with `record_id > afterId`, ordered by PK, limited to `pageSize`.
+ * Pass `""` as `afterId` for the first page.
+ */
+ queryL0RecordsCursor(afterId: string, pageSize: number): L0RecordRow[] {
+ if (this.degraded) return [];
+ try {
+ return this.stmtL0QueryMigrationCursor.all(afterId, pageSize) as unknown as L0RecordRow[];
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} [L0-query-cursor] FAILED (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return [];
+ }
+ }
+
+ // ── FTS5 search operations ──────────────────────────────────
+
+ /**
+ * Whether FTS5 full-text search is available.
+ * When `false`, callers should skip keyword-based recall entirely.
+ */
+ isFtsAvailable(): boolean {
+ return this.ftsAvailable;
+ }
+
+ /**
+ * FTS5 keyword search on L1 records.
+ * Returns top-`limit` results sorted by BM25 relevance (highest first).
+ *
+ * @param ftsQuery A pre-built FTS5 MATCH expression (from `buildFtsQuery()`).
+ * @param limit Maximum number of results to return.
+ *
+ * **Fault-tolerant**: returns an empty array on any error.
+ */
+ searchL1Fts(ftsQuery: string, limit = 20): FtsSearchResult[] {
+ if (this.degraded || !this.ftsAvailable) return [];
+ try {
+ const rows = this.stmtL1FtsSearch.all(ftsQuery, limit) as Array<{
+ record_id: string;
+ content: string;
+ type: string;
+ priority: number;
+ scene_name: string;
+ session_key: string;
+ session_id: string;
+ timestamp_str: string;
+ timestamp_start: string;
+ timestamp_end: string;
+ metadata_json: string;
+ rank: number;
+ }>;
+
+ return rows.map((r) => ({
+ record_id: r.record_id,
+ content: r.content,
+ type: r.type,
+ priority: r.priority,
+ scene_name: r.scene_name,
+ score: bm25RankToScore(r.rank),
+ timestamp_str: r.timestamp_str,
+ timestamp_start: r.timestamp_start,
+ timestamp_end: r.timestamp_end,
+ session_key: r.session_key,
+ session_id: r.session_id,
+ metadata_json: r.metadata_json,
+ }));
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} [L1-fts-search] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return [];
+ }
+ }
+
+ /**
+ * FTS5 keyword search on L0 conversation messages.
+ * Returns top-`limit` results sorted by BM25 relevance (highest first).
+ *
+ * @param ftsQuery A pre-built FTS5 MATCH expression (from `buildFtsQuery()`).
+ * @param limit Maximum number of results to return.
+ *
+ * **Fault-tolerant**: returns an empty array on any error.
+ */
+ searchL0Fts(ftsQuery: string, limit = VectorStore.FTS_DEFAULT_LIMIT): L0FtsSearchResult[] {
+ if (this.degraded || !this.ftsAvailable) return [];
+ try {
+ const rows = this.stmtL0FtsSearch.all(ftsQuery, limit) as Array<{
+ record_id: string;
+ message_text: string;
+ session_key: string;
+ session_id: string;
+ role: string;
+ recorded_at: string;
+ timestamp: number;
+ rank: number;
+ }>;
+
+ return rows.map((r) => ({
+ record_id: r.record_id,
+ session_key: r.session_key,
+ session_id: r.session_id,
+ role: r.role,
+ message_text: r.message_text,
+ score: bm25RankToScore(r.rank),
+ recorded_at: r.recorded_at,
+ timestamp: r.timestamp ?? 0,
+ }));
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} [L0-fts-search] FAILED (non-fatal, returning empty): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return [];
+ }
+ }
+
+ // ── FTS5 migration & rebuild ──────────────────────────────────────────────
+
+ /**
+ * Detect old FTS5 v1 schema (no `content_original` column) and drop the
+ * tables so they can be recreated with the v2 schema.
+ *
+ * FTS5 virtual tables do NOT support `ALTER TABLE ADD COLUMN`, so the only
+ * migration path is DROP + recreate + repopulate.
+ *
+ * @returns `true` if migration was performed (= FTS index needs rebuilding).
+ * @internal
+ */
+ private migrateFtsTablesIfNeeded(): boolean {
+ try {
+ // Check if l1_fts exists at all
+ const l1Exists = this.db
+ .prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name='l1_fts'")
+ .get();
+ if (!l1Exists) {
+ // Fresh install — tables will be created with v2 schema.
+ // Still need rebuild if there's existing data in l1_records.
+ const hasData = this.db.prepare("SELECT 1 FROM l1_records LIMIT 1").get();
+ return !!hasData;
+ }
+
+ // Check if the v2 column `content_original` exists.
+ // FTS5 tables appear in pragma_table_info with their column names.
+ const cols = this.db
+ .prepare("SELECT name FROM pragma_table_info('l1_fts')")
+ .all() as Array<{ name: string }>;
+ const hasV2Col = cols.some((c) => c.name === "content_original");
+
+ if (hasV2Col) {
+ return false; // Already v2 — no migration needed
+ }
+
+ // v1 → v2: drop both FTS tables (data will be repopulated by rebuildFtsIndex)
+ this.logger?.info(`${TAG} Migrating FTS5 tables from v1 to v2 (jieba segmented)`);
+ this.db.exec("DROP TABLE IF EXISTS l1_fts");
+ this.db.exec("DROP TABLE IF EXISTS l0_fts");
+ return true;
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} FTS migration check failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ return false;
+ }
+ }
+
+ /**
+ * Rebuild the FTS5 index from scratch by reading all records from the
+ * metadata tables and re-inserting them with jieba-segmented text.
+ *
+ * Called automatically after:
+ * - Schema migration from v1 to v2
+ * - Fresh table creation when existing data exists
+ *
+ * Safe to call multiple times (idempotent — clears FTS tables first).
+ */
+ rebuildFtsIndex(): void {
+ if (!this.ftsAvailable) return;
+
+ try {
+ this.logger?.info(`${TAG} Rebuilding FTS5 index with jieba segmentation…`);
+
+ // ── Rebuild L1 FTS ──
+ // Clear existing FTS data
+ this.db.exec("DELETE FROM l1_fts");
+
+ // Read all L1 records from metadata table
+ const l1Rows = this.db
+ .prepare(`
+ SELECT record_id, content, type, priority, scene_name,
+ session_key, session_id, timestamp_str, timestamp_start, timestamp_end, metadata_json
+ FROM l1_records
+ `)
+ .all() as Array<{
+ record_id: string;
+ content: string;
+ type: string;
+ priority: number;
+ scene_name: string;
+ session_key: string;
+ session_id: string;
+ timestamp_str: string;
+ timestamp_start: string;
+ timestamp_end: string;
+ metadata_json: string;
+ }>;
+
+ let l1Count = 0;
+ for (const r of l1Rows) {
+ try {
+ this.stmtL1FtsInsert.run(
+ tokenizeForFts(r.content), // content — segmented
+ r.content, // content_original — raw
+ r.record_id,
+ r.type,
+ r.priority,
+ r.scene_name,
+ r.session_key,
+ r.session_id,
+ r.timestamp_str,
+ r.timestamp_start,
+ r.timestamp_end,
+ r.metadata_json,
+ );
+ l1Count++;
+ } catch (err) {
+ this.logger?.warn?.(
+ `${TAG} FTS rebuild skip L1 ${r.record_id}: ${err instanceof Error ? err.message : String(err)}`,
+ );
+ }
+ }
+
+ // ── Rebuild L0 FTS ──
+ this.db.exec("DELETE FROM l0_fts");
+
+ const l0Rows = this.db
+ .prepare(`
+ SELECT record_id, message_text, session_key, session_id, role, recorded_at, timestamp
+ FROM l0_conversations
+ `)
+ .all() as Array<{
+ record_id: string;
+ message_text: string;
+ session_key: string;
+ session_id: string;
+ role: string;
+ recorded_at: string;
+ timestamp: number;
+ }>;
+
+ let l0Count = 0;
+ for (const r of l0Rows) {
+ try {
+ this.stmtL0FtsInsert.run(
+ tokenizeForFts(r.message_text), // message_text — segmented
+ r.message_text, // message_text_original — raw
+ r.record_id,
+ r.session_key,
+ r.session_id,
+ r.role,
+ r.recorded_at,
+ r.timestamp,
+ );
+ l0Count++;
+ } catch (err) {
+ this.logger?.warn?.(
+ `${TAG} FTS rebuild skip L0 ${r.record_id}: ${err instanceof Error ? err.message : String(err)}`,
+ );
+ }
+ }
+
+ this.logger?.info(
+ `${TAG} FTS5 rebuild complete: L1=${l1Count}/${l1Rows.length}, L0=${l0Count}/${l0Rows.length}`,
+ );
+ } catch (err) {
+ this.logger?.warn(
+ `${TAG} FTS5 rebuild failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
+ );
+ }
+ }
+
+ // ============================
+ // IMemoryStore interface implementation
+ // ============================
+
+ /** Query the store's search capabilities. */
+ getCapabilities(): StoreCapabilities {
+ return {
+ vectorSearch: this.vecTablesReady,
+ ftsSearch: this.ftsAvailable,
+ nativeHybridSearch: false,
+ sparseVectors: false,
+ };
+ }
+
+ /**
+ * Close the database connection.
+ * Should be called on shutdown. Idempotent — safe to call multiple times.
+ */
+ close(): void {
+ if (this.closed) return;
+ this.closed = true;
+ try {
+ this.db.close();
+ } catch (err) {
+ this.logger?.warn?.(
+ `${TAG} Error closing database: ${err instanceof Error ? err.message : String(err)}`,
+ );
+ }
+ }
+}
diff --git a/src/store/tcvdb-client.ts b/src/store/tcvdb-client.ts
new file mode 100644
index 0000000..3594a59
--- /dev/null
+++ b/src/store/tcvdb-client.ts
@@ -0,0 +1,287 @@
+/**
+ * Tencent Cloud VectorDB HTTP Client.
+ *
+ * Thin wrapper around the VectorDB HTTP API. Handles authentication, timeouts,
+ * retries (5xx / timeout), and error normalization.
+ *
+ * API docs: https://cloud.tencent.com/document/product/1709
+ */
+
+import fs from "node:fs";
+import { request as undiciRequest, Agent as UndiciAgent } from "undici";
+import type { Dispatcher } from "undici";
+import type { StoreLogger } from "./types.js";
+
+// ============================
+// Types
+// ============================
+
+export interface TcvdbClientConfig {
+ /** Instance URL (e.g. "http://10.0.1.1:80") */
+ url: string;
+ /** Account name (default: "root") */
+ username: string;
+ /** API Key */
+ apiKey: string;
+ /** Database name */
+ database: string;
+ /** Request timeout in ms (default: 10000) */
+ timeout: number;
+ /** Path to CA certificate PEM file (for HTTPS connections) */
+ caPemPath?: string;
+}
+
+/** Standard VectorDB API response envelope. */
+interface ApiResponse {
+ code: number;
+ msg: string;
+ [key: string]: unknown;
+}
+
+/** Search/hybridSearch response shape. */
+export interface SearchResponse {
+ documents: Array>>;
+}
+
+/** Query response shape. */
+export interface QueryResponse {
+ documents: Array>;
+ count?: number;
+}
+
+/** Collection info from describeCollection. */
+export interface CollectionInfo {
+ collection: string;
+ database: string;
+ documentCount?: number;
+ embedding?: {
+ field: string;
+ vectorField: string;
+ model: string;
+ };
+ indexes?: Array>;
+ [key: string]: unknown;
+}
+
+export class TcvdbApiError extends Error {
+ readonly apiCode: number;
+ constructor(path: string, code: number, msg: string) {
+ super(`VectorDB ${path}: code=${code}, msg=${msg}`);
+ this.name = "TcvdbApiError";
+ this.apiCode = code;
+ }
+}
+
+// ============================
+// Client
+// ============================
+
+const TAG = "[memory-tdai][tcvdb-client]";
+const MAX_RETRIES = 2;
+
+export class TcvdbClient {
+ private readonly baseUrl: string;
+ private readonly authHeader: string;
+ private readonly database: string;
+ private readonly timeout: number;
+ private readonly logger?: StoreLogger;
+ /** undici dispatcher for HTTPS + custom CA. */
+ private readonly dispatcher?: Dispatcher;
+
+ constructor(config: TcvdbClientConfig, logger?: StoreLogger) {
+ this.baseUrl = config.url.replace(/\/+$/, "");
+ this.authHeader = `Bearer account=${config.username}&api_key=${config.apiKey}`;
+ this.database = config.database;
+ this.timeout = config.timeout;
+ this.logger = logger;
+
+ // Log connection info at construction time.
+ this.logger?.debug?.(`${TAG} url=${this.baseUrl} db=${this.database} timeout=${this.timeout}${this.baseUrl.startsWith("https://") ? ` https=true caPemPath=${config.caPemPath ?? "(none)"}` : ""}`);
+
+ // For HTTPS with a custom CA certificate, create a dedicated undici Agent.
+ // We use undici.request() instead of global fetch because fetch's
+ // `dispatcher` option is unreliable across Node versions.
+ if (this.baseUrl.startsWith("https://") && config.caPemPath) {
+ try {
+ const ca = fs.readFileSync(config.caPemPath, "utf-8");
+ this.dispatcher = new UndiciAgent({ connect: { ca } });
+ this.logger?.debug?.(`${TAG} HTTPS enabled with CA from ${config.caPemPath}`);
+ } catch (err) {
+ this.logger?.error(`${TAG} Failed to load CA PEM from ${config.caPemPath}: ${err instanceof Error ? err.message : String(err)}`);
+ }
+ }
+ }
+
+ // ── Generic request ─────────────────────────────────────
+
+ /**
+ * Send a POST request to VectorDB API.
+ * Handles auth, timeout, retries (5xx/timeout), and error unwrapping.
+ */
+ async request(path: string, body: Record): Promise {
+ let lastError: Error | undefined;
+
+ for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
+ try {
+ this.logger?.debug?.(`${TAG} → ${path} body=${JSON.stringify(body).slice(0, 500)}`);
+ const { statusCode, body: respBody } = await undiciRequest(`${this.baseUrl}${path}`, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ "Authorization": this.authHeader,
+ },
+ body: JSON.stringify(body),
+ signal: AbortSignal.timeout(this.timeout),
+ ...(this.dispatcher ? { dispatcher: this.dispatcher } : {}),
+ });
+
+ const text = await respBody.text();
+ const json = JSON.parse(text) as ApiResponse;
+ this.logger?.debug?.(`${TAG} ← ${path} status=${statusCode} code=${json.code} msg=${json.msg} keys=[${Object.keys(json).join(",")}]`);
+
+ if (json.code !== 0) {
+ const err = new TcvdbApiError(path, json.code, json.msg);
+ if (statusCode !== undefined && statusCode >= 400 && statusCode < 500) throw err;
+ lastError = err;
+ continue;
+ }
+
+ return json as unknown as T;
+ } catch (err) {
+ if (err instanceof TcvdbApiError && err.apiCode !== 0) throw err;
+ lastError = err instanceof Error ? err : new Error(String(err));
+ if (attempt < MAX_RETRIES) {
+ const delay = 500 * (attempt + 1);
+ this.logger?.debug?.(`${TAG} ${path} retry ${attempt + 1}/${MAX_RETRIES} in ${delay}ms`);
+ await new Promise((r) => setTimeout(r, delay));
+ }
+ }
+ }
+
+ throw lastError ?? new Error(`${TAG} ${path} failed after retries`);
+ }
+
+ // ── Database operations ─────────────────────────────────
+
+ async createDatabase(dbName?: string): Promise {
+ const name = dbName ?? this.database;
+ // SDK pattern: list first, create only if not found
+ const listResp = await this.request<{ databases: string[] }>("/database/list", {});
+ const exists = (listResp.databases ?? []).includes(name);
+ if (exists) {
+ this.logger?.debug?.(`${TAG} Database already exists: ${name}`);
+ return false;
+ }
+ await this.request("/database/create", { database: name });
+ this.logger?.info(`${TAG} Database created: ${name}`);
+ return true;
+ }
+
+ // ── Collection operations ───────────────────────────────
+
+ async createCollection(params: Record): Promise {
+ const name = String(params.collection ?? "");
+ // SDK pattern: try describe first, create only if not found (code 15302)
+ try {
+ await this.describeCollection(name);
+ this.logger?.debug?.(`${TAG} Collection already exists: ${name}`);
+ return;
+ } catch (err) {
+ if (!(err instanceof TcvdbApiError && err.apiCode === 15302)) {
+ throw err; // unexpected error
+ }
+ // 15302 = collection not found → proceed to create
+ }
+ try {
+ await this.request("/collection/create", {
+ database: this.database,
+ ...params,
+ });
+ this.logger?.info(`${TAG} Collection created: ${name}`);
+ } catch (err) {
+ // 15202 = collection already exists — race between describe and create.
+ // Semantically identical to "describe found it", so treat as success.
+ if (err instanceof TcvdbApiError && err.apiCode === 15202) {
+ this.logger?.debug?.(`${TAG} Collection already exists (race): ${name}`);
+ return;
+ }
+ throw err;
+ }
+ }
+
+ async describeCollection(collection: string): Promise {
+ const resp = await this.request<{ collection: CollectionInfo }>("/collection/describe", {
+ database: this.database,
+ collection,
+ });
+ return resp.collection;
+ }
+
+ // ── Document operations ─────────────────────────────────
+
+ async upsert(collection: string, documents: Record[]): Promise {
+ await this.request("/document/upsert", {
+ database: this.database,
+ collection,
+ buildIndex: true,
+ documents,
+ });
+ }
+
+ async search(collection: string, searchParams: Record): Promise {
+ return this.request("/document/search", {
+ database: this.database,
+ collection,
+ readConsistency: "strongConsistency",
+ search: searchParams,
+ });
+ }
+
+ async hybridSearch(collection: string, searchParams: Record): Promise {
+ return this.request("/document/hybridSearch", {
+ database: this.database,
+ collection,
+ readConsistency: "strongConsistency",
+ search: searchParams,
+ });
+ }
+
+ async query(collection: string, queryParams: Record): Promise {
+ return this.request("/document/query", {
+ database: this.database,
+ collection,
+ readConsistency: "strongConsistency",
+ query: queryParams,
+ });
+ }
+
+ async deleteDoc(collection: string, params: Record): Promise {
+ await this.request("/document/delete", {
+ database: this.database,
+ collection,
+ ...params,
+ });
+ }
+
+ /**
+ * Count documents matching an optional filter.
+ * Uses the dedicated /document/count endpoint.
+ */
+ async count(collection: string, filter?: string): Promise {
+ const query: Record = {};
+ if (filter) query.filter = filter;
+ const resp = await this.request<{ count: number }>("/document/count", {
+ database: this.database,
+ collection,
+ readConsistency: "strongConsistency",
+ query,
+ });
+ return resp.count ?? 0;
+ }
+
+ // ── Convenience getters ─────────────────────────────────
+
+ getDatabase(): string {
+ return this.database;
+ }
+}
diff --git a/src/store/tcvdb.ts b/src/store/tcvdb.ts
new file mode 100644
index 0000000..35fe6a9
--- /dev/null
+++ b/src/store/tcvdb.ts
@@ -0,0 +1,1180 @@
+/**
+ * TcvdbMemoryStore: Tencent Cloud VectorDB backend implementing IMemoryStore.
+ *
+ * Features:
+ * - Server-side dense embedding (embeddingItems via Collection embedding config)
+ * - Client-side sparse vectors (BM25 local encoder for hybridSearch)
+ * - Native hybridSearch (dense + sparse + RRFRerank)
+ * - Filter expressions for scalar field queries
+ * - Time fields stored as uint64 epoch ms (ISO ↔ epoch conversion internal)
+ *
+ * All methods are fault-tolerant: return empty/false on error, never throw.
+ */
+
+import type { MemoryRecord } from "../record/l1-writer.js";
+import type { EmbeddingProviderInfo } from "./embedding.js";
+import type {
+ IMemoryStore,
+ StoreCapabilities,
+ StoreInitResult,
+ L1SearchResult,
+ L1FtsResult,
+ L1RecordRow,
+ L1QueryFilter,
+ L0SearchResult,
+ L0FtsResult,
+ L0QueryRow,
+ L0SessionGroup,
+ ProfileRecord,
+ ProfileSyncRecord,
+ StoreLogger,
+} from "./types.js";
+import { TcvdbClient, TcvdbApiError } from "./tcvdb-client.js";
+import type { BM25LocalEncoder } from "./bm25-local.js";
+import type { SparseVector } from "@tencentdb-agent-memory/tcvdb-text";
+
+// ============================
+// Config & Constants
+// ============================
+
+export interface TcvdbMemoryStoreConfig {
+ url: string;
+ username: string;
+ apiKey: string;
+ database: string;
+ embeddingModel: string;
+ timeout: number;
+ /** Path to CA certificate PEM file (for HTTPS connections) */
+ caPemPath?: string;
+ logger?: StoreLogger;
+ bm25Encoder?: BM25LocalEncoder;
+}
+
+const TAG = "[memory-tdai][tcvdb]";
+
+/** Base collection suffixes (prefixed with database name at construction time). */
+const L1_COLLECTION_SUFFIX = "l1_memories";
+const L0_COLLECTION_SUFFIX = "l0_conversations";
+const PROFILES_COLLECTION_SUFFIX = "profiles";
+
+/** Max documents per /document/query page (VectorDB API limit). */
+const QUERY_PAGE_SIZE = 100;
+
+/** All L1 output fields returned by query/search (excludes vector/sparse_vector). */
+const L1_OUTPUT_FIELDS = [
+ "id", "text", "type", "priority", "scene_name",
+ "session_key", "session_id", "timestamp_str", "timestamp_start",
+ "timestamp_end", "metadata_json", "created_time_ms", "updated_time_ms",
+];
+
+/** All L0 output fields returned by query/search. */
+const L0_OUTPUT_FIELDS = [
+ "id", "message_text", "agent_id", "session_key", "session_id", "role",
+ "recorded_at_ms", "timestamp",
+];
+
+const PROFILE_OUTPUT_FIELDS = [
+ "id", "type", "filename", "content", "content_md5", "agent_id",
+ "version", "created_at_ms", "updated_at_ms",
+];
+
+const PROFILE_METADATA_OUTPUT_FIELDS = [
+ "id", "type", "filename", "content_md5", "agent_id",
+ "version", "created_at_ms", "updated_at_ms",
+];
+
+// ============================
+// Helpers
+// ============================
+
+function isoToEpochMs(iso: string): number {
+ if (!iso) return 0;
+ const ms = new Date(iso).getTime();
+ return Number.isFinite(ms) ? ms : 0;
+}
+
+function epochMsToIso(ms: number): string {
+ if (!ms || ms <= 0) return "";
+ return new Date(ms).toISOString();
+}
+
+/**
+ * Extract agent ID from a sessionKey like `agent::`.
+ * Returns empty string if the format doesn't match.
+ */
+function extractAgentId(sessionKey: string): string {
+ if (!sessionKey) return "";
+ const parts = sessionKey.split(":");
+ // Format: "agent::..." → parts[1]
+ if (parts.length >= 2 && parts[0] === "agent") {
+ return parts[1];
+ }
+ return "";
+}
+
+// ============================
+// TcvdbMemoryStore
+// ============================
+
+export class TcvdbMemoryStore implements IMemoryStore {
+ private readonly client: TcvdbClient;
+ private readonly embeddingModel: string;
+ private readonly logger?: StoreLogger;
+ private readonly bm25Encoder?: BM25LocalEncoder;
+ private readonly l1Collection: string;
+ private readonly l0Collection: string;
+ private readonly profilesCollection: string;
+ private degraded = false;
+
+ /** Promise that resolves when async init completes. */
+ private _initPromise: Promise | undefined;
+
+ constructor(config: TcvdbMemoryStoreConfig) {
+ this.client = new TcvdbClient({
+ url: config.url,
+ username: config.username,
+ apiKey: config.apiKey,
+ database: config.database,
+ timeout: config.timeout,
+ caPemPath: config.caPemPath,
+ }, config.logger);
+ this.embeddingModel = config.embeddingModel;
+ this.logger = config.logger;
+ this.bm25Encoder = config.bm25Encoder;
+
+ // Collection names are globally unique within a TCVDB instance,
+ // so prefix with database name to avoid cross-database collisions.
+ this.l1Collection = `${config.database}_${L1_COLLECTION_SUFFIX}`;
+ this.l0Collection = `${config.database}_${L0_COLLECTION_SUFFIX}`;
+ this.profilesCollection = `${config.database}_${PROFILES_COLLECTION_SUFFIX}`;
+ }
+
+ // ── Lifecycle ────────────────────────────────────────────
+
+ async init(_providerInfo?: EmbeddingProviderInfo): Promise {
+ // TCVDB init is async (HTTP). We store the promise so _ensureInit()
+ // can also await it as a defensive fallback in each data method.
+ this._initPromise = this._initAsync();
+ try {
+ await this._initPromise;
+ } catch (err) {
+ this.logger?.error(`${TAG} Async init failed: ${err instanceof Error ? err.message : String(err)}`);
+ this.degraded = true;
+ }
+ return { needsReindex: false };
+ }
+
+ /**
+ * Await async initialization. Call at the start of every async method.
+ * If init already completed (or failed → degraded), returns immediately.
+ */
+ private async _ensureInit(): Promise {
+ if (this._initPromise) {
+ await this._initPromise;
+ }
+ }
+
+ // ── Vector index definitions ─────────────────────────────
+ //
+ // Preferred: DISK_FLAT (lower memory, suitable for large-scale recall).
+ // Fallback: HNSW (for instances whose storage engine doesn't support DISK_FLAT).
+
+ private static readonly VECTOR_INDEX_DISK_FLAT: Record = {
+ fieldName: "vector", fieldType: "vector", indexType: "DISK_FLAT",
+ dimension: 1024, metricType: "COSINE",
+ };
+
+ private static readonly VECTOR_INDEX_HNSW: Record = {
+ fieldName: "vector", fieldType: "vector", indexType: "HNSW",
+ dimension: 1024, metricType: "COSINE",
+ params: { M: 16, efConstruction: 200 },
+ };
+
+ /**
+ * Detect whether a createCollection error indicates DISK_FLAT is unsupported.
+ * Matches on apiCode 15113 OR message containing "DISK_FLAT" + "not support".
+ */
+ private static isDiskFlatUnsupported(err: unknown): boolean {
+ if (!(err instanceof TcvdbApiError)) return false;
+ if (err.apiCode === 15113) return true;
+ const msg = err.message.toLowerCase();
+ return msg.includes("disk_flat") && (msg.includes("not support") || msg.includes("unsupported"));
+ }
+
+ /**
+ * Create a collection with DISK_FLAT vector index, falling back to HNSW
+ * if the storage engine doesn't support DISK_FLAT.
+ */
+ private async _createCollectionWithVectorFallback(
+ params: Record,
+ filterIndexes: Array>,
+ ): Promise {
+ const buildIndexes = (vectorIndex: Record) => [
+ { fieldName: "id", fieldType: "string", indexType: "primaryKey" },
+ vectorIndex,
+ { fieldName: "sparse_vector", fieldType: "sparseVector", indexType: "inverted", metricType: "IP" },
+ ...filterIndexes,
+ ];
+
+ try {
+ await this.client.createCollection({ ...params, indexes: buildIndexes(TcvdbMemoryStore.VECTOR_INDEX_DISK_FLAT) });
+ } catch (err) {
+ if (TcvdbMemoryStore.isDiskFlatUnsupported(err)) {
+ this.logger?.debug?.(`${TAG} DISK_FLAT not supported for ${String(params.collection)}, falling back to HNSW`);
+ await this.client.createCollection({ ...params, indexes: buildIndexes(TcvdbMemoryStore.VECTOR_INDEX_HNSW) });
+ } else {
+ throw err;
+ }
+ }
+ }
+
+ private async _initAsync(): Promise {
+ try {
+ // Create database (idempotent — returns true if just created, false if already existed)
+ const dbCreated = await this.client.createDatabase();
+
+ if (dbCreated) {
+ // TCVDB requires ~3s after database creation before collections can be created.
+ // TODO: defer collection creation to first use to avoid blocking plugin startup.
+ this.logger?.debug?.(`${TAG} Waiting 5s for database to become ready...`);
+ await new Promise((r) => setTimeout(r, 5_000));
+ }
+
+ // Create L1 collection (DISK_FLAT preferred, HNSW fallback)
+ await this._createCollectionWithVectorFallback(
+ {
+ collection: this.l1Collection,
+ shardNum: 1,
+ replicaNum: 2,
+ description: "L1 结构化记忆",
+ embedding: {
+ status: "enabled",
+ field: "text",
+ vectorField: "vector",
+ model: this.embeddingModel,
+ },
+ },
+ [
+ { fieldName: "type", fieldType: "string", indexType: "filter" },
+ { fieldName: "priority", fieldType: "uint64", indexType: "filter" },
+ { fieldName: "scene_name", fieldType: "string", indexType: "filter" },
+ { fieldName: "agent_id", fieldType: "string", indexType: "filter" },
+ { fieldName: "session_key", fieldType: "string", indexType: "filter" },
+ { fieldName: "session_id", fieldType: "string", indexType: "filter" },
+ { fieldName: "timestamp_start", fieldType: "string", indexType: "filter" },
+ { fieldName: "timestamp_end", fieldType: "string", indexType: "filter" },
+ { fieldName: "created_time_ms", fieldType: "uint64", indexType: "filter" },
+ { fieldName: "updated_time_ms", fieldType: "uint64", indexType: "filter" },
+ ],
+ );
+
+ // Create L0 collection (DISK_FLAT preferred, HNSW fallback)
+ await this._createCollectionWithVectorFallback(
+ {
+ collection: this.l0Collection,
+ shardNum: 1,
+ replicaNum: 2,
+ description: "L0 原始对话消息",
+ embedding: {
+ status: "enabled",
+ field: "message_text",
+ vectorField: "vector",
+ model: this.embeddingModel,
+ },
+ },
+ [
+ { fieldName: "agent_id", fieldType: "string", indexType: "filter" },
+ { fieldName: "session_key", fieldType: "string", indexType: "filter" },
+ { fieldName: "session_id", fieldType: "string", indexType: "filter" },
+ { fieldName: "role", fieldType: "string", indexType: "filter" },
+ { fieldName: "recorded_at_ms", fieldType: "uint64", indexType: "filter" },
+ { fieldName: "timestamp", fieldType: "int64", indexType: "filter" },
+ ],
+ );
+
+ await this.client.createCollection({
+ collection: this.profilesCollection,
+ shardNum: 1,
+ replicaNum: 2,
+ description: "L2 场景块 + L3 用户画像",
+ embedding: { status: "disabled" },
+ indexes: [
+ { fieldName: "id", fieldType: "string", indexType: "primaryKey" },
+ { fieldName: "vector", fieldType: "vector", indexType: "FLAT",
+ dimension: 1, metricType: "COSINE" },
+ { fieldName: "type", fieldType: "string", indexType: "filter" },
+ { fieldName: "filename", fieldType: "string", indexType: "filter" },
+ { fieldName: "content_md5", fieldType: "string", indexType: "filter" },
+ { fieldName: "agent_id", fieldType: "string", indexType: "filter" },
+ { fieldName: "created_at_ms", fieldType: "uint64", indexType: "filter" },
+ { fieldName: "updated_at_ms", fieldType: "uint64", indexType: "filter" },
+ { fieldName: "version", fieldType: "uint64", indexType: "filter" },
+ ],
+ });
+
+ this.logger?.debug?.(`${TAG} Initialized: db=${this.client.getDatabase()}, model=${this.embeddingModel}`);
+ } catch (err) {
+ // 15201 = database already exists — benign race in createDatabase().
+ // 15202 (collection already exists) is now handled inside TcvdbClient.createCollection(),
+ // so it should no longer reach here.
+ if (err instanceof TcvdbApiError && err.apiCode === 15201) {
+ this.logger?.debug?.(`${TAG} Init (benign): ${err.message}`);
+ return;
+ }
+ this.logger?.error(`${TAG} Init failed: ${err instanceof Error ? err.message : String(err)}`);
+ this.degraded = true;
+ }
+ }
+
+ isDegraded(): boolean {
+ return this.degraded;
+ }
+
+ getCapabilities(): StoreCapabilities {
+ const hasBm25 = !!this.bm25Encoder;
+ return {
+ vectorSearch: true,
+ ftsSearch: hasBm25,
+ nativeHybridSearch: hasBm25,
+ sparseVectors: hasBm25,
+ };
+ }
+
+ close(): void {
+ // HTTP client — nothing to close
+ }
+
+ // ── Internal: paginated query helper ────────────────────
+
+ /**
+ * Paginated /document/query that fetches all matching docs.
+ * TCVDB query API returns at most `limit` docs per call.
+ * We loop with offset until fewer docs than page size are returned.
+ */
+ private async _queryAllDocs(
+ collection: string,
+ filter?: string,
+ outputFields?: string[],
+ limit?: number,
+ sort?: Array>,
+ ): Promise>> {
+ const allDocs: Array> = [];
+ let offset = 0;
+ const pageSize = limit && limit < QUERY_PAGE_SIZE ? limit : QUERY_PAGE_SIZE;
+
+ // eslint-disable-next-line no-constant-condition
+ while (true) {
+ const queryParams: Record = {
+ retrieveVector: false,
+ limit: pageSize,
+ offset,
+ };
+ if (filter) queryParams.filter = filter;
+ if (outputFields) queryParams.outputFields = outputFields;
+ if (sort) queryParams.sort = sort;
+
+ const resp = await this.client.query(collection, queryParams);
+ const docs = resp.documents ?? [];
+ allDocs.push(...docs);
+
+ // Stop if: we got fewer than page size (last page), or we hit caller's limit
+ if (docs.length < pageSize) break;
+ if (limit && allDocs.length >= limit) break;
+
+ offset += docs.length;
+ }
+
+ // Trim to caller's limit if specified
+ return limit ? allDocs.slice(0, limit) : allDocs;
+ }
+
+ // ── L1 Write Operations ──────────────────────────────────
+
+ async upsertL1(record: MemoryRecord, _embedding?: Float32Array): Promise {
+ try {
+ await this._upsertL1Async(record);
+ return true;
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L1-upsert] FAILED id=${record.id}: ${err instanceof Error ? err.message : String(err)}`);
+ return false;
+ }
+ }
+
+ private async _upsertL1Async(record: MemoryRecord): Promise {
+ await this._ensureInit();
+ if (this.degraded) return;
+
+ const tsStr = record.timestamps[0] ?? "";
+ const tsStart = record.timestamps.length > 0
+ ? record.timestamps.reduce((a, b) => (a < b ? a : b)) : tsStr;
+ const tsEnd = record.timestamps.length > 0
+ ? record.timestamps.reduce((a, b) => (a > b ? a : b)) : tsStr;
+
+ const doc: Record = {
+ id: record.id,
+ text: record.content,
+ type: record.type,
+ priority: record.priority,
+ scene_name: record.scene_name,
+ agent_id: extractAgentId(record.sessionKey),
+ session_key: record.sessionKey,
+ session_id: record.sessionId,
+ timestamp_str: tsStr,
+ timestamp_start: tsStart,
+ timestamp_end: tsEnd,
+ created_time_ms: isoToEpochMs(record.createdAt),
+ updated_time_ms: isoToEpochMs(record.updatedAt),
+ metadata_json: JSON.stringify(record.metadata),
+ };
+
+ // BM25 sparse vector (if sidecar available)
+ if (this.bm25Encoder) {
+ const sparse = this.bm25Encoder.encodeTexts([record.content]);
+ if (sparse.length > 0 && sparse[0].length > 0) {
+ doc.sparse_vector = sparse[0];
+ }
+ }
+
+ await this.client.upsert(this.l1Collection, [doc]);
+ }
+
+ /**
+ * Batch upsert multiple L1 records in a single API call.
+ * Used by migration scripts to reduce request count.
+ */
+ async upsertL1Batch(records: MemoryRecord[]): Promise {
+ if (records.length === 0) return 0;
+ try {
+ await this._ensureInit();
+ if (this.degraded) return 0;
+
+ const docs = records.map((record) => {
+ const tsStr = record.timestamps[0] ?? "";
+ const tsStart = record.timestamps.length > 0
+ ? record.timestamps.reduce((a, b) => (a < b ? a : b)) : tsStr;
+ const tsEnd = record.timestamps.length > 0
+ ? record.timestamps.reduce((a, b) => (a > b ? a : b)) : tsStr;
+
+ const doc: Record = {
+ id: record.id,
+ text: record.content,
+ type: record.type,
+ priority: record.priority,
+ scene_name: record.scene_name,
+ agent_id: extractAgentId(record.sessionKey),
+ session_key: record.sessionKey,
+ session_id: record.sessionId,
+ timestamp_str: tsStr,
+ timestamp_start: tsStart,
+ timestamp_end: tsEnd,
+ created_time_ms: isoToEpochMs(record.createdAt),
+ updated_time_ms: isoToEpochMs(record.updatedAt),
+ metadata_json: JSON.stringify(record.metadata),
+ };
+
+ if (this.bm25Encoder) {
+ const sparse = this.bm25Encoder.encodeTexts([record.content]);
+ if (sparse.length > 0 && sparse[0].length > 0) {
+ doc.sparse_vector = sparse[0];
+ }
+ }
+ return doc;
+ });
+
+ await this.client.upsert(this.l1Collection, docs);
+ return records.length;
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L1-upsertBatch] FAILED (${records.length} records): ${err instanceof Error ? err.message : String(err)}`);
+ return 0;
+ }
+ }
+
+ async deleteL1(recordId: string): Promise {
+ try {
+ await this._ensureInit();
+ if (this.degraded) return false;
+ await this.client.deleteDoc(this.l1Collection, {
+ query: { documentIds: [recordId] },
+ });
+ return true;
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L1-delete] FAILED id=${recordId}: ${err instanceof Error ? err.message : String(err)}`);
+ return false;
+ }
+ }
+
+ async deleteL1Batch(recordIds: string[]): Promise {
+ if (recordIds.length === 0) return true;
+ try {
+ await this._ensureInit();
+ if (this.degraded) return false;
+ await this.client.deleteDoc(this.l1Collection, {
+ query: { documentIds: recordIds },
+ });
+ return true;
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L1-deleteBatch] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return false;
+ }
+ }
+
+ async deleteL1Expired(cutoffIso: string): Promise {
+ const cutoffMs = isoToEpochMs(cutoffIso);
+ if (cutoffMs <= 0) return 0;
+ try {
+ await this._ensureInit();
+ if (this.degraded) return 0;
+ await this.client.deleteDoc(this.l1Collection, {
+ query: { filter: `updated_time_ms < ${cutoffMs}` },
+ });
+ return 0; // actual count unknown from delete API
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L1-deleteExpired] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return 0;
+ }
+ }
+
+ // ── L1 Read Operations ───────────────────────────────────
+
+ async countL1(): Promise {
+ try {
+ await this._ensureInit();
+ if (this.degraded) return 0;
+ return await this.client.count(this.l1Collection);
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L1-count] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return 0;
+ }
+ }
+
+ async queryL1Records(filter?: L1QueryFilter): Promise {
+ try {
+ await this._ensureInit();
+ if (this.degraded) return [];
+
+ // Build TCVDB filter expression from L1QueryFilter
+ const conditions: string[] = [];
+ if (filter?.sessionKey) conditions.push(`session_key = "${filter.sessionKey}"`);
+ if (filter?.sessionId) conditions.push(`session_id = "${filter.sessionId}"`);
+ if (filter?.updatedAfter) {
+ const afterMs = isoToEpochMs(filter.updatedAfter);
+ if (afterMs > 0) conditions.push(`updated_time_ms > ${afterMs}`);
+ }
+ const filterExpr = conditions.length > 0 ? conditions.join(" and ") : undefined;
+
+ const docs = await this._queryAllDocs(
+ this.l1Collection,
+ filterExpr,
+ L1_OUTPUT_FIELDS,
+ undefined, // no limit — fetch all matching
+ [{ fieldName: "updated_time_ms", direction: "asc" }],
+ );
+
+ return docs.map((doc) => ({
+ record_id: String(doc.id ?? ""),
+ content: String(doc.text ?? ""),
+ type: String(doc.type ?? ""),
+ priority: Number(doc.priority ?? 0),
+ scene_name: String(doc.scene_name ?? ""),
+ session_key: String(doc.session_key ?? ""),
+ session_id: String(doc.session_id ?? ""),
+ timestamp_str: String(doc.timestamp_str ?? ""),
+ timestamp_start: String(doc.timestamp_start ?? ""),
+ timestamp_end: String(doc.timestamp_end ?? ""),
+ created_time: epochMsToIso(Number(doc.created_time_ms ?? 0)),
+ updated_time: epochMsToIso(Number(doc.updated_time_ms ?? 0)),
+ metadata_json: String(doc.metadata_json ?? "{}"),
+ }));
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L1-query] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return [];
+ }
+ }
+
+ async getAllL1Texts(): Promise> {
+ try {
+ await this._ensureInit();
+ if (this.degraded) return [];
+
+ const docs = await this._queryAllDocs(
+ this.l1Collection,
+ undefined,
+ ["id", "text", "updated_time_ms"],
+ );
+
+ return docs.map((doc) => ({
+ record_id: String(doc.id ?? ""),
+ content: String(doc.text ?? ""),
+ updated_time: epochMsToIso(Number(doc.updated_time_ms ?? 0)),
+ }));
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L1-getAllTexts] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return [];
+ }
+ }
+
+ // ── L1 Search Operations ─────────────────────────────────
+
+ async searchL1Vector(_queryEmbedding: Float32Array, topK?: number, queryText?: string): Promise {
+ // TCVDB uses server-side embedding — delegate to hybrid search with text
+ if (queryText) {
+ return this.searchL1HybridAsync({ queryText, topK });
+ }
+ // No queryText and TCVDB can't use client embeddings directly via embeddingItems
+ // Return empty — callers should pass queryText for TCVDB
+ return [];
+ }
+
+ async searchL1Fts(ftsQuery: string, limit?: number): Promise {
+ // TCVDB has no pure FTS — use hybrid search with sparse-only path
+ // The ftsQuery is raw text, use it as queryText for hybrid
+ if (!ftsQuery) return [];
+ const results = await this.searchL1HybridAsync({ queryText: ftsQuery, topK: limit });
+ // L1SearchResult and L1FtsResult have identical shapes
+ return results;
+ }
+
+ async searchL1Hybrid(params: {
+ query?: string;
+ queryEmbedding?: Float32Array;
+ sparseVector?: SparseVector;
+ topK?: number;
+ }): Promise {
+ const queryText = params.query;
+ if (!queryText) return [];
+ return this.searchL1HybridAsync({ queryText, topK: params.topK });
+ }
+
+ /**
+ * Async L1 hybrid search — the real implementation.
+ * Call this directly from async contexts (hooks, tools).
+ */
+ async searchL1HybridAsync(params: {
+ queryText: string;
+ topK?: number;
+ }): Promise {
+ const { queryText, topK = 10 } = params;
+ if (!queryText) return [];
+
+ try {
+ await this._ensureInit();
+ if (this.degraded) return [];
+
+ // Build search params
+ const searchParams: Record = {
+ limit: topK,
+ outputFields: L1_OUTPUT_FIELDS,
+ };
+
+ // ann: use embedding field name "text" for server-side embedding
+ // (per SDK: AnnSearch(field_name="text", data='query string'))
+ const ann = [{
+ fieldName: "text",
+ data: [queryText], // embeddingItems — server-side embedding
+ limit: topK,
+ }];
+
+ let match: Array> | undefined;
+ if (this.bm25Encoder) {
+ const sparse = this.bm25Encoder.encodeQueries([queryText]);
+ if (sparse.length > 0 && sparse[0].length > 0) {
+ match = [{
+ fieldName: "sparse_vector",
+ data: [sparse[0]], // SDK wraps single sparse vector in array
+ limit: topK,
+ }];
+ }
+ }
+
+ if (match) {
+ // Full hybrid: dense + sparse + RRF
+ searchParams.ann = ann;
+ searchParams.match = match;
+ searchParams.rerank = { method: "rrf", k: 60 };
+
+ const resp = await this.client.hybridSearch(this.l1Collection, searchParams);
+ return this._parseL1SearchResults(resp.documents);
+ } else {
+ // Dense-only fallback (BM25 unavailable) — use /document/search with embeddingItems
+ const denseSearch: Record = {
+ embeddingItems: [queryText],
+ limit: topK,
+ retrieveVector: false,
+ outputFields: L1_OUTPUT_FIELDS,
+ };
+ const resp = await this.client.search(this.l1Collection, denseSearch);
+ return this._parseL1SearchResults(resp.documents);
+ }
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L1-hybridSearch] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return [];
+ }
+ }
+
+ // ── L0 Write Operations ──────────────────────────────────
+
+ async upsertL0(record: { id: string; sessionKey: string; sessionId: string; role: string; messageText: string; recordedAt: string; timestamp: number }, _embedding?: Float32Array): Promise {
+ try {
+ await this._upsertL0Async(record);
+ return true;
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L0-upsert] FAILED id=${record.id}: ${err instanceof Error ? err.message : String(err)}`);
+ return false;
+ }
+ }
+
+ private async _upsertL0Async(record: { id: string; sessionKey: string; sessionId: string; role: string; messageText: string; recordedAt: string; timestamp: number }): Promise {
+ await this._ensureInit();
+ if (this.degraded) return;
+
+ const doc: Record = {
+ id: record.id,
+ message_text: record.messageText,
+ agent_id: extractAgentId(record.sessionKey),
+ session_key: record.sessionKey,
+ session_id: record.sessionId,
+ role: record.role,
+ recorded_at_ms: isoToEpochMs(record.recordedAt),
+ timestamp: record.timestamp,
+ };
+
+ if (this.bm25Encoder) {
+ const sparse = this.bm25Encoder.encodeTexts([record.messageText]);
+ if (sparse.length > 0 && sparse[0].length > 0) {
+ doc.sparse_vector = sparse[0];
+ }
+ }
+
+ await this.client.upsert(this.l0Collection, [doc]);
+ }
+
+ /**
+ * Batch upsert multiple L0 records in a single API call.
+ * Used by migration scripts to reduce request count.
+ */
+ async upsertL0Batch(records: Array<{ id: string; sessionKey: string; sessionId: string; role: string; messageText: string; recordedAt: string; timestamp: number }>): Promise {
+ if (records.length === 0) return 0;
+ try {
+ await this._ensureInit();
+ if (this.degraded) return 0;
+
+ const docs = records.map((record) => {
+ const doc: Record = {
+ id: record.id,
+ message_text: record.messageText,
+ agent_id: extractAgentId(record.sessionKey),
+ session_key: record.sessionKey,
+ session_id: record.sessionId,
+ role: record.role,
+ recorded_at_ms: isoToEpochMs(record.recordedAt),
+ timestamp: record.timestamp,
+ };
+
+ if (this.bm25Encoder) {
+ const sparse = this.bm25Encoder.encodeTexts([record.messageText]);
+ if (sparse.length > 0 && sparse[0].length > 0) {
+ doc.sparse_vector = sparse[0];
+ }
+ }
+ return doc;
+ });
+
+ await this.client.upsert(this.l0Collection, docs);
+ return records.length;
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L0-upsertBatch] FAILED (${records.length} records): ${err instanceof Error ? err.message : String(err)}`);
+ return 0;
+ }
+ }
+
+ async deleteL0(recordId: string): Promise {
+ try {
+ await this._ensureInit();
+ if (this.degraded) return false;
+ await this.client.deleteDoc(this.l0Collection, {
+ query: { documentIds: [recordId] },
+ });
+ return true;
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L0-delete] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return false;
+ }
+ }
+
+ async deleteL0Expired(cutoffIso: string): Promise {
+ const cutoffMs = isoToEpochMs(cutoffIso);
+ if (cutoffMs <= 0) return 0;
+ try {
+ await this._ensureInit();
+ if (this.degraded) return 0;
+ await this.client.deleteDoc(this.l0Collection, {
+ query: { filter: `recorded_at_ms < ${cutoffMs}` },
+ });
+ return 0;
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L0-deleteExpired] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return 0;
+ }
+ }
+
+ // ── L0 Read Operations ───────────────────────────────────
+
+ async countL0(): Promise {
+ try {
+ await this._ensureInit();
+ if (this.degraded) return 0;
+ return await this.client.count(this.l0Collection);
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L0-count] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return 0;
+ }
+ }
+
+ async queryL0ForL1(sessionKey: string, afterRecordedAtMs?: number, limit = 50): Promise {
+ try {
+ await this._ensureInit();
+ if (this.degraded) return [];
+
+ const conditions: string[] = [`session_key = "${sessionKey}"`];
+ if (afterRecordedAtMs && afterRecordedAtMs > 0) {
+ conditions.push(`recorded_at_ms > ${afterRecordedAtMs}`);
+ }
+ const filterExpr = conditions.join(" and ");
+
+ const docs = await this._queryAllDocs(
+ this.l0Collection,
+ filterExpr,
+ L0_OUTPUT_FIELDS,
+ limit,
+ [{ fieldName: "recorded_at_ms", direction: "desc" }],
+ );
+
+ // Convert to L0QueryRow and reverse to chronological order (query is DESC, callers expect ASC)
+ const rows: L0QueryRow[] = docs.map((doc) => ({
+ record_id: String(doc.id ?? ""),
+ session_key: String(doc.session_key ?? ""),
+ session_id: String(doc.session_id ?? ""),
+ role: String(doc.role ?? ""),
+ message_text: String(doc.message_text ?? ""),
+ recorded_at: epochMsToIso(Number(doc.recorded_at_ms ?? 0)),
+ timestamp: Number(doc.timestamp ?? 0),
+ }));
+
+ return rows.reverse();
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L0-queryForL1] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return [];
+ }
+ }
+
+ async queryL0GroupedBySessionId(sessionKey: string, afterRecordedAtMs?: number, limit = 50): Promise {
+ try {
+ const rows = await this.queryL0ForL1(sessionKey, afterRecordedAtMs, limit);
+
+ // Group by session_id
+ const groupMap = new Map>();
+ for (const row of rows) {
+ const sid = row.session_id || "";
+ let group = groupMap.get(sid);
+ if (!group) {
+ group = [];
+ groupMap.set(sid, group);
+ }
+ group.push({
+ id: row.record_id,
+ role: row.role,
+ content: row.message_text,
+ timestamp: row.timestamp,
+ recordedAtMs: row.recorded_at ? Date.parse(row.recorded_at) || 0 : 0,
+ });
+ }
+
+ // Convert to array, sorted by earliest message timestamp
+ const groups: L0SessionGroup[] = [];
+ for (const [sessionId, messages] of groupMap) {
+ if (messages.length > 0) {
+ groups.push({ sessionId, messages });
+ }
+ }
+ groups.sort((a, b) => a.messages[0].timestamp - b.messages[0].timestamp);
+
+ return groups;
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L0-queryGrouped] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return [];
+ }
+ }
+
+ async getAllL0Texts(): Promise> {
+ try {
+ await this._ensureInit();
+ if (this.degraded) return [];
+
+ const docs = await this._queryAllDocs(
+ this.l0Collection,
+ undefined,
+ ["id", "message_text", "recorded_at_ms"],
+ );
+
+ return docs.map((doc) => ({
+ record_id: String(doc.id ?? ""),
+ message_text: String(doc.message_text ?? ""),
+ recorded_at: epochMsToIso(Number(doc.recorded_at_ms ?? 0)),
+ }));
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L0-getAllTexts] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return [];
+ }
+ }
+
+ // ── L0 Search Operations ─────────────────────────────────
+
+ async searchL0Vector(_queryEmbedding: Float32Array, topK?: number, queryText?: string): Promise {
+ // TCVDB uses server-side embedding — delegate to hybrid search with text
+ if (queryText) {
+ return this.searchL0HybridAsync({ queryText, topK });
+ }
+ return [];
+ }
+
+ async searchL0Fts(ftsQuery: string, limit?: number): Promise {
+ if (!ftsQuery) return [];
+ // Use hybrid search; L0SearchResult and L0FtsResult have identical shapes
+ return this.searchL0HybridAsync({ queryText: ftsQuery, topK: limit });
+ }
+
+ /**
+ * Async L0 hybrid search.
+ */
+ async searchL0HybridAsync(params: {
+ queryText: string;
+ topK?: number;
+ }): Promise {
+ const { queryText, topK = 10 } = params;
+ if (!queryText) return [];
+
+ try {
+ await this._ensureInit();
+ if (this.degraded) return [];
+
+ const searchParams: Record = {
+ limit: topK,
+ outputFields: L0_OUTPUT_FIELDS,
+ };
+
+ // ann: use embedding field name "message_text" for L0 server-side embedding
+ const ann = [{
+ fieldName: "message_text",
+ data: [queryText],
+ limit: topK,
+ }];
+
+ let match: Array> | undefined;
+ if (this.bm25Encoder) {
+ const sparse = this.bm25Encoder.encodeQueries([queryText]);
+ if (sparse.length > 0 && sparse[0].length > 0) {
+ match = [{
+ fieldName: "sparse_vector",
+ data: [sparse[0]],
+ limit: topK,
+ }];
+ }
+ }
+
+ if (match) {
+ searchParams.ann = ann;
+ searchParams.match = match;
+ searchParams.rerank = { method: "rrf", k: 60 };
+ const resp = await this.client.hybridSearch(this.l0Collection, searchParams);
+ return this._parseL0SearchResults(resp.documents);
+ } else {
+ const denseSearch: Record = {
+ embeddingItems: [queryText],
+ limit: topK,
+ retrieveVector: false,
+ outputFields: L0_OUTPUT_FIELDS,
+ };
+ const resp = await this.client.search(this.l0Collection, denseSearch);
+ return this._parseL0SearchResults(resp.documents);
+ }
+ } catch (err) {
+ this.logger?.warn(`${TAG} [L0-hybridSearch] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return [];
+ }
+ }
+
+ async pullProfiles(): Promise {
+ try {
+ await this._ensureInit();
+ if (this.degraded) return [];
+
+ const docs = await this._queryAllDocs(
+ this.profilesCollection,
+ undefined,
+ PROFILE_OUTPUT_FIELDS,
+ );
+
+ return docs.map((doc) => ({
+ id: String(doc.id ?? ""),
+ type: doc.type === "l3" ? "l3" : "l2",
+ filename: String(doc.filename ?? ""),
+ content: String(doc.content ?? ""),
+ contentMd5: String(doc.content_md5 ?? ""),
+ agentId: String(doc.agent_id ?? "") || undefined,
+ version: Number(doc.version ?? 0),
+ createdAtMs: Number(doc.created_at_ms ?? 0),
+ updatedAtMs: Number(doc.updated_at_ms ?? 0),
+ }));
+ } catch (err) {
+ this.logger?.warn(`${TAG} [profiles-pull] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ return [];
+ }
+ }
+
+ async syncProfiles(records: ProfileSyncRecord[]): Promise {
+ if (records.length === 0) return;
+
+ try {
+ await this._ensureInit();
+ if (this.degraded) return;
+
+ const remoteDocs = await this._queryAllDocs(
+ this.profilesCollection,
+ undefined,
+ PROFILE_METADATA_OUTPUT_FIELDS,
+ );
+ const remoteMap = new Map(
+ remoteDocs.map((doc) => [String(doc.id ?? ""), doc] as const),
+ );
+ const now = Date.now();
+ const upserts: Array> = [];
+
+ for (const record of records) {
+ const current = remoteMap.get(record.id);
+ if (!current) {
+ const createdAtMs = record.createdAtMs > 0 ? record.createdAtMs : now;
+ upserts.push({
+ id: record.id,
+ vector: [0],
+ type: record.type,
+ filename: record.filename,
+ content: record.content,
+ content_md5: record.contentMd5,
+ agent_id: record.agentId ?? "",
+ version: 1,
+ created_at_ms: createdAtMs,
+ updated_at_ms: now,
+ });
+ continue;
+ }
+
+ const currentMd5 = String(current.content_md5 ?? "");
+ const currentVersion = Number(current.version ?? 0);
+ const currentCreatedAtMs = Number(current.created_at_ms ?? 0) || now;
+
+ if (currentMd5 === record.contentMd5) {
+ continue;
+ }
+
+ if ((record.baselineVersion ?? 0) !== currentVersion) {
+ this.logger?.warn(
+ `${TAG} [profiles-sync] Conflict for ${record.filename}: remote version advanced from ${record.baselineVersion ?? 0} to ${currentVersion}, skipping sync`,
+ );
+ continue;
+ }
+
+ upserts.push({
+ id: record.id,
+ vector: [0],
+ type: record.type,
+ filename: record.filename,
+ content: record.content,
+ content_md5: record.contentMd5,
+ agent_id: record.agentId ?? "",
+ version: currentVersion + 1,
+ created_at_ms: currentCreatedAtMs,
+ updated_at_ms: now,
+ });
+ }
+
+ if (upserts.length > 0) {
+ await this.client.upsert(this.profilesCollection, upserts);
+ }
+ } catch (err) {
+ this.logger?.warn(`${TAG} [profiles-sync] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ }
+ }
+
+ async deleteProfiles(recordIds: string[]): Promise {
+ if (recordIds.length === 0) return;
+
+ try {
+ await this._ensureInit();
+ if (this.degraded) return;
+ await this.client.deleteDoc(this.profilesCollection, {
+ query: { documentIds: recordIds },
+ });
+ } catch (err) {
+ this.logger?.warn(`${TAG} [profiles-delete] FAILED: ${err instanceof Error ? err.message : String(err)}`);
+ }
+ }
+
+ // ── Re-index ─────────────────────────────────────────────
+
+ async reindexAll(
+ _embedFn: (text: string) => Promise,
+ _onProgress?: (done: number, total: number, layer: "L1" | "L0") => void,
+ ): Promise<{ l1Count: number; l0Count: number }> {
+ // TCVDB uses server-side embedding — reindex means rebuild Collection.
+ // Not implemented in Phase 2-3 (requires drop + recreate + re-upsert from JSONL).
+ this.logger?.info(`${TAG} reindexAll: TCVDB uses server-side embedding, skipping`);
+ return { l1Count: 0, l0Count: 0 };
+ }
+
+ isFtsAvailable(): boolean {
+ return !!this.bm25Encoder;
+ }
+
+ // ── Internal: parse search results ───────────────────────
+
+ private _parseL1SearchResults(docArrays: Array>>): L1SearchResult[] {
+ const results: L1SearchResult[] = [];
+ // hybridSearch/search returns [[doc, doc, ...]] (one array per query)
+ const docs = docArrays?.[0] ?? [];
+ for (const doc of docs) {
+ results.push({
+ record_id: String(doc.id ?? ""),
+ content: String(doc.text ?? ""),
+ type: String(doc.type ?? ""),
+ priority: Number(doc.priority ?? 0),
+ scene_name: String(doc.scene_name ?? ""),
+ score: Number(doc.score ?? 0),
+ timestamp_str: String(doc.timestamp_str ?? ""),
+ timestamp_start: String(doc.timestamp_start ?? ""),
+ timestamp_end: String(doc.timestamp_end ?? ""),
+ session_key: String(doc.session_key ?? ""),
+ session_id: String(doc.session_id ?? ""),
+ metadata_json: String(doc.metadata_json ?? "{}"),
+ });
+ }
+ return results;
+ }
+
+ private _parseL0SearchResults(docArrays: Array>>): L0SearchResult[] {
+ const results: L0SearchResult[] = [];
+ const docs = docArrays?.[0] ?? [];
+ for (const doc of docs) {
+ results.push({
+ record_id: String(doc.id ?? ""),
+ session_key: String(doc.session_key ?? ""),
+ session_id: String(doc.session_id ?? ""),
+ role: String(doc.role ?? ""),
+ message_text: String(doc.message_text ?? ""),
+ score: Number(doc.score ?? 0),
+ recorded_at: epochMsToIso(Number(doc.recorded_at_ms ?? 0)),
+ timestamp: Number(doc.timestamp ?? 0),
+ });
+ }
+ return results;
+ }
+}
diff --git a/src/store/types.ts b/src/store/types.ts
new file mode 100644
index 0000000..cfcb50a
--- /dev/null
+++ b/src/store/types.ts
@@ -0,0 +1,328 @@
+/**
+ * Memory Store Abstraction Layer — Core Types & Interfaces.
+ *
+ * This module defines the storage contracts that all backend implementations
+ * (SQLite local, Tencent Cloud VectorDB, etc.) must satisfy.
+ *
+ * Design principles:
+ * 1. **Backend-agnostic**: Upper-layer modules (hooks, tools, pipeline, record)
+ * depend only on these interfaces — never on concrete implementations.
+ * 2. **Capability-based**: Features like vector search, FTS, and hybrid search
+ * are expressed as capability flags so callers can gracefully degrade.
+ * 3. **Fault-tolerant**: All methods return empty results or `false` on
+ * failure rather than throwing, unless explicitly documented otherwise.
+ * 4. **Sync-first**: Matches current SQLite DatabaseSync usage. TCVDB backend
+ * adapts internally without changing these signatures.
+ */
+
+import type { MemoryRecord } from "../record/l1-writer.js";
+import type { EmbeddingProviderInfo } from "./embedding.js";
+
+// Re-export so consumers can import everything from types.ts
+export type { MemoryRecord, EmbeddingProviderInfo };
+
+// ============================
+// Common Types
+// ============================
+
+/** Minimal logger interface accepted by store implementations. */
+export interface StoreLogger {
+ debug?: (message: string) => void;
+ info: (message: string) => void;
+ warn: (message: string) => void;
+ error: (message: string) => void;
+}
+
+// ============================
+// L1 Types (Structured Memories)
+// ============================
+
+/** Result from an L1 vector similarity search. */
+export interface L1SearchResult {
+ record_id: string;
+ content: string;
+ type: string;
+ priority: number;
+ scene_name: string;
+ /** Similarity score (0–1, higher is better). */
+ score: number;
+ timestamp_str: string;
+ timestamp_start: string;
+ timestamp_end: string;
+ session_key: string;
+ session_id: string;
+ metadata_json: string;
+}
+
+/** Result from an L1 FTS keyword search. */
+export interface L1FtsResult {
+ record_id: string;
+ content: string;
+ type: string;
+ priority: number;
+ scene_name: string;
+ /** BM25-derived score (0–1, higher is better). */
+ score: number;
+ timestamp_str: string;
+ timestamp_start: string;
+ timestamp_end: string;
+ session_key: string;
+ session_id: string;
+ metadata_json: string;
+}
+
+/** Filter options for querying L1 records. */
+export interface L1QueryFilter {
+ sessionKey?: string;
+ sessionId?: string;
+ /** Only return records with updated_time strictly after this ISO 8601 UTC timestamp. */
+ updatedAfter?: string;
+}
+
+/** Row shape returned by L1 query methods. */
+export interface L1RecordRow {
+ record_id: string;
+ content: string;
+ type: string;
+ priority: number;
+ scene_name: string;
+ session_key: string;
+ session_id: string;
+ timestamp_str: string;
+ timestamp_start: string;
+ timestamp_end: string;
+ created_time: string;
+ updated_time: string;
+ metadata_json: string;
+}
+
+// ============================
+// L0 Types (Raw Conversations)
+// ============================
+
+/** An L0 conversation message record for vector indexing. */
+export interface L0Record {
+ id: string;
+ sessionKey: string;
+ sessionId: string;
+ role: string;
+ messageText: string;
+ recordedAt: string;
+ /** Original message timestamp (epoch ms). */
+ timestamp: number;
+}
+
+/** Result from an L0 vector similarity search. */
+export interface L0SearchResult {
+ record_id: string;
+ session_key: string;
+ session_id: string;
+ role: string;
+ message_text: string;
+ /** Similarity score (0–1, higher is better). */
+ score: number;
+ recorded_at: string;
+ timestamp: number;
+}
+
+/** Result from an L0 FTS keyword search. */
+export interface L0FtsResult {
+ record_id: string;
+ session_key: string;
+ session_id: string;
+ role: string;
+ message_text: string;
+ /** BM25-derived score (0–1, higher is better). */
+ score: number;
+ recorded_at: string;
+ timestamp: number;
+}
+
+/** Raw L0 row returned by query methods (used by L1 runner). */
+export interface L0QueryRow {
+ record_id: string;
+ session_key: string;
+ session_id: string;
+ role: string;
+ message_text: string;
+ recorded_at: string;
+ timestamp: number;
+}
+
+/** L0 messages grouped by session ID (for L1 runner). */
+export interface L0SessionGroup {
+ sessionId: string;
+ messages: Array<{
+ id: string;
+ role: string;
+ content: string;
+ timestamp: number;
+ /** Epoch ms when this message was recorded into L0 (used by L1 cursor). */
+ recordedAtMs: number;
+ }>;
+}
+
+// ============================
+// Store Init Result
+// ============================
+
+/** Result of store initialization. */
+export interface StoreInitResult {
+ /** Whether embeddings need to be regenerated (provider/model change). */
+ needsReindex: boolean;
+ /** Human-readable reason (for logging). */
+ reason?: string;
+}
+
+// ============================
+// Capability Flags
+// ============================
+
+/**
+ * Describes what search capabilities a store backend supports.
+ * Callers use this to select search strategies and degrade gracefully.
+ */
+export interface StoreCapabilities {
+ /** Whether vector (embedding) search is available. */
+ vectorSearch: boolean;
+ /** Whether FTS (full-text keyword) search is available. */
+ ftsSearch: boolean;
+ /** Whether native hybrid search is supported (e.g., TCVDB hybridSearch). */
+ nativeHybridSearch: boolean;
+ /** Whether the store supports sparse vectors (BM25 encoding). */
+ sparseVectors: boolean;
+}
+
+// ============================
+// L2/L3 Profile Sync Types
+// ============================
+
+/** Canonical L2/L3 profile row shared between local cache and remote store. */
+export interface ProfileRecord {
+ /** Stable ID: `profile:v1:${sha256(scope + "\0" + type + "\0" + filename)}`. */
+ id: string;
+ type: "l2" | "l3";
+ filename: string;
+ content: string;
+ contentMd5: string;
+ agentId?: string;
+ version: number;
+ createdAtMs: number;
+ updatedAtMs: number;
+}
+
+/** Profile upsert payload with optimistic-lock baseline from the last pull. */
+export interface ProfileSyncRecord extends ProfileRecord {
+ baselineVersion?: number;
+}
+
+// ============================
+// IMemoryStore — The Core Abstraction
+// ============================
+
+/**
+ * Unified memory store interface.
+ *
+ * Implementations:
+ * - `SqliteMemoryStore` (sqlite.ts) — local SQLite + sqlite-vec + FTS5
+ * - `TcvdbMemoryStore` (tcvdb.ts) — Tencent Cloud VectorDB (future)
+ *
+ * All methods are fault-tolerant: they return empty results or `false` on
+ * failure rather than throwing, unless explicitly documented otherwise.
+ */
+/**
+ * Helper type: a value that may be sync or async.
+ * Callers should always `await` the result — it's safe for both sync and async values.
+ */
+export type MaybePromise = T | Promise;
+
+export interface IMemoryStore {
+ // ── Capabilities ───────────────────────────────────────────
+
+ /**
+ * Whether this store supports deferred (background) embedding updates.
+ *
+ * When `true`, auto-capture writes metadata-only via `upsertL0(record, undefined)`
+ * and later calls `updateL0Embedding()` in a fire-and-forget background task.
+ * When `false` or absent, embedding is computed inline and passed to `upsertL0()`.
+ */
+ readonly supportsDeferredEmbedding?: boolean;
+
+ // ── Lifecycle (always sync) ──────────────────────────────
+
+ init(providerInfo?: EmbeddingProviderInfo): MaybePromise;
+ isDegraded(): boolean;
+ getCapabilities(): StoreCapabilities;
+ close(): void;
+
+ // ── L1 Write ─────────────────────────────────────────────
+
+ upsertL1(record: MemoryRecord, embedding?: Float32Array): MaybePromise;
+ deleteL1(recordId: string): MaybePromise;
+ deleteL1Batch(recordIds: string[]): MaybePromise;
+ deleteL1Expired(cutoffIso: string): MaybePromise;
+
+ // ── L1 Read ──────────────────────────────────────────────
+
+ countL1(): MaybePromise;
+ queryL1Records(filter?: L1QueryFilter): MaybePromise;
+ getAllL1Texts(): MaybePromise>;
+
+ // ── L1 Search ────────────────────────────────────────────
+
+ searchL1Vector(queryEmbedding: Float32Array, topK?: number, queryText?: string): MaybePromise;
+ searchL1Fts(ftsQuery: string, limit?: number): MaybePromise;
+ searchL1Hybrid?(params: {
+ query?: string;
+ queryEmbedding?: Float32Array;
+ sparseVector?: Array<[number, number]>;
+ topK?: number;
+ }): MaybePromise;
+
+ // ── L0 Write ─────────────────────────────────────────────
+
+ upsertL0(record: L0Record, embedding?: Float32Array): MaybePromise;
+ /** Update only the vector embedding for an existing L0 record (sqlite background path). */
+ updateL0Embedding?(recordId: string, embedding: Float32Array): MaybePromise;
+ deleteL0(recordId: string): MaybePromise;
+ deleteL0Expired(cutoffIso: string): MaybePromise;
+
+ // ── L0 Read ──────────────────────────────────────────────
+
+ countL0(): MaybePromise;
+ queryL0ForL1(sessionKey: string, afterRecordedAtMs?: number, limit?: number): MaybePromise;
+ queryL0GroupedBySessionId(sessionKey: string, afterRecordedAtMs?: number, limit?: number): MaybePromise;
+ getAllL0Texts(): MaybePromise>;
+
+ // ── L0 Search ────────────────────────────────────────────
+
+ searchL0Vector(queryEmbedding: Float32Array, topK?: number, queryText?: string): MaybePromise;
+ searchL0Fts(ftsQuery: string, limit?: number): MaybePromise;
+
+ pullProfiles?(): Promise;
+ syncProfiles?(records: ProfileSyncRecord[]): Promise;
+ deleteProfiles?(recordIds: string[]): Promise;
+
+ // ── Re-index ─────────────────────────────────────────────
+
+ reindexAll(
+ embedFn: (text: string) => Promise,
+ onProgress?: (done: number, total: number, layer: "L1" | "L0") => void,
+ ): Promise<{ l1Count: number; l0Count: number }>;
+
+ // ── FTS (always sync — cached flag) ──────────────────────
+
+ isFtsAvailable(): boolean;
+}
+
+// ============================
+// IEmbeddingService — re-exported from embedding.ts for convenience
+// ============================
+
+/**
+ * Re-export EmbeddingService as IEmbeddingService for backward compatibility.
+ * The canonical definition lives in `./embedding.ts`. All concrete implementations
+ * (LocalEmbeddingService, OpenAIEmbeddingService, NoopEmbeddingService) implement
+ * the EmbeddingService interface from embedding.ts.
+ */
+export type { EmbeddingService as IEmbeddingService } from "./embedding.js";
diff --git a/src/tools/conversation-search.ts b/src/tools/conversation-search.ts
index 8947e0e..ac4f3b1 100644
--- a/src/tools/conversation-search.ts
+++ b/src/tools/conversation-search.ts
@@ -10,8 +10,8 @@
* The tool is registered via `api.registerTool()` in index.ts.
*/
-import type { VectorStore, L0VectorSearchResult } from "../store/vector-store.js";
-import { buildFtsQuery } from "../store/vector-store.js";
+import type { IMemoryStore, L0SearchResult } from "../store/types.js";
+import { buildFtsQuery } from "../store/sqlite.js";
import type { EmbeddingService } from "../store/embedding.js";
// ============================
@@ -90,7 +90,7 @@ export async function executeConversationSearch(params: {
query: string;
limit: number;
sessionKey?: string;
- vectorStore?: VectorStore;
+ vectorStore?: IMemoryStore;
embeddingService?: EmbeddingService;
logger?: Logger;
}): Promise {
@@ -152,7 +152,7 @@ export async function executeConversationSearch(params: {
return [];
}
logger?.debug?.(`${TAG} [hybrid-fts] FTS5 query: "${ftsQuery}"`);
- const ftsResults = vectorStore.ftsSearchL0(ftsQuery, candidateK);
+ const ftsResults = await vectorStore.searchL0Fts(ftsQuery, candidateK);
logger?.debug?.(`${TAG} [hybrid-fts] FTS5 returned ${ftsResults.length} candidates`);
return ftsResults.map((r) => ({
id: r.record_id,
@@ -179,7 +179,7 @@ export async function executeConversationSearch(params: {
logger?.debug?.(
`${TAG} [hybrid-vec] Embedding OK, dims=${queryEmbedding.length}, searching top-${candidateK}...`,
);
- const vecResults: L0VectorSearchResult[] = vectorStore.searchL0(queryEmbedding, candidateK);
+ const vecResults: L0SearchResult[] = await vectorStore.searchL0Vector(queryEmbedding, candidateK, query);
logger?.debug?.(`${TAG} [hybrid-vec] Vector search returned ${vecResults.length} candidates`);
return vecResults.map((r) => ({
id: r.record_id,
diff --git a/src/tools/memory-search.ts b/src/tools/memory-search.ts
index 3725ced..dc9d2c2 100644
--- a/src/tools/memory-search.ts
+++ b/src/tools/memory-search.ts
@@ -10,8 +10,8 @@
* The tool is registered via `api.registerTool()` in index.ts.
*/
-import type { VectorStore, VectorSearchResult } from "../store/vector-store.js";
-import { buildFtsQuery } from "../store/vector-store.js";
+import type { IMemoryStore, L1SearchResult } from "../store/types.js";
+import { buildFtsQuery } from "../store/sqlite.js";
import type { EmbeddingService } from "../store/embedding.js";
// ============================
@@ -90,7 +90,7 @@ export async function executeMemorySearch(params: {
limit: number;
type?: string;
scene?: string;
- vectorStore?: VectorStore;
+ vectorStore?: IMemoryStore;
embeddingService?: EmbeddingService;
logger?: Logger;
}): Promise {
@@ -153,7 +153,7 @@ export async function executeMemorySearch(params: {
return [];
}
logger?.debug?.(`${TAG} [hybrid-fts] FTS5 query: "${ftsQuery}"`);
- const ftsResults = vectorStore.ftsSearchL1(ftsQuery, candidateK);
+ const ftsResults = await vectorStore.searchL1Fts(ftsQuery, candidateK);
logger?.debug?.(`${TAG} [hybrid-fts] FTS5 returned ${ftsResults.length} candidates`);
return ftsResults.map((r) => ({
id: r.record_id,
@@ -182,7 +182,7 @@ export async function executeMemorySearch(params: {
logger?.debug?.(
`${TAG} [hybrid-vec] Embedding OK, dims=${queryEmbedding.length}, searching top-${candidateK}...`,
);
- const vecResults: VectorSearchResult[] = vectorStore.search(queryEmbedding, candidateK);
+ const vecResults: L1SearchResult[] = await vectorStore.searchL1Vector(queryEmbedding, candidateK, query);
logger?.debug?.(`${TAG} [hybrid-vec] Vector search returned ${vecResults.length} candidates`);
return vecResults.map((r) => ({
id: r.record_id,
diff --git a/src/utils/checkpoint.ts b/src/utils/checkpoint.ts
index 4da746f..301fc0d 100644
--- a/src/utils/checkpoint.ts
+++ b/src/utils/checkpoint.ts
@@ -297,63 +297,6 @@ export class CheckpointManager {
// Public API — mutating (all serialized via file lock)
// ============================
- /**
- * Advance the captured timestamp after successful upload/recording.
- * Also updates total_processed and persona counters.
- *
- * NOTE: This advances the GLOBAL cursor (`Checkpoint.last_captured_timestamp`).
- * For per-session cursor advancement, use `advanceSessionCapturedTimestamp()`.
- * The global cursor is kept for aggregate stats / backward compat, but should
- * NOT be used as the L0 incremental-capture filter (use per-session instead).
- */
- async advanceCapturedTimestamp(maxTimestamp: number, messageCount: number): Promise {
- const cp = await this.mutate((cp) => {
- cp.last_captured_timestamp = maxTimestamp;
- cp.total_processed += messageCount;
- cp.memories_since_last_persona += messageCount;
- });
- this.logger.info(
- `[checkpoint] advanceCapturedTimestamp: -> ${maxTimestamp} (+${messageCount} msgs), ` +
- `total_processed=${cp.total_processed}, memories_since_last_persona=${cp.memories_since_last_persona}`,
- );
- }
-
- /**
- * Advance the per-session L0 capture cursor after recording messages.
- * This is the **primary** cursor for incremental L0 recording — each session
- * tracks its own progress independently, preventing cross-session cursor drift.
- *
- * Also updates the global cursor / total_processed for aggregate stats.
- */
- async advanceSessionCapturedTimestamp(
- sessionKey: string,
- maxTimestamp: number,
- messageCount: number,
- ): Promise {
- const cp = await this.mutate((cp) => {
- // Per-session cursor (runner-owned)
- const state = this.getRunnerState(cp, sessionKey);
- state.last_captured_timestamp = maxTimestamp;
- // Global stats (aggregate only — not used for filtering)
- cp.last_captured_timestamp = Math.max(cp.last_captured_timestamp, maxTimestamp);
- cp.total_processed += messageCount;
- cp.memories_since_last_persona += messageCount;
- });
- this.logger.info(
- `[checkpoint] advanceSessionCapturedTimestamp session=${sessionKey}: -> ${maxTimestamp} ` +
- `(+${messageCount} msgs), total_processed=${cp.total_processed}`,
- );
- }
-
- /**
- * Increment L0 conversation count.
- */
- async incrementL0ConversationCount(): Promise {
- await this.mutate((cp) => {
- cp.l0_conversations_count += 1;
- });
- }
-
// ============================
// Persona methods (L3)
// ============================
@@ -460,17 +403,20 @@ export class CheckpointManager {
/**
* Mark L1 extraction completed: reset sinceL1 counter, advance L1 cursor,
* and optionally save the last scene name for cross-batch continuity.
+ *
+ * @param cursorRecordedAtMs - The max recorded_at epoch ms of processed L0 messages.
+ * This becomes the new `last_l1_cursor` value (recorded_at semantics, not conversation timestamp).
*/
async markL1ExtractionComplete(
sessionKey: string,
memoriesExtracted: number,
- cursorTimestamp?: number,
+ cursorRecordedAtMs?: number,
lastSceneName?: string,
): Promise {
await this.mutate((cp) => {
const state = this.getRunnerState(cp, sessionKey);
- if (cursorTimestamp) {
- state.last_l1_cursor = cursorTimestamp;
+ if (cursorRecordedAtMs) {
+ state.last_l1_cursor = cursorRecordedAtMs;
}
if (lastSceneName !== undefined) {
state.last_scene_name = lastSceneName;
@@ -480,7 +426,7 @@ export class CheckpointManager {
});
this.logger.info(
`[checkpoint] markL1ExtractionComplete session=${sessionKey}: ` +
- `extracted=${memoriesExtracted}, cursor=${cursorTimestamp ?? "(unchanged)"}, ` +
+ `extracted=${memoriesExtracted}, cursor=${cursorRecordedAtMs ?? "(unchanged)"}, ` +
`lastScene="${lastSceneName ?? "(unchanged)"}"`,
);
}
@@ -533,7 +479,6 @@ export class CheckpointManager {
// Global stats (aggregate only — not used for filtering)
cp.last_captured_timestamp = Math.max(cp.last_captured_timestamp, result.maxTimestamp);
cp.total_processed += result.messageCount;
- cp.memories_since_last_persona += result.messageCount;
// Increment L0 conversation count (was a separate mutate() call before)
cp.l0_conversations_count += 1;
}
diff --git a/src/utils/clean-context-runner.ts b/src/utils/clean-context-runner.ts
index e4008d1..64f490d 100644
--- a/src/utils/clean-context-runner.ts
+++ b/src/utils/clean-context-runner.ts
@@ -14,6 +14,7 @@ import fsSync from "node:fs";
import path from "node:path";
import os from "node:os";
import { fileURLToPath, pathToFileURL } from "node:url";
+import type { OpenClawPluginApi } from "openclaw/plugin-sdk/core";
import { report } from "../report/reporter.js";
/**
@@ -55,7 +56,42 @@ interface RunnerLogger {
}
// Dynamic import type — runEmbeddedPiAgent is an internal API
-type RunEmbeddedPiAgentFn = (params: Record) => Promise;
+// Prefer the public plugin runtime signature so host-injected runtimes stay assignable.
+type RunEmbeddedPiAgentFn = OpenClawPluginApi["runtime"]["agent"]["runEmbeddedPiAgent"];
+
+export interface EmbeddedAgentRuntimeLike {
+ runEmbeddedPiAgent?: RunEmbeddedPiAgentFn;
+}
+
+let _preferredAgentRuntime: EmbeddedAgentRuntimeLike | undefined;
+
+export function setPreferredEmbeddedAgentRuntime(
+ agentRuntime: EmbeddedAgentRuntimeLike | undefined,
+): void {
+ _preferredAgentRuntime = agentRuntime;
+}
+
+function resolveInjectedRunEmbeddedPiAgent(
+ agentRuntime?: EmbeddedAgentRuntimeLike,
+): RunEmbeddedPiAgentFn | undefined {
+ const candidate =
+ agentRuntime?.runEmbeddedPiAgent ?? _preferredAgentRuntime?.runEmbeddedPiAgent;
+ return typeof candidate === "function" ? candidate : undefined;
+}
+
+async function resolveRunEmbeddedPiAgent(
+ agentRuntime: EmbeddedAgentRuntimeLike | undefined,
+ logger?: RunnerLogger,
+): Promise {
+ const injected = resolveInjectedRunEmbeddedPiAgent(agentRuntime);
+ if (injected) {
+ logger?.debug?.(
+ `${TAG} resolveRunEmbeddedPiAgent: using injected runtime.agent.runEmbeddedPiAgent`,
+ );
+ return injected;
+ }
+ return loadRunEmbeddedPiAgent(logger);
+}
// ── Core import (mirrors voice-call/core-bridge.ts — dist/ only, no jiti) ──
@@ -123,7 +159,17 @@ function loadRunEmbeddedPiAgent(logger?: RunnerLogger): Promise {
logger?.warn(`${TAG} prewarmEmbeddedAgent: failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`);
});
@@ -232,6 +278,8 @@ export interface CleanContextRunnerOptions {
* automatically falls back to the main config's `agents.defaults.model`.
*/
modelRef?: string;
+ /** Preferred runtime seam. When absent, falls back to the legacy dist bridge. */
+ agentRuntime?: EmbeddedAgentRuntimeLike;
/** Allow the LLM to use tools (read_file, write_to_file, etc). Default: false */
enableTools?: boolean;
/** Logger instance for detailed tracing */
@@ -318,11 +366,14 @@ export class CleanContextRunner {
try {
const sessionFile = path.join(tmpDir, "session.json");
- // Phase 1: Load runEmbeddedPiAgent (fast if dist/ exists or already cached)
+ // Phase 1: Resolve runEmbeddedPiAgent (prefer runtime, fallback to legacy dist bridge)
const importStartMs = Date.now();
- const runEmbeddedPiAgent = await loadRunEmbeddedPiAgent(this.logger);
+ const runEmbeddedPiAgent = await resolveRunEmbeddedPiAgent(
+ this.options.agentRuntime,
+ this.logger,
+ );
const importElapsedMs = Date.now() - importStartMs;
- this.logger?.debug?.(`${TAG} run() dynamic import phase: ${importElapsedMs}ms`);
+ this.logger?.debug?.(`${TAG} run() runner resolution phase: ${importElapsedMs}ms`);
// Derive a config with plugins disabled to prevent loadOpenClawPlugins
// from re-registering plugins when the workspaceDir differs from the
@@ -347,10 +398,10 @@ export class CleanContextRunner {
},
};
- // Build the effective prompt:
- // If systemPrompt is provided, pass it as a separate parameter to the agent
- // and use `prompt` as the user message. Fallback: prepend to prompt if the
- // embedded agent doesn't support systemPrompt natively.
+ // Build the effective prompt.
+ // Keep prepending the optional systemPrompt into the user-visible prompt so
+ // runtime and legacy fallback paths preserve the same behavior without
+ // relying on a newer native extraSystemPrompt contract.
const effectivePrompt = params.systemPrompt
? `${params.systemPrompt}\n\n---\n\n${params.prompt}`
: params.prompt;
@@ -368,7 +419,6 @@ export class CleanContextRunner {
workspaceDir: cleanWorkspace,
config: cleanConfig,
prompt: effectivePrompt,
- systemPrompt: params.systemPrompt,
timeoutMs: params.timeoutMs ?? 120_000,
runId,
provider: this.resolvedProvider,
diff --git a/src/utils/manifest.ts b/src/utils/manifest.ts
new file mode 100644
index 0000000..20b8e36
--- /dev/null
+++ b/src/utils/manifest.ts
@@ -0,0 +1,159 @@
+/**
+ * Manifest — self-describing metadata for a memory-tdai data directory.
+ *
+ * Lives at `/.metadata/manifest.json`.
+ *
+ * - **store**: written once on first successful store init; never overwritten.
+ * On subsequent starts the current config is compared against the persisted
+ * store binding — mismatches are logged as warnings.
+ * - **seed**: written once when a seed run completes; null for live-runtime dirs.
+ *
+ * This file is informational / read-only from the user's perspective.
+ * The plugin reads it on startup for consistency checks.
+ */
+
+import fs from "node:fs";
+import path from "node:path";
+
+// ============================
+// Types
+// ============================
+
+export interface ManifestStoreInfo {
+ type: "sqlite" | "tcvdb";
+ sqlite?: {
+ /** Relative path to the SQLite DB file (relative to dataDir). */
+ path: string;
+ };
+ tcvdb?: {
+ url: string;
+ database: string;
+ /** User-friendly alias (optional). */
+ alias?: string;
+ };
+}
+
+export interface ManifestSeedInfo {
+ /** Original input file name (basename only). */
+ inputFile?: string;
+ sessions: number;
+ rounds: number;
+ messages: number;
+ startedAt: string;
+ completedAt: string;
+}
+
+export interface Manifest {
+ /** Schema version for future migrations. */
+ version: 1;
+ /** Timestamp when the manifest was first created. */
+ createdAt: string;
+ /** Store binding — written once on first init. */
+ store: ManifestStoreInfo;
+ /** Seed run info — null for live-runtime directories. */
+ seed: ManifestSeedInfo | null;
+}
+
+// ============================
+// Paths
+// ============================
+
+const METADATA_DIR = ".metadata";
+const MANIFEST_FILE = "manifest.json";
+
+export function manifestPath(dataDir: string): string {
+ return path.join(dataDir, METADATA_DIR, MANIFEST_FILE);
+}
+
+// ============================
+// Read / Write
+// ============================
+
+/**
+ * Read an existing manifest from disk. Returns `null` if not found or unparseable.
+ */
+export function readManifest(dataDir: string): Manifest | null {
+ const p = manifestPath(dataDir);
+ try {
+ if (!fs.existsSync(p)) return null;
+ const raw = fs.readFileSync(p, "utf-8");
+ return JSON.parse(raw) as Manifest;
+ } catch {
+ return null;
+ }
+}
+
+/**
+ * Write a manifest to disk (creates `.metadata/` if needed).
+ */
+export function writeManifest(dataDir: string, manifest: Manifest): void {
+ const dir = path.join(dataDir, METADATA_DIR);
+ fs.mkdirSync(dir, { recursive: true });
+ fs.writeFileSync(
+ manifestPath(dataDir),
+ JSON.stringify(manifest, null, 2) + "\n",
+ "utf-8",
+ );
+}
+
+// ============================
+// Store binding helpers
+// ============================
+
+export interface StoreConfigSnapshot {
+ type: "sqlite" | "tcvdb";
+ sqlitePath?: string;
+ tcvdbUrl?: string;
+ tcvdbDatabase?: string;
+ tcvdbAlias?: string;
+}
+
+/**
+ * Build a ManifestStoreInfo from the current store config snapshot.
+ */
+export function buildStoreInfo(snapshot: StoreConfigSnapshot): ManifestStoreInfo {
+ const info: ManifestStoreInfo = { type: snapshot.type };
+ if (snapshot.type === "sqlite") {
+ info.sqlite = { path: snapshot.sqlitePath ?? "vectors.db" };
+ } else {
+ info.tcvdb = {
+ url: snapshot.tcvdbUrl!,
+ database: snapshot.tcvdbDatabase!,
+ alias: snapshot.tcvdbAlias || undefined,
+ };
+ }
+ return info;
+}
+
+/**
+ * Compare the persisted store binding against the current config.
+ * Returns a list of human-readable mismatch descriptions (empty = all good).
+ */
+export function diffStoreBinding(
+ persisted: ManifestStoreInfo,
+ current: ManifestStoreInfo,
+): string[] {
+ const diffs: string[] = [];
+
+ if (persisted.type !== current.type) {
+ diffs.push(`store type changed: ${persisted.type} → ${current.type}`);
+ return diffs; // no point comparing fields across different types
+ }
+
+ if (persisted.type === "sqlite" && current.type === "sqlite") {
+ if (persisted.sqlite?.path !== current.sqlite?.path) {
+ diffs.push(`sqlite path changed: ${persisted.sqlite?.path} → ${current.sqlite?.path}`);
+ }
+ }
+
+ if (persisted.type === "tcvdb" && current.type === "tcvdb") {
+ if (persisted.tcvdb?.url !== current.tcvdb?.url) {
+ diffs.push(`tcvdb url changed: ${persisted.tcvdb?.url} → ${current.tcvdb?.url}`);
+ }
+ if (persisted.tcvdb?.database !== current.tcvdb?.database) {
+ diffs.push(`tcvdb database changed: ${persisted.tcvdb?.database} → ${current.tcvdb?.database}`);
+ }
+ }
+
+ return diffs;
+}
diff --git a/src/utils/memory-cleaner.ts b/src/utils/memory-cleaner.ts
index 24ccf8e..ebb9008 100644
--- a/src/utils/memory-cleaner.ts
+++ b/src/utils/memory-cleaner.ts
@@ -1,7 +1,7 @@
import fs from "node:fs/promises";
import path from "node:path";
-import type { VectorStore } from "../store/vector-store.js";
+import type { IMemoryStore } from "../store/types.js";
import { ManagedTimer } from "./managed-timer.js";
interface Logger {
@@ -16,7 +16,7 @@ export interface MemoryCleanerOptions {
retentionDays: number;
cleanTime: string;
logger?: Logger;
- vectorStore?: VectorStore;
+ vectorStore?: IMemoryStore;
}
interface CleanupStats {
@@ -33,14 +33,14 @@ const L1_DIR_NAME = "records";
export class LocalMemoryCleaner {
private readonly timer: ManagedTimer;
private destroyed = false;
- private vectorStore?: VectorStore;
+ private vectorStore?: IMemoryStore;
constructor(private readonly opts: MemoryCleanerOptions) {
this.timer = new ManagedTimer("memory-tdai-cleaner", () => this.destroyed);
this.vectorStore = opts.vectorStore;
}
- setVectorStore(vectorStore: VectorStore | undefined): void {
+ setVectorStore(vectorStore: IMemoryStore | undefined): void {
this.vectorStore = vectorStore;
}
@@ -51,10 +51,10 @@ export class LocalMemoryCleaner {
const tz = Intl.DateTimeFormat().resolvedOptions().timeZone || "unknown";
const utcOffset = formatUtcOffset(-now.getTimezoneOffset());
- this.opts.logger?.info(
+ this.opts.logger?.debug?.(
`${TAG} Enabled: retentionDays=${this.opts.retentionDays}, cleanTime=${this.opts.cleanTime}, dirs=[${L0_DIR_NAME}, ${L1_DIR_NAME}]`,
);
- this.opts.logger?.info(
+ this.opts.logger?.debug?.(
`${TAG} Runtime clock: nowLocal=${formatLocalDateTime(now)}, nowIso=${now.toISOString()}, tz=${tz}, utcOffset=${utcOffset}`,
);
@@ -110,7 +110,7 @@ export class LocalMemoryCleaner {
let failedL1DbCleanup = 0;
try {
- removedL0 = vectorStore.deleteL0ExpiredByRecordedAt(cutoffIso);
+ removedL0 = await vectorStore.deleteL0Expired(cutoffIso);
} catch (err) {
failedL0DbCleanup = 1;
this.opts.logger?.warn(
@@ -119,7 +119,7 @@ export class LocalMemoryCleaner {
}
try {
- removedL1 = vectorStore.deleteL1ExpiredByUpdatedTime(cutoffIso);
+ removedL1 = await vectorStore.deleteL1Expired(cutoffIso);
} catch (err) {
failedL1DbCleanup = 1;
this.opts.logger?.warn(
@@ -150,7 +150,7 @@ export class LocalMemoryCleaner {
const passedToday = targetToday <= nowMs;
const delayMs = Math.max(0, next - nowMs);
- this.opts.logger?.info(
+ this.opts.logger?.debug?.(
`${TAG} Schedule next run: nowLocal=${formatLocalDateTime(now)}, cleanTime=${this.opts.cleanTime}, targetTodayLocal=${formatLocalDateTime(new Date(targetToday))}, passedToday=${passedToday}, nextRunLocal=${formatLocalDateTime(new Date(next))}, nextRunIso=${new Date(next).toISOString()}, delayMs=${delayMs}`,
);
diff --git a/src/utils/pipeline-factory.ts b/src/utils/pipeline-factory.ts
new file mode 100644
index 0000000..ff5365b
--- /dev/null
+++ b/src/utils/pipeline-factory.ts
@@ -0,0 +1,720 @@
+/**
+ * Pipeline factory: shared infrastructure for creating and wiring
+ * MemoryPipelineManager instances with VectorStore, EmbeddingService,
+ * L1 runner, L2 runner, L3 runner, and persister.
+ *
+ * Used by both:
+ * - `index.ts` (live plugin runtime)
+ * - `seed-runtime.ts` (standalone seed CLI command)
+ *
+ * This avoids duplicating VectorStore init, L1/L2/L3 extraction logic,
+ * persister wiring, and destroy sequences across multiple callers.
+ */
+
+import fs from "node:fs";
+import path from "node:path";
+import type { MemoryTdaiConfig } from "../config.js";
+import { MemoryPipelineManager } from "./pipeline-manager.js";
+import type { L2Runner, L3Runner } from "./pipeline-manager.js";
+import { SessionFilter } from "./session-filter.js";
+import { extractL1Memories } from "../record/l1-extractor.js";
+import { readConversationMessagesGroupedBySessionId } from "../conversation/l0-recorder.js";
+import type { ConversationMessage } from "../conversation/l0-recorder.js";
+import { CheckpointManager } from "./checkpoint.js";
+import type { PipelineSessionState } from "./checkpoint.js";
+import { createStoreBundle } from "../store/factory.js";
+import type { IMemoryStore } from "../store/types.js";
+import type { EmbeddingService } from "../store/embedding.js";
+import {
+ readManifest,
+ writeManifest,
+ buildStoreInfo,
+ diffStoreBinding,
+ type Manifest,
+} from "./manifest.js";
+import { SceneExtractor } from "../scene/scene-extractor.js";
+import { PersonaTrigger } from "../persona/persona-trigger.js";
+import { PersonaGenerator } from "../persona/persona-generator.js";
+import { pullProfilesToLocal, syncLocalProfilesToStore } from "../profile/profile-sync.js";
+
+const TAG = "[memory-tdai] [pipeline-factory]";
+
+function supportsProfileSyncWrite(store?: IMemoryStore): boolean {
+ return !!(store?.syncProfiles || store?.deleteProfiles);
+}
+
+// ============================
+// Logger interface
+// ============================
+
+export interface PipelineLogger {
+ debug?: (message: string) => void;
+ info: (message: string) => void;
+ warn: (message: string) => void;
+ error: (message: string) => void;
+}
+
+// ============================
+// Factory options
+// ============================
+
+export interface PipelineFactoryOptions {
+ /** Plugin data directory (L0, records, scene_blocks, vectors.db, etc.). */
+ pluginDataDir: string;
+ /** Parsed memory-tdai config. */
+ cfg: MemoryTdaiConfig;
+ /** OpenClaw config object (needed for LLM calls in L1). */
+ openclawConfig: unknown;
+ /** Logger instance. */
+ logger: PipelineLogger;
+ /** Session filter (optional, defaults to empty). */
+ sessionFilter?: SessionFilter;
+}
+
+// ============================
+// Factory result
+// ============================
+
+export interface PipelineInstance {
+ /** The pipeline scheduler. */
+ scheduler: MemoryPipelineManager;
+ /** VectorStore (undefined if init failed or degraded). */
+ vectorStore: IMemoryStore | undefined;
+ /** EmbeddingService (undefined if not configured or init failed). */
+ embeddingService: EmbeddingService | undefined;
+ /**
+ * Destroy all resources (scheduler, VectorStore, EmbeddingService).
+ * Call this on shutdown / cleanup.
+ */
+ destroy: () => Promise;
+}
+
+// ============================
+// Data directory init
+// ============================
+
+/**
+ * Ensure all required data subdirectories exist under `pluginDataDir`.
+ * Safe to call multiple times (mkdirSync with `recursive: true`).
+ */
+export function initDataDirectories(dataDir: string): void {
+ const dirs = ["conversations", "records", "scene_blocks", ".metadata", ".backup"];
+ for (const sub of dirs) {
+ fs.mkdirSync(path.join(dataDir, sub), { recursive: true });
+ }
+}
+
+// ============================
+// Store init (once-async singleton)
+// ============================
+
+export interface StoreInitResult {
+ vectorStore: IMemoryStore | undefined;
+ embeddingService: EmbeddingService | undefined;
+ /** Whether a background re-index is needed (embedding config changed). */
+ needsReindex: boolean;
+ reindexReason?: string;
+}
+
+/**
+ * Cached store init promises — keyed by `pluginDataDir` so that different
+ * data directories (e.g. live runtime vs. seed output) each get their own
+ * store instance, while concurrent callers for the *same* directory share
+ * one initialization.
+ */
+const _storeInitCache = new Map>();
+
+/**
+ * Initialize store backend and (optionally) EmbeddingService.
+ *
+ * **Once-async semantics per dataDir**: the first call for a given
+ * `pluginDataDir` creates the store and caches the result; subsequent
+ * calls with the same dir return the cached Promise immediately.
+ * Call `resetStores()` during shutdown to clear the cache.
+ *
+ * Supports both SQLite (sync init) and TCVDB (async init) backends.
+ */
+export function initStores(
+ cfg: MemoryTdaiConfig,
+ pluginDataDir: string,
+ logger: PipelineLogger,
+): Promise {
+ const key = pluginDataDir;
+ if (!_storeInitCache.has(key)) {
+ _storeInitCache.set(key, _doInitStores(cfg, pluginDataDir, logger));
+ }
+ return _storeInitCache.get(key)!;
+}
+
+/**
+ * Reset the cached store singleton(s).
+ *
+ * Call this during `gateway_stop` (after closing the actual store/embedding
+ * resources) so that a subsequent `register()` on hot-restart can
+ * re-initialize fresh instances.
+ *
+ * @param pluginDataDir If provided, only clear the cache for that dir.
+ * If omitted, clear all cached stores.
+ */
+export function resetStores(pluginDataDir?: string): void {
+ if (pluginDataDir) {
+ _storeInitCache.delete(pluginDataDir);
+ } else {
+ _storeInitCache.clear();
+ }
+}
+
+/**
+ * Internal: actual store initialization logic (called once by the cache).
+ */
+async function _doInitStores(
+ cfg: MemoryTdaiConfig,
+ pluginDataDir: string,
+ logger: PipelineLogger,
+): Promise {
+ let vectorStore: IMemoryStore | undefined;
+ let embeddingService: EmbeddingService | undefined;
+ let needsReindex = false;
+ let reindexReason: string | undefined;
+
+ try {
+ const bundle = createStoreBundle(cfg, {
+ dataDir: pluginDataDir,
+ logger,
+ });
+ vectorStore = bundle.store;
+ embeddingService = bundle.embedding ?? undefined;
+
+ const providerInfo = embeddingService?.getProviderInfo();
+ const initResult = await vectorStore.init(providerInfo);
+
+ if (vectorStore.isDegraded()) {
+ logger.warn(`${TAG} Store is in degraded mode, falling back to keyword dedup`);
+ vectorStore = undefined;
+ embeddingService = undefined;
+ } else {
+ logger.debug?.(
+ `${TAG} Store initialized: backend=${cfg.storeBackend}, provider=${cfg.embedding.provider}`,
+ );
+ needsReindex = initResult.needsReindex;
+ reindexReason = initResult.reason;
+
+ // ── Manifest: first-write + config-drift detection ──
+ try {
+ const currentStoreInfo = buildStoreInfo(bundle.storeSnapshot);
+ const existing = readManifest(pluginDataDir);
+
+ if (!existing) {
+ // First init — write manifest
+ const manifest: Manifest = {
+ version: 1,
+ createdAt: new Date().toISOString(),
+ store: currentStoreInfo,
+ seed: null,
+ };
+ writeManifest(pluginDataDir, manifest);
+ logger.debug?.(`${TAG} Manifest created: ${JSON.stringify(currentStoreInfo)}`);
+ } else {
+ // Compare persisted store binding against current config
+ const diffs = diffStoreBinding(existing.store, currentStoreInfo);
+ if (diffs.length > 0) {
+ logger.warn(
+ `${TAG} ⚠️ Store config has changed since this data directory was created! ` +
+ `Diffs: ${diffs.join("; ")}. ` +
+ `Local JSONL data may not match the current store. ` +
+ `Consider re-seeding or migrating data.`,
+ );
+ }
+ }
+ } catch (err) {
+ logger.warn(`${TAG} Failed to read/write manifest (non-fatal): ${err instanceof Error ? err.message : String(err)}`);
+ }
+ }
+ } catch (err) {
+ logger.warn(
+ `${TAG} Store init failed; vector/FTS recall and dedup conflict detection will be unavailable: ${err instanceof Error ? err.message : String(err)}`,
+ );
+ vectorStore = undefined;
+ embeddingService = undefined;
+ }
+
+ return { vectorStore, embeddingService, needsReindex, reindexReason };
+}
+
+// ============================
+// L1 Runner factory
+// ============================
+
+/**
+ * Create the standard L1 runner function.
+ *
+ * Reads L0 messages (from VectorStore DB or JSONL fallback), groups by sessionId,
+ * runs extractL1Memories for each group, and updates the checkpoint cursor.
+ */
+export function createL1Runner(opts: {
+ pluginDataDir: string;
+ cfg: MemoryTdaiConfig;
+ openclawConfig: unknown;
+ vectorStore: IMemoryStore | undefined;
+ embeddingService: EmbeddingService | undefined;
+ logger: PipelineLogger;
+ /**
+ * Getter for the plugin instance ID used for metric reporting.
+ * Called at runner execution time (not at creation time) so that the ID is
+ * available even when the runner is wired before instanceId is resolved.
+ * Metrics are skipped when the getter returns undefined.
+ */
+ getInstanceId?: () => string | undefined;
+}): (params: { sessionKey: string }) => Promise<{ processedCount: number }> {
+ const { pluginDataDir, cfg, openclawConfig, vectorStore, embeddingService, logger, getInstanceId } = opts;
+ const config = openclawConfig as Record | undefined;
+
+ return async ({ sessionKey }) => {
+ if (!config) {
+ logger.debug?.(`${TAG} [l1] No OpenClaw config, skipping L1 extraction`);
+ return { processedCount: 0 };
+ }
+
+ const checkpoint = new CheckpointManager(pluginDataDir, logger);
+ const cp = await checkpoint.read();
+ const runnerState = checkpoint.getRunnerState(cp, sessionKey);
+
+ logger.info(
+ `${TAG} [l1] Session ${sessionKey}: l1_cursor=${runnerState.last_l1_cursor || "(start)"}`,
+ );
+
+ try {
+ let groups: Array<{ sessionId: string; messages: ConversationMessage[] }>;
+ let maxRecordedAtMs = 0;
+
+ if (vectorStore && !vectorStore.isDegraded()) {
+ const l1Cursor = runnerState.last_l1_cursor > 0
+ ? runnerState.last_l1_cursor
+ : undefined;
+ const dbGroups = await vectorStore.queryL0GroupedBySessionId(sessionKey, l1Cursor);
+ groups = dbGroups.map((g) => ({
+ sessionId: g.sessionId,
+ messages: g.messages.map((m) => ({
+ id: m.id,
+ role: m.role as "user" | "assistant",
+ content: m.content,
+ timestamp: m.timestamp,
+ })),
+ }));
+ // Compute max recordedAtMs across all groups for cursor advancement
+ for (const g of dbGroups) {
+ for (const m of g.messages) {
+ if (m.recordedAtMs > maxRecordedAtMs) maxRecordedAtMs = m.recordedAtMs;
+ }
+ }
+ logger.debug?.(`${TAG} [l1] L0 data source: VectorStore DB`);
+ } else {
+ logger.debug?.(`${TAG} [l1] L0 data source: JSONL files (VectorStore unavailable)`);
+ const jsonlGroups = await readConversationMessagesGroupedBySessionId(
+ sessionKey,
+ pluginDataDir,
+ runnerState.last_l1_cursor || undefined,
+ logger,
+ 50,
+ );
+ groups = jsonlGroups.map((g) => ({
+ sessionId: g.sessionId,
+ messages: g.messages,
+ }));
+ // Compute max recordedAtMs from JSONL groups
+ for (const g of jsonlGroups) {
+ for (const m of g.messages) {
+ if (m.recordedAtMs > maxRecordedAtMs) maxRecordedAtMs = m.recordedAtMs;
+ }
+ }
+ }
+
+ if (groups.length === 0) {
+ logger.debug?.(`${TAG} [l1] No new L0 messages for session ${sessionKey}`);
+ return { processedCount: 0 };
+ }
+
+ const totalMessages = groups.reduce((sum, g) => sum + g.messages.length, 0);
+ logger.info(
+ `${TAG} [l1] Processing ${totalMessages} L0 messages across ${groups.length} sessionId group(s) for session ${sessionKey}`,
+ );
+
+ let totalExtracted = 0;
+ let totalStored = 0;
+ let lastSceneName: string | undefined;
+
+ for (const group of groups) {
+ logger.debug?.(
+ `${TAG} [l1] Group sessionId=${group.sessionId || "(empty)"}: ${group.messages.length} messages`,
+ );
+
+ const l1Result = await extractL1Memories({
+ messages: group.messages,
+ sessionKey,
+ sessionId: group.sessionId,
+ baseDir: pluginDataDir,
+ config,
+ options: {
+ enableDedup: cfg.extraction.enableDedup,
+ maxMemoriesPerSession: cfg.extraction.maxMemoriesPerSession,
+ model: cfg.extraction.model,
+ previousSceneName: lastSceneName ?? (runnerState.last_scene_name || undefined),
+ vectorStore,
+ embeddingService,
+ conflictRecallTopK: cfg.embedding.conflictRecallTopK,
+ },
+ logger,
+ instanceId: getInstanceId?.(),
+ });
+
+ totalExtracted += l1Result.extractedCount;
+ totalStored += l1Result.storedCount;
+ if (l1Result.lastSceneName) {
+ lastSceneName = l1Result.lastSceneName;
+ }
+ }
+
+ // Use maxRecordedAtMs (write time) as cursor — always positive, TCVDB-safe
+ await checkpoint.markL1ExtractionComplete(sessionKey, totalStored, maxRecordedAtMs || undefined, lastSceneName);
+ logger.info(
+ `${TAG} [l1] L1 complete: extracted=${totalExtracted}, stored=${totalStored} (${groups.length} group(s))`,
+ );
+
+ return { processedCount: totalMessages };
+ } catch (err) {
+ logger.error(`${TAG} [l1] L1 failed: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
+ throw err;
+ }
+ };
+}
+
+// ============================
+// Persister factory
+// ============================
+
+/**
+ * Create the standard pipeline state persister.
+ * Saves pipeline session states to the checkpoint file.
+ */
+export function createPersister(
+ pluginDataDir: string,
+ logger: PipelineLogger,
+): (states: Record) => Promise {
+ return async (states) => {
+ const checkpoint = new CheckpointManager(pluginDataDir, logger);
+ await checkpoint.mergePipelineStates(states);
+ };
+}
+
+// ============================
+// L2 Runner factory
+// ============================
+
+/**
+ * Create the standard L2 runner function (scene extraction).
+ *
+ * Reads L1 memory records (incremental via VectorStore or JSONL fallback),
+ * runs SceneExtractor, and returns the latest cursor for pipeline-manager
+ * to track incremental progress.
+ *
+ * Used by both `index.ts` (live runtime) and `seed-runtime.ts` (seed CLI).
+ */
+export function createL2Runner(opts: {
+ pluginDataDir: string;
+ cfg: MemoryTdaiConfig;
+ openclawConfig: unknown;
+ vectorStore: IMemoryStore | undefined;
+ logger: PipelineLogger;
+ instanceId?: string;
+}): L2Runner {
+ const { pluginDataDir, cfg, openclawConfig, vectorStore, logger, instanceId } = opts;
+ let profileBaseline = new Map();
+
+ return async (sessionKey: string, cursor?: string) => {
+ logger.debug?.(
+ `${TAG} [L2] session=${sessionKey}, updatedAfter=${cursor ?? "(full)"}`,
+ );
+
+ let records: Array<{ content: string; created_at: string; id: string; updatedAt: string }>;
+
+ if (vectorStore?.pullProfiles && !vectorStore.isDegraded()) {
+ profileBaseline = await pullProfilesToLocal(pluginDataDir, vectorStore, logger);
+ }
+
+ if (vectorStore && !vectorStore.isDegraded()) {
+ const { queryMemoryRecords } = await import("../record/l1-reader.js");
+ const memRecords = await queryMemoryRecords(vectorStore, {
+ sessionKey,
+ updatedAfter: cursor,
+ }, logger);
+
+ if (memRecords.length === 0) {
+ logger.debug?.(
+ `${TAG} [L2] No new L1 records since cursor (session=${sessionKey}, updatedAfter=${cursor ?? "(full)"}), skipping scene extraction`,
+ );
+ return;
+ }
+
+ logger.debug?.(
+ `${TAG} [L2] Incremental query returned ${memRecords.length} record(s) (session=${sessionKey})`,
+ );
+
+ records = memRecords.map((r) => ({
+ content: r.content,
+ created_at: r.createdAt,
+ id: r.id,
+ updatedAt: r.updatedAt,
+ }));
+ } else {
+ logger.debug?.(`${TAG} [L2] VectorStore unavailable, falling back to JSONL read (session=${sessionKey})`);
+ const { readMemoryRecords } = await import("../record/l1-reader.js");
+ let sessionRecords = await readMemoryRecords(sessionKey, pluginDataDir, logger);
+
+ if (cursor) {
+ const beforeCount = sessionRecords.length;
+ sessionRecords = sessionRecords.filter((r) => {
+ const t = r.updatedAt || r.createdAt || "";
+ return t > cursor;
+ });
+ logger.debug?.(
+ `${TAG} [L2] JSONL time filter: ${beforeCount} → ${sessionRecords.length} record(s) (updatedAfter=${cursor})`,
+ );
+ }
+
+ if (sessionRecords.length === 0) {
+ logger.debug?.(`${TAG} [L2] No new L1 records found (JSONL fallback, session=${sessionKey}), skipping scene extraction`);
+ return;
+ }
+
+ records = sessionRecords.map((r) => ({
+ content: r.content,
+ created_at: r.createdAt,
+ id: r.id,
+ updatedAt: r.updatedAt,
+ }));
+ }
+
+ const extractor = new SceneExtractor({
+ dataDir: pluginDataDir,
+ config: openclawConfig!,
+ model: cfg.persona.model,
+ maxScenes: cfg.persona.maxScenes,
+ sceneBackupCount: cfg.persona.sceneBackupCount,
+ logger,
+ instanceId,
+ });
+
+ const memories = records.map((r) => ({
+ content: r.content,
+ created_at: r.created_at,
+ id: r.id,
+ }));
+
+ const preCheckpoint = new CheckpointManager(pluginDataDir, logger);
+ const preState = await preCheckpoint.read();
+ const preScenesProcessed = preState.scenes_processed;
+ const preMemoriesSince = preState.memories_since_last_persona;
+ const preTotalProcessed = preState.total_processed;
+
+ const extractResult = await extractor.extract(memories);
+ if (extractResult.success && extractResult.memoriesProcessed > 0) {
+ const checkpoint = new CheckpointManager(pluginDataDir, logger);
+ const postState = await checkpoint.read();
+ if (
+ postState.scenes_processed < preScenesProcessed ||
+ postState.total_processed < preTotalProcessed
+ ) {
+ logger.warn(
+ `${TAG} [L2] ⚠️ Checkpoint corruption detected! ` +
+ `scenes_processed: ${preScenesProcessed} → ${postState.scenes_processed}, ` +
+ `total_processed: ${preTotalProcessed} → ${postState.total_processed}, ` +
+ `memories_since: ${preMemoriesSince} → ${postState.memories_since_last_persona}. ` +
+ `Repairing...`,
+ );
+ await checkpoint.write({
+ ...postState,
+ scenes_processed: Math.max(postState.scenes_processed, preScenesProcessed),
+ total_processed: Math.max(postState.total_processed, preTotalProcessed),
+ memories_since_last_persona: Math.max(postState.memories_since_last_persona, preMemoriesSince),
+ });
+ logger.info(`${TAG} [L2] Checkpoint repaired`);
+ }
+
+ if (vectorStore && supportsProfileSyncWrite(vectorStore)) {
+ await syncLocalProfilesToStore(pluginDataDir, vectorStore, profileBaseline, logger);
+ }
+ await checkpoint.incrementScenesProcessed();
+
+ const latestCursor = records.reduce((latest, r) => {
+ return r.updatedAt > latest ? r.updatedAt : latest;
+ }, "");
+
+ logger.debug?.(
+ `${TAG} [L2] Extraction complete: processed=${extractResult.memoriesProcessed}, latestCursor=${latestCursor}`,
+ );
+
+ return { latestCursor: latestCursor || undefined };
+ }
+ };
+}
+
+// ============================
+// L3 Runner factory
+// ============================
+
+/**
+ * Create the standard L3 runner function (persona generation).
+ *
+ * Uses PersonaTrigger to check if generation is needed, then runs
+ * PersonaGenerator. Used by both `index.ts` and `seed-runtime.ts`.
+ */
+export function createL3Runner(opts: {
+ pluginDataDir: string;
+ cfg: MemoryTdaiConfig;
+ openclawConfig: unknown;
+ vectorStore?: IMemoryStore;
+ logger: PipelineLogger;
+ instanceId?: string;
+}): L3Runner {
+ const { pluginDataDir, cfg, openclawConfig, vectorStore, logger, instanceId } = opts;
+
+ return async () => {
+ const trigger = new PersonaTrigger({
+ dataDir: pluginDataDir,
+ interval: cfg.persona.triggerEveryN,
+ logger,
+ });
+
+ const { should, reason } = await trigger.shouldGenerate();
+ if (!should) {
+ logger.debug?.(`${TAG} [L3] Persona generation not needed`);
+ return;
+ }
+
+ if (!openclawConfig) {
+ logger.warn(`${TAG} [L3] No OpenClaw config, skipping persona generation`);
+ return;
+ }
+
+ // Pull remote profiles to establish fresh baseline before generation.
+ // This ensures syncLocalProfilesToStore() has correct baselineVersion
+ // for the optimistic-lock check instead of defaulting to 0.
+ let profileBaseline = new Map();
+ if (vectorStore?.pullProfiles && !vectorStore.isDegraded()) {
+ profileBaseline = await pullProfilesToLocal(pluginDataDir, vectorStore, logger);
+ }
+
+ logger.info(`${TAG} [L3] Starting persona generation: ${reason}`);
+ const generator = new PersonaGenerator({
+ dataDir: pluginDataDir,
+ config: openclawConfig,
+ model: cfg.persona.model,
+ backupCount: cfg.persona.backupCount,
+ logger,
+ instanceId,
+ });
+ const genResult = await generator.generateLocalPersona(reason);
+ if (!genResult) {
+ logger.info(`${TAG} [L3] Persona generation skipped (no changes)`);
+ return;
+ }
+
+ if (vectorStore && supportsProfileSyncWrite(vectorStore)) {
+ await syncLocalProfilesToStore(pluginDataDir, vectorStore, profileBaseline, logger);
+ }
+
+ const checkpoint = new CheckpointManager(pluginDataDir, logger);
+ const cp = await checkpoint.read();
+ await checkpoint.markPersonaGenerated(cp.total_processed);
+ logger.info(`${TAG} [L3] Persona generation succeeded`);
+ };
+}
+
+// ============================
+// Pipeline Manager factory
+// ============================
+
+/**
+ * Create a MemoryPipelineManager with the standard config mapping.
+ */
+export function createPipelineManager(
+ cfg: MemoryTdaiConfig,
+ logger: PipelineLogger,
+ sessionFilter?: SessionFilter,
+): MemoryPipelineManager {
+ return new MemoryPipelineManager(
+ {
+ everyNConversations: cfg.pipeline.everyNConversations,
+ enableWarmup: cfg.pipeline.enableWarmup,
+ l1: { idleTimeoutSeconds: cfg.pipeline.l1IdleTimeoutSeconds },
+ l2: {
+ delayAfterL1Seconds: cfg.pipeline.l2DelayAfterL1Seconds,
+ minIntervalSeconds: cfg.pipeline.l2MinIntervalSeconds,
+ maxIntervalSeconds: cfg.pipeline.l2MaxIntervalSeconds,
+ sessionActiveWindowHours: cfg.pipeline.sessionActiveWindowHours,
+ },
+ },
+ logger,
+ sessionFilter ?? new SessionFilter([]),
+ );
+}
+
+// ============================
+// Full pipeline factory
+// ============================
+
+/**
+ * Create a fully wired pipeline instance: VectorStore + EmbeddingService +
+ * MemoryPipelineManager with L1 runner and persister attached.
+ *
+ * This is the high-level entry point used by both `index.ts` and `seed-runtime.ts`.
+ * Callers should attach L2/L3 runners after creation using `createL2Runner()`
+ * and `createL3Runner()` from this module.
+ */
+export async function createPipeline(opts: PipelineFactoryOptions): Promise {
+ const { pluginDataDir, cfg, openclawConfig, logger, sessionFilter } = opts;
+
+ // Ensure data directories exist
+ initDataDirectories(pluginDataDir);
+
+ // Initialize stores (once-async: reuses cached result if already initialized)
+ const stores = await initStores(cfg, pluginDataDir, logger);
+ const { vectorStore, embeddingService } = stores;
+
+ // Create pipeline manager
+ const scheduler = createPipelineManager(cfg, logger, sessionFilter);
+
+ // Wire L1 runner
+ scheduler.setL1Runner(createL1Runner({
+ pluginDataDir,
+ cfg,
+ openclawConfig,
+ vectorStore,
+ embeddingService,
+ logger,
+ }));
+
+ // Wire persister
+ scheduler.setPersister(createPersister(pluginDataDir, logger));
+
+ // Destroy function
+ const destroy = async () => {
+ logger.info(`${TAG} Destroying pipeline...`);
+ await scheduler.destroy();
+ if (vectorStore) {
+ logger.info(`${TAG} Closing VectorStore`);
+ vectorStore.close();
+ }
+ if (embeddingService?.close) {
+ try {
+ logger.info(`${TAG} Closing EmbeddingService`);
+ await embeddingService.close();
+ } catch (err) {
+ logger.warn(`${TAG} Error closing EmbeddingService: ${err instanceof Error ? err.message : String(err)}`);
+ }
+ }
+ logger.info(`${TAG} Pipeline destroyed`);
+ };
+
+ return { scheduler, vectorStore, embeddingService, destroy };
+}
diff --git a/src/utils/pipeline-manager.ts b/src/utils/pipeline-manager.ts
index 01c47c9..76c1c73 100644
--- a/src/utils/pipeline-manager.ts
+++ b/src/utils/pipeline-manager.ts
@@ -262,7 +262,7 @@ export class MemoryPipelineManager {
this.logger = logger;
this.sessionFilter = sessionFilter ?? new SessionFilter();
- this.logger?.info(
+ this.logger?.debug?.(
`${TAG} Initialized: everyNConversations=${config.everyNConversations}, ` +
`warmup=${config.enableWarmup ? "enabled" : "disabled"}, ` +
`l1IdleTimeout=${config.l1.idleTimeoutSeconds}s, ` +
@@ -1076,12 +1076,22 @@ export class MemoryPipelineManager {
return this.destroyed;
}
- /** Queue sizes for monitoring. */
- getQueueSizes(): { l1: number; l2: number; l3: number } {
+ /** Queue sizes and running state for monitoring. */
+ getQueueSizes(): {
+ l1: number; l2: number; l3: number;
+ l1Pending: boolean; l2Pending: boolean; l3Pending: boolean;
+ l1Idle: boolean; l2Idle: boolean; l3Idle: boolean;
+ } {
return {
l1: this.l1Queue.size,
l2: this.l2Queue.size,
l3: this.l3Queue.size,
+ l1Pending: this.l1Queue.pending,
+ l2Pending: this.l2Queue.pending,
+ l3Pending: this.l3Queue.pending,
+ l1Idle: this.l1Queue.idle,
+ l2Idle: this.l2Queue.idle,
+ l3Idle: this.l3Queue.idle,
};
}
}
diff --git a/src/utils/sanitize.ts b/src/utils/sanitize.ts
index 05097bc..93f976c 100644
--- a/src/utils/sanitize.ts
+++ b/src/utils/sanitize.ts
@@ -38,6 +38,9 @@ export function sanitizeText(text: string): string {
// Remove framework reply directive tags: [[reply_to_current]], [[reply_to_xxx]], etc.
cleaned = cleaned.replace(/\[\[reply_to[^\]]*\]\]\s*/g, "");
+ // Remove injected skill-selection wrappers, e.g. ¥¥[... ]¥¥
+ cleaned = cleaned.replace(/¥¥\[[\s\S]*?\]¥¥/g, "");
+
// Remove line-leading timestamps, e.g. "[Tue 2026-03-24 03:48 UTC]"
// or "[Tue 2026-03-24 20:21 GMT+8]", "[Thu 2026-03-24 01:51 GMT+5:30]"
// Matches brackets containing word chars, digits, hyphens, colons, plus signs,
diff --git a/src/utils/serial-queue.ts b/src/utils/serial-queue.ts
index dca2b94..63d179b 100644
--- a/src/utils/serial-queue.ts
+++ b/src/utils/serial-queue.ts
@@ -50,6 +50,11 @@ export class SerialQueue {
return this.running;
}
+ /** Whether the queue is idle (no queued tasks and nothing running). */
+ get idle(): boolean {
+ return this.queue.length === 0 && !this.running;
+ }
+
/** Add a task to the queue. Returns the task's result promise. */
add(task: Task): Promise {
return new Promise((resolve, reject) => {