mirror of
https://github.com/TencentCloud/TencentDB-Agent-Memory
synced 2026-07-10 12:34:27 +00:00
feat: release v0.2.2 — TCVDB backend, BM25 hybrid retrieval, pipeline refactor
This commit is contained in:
chrishuan
@@ -10,5 +10,10 @@ workspace/
|
||||
# Test caches
|
||||
__tests__/soak/.model-cache/
|
||||
|
||||
# Migration build output
|
||||
scripts/export-tencent-vdb/dist/
|
||||
scripts/migrate-sqlite-to-tcvdb/dist/
|
||||
scripts/read-local-memory/dist/
|
||||
|
||||
node_modules/
|
||||
benchmark-runs/
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
# 测试文件
|
||||
*.test.ts
|
||||
*.test.js
|
||||
*.spec.ts
|
||||
*.spec.js
|
||||
__tests__/
|
||||
|
||||
# 开发文档与辅助
|
||||
docs/
|
||||
benchmark-runs/
|
||||
workspace/
|
||||
|
||||
# 环境与配置
|
||||
.env
|
||||
.env.*
|
||||
.gitignore
|
||||
.codebuddy/
|
||||
.coding-ci.yaml
|
||||
|
||||
# 运行时产物
|
||||
node_modules/
|
||||
*.tgz
|
||||
@@ -4,6 +4,101 @@
|
||||
|
||||
---
|
||||
|
||||
## [0.2.2] - 2026-04-17
|
||||
|
||||
### 🐛 修复
|
||||
|
||||
- 修复因未声明 `undici` 依赖导致 TCVDB 客户端加载失败的问题（开发环境之前依赖 monorepo 根 `node_modules` 的传递解析）
|
||||
- 将插件注册阶段的大量 INFO 日志降级为 DEBUG，避免 CLI 模式下输出过多无关日志
|
||||
|
||||
## [0.2.1] - 2026-04-16 (deprecated)
|
||||
|
||||
> NOTE: 此版本由于存在 undici 依赖导致插件启动失败的问题，已废弃
|
||||
> 相关问题在 0.2.2 及以后版本中已修复
|
||||
|
||||
### 🚀 新功能
|
||||
|
||||
- TCVDB 新增 HTTPS 连接支持，可通过插件配置 `caPemPath` 或迁移脚本参数 `--tcvdb-ca-pem` 指定自定义 CA 证书 PEM 文件
|
||||
- `read-local-memory` 脚本新增 L2 单文件查询，并将 L0 / L1 查询切换为直接从 `vectors.db` 读取，支持 SQL 层过滤、排序与分页
|
||||
|
||||
### ✨ 改进
|
||||
|
||||
- TCVDB 的 L0 / L1 向量索引默认调整为 `DISK_FLAT`，并在不支持该索引类型的实例上自动回退到 `HNSW`
|
||||
- 默认服务端 embedding 模型调整为 `bge-large-zh`
|
||||
- TCVDB 所有读接口统一启用 `readConsistency: "strongConsistency"`，消除 read-after-write 不一致
|
||||
- 健康检测脚本 VDB 连接支持 HTTPS 自签证书
|
||||
|
||||
### 🐛 修复
|
||||
|
||||
- 修复 L3 persona sync 因未拉取远端 baseline 导致版本冲突跳过写入的问题
|
||||
- 修复 `memories_since_last_persona` 被 L0 和 L1 双重计数导致 persona 触发阈值膨胀的问题
|
||||
- 移除 `CheckpointManager` 中已被 `captureAtomically()` 替代的废弃方法
|
||||
|
||||
---
|
||||
|
||||
## [0.2.0] - 2026-04-15
|
||||
|
||||
### 🚀 新功能
|
||||
|
||||
**腾讯云向量数据库（TCVDB）存储后端**
|
||||
|
||||
- 新增腾讯云向量数据库存储后端，支持向量 + BM25 混合召回
|
||||
- 支持 SQLite 与 TCVDB 之间的索引结构同步
|
||||
- L2 场景 / L3 画像支持在本地缓存与向量数据库之间双向同步
|
||||
- 插件配置（manifest）暴露 `storeBackend`、`tcvdb`、`bm25`、`embedding.timeoutMs` 等配置项
|
||||
|
||||
**本地 BM25 关键字检索**
|
||||
|
||||
- 使用本地 tcvdb-text 编码器替代原有的 BM25 HTTP sidecar 服务，消除外部依赖
|
||||
|
||||
**Seed 数据导入工具**
|
||||
|
||||
- 新增 CLI `seed` 命令，支持从外部数据批量导入记忆
|
||||
- 提取共享的 pipeline-factory，供 seed 和正常运行时复用
|
||||
- 支持 ISO 8601 时间戳格式（移除 JSONL 支持）
|
||||
|
||||
**数据迁移与运维工具**
|
||||
|
||||
- 新增 SQLite → 腾讯云向量数据库迁移脚本，支持 `--help` / `-h` 展示完整参数说明和使用示例
|
||||
- 新增 VDB 数据导出脚本（含预编译 JS 和 CLI 启动器）
|
||||
- 新增本地 Memory 数据查询脚本
|
||||
- 注册全部 CLI bin 入口：`migrate-sqlite-to-tcvdb`、`export-tencent-vdb`、`read-local-memory`
|
||||
|
||||
**记忆搜索工具调用限制**
|
||||
|
||||
- `tdai_memory_search` + `tdai_conversation_search` 增加每轮合计最多 3 次的调用次数限制，通过 tool description 和召回引导提示词约束模型行为，防止陷入无效重复搜索
|
||||
|
||||
### 🐛 修复
|
||||
|
||||
- 修复 L2 场景合并（MERGE）无法删除旧文件的问题：OpenClaw 4.1+ 的 write 工具拒绝空白内容，改用 `[DELETED]` 标记实现软删除，SceneExtractor cleanup 阶段同步识别并清理
|
||||
- 修复 L2 抽取产生孤立 BATCH/ARCHIVE 文件的问题，统一 maxScenes 上限为 15
|
||||
- 修复 L3 启动时重复拉取 profile 的问题
|
||||
- 过滤 skill wrapper 噪声标记（`¥¥[...]¥¥`）
|
||||
- 处理 `createCollection` 并发竞态（错误码 15202）
|
||||
|
||||
### ♻️ 重构
|
||||
|
||||
- Pipeline checkpoint 游标语义从 timestamp 改为 update_at
|
||||
- Runner 改用 `api.runtime.agent.runEmbeddedPiAgent`，避免跨环境导入失败
|
||||
- 统一脚本构建流程：新增 `build:scripts` 一键编译命令，`prepack` 钩子确保 `npm pack` 前自动编译全部脚本产物
|
||||
|
||||
### 📚 文档
|
||||
|
||||
- 新增 AI Agent 长期记忆插件设计与实现技术文档
|
||||
- 新增项目指南、研发系统分层架构文档
|
||||
- 新增 VDB 存储设计文档及迁移指南
|
||||
|
||||
---
|
||||
|
||||
<details>
|
||||
<summary>预发布版本</summary>
|
||||
|
||||
## [0.2.0-beta.1] - 2026-04-14
|
||||
|
||||
*此版本的内容已合并至 [0.2.0] 正式版。*
|
||||
|
||||
</details>
|
||||
|
||||
## [0.1.4] - 2026-04-10
|
||||
|
||||
### 🚀 Features
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
---
|
||||
name: openclaw-diagnostic-export
|
||||
description: 帮助用户导出 OpenClaw + memory-tencentdb（原 memory-tdai）记忆插件的现场诊断数据，用于排查问题。当用户提到"导出诊断数据""export diagnostic""现场数据""排查问题""导出日志""收集现场""打包现场数据"时应触发。
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
## 目的
|
||||
|
||||
将 OpenClaw 日志、记忆插件数据（L0~L3）、脱敏后的配置打包为本地压缩包，由用户确认后手动发送给研发团队排查问题。
|
||||
|
||||
> **名称说明**：插件已从 `@tdai/memory-tdai` 更名为 `@tencentdb-agent-memory/memory-tencentdb`，但数据目录始终为 `~/.openclaw/memory-tdai/`（代码中硬编码）。本 skill 中所有对 `memory-tdai` 目录的引用均指实际数据目录路径，与插件 ID 无关。
|
||||
|
||||
## 导出工作流
|
||||
|
||||
### Step 1: 确认环境
|
||||
|
||||
在导出前，先确认 OpenClaw 工作目录存在且可访问：
|
||||
|
||||
```bash
|
||||
# 探测工作目录（优先级：环境变量 > ~/.openclaw > ~/.clawdbot）
|
||||
OPENCLAW_DIR="${OPENCLAW_STATE_DIR:-$HOME/.openclaw}"
|
||||
[ -d "$OPENCLAW_DIR" ] || OPENCLAW_DIR="$HOME/.clawdbot"
|
||||
ls -la "$OPENCLAW_DIR/" 2>/dev/null && echo "✅ 找到: $OPENCLAW_DIR" || echo "❌ 未找到 OpenClaw 工作目录"
|
||||
```
|
||||
|
||||
确认 memory-tdai 子目录存在：
|
||||
|
||||
```bash
|
||||
ls -la "$OPENCLAW_DIR/memory-tdai/" 2>/dev/null
|
||||
```
|
||||
|
||||
### Step 2: 执行导出脚本
|
||||
|
||||
运行项目 `scripts/` 目录下的导出脚本：
|
||||
|
||||
```bash
|
||||
bash scripts/export-diagnostic.sh
|
||||
```
|
||||
|
||||
> 脚本位于本项目的 `scripts/export-diagnostic.sh`，如果通过 `pnpm` 或其他方式运行，需确保工作目录在项目根目录下。
|
||||
|
||||
脚本默认将压缩包输出到 `~/Downloads/openclaw-diagnostic-<timestamp>.tar.gz`。
|
||||
|
||||
如需指定其他输出目录：
|
||||
|
||||
```bash
|
||||
bash scripts/export-diagnostic.sh /tmp
|
||||
```
|
||||
|
||||
### Step 3: 确认导出结果
|
||||
|
||||
脚本执行完成后，检查输出：
|
||||
|
||||
1. **确认压缩包已生成** — 脚本末尾会打印压缩包路径和大小
|
||||
2. **向用户说明包含内容**：
|
||||
|
||||
| 文件/目录 | 内容 | 隐私风险 |
|
||||
|-----------|------|---------|
|
||||
| `env-info.txt` | 系统版本、OpenClaw 版本、目录结构、磁盘占用 | 低 |
|
||||
| `logs/` | OpenClaw 网关日志 + 滚动日志（最近 3 天，每文件最多 5000 行） | 低 |
|
||||
| `memory-tdai/` | 记忆插件全量数据：L0 对话、L1 记忆、L2 场景、L3 画像、SQLite 数据库、checkpoint | **高** — 包含用户对话原文 |
|
||||
| `openclaw-config-redacted.json` | 脱敏后的配置（已移除 API Key/Token/Password/Secret，models/channels/env 整体替换） | 低 |
|
||||
| `plugins-info.txt` | 已安装插件列表和版本 | 低 |
|
||||
|
||||
3. **提醒用户**：
|
||||
- 配置文件已自动脱敏，API Key、Token 等敏感信息已被替换为 `***REDACTED***`
|
||||
- **记忆数据（memory-tdai/）包含用户对话原文**，请确认可以分享后再发送
|
||||
- 压缩包存放在本地，**不会自动上传**，需要用户手动发送给研发团队
|
||||
|
||||
### Step 4: 告知用户后续操作
|
||||
|
||||
导出完成后，告知用户：
|
||||
|
||||
1. 压缩包已保存在本地（打印具体路径）
|
||||
2. 请检查内容后，通过企微/邮件等方式手动发送给研发团队
|
||||
3. 如只需部分数据（如仅日志或仅配置），可解压后选择性发送
|
||||
|
||||
## 导出内容详解
|
||||
|
||||
### OpenClaw 日志位置
|
||||
|
||||
| 日志类型 | 路径 | 说明 |
|
||||
|---------|------|------|
|
||||
| 网关 stdout | `~/.openclaw/logs/gateway.log` | 网关守护进程标准输出 |
|
||||
| 网关 stderr | `~/.openclaw/logs/gateway.err.log` | 网关守护进程错误输出 |
|
||||
| 滚动日志 | `/tmp/openclaw/openclaw-YYYY-MM-DD.log` | 按日期滚动，JSON Lines 格式，24h 自动清理 |
|
||||
| 配置审计 | `~/.openclaw/logs/config-audit.jsonl` | 配置写入审计记录 |
|
||||
| 命令日志 | `~/.openclaw/logs/commands.log` | 命令事件日志（hook 可选） |
|
||||
|
||||
### 记忆插件数据结构
|
||||
|
||||
```
|
||||
~/.openclaw/memory-tdai/
|
||||
├── conversations/ — L0 原始对话（每日 JSONL 分片）
|
||||
├── records/ — L1 结构化记忆（每日 JSONL 分片）
|
||||
├── scene_blocks/ — L2 场景 Markdown 文件
|
||||
├── persona.md — L3 用户画像
|
||||
├── vectors.db — SQLite 数据库（向量 + 全文索引）
|
||||
├── .metadata/ — checkpoint、scene_index.json
|
||||
└── .backup/ — 滚动备份
|
||||
```
|
||||
|
||||
### 配置脱敏规则
|
||||
|
||||
导出脚本对 `openclaw.json` 执行以下脱敏：
|
||||
|
||||
| 规则 | 处理方式 |
|
||||
|------|---------|
|
||||
| 字段名匹配 `apiKey/token/password/secret/credential` 且值为字符串 | 替换为 `***REDACTED(Nchars)***` |
|
||||
| SecretRef 对象（含 source/provider/id） | id 替换为 `***REDACTED***` |
|
||||
| 顶层 `models`、`secrets`、`channels`、`env` 块 | 整体替换为 `***REDACTED_SECTION***` |
|
||||
| `gateway.auth` 下的 token/password | 替换为 `***REDACTED***` |
|
||||
| 其余字段（含 `plugins` 完整配置） | **保留原样**（插件配置是排查重点） |
|
||||
|
||||
## 手动导出（脚本不可用时的备选方案）
|
||||
|
||||
如果导出脚本无法执行（如 Node.js 不可用），按以下步骤手动收集：
|
||||
|
||||
```bash
|
||||
# 1. 创建导出目录
|
||||
EXPORT_DIR=~/Downloads/openclaw-diagnostic-$(date +%Y%m%d-%H%M%S)
|
||||
mkdir -p "$EXPORT_DIR"
|
||||
|
||||
# 2. 复制日志
|
||||
cp -r ~/.openclaw/logs/ "$EXPORT_DIR/logs/" 2>/dev/null
|
||||
cp /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log "$EXPORT_DIR/" 2>/dev/null
|
||||
|
||||
# 3. 复制记忆插件数据
|
||||
cp -r ~/.openclaw/memory-tdai/ "$EXPORT_DIR/memory-tdai/" 2>/dev/null
|
||||
|
||||
# 4. 手动脱敏配置（⚠️ 必须手动删除敏感字段！）
|
||||
# 复制配置并用编辑器删除 models/secrets/channels 块和所有 apiKey/token 值
|
||||
cp ~/.openclaw/openclaw.json "$EXPORT_DIR/openclaw-config-NEEDS-MANUAL-REDACTION.json"
|
||||
|
||||
# 5. 打包
|
||||
cd ~/Downloads && tar -czf "$EXPORT_DIR.tar.gz" "$(basename $EXPORT_DIR)"
|
||||
|
||||
echo "⚠️ 请务必在发送前手动检查并删除配置中的敏感信息！"
|
||||
```
|
||||
|
||||
## 常见问题排查线索
|
||||
|
||||
导出数据后，研发团队通常关注以下方面：
|
||||
|
||||
| 排查方向 | 查看文件 | 关键信息 |
|
||||
|---------|---------|---------|
|
||||
| 插件是否加载 | `logs/` 中搜索 `[memory-tdai]` | 插件注册、配置解析日志（注：日志标签仍为 `[memory-tdai]`，与插件 ID 无关） |
|
||||
| 记忆召回是否工作 | `logs/` 中搜索 `[recall]` | 搜索策略、耗时、命中数 |
|
||||
| L1 提取是否触发 | `logs/` 中搜索 `[pipeline]` | 调度触发、L1/L2/L3 执行状态 |
|
||||
| 向量搜索是否可用 | `openclaw-config-redacted.json` 的 `plugins.entries` | embedding 配置是否正确 |
|
||||
| 数据量/磁盘占用 | `env-info.txt` | du 输出、文件数量 |
|
||||
| checkpoint 状态 | `memory-tdai/.metadata/recall_checkpoint.json` | 进度、游标、计数器 |
|
||||
@@ -0,0 +1,239 @@
|
||||
---
|
||||
name: openclaw-memory-tencentdb-migration
|
||||
description: 帮助存量用户将 OpenClaw 记忆插件从旧包 @tdai/memory-tdai 迁移到新包 @tencentdb-agent-memory/memory-tencentdb。当用户提到"插件迁移""更换记忆插件包名""memory-tdai 升级""包名变更"或出现旧包相关安装报错时应触发。
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
## 目的
|
||||
|
||||
帮助已安装 `@tdai/memory-tdai`（旧包名）的存量用户，平滑迁移到 `@tencentdb-agent-memory/memory-tencentdb`（新包名），确保已有记忆数据不丢失、配置完整还原。
|
||||
|
||||
## 背景
|
||||
|
||||
- **旧包名**：`@tdai/memory-tdai`（插件 ID：`memory-tdai`）
|
||||
- **新包名**：`@tencentdb-agent-memory/memory-tencentdb`（插件 ID：`memory-tencentdb`）
|
||||
- 新旧插件共用相同的数据目录（`~/.openclaw/memory-tdai/`），卸载旧插件**不会删除数据目录**，已有记忆数据不受影响
|
||||
- 卸载旧插件**会删除** `openclaw.json` 中该插件的配置段，需提前备份
|
||||
|
||||
## 适用场景
|
||||
|
||||
- 用户已安装 `@tdai/memory-tdai`，需迁移到新包名
|
||||
- 用户执行 `openclaw plugins install @tdai/memory-tdai` 报 404 / not found
|
||||
- 用户被告知旧包已废弃，需要迁移
|
||||
|
||||
## 不适用场景
|
||||
|
||||
- 用户从未安装过记忆插件（应使用 `openclaw-memory-tencentdb-setup` skill）
|
||||
- 用户使用的是其他记忆插件（如 `openclaw-mem0`）
|
||||
|
||||
## 标准工作流
|
||||
|
||||
### 1) 确认当前状态
|
||||
|
||||
确认旧插件是否已安装：
|
||||
|
||||
```bash
|
||||
openclaw plugins list | grep -i memory
|
||||
```
|
||||
|
||||
预期看到 `memory-tdai` 或 `@tdai/memory-tdai` 处于 loaded 状态。
|
||||
|
||||
如果未看到旧插件，跳过迁移流程，直接使用 `openclaw-memory-tencentdb-setup` skill 进行全新安装。
|
||||
|
||||
### 2) 备份现有配置（关键步骤）
|
||||
|
||||
卸载旧插件会删除 `openclaw.json` 中的配置段。**必须先备份**。
|
||||
|
||||
执行以下命令提取旧插件配置：
|
||||
|
||||
```bash
|
||||
cat ~/.openclaw/openclaw.json | python3 -c "
|
||||
import sys, json
|
||||
cfg = json.load(sys.stdin)
|
||||
plugins = cfg.get('plugins', {}).get('entries', {})
|
||||
old_cfg = plugins.get('memory-tdai', {})
|
||||
if old_cfg:
|
||||
print(json.dumps(old_cfg, indent=2, ensure_ascii=False))
|
||||
with open('/tmp/memory-tdai-config-backup.json', 'w') as f:
|
||||
json.dump(old_cfg, f, indent=2, ensure_ascii=False)
|
||||
print('\n✅ 配置已备份到 /tmp/memory-tdai-config-backup.json')
|
||||
else:
|
||||
print('⚠️ 未找到 memory-tdai 配置段（可能使用默认配置）')
|
||||
"
|
||||
```
|
||||
|
||||
**特别关注以下配置是否存在（如有则必须记录）**：
|
||||
|
||||
- `embedding` 配置（`provider`、`baseUrl`、`apiKey`、`model`、`dimensions`、`proxyUrl`）
|
||||
- `extraction.model`（提取使用的模型）
|
||||
- `persona.model`（画像使用的模型）
|
||||
- `capture.excludeAgents`（排除的 agent 列表）
|
||||
- `capture.l0l1RetentionDays`（数据保留天数）
|
||||
|
||||
### 3) 确认数据目录存在
|
||||
|
||||
```bash
|
||||
ls -la ~/.openclaw/memory-tdai/
|
||||
```
|
||||
|
||||
预期看到：`conversations/`、`records/`、`scene_blocks/`、`vectors.db`、`persona.md` 等文件。
|
||||
|
||||
记录当前数据量作为迁移后验证依据：
|
||||
|
||||
```bash
|
||||
echo "=== 迁移前数据统计 ==="
|
||||
wc -l ~/.openclaw/memory-tdai/conversations/*.jsonl 2>/dev/null || echo "无对话数据"
|
||||
wc -l ~/.openclaw/memory-tdai/records/*.jsonl 2>/dev/null || echo "无记录数据"
|
||||
ls ~/.openclaw/memory-tdai/scene_blocks/*.md 2>/dev/null | wc -l | xargs -I{} echo "场景块: {} 个"
|
||||
wc -c ~/.openclaw/memory-tdai/persona.md 2>/dev/null || echo "无 persona"
|
||||
```
|
||||
|
||||
### 4) 卸载旧插件
|
||||
|
||||
```bash
|
||||
openclaw plugins uninstall memory-tdai
|
||||
```
|
||||
|
||||
执行后确认：
|
||||
|
||||
- `openclaw.json` 中 `memory-tdai` 配置段已被删除（预期行为）
|
||||
- `~/.openclaw/memory-tdai/` 数据目录**仍然存在**（不会被删除）
|
||||
|
||||
```bash
|
||||
# 验证数据目录仍在
|
||||
ls ~/.openclaw/memory-tdai/ && echo "✅ 数据目录完好" || echo "❌ 数据目录丢失！"
|
||||
```
|
||||
|
||||
### 5) 安装新插件
|
||||
|
||||
```bash
|
||||
openclaw plugins install @tencentdb-agent-memory/memory-tencentdb
|
||||
```
|
||||
|
||||
### 6) 还原配置
|
||||
|
||||
将步骤 2 备份的配置写回 `openclaw.json`，注意新插件的配置 key 是 `memory-tencentdb`：
|
||||
|
||||
```bash
|
||||
python3 -c "
|
||||
import json, os
|
||||
|
||||
# 读取备份配置
|
||||
backup_path = '/tmp/memory-tdai-config-backup.json'
|
||||
if os.path.exists(backup_path):
|
||||
with open(backup_path) as f:
|
||||
old_cfg = json.load(f)
|
||||
print('📋 备份配置内容：')
|
||||
print(json.dumps(old_cfg, indent=2, ensure_ascii=False))
|
||||
else:
|
||||
old_cfg = {'enabled': True}
|
||||
print('⚠️ 未找到备份，使用最小配置')
|
||||
|
||||
# 读取当前 openclaw.json
|
||||
config_path = os.path.expanduser('~/.openclaw/openclaw.json')
|
||||
with open(config_path) as f:
|
||||
cfg = json.load(f)
|
||||
|
||||
# 写入新插件配置
|
||||
cfg.setdefault('plugins', {}).setdefault('entries', {})['memory-tencentdb'] = old_cfg
|
||||
|
||||
with open(config_path, 'w') as f:
|
||||
json.dump(cfg, f, indent=2, ensure_ascii=False)
|
||||
|
||||
print('\n✅ 配置已写入 memory-tencentdb')
|
||||
"
|
||||
```
|
||||
|
||||
如果备份丢失或用户需要手动恢复，至少确保写入最小配置：
|
||||
|
||||
```json
|
||||
{
|
||||
"memory-tencentdb": {
|
||||
"enabled": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 7) 重启 Gateway 并验证
|
||||
|
||||
```bash
|
||||
openclaw gateway restart
|
||||
```
|
||||
|
||||
检查项：
|
||||
|
||||
- Gateway 日志中出现 `[memory-tdai]` 前缀（注：日志标签仍为 memory-tdai，这是正常的）
|
||||
- 数据目录内容未变化
|
||||
|
||||
```bash
|
||||
echo "=== 迁移后验证 ==="
|
||||
# 确认新插件已加载
|
||||
openclaw plugins list | grep -i memory
|
||||
|
||||
# 确认数据量与迁移前一致
|
||||
wc -l ~/.openclaw/memory-tdai/conversations/*.jsonl 2>/dev/null
|
||||
wc -l ~/.openclaw/memory-tdai/records/*.jsonl 2>/dev/null
|
||||
```
|
||||
|
||||
### 8) 功能冒烟验证
|
||||
|
||||
执行一次对话确认记忆链路正常：
|
||||
|
||||
1. 发送一条包含个人信息的消息（如偏好、习惯）
|
||||
2. 确认日志中有 `[before_prompt_build]` 和 `[agent_end]` 相关输出
|
||||
3. 如有 embedding 配置，确认向量检索正常（日志无 embedding 报错）
|
||||
|
||||
## 回滚方案
|
||||
|
||||
如迁移后出现问题，可快速回滚：
|
||||
|
||||
```bash
|
||||
# 1. 卸载新插件
|
||||
openclaw plugins uninstall memory-tencentdb
|
||||
|
||||
# 2. 重新安装旧插件（如 npm 源仍可用）
|
||||
openclaw plugins install @tdai/memory-tdai
|
||||
|
||||
# 3. 手动还原配置（从备份）
|
||||
# 将 /tmp/memory-tdai-config-backup.json 内容写回 openclaw.json 的 memory-tdai 段
|
||||
|
||||
# 4. 重启
|
||||
openclaw gateway restart
|
||||
```
|
||||
|
||||
## 故障排查
|
||||
|
||||
| 现象 | 可能原因 | 解决方案 |
|
||||
|------|----------|----------|
|
||||
| 新插件无日志输出 | 配置中 `enabled` 未设为 `true` | 检查 `openclaw.json` 中 `memory-tencentdb.enabled` |
|
||||
| 安装新插件报错 | npm 源不可用 | 检查网络 / npm registry 配置 |
|
||||
| 迁移后无历史记忆 | 配置还原不完整 | 对比 `/tmp/memory-tdai-config-backup.json` 与当前配置 |
|
||||
| embedding 报错 | `apiKey` 等配置丢失 | 从备份中还原 `embedding` 配置段 |
|
||||
| 数据目录为空 | 卸载时异常删除（极少见） | 检查 `~/.openclaw/memory-tdai/` 是否存在 |
|
||||
|
||||
## 安全与合规约束
|
||||
|
||||
- 备份文件 `/tmp/memory-tdai-config-backup.json` 可能包含 `apiKey`，迁移完成后建议删除：`rm /tmp/memory-tdai-config-backup.json`
|
||||
- 不在聊天、日志中明文展示 `apiKey`
|
||||
- 仅修改 `memory-tencentdb` 配置段，不影响用户其它插件
|
||||
|
||||
## 完成定义（Definition of Done）
|
||||
|
||||
迁移完成需同时满足：
|
||||
|
||||
- [x] 旧插件 `@tdai/memory-tdai` 已卸载
|
||||
- [x] 新插件 `@tencentdb-agent-memory/memory-tencentdb` 已安装并加载
|
||||
- [x] `openclaw.json` 中存在完整的 `memory-tencentdb` 配置（含用户自定义的 embedding 等配置）
|
||||
- [x] Gateway 已重启
|
||||
- [x] 日志中出现 `[memory-tdai]` 前缀
|
||||
- [x] 数据目录完好，数据量与迁移前一致
|
||||
- [x] 至少 1 次对话验证记忆链路正常
|
||||
- [x] 已清理备份文件中的敏感信息
|
||||
|
||||
## 交付话术模板
|
||||
|
||||
> 已完成记忆插件迁移：
|
||||
> - 旧插件 `@tdai/memory-tdai` → 新插件 `@tencentdb-agent-memory/memory-tencentdb`
|
||||
> - 已有记忆数据完整保留（对话/记录/场景块/向量库均未受影响）
|
||||
> - 配置已从旧插件完整还原（含 embedding / extraction / persona 等自定义配置）
|
||||
> - Gateway 已重启，记忆链路验证正常
|
||||
@@ -0,0 +1,201 @@
|
||||
---
|
||||
name: openclaw-memory-tencentdb-setup
|
||||
description: 用于在 OpenClaw 环境中安装、配置并验证 @tencentdb-agent-memory/memory-tencentdb 插件。当用户提到"安装记忆插件""配置 memory-tencentdb""开启长期记忆/召回"或出现相关报错时应触发。
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
## 目的
|
||||
|
||||
在不依赖外部托管记忆服务的前提下，为 OpenClaw 提供可持续的本地长期记忆能力（L0→L1→L2→L3），并完成从安装、配置到验收的一次性闭环。
|
||||
|
||||
## 适用场景
|
||||
|
||||
- 用户要求在 OpenClaw 中安装或启用 `memory-tencentdb`
|
||||
- 用户需要配置召回、提取、画像、清理等参数
|
||||
- 用户反馈"插件已装但无记忆 / 无召回 / 无向量检索"
|
||||
|
||||
## 不适用场景
|
||||
|
||||
- 用户只需要解释 memory 理念，不要求实际落地
|
||||
- 用户要接入非 OpenClaw 宿主（先确认目标框架）
|
||||
|
||||
## 标准工作流
|
||||
|
||||
### 1) 环境预检
|
||||
|
||||
先确认基础版本满足要求：
|
||||
|
||||
- OpenClaw: `>= 2026.3.13`
|
||||
- Node.js: `>= 22.16.0`
|
||||
|
||||
执行：
|
||||
|
||||
```bash
|
||||
openclaw --version
|
||||
node -v
|
||||
```
|
||||
|
||||
若版本不满足，先升级再继续。
|
||||
|
||||
### 2) 安装插件
|
||||
|
||||
执行安装命令：
|
||||
|
||||
```bash
|
||||
openclaw plugins install @tencentdb-agent-memory/memory-tencentdb
|
||||
```
|
||||
|
||||
如已安装则执行更新：
|
||||
|
||||
```bash
|
||||
openclaw plugins update memory-tencentdb
|
||||
```
|
||||
|
||||
### 3) 写入最小配置
|
||||
|
||||
编辑 `~/.openclaw/openclaw.json`，确保存在：
|
||||
|
||||
```json
|
||||
{
|
||||
"memory-tencentdb": {
|
||||
"enabled": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
说明：该插件支持零配置启动；不补充其它字段也能运行基础能力。
|
||||
|
||||
### 4) 按需追加推荐配置（生产常用）
|
||||
|
||||
根据用户需求补充如下分组：
|
||||
|
||||
- `capture`: 对话捕获与保留策略
|
||||
- `extraction`: L1 提取与去重
|
||||
- `pipeline`: L1→L2→L3 调度
|
||||
- `recall`: 召回数量、阈值、策略
|
||||
- `persona`: 场景与画像触发参数
|
||||
- `embedding`: 向量检索配置（远端 OpenAI 兼容）
|
||||
|
||||
推荐模板：
|
||||
|
||||
```json
|
||||
{
|
||||
"memory-tencentdb": {
|
||||
"capture": {
|
||||
"enabled": true,
|
||||
"excludeAgents": [],
|
||||
"l0l1RetentionDays": 90,
|
||||
"cleanTime": "03:00"
|
||||
},
|
||||
"extraction": {
|
||||
"enabled": true,
|
||||
"enableDedup": true,
|
||||
"maxMemoriesPerSession": 10,
|
||||
"model": "provider/model"
|
||||
},
|
||||
"pipeline": {
|
||||
"everyNConversations": 5,
|
||||
"enableWarmup": true,
|
||||
"l1IdleTimeoutSeconds": 60,
|
||||
"l2DelayAfterL1Seconds": 90,
|
||||
"l2MinIntervalSeconds": 300,
|
||||
"l2MaxIntervalSeconds": 1800,
|
||||
"sessionActiveWindowHours": 24
|
||||
},
|
||||
"recall": {
|
||||
"enabled": true,
|
||||
"maxResults": 5,
|
||||
"scoreThreshold": 0.3,
|
||||
"strategy": "hybrid"
|
||||
},
|
||||
"persona": {
|
||||
"triggerEveryN": 50,
|
||||
"maxScenes": 15,
|
||||
"backupCount": 3,
|
||||
"sceneBackupCount": 10,
|
||||
"model": "provider/model"
|
||||
},
|
||||
"embedding": {
|
||||
"enabled": true,
|
||||
"provider": "openai",
|
||||
"baseUrl": "https://api.openai.com/v1",
|
||||
"apiKey": "${EMBEDDING_API_KEY}",
|
||||
"model": "text-embedding-3-small",
|
||||
"dimensions": 1536,
|
||||
"conflictRecallTopK": 5
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5) 关键配置规则（避免隐性失败）
|
||||
|
||||
- `embedding.provider = "none"` 时，向量能力会禁用，仅保留关键词路径。
|
||||
- 若配置远端 `provider`（如 `openai` / `deepseek`），必须同时提供：
|
||||
- `apiKey`
|
||||
- `baseUrl`
|
||||
- `model`
|
||||
- `dimensions`
|
||||
- 上述任一缺失时，插件会继续运行，但自动降级为非向量模式。
|
||||
- `l0l1RetentionDays`：
|
||||
- `0` 表示不清理
|
||||
- 非 `0` 时建议 `>=3`
|
||||
- 若设为 `1~2`，需显式开启 `allowAggressiveCleanup`
|
||||
|
||||
### 6) 重启并验证生效
|
||||
|
||||
执行：
|
||||
|
||||
```bash
|
||||
openclaw gateway restart
|
||||
```
|
||||
|
||||
检查项：
|
||||
|
||||
- Gateway 日志中出现 `[memory-tdai]` 前缀
|
||||
- 数据目录已创建：`~/.openclaw/state/memory-tdai/`
|
||||
- 至少包含：`conversations/`、`records/`、`scene_blocks/`、`vectors.db`
|
||||
|
||||
### 7) 功能冒烟测试
|
||||
|
||||
执行一次最小对话回路并验证：
|
||||
|
||||
1. 连续对话 2~3 轮，提供可记忆信息（偏好、约束、背景）。
|
||||
2. 发起新一轮对话，观察是否出现召回上下文注入。
|
||||
3. 在 Agent 中调用：
|
||||
- `tdai_memory_search`
|
||||
- `tdai_conversation_search`
|
||||
4. 确认能检索到刚刚产生的内容。
|
||||
|
||||
## 故障排查速查
|
||||
|
||||
- 插件无日志：检查 `openclaw.json` 中 `memory-tencentdb.enabled` 是否为 `true`，并确认已重启 Gateway。
|
||||
- 有记录无召回：检查 `recall.enabled`、`scoreThreshold` 是否过高。
|
||||
- 无向量结果：检查 `embedding` 四元组（`apiKey/baseUrl/model/dimensions`）是否齐全。
|
||||
- 清理过猛导致历史过少：检查 `l0l1RetentionDays` 与 `allowAggressiveCleanup`。
|
||||
- 配置已改但行为不变：确认修改的是 `~/.openclaw/openclaw.json`，并再次重启 Gateway。
|
||||
|
||||
## 安全与合规约束
|
||||
|
||||
- 将 `apiKey` 视为敏感信息；不在聊天、日志、截图中明文扩散。
|
||||
- 优先使用环境变量注入密钥；配置示例中仅保留占位符。
|
||||
- 仅修改 `memory-tencentdb` 对应配置段，避免覆盖用户其它插件配置。
|
||||
|
||||
## 完成定义（Definition of Done）
|
||||
|
||||
在结束任务前，必须同时满足：
|
||||
|
||||
- 插件安装/更新命令执行成功
|
||||
- `openclaw.json` 已存在有效 `memory-tencentdb` 配置
|
||||
- Gateway 已重启
|
||||
- `[memory-tdai]` 日志可见
|
||||
- 数据目录与关键文件已生成
|
||||
- 至少 1 次检索工具调用成功返回结果
|
||||
|
||||
## 交付话术模板
|
||||
|
||||
可在完成后向用户输出：
|
||||
|
||||
- 已完成 `memory-tencentdb` 安装与配置，并重启 Gateway。
|
||||
- 已验证日志与数据目录生效，记忆链路可用。
|
||||
- 如需下一步优化，可继续调优 `recall.scoreThreshold`、`pipeline.everyNConversations`、`persona.triggerEveryN` 与 `embedding` 模型参数。
|
||||
@@ -10,7 +10,6 @@
|
||||
* All processing is local, zero external API dependencies.
|
||||
*/
|
||||
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { createRequire } from "node:module";
|
||||
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/core";
|
||||
@@ -19,54 +18,33 @@ import type { MemoryTdaiConfig } from "./src/config.js";
|
||||
import { performAutoRecall } from "./src/hooks/auto-recall.js";
|
||||
import { performAutoCapture } from "./src/hooks/auto-capture.js";
|
||||
import { MemoryPipelineManager } from "./src/utils/pipeline-manager.js";
|
||||
import { SceneExtractor } from "./src/scene/scene-extractor.js";
|
||||
import { CheckpointManager } from "./src/utils/checkpoint.js";
|
||||
import { PersonaTrigger } from "./src/persona/persona-trigger.js";
|
||||
import { PersonaGenerator } from "./src/persona/persona-generator.js";
|
||||
import { prewarmEmbeddedAgent } from "./src/utils/clean-context-runner.js";
|
||||
import {
|
||||
prewarmEmbeddedAgent,
|
||||
setPreferredEmbeddedAgentRuntime,
|
||||
} from "./src/utils/clean-context-runner.js";
|
||||
import { SessionFilter } from "./src/utils/session-filter.js";
|
||||
import { extractL1Memories } from "./src/record/l1-extractor.js";
|
||||
import { readConversationMessagesGroupedBySessionId } from "./src/conversation/l0-recorder.js";
|
||||
import type { ConversationMessage } from "./src/conversation/l0-recorder.js";
|
||||
import { VectorStore } from "./src/store/vector-store.js";
|
||||
import { createEmbeddingService } from "./src/store/embedding.js";
|
||||
import type { IMemoryStore } from "./src/store/types.js";
|
||||
import type { EmbeddingService } from "./src/store/embedding.js";
|
||||
import { executeMemorySearch, formatSearchResponse } from "./src/tools/memory-search.js";
|
||||
import { executeConversationSearch, formatConversationSearchResponse } from "./src/tools/conversation-search.js";
|
||||
import { LocalMemoryCleaner } from "./src/utils/memory-cleaner.js";
|
||||
import { getOrCreateInstanceId, initReporter, report } from "./src/report/reporter.js";
|
||||
import { registerMemoryTdaiCli } from "./src/cli/index.js";
|
||||
import {
|
||||
initDataDirectories,
|
||||
initStores,
|
||||
resetStores,
|
||||
createPipelineManager,
|
||||
createL1Runner,
|
||||
createPersister,
|
||||
createL2Runner,
|
||||
createL3Runner,
|
||||
} from "./src/utils/pipeline-factory.js";
|
||||
import { getOrCreateInstanceId, initReporter, report, resetReporter } from "./src/report/reporter.js";
|
||||
import { ensureL2L3Local } from "./src/profile/profile-sync.js";
|
||||
|
||||
const TAG = "[memory-tdai]";
|
||||
|
||||
/**
|
||||
* Initialize all required data directories under the plugin data root.
|
||||
*
|
||||
* Called once at plugin registration time so downstream modules
|
||||
* (L0 recorder, L1 writer, scene extractor, persona generator, etc.)
|
||||
* don't need to lazily mkdir on every write — the directories are
|
||||
* guaranteed to exist from startup.
|
||||
*
|
||||
* Directory layout:
|
||||
* <pluginDataDir>/
|
||||
* ├── conversations/ — L0 daily JSONL shards (one message per line)
|
||||
* ├── records/ — L1 daily JSONL shards (extracted memories)
|
||||
* ├── scene_blocks/ — L2 scene block .md files (LLM-managed)
|
||||
* ├── .metadata/ — checkpoint, scene_index.json
|
||||
* └── .backup/ — rotating backups (persona, scene_blocks)
|
||||
*/
|
||||
function initDataDirectories(dataDir: string): void {
|
||||
const dirs = [
|
||||
"conversations",
|
||||
"records",
|
||||
"scene_blocks",
|
||||
".metadata",
|
||||
".backup",
|
||||
];
|
||||
for (const sub of dirs) {
|
||||
fs.mkdirSync(path.join(dataDir, sub), { recursive: true });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Epoch ms when the plugin was registered (cold-start timestamp).
|
||||
* Used as a fallback cursor in performAutoCapture when no checkpoint
|
||||
@@ -151,20 +129,20 @@ function sweepStaleCaches(): void {
|
||||
|
||||
export default function register(api: OpenClawPluginApi) {
|
||||
pluginStartTimestamp = Date.now();
|
||||
setPreferredEmbeddedAgentRuntime(api.runtime.agent);
|
||||
// Reset reporter singleton so config changes take effect on hot-reload.
|
||||
resetReporter();
|
||||
const _require = createRequire(import.meta.url);
|
||||
const pluginVersion = (() => { try { return (_require("./package.json") as { version?: string }).version ?? "unknown"; } catch { return "unknown"; } })();
|
||||
api.logger.info(
|
||||
api.logger.debug?.(
|
||||
`${TAG} Registering plugin ... ` +
|
||||
`startTimestamp=${pluginStartTimestamp} (${new Date(pluginStartTimestamp).toISOString()})`,
|
||||
);
|
||||
|
||||
// Persistent instance ID for metric reporting (populated async below)
|
||||
let instanceId: string | undefined;
|
||||
|
||||
let cfg: MemoryTdaiConfig;
|
||||
try {
|
||||
cfg = parseConfig(api.pluginConfig as Record<string, unknown> | undefined);
|
||||
api.logger.info(
|
||||
api.logger.debug?.(
|
||||
`${TAG} Config parsed: ` +
|
||||
`capture=${cfg.capture.enabled}, ` +
|
||||
`recall=${cfg.recall.enabled}(maxResults=${cfg.recall.maxResults}), ` +
|
||||
@@ -186,12 +164,26 @@ export default function register(api: OpenClawPluginApi) {
|
||||
// Resolve plugin data directory via runtime API (avoid importing internal paths directly)
|
||||
const pluginDataDir = path.join(api.runtime.state.resolveStateDir(), "memory-tdai");
|
||||
initDataDirectories(pluginDataDir);
|
||||
api.logger.info(`${TAG} Data dir: ${pluginDataDir} (all subdirectories initialized)`);
|
||||
api.logger.debug?.(`${TAG} Data dir: ${pluginDataDir} (all subdirectories initialized)`);
|
||||
|
||||
// Kick off instanceId resolution immediately after data dir is ready.
|
||||
// getOrCreateInstanceId only reads/writes a small UUID file and caches the
|
||||
// result — starting it here means it will almost certainly be settled before
|
||||
// the first L1 runner fires, avoiding the need to defer metric reporting.
|
||||
let instanceId: string | undefined;
|
||||
getOrCreateInstanceId(pluginDataDir).then((id) => {
|
||||
instanceId = id;
|
||||
// initReporter is guarded by a "already initialised" check, so calling it
|
||||
// here is safe even if the registration-complete call below fires first.
|
||||
initReporter({ enabled: cfg.report.enabled, type: cfg.report.type, logger: api.logger, instanceId: id, pluginVersion });
|
||||
}).catch((err) => {
|
||||
api.logger.warn(`${TAG} Failed to initialize instanceId for metrics: ${err instanceof Error ? err.message : String(err)}`);
|
||||
});
|
||||
|
||||
// Unified session/agent filter: combines internal-session detection + user-configured excludeAgents
|
||||
const sessionFilter = new SessionFilter(cfg.capture.excludeAgents);
|
||||
if (cfg.capture.excludeAgents.length > 0) {
|
||||
api.logger.info(`${TAG} Agent exclude patterns: ${cfg.capture.excludeAgents.join(", ")}`);
|
||||
api.logger.debug?.(`${TAG} Agent exclude patterns: ${cfg.capture.excludeAgents.join(", ")}`);
|
||||
}
|
||||
|
||||
// Daily local JSONL cleaner (L0/L1), enabled only when retentionDays is configured.
|
||||
@@ -205,13 +197,13 @@ export default function register(api: OpenClawPluginApi) {
|
||||
logger: api.logger,
|
||||
});
|
||||
sharedMemoryCleaner.start();
|
||||
api.logger.info(`${TAG} Memory cleaner started (singleton)`);
|
||||
api.logger.debug?.(`${TAG} Memory cleaner started (singleton)`);
|
||||
} else {
|
||||
api.logger.info(`${TAG} Memory cleaner already started in this process, reusing existing instance`);
|
||||
api.logger.debug?.(`${TAG} Memory cleaner already started in this process, reusing existing instance`);
|
||||
}
|
||||
memoryCleaner = sharedMemoryCleaner;
|
||||
} else {
|
||||
api.logger.info(`${TAG} Memory cleaner disabled (retentionDays not configured)`);
|
||||
api.logger.debug?.(`${TAG} Memory cleaner disabled (retentionDays not configured)`);
|
||||
}
|
||||
|
||||
// Hardcoded actor ID (legacy, to be removed)
|
||||
@@ -228,7 +220,7 @@ export default function register(api: OpenClawPluginApi) {
|
||||
// ============================
|
||||
|
||||
// Shared references for tools (populated when extraction scheduler creates them)
|
||||
let sharedVectorStore: VectorStore | undefined;
|
||||
let sharedVectorStore: IMemoryStore | undefined;
|
||||
let sharedEmbeddingService: EmbeddingService | undefined;
|
||||
|
||||
/**
|
||||
@@ -272,12 +264,14 @@ export default function register(api: OpenClawPluginApi) {
|
||||
};
|
||||
|
||||
// tdai_memory_search — Agent-callable L1 memory search tool
|
||||
// TODO: implement hard per-turn call limit via before_tool_call hook + execute early-return (方案 D)
|
||||
api.registerTool(
|
||||
{
|
||||
name: "tdai_memory_search",
|
||||
label: "Memory Search",
|
||||
description:
|
||||
"Search through the user's long-term memories. Use this when you need to recall specific information about the user's preferences, past events, instructions, or context from previous conversations. Returns relevant memory records ranked by relevance.",
|
||||
"Search through the user's long-term memories. Use this when you need to recall specific information about the user's preferences, past events, instructions, or context from previous conversations. Returns relevant memory records ranked by relevance. " +
|
||||
"Limit: tdai_memory_search and tdai_conversation_search share a combined limit of 3 calls per turn. Stop searching after 3 total attempts.",
|
||||
parameters: {
|
||||
type: "object",
|
||||
properties: {
|
||||
@@ -367,6 +361,7 @@ export default function register(api: OpenClawPluginApi) {
|
||||
);
|
||||
|
||||
// tdai_conversation_search — Agent-callable L0 conversation search tool
|
||||
// TODO: implement hard per-turn call limit via before_tool_call hook + execute early-return (方案 D)
|
||||
api.registerTool(
|
||||
{
|
||||
name: "tdai_conversation_search",
|
||||
@@ -375,7 +370,8 @@ export default function register(api: OpenClawPluginApi) {
|
||||
"Search through past conversation history (raw dialogue records). " +
|
||||
"Use this when tdai_memory_search (structured memories) doesn't have the information you need, " +
|
||||
"or when you want to find specific past conversations, dialogue context, or exact words " +
|
||||
"the user said before. Returns relevant individual messages ranked by relevance.",
|
||||
"the user said before. Returns relevant individual messages ranked by relevance. " +
|
||||
"Limit: tdai_memory_search and tdai_conversation_search share a combined limit of 3 calls per turn. Stop searching after 3 total attempts.",
|
||||
parameters: {
|
||||
type: "object",
|
||||
properties: {
|
||||
@@ -464,7 +460,7 @@ export default function register(api: OpenClawPluginApi) {
|
||||
// (migrated from legacy before_agent_start to before_prompt_build so that
|
||||
// event.messages is guaranteed to be available — session is already loaded)
|
||||
if (cfg.recall.enabled) {
|
||||
api.logger.info(`${TAG} Registering before_prompt_build hook (auto-recall)`);
|
||||
api.logger.debug?.(`${TAG} Registering before_prompt_build hook (auto-recall)`);
|
||||
api.on("before_prompt_build", async (event, ctx) => {
|
||||
const startMs = Date.now();
|
||||
api.logger.debug?.(`${TAG} [before_prompt_build] Hook triggered`);
|
||||
@@ -619,343 +615,128 @@ export default function register(api: OpenClawPluginApi) {
|
||||
scheduler.start({});
|
||||
}
|
||||
|
||||
// Pre-warm the embedded agent import so the first extraction run doesn't
|
||||
// pay the cold-start cost (~35s jiti compile → <50ms with dist/ path).
|
||||
prewarmEmbeddedAgent(api.logger);
|
||||
// Pre-warm the embedded agent entrypoint. When runtime already exposes
|
||||
// runEmbeddedPiAgent this becomes a no-op; otherwise it still preloads
|
||||
// the legacy dist bridge to reduce first-run cold start.
|
||||
prewarmEmbeddedAgent(api.logger, api.runtime.agent);
|
||||
};
|
||||
|
||||
if (cfg.extraction.enabled) {
|
||||
// === Initialize VectorStore (always) + EmbeddingService (only when embedding enabled) ===
|
||||
let vectorStore: VectorStore | undefined;
|
||||
// === Store + scheduler initialization (async, runs eagerly) ===
|
||||
// Wrapped in an async IIFE because register() is synchronous.
|
||||
// initStores() is once-async: the first call creates the store,
|
||||
// subsequent calls (e.g. from seed CLI) reuse the cached result.
|
||||
let vectorStore: IMemoryStore | undefined;
|
||||
let embeddingService: EmbeddingService | undefined;
|
||||
|
||||
// VectorStore is always created as the metadata store for L0/L1 records.
|
||||
// It works as a pure SQLite store even without embedding — keyword search,
|
||||
// L0/L1 reads, and pipeline queries all use structured SQL, not vectors.
|
||||
try {
|
||||
const dims = cfg.embedding.dimensions; // 0 when provider="none" → vec0 tables deferred
|
||||
const dbPath = path.join(pluginDataDir, "vectors.db");
|
||||
vectorStore = new VectorStore(dbPath, dims, api.logger);
|
||||
const storeReady = (async () => {
|
||||
const stores = await initStores(cfg, pluginDataDir, api.logger);
|
||||
vectorStore = stores.vectorStore;
|
||||
embeddingService = stores.embeddingService;
|
||||
|
||||
// Create EmbeddingService only when embedding is enabled (remote provider configured)
|
||||
if (cfg.embedding.enabled) {
|
||||
// Share with tools immediately
|
||||
sharedVectorStore = vectorStore;
|
||||
sharedEmbeddingService = embeddingService;
|
||||
|
||||
// Keep cleaner's SQLite handle updated (singleton cleaner may start earlier).
|
||||
memoryCleaner?.setVectorStore(vectorStore);
|
||||
|
||||
if (vectorStore?.pullProfiles) {
|
||||
try {
|
||||
if (cfg.embedding.provider !== "local" && cfg.embedding.apiKey) {
|
||||
// Remote embedding provider (OpenAI-compatible API: OpenAI, Azure, self-hosted, etc.)
|
||||
embeddingService = createEmbeddingService({
|
||||
provider: cfg.embedding.provider,
|
||||
baseUrl: cfg.embedding.baseUrl,
|
||||
apiKey: cfg.embedding.apiKey,
|
||||
model: cfg.embedding.model,
|
||||
dimensions: cfg.embedding.dimensions,
|
||||
proxyUrl: cfg.embedding.proxyUrl,
|
||||
maxInputChars: cfg.embedding.maxInputChars,
|
||||
timeoutMs: cfg.embedding.timeoutMs,
|
||||
}, api.logger);
|
||||
} else {
|
||||
// Local provider (node-llama-cpp) — preserved internally but not reachable from user config
|
||||
embeddingService = createEmbeddingService({
|
||||
provider: "local",
|
||||
modelPath: cfg.embedding.model || undefined,
|
||||
modelCacheDir: cfg.embedding.modelCacheDir,
|
||||
}, api.logger);
|
||||
}
|
||||
await ensureL2L3Local(pluginDataDir, vectorStore, api.logger);
|
||||
} catch (err) {
|
||||
api.logger.warn(
|
||||
`${TAG} EmbeddingService init failed, continuing with keyword-only mode: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
embeddingService = undefined;
|
||||
api.logger.warn(`${TAG} Startup L2/L3 pull failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
} else {
|
||||
api.logger.info(`${TAG} Embedding disabled by config, VectorStore will serve as metadata-only store`);
|
||||
}
|
||||
|
||||
// Init VectorStore with provider info (undefined when no embedding → skips provider change detection)
|
||||
const providerInfo = embeddingService?.getProviderInfo();
|
||||
const initResult = vectorStore.init(providerInfo);
|
||||
|
||||
// If VectorStore entered degraded mode (e.g. sqlite-vec load failed),
|
||||
// treat it as unavailable and fall back to keyword-only mode.
|
||||
if (vectorStore.isDegraded()) {
|
||||
api.logger.warn(
|
||||
`${TAG} VectorStore is in degraded mode, falling back to keyword dedup`,
|
||||
);
|
||||
vectorStore = undefined;
|
||||
embeddingService = undefined;
|
||||
} else {
|
||||
// If embedding provider/model/dimensions changed, re-embed all existing texts
|
||||
if (stores.needsReindex && embeddingService && vectorStore) {
|
||||
const svc = embeddingService;
|
||||
const vs = vectorStore;
|
||||
api.logger.info(
|
||||
`${TAG} VectorStore initialized: ${dbPath} (${dims}D, provider=${cfg.embedding.provider})`,
|
||||
`${TAG} Embedding config changed (${stores.reindexReason}). ` +
|
||||
`Starting background re-embed of all stored texts...`,
|
||||
);
|
||||
|
||||
// If embedding provider/model/dimensions changed, re-embed all existing texts
|
||||
if (initResult.needsReindex && embeddingService) {
|
||||
const svc = embeddingService; // capture for async closure
|
||||
const vs = vectorStore; // capture for async closure
|
||||
vs.reindexAll(
|
||||
(text) => svc.embed(text),
|
||||
(done, total, layer) => {
|
||||
if (done === total || done % 50 === 0) {
|
||||
api.logger.debug?.(`${TAG} Re-embed progress: ${layer} ${done}/${total}`);
|
||||
}
|
||||
},
|
||||
).then(({ l1Count, l0Count }) => {
|
||||
api.logger.info(
|
||||
`${TAG} Embedding config changed (${initResult.reason}). ` +
|
||||
`Starting background re-embed of all stored texts...`,
|
||||
`${TAG} Re-embed complete: L1=${l1Count} records, L0=${l0Count} messages`,
|
||||
);
|
||||
// Run re-embed asynchronously so it doesn't block plugin startup
|
||||
vs.reindexAll(
|
||||
(text) => svc.embed(text),
|
||||
(done, total, layer) => {
|
||||
if (done === total || done % 50 === 0) {
|
||||
api.logger.debug?.(`${TAG} Re-embed progress: ${layer} ${done}/${total}`);
|
||||
}
|
||||
},
|
||||
).then(({ l1Count, l0Count }) => {
|
||||
api.logger.info(
|
||||
`${TAG} Re-embed complete: L1=${l1Count} records, L0=${l0Count} messages`,
|
||||
);
|
||||
}).catch((err) => {
|
||||
api.logger.error(
|
||||
`${TAG} Re-embed failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
});
|
||||
}
|
||||
}).catch((err) => {
|
||||
api.logger.error(
|
||||
`${TAG} Re-embed failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
api.logger.warn(
|
||||
`${TAG} VectorStore init failed; vector/FTS recall and dedup conflict detection will be unavailable: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
vectorStore = undefined;
|
||||
embeddingService = undefined;
|
||||
}
|
||||
})();
|
||||
|
||||
// Share vectorStore/embeddingService with tdai_memory_search tool
|
||||
sharedVectorStore = vectorStore;
|
||||
sharedEmbeddingService = embeddingService;
|
||||
// === Create pipeline manager (sync — does not need store) ===
|
||||
scheduler = createPipelineManager(cfg, api.logger, sessionFilter);
|
||||
|
||||
// Keep cleaner's SQLite handle updated (singleton cleaner may start earlier).
|
||||
memoryCleaner?.setVectorStore(vectorStore);
|
||||
// Wire runners after store is ready
|
||||
storeReady.then(() => {
|
||||
// L1 runner via shared factory
|
||||
scheduler!.setL1Runner(createL1Runner({
|
||||
pluginDataDir,
|
||||
cfg,
|
||||
openclawConfig: api.config,
|
||||
vectorStore,
|
||||
embeddingService,
|
||||
logger: api.logger,
|
||||
getInstanceId: () => instanceId,
|
||||
}));
|
||||
|
||||
// === Create pipeline manager ===
|
||||
scheduler = new MemoryPipelineManager(
|
||||
{
|
||||
everyNConversations: cfg.pipeline.everyNConversations,
|
||||
enableWarmup: cfg.pipeline.enableWarmup,
|
||||
l1: { idleTimeoutSeconds: cfg.pipeline.l1IdleTimeoutSeconds },
|
||||
l2: {
|
||||
delayAfterL1Seconds: cfg.pipeline.l2DelayAfterL1Seconds,
|
||||
minIntervalSeconds: cfg.pipeline.l2MinIntervalSeconds,
|
||||
maxIntervalSeconds: cfg.pipeline.l2MaxIntervalSeconds,
|
||||
sessionActiveWindowHours: cfg.pipeline.sessionActiveWindowHours,
|
||||
},
|
||||
},
|
||||
api.logger,
|
||||
sessionFilter,
|
||||
);
|
||||
// Persister via shared factory
|
||||
scheduler!.setPersister(createPersister(pluginDataDir, api.logger));
|
||||
|
||||
// L1 runner: read L0 from DB (primary) or JSONL (fallback) → local LLM extraction → L1 JSONL + VectorStore
|
||||
scheduler.setL1Runner(async ({ sessionKey }) => {
|
||||
// L1 reads L0 data from VectorStore DB (primary, indexed query).
|
||||
// Fallback: read from L0 JSONL files when VectorStore is unavailable.
|
||||
if (!api.config) {
|
||||
api.logger.debug?.(`${TAG} [pipeline-l1] No OpenClaw config, skipping L1 extraction`);
|
||||
return { processedCount: 0 };
|
||||
}
|
||||
|
||||
const checkpoint = new CheckpointManager(pluginDataDir, api.logger);
|
||||
const cp = await checkpoint.read();
|
||||
const runnerState = checkpoint.getRunnerState(cp, sessionKey);
|
||||
|
||||
api.logger.info(
|
||||
`${TAG} [pipeline-l1] Session ${sessionKey}: ` +
|
||||
`l1_cursor=${runnerState.last_l1_cursor || "(start)"}`,
|
||||
);
|
||||
|
||||
try {
|
||||
// Read L0 messages since last L1 cursor, grouped by sessionId.
|
||||
// Within the same sessionKey, different sessionIds represent different
|
||||
// conversation instances (e.g. after /reset). Each group is extracted
|
||||
// independently so its sessionId is correctly associated with L1 records.
|
||||
//
|
||||
// Primary path: read from VectorStore DB (indexed query, fast).
|
||||
// Fallback: read from L0 JSONL files (scan + parse, slower).
|
||||
let groups: Array<{ sessionId: string; messages: ConversationMessage[] }>;
|
||||
|
||||
if (vectorStore && !vectorStore.isDegraded()) {
|
||||
// DB path: fast indexed query
|
||||
// NOTE: When last_l1_cursor is 0 (first L1 run), we pass undefined
|
||||
// to query all messages — but only those captured AFTER plugin start
|
||||
// (L0 capture uses pluginStartTimestamp as floor, so DB won't contain
|
||||
// pre-existing messages). This is safe because auto-capture already
|
||||
// filters out messages older than pluginStartTimestamp.
|
||||
const l1Cursor = runnerState.last_l1_cursor > 0
|
||||
? runnerState.last_l1_cursor
|
||||
: undefined;
|
||||
const dbGroups = vectorStore.queryL0GroupedBySessionId(
|
||||
sessionKey,
|
||||
l1Cursor,
|
||||
);
|
||||
// Cast role from string to "user" | "assistant" (DB stores as string)
|
||||
groups = dbGroups.map((g) => ({
|
||||
sessionId: g.sessionId,
|
||||
messages: g.messages.map((m) => ({
|
||||
id: m.id,
|
||||
role: m.role as "user" | "assistant",
|
||||
content: m.content,
|
||||
timestamp: m.timestamp,
|
||||
})),
|
||||
}));
|
||||
api.logger.debug?.(
|
||||
`${TAG} [pipeline-l1] L0 data source: VectorStore DB`,
|
||||
);
|
||||
} else {
|
||||
// Fallback: JSONL files
|
||||
api.logger.debug?.(
|
||||
`${TAG} [pipeline-l1] L0 data source: JSONL files (VectorStore unavailable)`,
|
||||
);
|
||||
const jsonlGroups = await readConversationMessagesGroupedBySessionId(
|
||||
sessionKey,
|
||||
// L2 runner: read L1 records (incremental) → SceneExtractor
|
||||
scheduler!.setL2Runner(async (sessionKey: string, cursor?: string) => {
|
||||
try {
|
||||
const l2Runner = createL2Runner({
|
||||
pluginDataDir,
|
||||
runnerState.last_l1_cursor || undefined,
|
||||
api.logger,
|
||||
50, // Match DB path limit (queryL0ForL1 default)
|
||||
);
|
||||
// Convert SessionIdMessageGroup[] to the same shape
|
||||
groups = jsonlGroups.map((g) => ({
|
||||
sessionId: g.sessionId,
|
||||
messages: g.messages,
|
||||
}));
|
||||
}
|
||||
|
||||
if (groups.length === 0) {
|
||||
api.logger.debug?.(`${TAG} [pipeline-l1] No new L0 messages for session ${sessionKey}`);
|
||||
return { processedCount: 0 };
|
||||
}
|
||||
|
||||
const totalMessages = groups.reduce((sum, g) => sum + g.messages.length, 0);
|
||||
api.logger.info(
|
||||
`${TAG} [pipeline-l1] Processing ${totalMessages} L0 messages across ${groups.length} sessionId group(s) for session ${sessionKey}`,
|
||||
);
|
||||
|
||||
let totalExtracted = 0;
|
||||
let totalStored = 0;
|
||||
let lastSceneName: string | undefined;
|
||||
let maxTimestamp = 0;
|
||||
|
||||
for (const group of groups) {
|
||||
api.logger.debug?.(
|
||||
`${TAG} [pipeline-l1] Group sessionId=${group.sessionId || "(empty)"}: ${group.messages.length} messages`,
|
||||
);
|
||||
|
||||
const l1Result = await extractL1Memories({
|
||||
messages: group.messages,
|
||||
sessionKey,
|
||||
sessionId: group.sessionId,
|
||||
baseDir: pluginDataDir,
|
||||
config: api.config,
|
||||
options: {
|
||||
enableDedup: cfg.extraction.enableDedup,
|
||||
maxMemoriesPerSession: cfg.extraction.maxMemoriesPerSession,
|
||||
model: cfg.extraction.model,
|
||||
previousSceneName: lastSceneName ?? (runnerState.last_scene_name || undefined),
|
||||
vectorStore,
|
||||
embeddingService,
|
||||
conflictRecallTopK: cfg.embedding.conflictRecallTopK,
|
||||
},
|
||||
cfg,
|
||||
openclawConfig: api.config,
|
||||
vectorStore,
|
||||
logger: api.logger,
|
||||
instanceId,
|
||||
});
|
||||
|
||||
totalExtracted += l1Result.extractedCount;
|
||||
totalStored += l1Result.storedCount;
|
||||
if (l1Result.lastSceneName) {
|
||||
lastSceneName = l1Result.lastSceneName;
|
||||
}
|
||||
|
||||
const groupMaxTs = Math.max(...group.messages.map((m) => m.timestamp));
|
||||
maxTimestamp = Math.max(maxTimestamp, groupMaxTs);
|
||||
return await l2Runner(sessionKey, cursor);
|
||||
} catch (err) {
|
||||
api.logger.error(`${TAG} [pipeline-l2] L2 failed: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
|
||||
throw err;
|
||||
}
|
||||
});
|
||||
|
||||
// Update checkpoint on disk — cursor is the global max timestamp across all groups
|
||||
await checkpoint.markL1ExtractionComplete(
|
||||
sessionKey,
|
||||
totalStored,
|
||||
maxTimestamp,
|
||||
lastSceneName,
|
||||
);
|
||||
|
||||
api.logger.info(
|
||||
`${TAG} [pipeline-l1] L1 complete: extracted=${totalExtracted}, stored=${totalStored} (${groups.length} group(s))`,
|
||||
);
|
||||
|
||||
return { processedCount: totalMessages };
|
||||
} catch (err) {
|
||||
api.logger.error(`${TAG} [pipeline-l1] L1 failed: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
|
||||
// ── error_degradation metric ──
|
||||
if (instanceId) {
|
||||
report("error_degradation", {
|
||||
module: "l1-extraction",
|
||||
action: "extractL1Memories",
|
||||
errorType: "exception",
|
||||
errorMessage: err instanceof Error ? err.message : String(err),
|
||||
degradedTo: null,
|
||||
impact: "blocking",
|
||||
// L3 runner: persona trigger + generation
|
||||
scheduler!.setL3Runner(async () => {
|
||||
try {
|
||||
const l3Runner = createL3Runner({
|
||||
pluginDataDir,
|
||||
cfg,
|
||||
openclawConfig: api.config,
|
||||
vectorStore,
|
||||
logger: api.logger,
|
||||
instanceId,
|
||||
});
|
||||
await l3Runner();
|
||||
} catch (err) {
|
||||
api.logger.error(`${TAG} [pipeline-l3] Failed: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
|
||||
}
|
||||
throw err; // rethrow so pipeline-manager can retry
|
||||
}
|
||||
});
|
||||
}).catch((err) => {
|
||||
api.logger.error(
|
||||
`${TAG} Store init failed; vector/FTS recall and dedup will be unavailable: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
});
|
||||
|
||||
// Persister: saves pipeline session states to checkpoint
|
||||
scheduler.setPersister(async (states) => {
|
||||
const checkpoint = new CheckpointManager(pluginDataDir, api.logger);
|
||||
await checkpoint.mergePipelineStates(states);
|
||||
});
|
||||
|
||||
// L2 runner: read L1 records (incremental) → SceneExtractor
|
||||
scheduler.setL2Runner(async (sessionKey: string, cursor?: string) => {
|
||||
try {
|
||||
return await runLocalL2Extraction(api, cfg, pluginDataDir, sessionKey, vectorStore, cursor, instanceId);
|
||||
} catch (err) {
|
||||
api.logger.error(`${TAG} [pipeline-l2] L2 failed: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
|
||||
throw err; // rethrow so pipeline-manager can handle retry/fallback (consistent with L1 runner)
|
||||
}
|
||||
});
|
||||
|
||||
// L3 runner: persona trigger + generation
|
||||
scheduler.setL3Runner(async () => {
|
||||
try {
|
||||
const trigger = new PersonaTrigger({
|
||||
dataDir: pluginDataDir,
|
||||
interval: cfg.persona.triggerEveryN,
|
||||
logger: api.logger,
|
||||
});
|
||||
|
||||
const { should, reason } = await trigger.shouldGenerate();
|
||||
if (!should) {
|
||||
api.logger.debug?.(`${TAG} [pipeline-l3] Persona generation not needed`);
|
||||
return;
|
||||
}
|
||||
|
||||
if (!api.config) {
|
||||
api.logger.warn(`${TAG} [pipeline-l3] No OpenClaw config, skipping persona generation`);
|
||||
return;
|
||||
}
|
||||
|
||||
api.logger.info(`${TAG} [pipeline-l3] Starting persona generation: ${reason}`);
|
||||
const generator = new PersonaGenerator({
|
||||
dataDir: pluginDataDir,
|
||||
config: api.config,
|
||||
model: cfg.persona.model,
|
||||
backupCount: cfg.persona.backupCount,
|
||||
logger: api.logger,
|
||||
instanceId,
|
||||
});
|
||||
const genResult = await generator.generate(reason);
|
||||
api.logger.info(`${TAG} [pipeline-l3] Persona generation ${genResult ? "succeeded" : "skipped (no changes)"}`);
|
||||
} catch (err) {
|
||||
api.logger.error(`${TAG} [pipeline-l3] Failed: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
|
||||
}
|
||||
});
|
||||
|
||||
// Capture vectorStore reference for cleanup
|
||||
const vectorStoreRef = vectorStore;
|
||||
|
||||
// Register a SINGLE gateway_stop hook for ordered shutdown.
|
||||
// Order: memoryCleaner → scheduler → vectorStore → embeddingService
|
||||
// Order: memoryCleaner → scheduler → vectorStore → embeddingService → resetStores
|
||||
// (memoryCleaner may use VectorStore during cleanup, so it must stop first)
|
||||
//
|
||||
// The entire hook is wrapped with a 3 s timeout to guarantee we never
|
||||
@@ -965,8 +746,11 @@ export default function register(api: OpenClawPluginApi) {
|
||||
const GATEWAY_STOP_TIMEOUT_MS = 3_000;
|
||||
const hookStartMs = Date.now();
|
||||
|
||||
// Ensure store init has completed before tearing down
|
||||
await storeReady.catch(() => {});
|
||||
|
||||
const doCleanup = async (): Promise<void> => {
|
||||
// 1. Stop the memory cleaner
|
||||
// 1. Stop the memory cleaner first (it may be running deleteL1ExpiredByUpdatedTime)
|
||||
if (memoryCleaner) {
|
||||
try {
|
||||
memoryCleaner.destroy();
|
||||
@@ -983,16 +767,20 @@ export default function register(api: OpenClawPluginApi) {
|
||||
const t = Date.now();
|
||||
await scheduler.destroy();
|
||||
api.logger.info(`${TAG} [gateway_stop] Scheduler destroyed (${Date.now() - t}ms)`);
|
||||
} else {
|
||||
api.logger.info(`${TAG} [gateway_stop] Scheduler was never started, skipping destroy`);
|
||||
}
|
||||
|
||||
// 3. Close VectorStore
|
||||
if (vectorStoreRef) {
|
||||
vectorStoreRef.close();
|
||||
// 3. Close VectorStore last (after all consumers are done)
|
||||
if (vectorStore) {
|
||||
api.logger.info(`${TAG} [gateway_stop] Closing VectorStore`);
|
||||
vectorStore.close();
|
||||
}
|
||||
|
||||
// 4. Release embedding service resources
|
||||
// 4. Release embedding service resources (model memory, GPU, etc.)
|
||||
if (embeddingService?.close) {
|
||||
try {
|
||||
api.logger.info(`${TAG} [gateway_stop] Closing EmbeddingService`);
|
||||
await embeddingService.close();
|
||||
} catch (err) {
|
||||
api.logger.warn(`${TAG} [gateway_stop] EmbeddingService close error: ${err instanceof Error ? err.message : String(err)}`);
|
||||
@@ -1021,11 +809,14 @@ export default function register(api: OpenClawPluginApi) {
|
||||
if (timeoutId !== undefined) clearTimeout(timeoutId);
|
||||
}
|
||||
|
||||
// 5. Reset store singleton cache so hot-restart can re-initialize
|
||||
resetStores();
|
||||
|
||||
api.logger.info(`${TAG} [gateway_stop] Cleanup finished, all resources released (${Date.now() - hookStartMs}ms)`);
|
||||
});
|
||||
}
|
||||
|
||||
api.logger.info(`${TAG} Registering agent_end hook (auto-capture)`);
|
||||
api.logger.debug?.(`${TAG} Registering agent_end hook (auto-capture)`);
|
||||
api.on("agent_end", async (event, ctx) => {
|
||||
const startMs = Date.now();
|
||||
api.logger.debug?.(`${TAG} [agent_end] Hook triggered`);
|
||||
@@ -1139,7 +930,7 @@ export default function register(api: OpenClawPluginApi) {
|
||||
}
|
||||
});
|
||||
} else {
|
||||
api.logger.info(`${TAG} Auto-capture disabled`);
|
||||
api.logger.debug?.(`${TAG} Auto-capture disabled`);
|
||||
}
|
||||
|
||||
// memoryCleaner gateway_stop is handled in the unified handler above (inside extraction.enabled block).
|
||||
@@ -1159,177 +950,30 @@ export default function register(api: OpenClawPluginApi) {
|
||||
});
|
||||
}
|
||||
|
||||
api.logger.info(
|
||||
// ============================
|
||||
// CLI registration
|
||||
// ============================
|
||||
|
||||
api.registerCli(
|
||||
({ program, config, logger: cliLogger }) => {
|
||||
const memoryTdai = program
|
||||
.command("memory-tdai")
|
||||
.description("memory-tdai plugin commands (seed, query, stats)");
|
||||
|
||||
registerMemoryTdaiCli(memoryTdai, {
|
||||
config,
|
||||
pluginConfig: api.pluginConfig,
|
||||
stateDir: api.runtime.state.resolveStateDir(),
|
||||
logger: cliLogger,
|
||||
});
|
||||
},
|
||||
{ commands: ["memory-tdai"] },
|
||||
);
|
||||
|
||||
api.logger.debug?.(
|
||||
`${TAG} Plugin registration complete (v3). ` +
|
||||
`startTimestamp=${pluginStartTimestamp} (${new Date(pluginStartTimestamp).toISOString()})`,
|
||||
);
|
||||
|
||||
// Resolve persistent instance ID for metric reporting (used by agent_turn, error_degradation, etc.)
|
||||
getOrCreateInstanceId(pluginDataDir).then((id) => {
|
||||
instanceId = id;
|
||||
initReporter({ enabled: cfg.report.enabled, type: cfg.report.type, logger: api.logger, instanceId: id, pluginVersion });
|
||||
}).catch((err) => {
|
||||
api.logger.warn(`${TAG} Failed to initialize instanceId for metrics: ${err instanceof Error ? err.message : String(err)}`);
|
||||
});
|
||||
}
|
||||
|
||||
// ============================
|
||||
// L2 extraction implementations
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Local L2 extraction: read L1 records → SceneExtractor.
|
||||
*
|
||||
* Uses **incremental** reads when VectorStore is available:
|
||||
* 1. Receive the pipeline cursor (`last_extraction_updated_time`) from pipeline-manager
|
||||
* 2. Query only L1 records updated AFTER that cursor via `queryMemoryRecords`
|
||||
* 3. Return the latest `updatedAt` from the batch so pipeline-manager can advance the cursor
|
||||
*
|
||||
* Falls back to JSONL read (with client-side time filtering) when VectorStore is unavailable.
|
||||
*/
|
||||
async function runLocalL2Extraction(
|
||||
api: OpenClawPluginApi,
|
||||
cfg: MemoryTdaiConfig,
|
||||
pluginDataDir: string,
|
||||
sessionKey: string,
|
||||
vectorStore?: VectorStore,
|
||||
updatedAfter?: string,
|
||||
instanceId?: string,
|
||||
): Promise<{ latestCursor?: string } | void> {
|
||||
api.logger.debug?.(
|
||||
`${TAG} [L2-local] session=${sessionKey}, updatedAfter=${updatedAfter ?? "(full)"}`,
|
||||
);
|
||||
|
||||
let records: Array<{ content: string; created_at: string; id: string; updatedAt: string }>;
|
||||
|
||||
// Prefer incremental SQLite query when VectorStore is available
|
||||
if (vectorStore && !vectorStore.isDegraded()) {
|
||||
const { queryMemoryRecords } = await import("./src/record/l1-reader.js");
|
||||
const memRecords = queryMemoryRecords(vectorStore, {
|
||||
sessionKey,
|
||||
updatedAfter,
|
||||
}, api.logger);
|
||||
|
||||
if (memRecords.length === 0) {
|
||||
api.logger.debug?.(
|
||||
`${TAG} [L2-local] No new L1 records since cursor (session=${sessionKey}, updatedAfter=${updatedAfter ?? "(full)"}), skipping scene extraction`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
api.logger.debug?.(
|
||||
`${TAG} [L2-local] Incremental query returned ${memRecords.length} record(s) (session=${sessionKey})`,
|
||||
);
|
||||
|
||||
records = memRecords.map((r) => ({
|
||||
content: r.content,
|
||||
created_at: r.createdAt,
|
||||
id: r.id,
|
||||
updatedAt: r.updatedAt,
|
||||
}));
|
||||
} else {
|
||||
// Fallback: read JSONL files with client-side time filtering
|
||||
api.logger.debug?.(`${TAG} [L2-local] VectorStore unavailable, falling back to JSONL read`);
|
||||
const { readAllMemoryRecords } = await import("./src/record/l1-reader.js");
|
||||
let allRecords = await readAllMemoryRecords(pluginDataDir, api.logger);
|
||||
|
||||
// Apply updatedAfter filter on JSONL records (same semantics as SQLite path)
|
||||
if (updatedAfter) {
|
||||
const beforeCount = allRecords.length;
|
||||
allRecords = allRecords.filter((r) => {
|
||||
const t = r.updatedAt || r.createdAt || "";
|
||||
return t > updatedAfter;
|
||||
});
|
||||
api.logger.debug?.(
|
||||
`${TAG} [L2-local] JSONL time filter: ${beforeCount} → ${allRecords.length} record(s) (updatedAfter=${updatedAfter})`,
|
||||
);
|
||||
}
|
||||
|
||||
if (allRecords.length === 0) {
|
||||
api.logger.debug?.(`${TAG} [L2-local] No new L1 records found (JSONL fallback), skipping scene extraction`);
|
||||
return;
|
||||
}
|
||||
|
||||
records = allRecords.map((r) => ({
|
||||
content: r.content,
|
||||
created_at: r.createdAt,
|
||||
id: r.id,
|
||||
updatedAt: r.updatedAt,
|
||||
}));
|
||||
}
|
||||
|
||||
const extractor = new SceneExtractor({
|
||||
dataDir: pluginDataDir,
|
||||
config: api.config!,
|
||||
model: cfg.persona.model,
|
||||
maxScenes: cfg.persona.maxScenes,
|
||||
sceneBackupCount: cfg.persona.sceneBackupCount,
|
||||
logger: api.logger,
|
||||
instanceId,
|
||||
});
|
||||
|
||||
const memories = records.map((r) => ({
|
||||
content: r.content,
|
||||
created_at: r.created_at,
|
||||
id: r.id,
|
||||
}));
|
||||
|
||||
// ── Checkpoint guard ──────────────────────────────────────────────
|
||||
// Snapshot critical counters BEFORE the LLM agent runs.
|
||||
// The LLM operates on checkpoint via raw file tools (write_to_file /
|
||||
// replace_in_file) which bypass CheckpointManager's file lock.
|
||||
// If the LLM accidentally overwrites the entire checkpoint (e.g. via
|
||||
// write_to_file), system-managed counters like scenes_processed and
|
||||
// memories_since_last_persona can be reset to stale values.
|
||||
// After extraction we detect and repair such corruption.
|
||||
const preCheckpoint = new CheckpointManager(pluginDataDir, api.logger);
|
||||
const preState = await preCheckpoint.read();
|
||||
const preScenesProcessed = preState.scenes_processed;
|
||||
const preMemoriesSince = preState.memories_since_last_persona;
|
||||
const preTotalProcessed = preState.total_processed;
|
||||
|
||||
const extractResult = await extractor.extract(memories);
|
||||
if (extractResult.success && extractResult.memoriesProcessed > 0) {
|
||||
const checkpoint = new CheckpointManager(pluginDataDir, api.logger);
|
||||
|
||||
// Detect and repair LLM-caused checkpoint corruption.
|
||||
// If the LLM wrote the entire checkpoint file (instead of using
|
||||
// replace_in_file on specific fields), system-managed counters may
|
||||
// have been overwritten with stale/zero values.
|
||||
const postState = await checkpoint.read();
|
||||
if (
|
||||
postState.scenes_processed < preScenesProcessed ||
|
||||
postState.total_processed < preTotalProcessed
|
||||
) {
|
||||
api.logger.warn(
|
||||
`${TAG} [L2-local] ⚠️ Checkpoint corruption detected! ` +
|
||||
`scenes_processed: ${preScenesProcessed} → ${postState.scenes_processed}, ` +
|
||||
`total_processed: ${preTotalProcessed} → ${postState.total_processed}, ` +
|
||||
`memories_since: ${preMemoriesSince} → ${postState.memories_since_last_persona}. ` +
|
||||
`Repairing...`,
|
||||
);
|
||||
await checkpoint.write({
|
||||
...postState,
|
||||
scenes_processed: Math.max(postState.scenes_processed, preScenesProcessed),
|
||||
total_processed: Math.max(postState.total_processed, preTotalProcessed),
|
||||
memories_since_last_persona: Math.max(postState.memories_since_last_persona, preMemoriesSince),
|
||||
});
|
||||
api.logger.info(`${TAG} [L2-local] Checkpoint repaired`);
|
||||
}
|
||||
|
||||
await checkpoint.incrementScenesProcessed();
|
||||
|
||||
// Return the max updatedAt from this batch as the new cursor
|
||||
const latestCursor = records.reduce((latest, r) => {
|
||||
return r.updatedAt > latest ? r.updatedAt : latest;
|
||||
}, "");
|
||||
|
||||
api.logger.debug?.(
|
||||
`${TAG} [L2-local] Extraction complete: processed=${extractResult.memoriesProcessed}, latestCursor=${latestCursor}`,
|
||||
);
|
||||
|
||||
return { latestCursor: latestCursor || undefined };
|
||||
}
|
||||
}
|
||||
|
||||
// ============================
|
||||
|
||||
+32
-3
@@ -6,6 +6,12 @@
|
||||
"type": "object",
|
||||
"additionalProperties": true,
|
||||
"properties": {
|
||||
"storeBackend": {
|
||||
"type": "string",
|
||||
"enum": ["sqlite", "tcvdb"],
|
||||
"default": "sqlite",
|
||||
"description": "存储后端：sqlite（本地 SQLite + sqlite-vec）或 tcvdb（腾讯云向量数据库）"
|
||||
},
|
||||
"capture": {
|
||||
"type": "object",
|
||||
"description": "对话自动捕获设置 (L0)",
|
||||
@@ -32,7 +38,7 @@
|
||||
"description": "场景归纳与用户画像设置 (L2/L3)",
|
||||
"properties": {
|
||||
"triggerEveryN": { "type": "number", "default": 50, "description": "每 N 条新记忆触发画像生成" },
|
||||
"maxScenes": { "type": "number", "default": 20, "description": "最大场景数" },
|
||||
"maxScenes": { "type": "number", "default": 15, "description": "最大场景数" },
|
||||
"backupCount": { "type": "number", "default": 3, "description": "画像备份保留数量" },
|
||||
"sceneBackupCount": { "type": "number", "default": 10, "description": "场景块备份保留数量" },
|
||||
"model": { "type": "string", "description": "画像生成模型 (格式: provider/model)，未填写时使用openclaw默认模型" }
|
||||
@@ -66,7 +72,7 @@
|
||||
"type": "object",
|
||||
"description": "向量搜索 (Embedding) 配置",
|
||||
"properties": {
|
||||
"enabled": { "type": "boolean", "default": true, "description": "是否启用向量搜索" },
|
||||
"enabled": { "type": "boolean", "default": true, "description": "是否启用向量搜索（若 provider=none，则实际会被禁用）" },
|
||||
"provider": { "type": "string", "default": "none", "description": "Embedding 服务提供者：填写兼容 OpenAI API 的远端服务名称（如 openai、deepseek 等）；不填或填 none 则禁用向量搜索" },
|
||||
"proxyUrl": { "type": "string", "description": "本地代理地址（仅 provider=qclaw 时必填）。配置后 embedding 请求将通过该代理转发，原始 baseUrl 作为 Remote-URL 头传递" },
|
||||
"baseUrl": { "type": "string", "description": "API Base URL（必填）：填写对应 provider 的 API 地址" },
|
||||
@@ -74,7 +80,30 @@
|
||||
"model": { "type": "string", "description": "模型名称（必填）" },
|
||||
"dimensions": { "type": "number", "description": "向量维度（必填，需与所选模型匹配）" },
|
||||
"conflictRecallTopK": { "type": "number", "default": 5, "description": "冲突检测时召回 Top-K 数" },
|
||||
"maxInputChars": { "type": "number", "default": 5000, "description": "Embedding 输入文本最大字符数，超出时截断并打印警告日志（默认 5000，适合大多数模型的 token 上限）" }
|
||||
"maxInputChars": { "type": "number", "default": 5000, "description": "Embedding 输入文本最大字符数，超出时截断并打印警告日志（默认 5000，适合大多数模型的 token 上限）" },
|
||||
"timeoutMs": { "type": "number", "default": 10000, "description": "单次 embedding API 调用超时（毫秒），超时后该次请求中止且不重试" }
|
||||
}
|
||||
},
|
||||
"tcvdb": {
|
||||
"type": "object",
|
||||
"description": "腾讯云向量数据库配置（仅 storeBackend=tcvdb 时生效）",
|
||||
"properties": {
|
||||
"url": { "type": "string", "description": "实例 URL（必填，如 http://10.0.1.1:8100）" },
|
||||
"username": { "type": "string", "default": "root", "description": "账户名" },
|
||||
"apiKey": { "type": "string", "description": "API Key（必填）" },
|
||||
"database": { "type": "string", "description": "数据库名（未填写时默认根据实例 ID 自动生成）" },
|
||||
"alias": { "type": "string", "description": "用户友好别名（可选，用于 database.json 中识别）" },
|
||||
"embeddingModel": { "type": "string", "default": "bge-large-zh", "description": "服务端 embedding 模型" },
|
||||
"timeout": { "type": "number", "default": 10000, "description": "请求超时（毫秒）" },
|
||||
"caPemPath": { "type": "string", "description": "CA 证书 PEM 文件路径（HTTPS 连接时使用）" }
|
||||
}
|
||||
},
|
||||
"bm25": {
|
||||
"type": "object",
|
||||
"description": "BM25 稀疏向量编码设置（主要用于 tcvdb 后端）",
|
||||
"properties": {
|
||||
"enabled": { "type": "boolean", "default": true, "description": "是否启用 BM25 稀疏向量编码" },
|
||||
"language": { "type": "string", "enum": ["zh", "en"], "default": "zh", "description": "分词语言：zh（中文）或 en（英文）" }
|
||||
}
|
||||
},
|
||||
"report": {
|
||||
|
||||
+38
-5
@@ -1,14 +1,33 @@
|
||||
{
|
||||
"name": "@tencentdb-agent-memory/memory-tencentdb",
|
||||
"version": "0.1.4",
|
||||
"version": "0.2.2",
|
||||
"description": "Four-layer local memory system plugin for OpenClaw — auto-captures, structures, and profiles conversational knowledge using local LLM + SQLite vector search (L0→L1→L2→L3 pipeline)",
|
||||
"type": "module",
|
||||
"main": "index.ts",
|
||||
"bin": {
|
||||
"migrate-sqlite-to-tcvdb": "./bin/migrate-sqlite-to-tcvdb.mjs",
|
||||
"export-tencent-vdb": "./bin/export-tencent-vdb.mjs",
|
||||
"read-local-memory": "./bin/read-local-memory.mjs"
|
||||
},
|
||||
"exports": {
|
||||
".": "./index.ts"
|
||||
},
|
||||
"scripts": {
|
||||
"build:scripts": "npm run build:migrate-sqlite-to-vdb && npm run build:export-tencent-vdb && npm run build:read-local-memory",
|
||||
"prepack": "npm run build:scripts",
|
||||
"build:migrate-sqlite-to-vdb": "tsc -p scripts/migrate-sqlite-to-tcvdb/tsconfig.json --noEmitOnError false",
|
||||
"migrate-sqlite-to-tcvdb": "node ./bin/migrate-sqlite-to-tcvdb.mjs",
|
||||
"build:export-tencent-vdb": "tsc --project scripts/export-tencent-vdb/tsconfig.json",
|
||||
"export-tencent-vdb": "node ./bin/export-tencent-vdb.mjs",
|
||||
"build:read-local-memory": "tsc --project scripts/read-local-memory/tsconfig.json",
|
||||
"read-local-memory": "node ./bin/read-local-memory.mjs"
|
||||
},
|
||||
"files": [
|
||||
"bin/",
|
||||
"index.ts",
|
||||
"scripts/migrate-sqlite-to-tcvdb/dist/",
|
||||
"scripts/export-tencent-vdb/dist/",
|
||||
"scripts/read-local-memory/dist/",
|
||||
"src/",
|
||||
"openclaw.plugin.json",
|
||||
"README.md",
|
||||
@@ -39,11 +58,14 @@
|
||||
},
|
||||
"dependencies": {
|
||||
"@node-rs/jieba": "^2.0.1",
|
||||
"sqlite-vec": "0.1.7-alpha.2"
|
||||
"@tencentdb-agent-memory/tcvdb-text": "^0.1.1",
|
||||
"json5": "^2.2.3",
|
||||
"sqlite-vec": "0.1.7-alpha.2",
|
||||
"undici": "8.0.2"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"openclaw": ">=2026.3.7",
|
||||
"node-llama-cpp": "^3.16.2"
|
||||
"node-llama-cpp": "^3.16.2",
|
||||
"openclaw": ">=2026.3.7"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"openclaw": {
|
||||
@@ -56,6 +78,17 @@
|
||||
"openclaw": {
|
||||
"extensions": [
|
||||
"./index.ts"
|
||||
]
|
||||
],
|
||||
"compat": {
|
||||
"pluginApi": ">=2026.3.13",
|
||||
"minGatewayVersion": ">=2026.3.13"
|
||||
},
|
||||
"build": {
|
||||
"openclawVersion": "2026.3.13",
|
||||
"pluginSdkVersion": "2026.3.13"
|
||||
},
|
||||
"bundle": {
|
||||
"stageRuntimeDependencies": true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Executable
+227
@@ -0,0 +1,227 @@
|
||||
#!/usr/bin/env bash
|
||||
# OpenClaw + memory-tencentdb（原 memory-tdai）诊断数据导出脚本
|
||||
# 注：插件已更名为 memory-tencentdb，但数据目录始终为 memory-tdai（代码硬编码）
|
||||
# 用法: bash export-diagnostic.sh [输出目录]
|
||||
# 默认输出到 ~/Downloads/openclaw-diagnostic-<timestamp>/
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# ── 参数 ──
|
||||
OUTPUT_BASE="${1:-$HOME/Downloads}"
|
||||
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
|
||||
EXPORT_DIR="${OUTPUT_BASE}/openclaw-diagnostic-${TIMESTAMP}"
|
||||
ARCHIVE_PATH="${EXPORT_DIR}.tar.gz"
|
||||
|
||||
# ── OpenClaw 工作目录探测 ──
|
||||
if [ -n "${OPENCLAW_STATE_DIR:-}" ]; then
|
||||
STATE_DIR="$OPENCLAW_STATE_DIR"
|
||||
elif [ -d "$HOME/.openclaw" ]; then
|
||||
STATE_DIR="$HOME/.openclaw"
|
||||
elif [ -d "$HOME/.clawdbot" ]; then
|
||||
STATE_DIR="$HOME/.clawdbot"
|
||||
else
|
||||
echo "❌ 未找到 OpenClaw 工作目录 (~/.openclaw 或 ~/.clawdbot)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "📂 OpenClaw 工作目录: $STATE_DIR"
|
||||
echo "📦 导出目录: $EXPORT_DIR"
|
||||
|
||||
mkdir -p "$EXPORT_DIR"
|
||||
|
||||
# ── 1. 收集环境信息 ──
|
||||
echo "🔍 收集环境信息..."
|
||||
{
|
||||
echo "=== 导出时间 ==="
|
||||
date -Iseconds 2>/dev/null || date
|
||||
echo ""
|
||||
echo "=== 系统信息 ==="
|
||||
echo "OS: $(uname -a)"
|
||||
echo "Node: $(node --version 2>/dev/null || echo 'not found')"
|
||||
echo "pnpm: $(pnpm --version 2>/dev/null || echo 'not found')"
|
||||
echo ""
|
||||
echo "=== OpenClaw 版本 ==="
|
||||
openclaw --version 2>/dev/null || pnpm openclaw --version 2>/dev/null || echo "(unknown)"
|
||||
echo ""
|
||||
echo "=== 工作目录 ==="
|
||||
echo "STATE_DIR: $STATE_DIR"
|
||||
echo ""
|
||||
echo "=== 目录结构 ==="
|
||||
ls -la "$STATE_DIR/" 2>/dev/null || echo "(empty)"
|
||||
echo ""
|
||||
echo "=== memory-tdai 目录结构 ==="
|
||||
ls -laR "$STATE_DIR/memory-tdai/" 2>/dev/null || echo "(not found)"
|
||||
echo ""
|
||||
echo "=== 磁盘占用 ==="
|
||||
du -sh "$STATE_DIR/memory-tdai/"* 2>/dev/null || echo "(not found)"
|
||||
} > "$EXPORT_DIR/env-info.txt" 2>&1
|
||||
|
||||
# ── 2. 收集 OpenClaw 日志 ──
|
||||
echo "📋 收集 OpenClaw 日志..."
|
||||
mkdir -p "$EXPORT_DIR/logs"
|
||||
|
||||
# 网关日志 (~/.openclaw/logs/)
|
||||
if [ -d "$STATE_DIR/logs" ]; then
|
||||
cp -r "$STATE_DIR/logs/" "$EXPORT_DIR/logs/gateway-logs/" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# 滚动日志 (/tmp/openclaw/)
|
||||
TMP_LOG_DIR="/tmp/openclaw"
|
||||
if [ -d "$TMP_LOG_DIR" ]; then
|
||||
mkdir -p "$EXPORT_DIR/logs/rolling-logs"
|
||||
# 只取最近 3 个日志文件
|
||||
ls -t "$TMP_LOG_DIR"/openclaw-*.log 2>/dev/null | head -3 | while read -r f; do
|
||||
# 每个文件只取最后 5000 行，避免过大
|
||||
tail -5000 "$f" > "$EXPORT_DIR/logs/rolling-logs/$(basename "$f")" 2>/dev/null || true
|
||||
done
|
||||
fi
|
||||
|
||||
# ── 3. 收集记忆插件数据 ──
|
||||
# 注：数据目录名为 memory-tdai（历史原因，插件更名为 memory-tencentdb 后未改目录名）
|
||||
echo "🧠 收集记忆插件数据..."
|
||||
MEMORY_DIR="$STATE_DIR/memory-tdai"
|
||||
if [ -d "$MEMORY_DIR" ]; then
|
||||
mkdir -p "$EXPORT_DIR/memory-tdai"
|
||||
|
||||
# L0 对话记录 (JSONL)
|
||||
if [ -d "$MEMORY_DIR/conversations" ]; then
|
||||
cp -r "$MEMORY_DIR/conversations/" "$EXPORT_DIR/memory-tdai/conversations/" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# L1 结构化记忆 (JSONL)
|
||||
if [ -d "$MEMORY_DIR/records" ]; then
|
||||
cp -r "$MEMORY_DIR/records/" "$EXPORT_DIR/memory-tdai/records/" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# L2 场景文件 (Markdown)
|
||||
if [ -d "$MEMORY_DIR/scene_blocks" ]; then
|
||||
cp -r "$MEMORY_DIR/scene_blocks/" "$EXPORT_DIR/memory-tdai/scene_blocks/" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# L3 用户画像
|
||||
[ -f "$MEMORY_DIR/persona.md" ] && cp "$MEMORY_DIR/persona.md" "$EXPORT_DIR/memory-tdai/" 2>/dev/null || true
|
||||
|
||||
# checkpoint + scene_index
|
||||
if [ -d "$MEMORY_DIR/.metadata" ]; then
|
||||
cp -r "$MEMORY_DIR/.metadata/" "$EXPORT_DIR/memory-tdai/.metadata/" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# SQLite 数据库（用于检查向量/FTS索引状态）
|
||||
[ -f "$MEMORY_DIR/vectors.db" ] && cp "$MEMORY_DIR/vectors.db" "$EXPORT_DIR/memory-tdai/" 2>/dev/null || true
|
||||
|
||||
# 备份目录（可选，可能较大）
|
||||
if [ -d "$MEMORY_DIR/.backup" ]; then
|
||||
cp -r "$MEMORY_DIR/.backup/" "$EXPORT_DIR/memory-tdai/.backup/" 2>/dev/null || true
|
||||
fi
|
||||
else
|
||||
echo " ⚠️ 未找到 memory-tdai 数据目录（memory-tencentdb 插件的数据也存储在此目录）"
|
||||
fi
|
||||
|
||||
# ── 4. 收集 OpenClaw 配置（脱敏） ──
|
||||
echo "🔧 收集 OpenClaw 配置（已脱敏）..."
|
||||
CONFIG_FILE="$STATE_DIR/openclaw.json"
|
||||
if [ -f "$CONFIG_FILE" ]; then
|
||||
# 使用 node 脱敏处理配置
|
||||
node -e "
|
||||
const fs = require('fs');
|
||||
const JSON5 = (() => { try { return require('json5'); } catch { return JSON; } })();
|
||||
const raw = fs.readFileSync('$CONFIG_FILE', 'utf-8');
|
||||
let cfg;
|
||||
try { cfg = JSON5.parse(raw); } catch { cfg = JSON.parse(raw); }
|
||||
|
||||
// 递归脱敏函数
|
||||
function redact(obj, path) {
|
||||
if (!obj || typeof obj !== 'object') return obj;
|
||||
if (Array.isArray(obj)) return obj.map((v, i) => redact(v, path + '[' + i + ']'));
|
||||
const result = {};
|
||||
for (const [k, v] of Object.entries(obj)) {
|
||||
const fullPath = path ? path + '.' + k : k;
|
||||
// 脱敏规则：API key、token、password、secret 类字段
|
||||
if (/api_?key|token|password|secret|credential/i.test(k) && typeof v === 'string') {
|
||||
result[k] = v.length > 0 ? '***REDACTED(' + v.length + 'chars)***' : '';
|
||||
}
|
||||
// 脱敏 SecretRef 对象
|
||||
else if (v && typeof v === 'object' && v.source && v.id && v.provider) {
|
||||
result[k] = { source: v.source, provider: v.provider, id: '***REDACTED***' };
|
||||
}
|
||||
// 整体跳过的顶层敏感块
|
||||
else if (['models', 'secrets', 'channels', 'env'].includes(k) && !path) {
|
||||
result[k] = '***REDACTED_SECTION(use openclaw config get ' + k + ' to inspect)***';
|
||||
}
|
||||
// gateway.auth 内的 token/password
|
||||
else if (path === 'gateway.auth' && /token|password/i.test(k)) {
|
||||
result[k] = typeof v === 'string' ? '***REDACTED***' : v;
|
||||
}
|
||||
else {
|
||||
result[k] = redact(v, fullPath);
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
const redacted = redact(cfg, '');
|
||||
// plugins 已经过递归 redact()，其中 apiKey/token/password/secret 等字段
|
||||
// 会被自动脱敏，同时保留 provider/model/enabled 等排查所需的非敏感配置
|
||||
|
||||
fs.writeFileSync('$EXPORT_DIR/openclaw-config-redacted.json', JSON.stringify(redacted, null, 2));
|
||||
console.log(' ✅ 配置已脱敏导出');
|
||||
" 2>&1 || {
|
||||
echo " ⚠️ Node 脱敏失败，使用 grep 粗略脱敏"
|
||||
# 粗略脱敏：删除包含敏感关键字的行
|
||||
grep -v -iE '(api.?key|token|password|secret|credential).*:.*"[^"]{8,}"' "$CONFIG_FILE" \
|
||||
| sed -E 's/"(models|secrets|channels|env)"\s*:\s*\{[^}]*\}/"__REDACTED_SECTION__"/g' \
|
||||
> "$EXPORT_DIR/openclaw-config-redacted.json" 2>/dev/null || true
|
||||
}
|
||||
else
|
||||
echo " ⚠️ 未找到配置文件"
|
||||
fi
|
||||
|
||||
# ── 5. 收集插件安装信息 ──
|
||||
echo "🔌 收集插件安装信息..."
|
||||
if [ -d "$STATE_DIR/extensions" ]; then
|
||||
{
|
||||
echo "=== 已安装插件 ==="
|
||||
ls -la "$STATE_DIR/extensions/" 2>/dev/null
|
||||
echo ""
|
||||
for ext_dir in "$STATE_DIR/extensions"/*/; do
|
||||
[ -d "$ext_dir" ] || continue
|
||||
pkg="$ext_dir/node_modules/openclaw/package.json"
|
||||
plugin_pkg="$ext_dir/package.json"
|
||||
echo "--- $(basename "$ext_dir") ---"
|
||||
if [ -f "$plugin_pkg" ]; then
|
||||
node -e "const p=require('$plugin_pkg'); console.log('name:', p.name, 'version:', p.version)" 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
} > "$EXPORT_DIR/plugins-info.txt" 2>&1
|
||||
fi
|
||||
|
||||
# ── 6. 打包 ──
|
||||
echo "📦 打包中..."
|
||||
cd "$(dirname "$EXPORT_DIR")"
|
||||
tar -czf "$ARCHIVE_PATH" "$(basename "$EXPORT_DIR")"
|
||||
|
||||
# 计算大小
|
||||
ARCHIVE_SIZE=$(du -sh "$ARCHIVE_PATH" | cut -f1)
|
||||
|
||||
echo ""
|
||||
echo "═══════════════════════════════════════════════════"
|
||||
echo " ✅ 诊断数据导出完成"
|
||||
echo "═══════════════════════════════════════════════════"
|
||||
echo ""
|
||||
echo " 📦 压缩包: $ARCHIVE_PATH"
|
||||
echo " 📏 大小: $ARCHIVE_SIZE"
|
||||
echo ""
|
||||
echo " 包含内容:"
|
||||
echo " - env-info.txt — 环境信息、目录结构"
|
||||
echo " - logs/ — OpenClaw 网关日志 + 滚动日志"
|
||||
echo " - memory-tdai/ — 记忆插件全量数据 (L0~L3 + SQLite)"
|
||||
echo " - openclaw-config-redacted.json — 脱敏后的配置文件"
|
||||
echo " - plugins-info.txt — 插件安装信息"
|
||||
echo ""
|
||||
echo " ⚠️ 安全提示:"
|
||||
echo " - 配置文件已自动脱敏（API Key、Token、Password 等已移除）"
|
||||
echo " - models/secrets/channels/env 等敏感配置块已整体替换"
|
||||
echo " - 记忆数据中可能包含用户对话内容，请确认后再发送"
|
||||
echo ""
|
||||
echo " 📤 请手动检查后发送给研发团队"
|
||||
echo "═══════════════════════════════════════════════════"
|
||||
+64
-1
@@ -127,6 +127,37 @@ export interface MemoryCleanupConfig {
|
||||
cleanTime: string;
|
||||
}
|
||||
|
||||
/** BM25 sparse vector encoding configuration (local @tencentdb-agent-memory/tcvdb-text). */
|
||||
export interface BM25Config {
|
||||
/** Whether BM25 sparse encoding is enabled (default: true) */
|
||||
enabled: boolean;
|
||||
/** Language for BM25 pre-trained params: "zh" or "en" (default: "zh") */
|
||||
language: "zh" | "en";
|
||||
}
|
||||
|
||||
/** Tencent Cloud VectorDB configuration. */
|
||||
export interface TcvdbConfig {
|
||||
/** Instance URL (e.g. "http://10.0.1.1:80" or external domain) */
|
||||
url: string;
|
||||
/** Account name (default: "root") */
|
||||
username: string;
|
||||
/** API Key */
|
||||
apiKey: string;
|
||||
/** Database name (auto-generated from instance_id if empty) */
|
||||
database: string;
|
||||
/** User-friendly alias for this database (optional, for identification in database.json) */
|
||||
alias: string;
|
||||
/** Built-in embedding model (default: "bge-large-zh") */
|
||||
embeddingModel: string;
|
||||
/** Request timeout in ms (default: 10000) */
|
||||
timeout: number;
|
||||
/** Path to CA certificate PEM file (for HTTPS connections) */
|
||||
caPemPath?: string;
|
||||
}
|
||||
|
||||
/** Storage backend type. */
|
||||
export type StoreBackend = "sqlite" | "tcvdb";
|
||||
|
||||
/** Report settings — controls metric/event reporting. */
|
||||
export interface ReportConfig {
|
||||
/** Enable reporting (default: true) */
|
||||
@@ -143,6 +174,13 @@ export interface MemoryTdaiConfig {
|
||||
pipeline: PipelineTriggerConfig;
|
||||
recall: RecallConfig;
|
||||
embedding: EmbeddingConfig;
|
||||
/** Storage backend: "sqlite" (default) or "tcvdb" */
|
||||
storeBackend: StoreBackend;
|
||||
/** Tencent Cloud VectorDB configuration (required when storeBackend = "tcvdb") */
|
||||
tcvdb: TcvdbConfig;
|
||||
/** BM25 sparse vector encoding (local @tencentdb-agent-memory/tcvdb-text) */
|
||||
bm25: BM25Config;
|
||||
/** Local JSONL cleanup settings */
|
||||
memoryCleanup: MemoryCleanupConfig;
|
||||
report: ReportConfig;
|
||||
}
|
||||
@@ -272,6 +310,16 @@ export function parseConfig(raw: Record<string, unknown> | undefined): MemoryTda
|
||||
|
||||
const cleanTime = normalizeCleanTime(str(captureGroup, "cleanTime")) ?? "03:00";
|
||||
|
||||
// --- BM25 (local @tencentdb-agent-memory/tcvdb-text encoder) ---
|
||||
const bm25Group = obj(c, "bm25");
|
||||
|
||||
// --- Store backend ---
|
||||
const storeBackendRaw = str(c, "storeBackend") ?? "sqlite";
|
||||
const storeBackend: StoreBackend = storeBackendRaw === "tcvdb" ? "tcvdb" : "sqlite";
|
||||
|
||||
// --- TCVDB config ---
|
||||
const tcvdbGroup = obj(c, "tcvdb");
|
||||
|
||||
const memoryCleanup: MemoryCleanupConfig = {
|
||||
retentionDays,
|
||||
enabled: retentionDays != null,
|
||||
@@ -293,7 +341,7 @@ export function parseConfig(raw: Record<string, unknown> | undefined): MemoryTda
|
||||
},
|
||||
persona: {
|
||||
triggerEveryN: num(personaGroup, "triggerEveryN") ?? 50,
|
||||
maxScenes: num(personaGroup, "maxScenes") ?? 20,
|
||||
maxScenes: num(personaGroup, "maxScenes") ?? 15,
|
||||
backupCount: num(personaGroup, "backupCount") ?? 3,
|
||||
sceneBackupCount: num(personaGroup, "sceneBackupCount") ?? 10,
|
||||
model: optStr(personaGroup, "model"),
|
||||
@@ -328,6 +376,21 @@ export function parseConfig(raw: Record<string, unknown> | undefined): MemoryTda
|
||||
modelCacheDir: optStr(embeddingGroup, "modelCacheDir"),
|
||||
configError: embeddingConfigError,
|
||||
},
|
||||
storeBackend,
|
||||
tcvdb: {
|
||||
url: str(tcvdbGroup, "url") ?? "",
|
||||
username: str(tcvdbGroup, "username") ?? "root",
|
||||
apiKey: str(tcvdbGroup, "apiKey") ?? "",
|
||||
database: str(tcvdbGroup, "database") ?? "",
|
||||
alias: str(tcvdbGroup, "alias") ?? "",
|
||||
embeddingModel: str(tcvdbGroup, "embeddingModel") ?? "bge-large-zh",
|
||||
timeout: num(tcvdbGroup, "timeout") ?? 10000,
|
||||
caPemPath: str(tcvdbGroup, "caPemPath") || undefined,
|
||||
},
|
||||
bm25: {
|
||||
enabled: bool(bm25Group, "enabled") ?? true,
|
||||
language: (str(bm25Group, "language") === "en" ? "en" : "zh") as "zh" | "en",
|
||||
},
|
||||
memoryCleanup,
|
||||
report: {
|
||||
enabled: bool(obj(c, "report"), "enabled") ?? false,
|
||||
|
||||
@@ -160,7 +160,7 @@ export async function recordConversation(params: {
|
||||
// - If a message lacks a timestamp field, extractUserAssistantMessages()
|
||||
// assigns Date.now() at extraction time, which is always > previous cursor.
|
||||
const cursor = afterTimestamp ?? 0;
|
||||
const extracted = cursor > 0
|
||||
const extracted = cursor !== 0
|
||||
? allExtracted.filter((m) => m.timestamp > cursor)
|
||||
: allExtracted;
|
||||
|
||||
@@ -440,7 +440,7 @@ export async function readConversationMessages(
|
||||
*/
|
||||
export interface SessionIdMessageGroup {
|
||||
sessionId: string;
|
||||
messages: ConversationMessage[];
|
||||
messages: Array<ConversationMessage & { recordedAtMs: number }>;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -451,29 +451,32 @@ export interface SessionIdMessageGroup {
|
||||
* so that each group's sessionId is correctly associated with its extracted memories.
|
||||
*
|
||||
* When `limit` is provided, only the **newest** `limit` messages (across all groups)
|
||||
* are retained — matching the DB path's `ORDER BY timestamp DESC LIMIT ?` behavior.
|
||||
* are retained — matching the DB path's `ORDER BY recorded_at DESC LIMIT ?` behavior.
|
||||
* Groups that become empty after truncation are dropped.
|
||||
*
|
||||
* Groups are returned in chronological order (by earliest message timestamp).
|
||||
* Messages within each group are also in chronological order.
|
||||
*
|
||||
* @param afterRecordedAtMs - Epoch ms cursor: only messages with recordedAt > this are included.
|
||||
*/
|
||||
export async function readConversationMessagesGroupedBySessionId(
|
||||
sessionKey: string,
|
||||
baseDir: string,
|
||||
afterTimestamp?: number,
|
||||
afterRecordedAtMs?: number,
|
||||
logger?: Logger,
|
||||
limit?: number,
|
||||
): Promise<SessionIdMessageGroup[]> {
|
||||
const records = await readConversationRecords(sessionKey, baseDir, logger);
|
||||
|
||||
// Collect all messages with their sessionId, respecting afterTimestamp filter
|
||||
const allMessages: Array<{ sessionId: string; msg: ConversationMessage }> = [];
|
||||
// Collect all messages with their sessionId, filtering by recorded_at cursor
|
||||
const allMessages: Array<{ sessionId: string; msg: ConversationMessage & { recordedAtMs: number } }> = [];
|
||||
|
||||
for (const record of records) {
|
||||
const sid = record.sessionId || "";
|
||||
const recMs = Date.parse(record.recordedAt) || 0;
|
||||
if (afterRecordedAtMs && recMs <= afterRecordedAtMs) continue;
|
||||
for (const msg of record.messages) {
|
||||
if (afterTimestamp && msg.timestamp <= afterTimestamp) continue;
|
||||
allMessages.push({ sessionId: sid, msg });
|
||||
allMessages.push({ sessionId: sid, msg: { ...msg, recordedAtMs: recMs } });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -491,7 +494,7 @@ export async function readConversationMessagesGroupedBySessionId(
|
||||
}
|
||||
|
||||
// Re-group by sessionId
|
||||
const groupMap = new Map<string, ConversationMessage[]>();
|
||||
const groupMap = new Map<string, Array<ConversationMessage & { recordedAtMs: number }>>();
|
||||
for (const { sessionId, msg } of selected) {
|
||||
let group = groupMap.get(sessionId);
|
||||
if (!group) {
|
||||
|
||||
+82
-33
@@ -16,7 +16,7 @@ import { CheckpointManager } from "../utils/checkpoint.js";
|
||||
import type { MemoryPipelineManager } from "../utils/pipeline-manager.js";
|
||||
import { recordConversation } from "../conversation/l0-recorder.js";
|
||||
import type { ConversationMessage } from "../conversation/l0-recorder.js";
|
||||
import type { VectorStore, L0VectorRecord } from "../store/vector-store.js";
|
||||
import type { IMemoryStore, L0Record } from "../store/types.js";
|
||||
import type { EmbeddingService } from "../store/embedding.js";
|
||||
|
||||
const TAG = "[memory-tdai] [capture]";
|
||||
@@ -68,7 +68,7 @@ export async function performAutoCapture(params: {
|
||||
* prevents the first agent_end from dumping all session history into L0. */
|
||||
pluginStartTimestamp?: number;
|
||||
/** VectorStore for L0 vector indexing (optional). */
|
||||
vectorStore?: VectorStore;
|
||||
vectorStore?: IMemoryStore;
|
||||
/** EmbeddingService for L0 vector indexing (optional). */
|
||||
embeddingService?: EmbeddingService;
|
||||
}): Promise<AutoCaptureResult> {
|
||||
@@ -131,32 +131,42 @@ export async function performAutoCapture(params: {
|
||||
const tL0RecordEnd = performance.now();
|
||||
|
||||
// ============================
|
||||
// Step 1.5: L0 vector indexing — metadata written synchronously,
|
||||
// embedding done in background (non-blocking)
|
||||
// Step 1.5: L0 vector indexing
|
||||
// ============================
|
||||
// PERF FIX: Remote embedding API calls (2-3s each) were blocking
|
||||
// the agent_end hook, adding 5-9s latency per conversation round.
|
||||
// Now we:
|
||||
// 1. Write L0 metadata + FTS immediately (no embedding) — ~10ms
|
||||
// 2. Fire off background task to embed + update vectors (non-blocking)
|
||||
// This way the user gets their response immediately.
|
||||
// Two paths depending on store capabilities:
|
||||
//
|
||||
// A) Store supports updateL0Embedding (sqlite):
|
||||
// - Write metadata + FTS immediately WITHOUT embedding (~ms)
|
||||
// - Fire-and-forget background task: embedBatch + updateL0Embedding
|
||||
// - PERF: avoids blocking agent_end with 2-3s embedding calls
|
||||
//
|
||||
// B) Store does NOT support updateL0Embedding (VDB / remote):
|
||||
// - Embed synchronously, then upsertL0 with embedding in one call
|
||||
// - VDB backends handle embedding server-side or need it upfront
|
||||
const tL0VecStart = performance.now();
|
||||
let l0VectorsWritten = 0;
|
||||
let l0EmbedTotalMs = 0;
|
||||
let l0UpsertTotalMs = 0;
|
||||
logger?.debug?.(
|
||||
`${TAG} [L0-vec-index] Check: filteredMessages=${filteredMessages.length}, ` +
|
||||
`vectorStore=${vectorStore ? "available" : "UNAVAILABLE"}, ` +
|
||||
`embeddingService=${embeddingService ? "available" : "UNAVAILABLE"}`,
|
||||
);
|
||||
|
||||
// Pre-generate L0 records and write metadata synchronously (fast path)
|
||||
const l0Records: Array<{ record: L0VectorRecord; content: string }> = [];
|
||||
const supportsBgEmbed = vectorStore?.supportsDeferredEmbedding === true;
|
||||
|
||||
if (filteredMessages.length > 0 && vectorStore) {
|
||||
const now = new Date().toISOString();
|
||||
logger?.debug?.(`${TAG} [L0-vec-index] START indexing ${filteredMessages.length} message(s) for session ${sessionKey}`);
|
||||
const bgRecords: Array<{ recordId: string; content: string }> = [];
|
||||
logger?.debug?.(
|
||||
`${TAG} [L0-vec-index] START indexing ${filteredMessages.length} message(s) for session ${sessionKey} ` +
|
||||
`(mode=${supportsBgEmbed ? "async-bg" : "sync"})`,
|
||||
);
|
||||
|
||||
for (let i = 0; i < filteredMessages.length; i++) {
|
||||
const msg = filteredMessages[i];
|
||||
try {
|
||||
const l0Record: L0VectorRecord = {
|
||||
const l0Record: L0Record = {
|
||||
id: generateL0RecordId(sessionKey, i),
|
||||
sessionKey,
|
||||
sessionId: sessionId || "",
|
||||
@@ -166,11 +176,45 @@ export async function performAutoCapture(params: {
|
||||
timestamp: msg.timestamp,
|
||||
};
|
||||
|
||||
// Write metadata + FTS immediately WITHOUT embedding (fast, ~ms)
|
||||
const upsertOk = vectorStore.upsertL0(l0Record, undefined);
|
||||
let embedding: Float32Array | undefined;
|
||||
|
||||
if (!supportsBgEmbed && embeddingService) {
|
||||
// Path B (VDB): embed synchronously — needed for upsertL0
|
||||
// Skip local embed when using server-side embedding (NoopEmbeddingService, dims=0)
|
||||
if (embeddingService.getDimensions() === 0) {
|
||||
logger?.debug?.(
|
||||
`${TAG} [L0-vec-index] Server-side embedding (dims=0), skipping local embed for message ${i}`,
|
||||
);
|
||||
} else {
|
||||
const tEmbedStart = performance.now();
|
||||
try {
|
||||
embedding = await embeddingService.embed(msg.content);
|
||||
l0EmbedTotalMs += performance.now() - tEmbedStart;
|
||||
logger?.debug?.(
|
||||
`${TAG} [L0-vec-index] Embedding OK: dims=${embedding.length}, ` +
|
||||
`norm=${Math.sqrt(Array.from(embedding).reduce((s, v) => s + v * v, 0)).toFixed(4)}`,
|
||||
);
|
||||
} catch (embedErr) {
|
||||
l0EmbedTotalMs += performance.now() - tEmbedStart;
|
||||
logger?.warn(
|
||||
`${TAG} [L0-vec-index] Embedding FAILED for message ${i}, ` +
|
||||
`will write metadata only: ${embedErr instanceof Error ? embedErr.message : String(embedErr)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Path A (sqlite): pass undefined embedding — metadata + FTS only
|
||||
// Path B (VDB): pass embedding (may be undefined on failure)
|
||||
const tUpsertStart = performance.now();
|
||||
const upsertOk = await vectorStore.upsertL0(l0Record, supportsBgEmbed ? undefined : embedding);
|
||||
l0UpsertTotalMs += performance.now() - tUpsertStart;
|
||||
|
||||
if (upsertOk) {
|
||||
l0VectorsWritten++;
|
||||
l0Records.push({ record: l0Record, content: msg.content });
|
||||
if (supportsBgEmbed) {
|
||||
bgRecords.push({ recordId: l0Record.id, content: msg.content });
|
||||
}
|
||||
} else {
|
||||
logger?.warn(`${TAG} [L0-vec-index] upsertL0 returned false for message ${i}`);
|
||||
}
|
||||
@@ -178,38 +222,39 @@ export async function performAutoCapture(params: {
|
||||
logger?.warn?.(`${TAG} [L0-vec-index] FAILED for message ${i} (non-blocking): ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
}
|
||||
logger?.debug?.(`${TAG} [L0-vec-index] DONE: ${l0VectorsWritten}/${filteredMessages.length} metadata records written (sync)`);
|
||||
|
||||
// Fire-and-forget: batch embed + update vectors in background
|
||||
if (l0Records.length > 0 && embeddingService) {
|
||||
const bgVectorStore = vectorStore; // capture for closure
|
||||
const modeLabel = supportsBgEmbed ? "metadata-only, embed=background" : `embed=${l0EmbedTotalMs.toFixed(0)}ms, upsert=${l0UpsertTotalMs.toFixed(0)}ms`;
|
||||
logger?.debug?.(`${TAG} [L0-vec-index] DONE: ${l0VectorsWritten}/${filteredMessages.length} records written (${modeLabel})`);
|
||||
|
||||
// Path A only: fire-and-forget background embedding for sqlite stores
|
||||
if (supportsBgEmbed && bgRecords.length > 0 && embeddingService) {
|
||||
const bgVectorStore = vectorStore;
|
||||
const bgEmbeddingService = embeddingService;
|
||||
const bgRecords = [...l0Records]; // snapshot
|
||||
const bgSnapshot = [...bgRecords];
|
||||
const bgLogger = logger;
|
||||
|
||||
// Do NOT await — this runs in background after response is sent
|
||||
// Do NOT await — runs in background after response is sent
|
||||
void (async () => {
|
||||
const tBgStart = performance.now();
|
||||
try {
|
||||
// Use embedBatch for a single API call instead of N sequential calls
|
||||
const texts = bgRecords.map((r) => r.content);
|
||||
const texts = bgSnapshot.map((r) => r.content);
|
||||
const embeddings = await bgEmbeddingService.embedBatch(texts);
|
||||
|
||||
let bgUpdated = 0;
|
||||
for (let i = 0; i < bgRecords.length; i++) {
|
||||
for (let i = 0; i < bgSnapshot.length; i++) {
|
||||
try {
|
||||
const ok = bgVectorStore.updateL0Embedding(bgRecords[i].record.id, embeddings[i]);
|
||||
const ok = await bgVectorStore.updateL0Embedding!(bgSnapshot[i].recordId, embeddings[i]);
|
||||
if (ok) bgUpdated++;
|
||||
} catch (err) {
|
||||
bgLogger?.warn?.(
|
||||
`${TAG} [L0-vec-index-bg] Failed to update embedding for ${bgRecords[i].record.id}: ` +
|
||||
`${TAG} [L0-vec-index-bg] Failed to update embedding for ${bgSnapshot[i].recordId}: ` +
|
||||
`${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
const bgMs = performance.now() - tBgStart;
|
||||
bgLogger?.debug?.(
|
||||
`${TAG} [L0-vec-index-bg] Background embedding complete: ${bgUpdated}/${bgRecords.length} vectors updated (${bgMs.toFixed(0)}ms)`,
|
||||
`${TAG} [L0-vec-index-bg] Background embedding complete: ${bgUpdated}/${bgSnapshot.length} vectors updated (${bgMs.toFixed(0)}ms)`,
|
||||
);
|
||||
} catch (err) {
|
||||
const bgMs = performance.now() - tBgStart;
|
||||
@@ -235,11 +280,13 @@ export async function performAutoCapture(params: {
|
||||
logger?.debug?.(`${TAG} Scheduler notified of conversation round (sessionKey=${sessionKey})`);
|
||||
|
||||
const totalMs = performance.now() - tCaptureStart;
|
||||
const vecDetail = supportsBgEmbed
|
||||
? `metadata-only, embed=background, msgs=${filteredMessages.length}`
|
||||
: `embed=${l0EmbedTotalMs.toFixed(0)}ms, upsert=${l0UpsertTotalMs.toFixed(0)}ms, msgs=${filteredMessages.length}`;
|
||||
logger?.info(
|
||||
`${TAG} ⏱ Capture timing: total=${totalMs.toFixed(0)}ms, ` +
|
||||
`l0Record+checkpoint=${(tL0RecordEnd - tL0RecordStart).toFixed(0)}ms, ` +
|
||||
`l0VecIndex=${(tL0VecEnd - tL0VecStart).toFixed(0)}ms ` +
|
||||
`(metadata-only, embed=background, msgs=${filteredMessages.length}), ` +
|
||||
`l0VecIndex=${(tL0VecEnd - tL0VecStart).toFixed(0)}ms (${vecDetail}), ` +
|
||||
`notify=${(performance.now() - tNotifyStart).toFixed(0)}ms`,
|
||||
);
|
||||
|
||||
@@ -252,11 +299,13 @@ export async function performAutoCapture(params: {
|
||||
}
|
||||
|
||||
const totalMs = performance.now() - tCaptureStart;
|
||||
const vecDetail = supportsBgEmbed
|
||||
? `metadata-only, embed=background, msgs=${filteredMessages.length}`
|
||||
: `embed=${l0EmbedTotalMs.toFixed(0)}ms, upsert=${l0UpsertTotalMs.toFixed(0)}ms, msgs=${filteredMessages.length}`;
|
||||
logger?.info(
|
||||
`${TAG} ⏱ Capture timing: total=${totalMs.toFixed(0)}ms, ` +
|
||||
`l0Record+checkpoint=${(tL0RecordEnd - tL0RecordStart).toFixed(0)}ms, ` +
|
||||
`l0VecIndex=${(tL0VecEnd - tL0VecStart).toFixed(0)}ms ` +
|
||||
`(metadata-only, embed=background, msgs=${filteredMessages.length}), ` +
|
||||
`l0VecIndex=${(tL0VecEnd - tL0VecStart).toFixed(0)}ms (${vecDetail}), ` +
|
||||
`notify=${(performance.now() - tNotifyStart).toFixed(0)}ms`,
|
||||
);
|
||||
|
||||
|
||||
+21
-16
@@ -16,8 +16,8 @@ import type { MemoryTdaiConfig } from "../config.js";
|
||||
import { readSceneIndex } from "../scene/scene-index.js";
|
||||
import { generateSceneNavigation, stripSceneNavigation } from "../scene/scene-navigation.js";
|
||||
import type { MemoryRecord } from "../record/l1-reader.js";
|
||||
import type { VectorStore, VectorSearchResult, FtsSearchResult } from "../store/vector-store.js";
|
||||
import { buildFtsQuery } from "../store/vector-store.js";
|
||||
import type { IMemoryStore, L1SearchResult, L1FtsResult } from "../store/types.js";
|
||||
import { buildFtsQuery } from "../store/sqlite.js";
|
||||
import type { EmbeddingService } from "../store/embedding.js";
|
||||
import { sanitizeText } from "../utils/sanitize.js";
|
||||
|
||||
@@ -35,6 +35,11 @@ const MEMORY_TOOLS_GUIDE = `<memory-tools-guide>
|
||||
- **tdai_memory_search**：搜索结构化记忆（L1），适用于回忆用户偏好、历史事件节点、规则等关键信息。
|
||||
- **tdai_conversation_search**：搜索原始对话（L0），适用于查找具体消息原文、时间线、上下文细节；也可用于补充或校验 memory_search 的结果。
|
||||
- **read_file**（Scene Navigation 中的路径）：当已定位到相关情境，且需要该场景的完整画像、事件经过或阶段结论时使用。
|
||||
|
||||
### ⚠️ 调用次数限制
|
||||
每轮对话中，tdai_memory_search 和 tdai_conversation_search **合计最多调用 3 次**。
|
||||
- 首次搜索无结果时，可换关键词或换工具重试，但总调用次数不要超过 3 次。
|
||||
- 若 3 次搜索后仍无结果，说明该信息不在记忆中，请直接根据已有信息回复用户，不要继续搜索。
|
||||
</memory-tools-guide>`
|
||||
|
||||
/**
|
||||
@@ -82,7 +87,7 @@ export async function performAutoRecall(params: {
|
||||
cfg: MemoryTdaiConfig;
|
||||
pluginDataDir: string;
|
||||
logger?: Logger;
|
||||
vectorStore?: VectorStore;
|
||||
vectorStore?: IMemoryStore;
|
||||
embeddingService?: EmbeddingService;
|
||||
}): Promise<RecallResult | undefined> {
|
||||
const { cfg, logger } = params;
|
||||
@@ -112,7 +117,7 @@ async function performAutoRecallInner(params: {
|
||||
cfg: MemoryTdaiConfig;
|
||||
pluginDataDir: string;
|
||||
logger?: Logger;
|
||||
vectorStore?: VectorStore;
|
||||
vectorStore?: IMemoryStore;
|
||||
embeddingService?: EmbeddingService;
|
||||
}): Promise<RecallResult | undefined> {
|
||||
const { userText, cfg, pluginDataDir, logger, vectorStore, embeddingService } = params;
|
||||
@@ -269,7 +274,7 @@ async function searchMemoriesWithDetails(
|
||||
cfg: MemoryTdaiConfig,
|
||||
logger: Logger | undefined,
|
||||
strategy: "keyword" | "embedding" | "hybrid",
|
||||
vectorStore?: VectorStore,
|
||||
vectorStore?: IMemoryStore,
|
||||
embeddingService?: EmbeddingService,
|
||||
): Promise<{ lines: string[]; memories: RecalledMemory[]; timing: SearchTiming }> {
|
||||
const result = await searchMemories(userText, pluginDataDir, cfg, logger, strategy, vectorStore, embeddingService);
|
||||
@@ -305,7 +310,7 @@ async function searchMemories(
|
||||
cfg: MemoryTdaiConfig,
|
||||
logger: Logger | undefined,
|
||||
strategy: "keyword" | "embedding" | "hybrid",
|
||||
vectorStore?: VectorStore,
|
||||
vectorStore?: IMemoryStore,
|
||||
embeddingService?: EmbeddingService,
|
||||
): Promise<SearchResult> {
|
||||
const emptyResult: SearchResult = { lines: [], timing: { ftsMs: 0, embeddingMs: 0, ftsHits: 0, embeddingHits: 0 } };
|
||||
@@ -378,14 +383,14 @@ async function searchByKeyword(
|
||||
maxResults: number,
|
||||
threshold: number,
|
||||
logger?: Logger,
|
||||
vectorStore?: VectorStore,
|
||||
vectorStore?: IMemoryStore,
|
||||
): Promise<string[]> {
|
||||
// Prefer FTS5 if available
|
||||
if (vectorStore?.isFtsAvailable()) {
|
||||
const ftsQuery = buildFtsQuery(userText);
|
||||
if (ftsQuery) {
|
||||
logger?.debug?.(`${TAG} [keyword-fts] Using FTS5 BM25 search: query="${ftsQuery}"`);
|
||||
const ftsResults = vectorStore.ftsSearchL1(ftsQuery, maxResults * 2);
|
||||
const ftsResults = await vectorStore.searchL1Fts(ftsQuery, maxResults * 2);
|
||||
if (ftsResults.length > 0) {
|
||||
logger?.debug?.(
|
||||
`${TAG} [keyword-fts] FTS5 raw results (${ftsResults.length}): ` +
|
||||
@@ -428,7 +433,7 @@ async function searchByEmbedding(
|
||||
userText: string,
|
||||
maxResults: number,
|
||||
threshold: number,
|
||||
vectorStore: VectorStore,
|
||||
vectorStore: IMemoryStore,
|
||||
embeddingService: EmbeddingService,
|
||||
logger?: Logger,
|
||||
): Promise<string[]> {
|
||||
@@ -442,7 +447,7 @@ async function searchByEmbedding(
|
||||
`searching top-${maxResults * 2}...`,
|
||||
);
|
||||
// Retrieve more candidates for subsequent filtering
|
||||
const vecResults: VectorSearchResult[] = vectorStore.search(queryEmbedding, maxResults * 2);
|
||||
const vecResults: L1SearchResult[] = await vectorStore.searchL1Vector(queryEmbedding, maxResults * 2);
|
||||
|
||||
if (vecResults.length === 0) {
|
||||
logger?.debug?.(`${TAG} [embedding-search] Returned 0 results`);
|
||||
@@ -489,7 +494,7 @@ async function searchHybrid(
|
||||
_pluginDataDir: string,
|
||||
maxResults: number,
|
||||
_threshold: number,
|
||||
vectorStore: VectorStore,
|
||||
vectorStore: IMemoryStore,
|
||||
embeddingService: EmbeddingService,
|
||||
logger?: Logger,
|
||||
): Promise<SearchResult> {
|
||||
@@ -505,7 +510,7 @@ async function searchHybrid(
|
||||
if (vectorStore.isFtsAvailable()) {
|
||||
const ftsQuery = buildFtsQuery(userText);
|
||||
if (ftsQuery) {
|
||||
const ftsResults = vectorStore.ftsSearchL1(ftsQuery, candidateK);
|
||||
const ftsResults = await vectorStore.searchL1Fts(ftsQuery, candidateK);
|
||||
if (ftsResults.length > 0) {
|
||||
logger?.debug?.(`${TAG} [hybrid-keyword-fts] FTS5 found ${ftsResults.length} candidates`);
|
||||
// Convert FtsSearchResult to ScoredRecord for RRF merge
|
||||
@@ -547,12 +552,12 @@ async function searchHybrid(
|
||||
logger?.debug?.(
|
||||
`${TAG} [hybrid-embedding] Embedding OK, dims=${queryEmbedding.length}, searching top-${candidateK}...`,
|
||||
);
|
||||
const results = vectorStore.search(queryEmbedding, candidateK);
|
||||
const results = await vectorStore.searchL1Vector(queryEmbedding, candidateK, userText);
|
||||
logger?.debug?.(`${TAG} [hybrid-embedding] Got ${results.length} candidates`);
|
||||
return { results, ms: performance.now() - tStart };
|
||||
} catch (err) {
|
||||
logger?.warn?.(`${TAG} Hybrid: embedding part failed: ${err instanceof Error ? err.message : String(err)}`);
|
||||
return { results: [] as VectorSearchResult[], ms: performance.now() - tStart };
|
||||
return { results: [] as L1SearchResult[], ms: performance.now() - tStart };
|
||||
}
|
||||
})(),
|
||||
]);
|
||||
@@ -719,7 +724,7 @@ function recordToFormatable(record: MemoryRecord): FormatableMemory {
|
||||
* Build a FormatableMemory from a VectorSearchResult (embedding search path).
|
||||
* Handles empty/invalid metadata_json, empty timestamp_str gracefully.
|
||||
*/
|
||||
function vectorResultToFormatable(r: VectorSearchResult): FormatableMemory {
|
||||
function vectorResultToFormatable(r: L1SearchResult): FormatableMemory {
|
||||
let activityStart: string | undefined;
|
||||
let activityEnd: string | undefined;
|
||||
if (r.metadata_json && r.metadata_json !== "{}") {
|
||||
@@ -743,7 +748,7 @@ function vectorResultToFormatable(r: VectorSearchResult): FormatableMemory {
|
||||
* Build a FormatableMemory from an FtsSearchResult (FTS5 keyword search path).
|
||||
* Handles empty/invalid metadata_json, empty timestamp_str gracefully.
|
||||
*/
|
||||
function ftsResultToFormatable(r: FtsSearchResult): FormatableMemory {
|
||||
function ftsResultToFormatable(r: L1FtsResult): FormatableMemory {
|
||||
let activityStart: string | undefined;
|
||||
let activityEnd: string | undefined;
|
||||
if (r.metadata_json && r.metadata_json !== "{}") {
|
||||
|
||||
@@ -53,9 +53,9 @@ export class PersonaGenerator {
|
||||
}
|
||||
|
||||
/**
|
||||
* Execute persona generation.
|
||||
* Execute local persona generation without advancing checkpoint.
|
||||
*/
|
||||
async generate(triggerReason?: string): Promise<boolean> {
|
||||
async generateLocalPersona(triggerReason?: string): Promise<boolean> {
|
||||
const startMs = Date.now();
|
||||
this.logger?.debug?.(`${TAG} Starting generation: reason="${triggerReason ?? "none"}"`);
|
||||
|
||||
@@ -181,9 +181,6 @@ export class PersonaGenerator {
|
||||
const finalContent = nav ? `${personaText}\n\n${nav}\n` : personaText;
|
||||
await fs.writeFile(personaPath, finalContent, "utf-8");
|
||||
|
||||
// 12. Update checkpoint
|
||||
await cpManager.markPersonaGenerated(cp.total_processed);
|
||||
|
||||
const elapsedMs = Date.now() - startMs;
|
||||
this.logger?.info(`${TAG} Persona written (${finalContent.length} chars) in ${elapsedMs}ms`);
|
||||
|
||||
@@ -202,4 +199,17 @@ export class PersonaGenerator {
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Backward-compatible wrapper: local generation + checkpoint advance.
|
||||
*/
|
||||
async generate(triggerReason?: string): Promise<boolean> {
|
||||
const updated = await this.generateLocalPersona(triggerReason);
|
||||
if (!updated) return false;
|
||||
|
||||
const cpManager = new CheckpointManager(this.dataDir);
|
||||
const cp = await cpManager.read();
|
||||
await cpManager.markPersonaGenerated(cp.total_processed);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,213 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import type { IMemoryStore, ProfileRecord, ProfileSyncRecord } from "../store/types.js";
|
||||
import { readSceneIndex, syncSceneIndex } from "../scene/scene-index.js";
|
||||
import { generateSceneNavigation, stripSceneNavigation } from "../scene/scene-navigation.js";
|
||||
|
||||
const PROFILE_SCOPE = "global";
|
||||
|
||||
interface Logger {
|
||||
debug?: (message: string) => void;
|
||||
info: (message: string) => void;
|
||||
warn: (message: string) => void;
|
||||
error: (message: string) => void;
|
||||
}
|
||||
|
||||
export interface ProfileBaseline {
|
||||
version: number;
|
||||
contentMd5: string;
|
||||
createdAtMs: number;
|
||||
}
|
||||
|
||||
export function buildProfileStableId(scope: string, type: "l2" | "l3", filename: string): string {
|
||||
const hash = createHash("sha256")
|
||||
.update(`${scope}\u0000${type}\u0000${filename}`)
|
||||
.digest("hex");
|
||||
return `profile:v1:${hash}`;
|
||||
}
|
||||
|
||||
function md5(text: string): string {
|
||||
return createHash("md5").update(text).digest("hex");
|
||||
}
|
||||
|
||||
async function statTimes(filePath: string): Promise<{ createdAtMs: number; updatedAtMs: number }> {
|
||||
try {
|
||||
const stat = await fs.stat(filePath);
|
||||
return {
|
||||
createdAtMs: Math.floor(stat.birthtimeMs || stat.ctimeMs || Date.now()),
|
||||
updatedAtMs: Math.floor(stat.mtimeMs || Date.now()),
|
||||
};
|
||||
} catch {
|
||||
const now = Date.now();
|
||||
return { createdAtMs: now, updatedAtMs: now };
|
||||
}
|
||||
}
|
||||
|
||||
async function refreshPersonaNavigation(dataDir: string): Promise<void> {
|
||||
const personaPath = path.join(dataDir, "persona.md");
|
||||
let body: string;
|
||||
try {
|
||||
body = stripSceneNavigation(await fs.readFile(personaPath, "utf-8")).trim();
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
|
||||
if (!body) return;
|
||||
|
||||
const index = await readSceneIndex(dataDir);
|
||||
const nav = generateSceneNavigation(index);
|
||||
const finalContent = nav ? `${body}\n\n${nav}\n` : `${body}\n`;
|
||||
await fs.writeFile(personaPath, finalContent, "utf-8");
|
||||
}
|
||||
|
||||
export async function listLocalProfiles(dataDir: string): Promise<ProfileRecord[]> {
|
||||
const profiles: ProfileRecord[] = [];
|
||||
const blocksDir = path.join(dataDir, "scene_blocks");
|
||||
|
||||
try {
|
||||
const files = (await fs.readdir(blocksDir)).filter((file) => file.endsWith(".md")).sort();
|
||||
for (const filename of files) {
|
||||
const filePath = path.join(blocksDir, filename);
|
||||
const content = await fs.readFile(filePath, "utf-8");
|
||||
const { createdAtMs, updatedAtMs } = await statTimes(filePath);
|
||||
profiles.push({
|
||||
id: buildProfileStableId(PROFILE_SCOPE, "l2", filename),
|
||||
type: "l2",
|
||||
filename,
|
||||
content,
|
||||
contentMd5: md5(content),
|
||||
version: 0,
|
||||
createdAtMs,
|
||||
updatedAtMs,
|
||||
});
|
||||
}
|
||||
} catch {
|
||||
// ignore missing scene_blocks directory
|
||||
}
|
||||
|
||||
const personaPath = path.join(dataDir, "persona.md");
|
||||
try {
|
||||
const rawPersona = await fs.readFile(personaPath, "utf-8");
|
||||
const body = stripSceneNavigation(rawPersona).trim();
|
||||
if (body) {
|
||||
const { createdAtMs, updatedAtMs } = await statTimes(personaPath);
|
||||
profiles.push({
|
||||
id: buildProfileStableId(PROFILE_SCOPE, "l3", "persona.md"),
|
||||
type: "l3",
|
||||
filename: "persona.md",
|
||||
content: body,
|
||||
contentMd5: md5(body),
|
||||
version: 0,
|
||||
createdAtMs,
|
||||
updatedAtMs,
|
||||
});
|
||||
}
|
||||
} catch {
|
||||
// ignore missing persona file
|
||||
}
|
||||
|
||||
return profiles;
|
||||
}
|
||||
|
||||
export async function pullProfilesToLocal(
|
||||
dataDir: string,
|
||||
store: IMemoryStore,
|
||||
logger: Logger,
|
||||
): Promise<Map<string, ProfileBaseline>> {
|
||||
if (!store.pullProfiles) return new Map();
|
||||
|
||||
const records = await store.pullProfiles();
|
||||
const baseline = new Map<string, ProfileBaseline>();
|
||||
const tempDir = await fs.mkdtemp(path.join(dataDir, ".profiles-pull-"));
|
||||
const tempBlocksDir = path.join(tempDir, "scene_blocks");
|
||||
await fs.mkdir(tempBlocksDir, { recursive: true });
|
||||
|
||||
try {
|
||||
for (const record of records) {
|
||||
baseline.set(record.id, {
|
||||
version: record.version,
|
||||
contentMd5: record.contentMd5,
|
||||
createdAtMs: record.createdAtMs,
|
||||
});
|
||||
|
||||
if (record.type === "l2") {
|
||||
const target = path.join(tempBlocksDir, record.filename);
|
||||
await fs.writeFile(target, record.content, "utf-8");
|
||||
if (md5(record.content) !== record.contentMd5) {
|
||||
await fs.rm(target, { force: true });
|
||||
logger.warn(`[memory-tdai][profile-sync] MD5 mismatch for ${record.filename}`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (record.type === "l3") {
|
||||
const body = stripSceneNavigation(record.content).trim();
|
||||
await fs.writeFile(path.join(tempDir, "persona.md"), body, "utf-8");
|
||||
if (md5(body) !== record.contentMd5) {
|
||||
await fs.rm(path.join(tempDir, "persona.md"), { force: true });
|
||||
logger.warn(`[memory-tdai][profile-sync] MD5 mismatch for ${record.filename}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const localBlocksDir = path.join(dataDir, "scene_blocks");
|
||||
await fs.rm(localBlocksDir, { recursive: true, force: true });
|
||||
await fs.mkdir(path.dirname(localBlocksDir), { recursive: true });
|
||||
await fs.rename(tempBlocksDir, localBlocksDir);
|
||||
|
||||
const tempPersonaPath = path.join(tempDir, "persona.md");
|
||||
const localPersonaPath = path.join(dataDir, "persona.md");
|
||||
try {
|
||||
await fs.access(tempPersonaPath);
|
||||
await fs.rm(localPersonaPath, { force: true });
|
||||
await fs.rename(tempPersonaPath, localPersonaPath);
|
||||
} catch {
|
||||
await fs.rm(localPersonaPath, { force: true });
|
||||
}
|
||||
|
||||
await syncSceneIndex(dataDir);
|
||||
await refreshPersonaNavigation(dataDir);
|
||||
logger.debug?.(`[memory-tdai][profile-sync] Pulled ${records.length} profile(s) to local cache`);
|
||||
return baseline;
|
||||
} finally {
|
||||
await fs.rm(tempDir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
export async function syncLocalProfilesToStore(
|
||||
dataDir: string,
|
||||
store: IMemoryStore,
|
||||
baselineMap: Map<string, ProfileBaseline>,
|
||||
logger: Logger,
|
||||
): Promise<void> {
|
||||
const localProfiles = await listLocalProfiles(dataDir);
|
||||
const localIds = new Set(localProfiles.map((profile) => profile.id));
|
||||
|
||||
const syncRecords: ProfileSyncRecord[] = localProfiles
|
||||
.filter((profile) => baselineMap.get(profile.id)?.contentMd5 !== profile.contentMd5 || !baselineMap.has(profile.id))
|
||||
.map((profile) => ({
|
||||
...profile,
|
||||
baselineVersion: baselineMap.get(profile.id)?.version,
|
||||
}));
|
||||
|
||||
if (syncRecords.length > 0 && store.syncProfiles) {
|
||||
await store.syncProfiles(syncRecords);
|
||||
logger.info(`[memory-tdai][profile-sync] Synced ${syncRecords.length} changed profile(s)`);
|
||||
}
|
||||
|
||||
const deletedIds = [...baselineMap.keys()].filter((id) => !localIds.has(id));
|
||||
if (deletedIds.length > 0 && store.deleteProfiles) {
|
||||
await store.deleteProfiles(deletedIds);
|
||||
logger.info(`[memory-tdai][profile-sync] Deleted ${deletedIds.length} stale profile(s)`);
|
||||
}
|
||||
}
|
||||
|
||||
export async function ensureL2L3Local(
|
||||
dataDir: string,
|
||||
store: IMemoryStore,
|
||||
logger: Logger,
|
||||
): Promise<Map<string, ProfileBaseline>> {
|
||||
if (!store.pullProfiles) return new Map();
|
||||
return pullProfilesToLocal(dataDir, store, logger);
|
||||
}
|
||||
@@ -8,8 +8,10 @@
|
||||
*
|
||||
* Security: The LLM is sandboxed to scene_blocks/ only (workspaceDir = scene_blocks/).
|
||||
* It has NO visibility into checkpoint, scene_index, persona.md, or any other system file.
|
||||
* File deletion is achieved via "soft-delete" — writing an empty string to the file
|
||||
* — and the SceneExtractor subsequently removes empty files with fs.unlink.
|
||||
* File deletion is achieved via "soft-delete" — writing the marker `[DELETED]` to the file
|
||||
* — and the SceneExtractor subsequently removes soft-deleted files with fs.unlink.
|
||||
* Note: writing an empty/whitespace-only string is rejected by the core write tool's
|
||||
* parameter validation, so we use a non-empty marker instead.
|
||||
*
|
||||
* Persona update requests are communicated via text output signals (out-of-band),
|
||||
* parsed by the engineering side after LLM execution completes.
|
||||
@@ -22,6 +24,8 @@ export interface SceneExtractionPromptParams {
|
||||
sceneCountWarning?: string;
|
||||
/** List of existing scene filenames (relative, e.g. ["work.md", "hobby.md"]) */
|
||||
existingSceneFiles?: string[];
|
||||
/** Maximum number of scene blocks allowed */
|
||||
maxScenes: number;
|
||||
}
|
||||
|
||||
export function buildSceneExtractionPrompt(params: SceneExtractionPromptParams): string {
|
||||
@@ -31,6 +35,7 @@ export function buildSceneExtractionPrompt(params: SceneExtractionPromptParams):
|
||||
currentTimestamp,
|
||||
sceneCountWarning,
|
||||
existingSceneFiles,
|
||||
maxScenes,
|
||||
} = params;
|
||||
|
||||
const warningSection = sceneCountWarning
|
||||
@@ -51,7 +56,7 @@ export function buildSceneExtractionPrompt(params: SceneExtractionPromptParams):
|
||||
|
||||
### Layer 2 (Processing): Scene Diaries
|
||||
- **形态**：**不是清单，是连贯的叙事文档**
|
||||
- **逻辑**：将 L1 碎片融合进特定场景文件（强制限制在15个以内）
|
||||
- **逻辑**：将 L1 碎片融合进特定场景文件
|
||||
- **动作**：Create（创建）、Integrate（整合）、Rewrite（重写）
|
||||
- **禁止**：简单追加列表
|
||||
|
||||
@@ -63,6 +68,8 @@ ${warningSection}
|
||||
2. 现有 Block 映射表 (Existing Blocks Map): 包含当前所有记忆块（Markdown 文件）的文件名和摘要的列表。
|
||||
3. 当前时间 (Current Time): 用于生成元数据的具体时间戳。
|
||||
|
||||
**⚠️ 场景文件数量上限：${maxScenes} 个。处理完成后目录中的场景文件数量必须严格小于此上限。**
|
||||
|
||||
|
||||
### 1️⃣ New Memories List
|
||||
${memoriesJson}
|
||||
@@ -86,6 +93,8 @@ ${existingSceneFiles.map((f) => `- \`${f}\``).join("\n")}
|
||||
3. **创建新场景文件时**，直接使用文件名，如 \`新场景名.md\`
|
||||
4. **场景文件支持 replace_in_file**。对于局部更新（如只更新某个章节或 META 字段），可以使用 \`replace_in_file\` 进行精确替换。对于大范围重写或结构性变更，建议使用 \`read_file\` + \`write_to_file\` 整体重写。
|
||||
5. **场景索引和系统配置由工程系统自动维护**，你只需专注于操作 \`.md\` 场景文件
|
||||
6. **删除文件的唯一方式**：使用 \`write_to_file(filename, '[DELETED]')\` 将文件内容写为 \`[DELETED]\` 标记。系统会自动清理带有此标记的文件。**禁止**写入空字符串（会被系统拒绝）。**禁止**用 \`[ARCHIVE]\`、\`[CONSOLIDATED]\` 等其他标记替代删除——只有 \`[DELETED]\` 标记会触发系统清理。
|
||||
7. **禁止创建报告/整合/汇总类文件**。你的输出必须是有意义的场景叙事文件（如"技术架构与工程实践.md"、"日常生活与工作节奏.md"）。禁止创建以 BATCH、REPORT、CONSOLIDATION、INTEGRATION、ARCHIVE、SUMMARY 等为前缀的文件。
|
||||
|
||||
## 工作流与逻辑 (Workflow & Logic)
|
||||
在生成输出之前，你必须执行以下"思维链"过程：
|
||||
@@ -94,11 +103,12 @@ ${existingSceneFiles.map((f) => `- \`${f}\``).join("\n")}
|
||||
|
||||
**在处理任何记忆之前，你必须：**
|
||||
|
||||
1. **统计当前场景总数**：检查 "Existing Scene Blocks Summary" 中的场景数量
|
||||
2. **遵守分级预警，上限为15个block**：
|
||||
- 红色预警（≥ 15）：**必须先合并**，将最相似的 2-4 个场景合并为 1 个，然后再处理新记忆
|
||||
- 橙色预警（= 15-1）：**只能 UPDATE 现有场景，不能 CREATE 新场景**
|
||||
- 黄色预警（接近15）：**优先 UPDATE 或主动 MERGE 相似场景**
|
||||
1. **统计当前场景总数**：查看 "Existing Scene Blocks Summary" 顶部标注的当前场景总数
|
||||
2. **最终目标**：处理完成后，目录中的场景文件数量必须 **严格小于 ${maxScenes}**
|
||||
3. **遵守分级预警**：
|
||||
- 红色预警（≥ ${maxScenes}）：**必须先通过 MERGE 减少文件数量**，将最相似的 2-4 个场景合并为 1 个，**并删除被合并的旧文件**，直到文件数 < ${maxScenes} 后，再处理新记忆
|
||||
- 橙色预警（= ${maxScenes - 1}）：**只能 UPDATE 现有场景，不能 CREATE 新场景**
|
||||
- 黄色预警（接近 ${maxScenes}）：**优先 UPDATE 或主动 MERGE 相似场景**
|
||||
|
||||
**合并优先级**（当需要合并时，按以下顺序选择）：
|
||||
1. **主题高度重叠**：如"Python后端开发"和"Go后端开发" → 合并为"后端开发技术栈"
|
||||
@@ -120,10 +130,11 @@ ${existingSceneFiles.map((f) => `- \`${f}\``).join("\n")}
|
||||
1. **UPDATE（更新）**【首选策略】: 如果存在相关的 Block（基于摘要或文件名的相似性），先 read 文件内的具体信息，再锁定该 Block 进行更新（write 或 replace）
|
||||
2. **MERGE（合并）**:
|
||||
- 合并的新 block 应该是生成概括性更强的场景，包含已有的多个相似场景
|
||||
- **强制合并**：当前 Block 总数 **≥ 上限**时，必须先将多个相似记忆合并，或者删除最旧或者最不重要的 block
|
||||
- **强制合并**：当前 Block 总数 **≥ ${maxScenes}** 时，必须先将多个相似记忆合并
|
||||
- **主动合并**：即使未达上限，如果两个 Block 属于同一叙事弧线，也应合并以增加深度
|
||||
- **⚠️ 合并后必须删除旧文件**：被合并的旧场景文件必须通过 \`write_to_file(旧文件名, '[DELETED]')\` 写入删除标记。**仅仅打标记（如 [ARCHIVE]、[CONSOLIDATED]）不算删除，文件仍会占用配额。**
|
||||
3. **CREATE（新建）**【最后手段】:
|
||||
- **前提条件**：当前场景总数未达上限
|
||||
- **前提条件**：当前场景总数 < ${maxScenes}
|
||||
- **CREATE 前的强制验证**：必须先用 \`read_file\` 检查至少 2 个最相似的现有场景，确认新记忆确实无法融入后才能 CREATE。跳过验证直接 CREATE 是被禁止的
|
||||
- 如果话题是全新的且与现有内容区分度高，可以创建新 Block
|
||||
- **每次批处理最多新增 1 个场景**
|
||||
@@ -135,14 +146,15 @@ ${existingSceneFiles.map((f) => `- \`${f}\``).join("\n")}
|
||||
3. \`write_to_file('Python后端开发.md', B)\` → **整体重写该场景文件**
|
||||
或 \`replace_in_file('Python后端开发.md', old_section, new_section)\` → **局部更新某部分**
|
||||
|
||||
合并多个 block 的逻辑：
|
||||
**示例 B：合并多个 block（MERGE — 合并后必须删除旧文件）**
|
||||
**具体操作步骤（工具调用）**：
|
||||
1. \`read_file('Python后端开发.md')\` → 获取内容 A
|
||||
2. \`read_file('Go后端开发.md')\` → 获取内容 B
|
||||
3. 整合 A + B + 新记忆 → 生成新内容 C（\`heat = heatA + heatB + 1\`）
|
||||
4. \`write_to_file('后端开发技术栈.md', C)\` → 创建新文件，写入合并后的完整内容
|
||||
5. \`write_to_file('Python后端开发.md', '')\` → **清空旧文件 A（标记删除）**
|
||||
6. \`write_to_file('Go后端开发.md', '')\` → **清空旧文件 B（标记删除）**
|
||||
4. \`write_to_file('后端开发技术栈.md', C)\` → 创建合并后的新文件
|
||||
5. \`write_to_file('Python后端开发.md', '[DELETED]')\` → **⚠️ 删除旧文件 A（写 [DELETED] 标记）**
|
||||
6. \`write_to_file('Go后端开发.md', '[DELETED]')\` → **⚠️ 删除旧文件 B（写 [DELETED] 标记）**
|
||||
**关键**：步骤 5-6 是必须的！不执行删除 = 文件总数不减少 = 合并无效。
|
||||
|
||||
### 阶段 3：撰写与合成（核心任务）
|
||||
深度整合: 严禁简单的文本追加。你必须结合上下文（基于摘要或提供的原始内容）重写叙事，将新信息自然地融入其中。
|
||||
@@ -224,5 +236,5 @@ reason: 具体原因描述
|
||||
- 使用 \`read_file\` 读取需要更新的场景文件
|
||||
- 使用 \`write_to_file\` 创建新文件或**整体重写**已有场景文件
|
||||
- 使用 \`replace_in_file\` 对场景文件进行**局部更新**（如只更新某个章节）
|
||||
- **删除文件**：使用 \`write_to_file(filename, '')\` 将文件内容清空（系统会自动清理空文件，禁止使用其他删除方式）`;
|
||||
- **删除文件**：使用 \`write_to_file(filename, '[DELETED]')\` 将文件内容写为 **\`[DELETED]\` 标记**。系统会自动清理这些文件。**重要**：只有 \`[DELETED]\` 标记才会触发系统清理。写入空字符串会被系统拒绝，写入 \`[ARCHIVE]\`、\`[CONSOLIDATED]\` 等标记**不会删除文件**，文件会继续占用场景配额。`;
|
||||
}
|
||||
|
||||
+17
-12
@@ -16,8 +16,8 @@ import { CONFLICT_DETECTION_SYSTEM_PROMPT, formatBatchConflictPrompt } from "../
|
||||
import type { CandidateMatch } from "../prompts/l1-dedup.js";
|
||||
import { CleanContextRunner } from "../utils/clean-context-runner.js";
|
||||
import { sanitizeJsonForParse } from "../utils/sanitize.js";
|
||||
import type { VectorStore } from "../store/vector-store.js";
|
||||
import { buildFtsQuery } from "../store/vector-store.js";
|
||||
import type { IMemoryStore } from "../store/types.js";
|
||||
import { buildFtsQuery } from "../store/sqlite.js";
|
||||
import type { EmbeddingService } from "../store/embedding.js";
|
||||
|
||||
interface Logger {
|
||||
@@ -60,7 +60,7 @@ export async function batchDedup(params: {
|
||||
logger?: Logger;
|
||||
model?: string;
|
||||
/** Vector store for cosine similarity candidate recall */
|
||||
vectorStore?: VectorStore;
|
||||
vectorStore?: IMemoryStore;
|
||||
/** Embedding service for computing query vectors */
|
||||
embeddingService?: EmbeddingService;
|
||||
/** Top-K candidates per new memory (default: 5) */
|
||||
@@ -81,7 +81,7 @@ export async function batchDedup(params: {
|
||||
}));
|
||||
|
||||
// Determine what recall capabilities are available
|
||||
const hasVectorData = vectorStore && vectorStore.count() > 0;
|
||||
const hasVectorData = vectorStore && (await vectorStore.countL1()) > 0;
|
||||
const hasFts = vectorStore?.isFtsAvailable() ?? false;
|
||||
|
||||
// Fast path: no recall capability at all → skip dedup
|
||||
@@ -109,7 +109,7 @@ export async function batchDedup(params: {
|
||||
);
|
||||
// Degrade to FTS keyword recall
|
||||
if (hasFts) {
|
||||
matches = findCandidatesByFts(memories, vectorStore!, logger);
|
||||
matches = await findCandidatesByFts(memories, vectorStore!, logger);
|
||||
} else {
|
||||
logger?.debug?.(`${TAG} FTS not available either, skipping conflict detection`);
|
||||
return storeAll();
|
||||
@@ -118,7 +118,7 @@ export async function batchDedup(params: {
|
||||
} else if (hasFts) {
|
||||
// === Tier 2: FTS keyword recall ===
|
||||
logger?.debug?.(`${TAG} Using FTS keyword recall mode (no embedding service or no vector data)`);
|
||||
matches = findCandidatesByFts(memories, vectorStore!, logger);
|
||||
matches = await findCandidatesByFts(memories, vectorStore!, logger);
|
||||
} else {
|
||||
// Shouldn't reach here given the fast-path check above, but be defensive
|
||||
logger?.debug?.(`${TAG} No usable recall path, skipping conflict detection`);
|
||||
@@ -191,7 +191,7 @@ async function runLlmJudgment(
|
||||
*/
|
||||
async function findCandidatesByVector(
|
||||
memories: Array<ExtractedMemory & { record_id: string }>,
|
||||
vectorStore: VectorStore,
|
||||
vectorStore: IMemoryStore,
|
||||
embeddingService: EmbeddingService,
|
||||
topK: number,
|
||||
logger?: Logger,
|
||||
@@ -209,7 +209,7 @@ async function findCandidatesByVector(
|
||||
const queryVec = embeddings[i];
|
||||
|
||||
// Vector search top-K (request extra to account for self-batch filtering)
|
||||
const searchResults = vectorStore.search(queryVec, topK + memories.length);
|
||||
const searchResults = await vectorStore.searchL1Vector(queryVec, topK + memories.length, mem.content);
|
||||
|
||||
// Exclude records from current batch, convert to MemoryRecord format
|
||||
const candidates: MemoryRecord[] = searchResults
|
||||
@@ -245,18 +245,18 @@ async function findCandidatesByVector(
|
||||
* Uses the FTS index for efficient BM25-ranked keyword matching.
|
||||
* This replaces the old Jaccard word-overlap fallback entirely.
|
||||
*/
|
||||
function findCandidatesByFts(
|
||||
async function findCandidatesByFts(
|
||||
memories: Array<ExtractedMemory & { record_id: string }>,
|
||||
vectorStore: VectorStore,
|
||||
vectorStore: IMemoryStore,
|
||||
_logger?: Logger,
|
||||
): CandidateMatch[] {
|
||||
): Promise<CandidateMatch[]> {
|
||||
const newRecordIds = new Set(memories.map((m) => m.record_id));
|
||||
const matches: CandidateMatch[] = [];
|
||||
|
||||
for (const mem of memories) {
|
||||
const ftsQuery = buildFtsQuery(mem.content);
|
||||
if (ftsQuery) {
|
||||
const ftsResults = vectorStore.ftsSearchL1(ftsQuery, 10);
|
||||
const ftsResults = await vectorStore.searchL1Fts(ftsQuery, 10);
|
||||
// Filter out records from the current batch
|
||||
const candidates: MemoryRecord[] = ftsResults
|
||||
.filter((r) => !newRecordIds.has(r.record_id))
|
||||
@@ -333,6 +333,11 @@ function parseBatchResult(
|
||||
const d = item as Record<string, unknown>;
|
||||
|
||||
const recordId = String(d.record_id ?? "");
|
||||
// Skip entries with empty/missing record_id — they are LLM hallucinations
|
||||
if (!recordId) {
|
||||
logger?.debug?.(`${TAG} Skipping decision with empty record_id`);
|
||||
continue;
|
||||
}
|
||||
const action = String(d.action ?? "store");
|
||||
|
||||
if (!validActions.includes(action)) {
|
||||
|
||||
@@ -19,7 +19,7 @@ import { writeMemory, generateMemoryId } from "./l1-writer.js";
|
||||
import type { ExtractedMemory, MemoryRecord, MemoryType, DedupDecision } from "./l1-writer.js";
|
||||
import { CleanContextRunner } from "../utils/clean-context-runner.js";
|
||||
import { sanitizeJsonForParse, shouldExtractL1 } from "../utils/sanitize.js";
|
||||
import type { VectorStore } from "../store/vector-store.js";
|
||||
import type { IMemoryStore } from "../store/types.js";
|
||||
import type { EmbeddingService } from "../store/embedding.js";
|
||||
import { report } from "../report/reporter.js";
|
||||
|
||||
@@ -98,7 +98,7 @@ export async function extractL1Memories(params: {
|
||||
/** Previous scene name for continuity */
|
||||
previousSceneName?: string;
|
||||
/** Vector store for cosine similarity candidate recall */
|
||||
vectorStore?: VectorStore;
|
||||
vectorStore?: IMemoryStore;
|
||||
/** Embedding service for computing query vectors */
|
||||
embeddingService?: EmbeddingService;
|
||||
/** Top-K candidates for conflict recall (default: 5) */
|
||||
@@ -394,7 +394,7 @@ async function applyDecisions(params: {
|
||||
sessionKey: string;
|
||||
sessionId?: string;
|
||||
logger?: Logger;
|
||||
vectorStore?: VectorStore;
|
||||
vectorStore?: IMemoryStore;
|
||||
embeddingService?: EmbeddingService;
|
||||
}): Promise<MemoryRecord[]> {
|
||||
const { memoriesWithIds, decisions, baseDir, sessionKey, sessionId, logger, vectorStore, embeddingService } = params;
|
||||
@@ -447,7 +447,7 @@ async function storeAllDirectly(
|
||||
sessionKey: string,
|
||||
sessionId: string | undefined,
|
||||
logger?: Logger,
|
||||
vectorStore?: VectorStore,
|
||||
vectorStore?: IMemoryStore,
|
||||
embeddingService?: EmbeddingService,
|
||||
): Promise<MemoryRecord[]> {
|
||||
const storedRecords: MemoryRecord[] = [];
|
||||
|
||||
@@ -14,11 +14,11 @@
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import type { MemoryRecord, MemoryType, EpisodicMetadata } from "./l1-writer.js";
|
||||
import type { VectorStore, L1RecordRow, L1QueryFilter } from "../store/vector-store.js";
|
||||
import type { IMemoryStore, L1RecordRow, L1QueryFilter } from "../store/types.js";
|
||||
|
||||
// Re-export types that readers need
|
||||
export type { MemoryRecord, MemoryType, EpisodicMetadata } from "./l1-writer.js";
|
||||
export type { L1QueryFilter } from "../store/vector-store.js";
|
||||
export type { L1QueryFilter } from "../store/types.js";
|
||||
|
||||
interface Logger {
|
||||
debug?: (message: string) => void;
|
||||
@@ -44,17 +44,17 @@ const TAG = "[memory-tdai] [l1-reader]";
|
||||
*
|
||||
* Falls back to empty array if VectorStore is null or degraded.
|
||||
*/
|
||||
export function queryMemoryRecords(
|
||||
vectorStore: VectorStore | null | undefined,
|
||||
export async function queryMemoryRecords(
|
||||
vectorStore: IMemoryStore | null | undefined,
|
||||
filter?: L1QueryFilter,
|
||||
logger?: Logger,
|
||||
): MemoryRecord[] {
|
||||
): Promise<MemoryRecord[]> {
|
||||
if (!vectorStore) {
|
||||
logger?.warn(`${TAG} queryMemoryRecords: no VectorStore available, returning empty`);
|
||||
return [];
|
||||
}
|
||||
|
||||
const rows = vectorStore.queryL1Records(filter);
|
||||
const rows = await vectorStore.queryL1Records(filter);
|
||||
return rows.map(rowToMemoryRecord);
|
||||
}
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import crypto from "node:crypto";
|
||||
import type { VectorStore } from "../store/vector-store.js";
|
||||
import type { IMemoryStore } from "../store/types.js";
|
||||
import type { EmbeddingService } from "../store/embedding.js";
|
||||
|
||||
// ============================
|
||||
@@ -149,7 +149,7 @@ export async function writeMemory(params: {
|
||||
sessionId?: string;
|
||||
logger?: Logger;
|
||||
/** Optional vector store for dual-write (JSONL + vector DB) */
|
||||
vectorStore?: VectorStore;
|
||||
vectorStore?: IMemoryStore;
|
||||
/** Optional embedding service (required when vectorStore is provided) */
|
||||
embeddingService?: EmbeddingService;
|
||||
}): Promise<MemoryRecord | null> {
|
||||
@@ -208,7 +208,7 @@ export async function writeMemory(params: {
|
||||
// by memory-cleaner (which reconciles against VectorStore as source of truth).
|
||||
if (vectorStore) {
|
||||
try {
|
||||
vectorStore.deleteBatch(decision.target_ids);
|
||||
await vectorStore.deleteL1Batch(decision.target_ids);
|
||||
logger?.debug?.(`${TAG} VectorStore: deleted ${decision.target_ids.length} target record(s) for ${decision.action}`);
|
||||
} catch (err) {
|
||||
logger?.warn?.(
|
||||
@@ -251,7 +251,7 @@ export async function writeMemory(params: {
|
||||
}
|
||||
}
|
||||
|
||||
const upsertOk = vectorStore.upsert(record, embedding);
|
||||
const upsertOk = await vectorStore.upsertL1(record, embedding);
|
||||
logger?.debug?.(`${TAG} [vec-dual-write] upsert result=${upsertOk} id=${record.id}`);
|
||||
} catch (err) {
|
||||
// Vector write failure should NOT block the main JSONL write
|
||||
|
||||
+10
-2
@@ -19,7 +19,7 @@ let _reporter: IReporter | undefined;
|
||||
export function initReporter(opts: {
|
||||
enabled: boolean;
|
||||
type: string;
|
||||
logger: { info: (msg: string) => void };
|
||||
logger: { info: (msg: string) => void; debug?: (msg: string) => void };
|
||||
instanceId: string;
|
||||
pluginVersion: string;
|
||||
}): void {
|
||||
@@ -31,7 +31,7 @@ export function initReporter(opts: {
|
||||
break;
|
||||
// TODO: add new reporter type
|
||||
default:
|
||||
opts.logger.info(`[memory-tdai] Unknown reporter type "${opts.type}", disabled reporting`);
|
||||
opts.logger.debug?.(`[memory-tdai] Unknown reporter type "${opts.type}", disabled reporting`);
|
||||
break;
|
||||
}
|
||||
}
|
||||
@@ -40,6 +40,14 @@ export function setReporter(reporter: IReporter): void {
|
||||
_reporter = reporter;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset the reporter singleton so that the next `initReporter` call takes effect.
|
||||
* Must be called at plugin re-registration (hot-reload) to pick up config changes.
|
||||
*/
|
||||
export function resetReporter(): void {
|
||||
_reporter = undefined;
|
||||
}
|
||||
|
||||
export function report(event: string, data: ReportPayload): void {
|
||||
if (!_reporter) return;
|
||||
try {
|
||||
|
||||
@@ -89,7 +89,7 @@ export class SceneExtractor {
|
||||
|
||||
constructor(opts: SceneExtractorOptions) {
|
||||
this.dataDir = opts.dataDir;
|
||||
this.maxScenes = opts.maxScenes ?? 20;
|
||||
this.maxScenes = opts.maxScenes ?? 15;
|
||||
this.sceneBackupCount = opts.sceneBackupCount ?? 10;
|
||||
this.timeoutMs = opts.timeoutMs ?? 300_000; // 5 min — LLM may do multiple tool calls
|
||||
this.logger = opts.logger;
|
||||
@@ -190,6 +190,7 @@ export class SceneExtractor {
|
||||
currentTimestamp,
|
||||
sceneCountWarning,
|
||||
existingSceneFiles,
|
||||
maxScenes: this.maxScenes,
|
||||
});
|
||||
this.logger?.debug?.(`${TAG} extract() prompt built: ${prompt.length} chars (${Date.now() - promptStartMs}ms)`);
|
||||
|
||||
@@ -218,20 +219,34 @@ export class SceneExtractor {
|
||||
// Phase 5: Subsequent processing — safe cleanup of soft-deleted files
|
||||
//
|
||||
// Security: The LLM has no `exec` tool and cannot run shell commands.
|
||||
// Instead, it "deletes" files by writing empty content (soft-delete).
|
||||
// Here we detect and remove those empty files before syncing the index,
|
||||
// so syncSceneIndex won't re-index stale empty entries.
|
||||
// Instead, it "deletes" files by writing the marker `[DELETED]` to the file
|
||||
// (writing empty/whitespace-only content is rejected by core's write tool
|
||||
// parameter validation). Here we detect and remove those soft-deleted files
|
||||
// before syncing the index, so syncSceneIndex won't re-index stale entries.
|
||||
//
|
||||
// We also detect "META-only" files — files that contain only a META header
|
||||
// (e.g. [ARCHIVE] or [CONSOLIDATED] markers) but no actual scene content.
|
||||
// These are artifacts of LLM merges that didn't properly delete old files.
|
||||
const cleanupStartMs = Date.now();
|
||||
let cleanedCount = 0;
|
||||
try {
|
||||
const allFiles = (await fs.readdir(sceneBlocksDir)).filter((f) => f.endsWith(".md"));
|
||||
for (const file of allFiles) {
|
||||
const filePath = path.join(sceneBlocksDir, file);
|
||||
const content = await fs.readFile(filePath, "utf-8");
|
||||
if (content.trim().length === 0) {
|
||||
const raw = await fs.readFile(filePath, "utf-8");
|
||||
if (raw.trim().length === 0 || raw.trim() === "[DELETED]") {
|
||||
// Empty file or [DELETED] marker — soft-delete
|
||||
await fs.unlink(filePath);
|
||||
cleanedCount++;
|
||||
this.logger?.debug?.(`${TAG} extract() removed soft-deleted file: ${file}`);
|
||||
} else {
|
||||
// Check if file has only META header but no actual content
|
||||
const block = parseSceneBlock(raw, file);
|
||||
if (!block.content || block.content.trim().length === 0) {
|
||||
await fs.unlink(filePath);
|
||||
cleanedCount++;
|
||||
this.logger?.debug?.(`${TAG} extract() removed META-only file (no content): ${file}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (cleanupErr) {
|
||||
|
||||
@@ -0,0 +1,435 @@
|
||||
/**
|
||||
* Input loading, validation, normalization, and timestamp handling for the `seed` command.
|
||||
*
|
||||
* Responsibilities:
|
||||
* 1. Load raw JSON from file
|
||||
* 2. Detect Format A (`{ sessions: [...] }`) vs Format B (`[...]`)
|
||||
* 3. Six-layer validation (file → top-level → session → round → message → timestamp consistency)
|
||||
* 4. Normalize into NormalizedInput with auto-generated sessionIds
|
||||
* 5. Timestamp all-or-none check + fill strategy
|
||||
*/
|
||||
|
||||
import fs from "node:fs";
|
||||
import crypto from "node:crypto";
|
||||
import type {
|
||||
RawSession,
|
||||
FormatA,
|
||||
ValidationError,
|
||||
NormalizedInput,
|
||||
NormalizedSession,
|
||||
NormalizedRound,
|
||||
NormalizedMessage,
|
||||
SeedCommandOptions,
|
||||
} from "./types.js";
|
||||
|
||||
// ============================
|
||||
// Public API
|
||||
// ============================
|
||||
|
||||
export interface LoadAndValidateResult {
|
||||
/** Normalized input ready for pipeline consumption. */
|
||||
input: NormalizedInput;
|
||||
/** Whether the user needs to confirm timestamp auto-fill. */
|
||||
needsTimestampConfirmation: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Load, validate, and normalize seed input from a file.
|
||||
*
|
||||
* Throws on fatal validation errors with a human-readable message
|
||||
* that includes all collected errors.
|
||||
*/
|
||||
export function loadAndValidateInput(
|
||||
opts: Pick<SeedCommandOptions, "input" | "sessionKey" | "strictRoundRole">,
|
||||
): LoadAndValidateResult {
|
||||
// Layer 1: File — read + parse
|
||||
const raw = loadRawInput(opts.input);
|
||||
|
||||
// Layer 2: Top-level — detect A vs B
|
||||
const sessions = extractSessions(raw);
|
||||
|
||||
// Layers 3-5: session / round / message validation
|
||||
const errors: ValidationError[] = [];
|
||||
validateSessions(sessions, opts.strictRoundRole, errors);
|
||||
|
||||
if (errors.length > 0) {
|
||||
throw new SeedValidationError(errors);
|
||||
}
|
||||
|
||||
// Layer 6: Timestamp consistency (all-have / all-missing / mixed → error)
|
||||
const tsResult = checkTimestampConsistency(sessions);
|
||||
if (tsResult.status === "mixed") {
|
||||
throw new SeedValidationError([{
|
||||
stage: "timestamp_consistency",
|
||||
message:
|
||||
"Timestamp consistency check failed: some messages have timestamps while others do not. " +
|
||||
"All messages must either have timestamps or none must have timestamps.",
|
||||
}]);
|
||||
}
|
||||
|
||||
// Normalize
|
||||
const normalized = normalizeSessions(sessions, opts.sessionKey);
|
||||
|
||||
return {
|
||||
input: {
|
||||
sessions: normalized.sessions,
|
||||
totalRounds: normalized.totalRounds,
|
||||
totalMessages: normalized.totalMessages,
|
||||
hasTimestamps: tsResult.status === "all_present",
|
||||
},
|
||||
needsTimestampConfirmation: tsResult.status === "all_missing",
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Fill timestamps for all messages when the input has no timestamps.
|
||||
*
|
||||
* Uses a single monotonically increasing counter across ALL sessions
|
||||
* to guarantee global timestamp ordering. This is critical when multiple
|
||||
* sessions share the same sessionKey — the L0 capture cursor (advanced
|
||||
* per-session) would filter out later sessions whose timestamps fall
|
||||
* below the cursor if ordering were not globally monotonic.
|
||||
*/
|
||||
export function fillTimestamps(input: NormalizedInput): void {
|
||||
let currentTs = Date.now();
|
||||
for (const session of input.sessions) {
|
||||
for (const round of session.rounds) {
|
||||
for (let i = 0; i < round.messages.length; i++) {
|
||||
// Small offset per message to maintain strict ordering
|
||||
round.messages[i]!.timestamp = currentTs;
|
||||
currentTs += 100;
|
||||
}
|
||||
}
|
||||
}
|
||||
input.hasTimestamps = true;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Validation error class
|
||||
// ============================
|
||||
|
||||
export class SeedValidationError extends Error {
|
||||
public readonly errors: ValidationError[];
|
||||
|
||||
constructor(errors: ValidationError[]) {
|
||||
const summary = errors.map((e) => formatValidationError(e)).join("\n");
|
||||
super(`Seed input validation failed (${errors.length} error(s)):\n${summary}`);
|
||||
this.name = "SeedValidationError";
|
||||
this.errors = errors;
|
||||
}
|
||||
}
|
||||
|
||||
function formatValidationError(e: ValidationError): string {
|
||||
const parts: string[] = [` [${e.stage}]`];
|
||||
if (e.sourceIndex != null) parts.push(`session[${e.sourceIndex}]`);
|
||||
if (e.sessionKey) parts.push(`key="${e.sessionKey}"`);
|
||||
if (e.roundIndex != null) parts.push(`round[${e.roundIndex}]`);
|
||||
if (e.messageIndex != null) parts.push(`msg[${e.messageIndex}]`);
|
||||
parts.push(e.message);
|
||||
return parts.join(" ");
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Layer 1: File loading
|
||||
// ============================
|
||||
|
||||
function loadRawInput(filePath: string): unknown {
|
||||
if (!fs.existsSync(filePath)) {
|
||||
throw new SeedValidationError([{
|
||||
stage: "file",
|
||||
message: `Input file not found: ${filePath}`,
|
||||
}]);
|
||||
}
|
||||
|
||||
const content = fs.readFileSync(filePath, "utf-8").trim();
|
||||
if (!content) {
|
||||
throw new SeedValidationError([{
|
||||
stage: "file",
|
||||
message: "Input file is empty.",
|
||||
}]);
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(content);
|
||||
} catch (err) {
|
||||
throw new SeedValidationError([{
|
||||
stage: "file",
|
||||
message: `JSON parse error: ${err instanceof Error ? err.message : String(err)}`,
|
||||
}]);
|
||||
}
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Layer 2: Top-level format detection
|
||||
// ============================
|
||||
|
||||
function extractSessions(raw: unknown): RawSession[] {
|
||||
// Format A: { sessions: [...] }
|
||||
if (
|
||||
raw != null &&
|
||||
typeof raw === "object" &&
|
||||
!Array.isArray(raw) &&
|
||||
"sessions" in raw
|
||||
) {
|
||||
const obj = raw as FormatA;
|
||||
if (!Array.isArray(obj.sessions)) {
|
||||
throw new SeedValidationError([{
|
||||
stage: "top_level",
|
||||
message: 'Format A detected but "sessions" is not an array.',
|
||||
}]);
|
||||
}
|
||||
return obj.sessions;
|
||||
}
|
||||
|
||||
// Format B: [...]
|
||||
if (Array.isArray(raw)) {
|
||||
return raw as RawSession[];
|
||||
}
|
||||
|
||||
throw new SeedValidationError([{
|
||||
stage: "top_level",
|
||||
message:
|
||||
"Unrecognized input format. Expected either:\n" +
|
||||
' Format A: { "sessions": [...] }\n' +
|
||||
" Format B: [ { sessionKey, conversations }, ... ]",
|
||||
}]);
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Layers 3-5: session / round / message validation
|
||||
// ============================
|
||||
|
||||
function validateSessions(
|
||||
sessions: RawSession[],
|
||||
strictRoundRole: boolean,
|
||||
errors: ValidationError[],
|
||||
): void {
|
||||
if (sessions.length === 0) {
|
||||
errors.push({
|
||||
stage: "session",
|
||||
message: "No sessions found in input.",
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
for (let si = 0; si < sessions.length; si++) {
|
||||
const session = sessions[si]!;
|
||||
|
||||
// Layer 3: session validation
|
||||
if (!session.sessionKey || typeof session.sessionKey !== "string" || session.sessionKey.trim() === "") {
|
||||
errors.push({
|
||||
stage: "session",
|
||||
sourceIndex: si,
|
||||
message: '"sessionKey" is required and must be a non-empty string.',
|
||||
});
|
||||
}
|
||||
|
||||
if (!Array.isArray(session.conversations)) {
|
||||
errors.push({
|
||||
stage: "session",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
message: '"conversations" must be a two-dimensional array (array of rounds).',
|
||||
});
|
||||
continue; // Can't validate rounds
|
||||
}
|
||||
|
||||
// Check that conversations is a 2D array
|
||||
for (let ri = 0; ri < session.conversations.length; ri++) {
|
||||
const round = session.conversations[ri];
|
||||
|
||||
// Layer 4: round validation
|
||||
if (!Array.isArray(round)) {
|
||||
errors.push({
|
||||
stage: "round",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
message: "Round must be an array of messages.",
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
if (round.length === 0) {
|
||||
errors.push({
|
||||
stage: "round",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
message: "Round must be a non-empty array.",
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
// Strict round-role: each round must have at least one user and one assistant
|
||||
if (strictRoundRole) {
|
||||
const roles = new Set(round.map((m) => m.role));
|
||||
if (!roles.has("user")) {
|
||||
errors.push({
|
||||
stage: "round",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
message: '--strict-round-role: round must contain at least one "user" message.',
|
||||
});
|
||||
}
|
||||
if (!roles.has("assistant")) {
|
||||
errors.push({
|
||||
stage: "round",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
message: '--strict-round-role: round must contain at least one "assistant" message.',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Layer 5: message validation
|
||||
for (let mi = 0; mi < round.length; mi++) {
|
||||
const msg = round[mi]!;
|
||||
|
||||
if (!msg.role || typeof msg.role !== "string") {
|
||||
errors.push({
|
||||
stage: "message",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
messageIndex: mi,
|
||||
message: '"role" is required and must be a non-empty string.',
|
||||
});
|
||||
}
|
||||
|
||||
if (!msg.content || typeof msg.content !== "string" || msg.content.trim() === "") {
|
||||
errors.push({
|
||||
stage: "message",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
messageIndex: mi,
|
||||
message: '"content" is required and must be a non-empty string.',
|
||||
});
|
||||
}
|
||||
|
||||
if (msg.timestamp !== undefined) {
|
||||
if (typeof msg.timestamp === "number") {
|
||||
if (!Number.isInteger(msg.timestamp)) {
|
||||
errors.push({
|
||||
stage: "message",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
messageIndex: mi,
|
||||
message: '"timestamp" must be an integer (epoch milliseconds). Negative values are allowed for dates before 1970.',
|
||||
});
|
||||
}
|
||||
} else if (typeof msg.timestamp === "string") {
|
||||
if (Number.isNaN(new Date(msg.timestamp).getTime())) {
|
||||
errors.push({
|
||||
stage: "message",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
messageIndex: mi,
|
||||
message: `"timestamp" string is not a valid ISO 8601 date: "${msg.timestamp}".`,
|
||||
});
|
||||
}
|
||||
} else {
|
||||
errors.push({
|
||||
stage: "message",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
messageIndex: mi,
|
||||
message: '"timestamp" must be a number (epoch ms) or an ISO 8601 string.',
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Layer 6: Timestamp consistency
|
||||
// ============================
|
||||
|
||||
interface TimestampCheckResult {
|
||||
status: "all_present" | "all_missing" | "mixed";
|
||||
}
|
||||
|
||||
function checkTimestampConsistency(sessions: RawSession[]): TimestampCheckResult {
|
||||
let hasTs = false;
|
||||
let missingTs = false;
|
||||
|
||||
for (const session of sessions) {
|
||||
if (!Array.isArray(session.conversations)) continue;
|
||||
for (const round of session.conversations) {
|
||||
if (!Array.isArray(round)) continue;
|
||||
for (const msg of round) {
|
||||
if (msg.timestamp !== undefined && msg.timestamp !== null) {
|
||||
hasTs = true;
|
||||
} else {
|
||||
missingTs = true;
|
||||
}
|
||||
// Early exit on mixed
|
||||
if (hasTs && missingTs) {
|
||||
return { status: "mixed" };
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (hasTs && !missingTs) return { status: "all_present" };
|
||||
if (!hasTs && missingTs) return { status: "all_missing" };
|
||||
// No messages at all — treat as all_missing (will be caught by session validation)
|
||||
return { status: "all_missing" };
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Normalization
|
||||
// ============================
|
||||
|
||||
function normalizeSessions(
|
||||
sessions: RawSession[],
|
||||
fallbackSessionKey?: string,
|
||||
): { sessions: NormalizedSession[]; totalRounds: number; totalMessages: number } {
|
||||
const normalized: NormalizedSession[] = [];
|
||||
let totalRounds = 0;
|
||||
let totalMessages = 0;
|
||||
|
||||
for (let si = 0; si < sessions.length; si++) {
|
||||
const raw = sessions[si]!;
|
||||
|
||||
const sessionKey = raw.sessionKey || fallbackSessionKey || "seed-user";
|
||||
const sessionId = raw.sessionId || crypto.randomUUID();
|
||||
|
||||
const rounds: NormalizedRound[] = [];
|
||||
for (const rawRound of raw.conversations) {
|
||||
if (!Array.isArray(rawRound)) continue;
|
||||
|
||||
const messages: NormalizedMessage[] = rawRound.map((msg) => ({
|
||||
role: msg.role,
|
||||
content: msg.content,
|
||||
// Normalize timestamp: ISO string → epoch ms, number → pass-through, missing → 0 (filled later)
|
||||
timestamp: msg.timestamp == null
|
||||
? 0
|
||||
: typeof msg.timestamp === "string"
|
||||
? new Date(msg.timestamp).getTime()
|
||||
: msg.timestamp,
|
||||
}));
|
||||
|
||||
rounds.push({ messages });
|
||||
totalMessages += messages.length;
|
||||
}
|
||||
|
||||
totalRounds += rounds.length;
|
||||
normalized.push({
|
||||
sessionKey,
|
||||
sessionId,
|
||||
rounds,
|
||||
sourceIndex: si,
|
||||
});
|
||||
}
|
||||
|
||||
return { sessions: normalized, totalRounds, totalMessages };
|
||||
}
|
||||
@@ -0,0 +1,394 @@
|
||||
/**
|
||||
* Seed runtime: L0→L1→L2→L3 orchestration for the `seed` command.
|
||||
*
|
||||
* Uses the shared pipeline-factory for VectorStore/EmbeddingService init,
|
||||
* L1 runner, L2 runner, L3 runner, and persister wiring — keeping this
|
||||
* module focused on seed-specific concerns:
|
||||
* - Synchronous per-round L0 capture with progress reporting
|
||||
* - waitForL1Idle polling (L1 only — see FIXME below)
|
||||
* - Ctrl+C graceful shutdown
|
||||
*
|
||||
* FIXME: Currently we only wait for L1 to become idle before destroying the
|
||||
* pipeline. L2 (scene extraction) and L3 (persona generation) may still be
|
||||
* in-flight when `pipeline.destroy()` is called. This is intentional for now
|
||||
* to avoid excessively long seed runs, but means seed output may not include
|
||||
* the latest L2/L3 artifacts. Re-evaluate adding a full L1+L2+L3 idle wait
|
||||
* once pipeline-manager exposes reliable L2/L3 idle signals.
|
||||
*/
|
||||
|
||||
import path from "node:path";
|
||||
import { parseConfig } from "../config.js";
|
||||
import type { MemoryTdaiConfig } from "../config.js";
|
||||
import { performAutoCapture } from "../hooks/auto-capture.js";
|
||||
import { createPipeline, createL2Runner, createL3Runner } from "../utils/pipeline-factory.js";
|
||||
import type { PipelineInstance, PipelineLogger } from "../utils/pipeline-factory.js";
|
||||
import { readManifest, writeManifest } from "../utils/manifest.js";
|
||||
import type { MemoryPipelineManager } from "../utils/pipeline-manager.js";
|
||||
import type {
|
||||
NormalizedInput,
|
||||
SeedProgress,
|
||||
SeedSummary,
|
||||
} from "./types.js";
|
||||
|
||||
const TAG = "[memory-tdai] [seed]";
|
||||
|
||||
// ============================
|
||||
// Seed pipeline options
|
||||
// ============================
|
||||
|
||||
export interface SeedRuntimeOptions {
|
||||
/** Directory to store all seed output (L0, checkpoint, vectors.db). */
|
||||
outputDir: string;
|
||||
/** OpenClaw config object (needed for LLM calls in L1). */
|
||||
openclawConfig: unknown;
|
||||
/** Raw plugin config (same shape as api.pluginConfig). */
|
||||
pluginConfig?: Record<string, unknown>;
|
||||
/** Original input file path (for manifest traceability). */
|
||||
inputFile?: string;
|
||||
/** Logger instance. */
|
||||
logger: PipelineLogger;
|
||||
/** Progress callback (called after each round). */
|
||||
onProgress?: (progress: SeedProgress) => void;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Seed pipeline creation
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Create a seed pipeline using the shared factory, with L2/L3 runners
|
||||
* wired via shared factory functions (same logic as index.ts live runtime).
|
||||
*/
|
||||
async function createSeedPipeline(opts: SeedRuntimeOptions): Promise<{ pipeline: PipelineInstance; cfg: MemoryTdaiConfig }> {
|
||||
const { outputDir, openclawConfig, pluginConfig, logger } = opts;
|
||||
|
||||
// Parse config — all values come from pluginConfig (or parseConfig defaults)
|
||||
const cfg = parseConfig(pluginConfig);
|
||||
|
||||
logger.info(
|
||||
`${TAG} Creating seed pipeline: outputDir=${outputDir}, ` +
|
||||
`everyN=${cfg.pipeline.everyNConversations}, l1Idle=${cfg.pipeline.l1IdleTimeoutSeconds}s, ` +
|
||||
`l2Delay=${cfg.pipeline.l2DelayAfterL1Seconds}s, l2Min=${cfg.pipeline.l2MinIntervalSeconds}s, l2Max=${cfg.pipeline.l2MaxIntervalSeconds}s`,
|
||||
);
|
||||
|
||||
// Use shared factory for everything: store init, L1 runner, persister, destroy
|
||||
const pipeline = await createPipeline({
|
||||
pluginDataDir: outputDir,
|
||||
cfg,
|
||||
openclawConfig,
|
||||
logger,
|
||||
});
|
||||
|
||||
// Wire L2 runner via shared factory (same logic as index.ts live runtime)
|
||||
pipeline.scheduler.setL2Runner(createL2Runner({
|
||||
pluginDataDir: outputDir,
|
||||
cfg,
|
||||
openclawConfig,
|
||||
vectorStore: pipeline.vectorStore,
|
||||
logger,
|
||||
}));
|
||||
|
||||
// Wire L3 runner via shared factory (same logic as index.ts live runtime)
|
||||
pipeline.scheduler.setL3Runner(createL3Runner({
|
||||
pluginDataDir: outputDir,
|
||||
cfg,
|
||||
openclawConfig,
|
||||
vectorStore: pipeline.vectorStore,
|
||||
logger,
|
||||
}));
|
||||
|
||||
return { pipeline, cfg };
|
||||
}
|
||||
|
||||
// ============================
|
||||
// waitForL1Idle
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Poll pipeline queue status until L1 is idle for a given session.
|
||||
* Modeled after benchmark-ingest.ts waitForPipelineIdle() but focused on L1 only.
|
||||
*/
|
||||
async function waitForL1Idle(
|
||||
scheduler: MemoryPipelineManager,
|
||||
sessionKeys: string[],
|
||||
logger: PipelineLogger,
|
||||
opts: {
|
||||
pollIntervalMs?: number;
|
||||
stableRounds?: number;
|
||||
maxWaitMs?: number;
|
||||
} = {},
|
||||
): Promise<void> {
|
||||
const pollInterval = opts.pollIntervalMs ?? 1_000;
|
||||
const stableRounds = opts.stableRounds ?? 3;
|
||||
const maxWait = opts.maxWaitMs ?? 300_000; // 5 min default
|
||||
|
||||
const startTime = Date.now();
|
||||
let consecutiveIdle = 0;
|
||||
|
||||
while (true) {
|
||||
const elapsed = Date.now() - startTime;
|
||||
if (elapsed > maxWait) {
|
||||
logger.warn(`${TAG} [waitL1] Max wait time reached (${(maxWait / 1000).toFixed(0)}s), proceeding`);
|
||||
break;
|
||||
}
|
||||
|
||||
const queues = scheduler.getQueueSizes();
|
||||
|
||||
// Check per-session: buffered messages + conversation count
|
||||
let totalBuffered = 0;
|
||||
let totalConversationCount = 0;
|
||||
for (const key of sessionKeys) {
|
||||
totalBuffered += scheduler.getBufferedMessageCount(key);
|
||||
const state = scheduler.getSessionState(key);
|
||||
if (state) {
|
||||
totalConversationCount += state.conversation_count;
|
||||
}
|
||||
}
|
||||
|
||||
const isIdle =
|
||||
queues.l1Idle &&
|
||||
totalBuffered === 0 &&
|
||||
totalConversationCount === 0;
|
||||
|
||||
if (isIdle) {
|
||||
consecutiveIdle++;
|
||||
if (consecutiveIdle >= stableRounds) {
|
||||
logger.debug?.(`${TAG} [waitL1] L1 stable for ${stableRounds} consecutive polls`);
|
||||
return;
|
||||
}
|
||||
} else {
|
||||
consecutiveIdle = 0;
|
||||
logger.debug?.(
|
||||
`${TAG} [waitL1] Waiting: l1Queue=${queues.l1}, l1Pending=${queues.l1Pending}, l1Idle=${queues.l1Idle}, ` +
|
||||
`buffered=${totalBuffered}, convCount=${totalConversationCount}`,
|
||||
);
|
||||
}
|
||||
|
||||
await new Promise((resolve) => setTimeout(resolve, pollInterval));
|
||||
}
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Main execution function
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Execute the seed pipeline: feed normalized input through L0 → L1.
|
||||
*
|
||||
* L2/L3 runners are wired but their completion is **not** awaited — see the
|
||||
* module-level FIXME. The pipeline is destroyed after L1 idle, so L2/L3 may
|
||||
* be interrupted mid-run.
|
||||
*
|
||||
* This is the core runtime called by `src/cli/commands/seed.ts` after
|
||||
* all input validation and user confirmation are complete.
|
||||
*/
|
||||
export async function executeSeed(
|
||||
input: NormalizedInput,
|
||||
opts: SeedRuntimeOptions,
|
||||
): Promise<SeedSummary> {
|
||||
const { logger, onProgress } = opts;
|
||||
const startTime = Date.now();
|
||||
|
||||
// Track interrupt signal
|
||||
let interrupted = false;
|
||||
const onSigint = () => {
|
||||
if (interrupted) {
|
||||
// Second Ctrl+C — force exit
|
||||
logger.warn(`${TAG} Force exit (second Ctrl+C)`);
|
||||
process.exit(1);
|
||||
}
|
||||
interrupted = true;
|
||||
logger.warn(`${TAG} Interrupt received, finishing current round and shutting down...`);
|
||||
};
|
||||
process.on("SIGINT", onSigint);
|
||||
|
||||
let pipeline: PipelineInstance | undefined;
|
||||
let totalL0Recorded = 0;
|
||||
let roundsProcessed = 0;
|
||||
|
||||
try {
|
||||
// Create and start pipeline (returns both the pipeline instance and the
|
||||
// seed-optimized config so we don't need to parse config again)
|
||||
const seed = await createSeedPipeline(opts);
|
||||
pipeline = seed.pipeline;
|
||||
const seedCfg = seed.cfg;
|
||||
|
||||
pipeline.scheduler.start({});
|
||||
logger.info(`${TAG} Pipeline started, processing ${input.sessions.length} session(s), ${input.totalRounds} round(s)`);
|
||||
|
||||
// Seed-specific: use 0 so the cold-start guard in captureAtomically()
|
||||
// does NOT filter out historical messages. In live mode Date.now()
|
||||
// prevents the first agent_end from dumping full session history,
|
||||
// but seed intentionally feeds all historical data.
|
||||
const captureStartTimestamp = 0;
|
||||
|
||||
// Process each session → each round
|
||||
// Key invariant: after every everyNConversations rounds we must wait for L1
|
||||
// to finish before feeding more rounds. Without this pause the for-loop
|
||||
// would dump all rounds into L0 back-to-back and L1 would only run once
|
||||
// with the full batch (defeating the "every N" batching semantics).
|
||||
const everyN = seedCfg.pipeline.everyNConversations;
|
||||
|
||||
for (const session of input.sessions) {
|
||||
if (interrupted) break;
|
||||
|
||||
logger.info(`${TAG} Session: key="${session.sessionKey}" id="${session.sessionId}" rounds=${session.rounds.length}`);
|
||||
|
||||
for (let ri = 0; ri < session.rounds.length; ri++) {
|
||||
if (interrupted) break;
|
||||
|
||||
const round = session.rounds[ri]!;
|
||||
roundsProcessed++;
|
||||
|
||||
// Build messages in the format expected by performAutoCapture.
|
||||
// Field must be named "timestamp" (not "ts") because l0-recorder's
|
||||
// extractUserAssistantMessages reads m.timestamp for incremental filtering.
|
||||
const messages = round.messages.map((m) => ({
|
||||
role: m.role,
|
||||
content: m.content,
|
||||
timestamp: m.timestamp,
|
||||
}));
|
||||
|
||||
try {
|
||||
const result = await performAutoCapture({
|
||||
messages,
|
||||
sessionKey: session.sessionKey,
|
||||
sessionId: session.sessionId,
|
||||
cfg: seedCfg,
|
||||
pluginDataDir: opts.outputDir,
|
||||
logger,
|
||||
scheduler: pipeline.scheduler,
|
||||
pluginStartTimestamp: captureStartTimestamp,
|
||||
vectorStore: pipeline.vectorStore,
|
||||
embeddingService: pipeline.embeddingService,
|
||||
});
|
||||
|
||||
totalL0Recorded += result.l0RecordedCount;
|
||||
} catch (err) {
|
||||
logger.error(
|
||||
`${TAG} L0 capture failed for session="${session.sessionKey}" round=${ri}: ` +
|
||||
`${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
}
|
||||
|
||||
// Report progress
|
||||
onProgress?.({
|
||||
currentRound: roundsProcessed,
|
||||
totalRounds: input.totalRounds,
|
||||
sessionKey: session.sessionKey,
|
||||
stage: "l0_captured",
|
||||
});
|
||||
|
||||
// After every N rounds, wait for the triggered L1 to finish before
|
||||
// feeding the next batch. This keeps L1 batches aligned with the
|
||||
// everyNConversations boundary instead of letting all rounds pile up.
|
||||
const roundInSession = ri + 1; // 1-based
|
||||
if (roundInSession % everyN === 0 && !interrupted) {
|
||||
onProgress?.({
|
||||
currentRound: roundsProcessed,
|
||||
totalRounds: input.totalRounds,
|
||||
sessionKey: session.sessionKey,
|
||||
stage: "l1_waiting",
|
||||
});
|
||||
|
||||
logger.info(
|
||||
`${TAG} Pausing after round ${roundInSession}/${session.rounds.length} ` +
|
||||
`for session="${session.sessionKey}" — waiting for L1 to drain`,
|
||||
);
|
||||
|
||||
await waitForL1Idle(
|
||||
pipeline.scheduler,
|
||||
[session.sessionKey],
|
||||
logger,
|
||||
{ pollIntervalMs: 500, stableRounds: 2, maxWaitMs: 120_000 },
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// After all rounds for this session, wait for any residual L1 work
|
||||
// (handles the tail when total rounds is not a multiple of everyN)
|
||||
if (!interrupted) {
|
||||
onProgress?.({
|
||||
currentRound: roundsProcessed,
|
||||
totalRounds: input.totalRounds,
|
||||
sessionKey: session.sessionKey,
|
||||
stage: "l1_waiting",
|
||||
});
|
||||
|
||||
await waitForL1Idle(
|
||||
pipeline.scheduler,
|
||||
[session.sessionKey],
|
||||
logger,
|
||||
{ pollIntervalMs: 1_000, stableRounds: 3, maxWaitMs: 300_000 },
|
||||
);
|
||||
|
||||
logger.info(`${TAG} L1 idle for session="${session.sessionKey}"`);
|
||||
}
|
||||
}
|
||||
|
||||
// Final wait for all sessions
|
||||
if (!interrupted) {
|
||||
const allKeys = input.sessions.map((s) => s.sessionKey);
|
||||
logger.info(`${TAG} Final L1 idle wait for all sessions...`);
|
||||
await waitForL1Idle(
|
||||
pipeline.scheduler,
|
||||
allKeys,
|
||||
logger,
|
||||
{ pollIntervalMs: 1_000, stableRounds: 3, maxWaitMs: 300_000 },
|
||||
);
|
||||
}
|
||||
} finally {
|
||||
process.removeListener("SIGINT", onSigint);
|
||||
|
||||
// Graceful shutdown
|
||||
if (pipeline) {
|
||||
try {
|
||||
await pipeline.destroy();
|
||||
} catch (err) {
|
||||
logger.error(`${TAG} Pipeline destroy error: ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const durationMs = Date.now() - startTime;
|
||||
|
||||
const summary: SeedSummary = {
|
||||
sessionsProcessed: input.sessions.length,
|
||||
roundsProcessed,
|
||||
messagesProcessed: input.totalMessages,
|
||||
l0RecordedCount: totalL0Recorded,
|
||||
durationMs,
|
||||
outputDir: opts.outputDir,
|
||||
};
|
||||
|
||||
if (interrupted) {
|
||||
logger.warn(`${TAG} Seed interrupted after ${roundsProcessed}/${input.totalRounds} rounds`);
|
||||
} else {
|
||||
logger.info(
|
||||
`${TAG} Seed complete: sessions=${summary.sessionsProcessed}, ` +
|
||||
`rounds=${summary.roundsProcessed}, messages=${summary.messagesProcessed}, ` +
|
||||
`l0Recorded=${summary.l0RecordedCount}, duration=${(durationMs / 1000).toFixed(1)}s`,
|
||||
);
|
||||
}
|
||||
|
||||
// Append seed info to manifest (non-fatal if it fails)
|
||||
try {
|
||||
const manifest = readManifest(opts.outputDir);
|
||||
if (manifest) {
|
||||
manifest.seed = {
|
||||
inputFile: opts.inputFile ? path.basename(opts.inputFile) : undefined,
|
||||
sessions: summary.sessionsProcessed,
|
||||
rounds: summary.roundsProcessed,
|
||||
messages: summary.messagesProcessed,
|
||||
startedAt: new Date(startTime).toISOString(),
|
||||
completedAt: new Date().toISOString(),
|
||||
};
|
||||
writeManifest(opts.outputDir, manifest);
|
||||
logger.info(`${TAG} Manifest updated with seed info`);
|
||||
}
|
||||
} catch (err) {
|
||||
logger.warn(`${TAG} Failed to update manifest with seed info (non-fatal): ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
|
||||
return summary;
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
/**
|
||||
* Shared type definitions for the `seed` command.
|
||||
*
|
||||
* Covers:
|
||||
* - Raw input shapes (Format A / B / JSONL)
|
||||
* - Normalized internal structures
|
||||
* - Validation error descriptors
|
||||
*/
|
||||
|
||||
// ============================
|
||||
// Raw input types (before validation)
|
||||
// ============================
|
||||
|
||||
/** A single message in a conversation round. */
|
||||
export interface RawMessage {
|
||||
role: string;
|
||||
content: string;
|
||||
/**
|
||||
* Epoch milliseconds (number) **or** ISO 8601 string (e.g. `"2024-04-01T12:00:00Z"`).
|
||||
* ISO strings are parsed via `new Date()` during normalization and
|
||||
* stored internally as epoch ms.
|
||||
*/
|
||||
timestamp?: number | string;
|
||||
}
|
||||
|
||||
/** A single session entry (shared between Format A wrapper and Format B array). */
|
||||
export interface RawSession {
|
||||
sessionKey: string;
|
||||
sessionId?: string;
|
||||
conversations: RawMessage[][];
|
||||
}
|
||||
|
||||
/** Format A: `{ sessions: [...] }` */
|
||||
export interface FormatA {
|
||||
sessions: RawSession[];
|
||||
}
|
||||
|
||||
/** Format B: `[...]` (top-level array of sessions) */
|
||||
export type FormatB = RawSession[];
|
||||
|
||||
// ============================
|
||||
// Normalized types (after validation)
|
||||
// ============================
|
||||
|
||||
export interface NormalizedMessage {
|
||||
role: string;
|
||||
content: string;
|
||||
/** Epoch ms — always present after normalization (filled if originally missing). */
|
||||
timestamp: number;
|
||||
}
|
||||
|
||||
export interface NormalizedRound {
|
||||
messages: NormalizedMessage[];
|
||||
}
|
||||
|
||||
export interface NormalizedSession {
|
||||
sessionKey: string;
|
||||
sessionId: string;
|
||||
rounds: NormalizedRound[];
|
||||
/** Index in the original input array (for progress reporting). */
|
||||
sourceIndex: number;
|
||||
}
|
||||
|
||||
export interface NormalizedInput {
|
||||
sessions: NormalizedSession[];
|
||||
/** Total number of rounds across all sessions. */
|
||||
totalRounds: number;
|
||||
/** Total number of messages across all sessions. */
|
||||
totalMessages: number;
|
||||
/** Whether timestamps were present in the original input. */
|
||||
hasTimestamps: boolean;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Validation
|
||||
// ============================
|
||||
|
||||
/** Stages where a validation error can occur. */
|
||||
export type ValidationStage =
|
||||
| "file"
|
||||
| "top_level"
|
||||
| "session"
|
||||
| "round"
|
||||
| "message"
|
||||
| "timestamp_consistency";
|
||||
|
||||
/** A single validation error with location context. */
|
||||
export interface ValidationError {
|
||||
stage: ValidationStage;
|
||||
sourceIndex?: number;
|
||||
sessionKey?: string;
|
||||
roundIndex?: number;
|
||||
messageIndex?: number;
|
||||
message: string;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Seed command options (from CLI)
|
||||
// ============================
|
||||
|
||||
export interface SeedCommandOptions {
|
||||
/** Path to input file (required). */
|
||||
input: string;
|
||||
/** Output directory (optional, auto-generated if missing). */
|
||||
outputDir?: string;
|
||||
/** Fallback session key when input lacks one. */
|
||||
sessionKey?: string;
|
||||
/** Strict round-role validation (each round must have user + assistant). */
|
||||
strictRoundRole: boolean;
|
||||
/** Skip interactive confirmations. */
|
||||
yes: boolean;
|
||||
/** Path to memory-tdai config override file (JSON, deep-merged on top of current plugin config). */
|
||||
configFile?: string;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Seed runtime types
|
||||
// ============================
|
||||
|
||||
/** Progress info emitted during seed execution. */
|
||||
export interface SeedProgress {
|
||||
/** Current round index (1-based, across all sessions). */
|
||||
currentRound: number;
|
||||
/** Total rounds. */
|
||||
totalRounds: number;
|
||||
/** Current session key. */
|
||||
sessionKey: string;
|
||||
/** Current stage description. */
|
||||
stage: string;
|
||||
}
|
||||
|
||||
/** Final summary after seed completes. */
|
||||
export interface SeedSummary {
|
||||
sessionsProcessed: number;
|
||||
roundsProcessed: number;
|
||||
messagesProcessed: number;
|
||||
l0RecordedCount: number;
|
||||
durationMs: number;
|
||||
outputDir: string;
|
||||
}
|
||||
@@ -0,0 +1,168 @@
|
||||
/**
|
||||
* BM25 Sparse Vector Encoding Client.
|
||||
*
|
||||
* HTTP client for the BM25 Python sidecar service (bm25_server.py).
|
||||
* Used by TCVDB backend to generate sparse vectors for hybridSearch.
|
||||
*
|
||||
* Two operations:
|
||||
* - `encodeTexts(texts)` — encode documents for upsert (TF-based)
|
||||
* - `encodeQueries(texts)` — encode queries for search (IDF-based)
|
||||
*
|
||||
* Graceful degradation: if the sidecar is unreachable, all methods
|
||||
* return empty arrays and `isHealthy()` returns false. Callers can
|
||||
* check health to dynamically downgrade to pure semantic search.
|
||||
*/
|
||||
|
||||
// ============================
|
||||
// Types
|
||||
// ============================
|
||||
|
||||
/** Sparse vector: array of [token_hash, weight] pairs. */
|
||||
export type SparseVector = Array<[number, number]>;
|
||||
|
||||
interface Logger {
|
||||
debug?: (message: string) => void;
|
||||
info: (message: string) => void;
|
||||
warn: (message: string) => void;
|
||||
error: (message: string) => void;
|
||||
}
|
||||
|
||||
export interface BM25ClientConfig {
|
||||
/** Sidecar service URL (default: "http://127.0.0.1:8084") */
|
||||
serviceUrl: string;
|
||||
/** Request timeout in ms (default: 5000) */
|
||||
timeout: number;
|
||||
}
|
||||
|
||||
interface EncodeResponse {
|
||||
vectors: SparseVector[];
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Implementation
|
||||
// ============================
|
||||
|
||||
const TAG = "[memory-tdai][bm25-client]";
|
||||
|
||||
export class BM25Client {
|
||||
private readonly baseUrl: string;
|
||||
private readonly timeout: number;
|
||||
private readonly logger?: Logger;
|
||||
|
||||
/** Cached health status to avoid repeated checks on every call. */
|
||||
private _healthy: boolean | undefined;
|
||||
private _lastHealthCheck = 0;
|
||||
private static readonly HEALTH_CHECK_INTERVAL_MS = 30_000; // re-check every 30s
|
||||
|
||||
constructor(config: BM25ClientConfig, logger?: Logger) {
|
||||
this.baseUrl = config.serviceUrl.replace(/\/+$/, "");
|
||||
this.timeout = config.timeout;
|
||||
this.logger = logger;
|
||||
}
|
||||
|
||||
/**
|
||||
* Encode document texts for upsert (TF-based BM25 scoring).
|
||||
* Returns one SparseVector per input text.
|
||||
* Returns empty array on error (non-throwing).
|
||||
*/
|
||||
async encodeTexts(texts: string[]): Promise<SparseVector[]> {
|
||||
if (texts.length === 0) return [];
|
||||
return this._encode("/encode_texts", texts);
|
||||
}
|
||||
|
||||
/**
|
||||
* Encode query texts for search (IDF-based BM25 scoring).
|
||||
* Returns one SparseVector per input text.
|
||||
* Returns empty array on error (non-throwing).
|
||||
*/
|
||||
async encodeQueries(texts: string[]): Promise<SparseVector[]> {
|
||||
if (texts.length === 0) return [];
|
||||
return this._encode("/encode_queries", texts);
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if the BM25 sidecar is reachable.
|
||||
* Result is cached for 30 seconds to avoid spamming health checks.
|
||||
*/
|
||||
async isHealthy(): Promise<boolean> {
|
||||
const now = Date.now();
|
||||
if (
|
||||
this._healthy !== undefined &&
|
||||
now - this._lastHealthCheck < BM25Client.HEALTH_CHECK_INTERVAL_MS
|
||||
) {
|
||||
return this._healthy;
|
||||
}
|
||||
|
||||
try {
|
||||
const resp = await fetch(`${this.baseUrl}/health`, {
|
||||
signal: AbortSignal.timeout(3000),
|
||||
});
|
||||
this._healthy = resp.ok;
|
||||
} catch {
|
||||
this._healthy = false;
|
||||
}
|
||||
this._lastHealthCheck = now;
|
||||
|
||||
if (!this._healthy) {
|
||||
this.logger?.warn(`${TAG} BM25 sidecar health check failed (${this.baseUrl})`);
|
||||
}
|
||||
|
||||
return this._healthy;
|
||||
}
|
||||
|
||||
// ── Internal ──────────────────────────────────────────────────
|
||||
|
||||
private async _encode(path: string, texts: string[]): Promise<SparseVector[]> {
|
||||
try {
|
||||
const resp = await fetch(`${this.baseUrl}${path}`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ texts }),
|
||||
signal: AbortSignal.timeout(this.timeout),
|
||||
});
|
||||
|
||||
if (!resp.ok) {
|
||||
const errBody = await resp.text().catch(() => "(unreadable)");
|
||||
this.logger?.warn(
|
||||
`${TAG} ${path} HTTP ${resp.status}: ${errBody.slice(0, 200)}`,
|
||||
);
|
||||
return [];
|
||||
}
|
||||
|
||||
const json = (await resp.json()) as EncodeResponse;
|
||||
return json.vectors ?? [];
|
||||
} catch (err) {
|
||||
// Mark unhealthy on connection errors
|
||||
this._healthy = false;
|
||||
this._lastHealthCheck = Date.now();
|
||||
|
||||
this.logger?.warn(
|
||||
`${TAG} ${path} failed: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
return [];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Factory
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Create a BM25Client if BM25 is enabled in config.
|
||||
* Returns undefined if disabled — callers should check before using.
|
||||
*/
|
||||
export function createBM25Client(
|
||||
config: { enabled: boolean; serviceUrl: string; timeout: number },
|
||||
logger?: Logger,
|
||||
): BM25Client | undefined {
|
||||
if (!config.enabled) {
|
||||
logger?.info(`${TAG} BM25 sparse encoding disabled`);
|
||||
return undefined;
|
||||
}
|
||||
logger?.info(`${TAG} BM25 client → ${config.serviceUrl}`);
|
||||
return new BM25Client(
|
||||
{ serviceUrl: config.serviceUrl, timeout: config.timeout },
|
||||
logger,
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
/**
|
||||
* Local BM25 Sparse Vector Encoder.
|
||||
*
|
||||
* Pure TypeScript replacement for the Python sidecar BM25 client.
|
||||
* Uses @tencentdb-agent-memory/tcvdb-text package for tokenization (jieba-wasm) and BM25 encoding.
|
||||
*
|
||||
* Two operations (same contract as the old BM25Client):
|
||||
* - `encodeTexts(texts)` — encode documents for upsert (TF-based)
|
||||
* - `encodeQueries(texts)` — encode queries for search (IDF-based)
|
||||
*/
|
||||
|
||||
import { BM25Encoder } from "@tencentdb-agent-memory/tcvdb-text";
|
||||
import type { SparseVector } from "@tencentdb-agent-memory/tcvdb-text";
|
||||
|
||||
export type { SparseVector };
|
||||
|
||||
interface Logger {
|
||||
debug?: (message: string) => void;
|
||||
info: (message: string) => void;
|
||||
warn: (message: string) => void;
|
||||
error: (message: string) => void;
|
||||
}
|
||||
|
||||
export interface BM25LocalConfig {
|
||||
/** Whether BM25 sparse encoding is enabled (default: true) */
|
||||
enabled: boolean;
|
||||
/** Language for BM25 pre-trained params: "zh" or "en" (default: "zh") */
|
||||
language?: "zh" | "en";
|
||||
}
|
||||
|
||||
const TAG = "[memory-tdai][bm25-local]";
|
||||
|
||||
// ============================
|
||||
// Implementation
|
||||
// ============================
|
||||
|
||||
export class BM25LocalEncoder {
|
||||
private readonly encoder: BM25Encoder;
|
||||
private readonly logger?: Logger;
|
||||
|
||||
constructor(language: "zh" | "en" = "zh", logger?: Logger) {
|
||||
this.logger = logger;
|
||||
this.encoder = BM25Encoder.default(language);
|
||||
logger?.debug?.(`${TAG} Initialized BM25 local encoder (language=${language})`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Encode document texts for upsert (TF-based BM25 scoring).
|
||||
* Returns one SparseVector per input text.
|
||||
*/
|
||||
encodeTexts(texts: string[]): SparseVector[] {
|
||||
if (texts.length === 0) return [];
|
||||
try {
|
||||
return this.encoder.encodeTexts(texts);
|
||||
} catch (err) {
|
||||
this.logger?.warn(
|
||||
`${TAG} encodeTexts failed: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Encode query texts for search (IDF-based BM25 scoring).
|
||||
* Returns one SparseVector per input text.
|
||||
*/
|
||||
encodeQueries(texts: string[]): SparseVector[] {
|
||||
if (texts.length === 0) return [];
|
||||
try {
|
||||
return this.encoder.encodeQueries(texts);
|
||||
} catch (err) {
|
||||
this.logger?.warn(
|
||||
`${TAG} encodeQueries failed: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
return [];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Factory
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Create a BM25LocalEncoder if BM25 is enabled in config.
|
||||
* Returns undefined if disabled — callers should check before using.
|
||||
*/
|
||||
export function createBM25Encoder(
|
||||
config: BM25LocalConfig,
|
||||
logger?: Logger,
|
||||
): BM25LocalEncoder | undefined {
|
||||
if (!config.enabled) {
|
||||
logger?.debug?.(`${TAG} BM25 sparse encoding disabled`);
|
||||
return undefined;
|
||||
}
|
||||
return new BM25LocalEncoder(config.language ?? "zh", logger);
|
||||
}
|
||||
+53
-5
@@ -149,10 +149,20 @@ function sanitizeAndNormalize(vec: number[] | Float32Array): Float32Array {
|
||||
*/
|
||||
type LocalInitState = "idle" | "initializing" | "ready" | "failed";
|
||||
|
||||
/** Function that dynamically imports node-llama-cpp. Overridable for testing. */
|
||||
export type ImportLlamaFn = () => Promise<{
|
||||
getLlama: (opts: { logLevel: number }) => Promise<unknown>;
|
||||
resolveModelFile: (model: string, cacheDir?: string) => Promise<string>;
|
||||
LlamaLogLevel: { error: number };
|
||||
}>;
|
||||
|
||||
const defaultImportLlama: ImportLlamaFn = () => import("node-llama-cpp") as unknown as ReturnType<ImportLlamaFn>;
|
||||
|
||||
export class LocalEmbeddingService implements EmbeddingService {
|
||||
private readonly modelPath: string;
|
||||
private readonly modelCacheDir?: string;
|
||||
private readonly logger?: Logger;
|
||||
private readonly importLlama: ImportLlamaFn;
|
||||
|
||||
// Initialization state machine
|
||||
private initState: LocalInitState = "idle";
|
||||
@@ -162,10 +172,11 @@ export class LocalEmbeddingService implements EmbeddingService {
|
||||
getEmbeddingFor: (text: string) => Promise<{ vector: Float32Array | number[] }>;
|
||||
} | null = null;
|
||||
|
||||
constructor(config?: LocalEmbeddingConfig, logger?: Logger) {
|
||||
constructor(config?: LocalEmbeddingConfig, logger?: Logger, importLlama?: ImportLlamaFn) {
|
||||
this.modelPath = config?.modelPath?.trim() || DEFAULT_LOCAL_MODEL;
|
||||
this.modelCacheDir = config?.modelCacheDir?.trim();
|
||||
this.logger = logger;
|
||||
this.importLlama = importLlama ?? defaultImportLlama;
|
||||
}
|
||||
|
||||
getDimensions(): number {
|
||||
@@ -307,7 +318,7 @@ export class LocalEmbeddingService implements EmbeddingService {
|
||||
this.logger?.debug?.(`${TAG} Loading node-llama-cpp for local embedding...`);
|
||||
|
||||
// Dynamic import — node-llama-cpp is a peer dependency of OpenClaw
|
||||
const { getLlama, resolveModelFile, LlamaLogLevel } = await import("node-llama-cpp");
|
||||
const { getLlama, resolveModelFile, LlamaLogLevel } = await this.importLlama();
|
||||
|
||||
const llama = await getLlama({ logLevel: LlamaLogLevel.error });
|
||||
this.logger?.debug?.(`${TAG} Llama instance created`);
|
||||
@@ -581,18 +592,55 @@ export function createEmbeddingService(
|
||||
): EmbeddingService {
|
||||
// Remote OpenAI-compatible provider: any provider value other than "local"
|
||||
if (config && config.provider !== "local" && "apiKey" in config && config.apiKey) {
|
||||
logger?.info(`${TAG} Using remote embedding (provider=${config.provider}, model=${config.model})`);
|
||||
logger?.debug?.(`${TAG} Using remote embedding (provider=${config.provider}, model=${config.model})`);
|
||||
return new OpenAIEmbeddingService(config as OpenAIEmbeddingConfig, logger);
|
||||
}
|
||||
|
||||
// Explicit local config
|
||||
if (config && config.provider === "local") {
|
||||
const localConfig = config as LocalEmbeddingConfig;
|
||||
logger?.info(`${TAG} Using local embedding (node-llama-cpp, model=${localConfig.modelPath ?? DEFAULT_LOCAL_MODEL})`);
|
||||
logger?.debug?.(`${TAG} Using local embedding (node-llama-cpp, model=${localConfig.modelPath ?? DEFAULT_LOCAL_MODEL})`);
|
||||
return new LocalEmbeddingService(localConfig, logger);
|
||||
}
|
||||
|
||||
// Fallback: no config or empty apiKey → use local
|
||||
logger?.info(`${TAG} No remote embedding configured, falling back to local embedding (node-llama-cpp)`);
|
||||
logger?.debug?.(`${TAG} No remote embedding configured, falling back to local embedding (node-llama-cpp)`);
|
||||
return new LocalEmbeddingService(undefined, logger);
|
||||
}
|
||||
|
||||
// ============================
|
||||
// NoopEmbeddingService (for server-side embedding backends)
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* No-op embedding service for backends with built-in server-side embedding
|
||||
* (e.g., TCVDB with Collection-level embedding config).
|
||||
*
|
||||
* All embed() calls return an empty Float32Array because the server generates
|
||||
* vectors automatically from the text field during upsert/search.
|
||||
*/
|
||||
export class NoopEmbeddingService implements EmbeddingService {
|
||||
embed(_text: string): Promise<Float32Array> {
|
||||
return Promise.resolve(new Float32Array(0));
|
||||
}
|
||||
|
||||
embedBatch(texts: string[]): Promise<Float32Array[]> {
|
||||
return Promise.resolve(texts.map(() => new Float32Array(0)));
|
||||
}
|
||||
|
||||
getDimensions(): number {
|
||||
return 0;
|
||||
}
|
||||
|
||||
getProviderInfo(): EmbeddingProviderInfo {
|
||||
return { provider: "noop", model: "server-side" };
|
||||
}
|
||||
|
||||
isReady(): boolean {
|
||||
return true;
|
||||
}
|
||||
|
||||
startWarmup(): void {
|
||||
// no-op
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
/**
|
||||
* Store Factory — creates the appropriate storage backend and embedding service
|
||||
* based on plugin configuration.
|
||||
*
|
||||
* Supports:
|
||||
* - "sqlite" (default): local SQLite + sqlite-vec + FTS5
|
||||
* - "tcvdb": Tencent Cloud VectorDB (server-side embedding + hybridSearch)
|
||||
*/
|
||||
|
||||
import path from "node:path";
|
||||
import type { MemoryTdaiConfig } from "../config.js";
|
||||
import type { IMemoryStore, IEmbeddingService, StoreLogger } from "./types.js";
|
||||
import { VectorStore } from "./sqlite.js";
|
||||
import { TcvdbMemoryStore } from "./tcvdb.js";
|
||||
import { createEmbeddingService, NoopEmbeddingService } from "./embedding.js";
|
||||
import type { EmbeddingService } from "./embedding.js";
|
||||
import { createBM25Encoder } from "./bm25-local.js";
|
||||
import type { BM25LocalEncoder } from "./bm25-local.js";
|
||||
|
||||
// Re-export for convenience
|
||||
export type { IMemoryStore, IEmbeddingService, StoreLogger, BM25LocalEncoder };
|
||||
|
||||
const TAG = "[memory-tdai][factory]";
|
||||
|
||||
export interface StoreBundle {
|
||||
store: IMemoryStore;
|
||||
embedding: IEmbeddingService;
|
||||
bm25Encoder?: BM25LocalEncoder;
|
||||
/** Snapshot of current store config for manifest writing. */
|
||||
storeSnapshot: import("../utils/manifest.js").StoreConfigSnapshot;
|
||||
}
|
||||
|
||||
/**
|
||||
* Create the storage backend, embedding service, and optional BM25 encoder
|
||||
* based on plugin configuration.
|
||||
*
|
||||
* @param config Fully resolved plugin config.
|
||||
* @param options.dataDir Plugin data directory.
|
||||
* @param options.logger Logger instance.
|
||||
*/
|
||||
export function createStoreBundle(
|
||||
config: MemoryTdaiConfig,
|
||||
options: { dataDir: string; logger?: StoreLogger },
|
||||
): StoreBundle {
|
||||
const { logger } = options;
|
||||
|
||||
// ── BM25 local encoder ──
|
||||
const bm25Encoder = createBM25Encoder(config.bm25, logger);
|
||||
|
||||
switch (config.storeBackend) {
|
||||
case "tcvdb": {
|
||||
const tcvdbCfg = config.tcvdb;
|
||||
if (!tcvdbCfg.url || !tcvdbCfg.apiKey) {
|
||||
throw new Error(`${TAG} TCVDB backend requires tcvdb.url and tcvdb.apiKey`);
|
||||
}
|
||||
if (!tcvdbCfg.database) {
|
||||
throw new Error(`${TAG} TCVDB backend requires tcvdb.database — please set a unique database name in your openclaw.json plugin config`);
|
||||
}
|
||||
const database = tcvdbCfg.database;
|
||||
const store = new TcvdbMemoryStore({
|
||||
url: tcvdbCfg.url,
|
||||
username: tcvdbCfg.username,
|
||||
apiKey: tcvdbCfg.apiKey,
|
||||
database,
|
||||
embeddingModel: tcvdbCfg.embeddingModel,
|
||||
timeout: tcvdbCfg.timeout,
|
||||
caPemPath: tcvdbCfg.caPemPath,
|
||||
logger,
|
||||
bm25Encoder: bm25Encoder ?? undefined,
|
||||
});
|
||||
|
||||
logger?.debug?.(
|
||||
`${TAG} Store created: backend=tcvdb, database=${database}, model=${tcvdbCfg.embeddingModel}, ` +
|
||||
`bm25=${bm25Encoder ? "enabled" : "disabled"}`,
|
||||
);
|
||||
|
||||
return {
|
||||
store,
|
||||
embedding: new NoopEmbeddingService(),
|
||||
bm25Encoder,
|
||||
storeSnapshot: {
|
||||
type: "tcvdb",
|
||||
tcvdbUrl: tcvdbCfg.url,
|
||||
tcvdbDatabase: database,
|
||||
tcvdbAlias: tcvdbCfg.alias || undefined,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
case "sqlite":
|
||||
default: {
|
||||
// ── Embedding service (only when enabled) ──
|
||||
let embeddingService: EmbeddingService | undefined;
|
||||
if (config.embedding.enabled && config.embedding.provider !== "local" && config.embedding.apiKey) {
|
||||
embeddingService = createEmbeddingService({
|
||||
provider: config.embedding.provider,
|
||||
baseUrl: config.embedding.baseUrl,
|
||||
apiKey: config.embedding.apiKey,
|
||||
model: config.embedding.model,
|
||||
dimensions: config.embedding.dimensions,
|
||||
maxInputChars: config.embedding.maxInputChars,
|
||||
}, logger);
|
||||
}
|
||||
|
||||
// dimensions from config (0 when provider="none" → vec0 deferred)
|
||||
const dims = config.embedding.dimensions;
|
||||
const dbPath = path.join(options.dataDir, "vectors.db");
|
||||
const store = new VectorStore(dbPath, dims, logger);
|
||||
|
||||
logger?.debug?.(
|
||||
`${TAG} Store created: backend=sqlite, dbPath=${dbPath}, dimensions=${dims}, ` +
|
||||
`embedding=${embeddingService ? "enabled" : "disabled"}, ` +
|
||||
`bm25=${bm25Encoder ? "enabled" : "disabled"}`,
|
||||
);
|
||||
|
||||
return {
|
||||
store,
|
||||
embedding: embeddingService as unknown as IEmbeddingService,
|
||||
bm25Encoder,
|
||||
storeSnapshot: {
|
||||
type: "sqlite",
|
||||
sqlitePath: path.relative(options.dataDir, dbPath),
|
||||
},
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
/**
|
||||
* Search utilities — shared helpers for memory search across backends.
|
||||
*
|
||||
* Contains:
|
||||
* - RRF (Reciprocal Rank Fusion) merge — used by SQLite hybrid search
|
||||
* (eliminates the 3x duplication in auto-recall, memory-search, conversation-search)
|
||||
* - FTS query building — re-exported from sqlite for convenience
|
||||
*/
|
||||
|
||||
// ============================
|
||||
// RRF (Reciprocal Rank Fusion)
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Standard RRF constant from the original RRF paper.
|
||||
* Higher k → more weight on lower-ranked items (smoother distribution).
|
||||
*/
|
||||
export const RRF_K = 60;
|
||||
|
||||
/**
|
||||
* Merge multiple ranked lists via Reciprocal Rank Fusion.
|
||||
*
|
||||
* Each item's RRF score = sum over all lists of 1/(k + rank + 1).
|
||||
* Items appearing in multiple lists get their scores summed.
|
||||
*
|
||||
* @param lists Array of ranked lists. Each list must have items with an `id` field.
|
||||
* @param k RRF constant (default: 60).
|
||||
* @returns Merged list sorted by descending RRF score, with `rrfScore` attached.
|
||||
*
|
||||
* @example
|
||||
* ```ts
|
||||
* const merged = rrfMerge(
|
||||
* [ftsResults, vecResults],
|
||||
* (item) => item.record_id,
|
||||
* );
|
||||
* ```
|
||||
*/
|
||||
export function rrfMerge<T>(
|
||||
lists: T[][],
|
||||
getId: (item: T) => string,
|
||||
k: number = RRF_K,
|
||||
): Array<T & { rrfScore: number }> {
|
||||
const map = new Map<string, { item: T; rrfScore: number }>();
|
||||
|
||||
for (const list of lists) {
|
||||
for (let rank = 0; rank < list.length; rank++) {
|
||||
const item = list[rank];
|
||||
const id = getId(item);
|
||||
const score = 1 / (k + rank + 1);
|
||||
const existing = map.get(id);
|
||||
if (existing) {
|
||||
existing.rrfScore += score;
|
||||
} else {
|
||||
map.set(id, { item, rrfScore: score });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return [...map.values()]
|
||||
.sort((a, b) => b.rrfScore - a.rrfScore)
|
||||
.map(({ item, rrfScore }) => ({ ...item, rrfScore }));
|
||||
}
|
||||
+2303
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,287 @@
|
||||
/**
|
||||
* Tencent Cloud VectorDB HTTP Client.
|
||||
*
|
||||
* Thin wrapper around the VectorDB HTTP API. Handles authentication, timeouts,
|
||||
* retries (5xx / timeout), and error normalization.
|
||||
*
|
||||
* API docs: https://cloud.tencent.com/document/product/1709
|
||||
*/
|
||||
|
||||
import fs from "node:fs";
|
||||
import { request as undiciRequest, Agent as UndiciAgent } from "undici";
|
||||
import type { Dispatcher } from "undici";
|
||||
import type { StoreLogger } from "./types.js";
|
||||
|
||||
// ============================
|
||||
// Types
|
||||
// ============================
|
||||
|
||||
export interface TcvdbClientConfig {
|
||||
/** Instance URL (e.g. "http://10.0.1.1:80") */
|
||||
url: string;
|
||||
/** Account name (default: "root") */
|
||||
username: string;
|
||||
/** API Key */
|
||||
apiKey: string;
|
||||
/** Database name */
|
||||
database: string;
|
||||
/** Request timeout in ms (default: 10000) */
|
||||
timeout: number;
|
||||
/** Path to CA certificate PEM file (for HTTPS connections) */
|
||||
caPemPath?: string;
|
||||
}
|
||||
|
||||
/** Standard VectorDB API response envelope. */
|
||||
interface ApiResponse {
|
||||
code: number;
|
||||
msg: string;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
/** Search/hybridSearch response shape. */
|
||||
export interface SearchResponse {
|
||||
documents: Array<Array<Record<string, unknown>>>;
|
||||
}
|
||||
|
||||
/** Query response shape. */
|
||||
export interface QueryResponse {
|
||||
documents: Array<Record<string, unknown>>;
|
||||
count?: number;
|
||||
}
|
||||
|
||||
/** Collection info from describeCollection. */
|
||||
export interface CollectionInfo {
|
||||
collection: string;
|
||||
database: string;
|
||||
documentCount?: number;
|
||||
embedding?: {
|
||||
field: string;
|
||||
vectorField: string;
|
||||
model: string;
|
||||
};
|
||||
indexes?: Array<Record<string, unknown>>;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
export class TcvdbApiError extends Error {
|
||||
readonly apiCode: number;
|
||||
constructor(path: string, code: number, msg: string) {
|
||||
super(`VectorDB ${path}: code=${code}, msg=${msg}`);
|
||||
this.name = "TcvdbApiError";
|
||||
this.apiCode = code;
|
||||
}
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Client
|
||||
// ============================
|
||||
|
||||
const TAG = "[memory-tdai][tcvdb-client]";
|
||||
const MAX_RETRIES = 2;
|
||||
|
||||
export class TcvdbClient {
|
||||
private readonly baseUrl: string;
|
||||
private readonly authHeader: string;
|
||||
private readonly database: string;
|
||||
private readonly timeout: number;
|
||||
private readonly logger?: StoreLogger;
|
||||
/** undici dispatcher for HTTPS + custom CA. */
|
||||
private readonly dispatcher?: Dispatcher;
|
||||
|
||||
constructor(config: TcvdbClientConfig, logger?: StoreLogger) {
|
||||
this.baseUrl = config.url.replace(/\/+$/, "");
|
||||
this.authHeader = `Bearer account=${config.username}&api_key=${config.apiKey}`;
|
||||
this.database = config.database;
|
||||
this.timeout = config.timeout;
|
||||
this.logger = logger;
|
||||
|
||||
// Log connection info at construction time.
|
||||
this.logger?.debug?.(`${TAG} url=${this.baseUrl} db=${this.database} timeout=${this.timeout}${this.baseUrl.startsWith("https://") ? ` https=true caPemPath=${config.caPemPath ?? "(none)"}` : ""}`);
|
||||
|
||||
// For HTTPS with a custom CA certificate, create a dedicated undici Agent.
|
||||
// We use undici.request() instead of global fetch because fetch's
|
||||
// `dispatcher` option is unreliable across Node versions.
|
||||
if (this.baseUrl.startsWith("https://") && config.caPemPath) {
|
||||
try {
|
||||
const ca = fs.readFileSync(config.caPemPath, "utf-8");
|
||||
this.dispatcher = new UndiciAgent({ connect: { ca } });
|
||||
this.logger?.debug?.(`${TAG} HTTPS enabled with CA from ${config.caPemPath}`);
|
||||
} catch (err) {
|
||||
this.logger?.error(`${TAG} Failed to load CA PEM from ${config.caPemPath}: ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Generic request ─────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Send a POST request to VectorDB API.
|
||||
* Handles auth, timeout, retries (5xx/timeout), and error unwrapping.
|
||||
*/
|
||||
async request<T = ApiResponse>(path: string, body: Record<string, unknown>): Promise<T> {
|
||||
let lastError: Error | undefined;
|
||||
|
||||
for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
|
||||
try {
|
||||
this.logger?.debug?.(`${TAG} → ${path} body=${JSON.stringify(body).slice(0, 500)}`);
|
||||
const { statusCode, body: respBody } = await undiciRequest(`${this.baseUrl}${path}`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": this.authHeader,
|
||||
},
|
||||
body: JSON.stringify(body),
|
||||
signal: AbortSignal.timeout(this.timeout),
|
||||
...(this.dispatcher ? { dispatcher: this.dispatcher } : {}),
|
||||
});
|
||||
|
||||
const text = await respBody.text();
|
||||
const json = JSON.parse(text) as ApiResponse;
|
||||
this.logger?.debug?.(`${TAG} ← ${path} status=${statusCode} code=${json.code} msg=${json.msg} keys=[${Object.keys(json).join(",")}]`);
|
||||
|
||||
if (json.code !== 0) {
|
||||
const err = new TcvdbApiError(path, json.code, json.msg);
|
||||
if (statusCode !== undefined && statusCode >= 400 && statusCode < 500) throw err;
|
||||
lastError = err;
|
||||
continue;
|
||||
}
|
||||
|
||||
return json as unknown as T;
|
||||
} catch (err) {
|
||||
if (err instanceof TcvdbApiError && err.apiCode !== 0) throw err;
|
||||
lastError = err instanceof Error ? err : new Error(String(err));
|
||||
if (attempt < MAX_RETRIES) {
|
||||
const delay = 500 * (attempt + 1);
|
||||
this.logger?.debug?.(`${TAG} ${path} retry ${attempt + 1}/${MAX_RETRIES} in ${delay}ms`);
|
||||
await new Promise((r) => setTimeout(r, delay));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
throw lastError ?? new Error(`${TAG} ${path} failed after retries`);
|
||||
}
|
||||
|
||||
// ── Database operations ─────────────────────────────────
|
||||
|
||||
async createDatabase(dbName?: string): Promise<boolean> {
|
||||
const name = dbName ?? this.database;
|
||||
// SDK pattern: list first, create only if not found
|
||||
const listResp = await this.request<{ databases: string[] }>("/database/list", {});
|
||||
const exists = (listResp.databases ?? []).includes(name);
|
||||
if (exists) {
|
||||
this.logger?.debug?.(`${TAG} Database already exists: ${name}`);
|
||||
return false;
|
||||
}
|
||||
await this.request("/database/create", { database: name });
|
||||
this.logger?.info(`${TAG} Database created: ${name}`);
|
||||
return true;
|
||||
}
|
||||
|
||||
// ── Collection operations ───────────────────────────────
|
||||
|
||||
async createCollection(params: Record<string, unknown>): Promise<void> {
|
||||
const name = String(params.collection ?? "");
|
||||
// SDK pattern: try describe first, create only if not found (code 15302)
|
||||
try {
|
||||
await this.describeCollection(name);
|
||||
this.logger?.debug?.(`${TAG} Collection already exists: ${name}`);
|
||||
return;
|
||||
} catch (err) {
|
||||
if (!(err instanceof TcvdbApiError && err.apiCode === 15302)) {
|
||||
throw err; // unexpected error
|
||||
}
|
||||
// 15302 = collection not found → proceed to create
|
||||
}
|
||||
try {
|
||||
await this.request("/collection/create", {
|
||||
database: this.database,
|
||||
...params,
|
||||
});
|
||||
this.logger?.info(`${TAG} Collection created: ${name}`);
|
||||
} catch (err) {
|
||||
// 15202 = collection already exists — race between describe and create.
|
||||
// Semantically identical to "describe found it", so treat as success.
|
||||
if (err instanceof TcvdbApiError && err.apiCode === 15202) {
|
||||
this.logger?.debug?.(`${TAG} Collection already exists (race): ${name}`);
|
||||
return;
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
async describeCollection(collection: string): Promise<CollectionInfo> {
|
||||
const resp = await this.request<{ collection: CollectionInfo }>("/collection/describe", {
|
||||
database: this.database,
|
||||
collection,
|
||||
});
|
||||
return resp.collection;
|
||||
}
|
||||
|
||||
// ── Document operations ─────────────────────────────────
|
||||
|
||||
async upsert(collection: string, documents: Record<string, unknown>[]): Promise<void> {
|
||||
await this.request("/document/upsert", {
|
||||
database: this.database,
|
||||
collection,
|
||||
buildIndex: true,
|
||||
documents,
|
||||
});
|
||||
}
|
||||
|
||||
async search(collection: string, searchParams: Record<string, unknown>): Promise<SearchResponse> {
|
||||
return this.request<SearchResponse>("/document/search", {
|
||||
database: this.database,
|
||||
collection,
|
||||
readConsistency: "strongConsistency",
|
||||
search: searchParams,
|
||||
});
|
||||
}
|
||||
|
||||
async hybridSearch(collection: string, searchParams: Record<string, unknown>): Promise<SearchResponse> {
|
||||
return this.request<SearchResponse>("/document/hybridSearch", {
|
||||
database: this.database,
|
||||
collection,
|
||||
readConsistency: "strongConsistency",
|
||||
search: searchParams,
|
||||
});
|
||||
}
|
||||
|
||||
async query(collection: string, queryParams: Record<string, unknown>): Promise<QueryResponse> {
|
||||
return this.request<QueryResponse>("/document/query", {
|
||||
database: this.database,
|
||||
collection,
|
||||
readConsistency: "strongConsistency",
|
||||
query: queryParams,
|
||||
});
|
||||
}
|
||||
|
||||
async deleteDoc(collection: string, params: Record<string, unknown>): Promise<void> {
|
||||
await this.request("/document/delete", {
|
||||
database: this.database,
|
||||
collection,
|
||||
...params,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Count documents matching an optional filter.
|
||||
* Uses the dedicated /document/count endpoint.
|
||||
*/
|
||||
async count(collection: string, filter?: string): Promise<number> {
|
||||
const query: Record<string, unknown> = {};
|
||||
if (filter) query.filter = filter;
|
||||
const resp = await this.request<{ count: number }>("/document/count", {
|
||||
database: this.database,
|
||||
collection,
|
||||
readConsistency: "strongConsistency",
|
||||
query,
|
||||
});
|
||||
return resp.count ?? 0;
|
||||
}
|
||||
|
||||
// ── Convenience getters ─────────────────────────────────
|
||||
|
||||
getDatabase(): string {
|
||||
return this.database;
|
||||
}
|
||||
}
|
||||
+1180
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,328 @@
|
||||
/**
|
||||
* Memory Store Abstraction Layer — Core Types & Interfaces.
|
||||
*
|
||||
* This module defines the storage contracts that all backend implementations
|
||||
* (SQLite local, Tencent Cloud VectorDB, etc.) must satisfy.
|
||||
*
|
||||
* Design principles:
|
||||
* 1. **Backend-agnostic**: Upper-layer modules (hooks, tools, pipeline, record)
|
||||
* depend only on these interfaces — never on concrete implementations.
|
||||
* 2. **Capability-based**: Features like vector search, FTS, and hybrid search
|
||||
* are expressed as capability flags so callers can gracefully degrade.
|
||||
* 3. **Fault-tolerant**: All methods return empty results or `false` on
|
||||
* failure rather than throwing, unless explicitly documented otherwise.
|
||||
* 4. **Sync-first**: Matches current SQLite DatabaseSync usage. TCVDB backend
|
||||
* adapts internally without changing these signatures.
|
||||
*/
|
||||
|
||||
import type { MemoryRecord } from "../record/l1-writer.js";
|
||||
import type { EmbeddingProviderInfo } from "./embedding.js";
|
||||
|
||||
// Re-export so consumers can import everything from types.ts
|
||||
export type { MemoryRecord, EmbeddingProviderInfo };
|
||||
|
||||
// ============================
|
||||
// Common Types
|
||||
// ============================
|
||||
|
||||
/** Minimal logger interface accepted by store implementations. */
|
||||
export interface StoreLogger {
|
||||
debug?: (message: string) => void;
|
||||
info: (message: string) => void;
|
||||
warn: (message: string) => void;
|
||||
error: (message: string) => void;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// L1 Types (Structured Memories)
|
||||
// ============================
|
||||
|
||||
/** Result from an L1 vector similarity search. */
|
||||
export interface L1SearchResult {
|
||||
record_id: string;
|
||||
content: string;
|
||||
type: string;
|
||||
priority: number;
|
||||
scene_name: string;
|
||||
/** Similarity score (0–1, higher is better). */
|
||||
score: number;
|
||||
timestamp_str: string;
|
||||
timestamp_start: string;
|
||||
timestamp_end: string;
|
||||
session_key: string;
|
||||
session_id: string;
|
||||
metadata_json: string;
|
||||
}
|
||||
|
||||
/** Result from an L1 FTS keyword search. */
|
||||
export interface L1FtsResult {
|
||||
record_id: string;
|
||||
content: string;
|
||||
type: string;
|
||||
priority: number;
|
||||
scene_name: string;
|
||||
/** BM25-derived score (0–1, higher is better). */
|
||||
score: number;
|
||||
timestamp_str: string;
|
||||
timestamp_start: string;
|
||||
timestamp_end: string;
|
||||
session_key: string;
|
||||
session_id: string;
|
||||
metadata_json: string;
|
||||
}
|
||||
|
||||
/** Filter options for querying L1 records. */
|
||||
export interface L1QueryFilter {
|
||||
sessionKey?: string;
|
||||
sessionId?: string;
|
||||
/** Only return records with updated_time strictly after this ISO 8601 UTC timestamp. */
|
||||
updatedAfter?: string;
|
||||
}
|
||||
|
||||
/** Row shape returned by L1 query methods. */
|
||||
export interface L1RecordRow {
|
||||
record_id: string;
|
||||
content: string;
|
||||
type: string;
|
||||
priority: number;
|
||||
scene_name: string;
|
||||
session_key: string;
|
||||
session_id: string;
|
||||
timestamp_str: string;
|
||||
timestamp_start: string;
|
||||
timestamp_end: string;
|
||||
created_time: string;
|
||||
updated_time: string;
|
||||
metadata_json: string;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// L0 Types (Raw Conversations)
|
||||
// ============================
|
||||
|
||||
/** An L0 conversation message record for vector indexing. */
|
||||
export interface L0Record {
|
||||
id: string;
|
||||
sessionKey: string;
|
||||
sessionId: string;
|
||||
role: string;
|
||||
messageText: string;
|
||||
recordedAt: string;
|
||||
/** Original message timestamp (epoch ms). */
|
||||
timestamp: number;
|
||||
}
|
||||
|
||||
/** Result from an L0 vector similarity search. */
|
||||
export interface L0SearchResult {
|
||||
record_id: string;
|
||||
session_key: string;
|
||||
session_id: string;
|
||||
role: string;
|
||||
message_text: string;
|
||||
/** Similarity score (0–1, higher is better). */
|
||||
score: number;
|
||||
recorded_at: string;
|
||||
timestamp: number;
|
||||
}
|
||||
|
||||
/** Result from an L0 FTS keyword search. */
|
||||
export interface L0FtsResult {
|
||||
record_id: string;
|
||||
session_key: string;
|
||||
session_id: string;
|
||||
role: string;
|
||||
message_text: string;
|
||||
/** BM25-derived score (0–1, higher is better). */
|
||||
score: number;
|
||||
recorded_at: string;
|
||||
timestamp: number;
|
||||
}
|
||||
|
||||
/** Raw L0 row returned by query methods (used by L1 runner). */
|
||||
export interface L0QueryRow {
|
||||
record_id: string;
|
||||
session_key: string;
|
||||
session_id: string;
|
||||
role: string;
|
||||
message_text: string;
|
||||
recorded_at: string;
|
||||
timestamp: number;
|
||||
}
|
||||
|
||||
/** L0 messages grouped by session ID (for L1 runner). */
|
||||
export interface L0SessionGroup {
|
||||
sessionId: string;
|
||||
messages: Array<{
|
||||
id: string;
|
||||
role: string;
|
||||
content: string;
|
||||
timestamp: number;
|
||||
/** Epoch ms when this message was recorded into L0 (used by L1 cursor). */
|
||||
recordedAtMs: number;
|
||||
}>;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Store Init Result
|
||||
// ============================
|
||||
|
||||
/** Result of store initialization. */
|
||||
export interface StoreInitResult {
|
||||
/** Whether embeddings need to be regenerated (provider/model change). */
|
||||
needsReindex: boolean;
|
||||
/** Human-readable reason (for logging). */
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Capability Flags
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Describes what search capabilities a store backend supports.
|
||||
* Callers use this to select search strategies and degrade gracefully.
|
||||
*/
|
||||
export interface StoreCapabilities {
|
||||
/** Whether vector (embedding) search is available. */
|
||||
vectorSearch: boolean;
|
||||
/** Whether FTS (full-text keyword) search is available. */
|
||||
ftsSearch: boolean;
|
||||
/** Whether native hybrid search is supported (e.g., TCVDB hybridSearch). */
|
||||
nativeHybridSearch: boolean;
|
||||
/** Whether the store supports sparse vectors (BM25 encoding). */
|
||||
sparseVectors: boolean;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// L2/L3 Profile Sync Types
|
||||
// ============================
|
||||
|
||||
/** Canonical L2/L3 profile row shared between local cache and remote store. */
|
||||
export interface ProfileRecord {
|
||||
/** Stable ID: `profile:v1:${sha256(scope + "\0" + type + "\0" + filename)}`. */
|
||||
id: string;
|
||||
type: "l2" | "l3";
|
||||
filename: string;
|
||||
content: string;
|
||||
contentMd5: string;
|
||||
agentId?: string;
|
||||
version: number;
|
||||
createdAtMs: number;
|
||||
updatedAtMs: number;
|
||||
}
|
||||
|
||||
/** Profile upsert payload with optimistic-lock baseline from the last pull. */
|
||||
export interface ProfileSyncRecord extends ProfileRecord {
|
||||
baselineVersion?: number;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// IMemoryStore — The Core Abstraction
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Unified memory store interface.
|
||||
*
|
||||
* Implementations:
|
||||
* - `SqliteMemoryStore` (sqlite.ts) — local SQLite + sqlite-vec + FTS5
|
||||
* - `TcvdbMemoryStore` (tcvdb.ts) — Tencent Cloud VectorDB (future)
|
||||
*
|
||||
* All methods are fault-tolerant: they return empty results or `false` on
|
||||
* failure rather than throwing, unless explicitly documented otherwise.
|
||||
*/
|
||||
/**
|
||||
* Helper type: a value that may be sync or async.
|
||||
* Callers should always `await` the result — it's safe for both sync and async values.
|
||||
*/
|
||||
export type MaybePromise<T> = T | Promise<T>;
|
||||
|
||||
export interface IMemoryStore {
|
||||
// ── Capabilities ───────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Whether this store supports deferred (background) embedding updates.
|
||||
*
|
||||
* When `true`, auto-capture writes metadata-only via `upsertL0(record, undefined)`
|
||||
* and later calls `updateL0Embedding()` in a fire-and-forget background task.
|
||||
* When `false` or absent, embedding is computed inline and passed to `upsertL0()`.
|
||||
*/
|
||||
readonly supportsDeferredEmbedding?: boolean;
|
||||
|
||||
// ── Lifecycle (always sync) ──────────────────────────────
|
||||
|
||||
init(providerInfo?: EmbeddingProviderInfo): MaybePromise<StoreInitResult>;
|
||||
isDegraded(): boolean;
|
||||
getCapabilities(): StoreCapabilities;
|
||||
close(): void;
|
||||
|
||||
// ── L1 Write ─────────────────────────────────────────────
|
||||
|
||||
upsertL1(record: MemoryRecord, embedding?: Float32Array): MaybePromise<boolean>;
|
||||
deleteL1(recordId: string): MaybePromise<boolean>;
|
||||
deleteL1Batch(recordIds: string[]): MaybePromise<boolean>;
|
||||
deleteL1Expired(cutoffIso: string): MaybePromise<number>;
|
||||
|
||||
// ── L1 Read ──────────────────────────────────────────────
|
||||
|
||||
countL1(): MaybePromise<number>;
|
||||
queryL1Records(filter?: L1QueryFilter): MaybePromise<L1RecordRow[]>;
|
||||
getAllL1Texts(): MaybePromise<Array<{ record_id: string; content: string; updated_time: string }>>;
|
||||
|
||||
// ── L1 Search ────────────────────────────────────────────
|
||||
|
||||
searchL1Vector(queryEmbedding: Float32Array, topK?: number, queryText?: string): MaybePromise<L1SearchResult[]>;
|
||||
searchL1Fts(ftsQuery: string, limit?: number): MaybePromise<L1FtsResult[]>;
|
||||
searchL1Hybrid?(params: {
|
||||
query?: string;
|
||||
queryEmbedding?: Float32Array;
|
||||
sparseVector?: Array<[number, number]>;
|
||||
topK?: number;
|
||||
}): MaybePromise<L1SearchResult[]>;
|
||||
|
||||
// ── L0 Write ─────────────────────────────────────────────
|
||||
|
||||
upsertL0(record: L0Record, embedding?: Float32Array): MaybePromise<boolean>;
|
||||
/** Update only the vector embedding for an existing L0 record (sqlite background path). */
|
||||
updateL0Embedding?(recordId: string, embedding: Float32Array): MaybePromise<boolean>;
|
||||
deleteL0(recordId: string): MaybePromise<boolean>;
|
||||
deleteL0Expired(cutoffIso: string): MaybePromise<number>;
|
||||
|
||||
// ── L0 Read ──────────────────────────────────────────────
|
||||
|
||||
countL0(): MaybePromise<number>;
|
||||
queryL0ForL1(sessionKey: string, afterRecordedAtMs?: number, limit?: number): MaybePromise<L0QueryRow[]>;
|
||||
queryL0GroupedBySessionId(sessionKey: string, afterRecordedAtMs?: number, limit?: number): MaybePromise<L0SessionGroup[]>;
|
||||
getAllL0Texts(): MaybePromise<Array<{ record_id: string; message_text: string; recorded_at: string }>>;
|
||||
|
||||
// ── L0 Search ────────────────────────────────────────────
|
||||
|
||||
searchL0Vector(queryEmbedding: Float32Array, topK?: number, queryText?: string): MaybePromise<L0SearchResult[]>;
|
||||
searchL0Fts(ftsQuery: string, limit?: number): MaybePromise<L0FtsResult[]>;
|
||||
|
||||
pullProfiles?(): Promise<ProfileRecord[]>;
|
||||
syncProfiles?(records: ProfileSyncRecord[]): Promise<void>;
|
||||
deleteProfiles?(recordIds: string[]): Promise<void>;
|
||||
|
||||
// ── Re-index ─────────────────────────────────────────────
|
||||
|
||||
reindexAll(
|
||||
embedFn: (text: string) => Promise<Float32Array>,
|
||||
onProgress?: (done: number, total: number, layer: "L1" | "L0") => void,
|
||||
): Promise<{ l1Count: number; l0Count: number }>;
|
||||
|
||||
// ── FTS (always sync — cached flag) ──────────────────────
|
||||
|
||||
isFtsAvailable(): boolean;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// IEmbeddingService — re-exported from embedding.ts for convenience
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Re-export EmbeddingService as IEmbeddingService for backward compatibility.
|
||||
* The canonical definition lives in `./embedding.ts`. All concrete implementations
|
||||
* (LocalEmbeddingService, OpenAIEmbeddingService, NoopEmbeddingService) implement
|
||||
* the EmbeddingService interface from embedding.ts.
|
||||
*/
|
||||
export type { EmbeddingService as IEmbeddingService } from "./embedding.js";
|
||||
@@ -10,8 +10,8 @@
|
||||
* The tool is registered via `api.registerTool()` in index.ts.
|
||||
*/
|
||||
|
||||
import type { VectorStore, L0VectorSearchResult } from "../store/vector-store.js";
|
||||
import { buildFtsQuery } from "../store/vector-store.js";
|
||||
import type { IMemoryStore, L0SearchResult } from "../store/types.js";
|
||||
import { buildFtsQuery } from "../store/sqlite.js";
|
||||
import type { EmbeddingService } from "../store/embedding.js";
|
||||
|
||||
// ============================
|
||||
@@ -90,7 +90,7 @@ export async function executeConversationSearch(params: {
|
||||
query: string;
|
||||
limit: number;
|
||||
sessionKey?: string;
|
||||
vectorStore?: VectorStore;
|
||||
vectorStore?: IMemoryStore;
|
||||
embeddingService?: EmbeddingService;
|
||||
logger?: Logger;
|
||||
}): Promise<ConversationSearchResult> {
|
||||
@@ -152,7 +152,7 @@ export async function executeConversationSearch(params: {
|
||||
return [];
|
||||
}
|
||||
logger?.debug?.(`${TAG} [hybrid-fts] FTS5 query: "${ftsQuery}"`);
|
||||
const ftsResults = vectorStore.ftsSearchL0(ftsQuery, candidateK);
|
||||
const ftsResults = await vectorStore.searchL0Fts(ftsQuery, candidateK);
|
||||
logger?.debug?.(`${TAG} [hybrid-fts] FTS5 returned ${ftsResults.length} candidates`);
|
||||
return ftsResults.map((r) => ({
|
||||
id: r.record_id,
|
||||
@@ -179,7 +179,7 @@ export async function executeConversationSearch(params: {
|
||||
logger?.debug?.(
|
||||
`${TAG} [hybrid-vec] Embedding OK, dims=${queryEmbedding.length}, searching top-${candidateK}...`,
|
||||
);
|
||||
const vecResults: L0VectorSearchResult[] = vectorStore.searchL0(queryEmbedding, candidateK);
|
||||
const vecResults: L0SearchResult[] = await vectorStore.searchL0Vector(queryEmbedding, candidateK, query);
|
||||
logger?.debug?.(`${TAG} [hybrid-vec] Vector search returned ${vecResults.length} candidates`);
|
||||
return vecResults.map((r) => ({
|
||||
id: r.record_id,
|
||||
|
||||
@@ -10,8 +10,8 @@
|
||||
* The tool is registered via `api.registerTool()` in index.ts.
|
||||
*/
|
||||
|
||||
import type { VectorStore, VectorSearchResult } from "../store/vector-store.js";
|
||||
import { buildFtsQuery } from "../store/vector-store.js";
|
||||
import type { IMemoryStore, L1SearchResult } from "../store/types.js";
|
||||
import { buildFtsQuery } from "../store/sqlite.js";
|
||||
import type { EmbeddingService } from "../store/embedding.js";
|
||||
|
||||
// ============================
|
||||
@@ -90,7 +90,7 @@ export async function executeMemorySearch(params: {
|
||||
limit: number;
|
||||
type?: string;
|
||||
scene?: string;
|
||||
vectorStore?: VectorStore;
|
||||
vectorStore?: IMemoryStore;
|
||||
embeddingService?: EmbeddingService;
|
||||
logger?: Logger;
|
||||
}): Promise<MemorySearchResult> {
|
||||
@@ -153,7 +153,7 @@ export async function executeMemorySearch(params: {
|
||||
return [];
|
||||
}
|
||||
logger?.debug?.(`${TAG} [hybrid-fts] FTS5 query: "${ftsQuery}"`);
|
||||
const ftsResults = vectorStore.ftsSearchL1(ftsQuery, candidateK);
|
||||
const ftsResults = await vectorStore.searchL1Fts(ftsQuery, candidateK);
|
||||
logger?.debug?.(`${TAG} [hybrid-fts] FTS5 returned ${ftsResults.length} candidates`);
|
||||
return ftsResults.map((r) => ({
|
||||
id: r.record_id,
|
||||
@@ -182,7 +182,7 @@ export async function executeMemorySearch(params: {
|
||||
logger?.debug?.(
|
||||
`${TAG} [hybrid-vec] Embedding OK, dims=${queryEmbedding.length}, searching top-${candidateK}...`,
|
||||
);
|
||||
const vecResults: VectorSearchResult[] = vectorStore.search(queryEmbedding, candidateK);
|
||||
const vecResults: L1SearchResult[] = await vectorStore.searchL1Vector(queryEmbedding, candidateK, query);
|
||||
logger?.debug?.(`${TAG} [hybrid-vec] Vector search returned ${vecResults.length} candidates`);
|
||||
return vecResults.map((r) => ({
|
||||
id: r.record_id,
|
||||
|
||||
+7
-62
@@ -297,63 +297,6 @@ export class CheckpointManager {
|
||||
// Public API — mutating (all serialized via file lock)
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Advance the captured timestamp after successful upload/recording.
|
||||
* Also updates total_processed and persona counters.
|
||||
*
|
||||
* NOTE: This advances the GLOBAL cursor (`Checkpoint.last_captured_timestamp`).
|
||||
* For per-session cursor advancement, use `advanceSessionCapturedTimestamp()`.
|
||||
* The global cursor is kept for aggregate stats / backward compat, but should
|
||||
* NOT be used as the L0 incremental-capture filter (use per-session instead).
|
||||
*/
|
||||
async advanceCapturedTimestamp(maxTimestamp: number, messageCount: number): Promise<void> {
|
||||
const cp = await this.mutate((cp) => {
|
||||
cp.last_captured_timestamp = maxTimestamp;
|
||||
cp.total_processed += messageCount;
|
||||
cp.memories_since_last_persona += messageCount;
|
||||
});
|
||||
this.logger.info(
|
||||
`[checkpoint] advanceCapturedTimestamp: -> ${maxTimestamp} (+${messageCount} msgs), ` +
|
||||
`total_processed=${cp.total_processed}, memories_since_last_persona=${cp.memories_since_last_persona}`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Advance the per-session L0 capture cursor after recording messages.
|
||||
* This is the **primary** cursor for incremental L0 recording — each session
|
||||
* tracks its own progress independently, preventing cross-session cursor drift.
|
||||
*
|
||||
* Also updates the global cursor / total_processed for aggregate stats.
|
||||
*/
|
||||
async advanceSessionCapturedTimestamp(
|
||||
sessionKey: string,
|
||||
maxTimestamp: number,
|
||||
messageCount: number,
|
||||
): Promise<void> {
|
||||
const cp = await this.mutate((cp) => {
|
||||
// Per-session cursor (runner-owned)
|
||||
const state = this.getRunnerState(cp, sessionKey);
|
||||
state.last_captured_timestamp = maxTimestamp;
|
||||
// Global stats (aggregate only — not used for filtering)
|
||||
cp.last_captured_timestamp = Math.max(cp.last_captured_timestamp, maxTimestamp);
|
||||
cp.total_processed += messageCount;
|
||||
cp.memories_since_last_persona += messageCount;
|
||||
});
|
||||
this.logger.info(
|
||||
`[checkpoint] advanceSessionCapturedTimestamp session=${sessionKey}: -> ${maxTimestamp} ` +
|
||||
`(+${messageCount} msgs), total_processed=${cp.total_processed}`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Increment L0 conversation count.
|
||||
*/
|
||||
async incrementL0ConversationCount(): Promise<void> {
|
||||
await this.mutate((cp) => {
|
||||
cp.l0_conversations_count += 1;
|
||||
});
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Persona methods (L3)
|
||||
// ============================
|
||||
@@ -460,17 +403,20 @@ export class CheckpointManager {
|
||||
/**
|
||||
* Mark L1 extraction completed: reset sinceL1 counter, advance L1 cursor,
|
||||
* and optionally save the last scene name for cross-batch continuity.
|
||||
*
|
||||
* @param cursorRecordedAtMs - The max recorded_at epoch ms of processed L0 messages.
|
||||
* This becomes the new `last_l1_cursor` value (recorded_at semantics, not conversation timestamp).
|
||||
*/
|
||||
async markL1ExtractionComplete(
|
||||
sessionKey: string,
|
||||
memoriesExtracted: number,
|
||||
cursorTimestamp?: number,
|
||||
cursorRecordedAtMs?: number,
|
||||
lastSceneName?: string,
|
||||
): Promise<void> {
|
||||
await this.mutate((cp) => {
|
||||
const state = this.getRunnerState(cp, sessionKey);
|
||||
if (cursorTimestamp) {
|
||||
state.last_l1_cursor = cursorTimestamp;
|
||||
if (cursorRecordedAtMs) {
|
||||
state.last_l1_cursor = cursorRecordedAtMs;
|
||||
}
|
||||
if (lastSceneName !== undefined) {
|
||||
state.last_scene_name = lastSceneName;
|
||||
@@ -480,7 +426,7 @@ export class CheckpointManager {
|
||||
});
|
||||
this.logger.info(
|
||||
`[checkpoint] markL1ExtractionComplete session=${sessionKey}: ` +
|
||||
`extracted=${memoriesExtracted}, cursor=${cursorTimestamp ?? "(unchanged)"}, ` +
|
||||
`extracted=${memoriesExtracted}, cursor=${cursorRecordedAtMs ?? "(unchanged)"}, ` +
|
||||
`lastScene="${lastSceneName ?? "(unchanged)"}"`,
|
||||
);
|
||||
}
|
||||
@@ -533,7 +479,6 @@ export class CheckpointManager {
|
||||
// Global stats (aggregate only — not used for filtering)
|
||||
cp.last_captured_timestamp = Math.max(cp.last_captured_timestamp, result.maxTimestamp);
|
||||
cp.total_processed += result.messageCount;
|
||||
cp.memories_since_last_persona += result.messageCount;
|
||||
// Increment L0 conversation count (was a separate mutate() call before)
|
||||
cp.l0_conversations_count += 1;
|
||||
}
|
||||
|
||||
@@ -14,6 +14,7 @@ import fsSync from "node:fs";
|
||||
import path from "node:path";
|
||||
import os from "node:os";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/core";
|
||||
import { report } from "../report/reporter.js";
|
||||
|
||||
/**
|
||||
@@ -55,7 +56,42 @@ interface RunnerLogger {
|
||||
}
|
||||
|
||||
// Dynamic import type — runEmbeddedPiAgent is an internal API
|
||||
type RunEmbeddedPiAgentFn = (params: Record<string, unknown>) => Promise<unknown>;
|
||||
// Prefer the public plugin runtime signature so host-injected runtimes stay assignable.
|
||||
type RunEmbeddedPiAgentFn = OpenClawPluginApi["runtime"]["agent"]["runEmbeddedPiAgent"];
|
||||
|
||||
export interface EmbeddedAgentRuntimeLike {
|
||||
runEmbeddedPiAgent?: RunEmbeddedPiAgentFn;
|
||||
}
|
||||
|
||||
let _preferredAgentRuntime: EmbeddedAgentRuntimeLike | undefined;
|
||||
|
||||
export function setPreferredEmbeddedAgentRuntime(
|
||||
agentRuntime: EmbeddedAgentRuntimeLike | undefined,
|
||||
): void {
|
||||
_preferredAgentRuntime = agentRuntime;
|
||||
}
|
||||
|
||||
function resolveInjectedRunEmbeddedPiAgent(
|
||||
agentRuntime?: EmbeddedAgentRuntimeLike,
|
||||
): RunEmbeddedPiAgentFn | undefined {
|
||||
const candidate =
|
||||
agentRuntime?.runEmbeddedPiAgent ?? _preferredAgentRuntime?.runEmbeddedPiAgent;
|
||||
return typeof candidate === "function" ? candidate : undefined;
|
||||
}
|
||||
|
||||
async function resolveRunEmbeddedPiAgent(
|
||||
agentRuntime: EmbeddedAgentRuntimeLike | undefined,
|
||||
logger?: RunnerLogger,
|
||||
): Promise<RunEmbeddedPiAgentFn> {
|
||||
const injected = resolveInjectedRunEmbeddedPiAgent(agentRuntime);
|
||||
if (injected) {
|
||||
logger?.debug?.(
|
||||
`${TAG} resolveRunEmbeddedPiAgent: using injected runtime.agent.runEmbeddedPiAgent`,
|
||||
);
|
||||
return injected;
|
||||
}
|
||||
return loadRunEmbeddedPiAgent(logger);
|
||||
}
|
||||
|
||||
// ── Core import (mirrors voice-call/core-bridge.ts — dist/ only, no jiti) ──
|
||||
|
||||
@@ -123,7 +159,17 @@ function loadRunEmbeddedPiAgent(logger?: RunnerLogger): Promise<RunEmbeddedPiAge
|
||||
* the cold-start penalty on the first actual extraction run.
|
||||
* Returns immediately (fire-and-forget) — errors are swallowed.
|
||||
*/
|
||||
export function prewarmEmbeddedAgent(logger?: RunnerLogger): void {
|
||||
export function prewarmEmbeddedAgent(
|
||||
logger?: RunnerLogger,
|
||||
agentRuntime?: EmbeddedAgentRuntimeLike,
|
||||
): void {
|
||||
if (resolveInjectedRunEmbeddedPiAgent(agentRuntime)) {
|
||||
logger?.debug?.(
|
||||
`${TAG} prewarmEmbeddedAgent: runtime capability already available, skipping legacy preload`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
loadRunEmbeddedPiAgent(logger).catch((err) => {
|
||||
logger?.warn(`${TAG} prewarmEmbeddedAgent: failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`);
|
||||
});
|
||||
@@ -232,6 +278,8 @@ export interface CleanContextRunnerOptions {
|
||||
* automatically falls back to the main config's `agents.defaults.model`.
|
||||
*/
|
||||
modelRef?: string;
|
||||
/** Preferred runtime seam. When absent, falls back to the legacy dist bridge. */
|
||||
agentRuntime?: EmbeddedAgentRuntimeLike;
|
||||
/** Allow the LLM to use tools (read_file, write_to_file, etc). Default: false */
|
||||
enableTools?: boolean;
|
||||
/** Logger instance for detailed tracing */
|
||||
@@ -318,11 +366,14 @@ export class CleanContextRunner {
|
||||
try {
|
||||
const sessionFile = path.join(tmpDir, "session.json");
|
||||
|
||||
// Phase 1: Load runEmbeddedPiAgent (fast if dist/ exists or already cached)
|
||||
// Phase 1: Resolve runEmbeddedPiAgent (prefer runtime, fallback to legacy dist bridge)
|
||||
const importStartMs = Date.now();
|
||||
const runEmbeddedPiAgent = await loadRunEmbeddedPiAgent(this.logger);
|
||||
const runEmbeddedPiAgent = await resolveRunEmbeddedPiAgent(
|
||||
this.options.agentRuntime,
|
||||
this.logger,
|
||||
);
|
||||
const importElapsedMs = Date.now() - importStartMs;
|
||||
this.logger?.debug?.(`${TAG} run() dynamic import phase: ${importElapsedMs}ms`);
|
||||
this.logger?.debug?.(`${TAG} run() runner resolution phase: ${importElapsedMs}ms`);
|
||||
|
||||
// Derive a config with plugins disabled to prevent loadOpenClawPlugins
|
||||
// from re-registering plugins when the workspaceDir differs from the
|
||||
@@ -347,10 +398,10 @@ export class CleanContextRunner {
|
||||
},
|
||||
};
|
||||
|
||||
// Build the effective prompt:
|
||||
// If systemPrompt is provided, pass it as a separate parameter to the agent
|
||||
// and use `prompt` as the user message. Fallback: prepend to prompt if the
|
||||
// embedded agent doesn't support systemPrompt natively.
|
||||
// Build the effective prompt.
|
||||
// Keep prepending the optional systemPrompt into the user-visible prompt so
|
||||
// runtime and legacy fallback paths preserve the same behavior without
|
||||
// relying on a newer native extraSystemPrompt contract.
|
||||
const effectivePrompt = params.systemPrompt
|
||||
? `${params.systemPrompt}\n\n---\n\n${params.prompt}`
|
||||
: params.prompt;
|
||||
@@ -368,7 +419,6 @@ export class CleanContextRunner {
|
||||
workspaceDir: cleanWorkspace,
|
||||
config: cleanConfig,
|
||||
prompt: effectivePrompt,
|
||||
systemPrompt: params.systemPrompt,
|
||||
timeoutMs: params.timeoutMs ?? 120_000,
|
||||
runId,
|
||||
provider: this.resolvedProvider,
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
/**
|
||||
* Manifest — self-describing metadata for a memory-tdai data directory.
|
||||
*
|
||||
* Lives at `<dataDir>/.metadata/manifest.json`.
|
||||
*
|
||||
* - **store**: written once on first successful store init; never overwritten.
|
||||
* On subsequent starts the current config is compared against the persisted
|
||||
* store binding — mismatches are logged as warnings.
|
||||
* - **seed**: written once when a seed run completes; null for live-runtime dirs.
|
||||
*
|
||||
* This file is informational / read-only from the user's perspective.
|
||||
* The plugin reads it on startup for consistency checks.
|
||||
*/
|
||||
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
// ============================
|
||||
// Types
|
||||
// ============================
|
||||
|
||||
export interface ManifestStoreInfo {
|
||||
type: "sqlite" | "tcvdb";
|
||||
sqlite?: {
|
||||
/** Relative path to the SQLite DB file (relative to dataDir). */
|
||||
path: string;
|
||||
};
|
||||
tcvdb?: {
|
||||
url: string;
|
||||
database: string;
|
||||
/** User-friendly alias (optional). */
|
||||
alias?: string;
|
||||
};
|
||||
}
|
||||
|
||||
export interface ManifestSeedInfo {
|
||||
/** Original input file name (basename only). */
|
||||
inputFile?: string;
|
||||
sessions: number;
|
||||
rounds: number;
|
||||
messages: number;
|
||||
startedAt: string;
|
||||
completedAt: string;
|
||||
}
|
||||
|
||||
export interface Manifest {
|
||||
/** Schema version for future migrations. */
|
||||
version: 1;
|
||||
/** Timestamp when the manifest was first created. */
|
||||
createdAt: string;
|
||||
/** Store binding — written once on first init. */
|
||||
store: ManifestStoreInfo;
|
||||
/** Seed run info — null for live-runtime directories. */
|
||||
seed: ManifestSeedInfo | null;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Paths
|
||||
// ============================
|
||||
|
||||
const METADATA_DIR = ".metadata";
|
||||
const MANIFEST_FILE = "manifest.json";
|
||||
|
||||
export function manifestPath(dataDir: string): string {
|
||||
return path.join(dataDir, METADATA_DIR, MANIFEST_FILE);
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Read / Write
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Read an existing manifest from disk. Returns `null` if not found or unparseable.
|
||||
*/
|
||||
export function readManifest(dataDir: string): Manifest | null {
|
||||
const p = manifestPath(dataDir);
|
||||
try {
|
||||
if (!fs.existsSync(p)) return null;
|
||||
const raw = fs.readFileSync(p, "utf-8");
|
||||
return JSON.parse(raw) as Manifest;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Write a manifest to disk (creates `.metadata/` if needed).
|
||||
*/
|
||||
export function writeManifest(dataDir: string, manifest: Manifest): void {
|
||||
const dir = path.join(dataDir, METADATA_DIR);
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
manifestPath(dataDir),
|
||||
JSON.stringify(manifest, null, 2) + "\n",
|
||||
"utf-8",
|
||||
);
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Store binding helpers
|
||||
// ============================
|
||||
|
||||
export interface StoreConfigSnapshot {
|
||||
type: "sqlite" | "tcvdb";
|
||||
sqlitePath?: string;
|
||||
tcvdbUrl?: string;
|
||||
tcvdbDatabase?: string;
|
||||
tcvdbAlias?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a ManifestStoreInfo from the current store config snapshot.
|
||||
*/
|
||||
export function buildStoreInfo(snapshot: StoreConfigSnapshot): ManifestStoreInfo {
|
||||
const info: ManifestStoreInfo = { type: snapshot.type };
|
||||
if (snapshot.type === "sqlite") {
|
||||
info.sqlite = { path: snapshot.sqlitePath ?? "vectors.db" };
|
||||
} else {
|
||||
info.tcvdb = {
|
||||
url: snapshot.tcvdbUrl!,
|
||||
database: snapshot.tcvdbDatabase!,
|
||||
alias: snapshot.tcvdbAlias || undefined,
|
||||
};
|
||||
}
|
||||
return info;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compare the persisted store binding against the current config.
|
||||
* Returns a list of human-readable mismatch descriptions (empty = all good).
|
||||
*/
|
||||
export function diffStoreBinding(
|
||||
persisted: ManifestStoreInfo,
|
||||
current: ManifestStoreInfo,
|
||||
): string[] {
|
||||
const diffs: string[] = [];
|
||||
|
||||
if (persisted.type !== current.type) {
|
||||
diffs.push(`store type changed: ${persisted.type} → ${current.type}`);
|
||||
return diffs; // no point comparing fields across different types
|
||||
}
|
||||
|
||||
if (persisted.type === "sqlite" && current.type === "sqlite") {
|
||||
if (persisted.sqlite?.path !== current.sqlite?.path) {
|
||||
diffs.push(`sqlite path changed: ${persisted.sqlite?.path} → ${current.sqlite?.path}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (persisted.type === "tcvdb" && current.type === "tcvdb") {
|
||||
if (persisted.tcvdb?.url !== current.tcvdb?.url) {
|
||||
diffs.push(`tcvdb url changed: ${persisted.tcvdb?.url} → ${current.tcvdb?.url}`);
|
||||
}
|
||||
if (persisted.tcvdb?.database !== current.tcvdb?.database) {
|
||||
diffs.push(`tcvdb database changed: ${persisted.tcvdb?.database} → ${current.tcvdb?.database}`);
|
||||
}
|
||||
}
|
||||
|
||||
return diffs;
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
|
||||
import type { VectorStore } from "../store/vector-store.js";
|
||||
import type { IMemoryStore } from "../store/types.js";
|
||||
import { ManagedTimer } from "./managed-timer.js";
|
||||
|
||||
interface Logger {
|
||||
@@ -16,7 +16,7 @@ export interface MemoryCleanerOptions {
|
||||
retentionDays: number;
|
||||
cleanTime: string;
|
||||
logger?: Logger;
|
||||
vectorStore?: VectorStore;
|
||||
vectorStore?: IMemoryStore;
|
||||
}
|
||||
|
||||
interface CleanupStats {
|
||||
@@ -33,14 +33,14 @@ const L1_DIR_NAME = "records";
|
||||
export class LocalMemoryCleaner {
|
||||
private readonly timer: ManagedTimer;
|
||||
private destroyed = false;
|
||||
private vectorStore?: VectorStore;
|
||||
private vectorStore?: IMemoryStore;
|
||||
|
||||
constructor(private readonly opts: MemoryCleanerOptions) {
|
||||
this.timer = new ManagedTimer("memory-tdai-cleaner", () => this.destroyed);
|
||||
this.vectorStore = opts.vectorStore;
|
||||
}
|
||||
|
||||
setVectorStore(vectorStore: VectorStore | undefined): void {
|
||||
setVectorStore(vectorStore: IMemoryStore | undefined): void {
|
||||
this.vectorStore = vectorStore;
|
||||
}
|
||||
|
||||
@@ -51,10 +51,10 @@ export class LocalMemoryCleaner {
|
||||
const tz = Intl.DateTimeFormat().resolvedOptions().timeZone || "unknown";
|
||||
const utcOffset = formatUtcOffset(-now.getTimezoneOffset());
|
||||
|
||||
this.opts.logger?.info(
|
||||
this.opts.logger?.debug?.(
|
||||
`${TAG} Enabled: retentionDays=${this.opts.retentionDays}, cleanTime=${this.opts.cleanTime}, dirs=[${L0_DIR_NAME}, ${L1_DIR_NAME}]`,
|
||||
);
|
||||
this.opts.logger?.info(
|
||||
this.opts.logger?.debug?.(
|
||||
`${TAG} Runtime clock: nowLocal=${formatLocalDateTime(now)}, nowIso=${now.toISOString()}, tz=${tz}, utcOffset=${utcOffset}`,
|
||||
);
|
||||
|
||||
@@ -110,7 +110,7 @@ export class LocalMemoryCleaner {
|
||||
let failedL1DbCleanup = 0;
|
||||
|
||||
try {
|
||||
removedL0 = vectorStore.deleteL0ExpiredByRecordedAt(cutoffIso);
|
||||
removedL0 = await vectorStore.deleteL0Expired(cutoffIso);
|
||||
} catch (err) {
|
||||
failedL0DbCleanup = 1;
|
||||
this.opts.logger?.warn(
|
||||
@@ -119,7 +119,7 @@ export class LocalMemoryCleaner {
|
||||
}
|
||||
|
||||
try {
|
||||
removedL1 = vectorStore.deleteL1ExpiredByUpdatedTime(cutoffIso);
|
||||
removedL1 = await vectorStore.deleteL1Expired(cutoffIso);
|
||||
} catch (err) {
|
||||
failedL1DbCleanup = 1;
|
||||
this.opts.logger?.warn(
|
||||
@@ -150,7 +150,7 @@ export class LocalMemoryCleaner {
|
||||
const passedToday = targetToday <= nowMs;
|
||||
const delayMs = Math.max(0, next - nowMs);
|
||||
|
||||
this.opts.logger?.info(
|
||||
this.opts.logger?.debug?.(
|
||||
`${TAG} Schedule next run: nowLocal=${formatLocalDateTime(now)}, cleanTime=${this.opts.cleanTime}, targetTodayLocal=${formatLocalDateTime(new Date(targetToday))}, passedToday=${passedToday}, nextRunLocal=${formatLocalDateTime(new Date(next))}, nextRunIso=${new Date(next).toISOString()}, delayMs=${delayMs}`,
|
||||
);
|
||||
|
||||
|
||||
@@ -0,0 +1,720 @@
|
||||
/**
|
||||
* Pipeline factory: shared infrastructure for creating and wiring
|
||||
* MemoryPipelineManager instances with VectorStore, EmbeddingService,
|
||||
* L1 runner, L2 runner, L3 runner, and persister.
|
||||
*
|
||||
* Used by both:
|
||||
* - `index.ts` (live plugin runtime)
|
||||
* - `seed-runtime.ts` (standalone seed CLI command)
|
||||
*
|
||||
* This avoids duplicating VectorStore init, L1/L2/L3 extraction logic,
|
||||
* persister wiring, and destroy sequences across multiple callers.
|
||||
*/
|
||||
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import type { MemoryTdaiConfig } from "../config.js";
|
||||
import { MemoryPipelineManager } from "./pipeline-manager.js";
|
||||
import type { L2Runner, L3Runner } from "./pipeline-manager.js";
|
||||
import { SessionFilter } from "./session-filter.js";
|
||||
import { extractL1Memories } from "../record/l1-extractor.js";
|
||||
import { readConversationMessagesGroupedBySessionId } from "../conversation/l0-recorder.js";
|
||||
import type { ConversationMessage } from "../conversation/l0-recorder.js";
|
||||
import { CheckpointManager } from "./checkpoint.js";
|
||||
import type { PipelineSessionState } from "./checkpoint.js";
|
||||
import { createStoreBundle } from "../store/factory.js";
|
||||
import type { IMemoryStore } from "../store/types.js";
|
||||
import type { EmbeddingService } from "../store/embedding.js";
|
||||
import {
|
||||
readManifest,
|
||||
writeManifest,
|
||||
buildStoreInfo,
|
||||
diffStoreBinding,
|
||||
type Manifest,
|
||||
} from "./manifest.js";
|
||||
import { SceneExtractor } from "../scene/scene-extractor.js";
|
||||
import { PersonaTrigger } from "../persona/persona-trigger.js";
|
||||
import { PersonaGenerator } from "../persona/persona-generator.js";
|
||||
import { pullProfilesToLocal, syncLocalProfilesToStore } from "../profile/profile-sync.js";
|
||||
|
||||
const TAG = "[memory-tdai] [pipeline-factory]";
|
||||
|
||||
function supportsProfileSyncWrite(store?: IMemoryStore): boolean {
|
||||
return !!(store?.syncProfiles || store?.deleteProfiles);
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Logger interface
|
||||
// ============================
|
||||
|
||||
export interface PipelineLogger {
|
||||
debug?: (message: string) => void;
|
||||
info: (message: string) => void;
|
||||
warn: (message: string) => void;
|
||||
error: (message: string) => void;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Factory options
|
||||
// ============================
|
||||
|
||||
export interface PipelineFactoryOptions {
|
||||
/** Plugin data directory (L0, records, scene_blocks, vectors.db, etc.). */
|
||||
pluginDataDir: string;
|
||||
/** Parsed memory-tdai config. */
|
||||
cfg: MemoryTdaiConfig;
|
||||
/** OpenClaw config object (needed for LLM calls in L1). */
|
||||
openclawConfig: unknown;
|
||||
/** Logger instance. */
|
||||
logger: PipelineLogger;
|
||||
/** Session filter (optional, defaults to empty). */
|
||||
sessionFilter?: SessionFilter;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Factory result
|
||||
// ============================
|
||||
|
||||
export interface PipelineInstance {
|
||||
/** The pipeline scheduler. */
|
||||
scheduler: MemoryPipelineManager;
|
||||
/** VectorStore (undefined if init failed or degraded). */
|
||||
vectorStore: IMemoryStore | undefined;
|
||||
/** EmbeddingService (undefined if not configured or init failed). */
|
||||
embeddingService: EmbeddingService | undefined;
|
||||
/**
|
||||
* Destroy all resources (scheduler, VectorStore, EmbeddingService).
|
||||
* Call this on shutdown / cleanup.
|
||||
*/
|
||||
destroy: () => Promise<void>;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Data directory init
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Ensure all required data subdirectories exist under `pluginDataDir`.
|
||||
* Safe to call multiple times (mkdirSync with `recursive: true`).
|
||||
*/
|
||||
export function initDataDirectories(dataDir: string): void {
|
||||
const dirs = ["conversations", "records", "scene_blocks", ".metadata", ".backup"];
|
||||
for (const sub of dirs) {
|
||||
fs.mkdirSync(path.join(dataDir, sub), { recursive: true });
|
||||
}
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Store init (once-async singleton)
|
||||
// ============================
|
||||
|
||||
export interface StoreInitResult {
|
||||
vectorStore: IMemoryStore | undefined;
|
||||
embeddingService: EmbeddingService | undefined;
|
||||
/** Whether a background re-index is needed (embedding config changed). */
|
||||
needsReindex: boolean;
|
||||
reindexReason?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Cached store init promises — keyed by `pluginDataDir` so that different
|
||||
* data directories (e.g. live runtime vs. seed output) each get their own
|
||||
* store instance, while concurrent callers for the *same* directory share
|
||||
* one initialization.
|
||||
*/
|
||||
const _storeInitCache = new Map<string, Promise<StoreInitResult>>();
|
||||
|
||||
/**
|
||||
* Initialize store backend and (optionally) EmbeddingService.
|
||||
*
|
||||
* **Once-async semantics per dataDir**: the first call for a given
|
||||
* `pluginDataDir` creates the store and caches the result; subsequent
|
||||
* calls with the same dir return the cached Promise immediately.
|
||||
* Call `resetStores()` during shutdown to clear the cache.
|
||||
*
|
||||
* Supports both SQLite (sync init) and TCVDB (async init) backends.
|
||||
*/
|
||||
export function initStores(
|
||||
cfg: MemoryTdaiConfig,
|
||||
pluginDataDir: string,
|
||||
logger: PipelineLogger,
|
||||
): Promise<StoreInitResult> {
|
||||
const key = pluginDataDir;
|
||||
if (!_storeInitCache.has(key)) {
|
||||
_storeInitCache.set(key, _doInitStores(cfg, pluginDataDir, logger));
|
||||
}
|
||||
return _storeInitCache.get(key)!;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset the cached store singleton(s).
|
||||
*
|
||||
* Call this during `gateway_stop` (after closing the actual store/embedding
|
||||
* resources) so that a subsequent `register()` on hot-restart can
|
||||
* re-initialize fresh instances.
|
||||
*
|
||||
* @param pluginDataDir If provided, only clear the cache for that dir.
|
||||
* If omitted, clear all cached stores.
|
||||
*/
|
||||
export function resetStores(pluginDataDir?: string): void {
|
||||
if (pluginDataDir) {
|
||||
_storeInitCache.delete(pluginDataDir);
|
||||
} else {
|
||||
_storeInitCache.clear();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Internal: actual store initialization logic (called once by the cache).
|
||||
*/
|
||||
async function _doInitStores(
|
||||
cfg: MemoryTdaiConfig,
|
||||
pluginDataDir: string,
|
||||
logger: PipelineLogger,
|
||||
): Promise<StoreInitResult> {
|
||||
let vectorStore: IMemoryStore | undefined;
|
||||
let embeddingService: EmbeddingService | undefined;
|
||||
let needsReindex = false;
|
||||
let reindexReason: string | undefined;
|
||||
|
||||
try {
|
||||
const bundle = createStoreBundle(cfg, {
|
||||
dataDir: pluginDataDir,
|
||||
logger,
|
||||
});
|
||||
vectorStore = bundle.store;
|
||||
embeddingService = bundle.embedding ?? undefined;
|
||||
|
||||
const providerInfo = embeddingService?.getProviderInfo();
|
||||
const initResult = await vectorStore.init(providerInfo);
|
||||
|
||||
if (vectorStore.isDegraded()) {
|
||||
logger.warn(`${TAG} Store is in degraded mode, falling back to keyword dedup`);
|
||||
vectorStore = undefined;
|
||||
embeddingService = undefined;
|
||||
} else {
|
||||
logger.debug?.(
|
||||
`${TAG} Store initialized: backend=${cfg.storeBackend}, provider=${cfg.embedding.provider}`,
|
||||
);
|
||||
needsReindex = initResult.needsReindex;
|
||||
reindexReason = initResult.reason;
|
||||
|
||||
// ── Manifest: first-write + config-drift detection ──
|
||||
try {
|
||||
const currentStoreInfo = buildStoreInfo(bundle.storeSnapshot);
|
||||
const existing = readManifest(pluginDataDir);
|
||||
|
||||
if (!existing) {
|
||||
// First init — write manifest
|
||||
const manifest: Manifest = {
|
||||
version: 1,
|
||||
createdAt: new Date().toISOString(),
|
||||
store: currentStoreInfo,
|
||||
seed: null,
|
||||
};
|
||||
writeManifest(pluginDataDir, manifest);
|
||||
logger.debug?.(`${TAG} Manifest created: ${JSON.stringify(currentStoreInfo)}`);
|
||||
} else {
|
||||
// Compare persisted store binding against current config
|
||||
const diffs = diffStoreBinding(existing.store, currentStoreInfo);
|
||||
if (diffs.length > 0) {
|
||||
logger.warn(
|
||||
`${TAG} ⚠️ Store config has changed since this data directory was created! ` +
|
||||
`Diffs: ${diffs.join("; ")}. ` +
|
||||
`Local JSONL data may not match the current store. ` +
|
||||
`Consider re-seeding or migrating data.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
logger.warn(`${TAG} Failed to read/write manifest (non-fatal): ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
logger.warn(
|
||||
`${TAG} Store init failed; vector/FTS recall and dedup conflict detection will be unavailable: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
vectorStore = undefined;
|
||||
embeddingService = undefined;
|
||||
}
|
||||
|
||||
return { vectorStore, embeddingService, needsReindex, reindexReason };
|
||||
}
|
||||
|
||||
// ============================
|
||||
// L1 Runner factory
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Create the standard L1 runner function.
|
||||
*
|
||||
* Reads L0 messages (from VectorStore DB or JSONL fallback), groups by sessionId,
|
||||
* runs extractL1Memories for each group, and updates the checkpoint cursor.
|
||||
*/
|
||||
export function createL1Runner(opts: {
|
||||
pluginDataDir: string;
|
||||
cfg: MemoryTdaiConfig;
|
||||
openclawConfig: unknown;
|
||||
vectorStore: IMemoryStore | undefined;
|
||||
embeddingService: EmbeddingService | undefined;
|
||||
logger: PipelineLogger;
|
||||
/**
|
||||
* Getter for the plugin instance ID used for metric reporting.
|
||||
* Called at runner execution time (not at creation time) so that the ID is
|
||||
* available even when the runner is wired before instanceId is resolved.
|
||||
* Metrics are skipped when the getter returns undefined.
|
||||
*/
|
||||
getInstanceId?: () => string | undefined;
|
||||
}): (params: { sessionKey: string }) => Promise<{ processedCount: number }> {
|
||||
const { pluginDataDir, cfg, openclawConfig, vectorStore, embeddingService, logger, getInstanceId } = opts;
|
||||
const config = openclawConfig as Record<string, unknown> | undefined;
|
||||
|
||||
return async ({ sessionKey }) => {
|
||||
if (!config) {
|
||||
logger.debug?.(`${TAG} [l1] No OpenClaw config, skipping L1 extraction`);
|
||||
return { processedCount: 0 };
|
||||
}
|
||||
|
||||
const checkpoint = new CheckpointManager(pluginDataDir, logger);
|
||||
const cp = await checkpoint.read();
|
||||
const runnerState = checkpoint.getRunnerState(cp, sessionKey);
|
||||
|
||||
logger.info(
|
||||
`${TAG} [l1] Session ${sessionKey}: l1_cursor=${runnerState.last_l1_cursor || "(start)"}`,
|
||||
);
|
||||
|
||||
try {
|
||||
let groups: Array<{ sessionId: string; messages: ConversationMessage[] }>;
|
||||
let maxRecordedAtMs = 0;
|
||||
|
||||
if (vectorStore && !vectorStore.isDegraded()) {
|
||||
const l1Cursor = runnerState.last_l1_cursor > 0
|
||||
? runnerState.last_l1_cursor
|
||||
: undefined;
|
||||
const dbGroups = await vectorStore.queryL0GroupedBySessionId(sessionKey, l1Cursor);
|
||||
groups = dbGroups.map((g) => ({
|
||||
sessionId: g.sessionId,
|
||||
messages: g.messages.map((m) => ({
|
||||
id: m.id,
|
||||
role: m.role as "user" | "assistant",
|
||||
content: m.content,
|
||||
timestamp: m.timestamp,
|
||||
})),
|
||||
}));
|
||||
// Compute max recordedAtMs across all groups for cursor advancement
|
||||
for (const g of dbGroups) {
|
||||
for (const m of g.messages) {
|
||||
if (m.recordedAtMs > maxRecordedAtMs) maxRecordedAtMs = m.recordedAtMs;
|
||||
}
|
||||
}
|
||||
logger.debug?.(`${TAG} [l1] L0 data source: VectorStore DB`);
|
||||
} else {
|
||||
logger.debug?.(`${TAG} [l1] L0 data source: JSONL files (VectorStore unavailable)`);
|
||||
const jsonlGroups = await readConversationMessagesGroupedBySessionId(
|
||||
sessionKey,
|
||||
pluginDataDir,
|
||||
runnerState.last_l1_cursor || undefined,
|
||||
logger,
|
||||
50,
|
||||
);
|
||||
groups = jsonlGroups.map((g) => ({
|
||||
sessionId: g.sessionId,
|
||||
messages: g.messages,
|
||||
}));
|
||||
// Compute max recordedAtMs from JSONL groups
|
||||
for (const g of jsonlGroups) {
|
||||
for (const m of g.messages) {
|
||||
if (m.recordedAtMs > maxRecordedAtMs) maxRecordedAtMs = m.recordedAtMs;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (groups.length === 0) {
|
||||
logger.debug?.(`${TAG} [l1] No new L0 messages for session ${sessionKey}`);
|
||||
return { processedCount: 0 };
|
||||
}
|
||||
|
||||
const totalMessages = groups.reduce((sum, g) => sum + g.messages.length, 0);
|
||||
logger.info(
|
||||
`${TAG} [l1] Processing ${totalMessages} L0 messages across ${groups.length} sessionId group(s) for session ${sessionKey}`,
|
||||
);
|
||||
|
||||
let totalExtracted = 0;
|
||||
let totalStored = 0;
|
||||
let lastSceneName: string | undefined;
|
||||
|
||||
for (const group of groups) {
|
||||
logger.debug?.(
|
||||
`${TAG} [l1] Group sessionId=${group.sessionId || "(empty)"}: ${group.messages.length} messages`,
|
||||
);
|
||||
|
||||
const l1Result = await extractL1Memories({
|
||||
messages: group.messages,
|
||||
sessionKey,
|
||||
sessionId: group.sessionId,
|
||||
baseDir: pluginDataDir,
|
||||
config,
|
||||
options: {
|
||||
enableDedup: cfg.extraction.enableDedup,
|
||||
maxMemoriesPerSession: cfg.extraction.maxMemoriesPerSession,
|
||||
model: cfg.extraction.model,
|
||||
previousSceneName: lastSceneName ?? (runnerState.last_scene_name || undefined),
|
||||
vectorStore,
|
||||
embeddingService,
|
||||
conflictRecallTopK: cfg.embedding.conflictRecallTopK,
|
||||
},
|
||||
logger,
|
||||
instanceId: getInstanceId?.(),
|
||||
});
|
||||
|
||||
totalExtracted += l1Result.extractedCount;
|
||||
totalStored += l1Result.storedCount;
|
||||
if (l1Result.lastSceneName) {
|
||||
lastSceneName = l1Result.lastSceneName;
|
||||
}
|
||||
}
|
||||
|
||||
// Use maxRecordedAtMs (write time) as cursor — always positive, TCVDB-safe
|
||||
await checkpoint.markL1ExtractionComplete(sessionKey, totalStored, maxRecordedAtMs || undefined, lastSceneName);
|
||||
logger.info(
|
||||
`${TAG} [l1] L1 complete: extracted=${totalExtracted}, stored=${totalStored} (${groups.length} group(s))`,
|
||||
);
|
||||
|
||||
return { processedCount: totalMessages };
|
||||
} catch (err) {
|
||||
logger.error(`${TAG} [l1] L1 failed: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
|
||||
throw err;
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Persister factory
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Create the standard pipeline state persister.
|
||||
* Saves pipeline session states to the checkpoint file.
|
||||
*/
|
||||
export function createPersister(
|
||||
pluginDataDir: string,
|
||||
logger: PipelineLogger,
|
||||
): (states: Record<string, PipelineSessionState>) => Promise<void> {
|
||||
return async (states) => {
|
||||
const checkpoint = new CheckpointManager(pluginDataDir, logger);
|
||||
await checkpoint.mergePipelineStates(states);
|
||||
};
|
||||
}
|
||||
|
||||
// ============================
|
||||
// L2 Runner factory
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Create the standard L2 runner function (scene extraction).
|
||||
*
|
||||
* Reads L1 memory records (incremental via VectorStore or JSONL fallback),
|
||||
* runs SceneExtractor, and returns the latest cursor for pipeline-manager
|
||||
* to track incremental progress.
|
||||
*
|
||||
* Used by both `index.ts` (live runtime) and `seed-runtime.ts` (seed CLI).
|
||||
*/
|
||||
export function createL2Runner(opts: {
|
||||
pluginDataDir: string;
|
||||
cfg: MemoryTdaiConfig;
|
||||
openclawConfig: unknown;
|
||||
vectorStore: IMemoryStore | undefined;
|
||||
logger: PipelineLogger;
|
||||
instanceId?: string;
|
||||
}): L2Runner {
|
||||
const { pluginDataDir, cfg, openclawConfig, vectorStore, logger, instanceId } = opts;
|
||||
let profileBaseline = new Map<string, { version: number; contentMd5: string; createdAtMs: number }>();
|
||||
|
||||
return async (sessionKey: string, cursor?: string) => {
|
||||
logger.debug?.(
|
||||
`${TAG} [L2] session=${sessionKey}, updatedAfter=${cursor ?? "(full)"}`,
|
||||
);
|
||||
|
||||
let records: Array<{ content: string; created_at: string; id: string; updatedAt: string }>;
|
||||
|
||||
if (vectorStore?.pullProfiles && !vectorStore.isDegraded()) {
|
||||
profileBaseline = await pullProfilesToLocal(pluginDataDir, vectorStore, logger);
|
||||
}
|
||||
|
||||
if (vectorStore && !vectorStore.isDegraded()) {
|
||||
const { queryMemoryRecords } = await import("../record/l1-reader.js");
|
||||
const memRecords = await queryMemoryRecords(vectorStore, {
|
||||
sessionKey,
|
||||
updatedAfter: cursor,
|
||||
}, logger);
|
||||
|
||||
if (memRecords.length === 0) {
|
||||
logger.debug?.(
|
||||
`${TAG} [L2] No new L1 records since cursor (session=${sessionKey}, updatedAfter=${cursor ?? "(full)"}), skipping scene extraction`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
logger.debug?.(
|
||||
`${TAG} [L2] Incremental query returned ${memRecords.length} record(s) (session=${sessionKey})`,
|
||||
);
|
||||
|
||||
records = memRecords.map((r) => ({
|
||||
content: r.content,
|
||||
created_at: r.createdAt,
|
||||
id: r.id,
|
||||
updatedAt: r.updatedAt,
|
||||
}));
|
||||
} else {
|
||||
logger.debug?.(`${TAG} [L2] VectorStore unavailable, falling back to JSONL read (session=${sessionKey})`);
|
||||
const { readMemoryRecords } = await import("../record/l1-reader.js");
|
||||
let sessionRecords = await readMemoryRecords(sessionKey, pluginDataDir, logger);
|
||||
|
||||
if (cursor) {
|
||||
const beforeCount = sessionRecords.length;
|
||||
sessionRecords = sessionRecords.filter((r) => {
|
||||
const t = r.updatedAt || r.createdAt || "";
|
||||
return t > cursor;
|
||||
});
|
||||
logger.debug?.(
|
||||
`${TAG} [L2] JSONL time filter: ${beforeCount} → ${sessionRecords.length} record(s) (updatedAfter=${cursor})`,
|
||||
);
|
||||
}
|
||||
|
||||
if (sessionRecords.length === 0) {
|
||||
logger.debug?.(`${TAG} [L2] No new L1 records found (JSONL fallback, session=${sessionKey}), skipping scene extraction`);
|
||||
return;
|
||||
}
|
||||
|
||||
records = sessionRecords.map((r) => ({
|
||||
content: r.content,
|
||||
created_at: r.createdAt,
|
||||
id: r.id,
|
||||
updatedAt: r.updatedAt,
|
||||
}));
|
||||
}
|
||||
|
||||
const extractor = new SceneExtractor({
|
||||
dataDir: pluginDataDir,
|
||||
config: openclawConfig!,
|
||||
model: cfg.persona.model,
|
||||
maxScenes: cfg.persona.maxScenes,
|
||||
sceneBackupCount: cfg.persona.sceneBackupCount,
|
||||
logger,
|
||||
instanceId,
|
||||
});
|
||||
|
||||
const memories = records.map((r) => ({
|
||||
content: r.content,
|
||||
created_at: r.created_at,
|
||||
id: r.id,
|
||||
}));
|
||||
|
||||
const preCheckpoint = new CheckpointManager(pluginDataDir, logger);
|
||||
const preState = await preCheckpoint.read();
|
||||
const preScenesProcessed = preState.scenes_processed;
|
||||
const preMemoriesSince = preState.memories_since_last_persona;
|
||||
const preTotalProcessed = preState.total_processed;
|
||||
|
||||
const extractResult = await extractor.extract(memories);
|
||||
if (extractResult.success && extractResult.memoriesProcessed > 0) {
|
||||
const checkpoint = new CheckpointManager(pluginDataDir, logger);
|
||||
const postState = await checkpoint.read();
|
||||
if (
|
||||
postState.scenes_processed < preScenesProcessed ||
|
||||
postState.total_processed < preTotalProcessed
|
||||
) {
|
||||
logger.warn(
|
||||
`${TAG} [L2] ⚠️ Checkpoint corruption detected! ` +
|
||||
`scenes_processed: ${preScenesProcessed} → ${postState.scenes_processed}, ` +
|
||||
`total_processed: ${preTotalProcessed} → ${postState.total_processed}, ` +
|
||||
`memories_since: ${preMemoriesSince} → ${postState.memories_since_last_persona}. ` +
|
||||
`Repairing...`,
|
||||
);
|
||||
await checkpoint.write({
|
||||
...postState,
|
||||
scenes_processed: Math.max(postState.scenes_processed, preScenesProcessed),
|
||||
total_processed: Math.max(postState.total_processed, preTotalProcessed),
|
||||
memories_since_last_persona: Math.max(postState.memories_since_last_persona, preMemoriesSince),
|
||||
});
|
||||
logger.info(`${TAG} [L2] Checkpoint repaired`);
|
||||
}
|
||||
|
||||
if (vectorStore && supportsProfileSyncWrite(vectorStore)) {
|
||||
await syncLocalProfilesToStore(pluginDataDir, vectorStore, profileBaseline, logger);
|
||||
}
|
||||
await checkpoint.incrementScenesProcessed();
|
||||
|
||||
const latestCursor = records.reduce((latest, r) => {
|
||||
return r.updatedAt > latest ? r.updatedAt : latest;
|
||||
}, "");
|
||||
|
||||
logger.debug?.(
|
||||
`${TAG} [L2] Extraction complete: processed=${extractResult.memoriesProcessed}, latestCursor=${latestCursor}`,
|
||||
);
|
||||
|
||||
return { latestCursor: latestCursor || undefined };
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// ============================
|
||||
// L3 Runner factory
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Create the standard L3 runner function (persona generation).
|
||||
*
|
||||
* Uses PersonaTrigger to check if generation is needed, then runs
|
||||
* PersonaGenerator. Used by both `index.ts` and `seed-runtime.ts`.
|
||||
*/
|
||||
export function createL3Runner(opts: {
|
||||
pluginDataDir: string;
|
||||
cfg: MemoryTdaiConfig;
|
||||
openclawConfig: unknown;
|
||||
vectorStore?: IMemoryStore;
|
||||
logger: PipelineLogger;
|
||||
instanceId?: string;
|
||||
}): L3Runner {
|
||||
const { pluginDataDir, cfg, openclawConfig, vectorStore, logger, instanceId } = opts;
|
||||
|
||||
return async () => {
|
||||
const trigger = new PersonaTrigger({
|
||||
dataDir: pluginDataDir,
|
||||
interval: cfg.persona.triggerEveryN,
|
||||
logger,
|
||||
});
|
||||
|
||||
const { should, reason } = await trigger.shouldGenerate();
|
||||
if (!should) {
|
||||
logger.debug?.(`${TAG} [L3] Persona generation not needed`);
|
||||
return;
|
||||
}
|
||||
|
||||
if (!openclawConfig) {
|
||||
logger.warn(`${TAG} [L3] No OpenClaw config, skipping persona generation`);
|
||||
return;
|
||||
}
|
||||
|
||||
// Pull remote profiles to establish fresh baseline before generation.
|
||||
// This ensures syncLocalProfilesToStore() has correct baselineVersion
|
||||
// for the optimistic-lock check instead of defaulting to 0.
|
||||
let profileBaseline = new Map<string, { version: number; contentMd5: string; createdAtMs: number }>();
|
||||
if (vectorStore?.pullProfiles && !vectorStore.isDegraded()) {
|
||||
profileBaseline = await pullProfilesToLocal(pluginDataDir, vectorStore, logger);
|
||||
}
|
||||
|
||||
logger.info(`${TAG} [L3] Starting persona generation: ${reason}`);
|
||||
const generator = new PersonaGenerator({
|
||||
dataDir: pluginDataDir,
|
||||
config: openclawConfig,
|
||||
model: cfg.persona.model,
|
||||
backupCount: cfg.persona.backupCount,
|
||||
logger,
|
||||
instanceId,
|
||||
});
|
||||
const genResult = await generator.generateLocalPersona(reason);
|
||||
if (!genResult) {
|
||||
logger.info(`${TAG} [L3] Persona generation skipped (no changes)`);
|
||||
return;
|
||||
}
|
||||
|
||||
if (vectorStore && supportsProfileSyncWrite(vectorStore)) {
|
||||
await syncLocalProfilesToStore(pluginDataDir, vectorStore, profileBaseline, logger);
|
||||
}
|
||||
|
||||
const checkpoint = new CheckpointManager(pluginDataDir, logger);
|
||||
const cp = await checkpoint.read();
|
||||
await checkpoint.markPersonaGenerated(cp.total_processed);
|
||||
logger.info(`${TAG} [L3] Persona generation succeeded`);
|
||||
};
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Pipeline Manager factory
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Create a MemoryPipelineManager with the standard config mapping.
|
||||
*/
|
||||
export function createPipelineManager(
|
||||
cfg: MemoryTdaiConfig,
|
||||
logger: PipelineLogger,
|
||||
sessionFilter?: SessionFilter,
|
||||
): MemoryPipelineManager {
|
||||
return new MemoryPipelineManager(
|
||||
{
|
||||
everyNConversations: cfg.pipeline.everyNConversations,
|
||||
enableWarmup: cfg.pipeline.enableWarmup,
|
||||
l1: { idleTimeoutSeconds: cfg.pipeline.l1IdleTimeoutSeconds },
|
||||
l2: {
|
||||
delayAfterL1Seconds: cfg.pipeline.l2DelayAfterL1Seconds,
|
||||
minIntervalSeconds: cfg.pipeline.l2MinIntervalSeconds,
|
||||
maxIntervalSeconds: cfg.pipeline.l2MaxIntervalSeconds,
|
||||
sessionActiveWindowHours: cfg.pipeline.sessionActiveWindowHours,
|
||||
},
|
||||
},
|
||||
logger,
|
||||
sessionFilter ?? new SessionFilter([]),
|
||||
);
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Full pipeline factory
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Create a fully wired pipeline instance: VectorStore + EmbeddingService +
|
||||
* MemoryPipelineManager with L1 runner and persister attached.
|
||||
*
|
||||
* This is the high-level entry point used by both `index.ts` and `seed-runtime.ts`.
|
||||
* Callers should attach L2/L3 runners after creation using `createL2Runner()`
|
||||
* and `createL3Runner()` from this module.
|
||||
*/
|
||||
export async function createPipeline(opts: PipelineFactoryOptions): Promise<PipelineInstance> {
|
||||
const { pluginDataDir, cfg, openclawConfig, logger, sessionFilter } = opts;
|
||||
|
||||
// Ensure data directories exist
|
||||
initDataDirectories(pluginDataDir);
|
||||
|
||||
// Initialize stores (once-async: reuses cached result if already initialized)
|
||||
const stores = await initStores(cfg, pluginDataDir, logger);
|
||||
const { vectorStore, embeddingService } = stores;
|
||||
|
||||
// Create pipeline manager
|
||||
const scheduler = createPipelineManager(cfg, logger, sessionFilter);
|
||||
|
||||
// Wire L1 runner
|
||||
scheduler.setL1Runner(createL1Runner({
|
||||
pluginDataDir,
|
||||
cfg,
|
||||
openclawConfig,
|
||||
vectorStore,
|
||||
embeddingService,
|
||||
logger,
|
||||
}));
|
||||
|
||||
// Wire persister
|
||||
scheduler.setPersister(createPersister(pluginDataDir, logger));
|
||||
|
||||
// Destroy function
|
||||
const destroy = async () => {
|
||||
logger.info(`${TAG} Destroying pipeline...`);
|
||||
await scheduler.destroy();
|
||||
if (vectorStore) {
|
||||
logger.info(`${TAG} Closing VectorStore`);
|
||||
vectorStore.close();
|
||||
}
|
||||
if (embeddingService?.close) {
|
||||
try {
|
||||
logger.info(`${TAG} Closing EmbeddingService`);
|
||||
await embeddingService.close();
|
||||
} catch (err) {
|
||||
logger.warn(`${TAG} Error closing EmbeddingService: ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
}
|
||||
logger.info(`${TAG} Pipeline destroyed`);
|
||||
};
|
||||
|
||||
return { scheduler, vectorStore, embeddingService, destroy };
|
||||
}
|
||||
@@ -262,7 +262,7 @@ export class MemoryPipelineManager {
|
||||
this.logger = logger;
|
||||
this.sessionFilter = sessionFilter ?? new SessionFilter();
|
||||
|
||||
this.logger?.info(
|
||||
this.logger?.debug?.(
|
||||
`${TAG} Initialized: everyNConversations=${config.everyNConversations}, ` +
|
||||
`warmup=${config.enableWarmup ? "enabled" : "disabled"}, ` +
|
||||
`l1IdleTimeout=${config.l1.idleTimeoutSeconds}s, ` +
|
||||
@@ -1076,12 +1076,22 @@ export class MemoryPipelineManager {
|
||||
return this.destroyed;
|
||||
}
|
||||
|
||||
/** Queue sizes for monitoring. */
|
||||
getQueueSizes(): { l1: number; l2: number; l3: number } {
|
||||
/** Queue sizes and running state for monitoring. */
|
||||
getQueueSizes(): {
|
||||
l1: number; l2: number; l3: number;
|
||||
l1Pending: boolean; l2Pending: boolean; l3Pending: boolean;
|
||||
l1Idle: boolean; l2Idle: boolean; l3Idle: boolean;
|
||||
} {
|
||||
return {
|
||||
l1: this.l1Queue.size,
|
||||
l2: this.l2Queue.size,
|
||||
l3: this.l3Queue.size,
|
||||
l1Pending: this.l1Queue.pending,
|
||||
l2Pending: this.l2Queue.pending,
|
||||
l3Pending: this.l3Queue.pending,
|
||||
l1Idle: this.l1Queue.idle,
|
||||
l2Idle: this.l2Queue.idle,
|
||||
l3Idle: this.l3Queue.idle,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -38,6 +38,9 @@ export function sanitizeText(text: string): string {
|
||||
// Remove framework reply directive tags: [[reply_to_current]], [[reply_to_xxx]], etc.
|
||||
cleaned = cleaned.replace(/\[\[reply_to[^\]]*\]\]\s*/g, "");
|
||||
|
||||
// Remove injected skill-selection wrappers, e.g. ¥¥[... ]¥¥
|
||||
cleaned = cleaned.replace(/¥¥\[[\s\S]*?\]¥¥/g, "");
|
||||
|
||||
// Remove line-leading timestamps, e.g. "[Tue 2026-03-24 03:48 UTC]"
|
||||
// or "[Tue 2026-03-24 20:21 GMT+8]", "[Thu 2026-03-24 01:51 GMT+5:30]"
|
||||
// Matches brackets containing word chars, digits, hyphens, colons, plus signs,
|
||||
|
||||
@@ -50,6 +50,11 @@ export class SerialQueue {
|
||||
return this.running;
|
||||
}
|
||||
|
||||
/** Whether the queue is idle (no queued tasks and nothing running). */
|
||||
get idle(): boolean {
|
||||
return this.queue.length === 0 && !this.running;
|
||||
}
|
||||
|
||||
/** Add a task to the queue. Returns the task's result promise. */
|
||||
add<T>(task: Task<T>): Promise<T> {
|
||||
return new Promise<T>((resolve, reject) => {
|
||||
|
||||
Reference in New Issue
Block a user