mortal/TencentDB-Agent-Memory
1
0
Fork 0
mirror of https://github.com/TencentCloud/TencentDB-Agent-Memory synced 2026-07-11 12:54:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: release v1.0.0-beta.1

Browse Source
This commit is contained in:
chrishuan
2026-05-29 17:33:12 +08:00
commit 36b0537676
290 changed files with 72303 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.dockerignore
+31
View File
@@ -0,0 +1,31 @@
node_modules
.git
__tests__
tests
docs
*.md
*.pdf
*.pptx
*.html
*.png
*.tgz
.DS_Store
.codebuddy
.ai_assets
benchmark-runs
deploy
sdk
docker
backup-*
pnpm-lock.yaml
install_hermes_ubuntu24.04_hongkong.sh
scripts/migrate-sqlite-to-tcvdb
scripts/sync-hermes-llm-to-tdai.sh
dump.rdb
vitest.config.ts
vitest.e2e.config.ts
kubb.config.ts
tsdown.config.ts
# 保留必要的 md 文件（被上面 *.md 排除了，但 README.md 等对运行无影响，不需要保留）
# 如果需要在容器中读取 README.md，删除上面的 *.md 行
.github/ISSUE_TEMPLATE/bug_report.yml
+84
View File
@@ -0,0 +1,84 @@
name: 🐛 Bug Report
description: Report an issue with the plugin | 报告插件使用中的问题
title: "[Bug] "
labels: ["bug"]
assignees: []
body:
- type: input
id: openclaw-version
attributes:
label: OpenClaw Version | OpenClaw 版本
placeholder: e.g. 2026.3.13 | 例如：2026.3.13
validations:
required: true
- type: input
id: plugin-version
attributes:
label: Plugin Version | 插件版本
description: TencentDB-Agent-Memory
placeholder: e.g. 0.1.0 | 例如：0.1.0
validations:
required: true
- type: input
id: os
attributes:
label: Operating System | 操作系统
placeholder: e.g. Linux / Ubuntu 22.04 / macOS 14 / Windows 11 / CentOS 7
validations:
required: true
- type: input
id: system-spec
attributes:
label: System Specification | 系统配置
description: Optional, e.g. CPU model, memory size | 可选，如 CPU、内存大小等，不清楚可略过
placeholder: Intel i7 16GB RAM / 8C16G | Intel i7 16GB内存 / 8核16G
validations:
required: false
- type: textarea
id: bug-description
attributes:
label: Describe the bug | 问题描述
description: A clear and concise description of the problem | 清晰简洁地描述你遇到的问题
placeholder: Please describe the issue in detail | 请详细说明问题现象...
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: To Reproduce | 复现步骤
description: Steps to reproduce the behavior | 复现该问题的操作步骤
placeholder: |
1. Go to / Call API | 执行操作/调用接口
2. Set parameters | 传入参数/配置
3. See error | 出现报错/异常
validations:
required: true
- type: textarea
id: expected-behavior
attributes:
label: Expected behavior | 预期行为
description: What you expected to happen | 你期望的正确结果
validations:
required: true
- type: textarea
id: logs
attributes:
label: Error Logs / Screenshots | 报错日志/截图
description: Please attach full logs or screenshots | 请附上完整报错日志或截图
validations:
required: false
- type: textarea
id: additional-context
attributes:
label: Additional context | 补充信息
description: Any other relevant information | 其他相关信息
validations:
required: false
.github/ISSUE_TEMPLATE/feature_request.yml
+42
View File
@@ -0,0 +1,42 @@
name: ✨ Feature Request
description: Propose a feature or suggestion | 提出一个功能需求或建议
title: "[Feature] "
labels: ["enhancement"]
assignees: []
body:
- type: textarea
id: related-problem
attributes:
label: Is this feature related to a problem? | 该功能需求是否与某个问题相关？请描述
description: A clear and concise description of the problem | 清晰简洁地描述问题是什么
placeholder: e.g. I'm frustrated when... | 例如：我总是在使用 XX 功能时感到不便...
validations:
required: true
- type: textarea
id: solution
attributes:
label: Describe the solution you'd like | 描述你期望的解决方案
description: A clear and concise description of what you want to happen | 清晰简洁地说明你希望实现的效果
placeholder: Describe the feature you want to add or improve | 请描述你希望新增或改进的功能...
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Describe alternatives you've considered | 描述你考虑过的其他方案
description: A clear and concise description of any alternative solutions | 清晰简洁地描述你考虑过的其他替代方案或功能
placeholder: Optional | 可选
validations:
required: false
- type: textarea
id: additional-context
attributes:
label: Additional context | 补充说明
description: Add any other context or screenshots about the feature | 在此添加与该功能需求相关的其他背景信息或截图
placeholder: Optional | 可选
validations:
required: false
.github/ISSUE_TEMPLATE/question.yml
+37
View File
@@ -0,0 +1,37 @@
name: ❓ Question / Consultation
description: Ask a question about usage or community | 使用咨询或社区相关问题
title: "[Question] "
labels: ["question"]
assignees: []
body:
- type: dropdown
id: question-type
attributes:
label: Question Category | 问题类别
description: Select the category that best describes your question | 选择最符合你问题的类别
options:
- Usage / How-to | 使用方式咨询
- Compatibility / Integration | 兼容性 / 集成确认
- Community / Group | 社区 / 群组相关
- Other | 其他
validations:
required: true
- type: textarea
id: question-description
attributes:
label: Your Question | 你的问题
description: Describe your question clearly | 请清晰描述你的问题
placeholder: e.g. Does the memory plugin work with OpenClaw? | 例如：记忆插件是否支持 OpenClaw？
validations:
required: true
- type: textarea
id: context
attributes:
label: Context / Background | 背景信息
description: Any context that helps us understand your question better | 有助于我们理解你问题的背景信息
placeholder: e.g. your use case, environment, what you've tried | 例如：你的使用场景、环境、已尝试的方案等
validations:
required: false
.github/PULL_REQUEST_TEMPLATE.md
+18
View File
@@ -0,0 +1,18 @@
## Description | 描述
<!-- Describe what this PR does | 请描述这个 PR 做了什么 -->
## Related Issue | 关联 Issue
<!-- Example: Fix #123 | 例如：Fix #123 -->
## Change Type | 修改类型
- [ ] Bug fix | Bug 修复
- [ ] New feature | 新功能
- [ ] Documentation update | 文档更新
- [ ] Code optimization | 代码优化
## Self-test Checklist | 自测清单
- [ ] Verified locally | 本地验证通过
- [ ] No existing features affected | 无影响现有功能
## Additional Notes | 其他说明
<!-- Optional | 可选 -->
.github/workflows/pr-ci.yml
+134
View File
@@ -0,0 +1,134 @@
name: CI
on:
pull_request:
branches: [main]
# Cancel previous runs for the same PR / branch
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
jobs:
# ── 1. Install dependencies ────────────────────────────────
install:
name: Install
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "22"
- name: Cache node_modules
uses: actions/cache@v4
id: cache-deps
with:
path: node_modules
key: deps-${{ runner.os }}-${{ hashFiles('package.json') }}
- name: npm install
if: steps.cache-deps.outputs.cache-hit != 'true'
run: npm install --ignore-scripts
# ── 2. Pack validation ─────────────────────────────────────
pack:
name: Pack
needs: install
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "22"
- name: Restore node_modules
uses: actions/cache@v4
with:
path: node_modules
key: deps-${{ runner.os }}-${{ hashFiles('package.json') }}
- name: npm pack (dry-run)
run: npm pack --dry-run
- name: npm pack
run: npm pack
- name: Upload .tgz artifact
uses: actions/upload-artifact@v4
with:
name: package-tarball
path: "*.tgz"
retention-days: 7
# ── 5. Plugin manifest validation ──────────────────────────
manifest:
name: Manifest
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Validate openclaw.plugin.json
run: |
echo "── Checking openclaw.plugin.json exists ──"
test -f openclaw.plugin.json || { echo "❌ openclaw.plugin.json not found"; exit 1; }
echo "── Checking valid JSON ──"
node -e "JSON.parse(require('fs').readFileSync('openclaw.plugin.json','utf8'))" || { echo "❌ Invalid JSON"; exit 1; }
echo "── Checking required fields ──"
node -e "
const m = JSON.parse(require('fs').readFileSync('openclaw.plugin.json','utf8'));
const errors = [];
if (!m.id || typeof m.id !== 'string') errors.push('missing or invalid \"id\"');
if (m.configSchema && typeof m.configSchema !== 'object') errors.push('\"configSchema\" must be an object');
if (errors.length) { console.error('❌ Manifest errors:', errors.join(', ')); process.exit(1); }
console.log('✅ id:', m.id);
if (m.name) console.log(' name:', m.name);
if (m.configSchema) console.log(' configSchema: present (' + Object.keys(m.configSchema.properties || {}).length + ' top-level props)');
"
echo "── Checking package.json openclaw metadata ──"
node -e "
const pkg = JSON.parse(require('fs').readFileSync('package.json','utf8'));
const oc = pkg.openclaw || {};
const errors = [];
if (!oc.extensions || !oc.extensions.length) errors.push('missing openclaw.extensions');
if (!oc.compat?.pluginApi) errors.push('missing openclaw.compat.pluginApi');
if (!oc.build?.openclawVersion) errors.push('missing openclaw.build.openclawVersion');
if (errors.length) { console.error('❌ Package metadata errors:', errors.join(', ')); process.exit(1); }
console.log('✅ openclaw metadata OK');
console.log(' pluginApi:', oc.compat.pluginApi);
console.log(' openclawVersion:', oc.build.openclawVersion);
"
# ── 6. Package size guard ──────────────────────────────────
size:
name: Size Guard
needs: pack
runs-on: ubuntu-latest
steps:
- name: Download tarball
uses: actions/download-artifact@v4
with:
name: package-tarball
- name: Check package size
run: |
TGZ=$(ls *.tgz | head -1)
SIZE=$(stat -c%s "$TGZ" 2>/dev/null || stat -f%z "$TGZ")
SIZE_KB=$((SIZE / 1024))
MAX_KB=2048
echo "📦 Package: $TGZ"
echo " Size: ${SIZE_KB} KB (limit: ${MAX_KB} KB)"
if [ "$SIZE_KB" -gt "$MAX_KB" ]; then
echo "❌ Package exceeds ${MAX_KB} KB size limit!"
echo " Consider checking if large files were accidentally included."
exit 1
else
echo "✅ Size OK"
fi
.gitignore
+98
View File
@@ -0,0 +1,98 @@
# Dependencies
node_modules/
# Runtime workspace
workspace/
# Environment variables
.env
# Test caches
__tests__/soak/.model-cache/
# Python cache
__pycache__/
*.pyc
*.pyo
*.pyd
# Migration build output
scripts/export-tencent-vdb/dist/
scripts/migrate-sqlite-to-tcvdb/dist/
scripts/read-local-memory/dist/
# Root-level build output (not used at runtime, not published)
dist/
node_modules/
benchmark-runs/
.codebuddy
.ai_assets
# Dev-only scripts & lockfile (not shipped in MR)
install-plugin.sh
package-lock.json
pnpm-lock.yaml
yarn.lock
test-offload.sh
test-offload-mmd.sh
test-offload-sessions.sh
# npm pack / release tarballs (never commit packaged outputs)
dist-pack/
*.tgz
*.tar.gz
*.whl
# TypeScript build intermediates
*.tsbuildinfo
*.js.map
# SDK build output (source is .ts/.py, not compiled output)
sdk/typescript/dist/
sdk/python/dist/
sdk/python/*.egg-info/
# Vitest cache
.vite/
# Environment (already covered above, explicit for clarity)
.env.*
!.env.example
# Eval scripts output
scripts/eval-report.json
# OS files
.DS_Store
# Generated binaries
vectors.db
# Office temp files
~$*
# Generated docs (HTML/PDF)
docs/*.html
docs/*.pdf
docs/*.png
docs/*.pptx
docs/*.mmd
# Log files
*.log
scene_blocks/
gateway.log
# 可观测性私人文档（不上传git）
docs/开发环境说明.md
docs/可观测性开发规范.md
docs/插件埋点列表.md
# DevCloud multipod test runtime artifact (contains real credentials, never commit)
__tests__/devcloud-multipod/.env.devcloud
# Local-only diagnostic / convention scripts (not shipped)
__tests__/devcloud-multipod/diag-instance.sh
scripts/COMMIT_CONVENTION.md
scripts/pre-commit-check.sh
.npmignore
+22
View File
@@ -0,0 +1,22 @@
# 测试文件
*.test.ts
*.test.js
*.spec.ts
*.spec.js
__tests__/
# 开发文档与辅助
docs/
benchmark-runs/
workspace/
# 环境与配置
.env
.env.*
.gitignore
.codebuddy/
.coding-ci.yaml
# 运行时产物
node_modules/
*.tgz
CHANGELOG.md
+412
View File
@@ -0,0 +1,412 @@
# Changelog
本文件记录 `@tencentdb-agent-memory/memory-tencentdb` 插件的所有显著变更，格式遵循 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)，版本号遵循 [Semantic Versioning](https://semver.org/)。
---
## [0.3.6] - 2026-05-27
### ✨ 新功能
- **Recall 上下文预算控制** ([#71](https://github.com/Tencent/TencentDB-Agent-Memory/pull/71) / [#70](https://github.com/Tencent/TencentDB-Agent-Memory/issues/70))：新增 `recall.maxCharsPerMemory` 与 `recall.maxTotalRecallChars` 两项配置，默认 `0` 不改变现有行为；设置为正整数后，会在 L1 召回完成、注入 `<relevant-memories>` 之前按分数顺序裁剪超长条目并丢弃溢出部分，避免长会话因记忆膨胀挤占上下文。已在 README、README_CN 与 `openclaw.plugin.json` 同步说明。
- **L1 / L2 / L3 提示词按用户输入语言自适应** ([#38](https://github.com/Tencent/TencentDB-Agent-Memory/issues/38))：`l1-extraction`、`l1-dedup`、`scene-extraction`、`persona-generation` 四个 prompt 中所有自由文本字段（`scene_name`、记忆 `content`、scene `.md` 标题/正文、`persona.md` 各章节）现在跟随用户消息的主导语言书写；JSON 字段名、枚举值、ISO 时间戳、`persona.md` 等结构化文件名继续保持英文作为稳定契约。无需配置 `locale`，任意语言（en / fr / ja / es / …）均可直接使用。
- **Embedding `sendDimensions` 可选关闭**：`OpenAIEmbeddingService` 默认仍会在请求体携带 `dimensions` 字段（兼容 OpenAI `text-embedding-3-*` Matryoshka 截断）；新增 `embedding.sendDimensions` 配置项，设置为 `false` 时省略该字段，可对接 BGE-M3 等不支持自定义维度的固定维度模型（原会被服务端 HTTP 400 拒绝 `does not support matryoshka representation`）。
- **Gateway 可选 Bearer 鉴权 + CORS 白名单**：新增 `server.apiKey` / `TDAI_GATEWAY_API_KEY` 配置项，设置后所有非 `/health` 路由需携带 `Authorization: Bearer <key>`（`crypto.timingSafeEqual` 防时序攻击）；新增 `server.corsOrigins` / `TDAI_CORS_ORIGINS` 配置项，显式指定允许的 CORS 来源列表（空列表 = 不发送 CORS 头，`"*"` = 保留旧版宽松行为）。启动时打印安全态势摘要，非回环地址 + 无 apiKey 时输出 WARN。两项均默认关闭，现有部署无需改动。Hermes Python 客户端同步支持 `MEMORY_TENCENTDB_GATEWAY_API_KEY` 环境变量自动附加 Bearer 头。
- **Offload `collect` 模式**：新增 `offload.mode: "collect"` 配置，仅执行数据采集（L0 捕获 + 向量写入）而不触发 L3 压缩，适用于纯数据积累阶段或调试场景。
### 🐛 修复
#### 数据安全 / 数据隔离
- **L2 LLM 提取失败导致 `scene_blocks/` 被清空 / 半写入** ([#88](https://github.com/Tencent/TencentDB-Agent-Memory/issues/88))：Phase 1 已对 `scene_blocks/` 做完整快照，但 LLM 抛错时 `catch` 直接 `return`，沙箱里的部分写入 / 删除不会回滚，后续 recall 因此看不到场景导航，降级为碎片召回。新增 `BackupManager.findLatestBackup` + `restoreLatestDirectory`，在 LLM 失败时自动从最新备份恢复；采用 fail-soft 设计：无备份时不动目标目录，恢复过程自身的错误也不会替换原始 LLM 错误。
- **Cleaner 安全加固**：`computeCutoffMsByLocalDay` 拒绝无效 cutoff（未来时间 / 距今不足 24h）；SQLite 与 TCVDB 在 `expired/total > 80%` 时阻止删除；`runOnce` 增加最小保留护栏（L0:50 / L1:20）并产出 `cleaner_summary` JSON 审计日志；新增 `__tests__/cleaner/verify-cleaner-safety.ts` E2E 校验。
- **场景文件名含空格导致 Persona Scene Navigation 引用失效**：LLM 在 L2 偶发用 `Daily Rhythm in Shanghai.md` 这类含空格的名字创建 scene block，导致 `persona.md` 中 `### Path: scene_blocks/<name>.md` 引用无法被下游 `\S+\.md` 风格解析器（health-checker 等）识别，soak 后健康检查必报 `Scene Navigation 存在但无场景引用`。修复：(1) `scene-extraction` prompt 增加"📛 文件命名规范（强制）"段，禁止空格 / 括号 / 引号等标点；(2) 新增 `core/scene/filename-normalizer.ts`，在 `SceneExtractor.extract` Phase 5b（cleanup 后、`syncSceneIndex` 前）自动归一化文件名（空格 → `-`、剥离危险标点、冲突时追加 `-2` 后缀），下游 PersonaGenerator / recall / profile-sync 自动使用干净名，无需改动。
#### OpenClaw 宿主兼容
- **`api.runtime.state` 在新版 OpenClaw 上为 `undefined` 导致注册时崩溃** ([#78](https://github.com/Tencent/TencentDB-Agent-Memory/issues/78) / [#85](https://github.com/Tencent/TencentDB-Agent-Memory/pull/85) / [#79](https://github.com/Tencent/TencentDB-Agent-Memory/pull/79))：为两处调用点加可选链 + fallback，优先调用宿主 `runtimeState.resolveStateDir()`，缺失时退到 `OPENCLAW_STATE_DIR` 环境变量，再退到 `~/.openclaw`。同时修复 cli-metadata 注册模式下 `runtime` 为空对象 `{}` 仍会触发 `TypeError` 的问题（在该模式下提前 return，只调用 `registerCli`）。
- **`contextEngine` slot ID 与插件名不一致**：`registerContextEngine` 的 ID 从 `openclaw-context-offload` 改为 `memory-tencentdb`，避免 `openclaw doctor --fix` 把 slot 重置；`setup-offload.sh` 中的 `CONTEXT_ENGINE_ID` 同步更新。
- **L1.5 settle 永不返回导致 L2 卡死**：在 L2 poll 中为 L1.5 settle 增加 60s 超时，当未配置 Context Engine slot 导致 `assemble` 永不被调用时，自动 force-settle 解锁 L2。
- **Standalone 文本任务仍暴露工具导致 DeepSeek 等后端 L1 抽取不稳定** ([#58](https://github.com/Tencent/TencentDB-Agent-Memory/issues/58) / [#59](https://github.com/Tencent/TencentDB-Agent-Memory/pull/59))：`enableTools=false` 时彻底不传工具列表（此前即便只读子集也会鼓励 OpenAI 兼容后端尝试 tool calling，DeepSeek 上尤其明显）。
- **L2 cold-start skip 被错误地更新 `l2LastRunTime`**：导致首次 skip 后必须等满 `l2MaxInterval` 才会真正运行 L2；现在仅在确实跑过的情况下更新时间戳。
#### Offload（Context Engine）稳定性
- **`sanitizeText` 误删 emoji / CJK Extension B / Math Bold 等非 BMP 字符** ([#30](https://github.com/Tencent/TencentDB-Agent-Memory/issues/30) / [#31](https://github.com/Tencent/TencentDB-Agent-Memory/pull/31))：`UNSAFE_CHAR_RE` 含 `[\uD800-\uDFFF]` 但缺 `u` flag，JS 按 UTF-16 code unit 处理时会把每个非 BMP 码点的两个 surrogate 各自 strip 掉。加上 `u` flag 后只匹配孤立（畸形）surrogate，emoji / 扩展 CJK / 数学加粗等正常恢复；新增 vitest 套件覆盖保留与剥离两类 case。
- **Emergency 截断在 `MIN_KEEP` 拒绝下死锁**：`EMERGENCY_MIN_MESSAGES_TO_KEEP` 由 4 降到 2；新增 `_emergencyTruncateOversized`，当 head/tail 删除均被阻塞时就地截断超大消息（保留 `tool_use` 块结构），最后兜底强制删除并配对清理 `toolResult`，在 LLM 可见的内容里加截断告示。
- **多轮 aggressive compression 累计耗时**：由 6 轮 `O(N × rounds)` 全量 tiktoken 改为单趟 `O(N × 1)` 直接计算到目标阈值的精确切点。615 条消息从 84s 降到 ~14s；tool 配对、user 消息保护、MMD 保留、stall 检测等安全机制全部保留。
- **FP-HEAD-DELETE 在多轮 FAST-SKIP 后误删新消息**：移除该路径，改用 `FP-BOUNDARY-DELETE`（基于上一轮 aggressive 边界的 O(1) 头部删除，index + fingerprint 双重验证、tool-pair 安全）+ `BOUNDARY-INCR-SKIP`（增量估算低于阈值时跳过 tiktoken）。重放场景下 `assemble` 38s → 122ms（310×）。
- **首次 assemble 慢**：为 `TAIL-ACCUMULATE` 增加 fast-token-estimate（基于字符，~51× 快于 tiktoken）前置短路，无边界且 fast estimate 明显高于阈值时跳过全量 tiktoken；同时为 `TAIL-ACCUMULATE` 增加向后 tool-pair 校正、user 消息保护、最少保留 10 条等安全检查。首次 assemble 29s → ~1.4s。
- **Token 计算精度**：`details` 字段加入 `INTERNAL_KEYS`（框架在送 LLM 前 strip 掉，不应计入 token）；新增 `_stripLargeFields()` 移除非内容大字段；就地截断后调用 `invalidateTokenCache(msg)` 修正 WeakMap 缓存陈旧问题；`bestTokens < 600` 时跳过截断防反向膨胀；stub 文本简化为纯英文避免触发模型内容过滤；`l3TiktokenEncoding` 默认从 `o200k_base` 改为 `cl100k_base`（匹配 DeepSeek / GLM / MiniMax 分词器）。
- **Offload 日志降级**：`AGGRESSIVE` / `EMERGENCY` 等多数日志降到 debug，仅当超过 10s 时才输出 `SLOW` warn；Opik tracer 初始化、`after_tool_call` 无 session 等"正常 fallback"场景日志同步降级，减少 `plugins list` 噪音。
#### Hermes / Docker 部署
- **`Dockerfile.hermes` 生成的 `config.yaml` 缺 `api_key`** ([#77](https://github.com/Tencent/TencentDB-Agent-Memory/issues/77) / [#81](https://github.com/Tencent/TencentDB-Agent-Memory/pull/81))：`provider: custom` 模式下 Hermes 从 `config.yaml.model.api_key` 读密钥，而原 CMD 脚本仅写入 `.env` 的 `OPENAI_API_KEY`，造成容器内首条对话即报 401。修复为同步写入 `model.api_key: "${MODEL_API_KEY}"`。
- **安装脚本多处问题** ([#18](https://github.com/Tencent/TencentDB-Agent-Memory/issues/18) / [#19](https://github.com/Tencent/TencentDB-Agent-Memory/issues/19) / [#20](https://github.com/Tencent/TencentDB-Agent-Memory/issues/20) / [#54](https://github.com/Tencent/TencentDB-Agent-Memory/pull/54) / [#55](https://github.com/Tencent/TencentDB-Agent-Memory/pull/55))：
- 支持 `HERMES_AGENT_DIR` 环境变量覆盖，适配 FHS 布局（把 hermes-agent 装在 `/usr/local/lib/hermes-agent` 等）。
- root 用户执行时不再 `su - root` 无限递归。
- `MEMORY_TENCENTDB_GATEWAY_CMD` 在 systemd 环境下用 `command -v node` 解析的绝对路径，`npx tsx` 改为 `node --import tsx/esm`，避免 nvm PATH 不可见时找不到 node。
### ✨ 改进
- **L1.5 settle 60s 超时保护**（详见上）。
- **OpenAI-style standalone runner** 在 `enableTools=false` 时不再传任何工具（详见上）。
- **`l3TiktokenEncoding` 默认改为 `cl100k_base`**，匹配主流国产/开源模型分词器。
- **Docker 文档**：补充 `cd docker/opensource` 前置步骤；新增 question/consultation issue 模板。
### 🧪 测试 / 内部
- 新增 `src/utils/backup.test.ts`：`findLatestBackup` 4 个用例 + `restoreLatestDirectory` 5 个用例，真 fs 沙箱。
- 新增 `src/core/scene/scene-extractor.restore.integration.test.ts`：用真临时目录 + 真 BackupManager 验证 LLM 失败时 `scene_blocks/` 被恢复（含 step 日志，沙箱保留供人工 inspect）。
- 新增 `src/utils/openclaw-state-dir.test.ts` + `index.test.ts` cli-metadata 模式安全性用例。
- 新增 `__tests__/cleaner/verify-cleaner-safety.ts` E2E（SQLite + live VDB）。
- 新增 `src/offload/fast-token-estimate.ts` + `benchmark-token-estimate.ts` 基线脚本。
### ⚠️ 配置项变化（向后兼容）
| Key | 默认值 | 说明 |
|---|---|---|
| `recall.maxCharsPerMemory` | `0` | `0`/未设置 = 不裁剪 |
| `recall.maxTotalRecallChars` | `0` | `0`/未设置 = 不裁剪 |
| `embedding.sendDimensions` | `true` | `false` 时不在请求体携带 `dimensions`，适配 BGE-M3 等 |
| `l3TiktokenEncoding` | `cl100k_base`（原 `o200k_base`） | 仅在显式依赖 `o200k_base` 时需手动覆盖回去 |
---
## [0.3.5] - 2026-05-15
### 🐛 修复
- **兼容 OpenClaw v2026.5.7 zod v4 子路径**：显式声明 `zod@^4.4.3` 依赖，解决 `@ai-sdk/provider-utils@4.x` 需要 `zod/v4` 子路径导出但宿主环境可能 hoist zod@3.x 引发 `Cannot find module zod/v4` 的运行时错误。
### ✨ 改进
- **L1→L2 延迟从 90s 降至 10s**：`l2DelayAfterL1Seconds` 默认值 90→10，冷启动用户不再需要等待 ~90s 才能看到 L2 场景提取结果，体感更及时。
### 📖 文档
- README 新增 Docker Quick Start 章节，说明模型 URL/Name 环境变量配置方式。
---
## [0.3.4] - 2026-05-12
### 🐛 修复
- **兼容 OpenClaw v2026.4.7 以下版本 L1 抽取空输出**：旧宿主不支持 `systemPromptOverride`，通过 `extraSystemPrompt` 回退注入系统提示，确保 LLM 按数据提取助手身份工作。
- **TCVDB hybrid 召回冗余双重 HTTP 调用**：`auto-recall` 对 TCVDB 发两次相同的 `hybridSearch` 请求（且 keyword 路径将 FTS5 OR 表达式错误传入 BM25 编码器）。新增 `nativeHybridSearch` 短路，TCVDB 单次调用即可完成 dense + sparse + RRF，recall 耗时减半（~50-120ms）。
- **L2 parser 对齐 Go 后端**：增加 mermaid fallback，修复 `first{...last}` JSON 提取逻辑。
### ✨ 改进
- **VDB HTTP 请求级计时**：`tcvdb-client` 每次请求打一条 info 计时日志（`/document/hybridSearch 85ms`），retry/失败细节保持 debug 级别。
- **启动路径误导性日志降级为 DEBUG**：store manifest 不一致、sqlite schema migration、profile-sync MD5 mismatch 等正常场景不再打 warn/info，避免 AI 误判。
- **L1 提取调试日志**：新增 `[l1-debug]` 系列（RESOLVE / INVOKE / RESULT / EMPTY_DUMP / ENTRY / NO_JSON），方便定位 LLM 调用链问题。
### 🔧 兼容性适配
- **OC 2026.4.23 Zod schema 兼容 patch 脚本**（`scripts/bugfix-20260423/`）：一键修复 `allowConversationAccess` 被 `.strict()` 拒绝的问题，含轻量版脚本、全自动脚本、手动 SOP 文档。
- Offload 日志去掉 `Backend` 前缀，默认超时为 120s。
### 📦 新功能
- **Offload Local Mode**：支持本地模式运行 offload（不依赖远端后端）。
- **Docker 一体化镜像**（`Dockerfile.hermes`）：单容器捆绑 Hermes Agent + memory_tencentdb 插件 + TDAI Memory Gateway，统一 `MODEL_*` 环境变量驱动。
### ✅ 测试
- 修复 `fault-injection` FI-05 mock config 缺 `embedding` 字段
- 修复 `cli.test` dependencies 断言适配新增依赖
- 跳过 `patch-effectiveness` 已删除的 `install-plugin.sh` 测试
---
## [0.3.3] - 2026-05-08
### 🐛 修复
- **加固 hook-policy 版本决策逻辑**：仅当宿主版本为严格 `x.y.z` 语义化版本、且 `>= 2026.4.24` 时才自动写入 `hooks.allowConversationAccess`；无法解析（如 `unknown`、beta、snapshot 等非标准版本）时一律跳过，避免对旧版本或非预期版本误写配置导致启动失败。
- hook-policy 关键路径补充 debug 日志（原始版本串、解析后版本、最小要求版本、是否 patch 的决策），方便线上排查。
### ✅ 测试
- 新增 `src/utils/ensure-hook-policy.test.ts`，覆盖标准版本、预发布、`unknown`、边界值等决策 case。
## [0.3.2] - 2026-05-08
### 🐛 修复
- 兼容 OpenClaw v2026.4.23 前的版本，防止写入的 hook 配置导致无法启动
- 修改 allowConversationAccess 到 2026.4.24+ 添加。
## [0.3.1-beta.1] - 2026-05-07
### 🐛 修复
- **兼容 OpenClaw v2026.4.23+ hook 权限策略**：该版本引入 `allowConversationAccess` 安全门控（[openclaw#70786](https://github.com/openclaw/openclaw/pull/70786)），导致非 bundled 插件的 `agent_end` hook 被静默拦截，整个 capture pipeline 失效。新增 `ensurePluginHookPolicy()` 自动检测并补全配置，优先通过 SDK 触发 gateway 自动重启，fallback 手动写入配置文件。
- **兼容 OpenClaw 2026.5.3+ 安装校验**：新增 tsdown 构建配置生成 `dist/index.mjs`，满足新版安装时对编译产物的强制校验（不再允许纯 TypeScript 入口）。
- **声明 `activation.onStartup`**：确保 gateway 在启动时加载本插件。
- **声明 `contracts.tools`**：注册 `tdai_memory_search`、`tdai_conversation_search` 工具名，满足 tool registration contract 要求。
---
## [0.3.0] - 2026-05-06
### 🚀 新功能
**运维管理工具（CTL）**
- 新增 `memory-tencentdb-ctl` 命令行管理工具，支持 standalone 与 hermes 两种运行模式
- 新增 `install-memory-tencentdb` 一键安装脚本
- CTL 新增 `config vdb-off` 命令，支持将 Gateway 存储从 VDB 回退到 SQLite
- Gateway 安装脚本支持将环境变量写入 `~/.hermes/.env`（systemd 场景）
**Offload 增强**
- Offload 启动时自动应用 `after_tool_call` patch，patch 失败时自动禁用 offload
- 新增 `setup-offload.sh` 一键启用/禁用 offload 脚本，支持 `--backend-api-key` 参数
- L0 捕获过滤：排除 offload 注入的 MMD 上下文块，避免将压缩中间产物误存为记忆
**Gateway 自愈与稳定性**
- Hermes 插件新增 watchdog + lazy probe 机制，Gateway 异常时自动恢复
- Gateway YAML 配置解析支持任意深度嵌套
### ✨ 改进
- 数据目录与安装目录统一整合至 `~/.memory-tencentdb/`
- 引入 `$HERMES_HOME` 环境变量约定，移除硬编码 `~/.hermes` 路径
- CTL hermes 配置编辑改为缩进感知，保持原始文件格式
- 运维脚本保留在 tarball 中但不再注册为 bin 命令（减少全局命令污染）
- init/destroy 生命周期日志降级为 debug 级别
- patch 脚本兼容 pnpm 安装环境，使用 Node.js 动态解析 openclaw 安装路径
### 🐛 修复
**Core 稳定性**
- 修复 `ensureSchedulerStarted` 并发调用下的竞态问题
- 修复 `/session/end` 错误销毁全局 scheduler 的问题（改为按 session_key 作用域）
- 修复关闭 store 时未等待后台 fire-and-forget 任务完成的问题
- 修复 `disable_offload` 未正确删除 `slots.contextEngine` 配置的问题
**Offload**
- 修复 slot 占用检测逻辑：仅在 `ok=false`（slot 被占用）时拒绝，API 异常不再误判为冲突
- 修复 `registerContextEngine` 抛异常时未禁用 offload 的问题
- 修复 slot 被占用时未完全禁用所有 offload 功能的问题
**L3 压缩**
- 修复 aggressive/emergency 压缩在用户消息位于队首时卡死的问题
- 修复消息被大量 offload 后压缩停滞的问题
**迁移工具**
- 修复源数据目录或 SQLite 不存在时迁移脚本崩溃的问题（改为优雅跳过）
- 修复源数据为空时 config/manifest 未写入的问题
**脚本与运维**
- 修复 `set -e` 环境下 `((VAR++))` 在 VAR=0 时导致脚本退出的问题
- 修复 patch 脚本误报 FAILED 计数的问题（跳过无 after_tool_call 上下文的候选项）
- 修复 Hermes 退出时未终止 Gateway 子进程的问题
### ♻️ 重构
- 统一 patch 检测逻辑：始终委托给 patch 脚本并通过退出码判定结果
---
## [0.3.0-beta.1] - 2026-04-23
### 🚀 新功能
**短期记忆压缩（Context Offload）**
- 新增 Offload 模块，支持长对话场景下的上下文压缩与记忆卸载
**架构重构：Core + Gateway 多框架支持**
- 重构为 `TdaiCore` 宿主无关的核心层 + 适配器模式，解耦 OpenClaw 框架依赖
- 新增 `HostAdapter` / `LLMRunner` / `LLMRunnerFactory` 抽象接口，支持不同宿主的 LLM 调用
- 新增 Hermes Gateway 适配器（`memory_tencentdb` Hermes Plugin），支持通过 Hermes 框架独立运行
- `TdaiCore` 提供统一的 `handleBeforeRecall()` / `handleTurnCommitted()` / `searchMemories()` 等 API
- Gateway 零配置自动发现：Hermes 插件自动检测配置和数据目录
- 数据目录所有权从插件移至 Gateway 层管理
**Recall 注入优化（Cache 友好）**
- L1 召回记忆从 `appendSystemContext` 移到 `prependContext`（用户消息前缀），避免每轮系统提示词变化导致 prompt cache bust
- Persona / Scene Navigation / Tools Guide 保持在 `appendSystemContext`（稳定内容，连续多轮 cache 命中）
- 注册 `before_message_write` 钩子，在 user message 持久化到 JSONL 前 strip `<relevant-memories>` 标签，防止历史消息中累积旧的召回内容
**分场景 Embedding 超时**
- 新增 `embedding.recallTimeoutMs`（recall 路径）和 `embedding.captureTimeoutMs`（capture 路径）配置
- recall 超时时 hybrid 策略自动降级为纯关键词搜索；capture 超时时 L1 dedup 降级为 FTS
- 向前兼容：不配置时 fallback 到全局 `embedding.timeoutMs`
### ✨ 改进
- CleanContextRunner 通过 `systemPromptOverride` 替换 OpenClaw 默认系统提示词，每次 L1/L2/L3 调用节省 ~4500 input tokens
- L2（场景提取）和 L3（画像生成）prompt 拆分为 `systemPrompt` + `userPrompt`，角色划分更清晰
- Pipeline 默认参数调整：`l1IdleTimeoutSeconds` 60→600s，`l2MinIntervalSeconds` 300→900s，`l2MaxIntervalSeconds` 1800→3600s
### 🐛 修复
- 修复 `pullProfilesToLocal` 并发竞争导致 `ENOTEMPTY` 错误（乐观无锁修法：rename 竞争失败时静默使用对方结果）
- 修复 `originalUserMessageCount` 数据链路断裂导致 L0 recorder 无法定位被污染的 user message
- 修复 `RecallResult` 类型定义缺少 `prependContext` 字段（`types.ts` 与 `auto-recall.ts` 不一致）
---
## [0.2.2] - 2026-04-17
### 🐛 修复
- 修复因未声明 `undici` 依赖导致 TCVDB 客户端加载失败的问题（开发环境之前依赖 monorepo 根 `node_modules` 的传递解析）
- 将插件注册阶段的大量 INFO 日志降级为 DEBUG，避免 CLI 模式下输出过多无关日志
## [0.2.1] - 2026-04-16 (deprecated)
> NOTE: 此版本由于存在 undici 依赖导致插件启动失败的问题，已废弃
> 相关问题在 0.2.2 及以后版本中已修复
### 🚀 新功能
- TCVDB 新增 HTTPS 连接支持，可通过插件配置 `caPemPath` 或迁移脚本参数 `--tcvdb-ca-pem` 指定自定义 CA 证书 PEM 文件
- `read-local-memory` 脚本新增 L2 单文件查询，并将 L0 / L1 查询切换为直接从 `vectors.db` 读取，支持 SQL 层过滤、排序与分页
### ✨ 改进
- TCVDB 的 L0 / L1 向量索引默认调整为 `DISK_FLAT`，并在不支持该索引类型的实例上自动回退到 `HNSW`
- 默认服务端 embedding 模型调整为 `bge-large-zh`
- TCVDB 所有读接口统一启用 `readConsistency: "strongConsistency"`，消除 read-after-write 不一致
- 健康检测脚本 VDB 连接支持 HTTPS 自签证书
### 🐛 修复
- 修复 L3 persona sync 因未拉取远端 baseline 导致版本冲突跳过写入的问题
- 修复 `memories_since_last_persona` 被 L0 和 L1 双重计数导致 persona 触发阈值膨胀的问题
- 移除 `CheckpointManager` 中已被 `captureAtomically()` 替代的废弃方法
---
## [0.2.0] - 2026-04-15
### 🚀 新功能
**腾讯云向量数据库（TCVDB）存储后端**
- 新增腾讯云向量数据库存储后端，支持向量 + BM25 混合召回
- 支持 SQLite 与 TCVDB 之间的索引结构同步
- L2 场景 / L3 画像支持在本地缓存与向量数据库之间双向同步
- 插件配置（manifest）暴露 `storeBackend`、`tcvdb`、`bm25`、`embedding.timeoutMs` 等配置项
**本地 BM25 关键字检索**
- 使用本地 tcvdb-text 编码器替代原有的 BM25 HTTP sidecar 服务，消除外部依赖
**Seed 数据导入工具**
- 新增 CLI `seed` 命令，支持从外部数据批量导入记忆
- 提取共享的 pipeline-factory，供 seed 和正常运行时复用
- 支持 ISO 8601 时间戳格式（移除 JSONL 支持）
**数据迁移与运维工具**
- 新增 SQLite → 腾讯云向量数据库迁移脚本，支持 `--help` / `-h` 展示完整参数说明和使用示例
- 新增 VDB 数据导出脚本（含预编译 JS 和 CLI 启动器）
- 新增本地 Memory 数据查询脚本
- 注册全部 CLI bin 入口：`migrate-sqlite-to-tcvdb`、`export-tencent-vdb`、`read-local-memory`
**记忆搜索工具调用限制**
- `tdai_memory_search` + `tdai_conversation_search` 增加每轮合计最多 3 次的调用次数限制，通过 tool description 和召回引导提示词约束模型行为，防止陷入无效重复搜索
### 🐛 修复
- 修复 L2 场景合并（MERGE）无法删除旧文件的问题：OpenClaw 4.1+ 的 write 工具拒绝空白内容，改用 `[DELETED]` 标记实现软删除，SceneExtractor cleanup 阶段同步识别并清理
- 修复 L2 抽取产生孤立 BATCH/ARCHIVE 文件的问题，统一 maxScenes 上限为 15
- 修复 L3 启动时重复拉取 profile 的问题
- 过滤 skill wrapper 噪声标记（`¥¥[...]¥¥`）
- 处理 `createCollection` 并发竞态（错误码 15202）
### ♻️ 重构
- Pipeline checkpoint 游标语义从 timestamp 改为 update_at
- Runner 改用 `api.runtime.agent.runEmbeddedPiAgent`，避免跨环境导入失败
- 统一脚本构建流程：新增 `build:scripts` 一键编译命令，`prepack` 钩子确保 `npm pack` 前自动编译全部脚本产物
### 📚 文档
- 新增 AI Agent 长期记忆插件设计与实现技术文档
- 新增项目指南、研发系统分层架构文档
- 新增 VDB 存储设计文档及迁移指南
---
<details>
<summary>预发布版本</summary>
## [0.2.0-beta.1] - 2026-04-14
*此版本的内容已合并至 [0.2.0] 正式版。*
</details>
## [0.1.4] - 2026-04-10
### 🚀 Features
- *(auto-recall)* Add recall hint text before memories
## [0.1.3] - 2026-04-09
### 🚀 功能
- *(memory-tdai)* 用 reporter 抽象替换 emitMetric
- *(L3)* L3 使用读写工具，防止模型输出 CoT
- *(memory)* 添加 embedding 截断、召回超时，以及从 L0 捕获中剔除代码块
- *(config)* Embedding 超时支持配置
- *(report)* 在 schema 中暴露 report 配置项，默认值改为 false
### 🐛 修复
- *(capture)* 跳过心跳/定时任务/自动化/调度类消息
- *(recall)* 召回完成时清除超时定时器，避免误报超时警告
### 💼 Other
- 重命名包名为 memory-tencentdb
- *(deps)* 将 node-llama-cpp 改为可选依赖
### ⚡ 性能
- *(auto-capture)* 将 L0 向量嵌入移至后台以降低延迟
### 📚 文档
- 添加 allowPromptInjection 配置警告说明
## [0.1.2] — 2026-03-26
### 更新内容
1. 优化对话捕获与记忆抽取过滤机制
## [0.1.1] — 2026-03-25
### 更新内容
1. 兼容 openclaw 2026.3.23 更新
## [0.1.0] — 2026-03-25
> 首个正式发布版本。本地优先的四层记忆系统（L0→L1→L2→L3），基于 SQLite + LLM 实现对话捕获、记忆提取、场景归纳与用户画像。
### 更新内容
1. 关键字检索增加 FTS5 全文索引，采用 jieba 分词
2. 未配置远程 embedding 服务时，默认不开启 embedding 能力（不自动使用本地 embedding，且封禁主动使用本地 embedding 的配置入口）
3. 优化 L2、L3 生成 prompt 以控制生成内容大小（减少 token 开销）
4. Pipeline 调度器优化文件锁用法
5. 避免全量读取 L0、L1 数据
CONTRIBUTING.md
+133
View File
@@ -0,0 +1,133 @@
# Contributing Guide
Thank you for your interest in the **TencentDB Agent Memory** project! We welcome all kinds of contributions from the community — whether it's reporting issues, improving documentation, or submitting code.
## How to Contribute
- **Report Bugs**: Describe the issue in GitHub Issues and provide steps to reproduce.
- **Request Features**: Describe your use case and proposed solution in Issues.
- **Improve Documentation**: Fix typos, clarify explanations, or add examples.
- **Submit Code**: Fix bugs, implement new features, or optimize performance.
## Getting Started
### Prerequisites
- Node.js >= 22.16.0
- npm or pnpm
- OpenClaw >= 2026.3.13
### Developing from Source
This project requires no build step. Node.js 22.16+ natively supports TypeScript type stripping, and OpenClaw directly loads `.ts` source files at runtime.
```bash
# Clone the repository
git clone https://github.com/Tencent/TencentDB-Agent-Memory.git
cd TencentDB-Agent-Memory
# Install dependencies
npm install
# Register the current directory as a local plugin in OpenClaw
openclaw plugins install --link .
```
`install --link` registers the current directory as a local plugin in OpenClaw. After modifying source code, simply restart the Gateway for changes to take effect.
### Project Structure
```
├── index.ts # Plugin entry point
├── openclaw.plugin.json # OpenClaw plugin manifest
├── src/
│ ├── config.ts # Configuration management
│ ├── conversation/ # L0 Conversation layer — raw dialogue capture
│ ├── record/ # L1 Record layer — structured information extraction
│ ├── scene/ # L2 Scene layer — scene summarization & aggregation
│ ├── persona/ # L3 Persona layer — user profile construction
│ ├── store/ # Storage layer — SQLite / vector database
│ ├── hooks/ # OpenClaw hooks integration
│ ├── prompts/ # LLM prompt templates
│ ├── tools/ # Tool functions
│ ├── utils/ # General utilities
│ └── report/ # Health check & reporting
├── hermes-plugin/ # Hermes agent plugin adapter
├── scripts/ # Helper scripts (Gateway control, etc.)
├── CHANGELOG.md # Changelog
└── README.md # Project documentation
```
## Submitting a Pull Request
1. **Fork** this repository and create your feature branch from `main`.
2. **Make changes** — keep each commit focused and atomic.
3. **Test** — ensure existing functionality is not affected.
4. **Update documentation** — if changes affect user-facing behavior, update the README or related docs.
5. **Open a PR** — describe the motivation, changes, and link related Issues.
### Branch Information
| Branch | Purpose |
|--------|---------|
| `main` | Default branch, PR target |
## Commit Message Convention
Use the following format for commit messages:
```
<type>(<scope>): <short summary>
<detailed description (optional)>
Closes #123
Signed-off-by: Your Name <your-email@example.com>
```
### Types
Aligned with the PR template Change Types:
| Type | Description | PR Change Type |
|------|-------------|----------------|
| `fix` | Bug fix | Bug fix |
| `feat` | New feature | New feature |
| `docs` | Documentation update | Documentation update |
| `perf` | Performance optimization | Code optimization |
| `refactor` | Code refactoring (no behavior change) | Code optimization |
| `test` | Test related | — |
| `chore` | Build / tooling / dependency changes | — |
### Scope Examples
`store`, `hooks`, `persona`, `scene`, `record`, `conversation`, `gateway`, `hermes`
## Code Style
- **TypeScript**: Follow the existing code style in the project for consistency.
- **Naming**: Use meaningful variable and function names, prefer English.
- **Comments**: Add comments at critical logic points explaining "why" rather than "what".
- **Import order**: Node.js built-in modules → third-party dependencies → internal project modules.
## Developer Certificate of Origin (DCO)
All commits must include a `Signed-off-by` line, indicating your agreement to the [Developer Certificate of Origin](https://developercertificate.org/):
```bash
git commit -s -m "feat(store): add batch insert support"
```
Commits without a valid `Signed-off-by` will not be merged.
## Security Issues
If you discover a security vulnerability, please report it via the email agentmemory@tencent.com and we will address it promptly.
## License
By submitting a contribution, you agree that your code will be licensed under the [MIT License](./LICENSE).
---
Thank you again for contributing! If you have any questions, feel free to discuss them in Issues.
CONTRIBUTING_CN.md
+133
View File
@@ -0,0 +1,133 @@
# 贡献指南
感谢你对 **TencentDB Agent Memory** 项目的关注！我们欢迎来自社区的各种贡献——无论是报告问题、改进文档还是提交代码。
## 贡献方式
- **报告 Bug**：在 GitHub Issues 中描述问题并提供复现步骤。
- **请求功能**：在 Issues 中描述使用场景和你期望的解决方案。
- **改进文档**：修复错别字、完善说明或补充示例。
- **提交代码**：修复 Bug、实现新功能或优化性能。
## 开发入门
### 前置条件
- Node.js >= 22.16.0
- npm 或 pnpm
- OpenClaw >= 2026.3.13
### 从源码开发
本项目无需编译。Node.js 22.16+ 原生支持 TypeScript 类型剥离，OpenClaw 直接加载 `.ts` 源码运行。
```bash
# 克隆仓库
git clone https://github.com/Tencent/TencentDB-Agent-Memory.git
cd TencentDB-Agent-Memory
# 安装依赖
npm install
# 将当前目录作为本地插件注册到 OpenClaw
openclaw plugins install --link .
```
`install --link` 会将当前目录作为本地插件注册到 OpenClaw，修改源码后重启 Gateway 即可生效。
### 项目结构
```
├── index.ts # 插件入口
├── openclaw.plugin.json # OpenClaw 插件清单
├── src/
│ ├── config.ts # 配置管理
│ ├── conversation/ # L0 对话层 — 原始对话捕获
│ ├── record/ # L1 记录层 — 结构化信息提取
│ ├── scene/ # L2 场景层 — 场景归纳与聚合
│ ├── persona/ # L3 画像层 — 用户画像构建
│ ├── store/ # 存储层 — SQLite/向量数据库
│ ├── hooks/ # OpenClaw 钩子集成
│ ├── prompts/ # LLM 提示词模板
│ ├── tools/ # 工具函数
│ ├── utils/ # 通用工具
│ └── report/ # 健康检测与报告
├── hermes-plugin/ # Hermes 智能体插件适配
├── scripts/ # 辅助脚本（Gateway 控制等）
├── CHANGELOG.md # 变更日志
└── README.md # 项目说明
```
## 提交 Pull Request
1. **Fork** 本仓库并基于 `main` 分支创建你的特性分支。
2. **进行修改** — 保持每个提交专注且原子化。
3. **测试** — 确保现有功能不受影响。
4. **更新文档** — 如果更改涉及用户可见行为，请同步更新 README 或相关文档。
5. **提交 PR** — 描述修改动机、变更内容，并关联相关 Issue。
### 分支说明
| 分支 | 用途 |
|------|------|
| `main` | 默认分支，PR 提交目标 |
## 提交信息规范
使用以下格式编写 commit message：
```
<类型>(<范围>): <简要描述>
<详细说明（可选）>
Closes #123
Signed-off-by: Your Name <your-email@example.com>
```
### 类型
与 PR 模板中的 Change Type 对应：
| 类型 | 说明 | 对应 PR Change Type |
|------|------|---------------------|
| `fix` | Bug 修复 | Bug fix |
| `feat` | 新功能 | New feature |
| `docs` | 文档更新 | Documentation update |
| `perf` | 代码优化 | Code optimization |
| `refactor` | 代码重构（不影响功能） | Code optimization |
| `test` | 测试相关 | — |
| `chore` | 构建/工具/依赖变更 | — |
### 范围示例
`store`、`hooks`、`persona`、`scene`、`record`、`conversation`、`gateway`、`hermes`
## 代码风格
- **TypeScript**：使用项目已有的代码风格，保持一致性。
- **命名**：使用有意义的变量名和函数名，优先使用英文。
- **注释**：关键逻辑处添加注释，说明"为什么"而非"做了什么"。
- **导入顺序**：Node.js 内置模块 → 第三方依赖 → 项目内部模块。
## 开发者来源证书 (DCO)
所有提交必须包含 `Signed-off-by` 行，表示你同意 [开发者来源证书](https://developercertificate.org/)：
```bash
git commit -s -m "feat(store): add batch insert support"
```
没有有效 `Signed-off-by` 的提交将不会被合并。
## 安全问题
如果你发现安全漏洞，请通过邮箱 agentmemory@tencent.com 报告，我们会尽快处理。
## 许可证
提交贡献即表示你同意你的代码将在 [MIT License](./LICENSE) 下许可。
---
再次感谢你的贡献！如有任何问题，欢迎在 Issues 中讨论。
LICENSE
+27
View File
@@ -0,0 +1,27 @@
Tencent is pleased to support the open source community by making TencentDB Agent Memory available.
Copyright (C) 2026 Tencent. All rights reserved.
TencentDB Agent Memory is licensed under the MIT.
Terms of the MIT:
--------------------------------------------------------------------
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
README.md
+472
View File
@@ -0,0 +1,472 @@
<div align="center">
<img src="./assets/images/logo.png" alt="TencentDB Agent Memory" width="880" />
### Agents remember,Humans innovate.
[![npm](https://img.shields.io/npm/v/@tencentdb-agent-memory/memory-tencentdb?color=blue)](https://www.npmjs.com/package/@tencentdb-agent-memory/memory-tencentdb)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](./LICENSE)
[![Node](https://img.shields.io/badge/node-%3E=22.16-brightgreen)](https://nodejs.org/)
[![OpenClaw](https://img.shields.io/badge/OpenClaw-%3E=2026.3.13-orange)](https://github.com/openclaw/openclaw)
[![Hermes](https://img.shields.io/badge/Hermes-Gateway-7B61FF)](https://hermes-agent.nousresearch.com/docs/)
[![Discord](https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&logoColor=white)](https://discord.gg/kDtHb5RW2)
[Highlights](#-highlights) · [Overview](#overview) · [Core Technology](#core-technology-reject-flat-storage-embrace-layering-and-symbolization) · [Features](#-features) · [Quick Start](#quick-start)
<div align="center">
[**English**](./README.md) · [简体中文](./README_CN.md)
</div>
</div>
---
## ✨ Highlights
> **TencentDB Agent Memory = symbolic short-term memory + layered long-term memory.**
>
> - **Symbolic short-term memory** offloads heavy tool logs and condenses them into compact Mermaid symbols, cutting token usage and improving task success.
> - **Layered long-term memory** distills fragmented conversations into structured personas and scenes, instead of flat vector piles.
When integrated with OpenClaw, it cuts token usage by up to **61.38%**, improves pass rate by **51.52%** (relative), and raises PersonaMem accuracy from **48%** to **76%**.
| Memory Capability | Benchmark | OpenClaw Success | With Plugin | Relative Δ | OpenClaw Tokens | With Plugin Tokens | Relative Δ |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **Short-term** | WideSearch | 33% | **50%** | **+51.52%** | 221.31M | **85.64M** | **−61.38%** |
| **Short-term** | SWE-bench | 58.4% | **64.2%** | **+9.93%** | 3474.1M | **2375.4M** | **−33.09%** |
| **Short-term** | AA-LCR | 44.0% | **47.5%** | **+7.95%** | 112.0M | **77.3M** | **−30.98%** |
| **Long-term** | PersonaMem | 48% | **76%** | **+59%** | — | — | — |
> These results are measured over continuous long-horizon sessions, not isolated turns. For example, SWE-bench runs 50 consecutive tasks per session to simulate the context-accumulation pressure of real-world long-horizon agents.
---
## Overview
**Memory is not about hoarding everything in the AI — it is about sparing humans from having to repeat themselves.**
In practice, we constantly re-explain the same SOPs, project background, tool conventions, and output formats to the Agent. Such information should not require repetition, nor should it be indiscriminately dumped into the context.
TencentDB Agent Memory helps the Agent learn your workflows, retain task context, and reuse past experience. We reject both brute-force history accumulation and irreversible lossy summarization. Instead, we design memory as a layered system: **symbolic memory** for in-task information overload, and **memory layering** for cross-session experience.
> **Let the Agent remember what should be remembered, so people can focus on judgment, creation, and work that truly matters.**
---
## Core Technology: Reject Flat Storage, Embrace Layering and Symbolization
Our architecture rests on two pillars: **memory layering** and **symbolic memory**. Together they ensure Agents do not merely "remember more", but "reason better".
### 1. Memory Layering: Progressive Disclosure with Heterogeneous Storage
Traditional memory systems shred data into fragments and dump them into a flat vector store. Recall degenerates into a blind search across disconnected fragments, with no macro-level guidance.
Whether it is long-term knowledge, short-term tasks, or future skill capabilities, memory should never be flat — both its formation and its recall must be hierarchical. TencentDB Agent Memory adopts **layering** as its unified architectural paradigm:
* **Short-term context layering.** The bottom layer archives raw tool outputs (`refs/*.md`); the middle layer extracts step-level summaries (`jsonl`); the top layer condenses state into a lightweight Mermaid canvas. The Agent only needs to attend to the top-layer structure in context, and drills down to the lower layers via `node_id` when an error occurs.
* **Long-term personalization layering.** In place of flat logs, we build a semantic pyramid: **L0 Conversation** (raw dialogue) → **L1 Atom** (atomic facts) → **L2 Scenario** (scene blocks) → **L3 Persona** (user profile). The Persona layer carries day-to-day preferences; the system drills down to Atoms only when details matter.
* **Skill generation layering.** Layering also applies to actions. The middle layer derives common solution patterns (**Scenario**) from bottom-layer execution traces (**Conversation**), and the top layer distills reusable Skills or standard SOPs (**Persona**).
<p align="center">
<img src="./assets/images/memory-pyramid-en.jpg" alt="TencentDB Agent Memory L0 to L3 semantic pyramid" width="860" />
</p>
**Heterogeneous storage and progressive disclosure.** A dual-layer storage strategy underpins this architecture. The bottom layer (facts, logs, traces) is persisted in databases for robust full-text retrieval; the top layer (personas, scenes, canvases) is stored as human-readable Markdown files for high information density and white-box inspection. **Lower layers preserve evidence; upper layers preserve structure.**
**Full traceability and lossless recovery.** Compression often sacrifices traceability. TencentDB Agent Memory avoids irreversible compression by maintaining a deterministic path from high-level abstractions back to ground-truth evidence. Whether it is an offloaded error log or a distilled user preference, the system guarantees a complete drill-down path: "top-layer symbol (Persona / canvas) → mid-layer index (Scenario / jsonl) → bottom-layer raw text (L0 Conversation / refs)".
<div align="center">
<img src="assets/images/flowchart1.png" alt="Retrievable and Recoverable Drill-Down Chain" />
</div>
### 2. Symbolic Memory: Maximum Semantics in Minimum Symbols (Mermaid Canvas)
In long tasks, the largest token consumers are verbose intermediate logs (search results, code, error traces). To address this, we combine **context offloading** with **symbolic memory**:
* **Mermaid symbol graph.** Instead of verbose prose or flat JSON, we encode task state transitions in high-density Mermaid syntax — precise enough for LLMs to parse, concise enough for humans to read.
* **History offloading.** Full tool logs are offloaded to external files; only a lightweight Mermaid task map remains in context.
* **`node_id` tracing.** The Agent reasons over the symbol graph; to verify a detail, it greps for the `node_id` and instantly retrieves the full raw text — cutting token cost while preserving full traceability.
```mermaid
graph LR
Log["Verbose Logs<br/>(hundreds of thousands of tokens)"] -->|"1. Offload full text"| FS[("External FS<br/>(refs/*.md)")]
Log -->|"2. Extract relations"| MMD["Mermaid Canvas<br/>(with node_id)"]
MMD -->|"3. Light injection"| Agent(("Agent Context<br/>(a few hundred tokens)"))
Agent -. "4. Recall via node_id" .-> FS
style Log fill:#f1f5f9,stroke:#94a3b8,stroke-dasharray: 5 5,color:#475569
style FS fill:#f8fafc,stroke:#cbd5e1,stroke-width:2px,color:#334155
style MMD fill:#eff6ff,stroke:#3b82f6,stroke-width:2px,color:#1e3a8a
style Agent fill:#fffbeb,stroke:#f59e0b,stroke-width:2px,color:#92400e
```
---
## Quick Start
This section covers **standalone local mode only**: the Memory Gateway runs locally (or inside the container), uses local `SQLite + BM25` by default, and does not depend on any cloud Memory service.
> By default, `TDAI_GATEWAY_API_KEY` is not set, so the Gateway does not enforce a fixed Bearer secret. The v2 SDK still sends non-empty `apiKey` and `serviceId` headers as part of the protocol; for standalone mode, use `apiKey = "local"` and `serviceId = "default"`.
### Option 1: Use the published standalone container images
The images are published on Docker Hub with multi-arch tags, so Docker automatically selects the correct architecture (amd64 / arm64). No local image build is required:
```bash
docker pull agentmemory/hermes-memory:1.0.0-beta
docker pull agentmemory/openclaw-memory:1.0.0-beta
```
#### Hermes + Memory (standalone)
```bash
docker run -d --name hermes-memory \
--restart unless-stopped \
-v hermes-memory-data:/opt/data \
-e MODEL_API_KEY="<LLM_API_KEY>" \
-e MODEL_BASE_URL="https://api.lkeap.cloud.tencent.com/v1" \
-e MODEL_NAME="deepseek-v3.2" \
-e MODEL_PROVIDER="custom" \
agentmemory/hermes-memory:1.0.0-beta
# Enter Hermes chat
docker exec -it hermes-memory hermes
# Check the in-container Memory Gateway
docker exec hermes-memory curl -s http://127.0.0.1:8420/health
```
> If you reuse an existing `hermes-memory-data` volume, the container keeps using `/opt/data/config.yaml` and `/opt/data/.env` from that volume. When changing `MODEL_API_KEY`, `MODEL_BASE_URL`, `MODEL_NAME`, or `MODEL_PROVIDER`, add `-e HERMES_CONFIG_OVERWRITE=1` to regenerate the config. Removing the volume also works, but it deletes local memory data.
#### OpenClaw + Memory (standalone)
```bash
docker run --rm -it --name openclaw-memory \
-p 18789:18789 \
-v openclaw-memory-data:/home/node/.openclaw \
-v openclaw-memory-store:/opt/data \
-e MODEL_API_KEY="<LLM_API_KEY>" \
-e MODEL_BASE_URL="<LLM_BASE_URL>" \
-e MODEL_NAME="<MODEL_ID>" \
-e OPENCLAW_CONFIG_OVERWRITE=1 \
agentmemory/openclaw-memory:1.0.0-beta \
openclaw-memory-tui
```
Both OpenClaw and Hermes standalone containers run their own in-container Memory Gateway at `127.0.0.1:8420`. Normal usage does not require publishing port `8420` to the host. Only add a non-conflicting mapping such as `-p 8421:8420` when you want to debug the in-container Gateway from the host.
### Option 2: Start the standalone Memory service directly
If you only want to run the Memory Gateway and call it from SDKs or your own Agent:
```bash
git clone <this-repo>
cd tdai-memory-openclaw-plugin
npm install
TDAI_GATEWAY_CONFIG="$PWD/tdai-gateway.standalone.yaml" \
TDAI_GATEWAY_HOST=127.0.0.1 \
TDAI_GATEWAY_PORT=8420 \
TDAI_DATA_DIR="$HOME/.memory-tencentdb/memory-tdai" \
TDAI_LLM_API_KEY="<LLM_API_KEY>" \
TDAI_LLM_BASE_URL="https://api.openai.com/v1" \
TDAI_LLM_MODEL="gpt-4o" \
node --import tsx/esm src/gateway/server.ts
```
Verify the service:
```bash
curl http://127.0.0.1:8420/health
```
By default, `TDAI_GATEWAY_API_KEY` is not configured, so standalone mode does not enforce a fixed shared secret. If you expose the Gateway to a network, set `TDAI_GATEWAY_API_KEY` explicitly and use the same value in clients.
This option is intended for users who want to build their own Agent Memory adapter: run only the Memory Gateway, then use the v2 SDK inside your Agent framework to write conversations, recall memories, and expose memory tools. This repository provides two adapter references:
- [OpenClaw adapter reference](./openclaw-plugin/README.md)
- [Hermes v2 adapter reference](./hermes-plugin/memory/memory_tencentdb_v2/README.md)
Full SDK API and local package build references: [TypeScript SDK docs](./sdk/typescript/README.md) / [Python SDK docs](./sdk/python/README.md).
### Option 3: Node.js SDK
Install:
```bash
npm install @tencentdb-agent-memory/memory-sdk-ts
```
Example:
```ts
import { MemoryClient } from "@tencentdb-agent-memory/memory-sdk-ts";
const client = new MemoryClient({
endpoint: "http://127.0.0.1:8420",
apiKey: "local", // standalone default; if TDAI_GATEWAY_API_KEY is set, use that value
serviceId: "default", // standalone default space
});
await client.addConversation({
session_id: "quickstart-session",
messages: [
{ role: "user", content: "I prefer TypeScript for tool scripts." },
{ role: "assistant", content: "Got it, I will remember that preference." },
],
});
const conversations = await client.queryConversation({
session_id: "quickstart-session",
limit: 10,
});
console.log(conversations.messages);
// L1 memories are extracted asynchronously. After enough turns or after the
// pipeline has run, you can search structured memories:
const memories = await client.searchAtomic({ query: "TypeScript preference", limit: 5 });
console.log(memories.items);
```
- [Node.js SDK docs](./sdk/typescript/README.md)
### Option 4: Python SDK
Install:
```bash
pip install tencentdb-agent-memory-sdk-python
```
Example:
```python
from tencentdb_agent_memory import MemoryClient
client = MemoryClient(
endpoint="http://127.0.0.1:8420",
api_key="local", # standalone default; if TDAI_GATEWAY_API_KEY is set, use that value
service_id="default", # standalone default space
)
client.add_conversation(
session_id="quickstart-session",
messages=[
{"role": "user", "content": "I prefer Python for data processing."},
{"role": "assistant", "content": "Got it, I will remember that preference."},
],
)
conversations = client.query_conversation(
session_id="quickstart-session",
limit=10,
)
print(conversations["messages"])
# L1 memories are extracted asynchronously. After enough turns or after the
# pipeline has run, you can search structured memories:
memories = client.search_atomic(query="Python preference", limit=5)
print(memories["items"])
```
- [Python SDK docs](./sdk/python/README.md)
---
## 🔒 Gateway Security (optional)
The Hermes Gateway listens on `:8420` and exposes capture / search / recall HTTP endpoints. Two opt-in switches let you turn it from "open localhost sidecar" into "authenticated network service". **Both default to off so existing deployments keep working unchanged.**
| Field | env | Default | Description |
| :--- | :--- | :--- | :--- |
| `server.apiKey` | `TDAI_GATEWAY_API_KEY` | _(unset)_ | When set, every route except `GET /health` requires `Authorization: Bearer <apiKey>`; missing or wrong tokens get HTTP 401. Comparison is constant-time. |
| `server.corsOrigins` | `TDAI_CORS_ORIGINS` (comma-separated) | `[]` | CORS allow-list. Empty list emits **no** `Access-Control-Allow-*` headers — browsers then block all cross-origin requests. Use `["*"]` only for local development. |
When `apiKey` is unset, the gateway prints a startup `WARN`. If it is bound to a non-loopback host (e.g. `0.0.0.0`) without an apiKey, a second louder warning is emitted.
Clients call protected routes with a Bearer token:
```bash
curl -H "Authorization: Bearer $TDAI_GATEWAY_API_KEY" \
-H "Content-Type: application/json" \
-d '{"query":"...","session_key":"..."}' \
http://127.0.0.1:8420/recall
```
`GET /health` stays open without a token so orchestrator probes (`docker healthcheck`, `kubectl liveness`) keep working.
### Hermes plugin side
The Hermes `memory_tencentdb` plugin is a **client** of the Gateway. To make it talk to a Gateway that has auth enabled, set:
```bash
export MEMORY_TENCENTDB_GATEWAY_API_KEY="<same-secret-as-gateway>"
```
The plugin will then attach `Authorization: Bearer <key>` to every request it sends to the Gateway. If the variable is unset, the plugin sends no auth header — which matches the Gateway's legacy default and is fine for a Gateway that has not opted into `TDAI_GATEWAY_API_KEY`.
Important: the plugin only handles the **client half**. Whether the Gateway actually enforces a Bearer check is decided on the Gateway side (`TDAI_GATEWAY_API_KEY` / `server.apiKey`). Configure the same secret on both ends — the plugin does not propagate the secret across, since the Gateway might be started by Docker, systemd, or any other means outside the plugin's control.
If `MEMORY_TENCENTDB_GATEWAY_API_KEY` is unset, the plugin also looks at `TDAI_GATEWAY_API_KEY` as a fallback — handy when both processes share an env file and the operator only wants to set one variable name. The Gateway never reads `MEMORY_TENCENTDB_GATEWAY_API_KEY`; that name is plugin-side only.
---
## 🔧 Configurable Parameters
**Every field has a sensible default — it runs with zero configuration.** When you want to tune, peel back the layers based on how deep you go.
<details>
<summary><b>🟢 Level 1 · Daily tuning</b> (covers 90% of use cases)</summary>
| Field | Default | Description |
| :--- | :--- | :--- |
| `storeBackend` | `"sqlite"` | Storage backend: `sqlite` |
| `recall.strategy` | `"hybrid"` | Recall strategy: `keyword` / `embedding` / `hybrid` (RRF fusion, recommended) |
| `recall.maxResults` | `5` | Number of items returned per recall |
| `recall.maxCharsPerMemory` | `0` | Max characters injected for one recalled L1 memory; `0` disables this guard |
| `recall.maxTotalRecallChars` | `0` | Total character budget for auto-recalled L1 memories; `0` disables this guard |
| `pipeline.everyNConversations` | `5` | Trigger an L1 memory extraction every N turns |
| `extraction.maxMemoriesPerSession` | `20` | Max memories extracted per L1 pass |
| `persona.triggerEveryN` | `50` | Generate the user persona every N new memories |
| `offload.enabled` | `false` | Whether to enable short-term compression |
</details>
<details>
<summary><b>🟡 Level 2 · Advanced tuning</b> (long task / long session)</summary>
| Field | Default | Description |
| :--- | :--- | :--- |
| `pipeline.enableWarmup` | `true` | Warm-up: a new session triggers from turn 1, doubling each time up to N (1→2→4→…) |
| `pipeline.l1IdleTimeoutSeconds` | `600` | Trigger L1 after the user has been idle for this many seconds |
| `pipeline.l2MinIntervalSeconds` | `900` | Minimum interval between two L2 passes within the same session |
| `recall.timeoutMs` | `5000` | Recall timeout; on timeout, skip injection without blocking the conversation |
| `extraction.enableDedup` | `true` | L1 vector dedup / conflict detection |
| `capture.excludeAgents` | `[]` | Glob patterns to exclude specific agents (e.g. `bench-judge-*`) |
| `capture.l0l1RetentionDays` | `0` | Local retention days for L0 / L1 files; `0` = never clean up |
| `offload.mildOffloadRatio` | `0.5` | Mild compression trigger ratio (of context window) |
| `offload.aggressiveCompressRatio` | `0.85` | Aggressive compression trigger ratio |
| `offload.mmdMaxTokenRatio` | `0.2` | Token budget ratio for MMD injection |
| `bm25.language` | `"zh"` | Tokenizer language: `zh` (jieba) / `en` |
</details>
<details>
<summary><b>🔴 Level 3 · Full parameter reference</b> (ops / custom models / remote embedding)</summary>
For all fields, types, and constraints see [`openclaw.plugin.json`](./openclaw.plugin.json)。
- `embedding.*` — remote embedding service (OpenAI-compatible API)
- `embedding.sendDimensions` (default `true`): whether to include the `dimensions` field in the request body. OpenAI `text-embedding-3-*` models rely on it for Matryoshka truncation, but some self-hosted / OSS models (e.g. **BGE-M3**) do not support custom dimensions and will reject the request with HTTP 400 `does not support matryoshka representation`. Set it to `false` for those backends, e.g.:
```json
{
"embedding": {
"enabled": true,
"provider": "openai",
"baseUrl": "http://your-host:your-port/v1",
"apiKey": "<KEY>",
"model": "bge-m3",
"dimensions": 1024,
"sendDimensions": false
}
}
```
- `llm.*` — standalone LLM mode (bypass OpenClaw's built-in model and run L1/L2/L3 with a designated API)
- `offload.backendUrl / backendApiKey` — offload the L1/L1.5/L2/L4 flow to a backend service
- `report.*` — metrics reporting
</details>
---
## 🤔 Features
### 1. Macro Personas + Micro Facts: A Unified Drill-Down Mechanism
The biggest risk in compression is saving tokens at the cost of losing the evidence. TencentDB Agent Memory therefore does not collapse history into an irreversible summary — it preserves a clear path from high-level abstraction back to ground-truth evidence.
| Question type | First look at | Drill down to |
| :--- | :--- | :--- |
| Daily preferences, voice, long-term goals | L3 Persona / L2 Scenario | L1 Atom / L0 Conversation when facts are needed |
| Specific facts, dates, project details | L1 Atom / L0 Conversation | Widen the time range, or fall back to semantic recall when results are sparse |
| Continuing a long-running task | Active Mermaid task canvas | Check the JSONL when the summary lacks detail, then `refs/*.md` for raw text |
| Resuming a historical task | Metadata task entry | Open the Mermaid canvas → locate the `node_id` → trace `result_ref` |
The upper layers carry judgment and direction; the lower layers carry evidence and precision. Short-term compression and long-term memory form a single closed loop: **collapsible and expandable, abstract yet auditable.**
### 2. White-Box Debuggability: Memory Is Not a Black Box
Most memory systems fall short here: when recall is wrong, all you see is a list of vector scores, with no way to tell where things went wrong. TencentDB Agent Memory keeps the key intermediates as readable files:
- L2 Scenario blocks are plain Markdown — open them and inspect.
- L3 Persona lives in `persona.md` and traces back to the Scenarios that produced it.
- Short-term task canvases are Mermaid — readable by both humans and Agents.
- Raw payloads, summaries, and nodes are linked by `result_ref` and `node_id`.
Debugging no longer means probing an opaque database — it becomes a deterministic walk along the chain "Persona → Scenario → Atom → Conversation" until the root cause surfaces.
**All of these layered memory artifacts live under `~/.openclaw/memory-tdai/` — feel free to open the directory and inspect each layer for yourself.**
### 3. Production-Ready Engineering: Not a Demo
| Capability | Description |
| :--- | :--- |
| OpenClaw plugin | Automatically captures, extracts, and recalls memory once installed |
| Hermes Gateway adapter | `TdaiCore + HostAdapter`, decoupled from the host framework |
| Local backend | `SQLite + sqlite-vec`, ready to use out of the box |
| Hybrid retrieval | BM25 + vector + RRF — supports both keyword and semantic recall |
| Agent tools | `tdai_memory_search` / `tdai_conversation_search` |
---
## Documentation
| Document | Contents |
| :--- | :--- |
| [`scripts/README.memory-tencentdb-ctl.md`](./scripts/README.memory-tencentdb-ctl.md) | Operations & management tooling |
| [`CHANGELOG.md`](./CHANGELOG.md) | Release notes and version history |
| [`openclaw.plugin.json`](./openclaw.plugin.json) | OpenClaw plugin manifest and configuration schema |
---
## Community & Contributing
We welcome every kind of contribution — bug reports, feature ideas, doc fixes, benchmark reproductions, ecosystem integrations, or a Pull Request. Agent memory is far from a solved problem, and we'd love to figure it out together.
- 🐞 **Found a bug or have a question?** Open an issue at [GitHub Issues](https://github.com/Tencent/TencentDB-Agent-Memory/issues) — we respond within 24 hours.
- 💡 **Have an idea to share?** Start a thread in [GitHub Discussions](https://github.com/Tencent/TencentDB-Agent-Memory/discussions).
- 🛠️ **Want to contribute code?** Please read [CONTRIBUTING.md](./CONTRIBUTING.md) first.
- 💬 **Want to chat with us?** Join our [Discord community](https://discord.gg/kDtHb5RW2) and talk to the early developers directly.
---
## Roadmap
- [x] Long-term personalized memory (L0 → L3)
- [x] Short-term context compression (Context Offload + Mermaid canvas)
- [x] Local SQLite backend and Tencent Cloud Vector Database (TCVDB) backend
- [x] OpenClaw plugin and Hermes Gateway integration
- [ ] Portable memory: cross-Agent / cross-framework / cross-device import, export, and live migration
- [ ] Automatic Skill generation
- [ ] Visual debugging and memory observability dashboard
---
<table>
<tr>
<td width="68%">
<b>If TencentDB Agent Memory has been useful to you, please give the project a ⭐ to support us.</b><br />
For any suggestions, feel free to open an issue and start the discussion.
</td>
<td width="32%" align="right">
<img src="./assets/images/star-helper.png" alt="Star TencentDB Agent Memory" width="260" />
</td>
</tr>
</table>
[MIT](./LICENSE) © TencentDB Agent Memory Team
README_CN.md
+474
View File
@@ -0,0 +1,474 @@
<div align="center">
<img src="./assets/images/logo.png" alt="TencentDB Agent Memory" width="880" />
### 让 Agent 沉淀经验，让人专注创造。
[![npm](https://img.shields.io/npm/v/@tencentdb-agent-memory/memory-tencentdb?color=blue)](https://www.npmjs.com/package/@tencentdb-agent-memory/memory-tencentdb)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](./LICENSE)
[![Node](https://img.shields.io/badge/node-%3E=22.16-brightgreen)](https://nodejs.org/)
[![OpenClaw](https://img.shields.io/badge/OpenClaw-%3E=2026.3.13-orange)](https://github.com/openclaw/openclaw)
[![Hermes](https://img.shields.io/badge/Hermes-Gateway-7B61FF)](https://hermes-agent.nousresearch.com/docs/)
[![Discord](https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&logoColor=white)](https://discord.gg/kDtHb5RW2)
[效果亮点](#-效果亮点) · [项目简介](#项目简介) · [核心技术](#核心技术拒绝平铺走向分层与符号化) · [方案特点](#-方案特点) · [快速开始](#快速开始)
<div align="center">
[English](./README.md) · [**简体中文**](./README_CN.md)
</div>
---
</div>
## ✨ 效果亮点
> **TencentDB Agent Memory = 符号化短期记忆 + 分层式长期记忆。**
>
> - **符号化短期记忆**：将厚重的工具日志分层卸载，逐步总结成轻量级 Mermaid 结构符号，大幅降低 Token 消耗的同时提升任务成功率。
> - **分层式长期记忆**：把碎片化对话层层提炼，沉淀出有层次的画像与场景，不再是扁平的向量堆砌。
**作为 OpenClaw 插件接入后**：最高节省 **61.38% Token**，通过率相对提升 **51.52%**；PersonaMem 准确率从 **48%** 提升到 **76%**。
| 记忆能力 | Benchmark | OpenClaw 成功率 | 加插件后成功率 | 相对变化 | OpenClaw Token 消耗 | 加插件后 Token 消耗 | 相对变化 |
| :------ | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **短期记忆** | WideSearch | 33% | **50%** | **+51.52%** | 221.31M | **85.64M** | **−61.38%** |
| **短期记忆** | SWE-bench | 58.4% | **64.2%** | **+9.93%** | 3474.1M | **2375.4M** | **−33.09%** |
| **短期记忆** | AA-LCR | 44.0% | **47.5%** | **+7.95%** | 112.0M | **77.3M** | **−30.98%** |
| **长期记忆** | PersonaMem | 48% | **76%** | **+59%** | — | — | — |
> 超长 Session 评测不是单题清空上下文，而是把多个任务拼接到同一个 Session 中连续执行。例如 SWE-bench 每个 Session 连续执行 50 个任务，用来模拟真实长程 Agent 的上下文累积压力。
---
## 项目简介
**Memory 不是为了让 AI 存下所有东西，而是为了让人不必重复所有事情。**
在实际使用中，我们往往需要反复告诉 Agent 固定 SOP、项目背景、工具习惯和输出格式。这些信息不应该每次重新解释，但同时也不应该被无脑、扁平地全部塞进上下文。
TencentDB Agent Memory 帮助 Agent 学会你的流程、保留任务上下文、复用历史经验。但我们**拒绝暴力的历史堆砌**，也**抛弃不可逆的暴力摘要**。我们将记忆设计为一套极具层次感的系统，以**符号化记忆**解决单次长任务的信息过载，以**记忆分层**解决跨会话的经验沉淀。
> **让 Agent 记住该记的，让人把注意力留给判断、创造和真正有价值的工作。**
<p align="center">
<img src="https://github.com/user-attachments/assets/53513290-2c77-45cf-b06f-b07544902bd8" width="360" alt="Agent Memory 微信社群二维码" />
<br/>
<sub>📱 扫码加入 <b>Agent Memory 微信社群</b>，与早期开发者直接对话</sub>
</p>
## 核心技术：拒绝平铺，走向分层与符号化
TencentDB Agent Memory 的设计理念围绕两个核心展开：**记忆分层** 与 **符号化记忆**。这不仅让 Agent “记得更多”，更重要的是“想得更清”。
### 1. 记忆分层：渐进式披露与异构存储
传统的记忆系统往往将所有数据切片并平铺为向量，召回时如同在一堆毫无关联的便利贴中寻找线索，缺乏宏观视角的指引。
我们认为，**不管是长期的知识、短期的任务，还是未来的经验能力，记忆都不应该平铺，生成和召回都必须有层次**。TencentDB Agent Memory 将“分层”作为整个架构的统一设计哲学：
* **短期记忆（上下文卸载/任务）的分层**：底层保留原始、厚重的工具调用结果（`refs/*.md`），中层抽取步骤摘要（`jsonl`），高层则浓缩为一张极度轻量的 Mermaid 任务画布。Agent 在上下文中仅需关注高层结构，遇错时再沿着 `node_id` 下钻到底层查证。
* **长期个性化（用户理解）的分层**：打破扁平的历史记录，建立 **L0 Conversation**（原始对话） → **L1 Atom**（结构化事实） → **L2 Scenario**（场景块） → **L3 Persona**（用户画像）的语义金字塔。平时靠高层 Persona 把握用户偏好，需要考证细节时再检索底层 Atom。
* **技能生成（Skill与动作沉淀，Roadmap）的分层**：记忆不应仅限于“知道什么”，还应包括“会做什么”。我们正在将分层延伸至动作域：从底层的执行轨迹（Traces）与报错日志中，中层归纳出共性的解决模式，高层最终提炼出可直接挂载复用的 Skill 或标准 SOP 代码。
<p align="center">
<img src="./assets/images/memory-pyramid.png" alt="TencentDB Agent Memory L0 to L3 semantic pyramid" width="860" />
</p>
**渐进式披露与异构存储**：为了支撑这种无处不在的分层，我们设计了底层数据库与上层文件系统结合的存储方案。底层（海量事实、日志、轨迹）存入数据库或归档文件，确保稳定与全量检索；高层（画像、场景、画布、Skill）存入业务可读的文件系统（Markdown），确保高信息密度、逻辑清晰与白盒可调。**低层保留证据，高层保留结构。**
**每一条信息都 100% 可找回、可恢复**：压缩或抽象最大的风险是“丢失证据”。得益于严格的索引映射机制，系统内没有任何一段摘要是“不可逆”的黑盒。无论是短期记忆中被卸载的一段报错日志，还是长期记忆里总结出的一条用户偏好，Agent 或开发者都可以沿着“高层符号（Persona / 画布） → 中层索引（Scenario / JSONL） → 底层原文（L0 Conversation / refs）”的链路进行完美溯源与恢复。
<div align="center">
<img src="assets/images/flowchart1.cn.png" alt="Retrievable and Recoverable Drill-Down Chain" />
</div>
### 2. 符号化记忆：用最少符号表达最多语义（Mermaid 画布）
长程任务中最消耗 Token 的往往是繁杂的过程日志（如搜索结果、代码、报错）。为此，我们结合 **上下文卸载 (Context Offloading)** 提出了 **符号化记忆**：
* **Mermaid 符号图谱**：取代冗长的自然语言或扁平的 JSON，我们使用高密度、强拓扑的 Mermaid 语法来描绘任务状态流转，既能被 LLM 精准理解，也方便人类阅览。
* **历史折叠与卸载**：完整工具日志被卸载到外部文件系统，上下文仅保留轻量级的 Mermaid 任务地图。
* **基于 `node_id` 的溯源**：Agent 看着符号图谱推理，如需核对细节，直接 grep 图谱上的 `node_id` 即可瞬间找回完整原文，既大幅降本又保全了 100% 可追溯性。
```mermaid
graph LR
Log["繁杂冗长的过程日志<br/>(几十万 Token)"] -->|"1. 卸载完整原文"| FS[("外部文件系统<br/>(refs/xxx.md)")]
Log -->|"2. 提取关系"| MMD["Mermaid 符号图谱<br/>(带 node_id)"]
MMD -->|"3. 轻量级注入"| Agent(("Agent 上下文<br/>(几百 Token)"))
Agent -. "4. 随时按 node_id 下钻恢复原文" .-> FS
style Log fill:#f1f5f9,stroke:#94a3b8,stroke-dasharray: 5 5
style FS fill:#f8fafc,stroke:#cbd5e1
style MMD fill:#eff6ff,stroke:#3b82f6,stroke-width:2px
style Agent fill:#fffbeb,stroke:#f59e0b,stroke-width:2px
```
---
## 快速开始
本节只介绍 **standalone 本地模式**：Memory Gateway 运行在本机或容器内，默认使用本地 `SQLite + BM25`，不依赖云端 Memory 服务。
> 默认不设置 `TDAI_GATEWAY_API_KEY`，因此 Gateway 不校验固定 Bearer 密钥。v2 SDK 仍会按协议发送非空 `apiKey` 和 `serviceId` header；standalone 约定使用 `apiKey = "local"`、`serviceId = "default"` 即可。
### 方式一： 直接使用远端 standalone 容器镜像
镜像已发布到 Docker Hub，并提供 multi-arch tag，会自动匹配当前系统架构（amd64 / arm64），无需本地构建：
```bash
docker pull agentmemory/hermes-memory:1.0.0-beta
docker pull agentmemory/openclaw-memory:1.0.0-beta
```
#### Hermes + Memory（standalone）
```bash
docker run -d --name hermes-memory \
--restart unless-stopped \
-v hermes-memory-data:/opt/data \
-e MODEL_API_KEY="<LLM密钥>" \
-e MODEL_BASE_URL="https://api.lkeap.cloud.tencent.com/v1" \
-e MODEL_NAME="deepseek-v3.2" \
-e MODEL_PROVIDER="custom" \
agentmemory/hermes-memory:1.0.0-beta
# 进入 Hermes 对话
docker exec -it hermes-memory hermes
# 查看容器内 Memory Gateway 健康状态
docker exec hermes-memory curl -s http://127.0.0.1:8420/health
```
> 如果复用了旧的 `hermes-memory-data` volume，容器会继续读取已有的 `/opt/data/config.yaml` 和 `/opt/data/.env`。当你更换 `MODEL_API_KEY`、`MODEL_BASE_URL`、`MODEL_NAME` 或 `MODEL_PROVIDER` 时，请在 `docker run` 中额外加入 `-e HERMES_CONFIG_OVERWRITE=1` 让配置重新生成；如果删除 volume，则会同时清空本地记忆数据。
#### OpenClaw + Memory（standalone）
```bash
docker run --rm -it --name openclaw-memory \
-p 18789:18789 \
-v openclaw-memory-data:/home/node/.openclaw \
-v openclaw-memory-store:/opt/data \
-e MODEL_API_KEY="<LLM密钥>" \
-e MODEL_BASE_URL="<LLM服务地址>" \
-e MODEL_NAME="<模型ID>" \
-e OPENCLAW_CONFIG_OVERWRITE=1 \
agentmemory/openclaw-memory:1.0.0-beta \
openclaw-memory-tui
```
OpenClaw 和 Hermes standalone 容器内部都会自带 `127.0.0.1:8420` Memory Gateway，正常使用不需要把 `8420` 映射到宿主机。只有当你要从宿主机直接调试容器内 Gateway 时，才额外加 `-p 8421:8420` 这类非冲突端口映射。
### 方式二： 直接启动 standalone Memory 服务
如果你只想启动 Memory Gateway，然后通过 SDK 或自己的 Agent 调用：
```bash
git clone <this-repo>
cd tdai-memory-openclaw-plugin
npm install
TDAI_GATEWAY_CONFIG="$PWD/tdai-gateway.standalone.yaml" \
TDAI_GATEWAY_HOST=127.0.0.1 \
TDAI_GATEWAY_PORT=8420 \
TDAI_DATA_DIR="$HOME/.memory-tencentdb/memory-tdai" \
TDAI_LLM_API_KEY="<LLM密钥>" \
TDAI_LLM_BASE_URL="https://api.openai.com/v1" \
TDAI_LLM_MODEL="gpt-4o" \
node --import tsx/esm src/gateway/server.ts
```
验证服务：
```bash
curl http://127.0.0.1:8420/health
```
默认不配置 `TDAI_GATEWAY_API_KEY`，即 standalone 服务不启用固定密钥校验；如果要暴露到公网或局域网，请显式设置 `TDAI_GATEWAY_API_KEY` 并在客户端使用同一个密钥。
该方案适合希望自己开发 Agent Memory 适配层的用户：你可以只启动 Memory Gateway，然后在自己的 Agent 框架里通过 v2 SDK 完成对话写入、记忆召回和工具暴露。当前项目已提供两个可参考的 adapter 实现：
- [OpenClaw 适配参考](./openclaw-plugin/README_CN.md)
- [Hermes v2 适配参考](./hermes-plugin/memory/memory_tencentdb_v2/README_CN.md)
SDK 完整接口与本地包构建参考：[TypeScript SDK 文档](./sdk/typescript/README_CN.md) / [Python SDK 文档](./sdk/python/README_CN.md)。
### 方式三： Node.js SDK 使用
安装：
```bash
npm install @tencentdb-agent-memory/memory-sdk-ts
```
示例：
```ts
import { MemoryClient } from "@tencentdb-agent-memory/memory-sdk-ts";
const client = new MemoryClient({
endpoint: "http://127.0.0.1:8420",
apiKey: "local", // standalone 默认即可；若设置了 TDAI_GATEWAY_API_KEY，改成对应密钥
serviceId: "default", // standalone 默认空间
});
await client.addConversation({
session_id: "quickstart-session",
messages: [
{ role: "user", content: "我喜欢用 TypeScript 写工具脚本。" },
{ role: "assistant", content: "好的，我会记住这个偏好。" },
],
});
const conversations = await client.queryConversation({
session_id: "quickstart-session",
limit: 10,
});
console.log(conversations.messages);
// L1 记忆是异步抽取的；对话足够多或等待 pipeline 后可检索：
const memories = await client.searchAtomic({ query: "TypeScript 偏好", limit: 5 });
console.log(memories.items);
```
- [Node.js SDK简介](./sdk/typescript/README_CN.md)
### 方式四： Python SDK 使用
安装：
```bash
pip install tencentdb-agent-memory-sdk-python
```
示例：
```python
from tencentdb_agent_memory import MemoryClient
client = MemoryClient(
endpoint="http://127.0.0.1:8420",
api_key="local", # standalone 默认即可；若设置了 TDAI_GATEWAY_API_KEY，改成对应密钥
service_id="default", # standalone 默认空间
)
client.add_conversation(
session_id="quickstart-session",
messages=[
{"role": "user", "content": "我喜欢用 Python 做数据处理。"},
{"role": "assistant", "content": "好的，我会记住这个偏好。"},
],
)
conversations = client.query_conversation(
session_id="quickstart-session",
limit=10,
)
print(conversations["messages"])
# L1 记忆是异步抽取的；对话足够多或等待 pipeline 后可检索：
memories = client.search_atomic(query="Python 偏好", limit=5)
print(memories["items"])
```
- [Python SDK简介](./sdk/python/README_CN.md)
---
## 🔒 Gateway 安全配置（可选）
Hermes Gateway 监听 `:8420`，对外提供 capture / search / recall 的 HTTP 接口。新增两个开关，可以把它从“开放的本地 sidecar”切换为“需要鉴权的网络服务”。**两个开关默认都关闭，已有部署的行为不变。**
| 字段 | env | 默认值 | 说明 |
| :--- | :--- | :--- | :--- |
| `server.apiKey` | `TDAI_GATEWAY_API_KEY` | _(未设置)_ | 设置后，除 `GET /health` 外的所有接口都要求 `Authorization: Bearer <apiKey>`；缺失或错误的 token 返回 HTTP 401。Token 比较使用常量时间算法，避免时序侧信道。 |
| `server.corsOrigins` | `TDAI_CORS_ORIGINS`（逗号分隔） | `[]` | CORS 白名单。空列表表示**不发送**任何 `Access-Control-Allow-*` 响应头，浏览器同源策略会自动阻止跨域请求。`["*"]` 仅供本地开发，不要用于生产。 |
当 `apiKey` 未设置时，Gateway 启动时会打印一条 `WARN`；如果同时还绑定在非 loopback 地址（例如 `0.0.0.0`），还会再打印一条更醒目的告警。
客户端在启用鉴权后用 Bearer token 调用：
```bash
curl -H "Authorization: Bearer $TDAI_GATEWAY_API_KEY" \
-H "Content-Type: application/json" \
-d '{"query":"...","session_key":"..."}' \
http://127.0.0.1:8420/recall
```
`GET /health` 永远无需 token，方便 `docker healthcheck` / `kubectl liveness` 等编排探针继续工作。
### Hermes 插件侧的配置
Hermes `memory_tencentdb` 插件本身是 Gateway 的**客户端**。当 Gateway 开启了鉴权后，在 Hermes 进程上设置：
```bash
export MEMORY_TENCENTDB_GATEWAY_API_KEY="<与 Gateway 同一份密钥>"
```
插件随后会在每一次发往 Gateway 的请求上附带 `Authorization: Bearer <key>`。该变量未设置时，插件不发送任何鉴权头，与 Gateway 维持开放模式的旧行为完全匹配——已有部署 0 影响。
需要明确的边界：**插件只负责 client 一侧**。Gateway 是否真的强制鉴权由 Gateway 端自己的 `TDAI_GATEWAY_API_KEY` / `server.apiKey` 决定。两端要使用相同的密钥才能匹配；插件不会把这个值传递给 Gateway——因为 Gateway 可能由 Docker、systemd 或其它独立机制拉起，插件没有也不应该去管这个。
若 `MEMORY_TENCENTDB_GATEWAY_API_KEY` 没设置，插件还会回退读取 `TDAI_GATEWAY_API_KEY`，方便两个进程共享同一个 env 文件、只设一个变量名的场景。Gateway 永远不会读 `MEMORY_TENCENTDB_GATEWAY_API_KEY`，那是插件侧专用名字。
---
## 🔧 可调参数
**所有字段均有合理默认值，零配置即可跑。** 如果要调优，可以按使用深度逐层展开。
<details>
<summary><b>🟢 Level 1 · 日常调参</b>（覆盖 90% 使用场景）</summary>
| 字段 | 默认 | 说明 |
| :--- | :--- | :--- |
| `storeBackend` | `"sqlite"` | 存储后端：`sqlite` |
| `recall.strategy` | `"hybrid"` | 召回策略：`keyword` / `embedding` / `hybrid`（RRF 融合，推荐） |
| `recall.maxResults` | `5` | 每次召回条数 |
| `recall.maxCharsPerMemory` | `0` | 单条 L1 记忆注入的最大字符数；`0` 表示不限制 |
| `recall.maxTotalRecallChars` | `0` | 每轮 auto-recall 注入的 L1 记忆总字符预算；`0` 表示不限制 |
| `pipeline.everyNConversations` | `5` | 每 N 轮对话触发一次 L1 记忆提取 |
| `extraction.maxMemoriesPerSession` | `20` | 单次 L1 最多提取多少条 |
| `persona.triggerEveryN` | `50` | 每 N 条新记忆触发用户画像生成 |
| `offload.enabled` | `false` | 是否启用短期记忆压缩 |
</details>
<details>
<summary><b>🟡 Level 2 · 进阶调优</b>（长任务 / 长 Session 场景）</summary>
| 字段 | 默认 | 说明 |
| :--- | :--- | :--- |
| `pipeline.enableWarmup` | `true` | Warm-up：新 session 从 1 轮起触发，每次翻倍至 N（1→2→4→…） |
| `pipeline.l1IdleTimeoutSeconds` | `600` | 用户停止对话多久后触发 L1 |
| `pipeline.l2MinIntervalSeconds` | `900` | 同 session 两次 L2 之间的最小间隔 |
| `recall.timeoutMs` | `5000` | 召回超时阈值，超时跳过注入不阻塞对话 |
| `extraction.enableDedup` | `true` | L1 向量去重 / 冲突检测 |
| `capture.excludeAgents` | `[]` | Glob 模式排除特定 Agent（如 `bench-judge-*`） |
| `capture.l0l1RetentionDays` | `0` | L0/L1 本地文件保留天数，`0` = 永不清理 |
| `offload.mildOffloadRatio` | `0.5` | 温和压缩触发比例（占 context window） |
| `offload.aggressiveCompressRatio` | `0.85` | 激进压缩触发比例 |
| `offload.mmdMaxTokenRatio` | `0.2` | MMD 注入 token 预算比例 |
| `bm25.language` | `"zh"` | 分词语言：`zh`（jieba） / `en` |
</details>
<details>
<summary><b>🔴 Level 3 · 完整参数表</b>（运维 / 自定义模型 / 远程 embedding）</summary>
完整字段、类型、约束见 [`openclaw.plugin.json`](./openclaw.plugin.json) 。
- `embedding.*` — 远程 embedding 服务（OpenAI 兼容 API）
- `embedding.sendDimensions`（默认 `true`）：是否在请求体中携带 `dimensions` 字段。OpenAI `text-embedding-3-*` 系列依赖该字段做 Matryoshka 维度截断；但部分自托管 / 开源模型（如 **BGE-M3**）不支持自定义维度，会以 HTTP 400 报 `does not support matryoshka representation` 拒绝请求。此时请显式设为 `false`，例如：
```json
{
"embedding": {
"enabled": true,
"provider": "openai",
"baseUrl": "http://your-host:your-port/v1",
"apiKey": "<KEY>",
"model": "bge-m3",
"dimensions": 1024,
"sendDimensions": false
}
}
```
- `llm.*` — 独立 LLM 模式（绕过 OpenClaw 内置模型，用指定 API 跑 L1/L2/L3）
- `offload.backendUrl / backendApiKey` — 将 L1/L1.5/L2/L4 offload 流程卸载到后端服务
- `report.*` — 指标上报
</details>
---
## 🤔 方案特点
### 1. 宏观画像 + 微观事实：同一套下钻机制降低幻觉
压缩最大的风险是“省了 Token，也丢了证据”。因此 TencentDB Agent Memory 没有把历史压成一段不可恢复的 summary，而是保留了从高层摘要回到底层证据的路径。
| 问题类型 | 优先使用 | 继续下钻 |
| :--- | :--- | :--- |
| 日常偏好、表达风格、长期目标 | L3 Persona / L2 Scenario | 需要事实时查 L1 Atom / L0 Conversation |
| 具体事实、时间、项目细节 | L1 Atom / L0 Conversation | 命中不足时扩大时间范围或语义检索 |
| 当前长任务继续执行 | Active MMD 任务画布 | 摘要不够时查 JSONL，再读 `refs/*.md` 原文 |
| 历史任务恢复 | Metadata 任务入口 | 打开 MMD → 找 node_id → 追 result_ref |
上层负责“情商”和方向，下层负责“证据”和精度。短期压缩和长期记忆在这里合成一条闭环：**能折叠，也能展开；能抽象，也能追证。**
### 2. 白盒可调试：记忆不是黑盒向量
很多记忆系统的问题是：召回错了，你只能看到一串向量分数，很难判断到底哪里错。TencentDB Agent Memory 把关键中间产物保存在可读文件里：
- L2 Scenario 块是 Markdown，可以直接打开检查。
- L3 Persona 存放在 `persona.md`，可以追溯到对应的 Scenario。
- 短期任务画布是 Mermaid，既能给人看，也能给 Agent 读。
- 原文、摘要、节点之间有 `result_ref` 和 `node_id` 关联。
这意味着调试不再是翻黑盒数据库，而是沿着“Persona → Scenario → Atom → Conversation”的链路逐层定位。
**这些分层记忆产物都存放在 `~/.openclaw/memory-tdai/` 下，可以直接打开目录逐层查看。**
### 3. 工程能力完整：不是 Demo，而是可接入的插件
| 能力 | 说明 |
| :--- | :--- |
| OpenClaw 插件 | 安装后即可自动捕获、提取、召回记忆 |
| Hermes Gateway 适配 | `TdaiCore + HostAdapter` 解耦宿主框架 |
| 本地后端 | `SQLite + sqlite-vec`，开箱即用 |
| 混合检索 | BM25 + 向量 + RRF，兼顾关键词和语义召回 |
| Agent 工具 | `tdai_memory_search` / `tdai_conversation_search` |
---
## 文档
| 文档 | 内容 |
| :--- | :--- |
| [`scripts/README.memory-tencentdb-ctl.md`](./scripts/README.memory-tencentdb-ctl.md) | 运维管理工具说明 |
| [`CHANGELOG.md`](./CHANGELOG.md) | 版本变更记录 |
| [`openclaw.plugin.json`](./openclaw.plugin.json) | OpenClaw 插件声明与配置 Schema |
---
## 社区与贡献
我们欢迎一切形式的贡献——Bug 反馈、功能建议、文档勘误、Benchmark 复现、生态集成，或者一个 Pull Request 都可以。Agent 记忆这件事远未有定论，希望和大家一起把它做出来。
- 🐞 **发现 Bug 或有疑问？** 欢迎到 [GitHub Issues](https://github.com/Tencent/TencentDB-Agent-Memory/issues) 提交，我们会在 24 小时内响应。
- 💡 **有想法想交流？** 欢迎在 [GitHub Discussions](https://github.com/Tencent/TencentDB-Agent-Memory/discussions) 发起讨论。
- 🛠️ **想贡献代码？** 请先阅读 [CONTRIBUTING.md](./CONTRIBUTING_CN.md)。
- 💬 **想加入交流群？** 扫码加入 **Agent Memory 微信社群**，与早期开发者直接对话。
<p align="center"><img src="https://github.com/user-attachments/assets/53513290-2c77-45cf-b06f-b07544902bd8" width="200" alt="Agent Memory 微信社群二维码" />
---
## Roadmap
- [x] 长期个性化记忆（L0 → L3）
- [x] 短期记忆压缩（Context Offload + Mermaid 画布）
- [x] 可用本地 SQLite 后端与腾讯云向量数据库 TCVDB 后端
- [x] OpenClaw 插件与 Hermes Gateway 适配
- [ ] 记忆可迁移：跨 Agent / 跨框架 / 跨设备的导入导出与热迁移
- [ ] Skill自动生成
- [ ] 可视化调试与记忆观测面板
---
<table>
<tr>
<td width="68%">
<b>如果 TencentDB Agent Memory 对你有所帮助，欢迎为项目点亮 ⭐ 支持。</b><br />
如果有任何建议，欢迎提出issue讨论。
</td>
<td width="32%" align="right">
<img src="./assets/images/star-helper.png" alt="Star TencentDB Agent Memory" width="260" />
</td>
</tr>
</table>
[MIT](./LICENSE) © TencentDB Agent Memory Team
SKILL-DIAGNOSTIC-EXPORT.md
+152
View File
@@ -0,0 +1,152 @@
---
name: openclaw-diagnostic-export
description: 帮助用户导出 OpenClaw + memory-tencentdb（原 memory-tdai）记忆插件的现场诊断数据，用于排查问题。当用户提到"导出诊断数据""export diagnostic""现场数据""排查问题""导出日志""收集现场""打包现场数据"时应触发。
version: 1.0.0
---
## 目的
将 OpenClaw 日志、记忆插件数据（L0~L3）、脱敏后的配置打包为本地压缩包，由用户确认后手动发送给研发团队排查问题。
> **名称说明**：插件已从 `@tdai/memory-tdai` 更名为 `@tencentdb-agent-memory/memory-tencentdb`，但数据目录始终为 `~/.openclaw/memory-tdai/`（代码中硬编码）。本 skill 中所有对 `memory-tdai` 目录的引用均指实际数据目录路径，与插件 ID 无关。
## 导出工作流
### Step 1: 确认环境
在导出前，先确认 OpenClaw 工作目录存在且可访问：
```bash
# 探测工作目录（优先级：环境变量 > ~/.openclaw > ~/.clawdbot）
OPENCLAW_DIR="${OPENCLAW_STATE_DIR:-$HOME/.openclaw}"
[ -d "$OPENCLAW_DIR" ] || OPENCLAW_DIR="$HOME/.clawdbot"
ls -la "$OPENCLAW_DIR/" 2>/dev/null && echo "✅ 找到: $OPENCLAW_DIR" || echo "❌ 未找到 OpenClaw 工作目录"
```
确认 memory-tdai 子目录存在：
```bash
ls -la "$OPENCLAW_DIR/memory-tdai/" 2>/dev/null
```
### Step 2: 执行导出脚本
运行项目 `scripts/` 目录下的导出脚本：
```bash
bash scripts/export-diagnostic.sh
```
> 脚本位于本项目的 `scripts/export-diagnostic.sh`，如果通过 `pnpm` 或其他方式运行，需确保工作目录在项目根目录下。
脚本默认将压缩包输出到 `~/Downloads/openclaw-diagnostic-<timestamp>.tar.gz`。
如需指定其他输出目录：
```bash
bash scripts/export-diagnostic.sh /tmp
```
### Step 3: 确认导出结果
脚本执行完成后，检查输出：
1. **确认压缩包已生成** — 脚本末尾会打印压缩包路径和大小
2. **向用户说明包含内容**：
| 文件/目录 | 内容 | 隐私风险 |
|-----------|------|---------|
| `env-info.txt` | 系统版本、OpenClaw 版本、目录结构、磁盘占用 | 低 |
| `logs/` | OpenClaw 网关日志 + 滚动日志（最近 3 天，每文件最多 5000 行） | 低 |
| `memory-tdai/` | 记忆插件全量数据：L0 对话、L1 记忆、L2 场景、L3 画像、SQLite 数据库、checkpoint | **高** — 包含用户对话原文 |
| `openclaw-config-redacted.json` | 脱敏后的配置（已移除 API Key/Token/Password/Secret，models/channels/env 整体替换） | 低 |
| `plugins-info.txt` | 已安装插件列表和版本 | 低 |
3. **提醒用户**：
- 配置文件已自动脱敏，API Key、Token 等敏感信息已被替换为 `***REDACTED***`
- **记忆数据（memory-tdai/）包含用户对话原文**，请确认可以分享后再发送
- 压缩包存放在本地，**不会自动上传**，需要用户手动发送给研发团队
### Step 4: 告知用户后续操作
导出完成后，告知用户：
1. 压缩包已保存在本地（打印具体路径）
2. 请检查内容后，通过企微/邮件等方式手动发送给研发团队
3. 如只需部分数据（如仅日志或仅配置），可解压后选择性发送
## 导出内容详解
### OpenClaw 日志位置
| 日志类型 | 路径 | 说明 |
|---------|------|------|
| 网关 stdout | `~/.openclaw/logs/gateway.log` | 网关守护进程标准输出 |
| 网关 stderr | `~/.openclaw/logs/gateway.err.log` | 网关守护进程错误输出 |
| 滚动日志 | `/tmp/openclaw/openclaw-YYYY-MM-DD.log` | 按日期滚动，JSON Lines 格式，24h 自动清理 |
| 配置审计 | `~/.openclaw/logs/config-audit.jsonl` | 配置写入审计记录 |
| 命令日志 | `~/.openclaw/logs/commands.log` | 命令事件日志（hook 可选） |
### 记忆插件数据结构
```
~/.openclaw/memory-tdai/
├── conversations/ — L0 原始对话（每日 JSONL 分片）
├── records/ — L1 结构化记忆（每日 JSONL 分片）
├── scene_blocks/ — L2 场景 Markdown 文件
├── persona.md — L3 用户画像
├── vectors.db — SQLite 数据库（向量 + 全文索引）
├── .metadata/ — checkpoint、scene_index.json
└── .backup/ — 滚动备份
```
### 配置脱敏规则
导出脚本对 `openclaw.json` 执行以下脱敏：
| 规则 | 处理方式 |
|------|---------|
| 字段名匹配 `apiKey/token/password/secret/credential` 且值为字符串 | 替换为 `***REDACTED(Nchars)***` |
| SecretRef 对象（含 source/provider/id） | id 替换为 `***REDACTED***` |
| 顶层 `models`、`secrets`、`channels`、`env` 块 | 整体替换为 `***REDACTED_SECTION***` |
| `gateway.auth` 下的 token/password | 替换为 `***REDACTED***` |
| 其余字段（含 `plugins` 完整配置） | **保留原样**（插件配置是排查重点） |
## 手动导出（脚本不可用时的备选方案）
如果导出脚本无法执行（如 Node.js 不可用），按以下步骤手动收集：
```bash
# 1. 创建导出目录
EXPORT_DIR=~/Downloads/openclaw-diagnostic-$(date +%Y%m%d-%H%M%S)
mkdir -p "$EXPORT_DIR"
# 2. 复制日志
cp -r ~/.openclaw/logs/ "$EXPORT_DIR/logs/" 2>/dev/null
cp /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log "$EXPORT_DIR/" 2>/dev/null
# 3. 复制记忆插件数据
cp -r ~/.openclaw/memory-tdai/ "$EXPORT_DIR/memory-tdai/" 2>/dev/null
# 4. 手动脱敏配置（⚠️ 必须手动删除敏感字段！）
# 复制配置并用编辑器删除 models/secrets/channels 块和所有 apiKey/token 值
cp ~/.openclaw/openclaw.json "$EXPORT_DIR/openclaw-config-NEEDS-MANUAL-REDACTION.json"
# 5. 打包
cd ~/Downloads && tar -czf "$EXPORT_DIR.tar.gz" "$(basename $EXPORT_DIR)"
echo "⚠️ 请务必在发送前手动检查并删除配置中的敏感信息！"
```
## 常见问题排查线索
导出数据后，研发团队通常关注以下方面：
| 排查方向 | 查看文件 | 关键信息 |
|---------|---------|---------|
| 插件是否加载 | `logs/` 中搜索 `[memory-tdai]` | 插件注册、配置解析日志（注：日志标签仍为 `[memory-tdai]`，与插件 ID 无关） |
| 记忆召回是否工作 | `logs/` 中搜索 `[recall]` | 搜索策略、耗时、命中数 |
| L1 提取是否触发 | `logs/` 中搜索 `[pipeline]` | 调度触发、L1/L2/L3 执行状态 |
| 向量搜索是否可用 | `openclaw-config-redacted.json` 的 `plugins.entries` | embedding 配置是否正确 |
| 数据量/磁盘占用 | `env-info.txt` | du 输出、文件数量 |
| checkpoint 状态 | `memory-tdai/.metadata/recall_checkpoint.json` | 进度、游标、计数器 |
SKILL-MIGRATION.md
+239
View File
@@ -0,0 +1,239 @@
---
name: openclaw-memory-tencentdb-migration
description: 帮助存量用户将 OpenClaw 记忆插件从旧包 @tdai/memory-tdai 迁移到新包 @tencentdb-agent-memory/memory-tencentdb。当用户提到"插件迁移""更换记忆插件包名""memory-tdai 升级""包名变更"或出现旧包相关安装报错时应触发。
version: 1.0.0
---
## 目的
帮助已安装 `@tdai/memory-tdai`（旧包名）的存量用户，平滑迁移到 `@tencentdb-agent-memory/memory-tencentdb`（新包名），确保已有记忆数据不丢失、配置完整还原。
## 背景
- **旧包名**：`@tdai/memory-tdai`（插件 ID：`memory-tdai`）
- **新包名**：`@tencentdb-agent-memory/memory-tencentdb`（插件 ID：`memory-tencentdb`）
- 新旧插件共用相同的数据目录（`~/.openclaw/memory-tdai/`），卸载旧插件**不会删除数据目录**，已有记忆数据不受影响
- 卸载旧插件**会删除** `openclaw.json` 中该插件的配置段，需提前备份
## 适用场景
- 用户已安装 `@tdai/memory-tdai`，需迁移到新包名
- 用户执行 `openclaw plugins install @tdai/memory-tdai` 报 404 / not found
- 用户被告知旧包已废弃，需要迁移
## 不适用场景
- 用户从未安装过记忆插件（应使用 `openclaw-memory-tencentdb-setup` skill）
- 用户使用的是其他记忆插件（如 `openclaw-mem0`）
## 标准工作流
### 1) 确认当前状态
确认旧插件是否已安装：
```bash
openclaw plugins list | grep -i memory
```
预期看到 `memory-tdai` 或 `@tdai/memory-tdai` 处于 loaded 状态。
如果未看到旧插件，跳过迁移流程，直接使用 `openclaw-memory-tencentdb-setup` skill 进行全新安装。
### 2) 备份现有配置（关键步骤）
卸载旧插件会删除 `openclaw.json` 中的配置段。**必须先备份**。
执行以下命令提取旧插件配置：
```bash
cat ~/.openclaw/openclaw.json | python3 -c "
import sys, json
cfg = json.load(sys.stdin)
plugins = cfg.get('plugins', {}).get('entries', {})
old_cfg = plugins.get('memory-tdai', {})
if old_cfg:
print(json.dumps(old_cfg, indent=2, ensure_ascii=False))
with open('/tmp/memory-tdai-config-backup.json', 'w') as f:
json.dump(old_cfg, f, indent=2, ensure_ascii=False)
print('\n✅ 配置已备份到 /tmp/memory-tdai-config-backup.json')
else:
print('⚠️ 未找到 memory-tdai 配置段（可能使用默认配置）')
"
```
**特别关注以下配置是否存在（如有则必须记录）**：
- `embedding` 配置（`provider`、`baseUrl`、`apiKey`、`model`、`dimensions`、`proxyUrl`）
- `extraction.model`（提取使用的模型）
- `persona.model`（画像使用的模型）
- `capture.excludeAgents`（排除的 agent 列表）
- `capture.l0l1RetentionDays`（数据保留天数）
### 3) 确认数据目录存在
```bash
ls -la ~/.openclaw/memory-tdai/
```
预期看到：`conversations/`、`records/`、`scene_blocks/`、`vectors.db`、`persona.md` 等文件。
记录当前数据量作为迁移后验证依据：
```bash
echo "=== 迁移前数据统计 ==="
wc -l ~/.openclaw/memory-tdai/conversations/*.jsonl 2>/dev/null || echo "无对话数据"
wc -l ~/.openclaw/memory-tdai/records/*.jsonl 2>/dev/null || echo "无记录数据"
ls ~/.openclaw/memory-tdai/scene_blocks/*.md 2>/dev/null | wc -l | xargs -I{} echo "场景块: {} 个"
wc -c ~/.openclaw/memory-tdai/persona.md 2>/dev/null || echo "无 persona"
```
### 4) 卸载旧插件
```bash
openclaw plugins uninstall memory-tdai
```
执行后确认：
- `openclaw.json` 中 `memory-tdai` 配置段已被删除（预期行为）
- `~/.openclaw/memory-tdai/` 数据目录**仍然存在**（不会被删除）
```bash
# 验证数据目录仍在
ls ~/.openclaw/memory-tdai/ && echo "✅ 数据目录完好" || echo "❌ 数据目录丢失！"
```
### 5) 安装新插件
```bash
openclaw plugins install @tencentdb-agent-memory/memory-tencentdb
```
### 6) 还原配置
将步骤 2 备份的配置写回 `openclaw.json`，注意新插件的配置 key 是 `memory-tencentdb`：
```bash
python3 -c "
import json, os
# 读取备份配置
backup_path = '/tmp/memory-tdai-config-backup.json'
if os.path.exists(backup_path):
with open(backup_path) as f:
old_cfg = json.load(f)
print('📋 备份配置内容：')
print(json.dumps(old_cfg, indent=2, ensure_ascii=False))
else:
old_cfg = {'enabled': True}
print('⚠️ 未找到备份，使用最小配置')
# 读取当前 openclaw.json
config_path = os.path.expanduser('~/.openclaw/openclaw.json')
with open(config_path) as f:
cfg = json.load(f)
# 写入新插件配置
cfg.setdefault('plugins', {}).setdefault('entries', {})['memory-tencentdb'] = old_cfg
with open(config_path, 'w') as f:
json.dump(cfg, f, indent=2, ensure_ascii=False)
print('\n✅ 配置已写入 memory-tencentdb')
"
```
如果备份丢失或用户需要手动恢复，至少确保写入最小配置：
```json
{
"memory-tencentdb": {
"enabled": true
}
}
```
### 7) 重启 Gateway 并验证
```bash
openclaw gateway restart
```
检查项：
- Gateway 日志中出现 `[memory-tdai]` 前缀（注：日志标签仍为 memory-tdai，这是正常的）
- 数据目录内容未变化
```bash
echo "=== 迁移后验证 ==="
# 确认新插件已加载
openclaw plugins list | grep -i memory
# 确认数据量与迁移前一致
wc -l ~/.openclaw/memory-tdai/conversations/*.jsonl 2>/dev/null
wc -l ~/.openclaw/memory-tdai/records/*.jsonl 2>/dev/null
```
### 8) 功能冒烟验证
执行一次对话确认记忆链路正常：
1. 发送一条包含个人信息的消息（如偏好、习惯）
2. 确认日志中有 `[before_prompt_build]` 和 `[agent_end]` 相关输出
3. 如有 embedding 配置，确认向量检索正常（日志无 embedding 报错）
## 回滚方案
如迁移后出现问题，可快速回滚：
```bash
# 1. 卸载新插件
openclaw plugins uninstall memory-tencentdb
# 2. 重新安装旧插件（如 npm 源仍可用）
openclaw plugins install @tdai/memory-tdai
# 3. 手动还原配置（从备份）
# 将 /tmp/memory-tdai-config-backup.json 内容写回 openclaw.json 的 memory-tdai 段
# 4. 重启
openclaw gateway restart
```
## 故障排查
| 现象 | 可能原因 | 解决方案 |
|------|----------|----------|
| 新插件无日志输出 | 配置中 `enabled` 未设为 `true` | 检查 `openclaw.json` 中 `memory-tencentdb.enabled` |
| 安装新插件报错 | npm 源不可用 | 检查网络 / npm registry 配置 |
| 迁移后无历史记忆 | 配置还原不完整 | 对比 `/tmp/memory-tdai-config-backup.json` 与当前配置 |
| embedding 报错 | `apiKey` 等配置丢失 | 从备份中还原 `embedding` 配置段 |
| 数据目录为空 | 卸载时异常删除（极少见） | 检查 `~/.openclaw/memory-tdai/` 是否存在 |
## 安全与合规约束
- 备份文件 `/tmp/memory-tdai-config-backup.json` 可能包含 `apiKey`，迁移完成后建议删除：`rm /tmp/memory-tdai-config-backup.json`
- 不在聊天、日志中明文展示 `apiKey`
- 仅修改 `memory-tencentdb` 配置段，不影响用户其它插件
## 完成定义（Definition of Done）
迁移完成需同时满足：
- [x] 旧插件 `@tdai/memory-tdai` 已卸载
- [x] 新插件 `@tencentdb-agent-memory/memory-tencentdb` 已安装并加载
- [x] `openclaw.json` 中存在完整的 `memory-tencentdb` 配置（含用户自定义的 embedding 等配置）
- [x] Gateway 已重启
- [x] 日志中出现 `[memory-tdai]` 前缀
- [x] 数据目录完好，数据量与迁移前一致
- [x] 至少 1 次对话验证记忆链路正常
- [x] 已清理备份文件中的敏感信息
## 交付话术模板
> 已完成记忆插件迁移：
> - 旧插件 `@tdai/memory-tdai` → 新插件 `@tencentdb-agent-memory/memory-tencentdb`
> - 已有记忆数据完整保留（对话/记录/场景块/向量库均未受影响）
> - 配置已从旧插件完整还原（含 embedding / extraction / persona 等自定义配置）
> - Gateway 已重启，记忆链路验证正常
SKILL.md
+201
View File
@@ -0,0 +1,201 @@
---
name: openclaw-memory-tencentdb-setup
description: 用于在 OpenClaw 环境中安装、配置并验证 @tencentdb-agent-memory/memory-tencentdb 插件。当用户提到"安装记忆插件""配置 memory-tencentdb""开启长期记忆/召回"或出现相关报错时应触发。
version: 1.0.0
---
## 目的
在不依赖外部托管记忆服务的前提下，为 OpenClaw 提供可持续的本地长期记忆能力（L0→L1→L2→L3），并完成从安装、配置到验收的一次性闭环。
## 适用场景
- 用户要求在 OpenClaw 中安装或启用 `memory-tencentdb`
- 用户需要配置召回、提取、画像、清理等参数
- 用户反馈"插件已装但无记忆 / 无召回 / 无向量检索"
## 不适用场景
- 用户只需要解释 memory 理念，不要求实际落地
- 用户要接入非 OpenClaw 宿主（先确认目标框架）
## 标准工作流
### 1) 环境预检
先确认基础版本满足要求：
- OpenClaw: `>= 2026.3.13`
- Node.js: `>= 22.16.0`
执行：
```bash
openclaw --version
node -v
```
若版本不满足，先升级再继续。
### 2) 安装插件
执行安装命令：
```bash
openclaw plugins install @tencentdb-agent-memory/memory-tencentdb
```
如已安装则执行更新：
```bash
openclaw plugins update memory-tencentdb
```
### 3) 写入最小配置
编辑 `~/.openclaw/openclaw.json`，确保存在：
```json
{
"memory-tencentdb": {
"enabled": true
}
}
```
说明：该插件支持零配置启动；不补充其它字段也能运行基础能力。
### 4) 按需追加推荐配置（生产常用）
根据用户需求补充如下分组：
- `capture`: 对话捕获与保留策略
- `extraction`: L1 提取与去重
- `pipeline`: L1→L2→L3 调度
- `recall`: 召回数量、阈值、策略
- `persona`: 场景与画像触发参数
- `embedding`: 向量检索配置（远端 OpenAI 兼容）
推荐模板：
```json
{
"memory-tencentdb": {
"capture": {
"enabled": true,
"excludeAgents": [],
"l0l1RetentionDays": 90,
"cleanTime": "03:00"
},
"extraction": {
"enabled": true,
"enableDedup": true,
"maxMemoriesPerSession": 10,
"model": "provider/model"
},
"pipeline": {
"everyNConversations": 5,
"enableWarmup": true,
"l1IdleTimeoutSeconds": 600,
"l2DelayAfterL1Seconds": 10,
"l2MinIntervalSeconds": 900,
"l2MaxIntervalSeconds": 3600,
"sessionActiveWindowHours": 24
},
"recall": {
"enabled": true,
"maxResults": 5,
"scoreThreshold": 0.3,
"strategy": "hybrid"
},
"persona": {
"triggerEveryN": 50,
"maxScenes": 15,
"backupCount": 3,
"sceneBackupCount": 10,
"model": "provider/model"
},
"embedding": {
"enabled": true,
"provider": "openai",
"baseUrl": "https://api.openai.com/v1",
"apiKey": "${EMBEDDING_API_KEY}",
"model": "text-embedding-3-small",
"dimensions": 1536,
"conflictRecallTopK": 5
}
}
}
```
### 5) 关键配置规则（避免隐性失败）
- `embedding.provider = "none"` 时，向量能力会禁用，仅保留关键词路径。
- 若配置远端 `provider`（如 `openai` / `deepseek`），必须同时提供：
- `apiKey`
- `baseUrl`
- `model`
- `dimensions`
- 上述任一缺失时，插件会继续运行，但自动降级为非向量模式。
- `l0l1RetentionDays`：
- `0` 表示不清理
- 非 `0` 时建议 `>=3`
- 若设为 `1~2`，需显式开启 `allowAggressiveCleanup`
### 6) 重启并验证生效
执行：
```bash
openclaw gateway restart
```
检查项：
- Gateway 日志中出现 `[memory-tdai]` 前缀
- 数据目录已创建：`~/.openclaw/state/memory-tdai/`
- 至少包含：`conversations/`、`records/`、`scene_blocks/`、`vectors.db`
### 7) 功能冒烟测试
执行一次最小对话回路并验证：
1. 连续对话 2~3 轮，提供可记忆信息（偏好、约束、背景）。
2. 发起新一轮对话，观察是否出现召回上下文注入。
3. 在 Agent 中调用：
- `tdai_memory_search`
- `tdai_conversation_search`
4. 确认能检索到刚刚产生的内容。
## 故障排查速查
- 插件无日志：检查 `openclaw.json` 中 `memory-tencentdb.enabled` 是否为 `true`，并确认已重启 Gateway。
- 有记录无召回：检查 `recall.enabled`、`scoreThreshold` 是否过高。
- 无向量结果：检查 `embedding` 四元组（`apiKey/baseUrl/model/dimensions`）是否齐全。
- 清理过猛导致历史过少：检查 `l0l1RetentionDays` 与 `allowAggressiveCleanup`。
- 配置已改但行为不变：确认修改的是 `~/.openclaw/openclaw.json`，并再次重启 Gateway。
## 安全与合规约束
- 将 `apiKey` 视为敏感信息；不在聊天、日志、截图中明文扩散。
- 优先使用环境变量注入密钥；配置示例中仅保留占位符。
- 仅修改 `memory-tencentdb` 对应配置段，避免覆盖用户其它插件配置。
## 完成定义（Definition of Done）
在结束任务前，必须同时满足：
- 插件安装/更新命令执行成功
- `openclaw.json` 已存在有效 `memory-tencentdb` 配置
- Gateway 已重启
- `[memory-tdai]` 日志可见
- 数据目录与关键文件已生成
- 至少 1 次检索工具调用成功返回结果
## 交付话术模板
可在完成后向用户输出：
- 已完成 `memory-tencentdb` 安装与配置，并重启 Gateway。
- 已验证日志与数据目录生效，记忆链路可用。
- 如需下一步优化，可继续调优 `recall.scoreThreshold`、`pipeline.everyNConversations`、`persona.triggerEveryN` 与 `embedding` 模型参数。
assets/images/flowchart1.cn.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 27 KiB

assets/images/flowchart1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

assets/images/flowchart2.cn.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 27 KiB

assets/images/flowchart2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

assets/images/logo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 129 KiB

assets/images/logo2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 212 KiB

assets/images/memory-pyramid-en.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 664 KiB

assets/images/memory-pyramid.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 611 KiB

assets/images/star-helper.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 201 KiB

bin/export-tencent-vdb.mjs
Executable
+20
View File
@@ -0,0 +1,20 @@
#!/usr/bin/env node
// 薄启动器：加载预编译好的 VDB 导出脚本。
// 构建：npm run build:export-vdb
// 使用：npm run export:vdb -- [参数] 或 node ./bin/export-tencent-vdb.mjs [参数]
import path from "node:path";
import { fileURLToPath } from "node:url";
import fs from "node:fs";
const thisDir = path.dirname(fileURLToPath(import.meta.url));
const entryScript = path.resolve(thisDir, "../scripts/export-tencent-vdb/dist/export-tencent-vdb.js");
if (!fs.existsSync(entryScript)) {
console.error("❌ 预编译产物不存在: " + entryScript);
console.error(" 请先执行: npm run build:export-tencent-vdb");
process.exit(1);
}
import(entryScript);
bin/migrate-sqlite-to-tcvdb.mjs
Executable
+12
View File
@@ -0,0 +1,12 @@
#!/usr/bin/env node
// Thin wrapper: runs the pre-compiled migration CLI entry.
// Build first: npm run build:migrate-sqlite-to-vdb (or pnpm build:migrate-sqlite-to-vdb)
import path from "node:path";
import { fileURLToPath } from "node:url";
const thisDir = path.dirname(fileURLToPath(import.meta.url));
const entryScript = path.resolve(thisDir, "../scripts/migrate-sqlite-to-tcvdb/dist/scripts/migrate-sqlite-to-tcvdb/cli-entry.js");
import(entryScript);
bin/read-local-memory.mjs
Executable
+21
View File
@@ -0,0 +1,21 @@
#!/usr/bin/env node
// 薄启动器：加载预编译好的本地 Memory 数据查询脚本。
// 构建：npm run build:read-local-memory
// 使用：npm run read-local-memory -- [参数] 或 node ./bin/read-local-memory.mjs [参数]
import path from "node:path";
import { fileURLToPath } from "node:url";
import fs from "node:fs";
const thisDir = path.dirname(fileURLToPath(import.meta.url));
const entryScript = path.resolve(thisDir, "../scripts/read-local-memory/dist/read-local-memory.js");
if (!fs.existsSync(entryScript)) {
console.error("❌ 预编译产物不存在: " + entryScript);
console.error(" 请先执行: npm run build:read-local-memory");
process.exit(1);
}
import(entryScript);
bin/seed-v2.mjs
Executable
+38
View File
@@ -0,0 +1,38 @@
#!/usr/bin/env node
// 薄启动器：seed-v2 通过 v2 API 把历史对话灌入 memory-tencentdb gateway。
//
// 优先用预编译产物（生产场景），找不到时 fallback 到 tsx 跑源码（开发场景）。
//
// 构建：npm run build:seed-v2
// 使用：
// npm run seed-v2 -- --input ./scripts/seed-v2/fixtures/minimal.json
// node ./bin/seed-v2.mjs --input fixture.json --endpoint http://127.0.0.1:18420
import path from "node:path";
import fs from "node:fs";
import { fileURLToPath, pathToFileURL } from "node:url";
import { spawnSync } from "node:child_process";
const thisDir = path.dirname(fileURLToPath(import.meta.url));
const distEntry = path.resolve(thisDir, "../scripts/seed-v2/dist/seed-v2.js");
const srcEntry = path.resolve(thisDir, "../scripts/seed-v2/seed-v2.ts");
if (fs.existsSync(distEntry)) {
// 预编译产物存在：直接 dynamic import
await import(pathToFileURL(distEntry).href);
} else if (fs.existsSync(srcEntry)) {
// 没编译过：fallback 到 tsx（开发期常见）
const result = spawnSync(
process.execPath,
["--import", "tsx", srcEntry, ...process.argv.slice(2)],
{ stdio: "inherit" },
);
process.exit(result.status ?? 1);
} else {
console.error("❌ neither dist nor source found:");
console.error(" " + distEntry);
console.error(" " + srcEntry);
console.error(" run: npm run build:seed-v2");
process.exit(1);
}
hermes-plugin/memory/memory_tencentdb/README.md
+299
View File
@@ -0,0 +1,299 @@
# memory-tencentdb Memory Provider (Hermes)
Hermes-side [`MemoryProvider`](../../../../../hermes-agent/agent/memory_provider.py)
adapter for the **memory-tencentdb** four-layer memory system
(L0 conversation capture → L1 episodic extraction → L2 scene blocks → L3 persona synthesis).
The heavy lifting — capture, extraction, storage, recall, pipeline scheduling —
runs in a Node.js **Gateway** sidecar (shipped by the same package as the
OpenClaw plugin). This Python provider is a thin HTTP client + process
supervisor that plugs the Gateway into Hermes's lifecycle.
## Architecture
```
Hermes Agent (Python)
└─ MemoryManager
└─ MemoryTencentdbProvider (this directory)
├─ GatewaySupervisor — starts / health-checks the sidecar
└─ MemoryTencentdbSdkClient — POST /recall, /capture, /search/*, /session/end
│
▼ HTTP (127.0.0.1:8420 by default)
memory-tencentdb Gateway (Node.js)
└─ memory-tencentdb Core
├─ L0 Conversation store (SQLite / TCVDB + JSONL)
├─ L1 Episodic extraction (LLM + vector dedup)
├─ L2 Scene blocks (Markdown under data dir)
├─ L3 Persona synthesis (persona.md)
└─ Storage backends: SQLite + sqlite-vec OR Tencent VectorDB
```
Hermes lifecycle → Gateway mapping:
| Hermes hook / call | Gateway endpoint | Behavior |
|-----------------------------|------------------|------------------------------------------------------------|
| `prefetch(query)` | `POST /recall` | Synchronous. Returns `<memory-context>` text for injection |
| `sync_turn(user, assistant)`| `POST /capture` | Fire-and-forget on a background daemon thread (max 4 in-flight) |
| `shutdown()` / `on_session_end` | `POST /session/end` | Flush pending pipeline work |
| `get_tool_schemas()` | — | Advertises two LLM tools (see below) |
Reliability features baked into the provider:
- **Circuit breaker** — 5 consecutive Gateway failures → pause all calls for 60 s.
- **Back-pressure on capture** — at most 4 in-flight `sync_turn` threads; a 5th
waits up to 5 s for the oldest one before starting (Gateway hangs can't grow
threads unboundedly).
- **Supervised startup** — if `MEMORY_TENCENTDB_GATEWAY_CMD` is set (or the
provider auto-discovers `src/gateway/server.ts`, see below), it starts the
sidecar, polls `/health` for up to 30 s, and tails `gateway.stderr.log` on
crash for diagnostics.
- **Zero-config auto-discovery** — when `MEMORY_TENCENTDB_GATEWAY_CMD` is
unset, the provider looks for `src/gateway/server.ts` next to the plugin
checkout (in-tree) and, as a last resort, under
`~/.memory-tencentdb/tdai-memory-openclaw-plugin/` (preferred),
`~/tdai-memory-openclaw-plugin/` (legacy), and
`~/.hermes/plugins/tdai-memory-openclaw-plugin/`.
A fresh `git clone` therefore usually works without any extra env wiring —
override with the env var when you need a non-standard layout.
## Installation Location
This directory (`hermes-plugin/memory/memory_tencentdb/`) is the **source of
truth** for the provider; Hermes does **not** load it from here. At startup
Hermes scans two locations for memory providers, in precedence order (see
`hermes-agent/plugins/memory/__init__.py`):
1. **Bundled** — `<hermes-agent-checkout>/plugins/memory/<name>/`
**This is the path memory_tencentdb ships under.** It sits alongside the
other in-tree providers (`byterover/`, `honcho/`, `mem0/`, `hindsight/`,
…). Bundled entries take precedence over user-installed ones on name
collision.
2. **User-installed** — `$HERMES_HOME/plugins/<name>/`, where
`$HERMES_HOME` defaults to `~/.hermes` (see
`hermes_constants.get_hermes_home()`). This path is for third-party
providers; we don't use it for memory_tencentdb.
**The trailing directory name must be exactly `memory_tencentdb`** — Hermes
uses that directory name as the provider key; it must match
`plugin.yaml::name` and the value of `memory.provider` in `config.yaml`.
(The hyphenated form `memory-tencentdb` is a *config-side alias*, not a
valid directory name.)
Pick one of the two installation styles:
**Install A — symlink (recommended for developers working on both repos
simultaneously):** keeps this repo as the single source of truth so
`git pull` in the plugin repo is immediately visible to Hermes.
```bash
# from the tdai-memory-openclaw-plugin checkout:
ln -s "$(pwd)/hermes-plugin/memory/memory_tencentdb" \
<hermes-agent-checkout>/plugins/memory/memory_tencentdb
```
**Install B — copy (shipped alongside hermes-agent):** freezes a specific
version of the provider inside the hermes-agent tree. This is how
memory_tencentdb is currently vendored in this repo pair — the two copies
under `tdai-memory-openclaw-plugin/hermes-plugin/memory/memory_tencentdb/`
and `hermes-agent/plugins/memory/memory_tencentdb/` are kept in sync
manually.
```bash
cp -r tdai-memory-openclaw-plugin/hermes-plugin/memory/memory_tencentdb \
hermes-agent/plugins/memory/memory_tencentdb
```
Verify Hermes sees the provider:
```bash
$ cd <hermes-agent-checkout>
$ python -c 'from plugins.memory import discover_memory_providers; \
[print(n, a) for n, _, a in discover_memory_providers()]'
memory_tencentdb True
...
```
If the provider does not appear:
- confirm the target path is `hermes-agent/plugins/memory/memory_tencentdb/`
(underscore, not hyphen);
- confirm `__init__.py` and `plugin.yaml` sit directly inside that dir;
- the discovery scan requires `__init__.py` to contain the literal string
`MemoryProvider` or `register_memory_provider` — both are present in
this provider, so this is a non-issue as long as the file is the one
from this repo.
> The **Gateway source code** (Node.js sidecar under `src/gateway/`) stays
> in the `tdai-memory-openclaw-plugin` checkout and does NOT need to be
> copied into hermes-agent — the Python provider auto-discovers it via
> the paths listed in Option A below, or via `MEMORY_TENCENTDB_GATEWAY_CMD`.
## Setup
### 1. Activate in Hermes (`~/.hermes/config.yaml`)
```yaml
memory:
provider: memory_tencentdb # canonical name
# Aliases accepted for backward compatibility: `memory-tencentdb`, `tdai`
```
### 2. Provide Gateway runtime + LLM credentials
At minimum the Gateway needs an OpenAI-compatible endpoint for L1/L2/L3
extraction. Set these in the Hermes process environment:
```bash
export MEMORY_TENCENTDB_LLM_API_KEY="sk-..."
export MEMORY_TENCENTDB_LLM_BASE_URL="https://api.openai.com/v1" # optional
export MEMORY_TENCENTDB_LLM_MODEL="gpt-4o" # optional
```
### 3. Start the Gateway
You have three options; pick whichever fits your deployment.
**Option A — Auto-discovery (zero-config).** If the plugin checkout sits at
one of the well-known paths, the provider will find `src/gateway/server.ts`
on its own and `Popen()` it as `node --import tsx <path>`. Searched paths, in
order:
1. In-tree: `<plugin-root>/src/gateway/server.ts` (when Hermes loads this
provider from a checkout of this repo).
2. `~/.memory-tencentdb/tdai-memory-openclaw-plugin/src/gateway/server.ts` (preferred install location)
3. `~/tdai-memory-openclaw-plugin/src/gateway/server.ts` (legacy)
4. `~/.hermes/plugins/tdai-memory-openclaw-plugin/src/gateway/server.ts`
No environment variables required beyond the LLM credentials above. A line
like
```
INFO plugins.memory.memory_tencentdb: memory-tencentdb Gateway command auto-discovered: /…/src/gateway/server.ts
```
will appear in `~/.hermes/logs/agent.log` on startup.
**Option B — Explicit auto-start.** Override or disable discovery by setting
the command yourself:
```bash
export MEMORY_TENCENTDB_GATEWAY_CMD="node --import tsx /abs/path/to/tdai-memory-openclaw-plugin/src/gateway/server.ts"
```
The provider will `Popen()` this command on `initialize()`, wait for
`GET /health` to report `ok`/`degraded`, and tail stderr on crash.
**Option C — Run it yourself.** Start the Gateway separately on the default
port (`127.0.0.1:8420`) before launching Hermes; the provider will detect it
via `/health` and skip the subprocess-launch path.
```bash
cd tdai-memory-openclaw-plugin
node --import tsx src/gateway/server.ts
```
> Storage backend (SQLite vs Tencent VectorDB), embedding config, pipeline
> cadence, recall strategy, etc. are all **Gateway-side** settings. For
> OpenClaw installs they live in `~/.openclaw/openclaw.json`; for standalone
> Hermes deployments configure the Gateway via its own config file or env.
> See the plugin's top-level [README](../../../README.md) for the full
> configuration schema.
## Environment Variables
### Gateway location
| Variable | Default | Description |
|-----------------------------------|---------------------|-----------------------------------------------------------|
| `MEMORY_TENCENTDB_GATEWAY_HOST` | `127.0.0.1` | Gateway host |
| `MEMORY_TENCENTDB_GATEWAY_PORT` | `8420` | Gateway port (must be 1..65535; invalid values fall back) |
| `MEMORY_TENCENTDB_GATEWAY_CMD` | — | If set, the provider auto-starts the Gateway with this command. If unset, the provider auto-discovers `src/gateway/server.ts` next to the checkout or under `$HOME` (see Option A above) |
| `MEMORY_TENCENTDB_LOG_DIR` | `~/.hermes/logs/memory_tencentdb` | Where the supervisor writes `gateway.stdout.log` / `gateway.stderr.log` |
### Gateway data directory (owned by the Gateway, not this provider)
The L0~L3 data directory is resolved **inside the Gateway** (`src/gateway/config.ts`),
not here. Priority:
1. `TDAI_DATA_DIR` env var
2. `data.baseDir` from a `tdai-gateway.yaml` / `tdai-gateway.json` config file
3. Default: `~/.memory-tencentdb/memory-tdai`
(Override the parent dir with `MEMORY_TENCENTDB_ROOT` if needed.)
4. Legacy fallback: if `~/.memory-tencentdb/memory-tdai` does not exist but the
pre-0.4 location `~/memory-tdai` does, the Gateway keeps using the legacy
dir and prints a one-line deprecation warning to stderr. Run
`install_hermes_memory_tencentdb.sh` to migrate it automatically.
Hermes forwards the inherited environment to the Gateway subprocess, so
setting `TDAI_DATA_DIR` before launching Hermes is enough to override it.
The old `MEMORY_TENCENTDB_DATA_DIR` env var is no longer read — it was never
consumed by the Gateway anyway (names did not match), so removing it just
eliminates a silent no-op.
### Gateway LLM (consumed by the Node sidecar, not by this provider)
| Variable | Default | Description |
|-----------------------------------|------------------------------|-------------------------------------|
| `MEMORY_TENCENTDB_LLM_API_KEY` | — | LLM API key (required for L1/L2/L3) |
| `MEMORY_TENCENTDB_LLM_BASE_URL` | `https://api.openai.com/v1` | OpenAI-compatible API base URL |
| `MEMORY_TENCENTDB_LLM_MODEL` | `gpt-4o` | Model name |
> ⚠️ Only `MEMORY_TENCENTDB_*` env vars are honored by this provider for the
> Gateway location and LLM credentials. Data-directory resolution is
> deliberately delegated to the Gateway via `TDAI_DATA_DIR` (see above) so
> the provider and the Gateway can never disagree about where L0~L3 live.
## LLM Tools
This provider exposes two tools to the model via `get_tool_schemas()`:
| Tool | Purpose | Args |
|----------------------------------------|---------------------------------------------------|-------------------------------------------------|
| `memory_tencentdb_memory_search` | Search L1 structured long-term memories | `query` (required), `limit` (1..20, default 5), `type` (`persona`/`episodic`/`instruction`) |
| `memory_tencentdb_conversation_search` | Search L0 raw conversation history | `query` (required), `limit` (1..20, default 5) |
Tool-call arguments are defensively coerced: `limit` accepts ints, numeric
strings, and floats, rejects bools, and is clamped to `[1, 20]` with a
warning on garbage input.
> These are the **only** tool names registered with the LLM. The old
> `tdai_memory_search` / `tdai_conversation_search` names are not served by
> this provider — if older transcripts reference them, `handle_tool_call`
> will return an "Unknown tool" error.
## Plugin Metadata (`plugin.yaml`)
```yaml
name: memory_tencentdb # canonical provider name
display_name: memory-tencentdb
hooks:
- on_memory_write # reserved; not yet mirrored to the Gateway
- on_session_end # triggers POST /session/end
aliases:
- tdai # legacy config value still resolves here
- memory-tencentdb # hyphenated form resolves here too
```
## Troubleshooting
- **"memory-tencentdb Gateway not available"** on startup: either
`MEMORY_TENCENTDB_GATEWAY_CMD` is unset *and* auto-discovery did not find
`src/gateway/server.ts` *and* nothing is listening on `8420`, or the
sidecar crashed. Check `~/.hermes/logs/memory_tencentdb/gateway.stderr.log`
(override with `MEMORY_TENCENTDB_LOG_DIR`). To confirm auto-discovery was
attempted, enable `DEBUG` logging and look for
`memory-tencentdb Gateway auto-discovery found no server.ts under: …`;
that log line enumerates every path that was searched.
- **Gateway starts from the wrong checkout**: auto-discovery walks a fixed
preference list (in-tree first, then `$HOME`). If you want to pin a
specific path, set `MEMORY_TENCENTDB_GATEWAY_CMD` explicitly — it always
wins over discovery.
- **Search tools silently missing from the LLM**: `get_tool_schemas()`
returns `[]` until either the Gateway is reachable or one of
`MEMORY_TENCENTDB_GATEWAY_CMD` / `MEMORY_TENCENTDB_GATEWAY_PORT` is set
in the environment. Set the env var so the tools are advertised
optimistically at registration time.
- **"circuit breaker tripped"** warnings: five consecutive Gateway errors
were observed. Calls are paused for 60 s; check Gateway health and logs.
- **Capture backlog warnings**: Gateway is slow or hung — `sync_turn` is
tracking ≥ 4 in-flight threads. Inspect Gateway logs for stuck L1
extractions or LLM timeouts.
hermes-plugin/memory/memory_tencentdb/__init__.py
+1130
View File
File diff suppressed because it is too large Load Diff
hermes-plugin/memory/memory_tencentdb/client.py
+196
View File
@@ -0,0 +1,196 @@
"""MemoryTencentdbSdkClient — HTTP client for the memory-tencentdb Gateway.
Wraps all Gateway API endpoints with timeout, retry, and error handling.
Thread-safe — can be shared across prefetch/sync threads.
"""
from __future__ import annotations
import json
import logging
import urllib.request
import urllib.error
from typing import Any, Dict, Optional
logger = logging.getLogger(__name__)
DEFAULT_TIMEOUT = 10 # seconds
class MemoryTencentdbSdkClient:
"""HTTP client for the memory-tencentdb Gateway sidecar."""
def __init__(
self,
base_url: str = "http://127.0.0.1:8420",
timeout: int = DEFAULT_TIMEOUT,
api_key: Optional[str] = None,
):
"""Construct the client.
Args:
base_url: Gateway base URL.
timeout: Default request timeout in seconds.
api_key: Optional Bearer token. When non-empty, every request
attaches ``Authorization: Bearer <api_key>``. When ``None``
or empty, no auth header is sent — this preserves the
pre-existing open-Gateway behaviour and is the right default
for any deployment where the Gateway has not opted into
``TDAI_GATEWAY_API_KEY`` yet.
The provider sources this value from
``MEMORY_TENCENTDB_GATEWAY_API_KEY`` (with
``TDAI_GATEWAY_API_KEY`` as a fallback). The Gateway must
be configured with the matching secret independently —
this client does not (and should not) propagate the value
across to the Gateway process.
"""
self._base_url = base_url.rstrip("/")
self._timeout = timeout
# Strip whitespace defensively — env vars often pick up trailing
# newlines from `echo` or YAML quoting; an exact-match Bearer
# comparison would otherwise reject a key that "looks right".
self._api_key = (api_key or "").strip() or None
def _build_headers(self, *, content_type: bool) -> Dict[str, str]:
"""Build request headers, conditionally adding Authorization.
Centralised so the auth header logic is stated once: every method
below goes through ``_post`` / ``_get`` which call this helper. If
you ever add a new HTTP verb, route it here.
"""
headers: Dict[str, str] = {}
if content_type:
headers["Content-Type"] = "application/json"
if self._api_key:
headers["Authorization"] = f"Bearer {self._api_key}"
return headers
def _post(self, path: str, body: Dict[str, Any], timeout: Optional[int] = None) -> Dict[str, Any]:
"""Make a POST request to the Gateway."""
url = f"{self._base_url}{path}"
data = json.dumps(body).encode("utf-8")
req = urllib.request.Request(
url,
data=data,
headers=self._build_headers(content_type=True),
method="POST",
)
try:
with urllib.request.urlopen(req, timeout=timeout or self._timeout) as resp:
return json.loads(resp.read().decode("utf-8"))
except urllib.error.HTTPError as e:
body_text = ""
try:
body_text = e.read().decode("utf-8", errors="replace")
except Exception:
pass
logger.warning("memory-tencentdb Gateway %s returned %d: %s", path, e.code, body_text[:500])
raise
except Exception as e:
logger.debug("memory-tencentdb Gateway %s failed: %s", path, e)
raise
def _get(self, path: str, timeout: Optional[int] = None) -> Dict[str, Any]:
"""Make a GET request to the Gateway."""
url = f"{self._base_url}{path}"
req = urllib.request.Request(
url,
headers=self._build_headers(content_type=False),
method="GET",
)
try:
with urllib.request.urlopen(req, timeout=timeout or self._timeout) as resp:
return json.loads(resp.read().decode("utf-8"))
except Exception as e:
logger.debug("memory-tencentdb Gateway GET %s failed: %s", path, e)
raise
# -- API methods ----------------------------------------------------------
def health(self, timeout: int = 3) -> Dict[str, Any]:
"""Check if the Gateway is healthy."""
return self._get("/health", timeout=timeout)
def recall(self, query: str, session_key: str, user_id: str = "") -> Dict[str, Any]:
"""Recall memories for a query (prefetch)."""
body: Dict[str, Any] = {"query": query, "session_key": session_key}
if user_id:
body["user_id"] = user_id
return self._post("/recall", body)
def capture(
self,
user_content: str,
assistant_content: str,
session_key: str,
session_id: str = "",
user_id: str = "",
) -> Dict[str, Any]:
"""Capture a conversation turn (sync_turn)."""
body: Dict[str, Any] = {
"user_content": user_content,
"assistant_content": assistant_content,
"session_key": session_key,
}
if session_id:
body["session_id"] = session_id
if user_id:
body["user_id"] = user_id
return self._post("/capture", body)
def search_memories(self, query: str, limit: int = 5, type_filter: str = "", scene: str = "") -> Dict[str, Any]:
"""Search L1 structured memories."""
body: Dict[str, Any] = {"query": query, "limit": limit}
if type_filter:
body["type"] = type_filter
if scene:
body["scene"] = scene
return self._post("/search/memories", body)
def search_conversations(self, query: str, limit: int = 5, session_key: str = "") -> Dict[str, Any]:
"""Search L0 raw conversations."""
body: Dict[str, Any] = {"query": query, "limit": limit}
if session_key:
body["session_key"] = session_key
return self._post("/search/conversations", body)
def end_session(self, session_key: str, user_id: str = "") -> Dict[str, Any]:
"""End a session and trigger flush."""
body: Dict[str, Any] = {"session_key": session_key}
if user_id:
body["user_id"] = user_id
return self._post("/session/end", body)
def seed(
self,
data: Any,
session_key: str = "",
strict_round_role: bool = False,
auto_fill_timestamps: bool = True,
config_override: Optional[Dict[str, Any]] = None,
timeout: int = 300,
) -> Dict[str, Any]:
"""Batch seed historical conversations into the memory pipeline.
Args:
data: Seed input — Format A ``{"sessions": [...]}`` or Format B ``[...]``.
session_key: Fallback session key when input sessions lack one.
strict_round_role: Require each round to have both user and assistant.
auto_fill_timestamps: Auto-fill missing timestamps (default True).
config_override: Plugin config overrides (deep-merged).
timeout: Request timeout in seconds (seed can be slow, default 300s).
Returns:
Summary dict with sessions_processed, rounds_processed, etc.
"""
body: Dict[str, Any] = {"data": data}
if session_key:
body["session_key"] = session_key
if strict_round_role:
body["strict_round_role"] = True
if not auto_fill_timestamps:
body["auto_fill_timestamps"] = False
if config_override:
body["config_override"] = config_override
return self._post("/seed", body, timeout=timeout)
hermes-plugin/memory/memory_tencentdb/plugin.yaml
+12
View File
@@ -0,0 +1,12 @@
name: memory_tencentdb
display_name: memory-tencentdb
version: 1.0.0
description: "memory-tencentdb four-layer memory — L0 conversation recording, L1 episodic extraction, L2 scene blocks, L3 persona synthesis via local Node.js Gateway."
hooks:
- on_memory_write
- on_session_end
# Legacy provider name — kept so that users whose config still says
# `memory.provider: tdai` continue to resolve to this provider.
aliases:
- tdai
- memory-tencentdb
hermes-plugin/memory/memory_tencentdb/supervisor.py
+329
View File
@@ -0,0 +1,329 @@
"""GatewaySupervisor — manages the memory-tencentdb Gateway Node.js sidecar process.
On initialize(), checks if the Gateway is already running. If not, starts
it as a subprocess and waits for /health to become available.
On shutdown(), sends a flush signal and waits for clean exit.
"""
from __future__ import annotations
import logging
import os
import shlex
import subprocess
import time
from typing import IO, Optional
from .client import MemoryTencentdbSdkClient
logger = logging.getLogger(__name__)
# Default Gateway address
DEFAULT_HOST = "127.0.0.1"
DEFAULT_PORT = 8420
# Health check parameters
HEALTH_CHECK_INTERVAL = 0.5 # seconds between checks
HEALTH_CHECK_MAX_WAIT = 30 # max seconds to wait for Gateway to start
HEALTH_CHECK_RETRIES = 3 # retries for is_running check
# Log file rotation parameters
LOG_TAIL_BYTES_ON_CRASH = 2048 # bytes of stderr log to surface on startup crash
class GatewaySupervisor:
"""Manages the memory-tencentdb Gateway sidecar lifecycle."""
def __init__(
self,
host: str = DEFAULT_HOST,
port: int = DEFAULT_PORT,
gateway_cmd: Optional[str] = None,
api_key: Optional[str] = None,
):
"""Construct the supervisor.
Args:
host: Gateway bind host.
port: Gateway bind port.
gateway_cmd: Shell command to spawn the Gateway. Falls back to
``MEMORY_TENCENTDB_GATEWAY_CMD`` env var when None.
api_key: Optional Gateway Bearer token used by the **client**
(every outbound request adds ``Authorization: Bearer <key>``).
The supervisor does NOT propagate this value to the spawned
Gateway's environment — turning auth on at the Gateway is the
operator's responsibility (set ``TDAI_GATEWAY_API_KEY`` /
``server.apiKey`` on the Gateway side directly, in the same
place you'd configure its port and data dir). Both ends must
see the same secret; the plugin only handles the client half.
``None`` / empty means "do not attach an Authorization
header", which preserves the legacy default.
"""
self._host = host
self._port = port
self._base_url = f"http://{host}:{port}"
self._api_key = (api_key or "").strip() or None
self._client = MemoryTencentdbSdkClient(
base_url=self._base_url,
timeout=5,
api_key=self._api_key,
)
self._process: Optional[subprocess.Popen] = None
# File handles for child's stdout/stderr. Kept open for the lifetime of
# the process so the kernel pipe buffer never fills up (otherwise the
# Gateway's event loop would block on write() after ~64 KB of logs).
self._stdout_log: Optional[IO[bytes]] = None
self._stderr_log: Optional[IO[bytes]] = None
self._stderr_log_path: Optional[str] = None
# Resolve Gateway command
# Priority: explicit arg > MEMORY_TENCENTDB_GATEWAY_CMD env
self._gateway_cmd = gateway_cmd or os.environ.get("MEMORY_TENCENTDB_GATEWAY_CMD", "")
def is_running(self) -> bool:
"""Check if the Gateway is currently responding to health checks."""
for _ in range(HEALTH_CHECK_RETRIES):
try:
result = self._client.health(timeout=2)
return result.get("status") in ("ok", "degraded")
except Exception:
time.sleep(0.2)
return False
def is_process_alive(self) -> bool:
"""Return True iff we have spawned a child and it has not exited.
Distinct from ``is_running()``:
* ``is_running`` performs a network health check — slow, but works
even when the Gateway was started externally (systemd, manual run).
* ``is_process_alive`` only inspects our own ``Popen`` handle — fast,
and lets the watchdog notice an exited child without paying for an
HTTP round-trip every tick.
Returns False when we never spawned a child, or when the child has
exited (``poll()`` returns a non-None code). The watchdog combines
both checks: ``is_process_alive() or is_running()`` — only when both
say "no" do we attempt a re-spawn.
"""
proc = self._process
if proc is None:
return False
return proc.poll() is None
def _reap_dead_process(self) -> None:
"""Drop the reference to a child we spawned that has since exited.
Called from ``ensure_running`` so that a re-spawn after a crash does
not leak the previous ``Popen`` handle (the kernel still owns the
zombie until ``wait()``-style call). Safe to call when the process
is still alive — it's a no-op in that case.
"""
proc = self._process
if proc is None:
return
if proc.poll() is None:
return # still alive
try:
# poll() already reaped the child via waitpid internally on POSIX,
# so there is nothing more to do here. Just drop our handle and
# close the log files we opened for this run.
rc = proc.returncode
logger.warning(
"memory-tencentdb Gateway: previous child exited (code=%s); "
"reaping before respawn.", rc,
)
finally:
self._process = None
self._close_log_handles()
def ensure_running(self) -> bool:
"""Ensure the Gateway is running. Start it if not.
Returns True if the Gateway is available, False if startup failed.
"""
if self.is_running():
logger.info("memory-tencentdb Gateway already running at %s", self._base_url)
return True
# If we previously spawned a child and it has since died, drop the
# stale Popen handle so the new spawn below isn't shadowed by a
# zombie reference. Without this, a crashed-then-respawned Gateway
# would keep ``self._process`` pointing at the dead PID forever and
# ``is_process_alive()`` would mislead the watchdog.
self._reap_dead_process()
# Try to start the Gateway
if not self._gateway_cmd:
logger.warning(
"memory-tencentdb Gateway is not running and no gateway command configured. "
"Set MEMORY_TENCENTDB_GATEWAY_CMD environment variable or pass gateway_cmd to supervisor. "
"memory-tencentdb memory will be unavailable."
)
return False
logger.info("Starting memory-tencentdb Gateway: %s", self._gateway_cmd)
try:
env = os.environ.copy()
env["MEMORY_TENCENTDB_GATEWAY_PORT"] = str(self._port)
env["MEMORY_TENCENTDB_GATEWAY_HOST"] = self._host
# Note: we deliberately do NOT inject TDAI_GATEWAY_API_KEY into
# the child's env from here. Whether the Gateway enforces auth is
# the operator's call — they configure it on the Gateway side
# (env, yaml, docker run, systemd unit) just like any other
# Gateway setting. The supervisor's ``api_key`` is purely the
# client-side Bearer token used for outbound requests.
# Redirect child stdout/stderr to log files instead of PIPE.
# Using PIPE without an active reader will deadlock the child once
# the pipe buffer (~64 KB) fills up. A log directory next to the
# data dir keeps logs inspectable on crash while eliminating the
# blocking risk entirely.
log_dir = self._resolve_log_dir()
try:
os.makedirs(log_dir, exist_ok=True)
except OSError as e:
logger.warning(
"memory-tencentdb Gateway: failed to create log dir %s (%s); "
"falling back to DEVNULL", log_dir, e,
)
log_dir = None
if log_dir is not None:
stdout_path = os.path.join(log_dir, "gateway.stdout.log")
stderr_path = os.path.join(log_dir, "gateway.stderr.log")
# Append mode: preserve previous runs for postmortem.
self._stdout_log = open(stdout_path, "ab", buffering=0)
self._stderr_log = open(stderr_path, "ab", buffering=0)
self._stderr_log_path = stderr_path
stdout_target: object = self._stdout_log
stderr_target: object = self._stderr_log
else:
stdout_target = subprocess.DEVNULL
stderr_target = subprocess.DEVNULL
self._process = subprocess.Popen(
shlex.split(self._gateway_cmd),
env=env,
stdout=stdout_target,
stderr=stderr_target,
start_new_session=True, # Detach from parent process group
)
except Exception as e:
logger.error("Failed to start memory-tencentdb Gateway: %s", e)
self._close_log_handles()
return False
# Wait for health check
return self._wait_for_health()
def _resolve_log_dir(self) -> str:
"""Pick a directory to store Gateway stdout/stderr logs.
Priority:
1. ``MEMORY_TENCENTDB_LOG_DIR`` env var
2. ``~/.hermes/logs/memory_tencentdb`` (hermes-style log location)
3. ``<cwd>/.memory-tencentdb-logs`` (last-resort fallback if $HOME
is not set — unusual on real systems, but e.g. hermetic tests)
Note: the supervisor intentionally does *not* derive this from the
Gateway's data dir — the Gateway owns that path and the supervisor
no longer tracks it. Keeping our log dir in the hermes log tree also
avoids interleaving Gateway logs with user-facing memory data.
"""
env_dir = os.environ.get("MEMORY_TENCENTDB_LOG_DIR")
if env_dir:
return env_dir
home = os.environ.get("HOME") or os.environ.get("USERPROFILE")
if home:
return os.path.join(home, ".hermes", "logs", "memory_tencentdb")
return os.path.join(os.getcwd(), ".memory-tencentdb-logs")
def _close_log_handles(self) -> None:
"""Close log file handles; safe to call multiple times."""
for attr in ("_stdout_log", "_stderr_log"):
handle: Optional[IO[bytes]] = getattr(self, attr, None)
if handle is not None:
try:
handle.close()
except Exception:
pass
setattr(self, attr, None)
def _tail_stderr_log(self, max_bytes: int = LOG_TAIL_BYTES_ON_CRASH) -> str:
"""Return the last `max_bytes` of the stderr log for crash diagnostics."""
path = self._stderr_log_path
if not path:
return ""
try:
size = os.path.getsize(path)
with open(path, "rb") as f:
if size > max_bytes:
f.seek(-max_bytes, os.SEEK_END)
return f.read().decode("utf-8", errors="replace")
except Exception:
return ""
def _wait_for_health(self) -> bool:
"""Wait for the Gateway to become healthy."""
start = time.monotonic()
while time.monotonic() - start < HEALTH_CHECK_MAX_WAIT:
# Check if process died
if self._process and self._process.poll() is not None:
rc = self._process.returncode
# stderr was redirected to a log file; tail it for diagnostics.
stderr = self._tail_stderr_log()[:500]
logger.error(
"memory-tencentdb Gateway process exited with code %d during startup. "
"stderr_log=%s tail=%s",
rc, self._stderr_log_path or "<none>", stderr,
)
self._close_log_handles()
return False
try:
result = self._client.health(timeout=2)
if result.get("status") in ("ok", "degraded"):
logger.info(
"memory-tencentdb Gateway is ready (took %.1fs)",
time.monotonic() - start,
)
return True
except Exception:
pass
time.sleep(HEALTH_CHECK_INTERVAL)
logger.error(
"memory-tencentdb Gateway did not become healthy within %ds",
HEALTH_CHECK_MAX_WAIT,
)
return False
def shutdown(self) -> None:
"""Shut down the managed Gateway process (if we started it)."""
if self._process is None:
return
logger.info("Shutting down memory-tencentdb Gateway...")
try:
# Send SIGTERM for graceful shutdown
self._process.terminate()
try:
self._process.wait(timeout=10)
except subprocess.TimeoutExpired:
logger.warning("memory-tencentdb Gateway did not exit in 10s, sending SIGKILL")
self._process.kill()
self._process.wait(timeout=5)
except Exception as e:
logger.warning("Error shutting down memory-tencentdb Gateway: %s", e)
finally:
self._process = None
self._close_log_handles()
@property
def client(self) -> MemoryTencentdbSdkClient:
"""Get the HTTP client for making API calls."""
return self._client
hermes-plugin/memory/memory_tencentdb/tests/__init__.py
View File
hermes-plugin/memory/memory_tencentdb/tests/test_gateway_shutdown_leak.py
+806
View File
@@ -0,0 +1,806 @@
"""End-to-end tests for the "A mode" Gateway shutdown contract.
Background
----------
When the ``memory_tencentdb`` provider runs under hermes and the Gateway
is launched **by** the hermes process (Mode A — supervisor as parent),
``provider.shutdown()`` used to leave the Gateway subprocess running.
Because the supervisor spawns the Gateway with ``start_new_session=True``,
an un-shutdown Gateway is reparented to PID 1 and survives as an orphan.
Two concrete bugs fell out of that:
1. Orphan Gateway processes accumulate across hermes restarts.
2. The next hermes process's ``is_running()`` health-check sees the stale
Gateway as healthy and *reuses it*, silently ignoring any config the
user rotated between restarts (e.g. a new LLM API key installed via
``memory-tencentdb-ctl --hermes config llm``).
The fix: ``provider.shutdown()`` now calls ``supervisor.shutdown()``.
This test module locks that contract in.
Test suite layout
-----------------
* :class:`GatewayShutdownLeakTest`
Core contract tests against a fake Python HTTP Gateway. Fast (≤ a few
seconds), no Node/pnpm/tsx dependency, safe for CI. Covers:
- ``test_provider_shutdown_should_stop_supervisor_gateway``
Supervisor-owned Gateway **must** die on provider.shutdown().
- ``test_external_gateway_is_not_killed``
If the Gateway was already running when the provider attached
(``ensure_running`` returns early without spawning), shutdown must
**not** terminate it — we only own what we started.
- ``test_second_provider_does_not_reuse_stale_gateway``
End-to-end reproduction of the "stale LLM config" user report:
provider-A starts a Gateway, shuts down, provider-B starts up;
provider-B must not silently reuse the old Gateway.
* :class:`RealGatewayShutdownTest`
Integration test against the actual Node Gateway under
``src/gateway/server.ts``. Validates graceful shutdown (SIGTERM-driven
``gateway.stop()`` runs, SQLite WAL is checkpointed so ``*-wal``/
``*-shm`` sidecars don't leak). Skipped by default because it requires
a working ``pnpm``/``tsx`` toolchain and ~30s to start; opt in via
``TDAI_E2E_REAL_GATEWAY=1``.
Run directly::
python3 hermes-plugin/memory/memory_tencentdb/tests/test_gateway_shutdown_leak.py
Or scope to one case::
python3 hermes-plugin/memory/memory_tencentdb/tests/test_gateway_shutdown_leak.py \\
GatewayShutdownLeakTest.test_external_gateway_is_not_killed
"""
from __future__ import annotations
import os
import pathlib
import shutil
import signal
import subprocess
import sys
import tempfile
import textwrap
import time
import unittest
from typing import Dict, List, Optional
# ---------------------------------------------------------------------------
# Path setup
# ---------------------------------------------------------------------------
# tdai-memory-openclaw-plugin / hermes-plugin / memory / memory_tencentdb / tests / THIS FILE
_PROJECT_ROOT = pathlib.Path(__file__).resolve().parents[4]
_HERMES_PLUGIN_ROOT = _PROJECT_ROOT / "hermes-plugin"
def _ensure_importable() -> Optional[str]:
"""Inject plugin + hermes-agent roots into ``sys.path``.
Returns an informational skip reason if hermes-agent can't be located,
otherwise None. Each test method checks the return value and skips if
set, so the whole file still imports cleanly in environments without
a hermes checkout.
"""
if str(_HERMES_PLUGIN_ROOT) not in sys.path:
sys.path.insert(0, str(_HERMES_PLUGIN_ROOT))
hermes_agent_root = os.environ.get("HERMES_AGENT_ROOT")
if not hermes_agent_root:
candidate = _PROJECT_ROOT.parent / "hermes-agent"
if candidate.is_dir():
hermes_agent_root = str(candidate)
if not hermes_agent_root or not pathlib.Path(hermes_agent_root, "agent").is_dir():
return (
"hermes-agent checkout not found — set HERMES_AGENT_ROOT to "
"point at a sibling hermes-agent repo to run this test."
)
if hermes_agent_root not in sys.path:
sys.path.insert(0, hermes_agent_root)
return None
# ---------------------------------------------------------------------------
# Fake Gateway (Python HTTP server) helpers
# ---------------------------------------------------------------------------
def _pick_free_port() -> int:
"""Ask the kernel for an ephemeral port."""
import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind(("127.0.0.1", 0))
return s.getsockname()[1]
def _make_fake_gateway_script(tmpdir: pathlib.Path, pid_file: pathlib.Path) -> pathlib.Path:
"""Write a small Python HTTP server that impersonates the Gateway.
Behaviour:
* On startup, writes its own PID into ``pid_file`` and also a
line-per-request log into ``<tmpdir>/gateway.trace`` so tests can
assert which instance answered which request.
* Serves ``GET /health`` with the Gateway's canonical JSON shape.
Echoes the ``MEMORY_TENCENTDB_LLM_API_KEY`` env var back in a
``fingerprint`` field so "stale config reuse" tests can see which
instance answered.
* SIGTERM handler: remove the pid file and exit cleanly — lets us
distinguish "supervisor sent SIGTERM" from "orphaned, still up".
"""
script = tmpdir / "fake_gateway.py"
trace = tmpdir / "gateway.trace"
script.write_text(textwrap.dedent(
f"""\
import hashlib, json, os, signal, sys
from http.server import BaseHTTPRequestHandler, HTTPServer
PID_FILE = {str(pid_file)!r}
TRACE = {str(trace)!r}
PORT = int(os.environ["MEMORY_TENCENTDB_GATEWAY_PORT"])
# Stamp startup so tests know this is the correct instance.
FINGERPRINT = hashlib.sha1(
os.environ.get("MEMORY_TENCENTDB_LLM_API_KEY", "").encode()
).hexdigest()[:12]
with open(PID_FILE, "w", encoding="utf-8") as f:
f.write(str(os.getpid()))
with open(TRACE, "a", encoding="utf-8") as f:
f.write(f"start pid={{os.getpid()}} fp={{FINGERPRINT}}\\n")
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
if self.path == "/health":
body = json.dumps({{
"status": "ok",
"version": "fake-v1",
"uptime": 1,
"fingerprint": FINGERPRINT,
"stores": {{
"vectorStore": True,
"embeddingService": True,
}},
}}).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
with open(TRACE, "a", encoding="utf-8") as f:
f.write(f"GET /health pid={{os.getpid()}} fp={{FINGERPRINT}}\\n")
else:
self.send_response(404)
self.end_headers()
def log_message(self, fmt, *args):
pass
def _term(_signum, _frame):
try:
os.unlink(PID_FILE)
except OSError:
pass
with open(TRACE, "a", encoding="utf-8") as f:
f.write(f"stop pid={{os.getpid()}}\\n")
sys.exit(0)
signal.signal(signal.SIGTERM, _term)
signal.signal(signal.SIGINT, _term)
HTTPServer(("127.0.0.1", PORT), Handler).serve_forever()
"""
))
return script
def _pid_alive(pid: int) -> bool:
"""Return True if the OS says this pid is still a live process."""
try:
os.kill(pid, 0)
except ProcessLookupError:
return False
except PermissionError:
return True
return True
def _wait_for_pid_file(pid_file: pathlib.Path, timeout: float = 5.0) -> int:
"""Poll until the fake gateway writes its pid file; return the pid."""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if pid_file.exists():
raw = pid_file.read_text().strip()
if raw:
return int(raw)
time.sleep(0.05)
raise TimeoutError(f"fake gateway did not write {pid_file} within {timeout}s")
def _wait_until_dead(pid: int, timeout: float = 5.0) -> bool:
"""Poll up to ``timeout`` seconds for the pid to disappear."""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if not _pid_alive(pid):
return True
time.sleep(0.05)
return False
def _kill_if_alive(pid: int) -> None:
"""Best-effort SIGTERM→SIGKILL for cleanup paths."""
if not _pid_alive(pid):
return
try:
os.kill(pid, signal.SIGTERM)
time.sleep(0.2)
if _pid_alive(pid):
os.kill(pid, signal.SIGKILL)
except ProcessLookupError:
pass
def _set_env(overrides: Dict[str, Optional[str]]) -> Dict[str, Optional[str]]:
"""Apply env overrides, returning a restore dict."""
prior: Dict[str, Optional[str]] = {k: os.environ.get(k) for k in overrides}
for k, v in overrides.items():
if v is None:
os.environ.pop(k, None)
else:
os.environ[k] = v
return prior
def _restore_env(prior: Dict[str, Optional[str]]) -> None:
for k, v in prior.items():
if v is None:
os.environ.pop(k, None)
else:
os.environ[k] = v
# ---------------------------------------------------------------------------
# Core contract tests — against fake Python HTTP Gateway
# ---------------------------------------------------------------------------
class GatewayShutdownLeakTest(unittest.TestCase):
"""Supervisor lifecycle contract (fast; no Node dependency)."""
def setUp(self) -> None:
skip = _ensure_importable()
if skip:
self.skipTest(skip)
self._tmpdir = pathlib.Path(tempfile.mkdtemp(prefix="tdai-shutdown-leak-"))
self._pid_file = self._tmpdir / "gateway.pid"
self._fake_script = _make_fake_gateway_script(self._tmpdir, self._pid_file)
self._rogue_pids: List[int] = []
def tearDown(self) -> None:
if self._pid_file.exists():
try:
pid = int(self._pid_file.read_text().strip())
except Exception:
pid = 0
if pid:
_kill_if_alive(pid)
for pid in self._rogue_pids:
_kill_if_alive(pid)
shutil.rmtree(self._tmpdir, ignore_errors=True)
# -- utilities ----------------------------------------------------------
def _fake_gateway_cmd(self) -> str:
return f"{sys.executable} {self._fake_script}"
def _spawn_external_gateway(self, port: int, api_key: str = "") -> int:
"""Start a fake Gateway *outside* the supervisor's control.
Simulates "Gateway already running when provider attaches" —
e.g. started manually by the user or by a previous process that
legitimately left it behind.
"""
env = os.environ.copy()
env["MEMORY_TENCENTDB_GATEWAY_PORT"] = str(port)
if api_key:
env["MEMORY_TENCENTDB_LLM_API_KEY"] = api_key
proc = subprocess.Popen(
[sys.executable, str(self._fake_script)],
env=env,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
start_new_session=True,
)
# wait for it to come up
pid = _wait_for_pid_file(self._pid_file, timeout=8.0)
self.assertEqual(pid, proc.pid)
self._rogue_pids.append(pid)
return pid
# -- tests --------------------------------------------------------------
def test_provider_shutdown_should_stop_supervisor_gateway(self) -> None:
"""A-mode contract: Gateway we started MUST die on shutdown()."""
from memory.memory_tencentdb import MemoryTencentdbProvider
port = _pick_free_port()
prior = _set_env({
"MEMORY_TENCENTDB_GATEWAY_HOST": "127.0.0.1",
"MEMORY_TENCENTDB_GATEWAY_PORT": str(port),
"MEMORY_TENCENTDB_GATEWAY_CMD": self._fake_gateway_cmd(),
})
try:
provider = MemoryTencentdbProvider()
provider.initialize(session_id="leak-test-session", user_id="tester")
pid = _wait_for_pid_file(self._pid_file, timeout=8.0)
self.assertTrue(_pid_alive(pid))
provider.shutdown()
died = _wait_until_dead(pid, timeout=3.0)
self.assertTrue(
died,
f"Gateway pid={pid} still alive 3s after provider.shutdown(); "
"supervisor teardown did not propagate.",
)
finally:
_restore_env(prior)
def test_external_gateway_is_not_killed(self) -> None:
"""Symmetry contract: don't kill what we didn't start.
If the Gateway was already serving on the configured port when
the provider attached, ``supervisor.ensure_running()`` returns
without spawning and leaves ``_process = None``. In that case
``shutdown()`` must be a no-op for the Gateway — killing it would
break anyone else already using it.
"""
from memory.memory_tencentdb import MemoryTencentdbProvider
port = _pick_free_port()
external_pid = self._spawn_external_gateway(port)
prior = _set_env({
"MEMORY_TENCENTDB_GATEWAY_HOST": "127.0.0.1",
"MEMORY_TENCENTDB_GATEWAY_PORT": str(port),
# Supply a CMD too — we want to prove the supervisor takes the
# is_running() fast path and *doesn't* spawn a second copy.
"MEMORY_TENCENTDB_GATEWAY_CMD": self._fake_gateway_cmd(),
})
try:
provider = MemoryTencentdbProvider()
provider.initialize(session_id="external-gw-session", user_id="tester")
# Sanity: the external Gateway is still the pid-file holder.
pid = int(self._pid_file.read_text().strip())
self.assertEqual(
pid, external_pid,
"Supervisor unexpectedly started a second Gateway; "
"is_running() fast path must be taken when a healthy "
"Gateway is already serving the port.",
)
provider.shutdown()
# External gateway must survive.
time.sleep(0.5)
self.assertTrue(
_pid_alive(external_pid),
f"External Gateway pid={external_pid} was killed by "
"provider.shutdown(); supervisor must only terminate "
"processes it started itself.",
)
finally:
_restore_env(prior)
def test_second_provider_does_not_reuse_stale_gateway(self) -> None:
"""Stale-config reproduction.
Mirrors the user report: rotate ``MEMORY_TENCENTDB_LLM_API_KEY``
between two hermes runs. The second provider must end up with a
Gateway whose env has the *new* key — i.e. a brand-new process,
not the first provider's leftover. The fake Gateway publishes
``fingerprint = sha1(api_key)[:12]`` over ``/health`` so we can
tell the two apart by a single HTTP call.
"""
from memory.memory_tencentdb import MemoryTencentdbProvider
from memory.memory_tencentdb.client import MemoryTencentdbSdkClient
port = _pick_free_port()
def _health_fingerprint() -> str:
client = MemoryTencentdbSdkClient(
base_url=f"http://127.0.0.1:{port}", timeout=2,
)
return client.health(timeout=2).get("fingerprint", "")
prior = _set_env({
"MEMORY_TENCENTDB_GATEWAY_HOST": "127.0.0.1",
"MEMORY_TENCENTDB_GATEWAY_PORT": str(port),
"MEMORY_TENCENTDB_GATEWAY_CMD": self._fake_gateway_cmd(),
"MEMORY_TENCENTDB_LLM_API_KEY": "old-key-AAA",
})
try:
# --- first provider run (the "before rotation" hermes) ---
provider_a = MemoryTencentdbProvider()
provider_a.initialize(session_id="sess-a", user_id="tester")
pid_a = _wait_for_pid_file(self._pid_file, timeout=8.0)
fp_a = _health_fingerprint()
self.assertTrue(fp_a, "first Gateway did not publish a fingerprint")
provider_a.shutdown()
self.assertTrue(
_wait_until_dead(pid_a, timeout=3.0),
"first Gateway still alive after provider_a.shutdown() — "
"stale-config bug would reappear.",
)
# --- user rotates the LLM key between hermes restarts ---
os.environ["MEMORY_TENCENTDB_LLM_API_KEY"] = "new-key-ZZZ"
# --- second provider run (the "after rotation" hermes) ---
provider_b = MemoryTencentdbProvider()
provider_b.initialize(session_id="sess-b", user_id="tester")
pid_b = _wait_for_pid_file(self._pid_file, timeout=8.0)
fp_b = _health_fingerprint()
self.assertNotEqual(
pid_a, pid_b,
"provider_b reused provider_a's Gateway pid — the "
"orphan survived shutdown and was picked up by "
"is_running() (classic stale-config bug).",
)
self.assertNotEqual(
fp_a, fp_b,
"provider_b's Gateway still reports the old key "
f"fingerprint ({fp_a}); the new env never reached a "
"fresh process.",
)
provider_b.shutdown()
self.assertTrue(_wait_until_dead(pid_b, timeout=3.0))
finally:
_restore_env(prior)
# ---------------------------------------------------------------------------
# Integration test — against the real Node Gateway (opt-in)
# ---------------------------------------------------------------------------
class RealGatewayShutdownTest(unittest.TestCase):
"""Opt-in integration test for graceful shutdown of the real Gateway.
Enabled only when ``TDAI_E2E_REAL_GATEWAY=1`` is set, because it:
* depends on ``pnpm`` / ``tsx`` being available on PATH,
* costs ~10-30s (Node cold start + first /health),
* writes to a temp SQLite data dir.
Verifies two properties that matter beyond "pid dies":
1. ``gateway.stop()`` actually ran — SIGTERM was delivered and the
in-process shutdown handler finished before ``process.exit(0)``.
Proxy signal: SQLite files are in a clean state (no leftover
``*-wal`` with unflushed bytes).
2. The process exits within a reasonable grace window.
"""
def setUp(self) -> None:
if os.environ.get("TDAI_E2E_REAL_GATEWAY") != "1":
self.skipTest(
"Real-Gateway test skipped; set TDAI_E2E_REAL_GATEWAY=1 "
"to enable (requires pnpm + tsx on PATH)."
)
skip = _ensure_importable()
if skip:
self.skipTest(skip)
server_ts = _PROJECT_ROOT / "src" / "gateway" / "server.ts"
if not server_ts.is_file():
self.skipTest(f"src/gateway/server.ts not found at {server_ts}")
if shutil.which("pnpm") is None:
self.skipTest("pnpm not on PATH")
self._tmpdir = pathlib.Path(tempfile.mkdtemp(prefix="tdai-real-gw-"))
self._data_dir = self._tmpdir / "data"
self._data_dir.mkdir()
def tearDown(self) -> None:
shutil.rmtree(self._tmpdir, ignore_errors=True)
def test_real_gateway_graceful_shutdown(self) -> None:
from memory.memory_tencentdb import MemoryTencentdbProvider
port = _pick_free_port()
gateway_cmd = (
f"sh -c 'cd {_PROJECT_ROOT} && exec pnpm exec tsx src/gateway/server.ts'"
)
prior = _set_env({
"MEMORY_TENCENTDB_GATEWAY_HOST": "127.0.0.1",
"MEMORY_TENCENTDB_GATEWAY_PORT": str(port),
"MEMORY_TENCENTDB_GATEWAY_CMD": gateway_cmd,
# The supervisor exports MEMORY_TENCENTDB_GATEWAY_{HOST,PORT}
# into the child env, but ``src/gateway/config.ts`` currently
# reads ``TDAI_GATEWAY_{HOST,PORT}``. Export both so this test
# is agnostic to that mismatch (which is tracked separately).
"TDAI_GATEWAY_HOST": "127.0.0.1",
"TDAI_GATEWAY_PORT": str(port),
"TDAI_DATA_DIR": str(self._data_dir),
# Supply a placeholder LLM key: /health doesn't need it, but
# unset keys make the L1 extractor log loud errors. A fake
# key keeps the log clean and has no effect on the shutdown
# path we're actually testing.
"TDAI_LLM_API_KEY": "sk-test-placeholder-not-used",
"MEMORY_TENCENTDB_LLM_API_KEY": "sk-test-placeholder-not-used",
})
try:
provider = MemoryTencentdbProvider()
provider.initialize(session_id="real-gw-sess", user_id="tester")
# Fail loudly if the Gateway didn't actually come up — otherwise
# a failed startup would mask the shutdown assertions below and
# let a regression slip through. Surface the stderr log tail
# (same location the supervisor uses) to make diagnosis easy.
if not provider._gateway_available: # noqa: SLF001 (test access)
log_path = pathlib.Path(
os.environ.get("HOME", "") or "/",
".hermes", "logs", "memory_tencentdb", "gateway.stderr.log",
)
tail = ""
if log_path.is_file():
data = log_path.read_bytes()
tail = data[-2048:].decode("utf-8", errors="replace")
self.fail(
"real Node Gateway failed to become healthy; cannot "
f"test shutdown. Recent stderr:\n{tail}"
)
# The supervisor stores the Popen object; reach in (test-only)
# to grab the pid so we can watch it across shutdown.
supervisor = provider._supervisor # noqa: SLF001 (test access)
self.assertIsNotNone(supervisor, "supervisor must be set after initialize()")
proc = supervisor._process # noqa: SLF001
self.assertIsNotNone(
proc,
"real Node Gateway was expected to be spawned by the supervisor; "
"got None — did the health check fail?",
)
pid = proc.pid
t0 = time.monotonic()
provider.shutdown()
elapsed = time.monotonic() - t0
self.assertTrue(
_wait_until_dead(pid, timeout=12.0),
f"real Node Gateway pid={pid} did not exit within 12s of "
"SIGTERM — graceful shutdown path hung.",
)
# Graceful stop should typically finish well under the 10s
# supervisor timeout; flag long waits so regressions are loud.
self.assertLess(
elapsed, 10.0,
f"provider.shutdown() took {elapsed:.1f}s — suspiciously "
"close to the SIGKILL fallback. Check gateway.stop() for "
"blocking work.",
)
# Graceful-exit witness: no stray SQLite WAL/SHM should remain
# under the data dir. If the Gateway was SIGKILL'd mid-write,
# these sidecars would be left behind with uncommitted bytes.
leftovers = sorted(
p for p in self._data_dir.rglob("*")
if p.suffix in (".db-wal", ".db-shm")
)
self.assertEqual(
leftovers, [],
f"found leftover SQLite sidecars after graceful shutdown: "
f"{[str(p) for p in leftovers]}",
)
finally:
_restore_env(prior)
def test_wal_checkpoint_after_capture_and_sigterm(self) -> None:
"""Write data via capture(), then SIGTERM — graceful close verified.
End-to-end proof that ``gateway.stop()`` actually runs (not just
"pid disappears") when the supervisor sends SIGTERM:
1. Start a fresh real Node Gateway pointed at a temp data dir.
2. Send several ``/capture`` calls to produce L0 data.
3. Confirm data was actually written to disk (JSONL and/or .db).
4. ``provider.shutdown()`` → SIGTERM → ``gateway.stop()`` →
``core.destroy()`` → ``vectorStore.close()`` (which runs
an implicit ``PRAGMA wal_checkpoint``).
5. Assert the process exited **cleanly** (exit code 0 via
SIGTERM handler, not 137 from SIGKILL).
6. Assert shutdown finished well under the 10s SIGKILL fallback.
7. If any ``.db`` files exist, assert no dirty WAL / SHM remain.
8. Confirm JSONL data files are intact (non-empty, valid JSON
lines) — proves L0 writes were fully flushed.
Note: when ``sqlite-vec`` is not available, VectorStore enters
degraded mode and L0 goes through JSONL only. The test adapts:
it always checks JSONL; WAL assertions only fire when ``.db``
files actually exist.
"""
from memory.memory_tencentdb import MemoryTencentdbProvider
from memory.memory_tencentdb.client import MemoryTencentdbSdkClient
import json as _json
port = _pick_free_port()
gateway_cmd = (
f"sh -c 'cd {_PROJECT_ROOT} && exec pnpm exec tsx src/gateway/server.ts'"
)
prior = _set_env({
"MEMORY_TENCENTDB_GATEWAY_HOST": "127.0.0.1",
"MEMORY_TENCENTDB_GATEWAY_PORT": str(port),
"MEMORY_TENCENTDB_GATEWAY_CMD": gateway_cmd,
"TDAI_GATEWAY_HOST": "127.0.0.1",
"TDAI_GATEWAY_PORT": str(port),
"TDAI_DATA_DIR": str(self._data_dir),
"TDAI_LLM_API_KEY": "sk-test-placeholder-not-used",
"MEMORY_TENCENTDB_LLM_API_KEY": "sk-test-placeholder-not-used",
})
try:
provider = MemoryTencentdbProvider()
provider.initialize(session_id="wal-ckpt-sess", user_id="wal-tester")
if not provider._gateway_available: # noqa: SLF001
log_path = pathlib.Path(
os.environ.get("HOME", "") or "/",
".hermes", "logs", "memory_tencentdb", "gateway.stderr.log",
)
tail = ""
if log_path.is_file():
data = log_path.read_bytes()
tail = data[-2048:].decode("utf-8", errors="replace")
self.fail(
"real Node Gateway failed to become healthy; cannot "
f"test WAL checkpoint. Recent stderr:\n{tail}"
)
supervisor = provider._supervisor # noqa: SLF001
proc = supervisor._process # noqa: SLF001
self.assertIsNotNone(proc, "Gateway process must be spawned")
pid = proc.pid
# ---- Step 2: write data via /capture ----
client = MemoryTencentdbSdkClient(
base_url=f"http://127.0.0.1:{port}", timeout=10,
)
n_captures = 5
for i in range(n_captures):
try:
client.capture(
user_content=f"Test message {i}: the quick brown fox",
assistant_content=f"Acknowledged message {i}.",
session_key="wal-ckpt-sess",
user_id="wal-tester",
)
except Exception:
# capture() may partially fail (e.g. LLM extraction) but
# L0 write still happens before extraction kicks in.
pass
# Give the Gateway a moment to flush writes.
time.sleep(0.5)
# ---- Step 3: confirm data was written to disk ----
# JSONL (always present, even when VectorStore is degraded):
jsonl_files = sorted(self._data_dir.rglob("*.jsonl"))
self.assertTrue(
len(jsonl_files) > 0,
f"expected at least one .jsonl file under {self._data_dir} "
f"after {n_captures} capture() calls.",
)
total_lines_before = 0
for jf in jsonl_files:
lines = [l for l in jf.read_text().splitlines() if l.strip()]
total_lines_before += len(lines)
# Validate each line is parseable JSON.
for idx, line in enumerate(lines):
try:
_json.loads(line)
except _json.JSONDecodeError:
self.fail(
f"invalid JSON on line {idx+1} of {jf}: {line[:120]}"
)
self.assertGreater(
total_lines_before, 0,
"JSONL files exist but contain no data lines.",
)
# .db files (only present when sqlite-vec loaded successfully):
db_files = sorted(self._data_dir.rglob("*.db"))
has_sqlite = len(db_files) > 0
# ---- Step 4: SIGTERM via provider.shutdown() ----
# Grab a reference to the Popen *before* supervisor.shutdown()
# sets it to None, so we can check returncode afterwards.
popen_ref = proc
t0 = time.monotonic()
provider.shutdown()
elapsed = time.monotonic() - t0
# ---- Step 5: verify clean exit (SIGTERM handler ran) ----
self.assertTrue(
_wait_until_dead(pid, timeout=12.0),
f"real Node Gateway pid={pid} did not exit within 12s.",
)
# returncode semantics:
# 0 → Node SIGTERM handler ran and called process.exit(0)
# -15 → SIGTERM killed the process directly (normal for
# multi-layer launchers like ``pnpm exec tsx``: the
# supervisor's terminate() hits pnpm, which exits on
# signal; tsx/node children then exit as a cascade)
# -9 → SIGKILL (supervisor had to force-kill after 10s
# timeout — that's a regression)
# positive → unexpected crash exit code
rc = popen_ref.returncode
self.assertIsNotNone(rc, "process should have exited")
self.assertNotEqual(
rc, -9,
"Gateway was SIGKILL'd (exit code -9) — the SIGTERM path "
"failed to terminate within the supervisor's 10s timeout. "
"Graceful shutdown is broken.",
)
# For direct-node launches rc==0 means the handler ran. For
# pnpm-wrapped launches rc==-15 is expected (pnpm doesn't trap
# SIGTERM). Both are acceptable; anything else is suspicious.
self.assertIn(
rc, (0, -15, -2), # 0=handler, -15=SIGTERM, -2=SIGINT
f"Gateway exited with unexpected code {rc}. Expected 0 "
"(graceful handler) or -15 (signal). Investigate.",
)
# ---- Step 6: timing ----
self.assertLess(
elapsed, 10.0,
f"provider.shutdown() took {elapsed:.1f}s — close to the "
"SIGKILL fallback; gateway.stop() may be blocked.",
)
# ---- Step 7: WAL/SHM cleanliness (only when .db exists) ----
if has_sqlite:
shm_leftovers = sorted(self._data_dir.rglob("*.db-shm"))
self.assertEqual(
shm_leftovers, [],
f"SHM files should not survive graceful shutdown: "
f"{[str(p) for p in shm_leftovers]}",
)
dirty_wals = sorted(
f for f in self._data_dir.rglob("*.db-wal")
if f.stat().st_size > 0
)
self.assertEqual(
dirty_wals, [],
f"non-empty WAL files found after graceful shutdown — "
f"wal_checkpoint was NOT completed: "
f"{[(str(f), f.stat().st_size) for f in dirty_wals]}",
)
# ---- Step 8: JSONL integrity post-shutdown ----
# The same JSONL files should still be intact and no smaller
# (gateway.stop → core.destroy should not truncate them).
total_lines_after = 0
for jf in jsonl_files:
if jf.exists():
lines = [l for l in jf.read_text().splitlines() if l.strip()]
total_lines_after += len(lines)
self.assertGreaterEqual(
total_lines_after, total_lines_before,
f"JSONL data shrank after shutdown "
f"(before={total_lines_before}, after={total_lines_after}); "
f"graceful shutdown may have corrupted L0 data.",
)
finally:
_restore_env(prior)
if __name__ == "__main__":
unittest.main(verbosity=2)
hermes-plugin/memory/memory_tencentdb/tests/test_memory_tencentdb_recovery.py
+435
View File
@@ -0,0 +1,435 @@
"""Tests for memory-tencentdb provider self-healing.
Verifies the fixes that prevent the "tdai dies and is never resurrected"
class of failures:
1. Watchdog thread starts on initialize() and resurrects a dead Gateway
even when no business request triggers the failure path.
2. Lazy probe (_ensure_alive_for_request) lets a request short-circuit
guard self-heal before returning empty, breaking the
"_gateway_available stuck at False" deadlock.
3. is_process_alive() correctly distinguishes "child has exited" from
"child still running but unhealthy".
4. shutdown() cleanly stops the watchdog and drops the supervisor so
subsequent recovery attempts are no-ops.
These tests use mocks for the supervisor / client so they neither spawn
real Node processes nor open network sockets.
"""
from __future__ import annotations
import os
import pathlib
import sys
import threading
import time
from typing import Optional
from unittest.mock import MagicMock
import pytest
# Inject plugin + hermes-agent roots into sys.path so the provider module
# can be imported regardless of whether tests are invoked from the
# tdai-memory-openclaw-plugin tree (where this file lives at
# hermes-plugin/memory/memory_tencentdb/tests/) or from a hermes-agent
# checkout (where the same file is under tests/plugins/memory/). Mirrors
# the layout used by ``test_gateway_shutdown_leak.py`` next door.
_THIS_FILE = pathlib.Path(__file__).resolve()
_HERE = _THIS_FILE.parent
# When checked into the plugin repo: parents[4] = repo root,
# hermes-plugin/ holds the importable ``plugins`` package.
# When checked into hermes-agent: the tests/ tree already sits under a
# repo root that exposes ``plugins`` directly, so the extra insertion is
# harmless (sys.path lookups stop at the first match).
for candidate in (
_HERE.parents[3] if len(_HERE.parents) >= 4 else None, # plugin repo: hermes-plugin/
_HERE.parents[4] if len(_HERE.parents) >= 5 else None, # hermes-agent root
_HERE.parents[2] if len(_HERE.parents) >= 3 else None, # fallback
):
if candidate is not None and (candidate / "plugins").is_dir():
if str(candidate) not in sys.path:
sys.path.insert(0, str(candidate))
# Optional: hermes-agent provides ``agent.memory_provider``. Tests can set
# HERMES_AGENT_ROOT to point at a sibling checkout if needed.
_hermes_root = os.environ.get("HERMES_AGENT_ROOT")
if not _hermes_root:
# Try the canonical sibling layout used by this monorepo.
sibling = _HERE.parents[4] / "hermes-agent" if len(_HERE.parents) >= 5 else None
if sibling is not None and (sibling / "agent").is_dir():
_hermes_root = str(sibling)
if _hermes_root and _hermes_root not in sys.path:
sys.path.insert(0, _hermes_root)
try:
from plugins.memory.memory_tencentdb import MemoryTencentdbProvider
from plugins.memory.memory_tencentdb import supervisor as supervisor_module
except ImportError as e: # pragma: no cover — env-dependent
pytest.skip(
f"memory_tencentdb provider not importable ({e}); set HERMES_AGENT_ROOT "
"to a hermes-agent checkout if running from the plugin repo.",
allow_module_level=True,
)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
class FakeSupervisor:
"""In-memory stand-in for GatewaySupervisor.
Lets tests script the sequence of (alive?, healthy?, ensure_running()
outcome) values without spawning subprocesses or opening sockets.
"""
def __init__(self) -> None:
self.alive = True
self.healthy = True
# If set, the next ensure_running() call flips alive+healthy back on.
self.respawn_succeeds = True
self.client = MagicMock(name="MemoryTencentdbSdkClient")
self.ensure_running_calls = 0
self.is_running_calls = 0
self.is_process_alive_calls = 0
self.shutdown_calls = 0
def is_running(self) -> bool:
self.is_running_calls += 1
return self.healthy
def is_process_alive(self) -> bool:
self.is_process_alive_calls += 1
return self.alive
def ensure_running(self) -> bool:
self.ensure_running_calls += 1
if self.respawn_succeeds:
self.alive = True
self.healthy = True
return True
return False
def shutdown(self) -> None:
self.shutdown_calls += 1
self.alive = False
self.healthy = False
def _wait_until(predicate, *, timeout: float = 3.0, interval: float = 0.02) -> bool:
"""Poll ``predicate`` until it returns truthy or ``timeout`` elapses."""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if predicate():
return True
time.sleep(interval)
return False
@pytest.fixture()
def fast_watchdog(monkeypatch):
"""Make the watchdog poll every 50 ms instead of 10 s.
Tests can then trigger a state change and assert the watchdog reacts
within a tight bound, keeping the suite fast.
"""
import plugins.memory.memory_tencentdb as mod
monkeypatch.setattr(mod, "_WATCHDOG_INTERVAL_SECS", 0.05)
monkeypatch.setattr(mod, "_WATCHDOG_SHUTDOWN_TIMEOUT_SECS", 0.5)
# Also collapse the request-path cooldown so tests do not need to wait
# 15 s between recovery attempts triggered from prefetch / sync_turn.
monkeypatch.setattr(mod, "_RECOVER_COOLDOWN_SECS", 0)
yield
@pytest.fixture()
def provider_with_fake_supervisor(monkeypatch, fast_watchdog):
"""Yield a MemoryTencentdbProvider wired to a FakeSupervisor.
We monkey-patch the GatewaySupervisor symbol used inside the provider
module so initialize() builds a FakeSupervisor instead of the real one.
The FakeSupervisor is exposed on the provider as ``_fake`` for tests
to manipulate.
"""
import plugins.memory.memory_tencentdb as mod
fake = FakeSupervisor()
def _factory(*args, **kwargs):
return fake
monkeypatch.setattr(mod, "GatewaySupervisor", _factory)
# Make the auto-discovery happy: pretend an env var is set so the
# provider does not try to walk the filesystem looking for server.ts.
monkeypatch.setenv("MEMORY_TENCENTDB_GATEWAY_CMD", "fake-cmd")
provider = MemoryTencentdbProvider()
provider.initialize(session_id="test-session", user_id="test-user")
provider._fake = fake # attach for test access
# initialize() may have spawned _background_start in another thread.
# Wait until the provider settles into the "available" state before
# tests start poking at it. The FakeSupervisor reports healthy from
# the get-go, so this should be quick.
_wait_until(lambda: provider._gateway_available, timeout=2.0)
try:
yield provider
finally:
provider.shutdown()
# ---------------------------------------------------------------------------
# Supervisor.is_process_alive
# ---------------------------------------------------------------------------
class _FakePopen:
def __init__(self, returncode: Optional[int] = None) -> None:
self._returncode = returncode
def poll(self):
return self._returncode
@property
def returncode(self):
return self._returncode
def test_is_process_alive_returns_false_without_spawn():
sup = supervisor_module.GatewaySupervisor(gateway_cmd="")
assert sup.is_process_alive() is False
def test_is_process_alive_true_when_running():
sup = supervisor_module.GatewaySupervisor(gateway_cmd="")
sup._process = _FakePopen(returncode=None)
assert sup.is_process_alive() is True
def test_is_process_alive_false_after_exit():
sup = supervisor_module.GatewaySupervisor(gateway_cmd="")
sup._process = _FakePopen(returncode=137)
assert sup.is_process_alive() is False
def test_reap_dead_process_drops_handle():
sup = supervisor_module.GatewaySupervisor(gateway_cmd="")
sup._process = _FakePopen(returncode=137)
sup._reap_dead_process()
assert sup._process is None
def test_reap_dead_process_keeps_alive_handle():
sup = supervisor_module.GatewaySupervisor(gateway_cmd="")
alive = _FakePopen(returncode=None)
sup._process = alive
sup._reap_dead_process()
assert sup._process is alive
# ---------------------------------------------------------------------------
# Watchdog: detects death, resurrects, and reattaches
# ---------------------------------------------------------------------------
def test_watchdog_starts_after_initialize(provider_with_fake_supervisor):
provider = provider_with_fake_supervisor
assert provider._watchdog_thread is not None
assert provider._watchdog_thread.is_alive()
def test_watchdog_detects_dead_gateway_and_resurrects(provider_with_fake_supervisor):
provider = provider_with_fake_supervisor
fake = provider._fake
# Simulate "tdai got SIGKILL'd": process is dead, port is silent.
fake.alive = False
fake.healthy = False
fake.respawn_succeeds = True
# Also force the provider into the "stuck" state to mimic the
# production deadlock described in the issue.
provider._gateway_available = False
# The watchdog (interval=50ms) should pick this up well within 1s.
assert _wait_until(
lambda: fake.ensure_running_calls >= 1, timeout=2.0
), "watchdog never called ensure_running on a dead Gateway"
# And after the respawn succeeds it must flip availability back on.
assert _wait_until(
lambda: provider._gateway_available, timeout=2.0
), "watchdog respawned but never restored _gateway_available"
# Client was reattached from the (post-respawn) supervisor instance.
assert provider._client is fake.client
def test_watchdog_picks_up_external_restart_without_respawning(
provider_with_fake_supervisor,
):
"""If something external (systemd, operator) brings tdai back, the
watchdog should NOT spawn a duplicate — it should just notice health
is back and reattach."""
provider = provider_with_fake_supervisor
fake = provider._fake
# Mark provider as "stuck False" but keep the Gateway healthy.
provider._gateway_available = False
fake.alive = True
fake.healthy = True
initial_respawns = fake.ensure_running_calls
assert _wait_until(
lambda: provider._gateway_available, timeout=2.0
), "watchdog never reattached to an externally-healthy Gateway"
assert fake.ensure_running_calls == initial_respawns, (
"watchdog spawned a duplicate even though the Gateway was healthy"
)
def test_watchdog_resets_circuit_breaker_on_recovery(provider_with_fake_supervisor):
provider = provider_with_fake_supervisor
fake = provider._fake
# Trip the breaker manually (mimics 5 consecutive request failures).
provider._consecutive_failures = 999
provider._breaker_open_until = time.monotonic() + 60
provider._gateway_available = False
fake.alive = False
fake.healthy = False
fake.respawn_succeeds = True
assert _wait_until(
lambda: provider._gateway_available and not provider._is_breaker_open(),
timeout=2.0,
), "watchdog recovered Gateway but did not reset the breaker"
def test_watchdog_stops_on_shutdown(provider_with_fake_supervisor):
provider = provider_with_fake_supervisor
thread = provider._watchdog_thread
assert thread is not None and thread.is_alive()
provider.shutdown()
# After shutdown, the thread must wind down promptly.
thread.join(timeout=1.0)
assert not thread.is_alive(), "watchdog kept running after shutdown()"
# ---------------------------------------------------------------------------
# Lazy probe: request path self-heals when stuck-False
# ---------------------------------------------------------------------------
def test_prefetch_recovers_when_stuck_false_and_breaker_closed(
provider_with_fake_supervisor,
):
"""The original bug: prefetch sees _gateway_available==False and
short-circuits to "" forever, never giving recovery a chance. After
the fix, prefetch should attempt a one-shot recovery and proceed."""
provider = provider_with_fake_supervisor
fake = provider._fake
# Stop the watchdog so it cannot sneak in and do the recovery for us;
# we want to assert that the *request path* is what triggers the heal.
provider._stop_watchdog()
# Park the provider in the stuck state.
provider._gateway_available = False
provider._client = None
fake.alive = False
fake.healthy = False
fake.respawn_succeeds = True
fake.client.recall.return_value = {"context": "memories from tdai"}
result = provider.prefetch(query="hello", session_id="test-session")
assert "memories from tdai" in result, (
"prefetch should self-heal and return real memories, got: %r" % result
)
assert provider._gateway_available
assert fake.ensure_running_calls >= 1
def test_prefetch_respects_open_breaker(provider_with_fake_supervisor):
"""Breaker should still take precedence — the lazy probe must not
turn every request into a respawn attempt during a confirmed outage."""
provider = provider_with_fake_supervisor
fake = provider._fake
provider._stop_watchdog()
provider._gateway_available = False
provider._consecutive_failures = 999
provider._breaker_open_until = time.monotonic() + 60
initial_respawns = fake.ensure_running_calls
assert provider.prefetch(query="hello") == ""
assert fake.ensure_running_calls == initial_respawns, (
"lazy probe ran ensure_running while breaker was open"
)
def test_handle_tool_call_recovers_when_stuck_false(provider_with_fake_supervisor):
provider = provider_with_fake_supervisor
fake = provider._fake
provider._stop_watchdog()
provider._gateway_available = False
provider._client = None
fake.respawn_succeeds = True
fake.client.search_memories.return_value = {"results": ["m1", "m2"]}
out = provider.handle_tool_call(
"memory_tencentdb_memory_search", {"query": "anything"}
)
assert "results" in out
assert provider._gateway_available
def test_sync_turn_recovers_when_stuck_false(provider_with_fake_supervisor):
provider = provider_with_fake_supervisor
fake = provider._fake
provider._stop_watchdog()
provider._gateway_available = False
provider._client = None
fake.respawn_succeeds = True
capture_called = threading.Event()
fake.client.capture.side_effect = lambda **kw: capture_called.set()
provider.sync_turn(user_content="u", assistant_content="a")
assert capture_called.wait(timeout=2.0), (
"sync_turn never reached the Gateway after lazy recovery"
)
assert provider._gateway_available
# ---------------------------------------------------------------------------
# Shutdown safety
# ---------------------------------------------------------------------------
def test_shutdown_drops_supervisor_blocks_recovery(provider_with_fake_supervisor):
provider = provider_with_fake_supervisor
fake = provider._fake
provider.shutdown()
# Even if a stale request came in after shutdown, _try_recover_gateway
# must refuse to run (supervisor is None).
before = fake.ensure_running_calls
assert provider._try_recover_gateway() is False
assert fake.ensure_running_calls == before
def test_shutdown_is_idempotent(provider_with_fake_supervisor):
provider = provider_with_fake_supervisor
provider.shutdown()
provider.shutdown() # must not raise
hermes-plugin/memory/memory_tencentdb_v2/README.md
+137
View File
@@ -0,0 +1,137 @@
# Hermes Adapter for TencentDB Agent Memory v2
[简体中文](./README_CN.md) · English
This directory is a reference implementation for integrating Hermes with TencentDB Agent Memory v2 API. It implements a Hermes `MemoryProvider` that talks to an already-running Memory Gateway through the Python SDK. It does not start or manage a Gateway subprocess.
For standalone local usage, the recommended Gateway endpoint is `http://127.0.0.1:8420`. The default local convention is `api_key = "local"` and `service_id = "default"`. If your Gateway enables `TDAI_GATEWAY_API_KEY`, use the same value as `TDAI_MEMORY_API_KEY`.
## Architecture
```text
Hermes Agent
└─ MemoryManager
└─ memory_tencentdb_v2 provider
├─ sync_turn() completed turn -> add_conversation (L0)
├─ prefetch() search memories/core/scenarios before prompt
├─ tdai_memory_search tool
├─ tdai_conversation_search tool
└─ tdai_read_scene tool
│
▼
tencentdb_agent_memory.MemoryClient
│ HTTP v2 API
▼
TencentDB Agent Memory Gateway (:8420 standalone, or remote service)
```
## v1 vs v2
| | v1 `memory_tencentdb` | v2 `memory_tencentdb_v2` |
|---|---|---|
| API | legacy `/recall`, `/capture`, `/search/*` | v2 `/v2/*` |
| HTTP client | raw `urllib.request` | `tencentdb_agent_memory` Python SDK (`httpx`) |
| Gateway lifecycle | may start a local Gateway subprocess | expects an external or container-managed Gateway |
| Standalone convention | localhost only | `endpoint=http://127.0.0.1:8420`, `api_key=local`, `service_id=default` |
| Tools | memory search, conversation search | plus `tdai_read_scene` |
## Quick Start
Recommended: run the installer from the repository root:
```bash
bash scripts/install-hermes-plugin-v2.sh
```
The script downloads and installs the Python SDK wheel, symlinks the `memory_tencentdb_v2` provider into Hermes' memory plugin directory, checks whether `~/.hermes/config.yaml` enables the provider, and writes `TDAI_MEMORY_ENDPOINT`, `TDAI_MEMORY_API_KEY`, and `TDAI_MEMORY_SERVICE_ID` to `~/.hermes/.env`. The SDK is installed into the Python environment Hermes actually uses: explicit `PYTHON_BIN` first, then `HERMES_VENV_DIR/bin/python` (default `~/.hermes/hermes-agent/venv/bin/python`), then system `python3` as a fallback. Override paths with `HERMES_HOME`, `HERMES_AGENT_DIR`, `HERMES_VENV_DIR`, `HERMES_MEMORY_PLUGIN_DIR`, `HERMES_ENV`, or `PYTHON_BIN` if needed.
For manual installation, follow the steps below.
### 1. Install the SDK
The Python SDK has not been published to PyPI yet. Download the wheel first, then install it:
```bash
curl -L -o tencentdb_agent_memory_sdk_python-0.1.0-py3-none-any.whl \
"https://cnb.cool/tencent/cloud/nosql/nosql-utilities/-/commit-assets/download/cc74bd6dbc931727da9ab6907b5ab1a07d7afd9d/tencentdb_agent_memory_sdk_python-0.1.0-py3-none-any.whl"
# Use the Python interpreter Hermes actually runs with; common path:
~/.hermes/hermes-agent/venv/bin/python -m pip install ./tencentdb_agent_memory_sdk_python-0.1.0-py3-none-any.whl
# If your Hermes uses another Python environment, use that interpreter instead:
# PYTHON_BIN=/path/to/hermes/python
# "$PYTHON_BIN" -m pip install ./tencentdb_agent_memory_sdk_python-0.1.0-py3-none-any.whl
```
The Python import path is `tencentdb_agent_memory`.
### 2. Configure environment
For standalone local Gateway:
```bash
export TDAI_MEMORY_ENDPOINT="http://127.0.0.1:8420"
export TDAI_MEMORY_API_KEY="local"
export TDAI_MEMORY_SERVICE_ID="default"
```
If the Gateway enables `TDAI_GATEWAY_API_KEY`, set `TDAI_MEMORY_API_KEY` to the same value.
### 3. Activate in Hermes (`~/.hermes/config.yaml`)
```yaml
memory:
provider: memory_tencentdb_v2
```
### 4. Install the provider into Hermes
Development symlink:
```bash
ln -s "$(pwd)/hermes-plugin/memory/memory_tencentdb_v2" \
<hermes-agent>/plugins/memory/memory_tencentdb_v2
```
Deployment copy:
```bash
cp -r hermes-plugin/memory/memory_tencentdb_v2 \
<hermes-agent>/plugins/memory/memory_tencentdb_v2
```
## Environment Variables
| Variable | Default | Description |
|---|---|---|
| `TDAI_MEMORY_ENDPOINT` | `http://127.0.0.1:8420` | Memory Gateway URL |
| `TDAI_MEMORY_API_KEY` | `local` in standalone examples | Bearer token sent by the SDK |
| `TDAI_MEMORY_SERVICE_ID` | `default` in standalone examples | Memory space ID, sent as `x-tdai-service-id` |
## Provider Responsibilities
| Method / Tool | Purpose |
|---|---|
| `initialize(session_id)` | Create the SDK client and bind Hermes session ID |
| `sync_turn(user_content, assistant_content)` | Write completed turns to L0 through `add_conversation()` |
| `prefetch(query)` | Search L1 memories and read L3/L2 context before the next prompt |
| `tdai_memory_search` | Agent-callable L1 memory search |
| `tdai_conversation_search` | Agent-callable L0 conversation search |
| `tdai_read_scene` | Agent-callable L2 scene read |
## Using This as an Adapter Template
When adapting another Python Agent framework, copy the same pattern:
1. Initialize a `MemoryClient(endpoint, api_key, service_id)`.
2. After each completed turn, call `add_conversation()` with user and assistant messages.
3. Before the next prompt, call `search_atomic()`, `read_core()`, and optionally `list_scenarios()`.
4. Format the recalled memory as a clearly labeled context block.
5. Expose tools for active memory search and scene reading.
6. Keep the adapter best-effort: Memory failures should not block the Agent's main response path.
## Reliability
- Circuit breaker: 5 consecutive failures trigger a 60-second cooldown.
- Thread-safe state: internal mutations are protected by a lock.
- Graceful degradation: failed prefetch/tool calls return empty or user-friendly results instead of crashing the Agent.
hermes-plugin/memory/memory_tencentdb_v2/README_CN.md
+137
View File
@@ -0,0 +1,137 @@
# Hermes 适配参考：TencentDB Agent Memory v2
简体中文 · [English](./README.md)
该目录是 Hermes 接入 TencentDB Agent Memory v2 API 的参考实现。它实现了一个 Hermes `MemoryProvider`，通过 Python SDK 连接一个已经运行的 Memory Gateway。该 provider 不启动、不管理 Gateway 子进程。
在 standalone 本地模式下，推荐的 Gateway 地址是 `http://127.0.0.1:8420`。默认约定是 `api_key = "local"`、`service_id = "default"`。如果 Gateway 显式启用了 `TDAI_GATEWAY_API_KEY`，则 `TDAI_MEMORY_API_KEY` 应使用同一个值。
## 架构
```text
Hermes Agent
└─ MemoryManager
└─ memory_tencentdb_v2 provider
├─ sync_turn() 完整对话轮次 -> add_conversation (L0)
├─ prefetch() prompt 前召回记忆/core/scenario
├─ tdai_memory_search 工具
├─ tdai_conversation_search 工具
└─ tdai_read_scene 工具
│
▼
tencentdb_agent_memory.MemoryClient
│ HTTP v2 API
▼
TencentDB Agent Memory Gateway（standalone :8420 或远端服务）
```
## v1 与 v2 对比
| | v1 `memory_tencentdb` | v2 `memory_tencentdb_v2` |
|---|---|---|
| API | 旧版 `/recall`、`/capture`、`/search/*` | v2 `/v2/*` |
| HTTP 客户端 | 原生 `urllib.request` | `tencentdb_agent_memory` Python SDK（`httpx`） |
| Gateway 生命周期 | 可能启动本地 Gateway 子进程 | 连接外部或容器管理的 Gateway |
| standalone 约定 | localhost only | `endpoint=http://127.0.0.1:8420`、`api_key=local`、`service_id=default` |
| 工具 | memory search、conversation search | 额外支持 `tdai_read_scene` |
## 快速开始
推荐从仓库根目录执行自动安装脚本：
```bash
bash scripts/install-hermes-plugin-v2.sh
```
脚本会下载并安装 Python SDK wheel，将 `memory_tencentdb_v2` provider 软链到 Hermes 的 memory 插件目录，检查 `~/.hermes/config.yaml` 是否启用 provider，并把 `TDAI_MEMORY_ENDPOINT`、`TDAI_MEMORY_API_KEY`、`TDAI_MEMORY_SERVICE_ID` 写入 `~/.hermes/.env`。SDK 会安装到 Hermes 实际使用的 Python 环境：优先使用显式传入的 `PYTHON_BIN`，其次使用 `HERMES_VENV_DIR/bin/python`（默认 `~/.hermes/hermes-agent/venv/bin/python`），最后才回退到系统 `python3`。如需覆盖路径，可设置 `HERMES_HOME`、`HERMES_AGENT_DIR`、`HERMES_VENV_DIR`、`HERMES_MEMORY_PLUGIN_DIR`、`HERMES_ENV` 或 `PYTHON_BIN`。
如需手动安装，可按下面步骤执行。
### 1. 安装 SDK
Python SDK 暂未发布到 PyPI，请先下载 wheel 包再安装：
```bash
curl -L -o tencentdb_agent_memory_sdk_python-0.1.0-py3-none-any.whl \
"https://cnb.cool/tencent/cloud/nosql/nosql-utilities/-/commit-assets/download/cc74bd6dbc931727da9ab6907b5ab1a07d7afd9d/tencentdb_agent_memory_sdk_python-0.1.0-py3-none-any.whl"
# 请使用 Hermes 实际运行时的 Python；常见路径如下：
~/.hermes/hermes-agent/venv/bin/python -m pip install ./tencentdb_agent_memory_sdk_python-0.1.0-py3-none-any.whl
# 如果你的 Hermes 不是使用该 venv，请改用对应的 python：
# PYTHON_BIN=/path/to/hermes/python
# "$PYTHON_BIN" -m pip install ./tencentdb_agent_memory_sdk_python-0.1.0-py3-none-any.whl
```
Python import 路径是 `tencentdb_agent_memory`。
### 2. 配置环境变量
standalone 本地 Gateway：
```bash
export TDAI_MEMORY_ENDPOINT="http://127.0.0.1:8420"
export TDAI_MEMORY_API_KEY="local"
export TDAI_MEMORY_SERVICE_ID="default"
```
如果 Gateway 启用了 `TDAI_GATEWAY_API_KEY`，请将 `TDAI_MEMORY_API_KEY` 设置成同一个值。
### 3. 在 Hermes 配置中启用（`~/.hermes/config.yaml`）
```yaml
memory:
provider: memory_tencentdb_v2
```
### 4. 安装 provider 到 Hermes
开发环境推荐软链：
```bash
ln -s "$(pwd)/hermes-plugin/memory/memory_tencentdb_v2" \
<hermes-agent>/plugins/memory/memory_tencentdb_v2
```
部署时也可以复制：
```bash
cp -r hermes-plugin/memory/memory_tencentdb_v2 \
<hermes-agent>/plugins/memory/memory_tencentdb_v2
```
## 环境变量
| 变量 | 默认值 | 说明 |
|---|---|---|
| `TDAI_MEMORY_ENDPOINT` | `http://127.0.0.1:8420` | Memory Gateway 地址 |
| `TDAI_MEMORY_API_KEY` | standalone 示例中为 `local` | SDK 发送的 Bearer token |
| `TDAI_MEMORY_SERVICE_ID` | standalone 示例中为 `default` | Memory 空间 ID，通过 `x-tdai-service-id` 发送 |
## Provider 职责
| 方法 / 工具 | 说明 |
|---|---|
| `initialize(session_id)` | 创建 SDK client，并绑定 Hermes session ID |
| `sync_turn(user_content, assistant_content)` | 完整对话结束后调用 `add_conversation()` 写入 L0 |
| `prefetch(query)` | 下一轮 prompt 前搜索 L1，并读取 L3/L2 上下文 |
| `tdai_memory_search` | Agent 可主动调用的 L1 记忆搜索 |
| `tdai_conversation_search` | Agent 可主动调用的 L0 对话搜索 |
| `tdai_read_scene` | Agent 可主动调用的 L2 场景读取 |
## 作为其它 Agent 的适配模板
如果你要适配其它 Python Agent 框架，可以复用同样模式：
1. 初始化 `MemoryClient(endpoint, api_key, service_id)`。
2. 在每个完整对话轮次结束后调用 `add_conversation()`。
3. 在下一轮 prompt 构建前调用 `search_atomic()`、`read_core()`，必要时调用 `list_scenarios()`。
4. 将召回结果格式化成清晰标注的上下文块。
5. 暴露主动工具，让 Agent 可以搜索记忆和读取场景。
6. 保持 best-effort：Memory 失败不应该阻塞 Agent 主回答链路。
## 可靠性
- 熔断：连续 5 次失败后进入 60 秒冷却。
- 线程安全：内部状态变更使用锁保护。
- 优雅降级：prefetch 或工具调用失败时返回空结果或友好错误，不让 Agent 崩溃。
hermes-plugin/memory/memory_tencentdb_v2/__init__.py
+393
View File
@@ -0,0 +1,393 @@
"""
memory_tencentdb_v2 — Hermes MemoryProvider backed by TencentDB Agent Memory v2 API.
Uses the official `tencentdb_agent_memory` Python SDK to communicate with the
Memory Gateway. Supports both local sidecar mode (127.0.0.1:8420) and remote
service mode (endpoint + api_key + service_id).
Key differences from v1 `memory_tencentdb`:
- Uses v2 REST API (`/v2/*`) with structured envelope responses
- Uses `tencentdb_agent_memory.MemoryClient` SDK instead of raw urllib
- Supports Bearer token authentication for multi-tenant service mode
- Exposes `tdai_read_scene` tool for on-demand L2 scene reading
- No local Gateway subprocess management (expects external Gateway)
Environment variables:
TDAI_MEMORY_ENDPOINT — Gateway URL (default: http://127.0.0.1:8420)
TDAI_MEMORY_API_KEY — API key for authentication (optional for local)
TDAI_MEMORY_SERVICE_ID — Service/Space ID (optional for local)
"""
from __future__ import annotations
import logging
import os
import threading
import time
from typing import Any, Dict, List, Optional
from agent.memory_provider import MemoryProvider
logger = logging.getLogger("memory_tencentdb_v2")
# ════════════════════════════════════════════
# SDK import (lazy to allow graceful degradation)
# ════════════════════════════════════════════
_sdk_available = False
_MemoryClient = None
_AsyncMemoryClient = None
_TDAMError = None
try:
from tencentdb_agent_memory import MemoryClient, AsyncMemoryClient, TDAMError
_MemoryClient = MemoryClient
_AsyncMemoryClient = AsyncMemoryClient
_TDAMError = TDAMError
_sdk_available = True
except ImportError:
logger.warning(
"tencentdb_agent_memory SDK not installed. "
"Install the wheel package from the Hermes adapter README."
)
class MemoryTencentdbV2Provider(MemoryProvider):
"""
Hermes MemoryProvider implementation using TencentDB Agent Memory v2 API.
"""
NAME = "memory_tencentdb_v2"
def __init__(self):
super().__init__()
self._client: Optional[Any] = None
self._session_id: str = ""
self._endpoint: str = ""
self._available: bool = False
self._lock = threading.Lock()
# Circuit breaker
self._consecutive_failures: int = 0
self._circuit_open_until: float = 0
self._max_failures: int = 5
self._circuit_timeout: float = 60.0 # seconds
# ════════════════════════════════════════════
# Identity
# ════════════════════════════════════════════
@property
def name(self) -> str:
return self.NAME
# ════════════════════════════════════════════
# Lifecycle
# ════════════════════════════════════════════
def is_available(self) -> bool:
return _sdk_available
def initialize(self, session_id: str, **kwargs) -> None:
self._session_id = session_id
self._endpoint = os.environ.get("TDAI_MEMORY_ENDPOINT", "http://127.0.0.1:8420")
api_key = os.environ.get("TDAI_MEMORY_API_KEY", "")
service_id = os.environ.get("TDAI_MEMORY_SERVICE_ID", "")
if not _sdk_available:
logger.error("tencentdb_agent_memory SDK not available, cannot initialize")
return
try:
self._client = _MemoryClient(
endpoint=self._endpoint,
api_key=api_key or "local",
service_id=service_id or "default",
)
self._available = True
logger.info(
f"Initialized: endpoint={self._endpoint}, "
f"service_id={service_id or 'default'}, "
f"session_id={session_id}"
)
except Exception as e:
logger.error(f"Failed to initialize client: {e}")
self._available = False
def shutdown(self) -> None:
if self._client and self._session_id:
try:
# Flush pipeline buffers for this session
self._safe_call("on_session_end", lambda: None)
except Exception:
pass
self._client = None
self._available = False
logger.info("Shutdown complete")
# ════════════════════════════════════════════
# Core: Recall (prefetch)
# ════════════════════════════════════════════
def prefetch(self, query: str, session_id: str = "") -> Optional[Dict[str, Any]]:
"""
Search L1 memories + L3 core for the given query.
Returns dict with 'prepend_context' and 'append_system_context'.
"""
sid = session_id or self._session_id
def _do():
# Search L1 memories (atomic)
memories_result = self._client.search_atomic(query=query, limit=5)
memories = memories_result.get("results", [])
# Read L3 core (formerly persona)
core_text = ""
try:
core_result = self._client.read_core()
core_text = core_result.get("content", "")
except Exception:
pass
# Read L2 scene navigation (list scenarios)
scene_nav = ""
try:
scenarios = self._client.list_scenarios()
entries = scenarios.get("entries", [])
if entries:
lines = []
for s in entries:
name = s.get("path", "").replace("scene_blocks/", "").replace(".md", "")
lines.append(f"- Scene: {name} ({s.get('size', 0)} bytes)")
scene_nav = "Available scenes:\n" + "\n".join(lines)
except Exception:
pass
# Build recall context
prepend = ""
if memories:
memory_lines = []
for m in memories:
content = m.get("content", "")
mtype = m.get("type", "unknown")
memory_lines.append(f"- [{mtype}] {content}")
prepend = (
"<relevant-memories>\n"
"以下是当前对话召回的相关记忆，仅作为参考：\n\n"
+ "\n".join(memory_lines)
+ "\n</relevant-memories>"
)
append_parts = []
if core_text:
append_parts.append(f"<user-core>\n{core_text}\n</user-core>")
if scene_nav:
append_parts.append(f"<scene-navigation>\n{scene_nav}\n</scene-navigation>")
return {
"prepend_context": prepend,
"append_system_context": "\n\n".join(append_parts) if append_parts else "",
}
return self._safe_call("prefetch", _do)
# ════════════════════════════════════════════
# Core: Capture (sync_turn)
# ════════════════════════════════════════════
def sync_turn(self, user_content: str, assistant_content: str, session_id: str = "") -> None:
"""Write a conversation turn to L0 via v2 API."""
sid = session_id or self._session_id
# v2 schema requires ISO 8601 datetime strings for timestamps.
from datetime import datetime, timezone
now = datetime.now(timezone.utc)
# Make user message slightly earlier than assistant for ordering
user_ts = now.replace(microsecond=max(0, now.microsecond - 1000)).isoformat().replace("+00:00", "Z")
assistant_ts = now.isoformat().replace("+00:00", "Z")
def _do():
messages = [
{"role": "user", "content": user_content, "timestamp": user_ts},
{"role": "assistant", "content": assistant_content, "timestamp": assistant_ts},
]
self._client.add_conversation(session_id=sid, messages=messages)
self._safe_call("sync_turn", _do)
# ════════════════════════════════════════════
# Core: Session end
# ════════════════════════════════════════════
def on_session_end(self, messages: Optional[List] = None) -> None:
"""Signal session end to flush pipeline buffers."""
# v2 API doesn't have explicit session/end — the pipeline handles it
# via timer-based flush. This is a no-op for v2.
logger.debug(f"Session end signaled for {self._session_id}")
# ════════════════════════════════════════════
# Tools: Agent-callable
# ════════════════════════════════════════════
def get_tool_schemas(self) -> List[Dict[str, Any]]:
"""Expose tools for the Agent to call."""
return [
{
"type": "function",
"function": {
"name": "tdai_memory_search",
"description": (
"Search through the user's long-term memories. "
"Returns relevant memory records ranked by relevance."
),
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"},
"limit": {"type": "integer", "description": "Max results (default 5)"},
},
"required": ["query"],
},
},
},
{
"type": "function",
"function": {
"name": "tdai_conversation_search",
"description": (
"Search through past conversation history. "
"Returns relevant messages ranked by relevance."
),
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"},
"limit": {"type": "integer", "description": "Max results (default 5)"},
},
"required": ["query"],
},
},
},
{
"type": "function",
"function": {
"name": "tdai_read_scene",
"description": (
"Read a scene block's full content by its name. "
"Use when you see a scene listed in Scene Navigation."
),
"parameters": {
"type": "object",
"properties": {
"scene_id": {
"type": "string",
"description": "Scene name (e.g. 'travel-plan')",
},
},
"required": ["scene_id"],
},
},
},
]
def handle_tool_call(self, tool_name: str, args: Dict[str, Any]) -> Optional[str]:
"""Handle Agent tool calls."""
if tool_name == "tdai_memory_search":
return self._handle_memory_search(args)
elif tool_name == "tdai_conversation_search":
return self._handle_conversation_search(args)
elif tool_name == "tdai_read_scene":
return self._handle_read_scene(args)
return None
def _handle_memory_search(self, args: Dict[str, Any]) -> str:
query = args.get("query", "")
limit = args.get("limit", 5)
def _do():
result = self._client.search_atomic(query=query, limit=limit)
items = result.get("results", [])
if not items:
return "No memories found for this query."
lines = []
for m in items:
lines.append(f"- [{m.get('type', '?')}] {m.get('content', '')}")
return "\n".join(lines)
return self._safe_call("memory_search", _do) or "Memory search failed."
def _handle_conversation_search(self, args: Dict[str, Any]) -> str:
query = args.get("query", "")
limit = args.get("limit", 5)
def _do():
result = self._client.search_conversation(query=query, limit=limit)
items = result.get("results", [])
if not items:
return "No conversations found for this query."
lines = []
for m in items:
role = m.get("role", "?")
content = m.get("content", "")
lines.append(f"[{role}] {content}")
return "\n".join(lines)
return self._safe_call("conversation_search", _do) or "Conversation search failed."
def _handle_read_scene(self, args: Dict[str, Any]) -> str:
scene_id = args.get("scene_id", "")
if not scene_id:
return "Error: scene_id is required"
path = scene_id if scene_id.endswith(".md") else f"{scene_id}.md"
def _do():
result = self._client.read_scenario(path=path)
content = result.get("content", "")
if not content:
return f"Scene '{scene_id}' is empty or not found."
return content
return self._safe_call("read_scene", _do) or f"Failed to read scene '{scene_id}'."
# ════════════════════════════════════════════
# Prompt
# ════════════════════════════════════════════
def system_prompt_block(self) -> str:
return "" # Core profile and scene nav injected via prefetch
# ════════════════════════════════════════════
# Circuit breaker + safe call wrapper
# ════════════════════════════════════════════
def _safe_call(self, label: str, fn):
"""Execute fn with circuit breaker protection."""
if not self._client:
logger.warning(f"[{label}] Client not initialized")
return None
# Check circuit breaker
if self._consecutive_failures >= self._max_failures:
if time.time() < self._circuit_open_until:
logger.warning(f"[{label}] Circuit open, skipping call")
return None
else:
logger.info(f"[{label}] Circuit half-open, retrying")
try:
result = fn()
with self._lock:
self._consecutive_failures = 0
return result
except Exception as e:
with self._lock:
self._consecutive_failures += 1
if self._consecutive_failures >= self._max_failures:
self._circuit_open_until = time.time() + self._circuit_timeout
logger.error(
f"[{label}] Circuit OPEN after {self._consecutive_failures} failures "
f"(timeout={self._circuit_timeout}s): {e}"
)
else:
logger.warning(f"[{label}] Failed ({self._consecutive_failures}/{self._max_failures}): {e}")
return None
hermes-plugin/memory/memory_tencentdb_v2/plugin.yaml
+9
View File
@@ -0,0 +1,9 @@
name: memory_tencentdb_v2
display_name: memory-tencentdb-v2
version: 1.0.0
description: "memory-tencentdb v2 API adapter — connects Hermes to a remote TDAI Memory Service via the official Python SDK. Supports multi-tenant, multi-space deployments."
hooks:
- on_session_end
aliases:
- memory-tencentdb-v2
- tdai-v2
hermes-plugin/memory/memory_tencentdb_v2/tests/__init__.py
View File
hermes-plugin/memory/memory_tencentdb_v2/tests/test_provider.py
+156
View File
@@ -0,0 +1,156 @@
"""
Tests for memory_tencentdb_v2 provider.
"""
import pytest
from unittest.mock import MagicMock, patch
import os
# Mock the SDK before importing provider
mock_client_cls = MagicMock()
mock_async_client_cls = MagicMock()
mock_error_cls = type("TDAMError", (Exception,), {"code": 0, "message": ""})
@pytest.fixture(autouse=True)
def patch_sdk(monkeypatch):
"""Patch the SDK import for all tests."""
import memory_tencentdb_v2 as mod
monkeypatch.setattr(mod, "_sdk_available", True)
monkeypatch.setattr(mod, "_MemoryClient", mock_client_cls)
monkeypatch.setattr(mod, "_TDAMError", mock_error_cls)
@pytest.fixture
def provider():
from memory_tencentdb_v2 import MemoryTencentdbV2Provider
p = MemoryTencentdbV2Provider()
return p
@pytest.fixture
def initialized_provider(provider, monkeypatch):
"""Provider with a mock client."""
monkeypatch.setenv("TDAI_MEMORY_ENDPOINT", "http://localhost:3100")
monkeypatch.setenv("TDAI_MEMORY_API_KEY", "test-key")
monkeypatch.setenv("TDAI_MEMORY_SERVICE_ID", "space-001")
mock_client = MagicMock()
mock_client_cls.return_value = mock_client
provider.initialize("test-session-001")
provider._client = mock_client
return provider, mock_client
class TestLifecycle:
def test_is_available_with_sdk(self, provider):
assert provider.is_available() is True
def test_initialize_creates_client(self, initialized_provider):
provider, mock_client = initialized_provider
assert provider._available is True
assert provider._session_id == "test-session-001"
def test_shutdown_clears_state(self, initialized_provider):
provider, _ = initialized_provider
provider.shutdown()
assert provider._client is None
assert provider._available is False
class TestPrefetch:
def test_prefetch_returns_memories_and_core(self, initialized_provider):
provider, mock_client = initialized_provider
mock_client.search_atomic.return_value = {
"results": [
{"content": "User likes Python", "type": "persona", "score": 0.9},
]
}
mock_client.read_core.return_value = {"content": "# User Profile\nSoftware engineer."}
mock_client.list_scenarios.return_value = {"entries": []}
result = provider.prefetch("What does the user like?")
assert result is not None
assert "Python" in result["prepend_context"]
assert "Software engineer" in result["append_system_context"]
def test_prefetch_empty_memories(self, initialized_provider):
provider, mock_client = initialized_provider
mock_client.search_atomic.return_value = {"results": []}
mock_client.read_core.return_value = {"content": ""}
mock_client.list_scenarios.return_value = {"entries": []}
result = provider.prefetch("anything")
assert result is not None
assert result["prepend_context"] == ""
class TestSyncTurn:
def test_sync_turn_calls_add_conversation(self, initialized_provider):
provider, mock_client = initialized_provider
mock_client.add_conversation.return_value = {"accepted_ids": ["m1", "m2"], "total_count": 2}
provider.sync_turn("Hello", "Hi there!", session_id="sess-1")
mock_client.add_conversation.assert_called_once()
args = mock_client.add_conversation.call_args
assert args.kwargs["session_id"] == "sess-1"
assert len(args.kwargs["messages"]) == 2
class TestTools:
def test_get_tool_schemas(self, initialized_provider):
provider, _ = initialized_provider
schemas = provider.get_tool_schemas()
names = [s["function"]["name"] for s in schemas]
assert "tdai_memory_search" in names
assert "tdai_conversation_search" in names
assert "tdai_read_scene" in names
def test_handle_memory_search(self, initialized_provider):
provider, mock_client = initialized_provider
mock_client.search_atomic.return_value = {
"results": [{"content": "Likes coffee", "type": "persona"}]
}
result = provider.handle_tool_call("tdai_memory_search", {"query": "coffee"})
assert "coffee" in result.lower()
def test_handle_read_scene(self, initialized_provider):
provider, mock_client = initialized_provider
mock_client.read_scenario.return_value = {"content": "# Travel\nGoing to Japan."}
result = provider.handle_tool_call("tdai_read_scene", {"scene_id": "travel-plan"})
assert "Japan" in result
def test_handle_unknown_tool(self, initialized_provider):
provider, _ = initialized_provider
assert provider.handle_tool_call("unknown_tool", {}) is None
class TestCircuitBreaker:
def test_circuit_opens_after_max_failures(self, initialized_provider):
provider, mock_client = initialized_provider
mock_client.search_atomic.side_effect = Exception("connection refused")
# Trigger max_failures times
for _ in range(5):
provider.prefetch("test")
assert provider._consecutive_failures >= 5
# Next call should be skipped (circuit open)
result = provider.prefetch("test")
assert result is None
def test_circuit_resets_on_success(self, initialized_provider):
provider, mock_client = initialized_provider
provider._consecutive_failures = 4
mock_client.search_atomic.return_value = {"results": []}
mock_client.read_core.return_value = {"content": ""}
mock_client.list_scenarios.return_value = {"entries": []}
provider.prefetch("test")
assert provider._consecutive_failures == 0
index.ts
+990
View File
@@ -0,0 +1,990 @@
/**
* memory-tdai v3: Four-layer memory system plugin for OpenClaw.
*
* Provides:
* - L0: Automatic conversation recording (local JSONL)
* - L1: Structured memory extraction (LLM + dedup)
* - L2: Scene block management (LLM scene extraction)
* - L3: Persona generation (LLM persona synthesis)
*
* All processing is local, zero external API dependencies.
*
* v3.1: Refactored to use TdaiCore + OpenClawHostAdapter.
* index.ts is now a thin shell that:
* - Registers tools and hooks with OpenClaw
* - Translates OpenClaw events into TdaiCore calls
* - Manages prompt caching and metric reporting
*
* Core memory logic lives in src/core/tdai-core.ts (host-neutral).
*/
import path from "node:path";
import fs from "node:fs";
import { createRequire } from "node:module";
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/core";
import { parseConfig } from "./src/config.js";
import type { MemoryTdaiConfig } from "./src/config.js";
import { executeReadCos, READ_COS_TOOL_SCHEMA, READ_COS_TOOL_NAME, READ_COS_TOOL_DESCRIPTION } from "./src/core/tools/read-cos.js";
import { LocalStorageBackend } from "./src/core/storage/local-backend.js";
import { StaticCredentialProvider } from "./src/core/storage/credential-provider.js";
import type { IStorageBackend } from "./src/core/storage/types.js";
import { registerOffload } from "./src/offload/index.js";
import {
setPreferredEmbeddedAgentRuntime,
prewarmEmbeddedAgent,
} from "./src/utils/clean-context-runner.js";
import { SessionFilter } from "./src/utils/session-filter.js";
import { LocalMemoryCleaner } from "./src/utils/memory-cleaner.js";
import { readCosToolEnvConfig } from "./src/utils/env-config.js";
import { registerMemoryTdaiCli } from "./src/cli/index.js";
import { initDataDirectories, resetStores } from "./src/utils/pipeline-factory.js";
import { getOrCreateInstanceId, initReporter, report, resetReporter } from "./src/core/report/reporter.js";
import { ensureL2L3Local } from "./src/core/profile/profile-sync.js";
// Core abstractions (host-neutral)
import { OpenClawHostAdapter } from "./src/adapters/openclaw/host-adapter.js";
import { TdaiCore } from "./src/core/tdai-core.js";
import {
ensurePluginHookPolicy,
decideHookPolicy,
} from "./src/utils/ensure-hook-policy.js";
import { resolveOpenClawStateDir } from "./src/utils/openclaw-state-dir.js";
const TAG = "[memory-tdai]";
/**
* Epoch ms when the plugin was registered (cold-start timestamp).
* Used as a fallback cursor in performAutoCapture when no checkpoint
* exists yet — prevents the first agent_end from dumping the entire
* session history into L0.
*/
let pluginStartTimestamp = 0;
/**
* Cache original user prompts and message counts across hooks.
* - text: clean user prompt before prependContext injection
* - ts: cache creation time (for TTL sweep)
* - messageCount: session message count at before_prompt_build time,
* used as fallback slice offset if timestamp cursor is unreliable
*/
const pendingOriginalPrompts = new Map<string, { text: string; ts: number; messageCount: number }>();
const PROMPT_CACHE_TTL_MS = 10 * 60 * 1000; // 10 minutes
const PROMPT_CACHE_MAX_SIZE = 10_000; // Hard limit to prevent unbounded growth in high-concurrency scenarios
/**
* Cache recall results (L1 memories + L3 Persona) from before_prompt_build
* for retrieval at agent_end, enabling the agent_turn metric event.
*
* Keyed by sessionKey — same correlation pattern as pendingOriginalPrompts.
*/
const pendingRecallCache = new Map<string, {
l1Memories: Array<{ content: string; score: number; type: string }>;
l3Persona: string | null;
strategy: string;
durationMs: number;
ts: number;
}>();
/**
* Cache recall completion timestamps per session.
* Used in agent_end to estimate LLM reasoning time:
* llmEstimatedMs ≈ agent_end_start - recall_end_ts
* Entries are cleaned up in agent_end after use; stale entries swept alongside prompt cache.
*/
const pendingRecallEndTimestamps = new Map<string, number>();
// 进程级单例，避免同一进程重复启动清理器导致并发清理竞态
let sharedMemoryCleaner: LocalMemoryCleaner | undefined;
/**
* Sweep both pendingOriginalPrompts and pendingRecallCache for stale entries.
* Unified from the original sweepStalePromptCache() to cover both Maps
* with identical TTL + hard-cap logic.
*/
function sweepStaleCaches(): void {
const now = Date.now();
// Clean pendingOriginalPrompts
for (const [key, entry] of pendingOriginalPrompts) {
if (now - entry.ts > PROMPT_CACHE_TTL_MS) {
pendingOriginalPrompts.delete(key);
pendingRecallEndTimestamps.delete(key);
}
}
// Clean pendingRecallCache
for (const [key, entry] of pendingRecallCache) {
if (now - entry.ts > PROMPT_CACHE_TTL_MS) {
pendingRecallCache.delete(key);
}
}
// Hard limit: evict oldest entries if either Map exceeds cap
if (pendingOriginalPrompts.size > PROMPT_CACHE_MAX_SIZE) {
const entries = [...pendingOriginalPrompts.entries()].sort((a, b) => a[1].ts - b[1].ts);
const toEvict = entries.slice(0, entries.length - PROMPT_CACHE_MAX_SIZE);
for (const [key] of toEvict) {
pendingOriginalPrompts.delete(key);
pendingRecallEndTimestamps.delete(key);
}
}
if (pendingRecallCache.size > PROMPT_CACHE_MAX_SIZE) {
const entries = [...pendingRecallCache.entries()].sort((a, b) => a[1].ts - b[1].ts);
const toEvict = entries.slice(0, entries.length - PROMPT_CACHE_MAX_SIZE);
for (const [key] of toEvict) {
pendingRecallCache.delete(key);
}
}
}
export default function register(api: OpenClawPluginApi) {
// ─── CLI metadata mode: register CLI commands only, skip all runtime init ───
// In this mode, runtime is `{} as PluginRuntime` (empty object).
// OpenClaw calls this to discover CLI subcommands without starting the full plugin.
if (api.registrationMode === "cli-metadata") {
api.registerCli(
({ program, config, logger: cliLogger }) => {
const memoryTdai = program
.command("memory-tdai")
.description("memory-tdai plugin commands (seed, query, stats)");
registerMemoryTdaiCli(memoryTdai, {
config,
pluginConfig: api.pluginConfig,
stateDir: resolveOpenClawStateDir((api.runtime as any)?.state),
logger: cliLogger,
});
},
{ commands: ["memory-tdai"] },
);
return;
}
// ─── Full / discovery mode: complete runtime initialization ───
pluginStartTimestamp = Date.now();
setPreferredEmbeddedAgentRuntime(api.runtime.agent);
// Reset reporter singleton so config changes take effect on hot-reload.
resetReporter();
const _require = createRequire(import.meta.url);
const pluginVersion = (() => { try { return (_require("./package.json") as { version?: string }).version ?? "unknown"; } catch { return "unknown"; } })();
api.logger.debug?.(
`${TAG} Registering plugin ... ` +
`startTimestamp=${pluginStartTimestamp} (${new Date(pluginStartTimestamp).toISOString()})`,
);
let cfg: MemoryTdaiConfig;
try {
// OpenClaw calls register() N times (plugin scan → gateway start →
// per-channel bootstrap → config reload). Each call receives the full
// pluginConfig from openclaw.json, so we parse it directly every time.
const rawPluginConfig = api.pluginConfig as Record<string, unknown> | undefined;
const rawKeys = rawPluginConfig ? Object.keys(rawPluginConfig) : [];
api.logger.debug?.(
`${TAG} pluginConfig received (${rawKeys.length} keys)`,
);
cfg = parseConfig(rawPluginConfig);
api.logger.debug?.(
`${TAG} Config parsed: ` +
`capture=${cfg.capture.enabled}, ` +
`recall=${cfg.recall.enabled}(maxResults=${cfg.recall.maxResults}), ` +
`extraction=${cfg.extraction.enabled}(dedup=${cfg.extraction.enableDedup}, maxMem=${cfg.extraction.maxMemoriesPerSession}), ` +
`pipeline=(everyN=${cfg.pipeline.everyNConversations}, warmup=${cfg.pipeline.enableWarmup}, l1Idle=${cfg.pipeline.l1IdleTimeoutSeconds}s, l2DelayAfterL1=${cfg.pipeline.l2DelayAfterL1Seconds}s, l2Min=${cfg.pipeline.l2MinIntervalSeconds}s, l2Max=${cfg.pipeline.l2MaxIntervalSeconds}s, activeWindow=${cfg.pipeline.sessionActiveWindowHours}h), ` +
`persona(triggerEvery=${cfg.persona.triggerEveryN}, backupCount=${cfg.persona.backupCount}, sceneBackupCount=${cfg.persona.sceneBackupCount}), ` +
`memoryCleanup(enabled=${cfg.memoryCleanup.enabled}, retentionDays=${cfg.memoryCleanup.retentionDays ?? "(disabled)"}, cleanTime=${cfg.memoryCleanup.cleanTime}), ` +
`offload(enabled=${cfg.offload.enabled}, backendUrl=${cfg.offload.backendUrl ?? "(none)"}, mildRatio=${cfg.offload.mildOffloadRatio}, aggressiveRatio=${cfg.offload.aggressiveCompressRatio}, retentionDays=${cfg.offload.offloadRetentionDays})`,
);
} catch (err) {
api.logger.error(`${TAG} Config parsing failed: ${err instanceof Error ? err.message : String(err)}`);
throw err;
}
// ============================
// Hook policy auto-patch (v2026.4.24+ compat)
// ============================
// `allowConversationAccess` hook policy was introduced in v2026.4.23;
// the zod schema fix landed in v2026.4.24. Older hosts don't understand
// the field and don't need it patched in.
//
// Note: `api.runtime.version` is only exposed on v2026.4.15+. On older
// hosts it is `undefined`; we MUST treat that as "does not need the
// patch" (old hosts have no gate), otherwise we would silently mutate
// the user's openclaw.json on every gateway start.
{
// Gate: only apply the auto-patch when host version >= 2026.4.24.
// decideHookPolicy() parses the leading x.y.z prefix numerically
// (ignoring `-beta.N`, `-N`, etc.) and returns apply=false for any
// version we cannot parse — which is the safe default on old hosts
// that don't expose `api.runtime.version`. See ensure-hook-policy.ts
// for the full policy + co-located unit tests.
const rawVersion = (api.runtime as any)?.version;
const decision = decideHookPolicy(rawVersion);
const parsedStr = decision.parsedXYZ ? decision.parsedXYZ.join(".") : "<unparsable>";
const minStr = decision.minXYZ.join(".");
if (!decision.apply) {
api.logger.debug?.(
`${TAG} Hook policy auto-patch skipped: ` +
`original=${JSON.stringify(rawVersion)}, parsed=${parsedStr}, min=${minStr}`,
);
} else {
api.logger.debug?.(
`${TAG} Hook policy auto-patch applying: ` +
`original=${JSON.stringify(rawVersion)}, parsed=${parsedStr} >= min=${minStr}`,
);
try {
ensurePluginHookPolicy({
rootConfig: api.config,
runtimeConfig: api.runtime?.config,
logger: api.logger,
});
} catch (err) {
api.logger.warn(`${TAG} Hook policy check failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`);
}
}
}
// If remote embedding config is incomplete, log a prominent error so the user knows
if (cfg.embedding.configError) {
api.logger.error(`${TAG} [EMBEDDING CONFIG ERROR] ${cfg.embedding.configError}`);
}
// Resolve plugin data directory via runtime API (avoid importing internal paths directly)
const openclawStateDir = resolveOpenClawStateDir((api.runtime as any)?.state);
const pluginDataDir = path.join(openclawStateDir, "memory-tdai");
initDataDirectories(pluginDataDir);
api.logger.debug?.(`${TAG} Data dir: ${pluginDataDir} (all subdirectories initialized)`);
// ============================
// Create OpenClawHostAdapter + TdaiCore
// ============================
const hostAdapter = new OpenClawHostAdapter({
api,
pluginDataDir,
openclawConfig: api.config,
});
const sessionFilter = new SessionFilter(cfg.capture.excludeAgents);
if (cfg.capture.excludeAgents.length > 0) {
api.logger.debug?.(`${TAG} Agent exclude patterns: ${cfg.capture.excludeAgents.join(", ")}`);
}
const core = new TdaiCore({
hostAdapter,
config: cfg,
sessionFilter,
});
// Initialize TdaiCore (async — store init, pipeline wiring)
const coreReady = core.initialize().then(() => {
// Keep cleaner's SQLite handle updated after store init
memoryCleaner?.setVectorStore(core.getVectorStore());
// Pull L2/L3 profiles if remote store supports it
const vs = core.getVectorStore();
if (vs?.pullProfiles) {
ensureL2L3Local(pluginDataDir, vs, api.logger).catch((err) => {
api.logger.warn(`${TAG} Startup L2/L3 pull failed (non-fatal): ${err instanceof Error ? err.message : String(err)}`);
});
}
}).catch((err) => {
api.logger.error(`${TAG} Core init failed: ${err instanceof Error ? err.message : String(err)}`);
});
// Kick off instanceId resolution immediately after data dir is ready.
let instanceId: string | undefined;
getOrCreateInstanceId(pluginDataDir).then((id) => {
instanceId = id;
core.setInstanceId(id);
initReporter({ enabled: cfg.report.enabled, type: cfg.report.type, logger: api.logger, instanceId: id, pluginVersion });
}).catch((err) => {
api.logger.warn(`${TAG} Failed to initialize instanceId for metrics: ${err instanceof Error ? err.message : String(err)}`);
});
// Daily local JSONL cleaner (L0/L1), enabled only when retentionDays is configured.
let memoryCleaner: LocalMemoryCleaner | undefined;
if (cfg.memoryCleanup.enabled && cfg.memoryCleanup.retentionDays != null) {
if (!sharedMemoryCleaner) {
sharedMemoryCleaner = new LocalMemoryCleaner({
baseDir: pluginDataDir,
retentionDays: cfg.memoryCleanup.retentionDays,
cleanTime: cfg.memoryCleanup.cleanTime,
logger: api.logger,
});
sharedMemoryCleaner.start();
api.logger.debug?.(`${TAG} Memory cleaner started (singleton)`);
} else {
api.logger.debug?.(`${TAG} Memory cleaner already started in this process, reusing existing instance`);
}
memoryCleaner = sharedMemoryCleaner;
} else {
api.logger.debug?.(`${TAG} Memory cleaner disabled (retentionDays not configured)`);
}
const resolveSessionKey = (sessionKey?: string): string | undefined => {
if (sessionKey) return sessionKey;
api.logger.warn(`${TAG} sessionKey is empty, skipping capture/recall to avoid unstable fallback key`);
return undefined;
};
/**
* Whether embedding warmup has been triggered.
* Deferred until first real conversation to avoid model downloads during CLI commands.
*/
let embeddingWarmupTriggered = false;
const ensureEmbeddingWarmup = (): void => {
const svc = core.getEmbeddingService();
if (!svc) return;
if (!embeddingWarmupTriggered) {
embeddingWarmupTriggered = true;
api.logger.debug?.(`${TAG} Triggering lazy embedding warmup on first conversation`);
svc.startWarmup();
return;
}
if (!svc.isReady()) {
api.logger.debug?.(`${TAG} Embedding not ready, re-triggering warmup (retry)`);
svc.startWarmup();
}
};
// ============================
// Tool registration — delegate to TdaiCore
// ============================
// tdai_memory_search — Agent-callable L1 memory search tool
// TODO: implement hard per-turn call limit via before_tool_call hook + execute early-return (方案 D)
if (cfg.recall.enabled || cfg.capture.enabled) {
api.registerTool(
{
name: "tdai_memory_search",
label: "Memory Search",
description:
"Search through the user's long-term memories. Use this when you need to recall specific information about the user's preferences, past events, instructions, or context from previous conversations. Returns relevant memory records ranked by relevance. " +
"Limit: tdai_memory_search and tdai_conversation_search share a combined limit of 3 calls per turn. Stop searching after 3 total attempts.",
parameters: {
type: "object",
properties: {
query: {
type: "string",
description: "Search query describing what you want to recall about the user",
},
limit: {
type: "number",
description: "Maximum number of results to return (default: 5, max: 20)",
},
type: {
type: "string",
enum: ["persona", "episodic", "instruction"],
description: "Optional filter by memory type: persona (identity/preferences), episodic (events/activities), instruction (user rules/commands)",
},
scene: {
type: "string",
description: "Optional filter by scene name",
},
},
required: ["query"],
},
async execute(_toolCallId: string, params: Record<string, unknown>) {
const startMs = Date.now();
const query = String(params.query ?? "");
const limit = Math.min(Math.max(Number(params.limit) || 5, 1), 20);
const typeFilter = typeof params.type === "string" ? params.type : undefined;
const sceneFilter = typeof params.scene === "string" ? params.scene : undefined;
api.logger.debug?.(
`${TAG} [tool] tdai_memory_search called: ` +
`query="${query.length > 80 ? query.slice(0, 80) + "…" : query}", ` +
`limit=${limit}, type=${typeFilter ?? "(all)"}, scene=${sceneFilter ?? "(all)"}`,
);
try {
const result = await core.searchMemories({ query, limit, type: typeFilter, scene: sceneFilter });
const elapsedMs = Date.now() - startMs;
api.logger.debug?.(
`${TAG} [tool] tdai_memory_search completed (${elapsedMs}ms): ` +
`total=${result.total}, strategy=${result.strategy}, ` +
`responseLength=${result.text.length} chars`,
);
report("tool_call", {
tool: "tdai_memory_search",
query, limit, typeFilter, sceneFilter,
resultCount: result.total,
strategy: result.strategy,
durationMs: elapsedMs,
success: true,
});
return {
content: [{ type: "text" as const, text: result.text }],
details: { count: result.total, strategy: result.strategy },
};
} catch (err) {
const elapsedMs = Date.now() - startMs;
const errMsg = err instanceof Error ? err.message : String(err);
api.logger.error(`${TAG} [tool] tdai_memory_search failed (${elapsedMs}ms): ${errMsg}`);
report("tool_call", {
tool: "tdai_memory_search",
query, limit, typeFilter, sceneFilter,
durationMs: elapsedMs,
success: false,
error: errMsg,
});
return {
content: [{ type: "text" as const, text: `Memory search failed: ${errMsg}` }],
details: { error: errMsg },
};
}
},
},
{ name: "tdai_memory_search" },
);
// tdai_conversation_search — Agent-callable L0 conversation search tool
// TODO: implement hard per-turn call limit via before_tool_call hook + execute early-return (方案 D)
api.registerTool(
{
name: "tdai_conversation_search",
label: "Conversation Search",
description:
"Search through past conversation history (raw dialogue records). " +
"Use this when tdai_memory_search (structured memories) doesn't have the information you need, " +
"or when you want to find specific past conversations, dialogue context, or exact words " +
"the user said before. Returns relevant individual messages ranked by relevance. " +
"Limit: tdai_memory_search and tdai_conversation_search share a combined limit of 3 calls per turn. Stop searching after 3 total attempts.",
parameters: {
type: "object",
properties: {
query: {
type: "string",
description: "Search query describing what conversation content you want to find",
},
limit: {
type: "number",
description: "Maximum number of messages to return (default: 5, max: 20)",
},
session_key: {
type: "string",
description: "Optional: filter results to a specific session",
},
},
required: ["query"],
},
async execute(_toolCallId: string, params: Record<string, unknown>) {
const startMs = Date.now();
const query = String(params.query ?? "");
const limit = Math.min(Math.max(Number(params.limit) || 5, 1), 20);
const sessionKeyFilter = typeof params.session_key === "string" ? params.session_key : undefined;
api.logger.debug?.(
`${TAG} [tool] tdai_conversation_search called: ` +
`query="${query.length > 80 ? query.slice(0, 80) + "…" : query}", ` +
`limit=${limit}, session_key=${sessionKeyFilter ?? "(all)"}`,
);
try {
const result = await core.searchConversations({ query, limit, sessionKey: sessionKeyFilter });
const elapsedMs = Date.now() - startMs;
api.logger.debug?.(
`${TAG} [tool] tdai_conversation_search completed (${elapsedMs}ms): ` +
`total=${result.total}, responseLength=${result.text.length} chars`,
);
report("tool_call", {
tool: "tdai_conversation_search",
query, limit, sessionKeyFilter,
resultCount: result.total,
durationMs: elapsedMs,
success: true,
});
return {
content: [{ type: "text" as const, text: result.text }],
details: { count: result.total },
};
} catch (err) {
const elapsedMs = Date.now() - startMs;
const errMsg = err instanceof Error ? err.message : String(err);
api.logger.error(`${TAG} [tool] tdai_conversation_search failed (${elapsedMs}ms): ${errMsg}`);
report("tool_call", {
tool: "tdai_conversation_search",
query, limit, sessionKeyFilter,
durationMs: elapsedMs,
success: false,
error: errMsg,
});
return {
content: [{ type: "text" as const, text: `Conversation search failed: ${errMsg}` }],
details: { error: errMsg },
};
}
},
},
{ name: "tdai_conversation_search" },
);
// tdai_read_cos — Agent-callable tool for reading files from COS/local storage
{
// Read COS config from {pluginDataDir}/cos.env
// Format: KEY=VALUE per line (standard .env)
const cosEnvPath = path.join(pluginDataDir, "cos.env");
const cosEnv: Record<string, string> = {};
if (fs.existsSync(cosEnvPath)) {
const lines = fs.readFileSync(cosEnvPath, "utf-8").split("\n");
for (const line of lines) {
const trimmed = line.trim();
if (!trimmed || trimmed.startsWith("#")) continue;
const eqIdx = trimmed.indexOf("=");
if (eqIdx > 0) {
cosEnv[trimmed.slice(0, eqIdx).trim()] = trimmed.slice(eqIdx + 1).trim();
}
}
}
// Priority: env vars > cos.env file
const {
cosSecretId,
cosSecretKey,
cosBucket,
cosRegion,
cosPrefix,
cosDomain,
} = readCosToolEnvConfig(cosEnv);
// Lazily resolve the storage backend on first call. Optional COS support
// is loaded dynamically; when unavailable, the tool falls back to local
// storage. The resolution is cached so subsequent calls are zero-cost.
let storagePromise: Promise<IStorageBackend> | null = null;
const resolveReadCosStorage = (): Promise<IStorageBackend> => {
if (storagePromise) return storagePromise;
storagePromise = (async (): Promise<IStorageBackend> => {
if (cosSecretId && cosSecretKey && cosBucket) {
try {
const { CosStorageBackend } = await import("./src/integrations/cos/cos-backend.js");
const backend = new CosStorageBackend({
credentialProvider: new StaticCredentialProvider({
secretId: cosSecretId,
secretKey: cosSecretKey,
bucket: cosBucket,
region: cosRegion,
prefix: cosPrefix,
}),
logger: api.logger,
cosEndpointDomain: cosDomain,
});
api.logger.info(`${TAG} read_cos: using COS backend (bucket=${cosBucket}, prefix=${cosPrefix})`);
return backend;
} catch (err) {
const storageDataDir = path.join(pluginDataDir, "cos-storage");
const backend = new LocalStorageBackend({ rootDir: storageDataDir, logger: api.logger });
api.logger.warn(
`${TAG} read_cos: COS integration not available (${err instanceof Error ? err.message : String(err)}), ` +
`falling back to local backend (${storageDataDir}).`,
);
return backend;
}
}
const storageDataDir = path.join(pluginDataDir, "cos-storage");
const backend = new LocalStorageBackend({ rootDir: storageDataDir, logger: api.logger });
api.logger.info(`${TAG} read_cos: no COS credentials, using local backend (${storageDataDir}). Place cos.env in ${pluginDataDir} to enable COS.`);
return backend;
})();
return storagePromise;
};
api.registerTool(
{
name: READ_COS_TOOL_NAME,
label: "Read COS File",
description: READ_COS_TOOL_DESCRIPTION,
parameters: READ_COS_TOOL_SCHEMA,
async execute(_toolCallId: string, params: Record<string, unknown>) {
const startMs = Date.now();
const filePath = String(params.path ?? "");
api.logger.debug?.(`${TAG} [tool] tdai_read_cos called: path="${filePath}"`);
const readCosStorage = await resolveReadCosStorage();
const result = await executeReadCos(
{ path: filePath },
readCosStorage,
api.logger,
);
const elapsedMs = Date.now() - startMs;
api.logger.debug?.(
`${TAG} [tool] tdai_read_cos completed (${elapsedMs}ms): ` +
`success=${result.success}, size=${result.size}`,
);
report("tool_call", {
tool: "tdai_read_cos",
path: filePath,
durationMs: elapsedMs,
success: result.success,
size: result.size,
error: result.error,
});
if (result.success) {
return {
content: [{ type: "text" as const, text: result.content }],
details: { path: result.path, size: result.size },
};
} else {
return {
content: [{ type: "text" as const, text: `Read failed: ${result.error}` }],
details: { path: result.path, error: result.error },
};
}
},
},
{ name: READ_COS_TOOL_NAME },
);
api.logger.info(`${TAG} Registered tool: tdai_read_cos (storage backend resolved lazily on first call)`);
}
} else {
api.logger.debug?.(`${TAG} Memory tools (tdai_memory_search, tdai_conversation_search) not registered — memory features disabled`);
}
// ============================
// Lifecycle hooks — delegate to TdaiCore
// ============================
// Before prompt build: auto-recall relevant memories
if (cfg.recall.enabled) {
api.logger.debug?.(`${TAG} Registering before_prompt_build hook (auto-recall)`);
api.on("before_prompt_build", async (event, ctx) => {
const startMs = Date.now();
api.logger.debug?.(`${TAG} [before_prompt_build] Hook triggered`);
const sessionKey = ctx.sessionKey;
if (sessionFilter.shouldSkipCtx(ctx)) {
api.logger.debug?.(`${TAG} [before_prompt_build] Skipping filtered session`);
return;
}
ensureEmbeddingWarmup();
// Cache original user prompt for agent_end
const rawPrompt = event.prompt;
const messages = Array.isArray(event.messages) ? event.messages : undefined;
if (sessionKey && rawPrompt) {
const messageCount = messages?.length ?? 0;
pendingOriginalPrompts.set(sessionKey, { text: rawPrompt, ts: Date.now(), messageCount });
api.logger.debug?.(`${TAG} [before_prompt_build] Cached original prompt (${rawPrompt.length} chars, msgCount=${messageCount})`);
}
sweepStaleCaches();
const userText = rawPrompt;
api.logger.debug?.(`${TAG} [before_prompt_build] userText length: ${userText?.length}`);
if (!userText) {
api.logger.debug?.(`${TAG} [before_prompt_build] No user text found, skipping recall`);
return;
}
const resolvedSessionKey = resolveSessionKey(sessionKey);
if (!resolvedSessionKey) {
return;
}
try {
await coreReady;
const recallStartMs = Date.now();
const result = await core.handleBeforeRecall(userText, resolvedSessionKey);
const elapsedMs = Date.now() - startMs;
const recallDurationMs = Date.now() - recallStartMs;
// Cache recall results for agent_turn metric (retrieved at agent_end)
if (sessionKey && result) {
pendingRecallCache.set(sessionKey, {
l1Memories: result.recalledL1Memories ?? [],
l3Persona: result.recalledL3Persona ?? null,
strategy: result.recallStrategy ?? "unknown",
durationMs: recallDurationMs,
ts: Date.now(),
});
}
// Record recall completion timestamp for LLM timing estimation in agent_end
if (resolvedSessionKey) {
pendingRecallEndTimestamps.set(resolvedSessionKey, Date.now());
}
if (result?.appendSystemContext || result?.prependContext) {
const appendLen = result.appendSystemContext?.length ?? 0;
const prependLen = result.prependContext?.length ?? 0;
api.logger.info(
`${TAG} [before_prompt_build] Recall complete (${elapsedMs}ms), ` +
`appendSystemContext=${appendLen} chars, prependContext=${prependLen} chars`,
);
} else {
api.logger.info(`${TAG} [before_prompt_build] Recall complete (${elapsedMs}ms), no context to inject`);
}
return result;
} catch (err) {
const elapsedMs = Date.now() - startMs;
api.logger.error(`${TAG} [before_prompt_build] Auto-recall failed after ${elapsedMs}ms: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
if (instanceId) {
report("error_degradation", {
module: "auto-recall",
action: "performAutoRecall",
errorType: "exception",
errorMessage: err instanceof Error ? err.message : String(err),
degradedTo: "no_recall",
impact: "non-blocking",
});
}
}
});
}
// Strip <relevant-memories> from user messages before they are persisted to
// the session JSONL. The current-turn LLM already saw the full prompt
// (effectivePrompt lives in memory), but we don't want recall artifacts
// polluting the historical transcript for future replays.
api.logger.debug?.(`${TAG} Registering before_message_write hook (strip <relevant-memories>)`);
api.on("before_message_write", (event) => {
const msg = event.message as { role?: string; content?: unknown };
const contentType = typeof msg.content === "string" ? "string" : Array.isArray(msg.content) ? "parts" : typeof msg.content;
api.logger.debug?.(`${TAG} [before_message_write] role=${msg.role}, contentType=${contentType}`);
if (msg.role !== "user") return;
// UserMessage.content: string | (TextContent | ImageContent)[]
const STRIP_RE = /<relevant-memories>[\s\S]*?<\/relevant-memories>\s*/g;
if (typeof msg.content === "string") {
if (!msg.content.includes("<relevant-memories>")) return;
const cleaned = msg.content.replace(STRIP_RE, "").trim();
if (cleaned === msg.content) return;
api.logger.debug?.(`${TAG} [before_message_write] Stripped: ${msg.content.length} → ${cleaned.length} chars`);
return { message: { ...event.message, content: cleaned } as typeof event.message };
}
if (Array.isArray(msg.content)) {
let totalStripped = 0;
const cleanedParts = (msg.content as Array<Record<string, unknown>>).map((part) => {
if (part.type !== "text" || typeof part.text !== "string") return part;
if (!(part.text as string).includes("<relevant-memories>")) return part;
const cleaned = (part.text as string).replace(STRIP_RE, "").trim();
totalStripped += (part.text as string).length - cleaned.length;
return { ...part, text: cleaned };
});
if (totalStripped === 0) return;
api.logger.debug?.(`${TAG} [before_message_write] Stripped from parts: removed ${totalStripped} chars`);
return { message: { ...event.message, content: cleanedParts } as unknown as typeof event.message };
}
});
// After agent end: auto-capture + L0 record + L1/L2/L3 schedule
if (cfg.capture.enabled) {
api.logger.debug?.(`${TAG} Registering agent_end hook (auto-capture)`);
api.on("agent_end", async (event, ctx) => {
const startMs = Date.now();
api.logger.debug?.(`${TAG} [agent_end] Hook triggered`);
const e = event as Record<string, unknown>;
if (!e.success) {
api.logger.info(`${TAG} [agent_end] Agent did not succeed, skipping capture`);
return;
}
const sessionKey = ctx.sessionKey;
const sessionId = ctx.sessionId;
if (sessionFilter.shouldSkipCtx(ctx)) {
api.logger.debug?.(`${TAG} [agent_end] Skipping filtered session`);
return;
}
const messages = (e.messages as unknown[]) ?? [];
const resolvedSessionKey = resolveSessionKey(sessionKey);
if (!resolvedSessionKey) {
return;
}
// Estimate LLM reasoning time: recallEnd → agentEnd start
const recallEndTs = pendingRecallEndTimestamps.get(resolvedSessionKey);
if (recallEndTs) {
const llmEstimatedMs = startMs - recallEndTs;
api.logger.info(
`${TAG} ⏱ Turn timing: recallEnd→agentEnd=${llmEstimatedMs}ms ` +
`(≈ LLM reasoning + prompt build + tool calls)`,
);
pendingRecallEndTimestamps.delete(resolvedSessionKey);
}
// Retrieve cached original prompt
const cachedPrompt = sessionKey ? pendingOriginalPrompts.get(sessionKey) : undefined;
const originalUserText = cachedPrompt?.text;
try {
await coreReady;
// Pre-warm the embedded agent on first conversation
if (!core.isSchedulerStarted()) {
prewarmEmbeddedAgent(api.logger, api.runtime.agent);
}
const captureResult = await core.handleTurnCommitted({
userText: originalUserText ?? "",
assistantText: "",
messages,
sessionKey: resolvedSessionKey,
sessionId: sessionId || undefined,
startedAt: pluginStartTimestamp,
originalUserMessageCount: cachedPrompt?.messageCount,
});
const captureMs = Date.now() - startMs;
api.logger.info(
`${TAG} [agent_end] Auto-capture complete (${captureMs}ms), ` +
`l0Recorded=${captureResult.l0RecordedCount}, ` +
`schedulerNotified=${captureResult.schedulerNotified}`,
);
// ── agent_turn metric ──
const cachedRecall = sessionKey ? pendingRecallCache.get(sessionKey) : undefined;
if (sessionKey) pendingRecallCache.delete(sessionKey);
if (instanceId) {
report("agent_turn", {
sessionKey: resolvedSessionKey,
userPrompt: originalUserText ?? null,
recalledL1Memories: cachedRecall?.l1Memories ?? [],
recalledL1Count: cachedRecall?.l1Memories?.length ?? 0,
recalledL3Persona: cachedRecall?.l3Persona ?? null,
recallStrategy: cachedRecall?.strategy ?? null,
recallDurationMs: cachedRecall?.durationMs ?? 0,
l0CapturedMessages: captureResult.filteredMessages.map((m) => ({
role: m.role,
content: m.content,
ts: m.timestamp,
})),
l0CapturedCount: captureResult.l0RecordedCount,
l0VectorsWritten: captureResult.l0VectorsWritten,
captureDurationMs: captureMs,
totalDurationMs: Date.now() - startMs,
});
}
} catch (err) {
const elapsedMs = Date.now() - startMs;
api.logger.error(`${TAG} [agent_end] Auto-capture failed after ${elapsedMs}ms: ${err instanceof Error ? err.stack ?? err.message : String(err)}`);
if (instanceId) {
report("error_degradation", {
module: "auto-capture",
action: "performAutoCapture",
errorType: "exception",
errorMessage: err instanceof Error ? err.message : String(err),
degradedTo: "no_capture",
impact: "non-blocking",
});
}
}
});
// gateway_stop: ordered shutdown via TdaiCore.destroy()
api.on("gateway_stop", async () => {
const GATEWAY_STOP_TIMEOUT_MS = 3_000;
const hookStartMs = Date.now();
await coreReady.catch(() => {});
const doCleanup = async (): Promise<void> => {
// 1. Stop memory cleaner first
if (memoryCleaner) {
try {
memoryCleaner.destroy();
if (sharedMemoryCleaner === memoryCleaner) {
sharedMemoryCleaner = undefined;
}
} catch (error) {
api.logger.error(`${TAG} [gateway_stop] memoryCleaner error: ${error instanceof Error ? error.message : String(error)}`);
}
}
// 2. Destroy TdaiCore (scheduler flush + VectorStore close + EmbeddingService close)
await core.destroy();
};
// Race cleanup against a hard timeout
let timeoutId: ReturnType<typeof setTimeout> | undefined;
try {
await Promise.race([
doCleanup(),
new Promise<never>((_, reject) => {
timeoutId = setTimeout(
() => reject(new Error("timeout")),
GATEWAY_STOP_TIMEOUT_MS,
);
}),
]);
} catch (err) {
api.logger.warn(
`${TAG} [gateway_stop] Aborted (${Date.now() - hookStartMs}ms): ${err instanceof Error ? err.message : String(err)}. ` +
`Pending work will recover on next startup.`,
);
} finally {
if (timeoutId !== undefined) clearTimeout(timeoutId);
}
resetStores();
api.logger.info(`${TAG} [gateway_stop] Cleanup finished, all resources released (${Date.now() - hookStartMs}ms)`);
});
} else {
api.logger.debug?.(`${TAG} Auto-capture disabled`);
}
// memoryCleaner gateway_stop for capture-enabled-but-extraction-disabled case
if (memoryCleaner && !cfg.extraction.enabled) {
api.on("gateway_stop", async () => {
const startMs = Date.now();
try {
memoryCleaner?.destroy();
if (sharedMemoryCleaner === memoryCleaner) {
sharedMemoryCleaner = undefined;
}
api.logger.info(`${TAG} [gateway_stop] Memory cleaner destroyed (${Date.now() - startMs}ms)`);
} catch (error) {
api.logger.error(`${TAG} [gateway_stop] Error during memory cleaner destruction (${Date.now() - startMs}ms): ${error instanceof Error ? error.message : String(error)}`);
}
});
}
// ============================
// Context Offload (conditional)
// ============================
if (cfg.offload.enabled) {
api.logger.debug?.(`${TAG} Offload enabled, registering offload module...`);
try {
registerOffload(api, cfg.offload);
api.logger.debug?.(`${TAG} Offload module registered successfully`);
} catch (err) {
api.logger.error(`${TAG} Offload module registration failed: ${err instanceof Error ? err.message : String(err)}`);
}
} else {
api.logger.debug?.(`${TAG} Offload disabled (offload.enabled=false)`);
}
// ============================
// CLI registration
// ============================
api.registerCli(
({ program, config, logger: cliLogger }) => {
const memoryTdai = program
.command("memory-tdai")
.description("memory-tdai plugin commands (seed, query, stats)");
registerMemoryTdaiCli(memoryTdai, {
config,
pluginConfig: api.pluginConfig,
stateDir: openclawStateDir,
logger: cliLogger,
});
},
{ commands: ["memory-tdai"] },
);
api.logger.debug?.(
`${TAG} Plugin registration complete (v3.1 — TdaiCore). ` +
`startTimestamp=${pluginStartTimestamp} (${new Date(pluginStartTimestamp).toISOString()})`,
);
}
kubb.config.ts
+35
View File
@@ -0,0 +1,35 @@
import { defineConfig } from "@kubb/core";
import { pluginOas } from "@kubb/plugin-oas";
import { pluginTs } from "@kubb/plugin-ts";
import { pluginZod } from "@kubb/plugin-zod";
export default defineConfig({
root: ".",
input: {
path: "./docs/plans/server/01-api-spec.yaml",
},
output: {
path: "./src/gateway/generated",
clean: true,
barrelType: false,
},
plugins: [
pluginOas({
generators: [],
}),
pluginTs({
output: {
path: "./types.ts", // single file
barrelType: false,
},
}),
pluginZod({
output: {
path: "./schemas.ts", // single file
barrelType: false,
},
typed: true,
importPath: "zod",
}),
],
});
openclaw-plugin/.gitignore
+4
View File
@@ -0,0 +1,4 @@
node_modules/
dist/
.env
*.tgz
openclaw-plugin/README.md
+223
View File
@@ -0,0 +1,223 @@
# OpenClaw Adapter for TencentDB Agent Memory
[简体中文](./README_CN.md) · English
This directory is a reference implementation for integrating an Agent framework with TencentDB Agent Memory v2 API. It is an OpenClaw client plugin: it does not run extraction, indexing, scene generation, or persona generation itself. Instead, it connects to an already-running Memory Gateway and uses the TypeScript SDK to capture conversations, recall memories, and expose memory tools to the Agent.
For standalone local usage, the recommended Memory Gateway endpoint is `http://127.0.0.1:8420`. The default local convention is `apiKey = "local"` and `serviceId = "default"`. If your Gateway enables `TDAI_GATEWAY_API_KEY`, use the same value as `server.apiKey`.
## Architecture
```text
OpenClaw runtime
└─ memory-tencentdb-client plugin
├─ hooks/capture.ts agent_end -> addConversation (L0)
├─ hooks/recall.ts before_prompt_build -> search + prompt injection
├─ tools/memory-search.ts tdai_memory_search -> searchAtomic (L1)
├─ tools/conversation-search.ts tdai_conversation_search -> searchConversation (L0)
└─ tools/read-cos.ts tdai_read_cos -> read L2/L3 artifacts
│
▼
@tencentdb-agent-memory/memory-sdk-ts
│ HTTP v2 API
▼
TencentDB Agent Memory Gateway (:8420 standalone, or remote service)
```
## Quick Start
Recommended: run the installer from the repository root:
```bash
bash scripts/install-openclaw-plugin-v2.sh
```
The script checks/installs the OpenClaw CLI, installs plugin dependencies, builds the plugin, and installs the current `openclaw-plugin` directory into OpenClaw through `openclaw plugins install -l`. By default it also updates `~/.openclaw/openclaw.json`: it sets `plugins.slots.memory = "memory-tencentdb-client"`, enables the plugin, and writes the standalone defaults for `server`, `recall`, and `capture`. The script auto-detects the OpenClaw version: on **2026.4.24+** it also writes `hooks.allowPromptInjection` / `hooks.allowConversationAccess` (required by non-bundled plugins to enable conversation capture); on **older versions** these fields are **omitted**, because gateways before 2026.4.24 use a strict zod schema that refuses to start with these fields present.
You can override defaults with environment variables: `OPENCLAW_CONFIG_FILE`, `TDAI_MEMORY_ENDPOINT`, `TDAI_MEMORY_API_KEY`, `TDAI_MEMORY_INSTANCE_ID`, `TDAI_MEMORY_RECALL_MAX_RESULTS`, and `TDAI_MEMORY_CAPTURE_ENABLED`. Set `WRITE_OPENCLAW_CONFIG=0` if you only want to install the plugin without modifying OpenClaw config.
For manual installation, follow the steps below.
### 1. Install OpenClaw
If OpenClaw is not installed yet, install the OpenClaw CLI first:
```bash
curl -fsSL https://get.openclaw.dev | bash
```
Verify that the command is available:
```bash
openclaw --version
```
### 2. Install dependencies
```bash
cd openclaw-plugin
npm install
```
The plugin depends on `@tencentdb-agent-memory/memory-sdk-ts`, the TypeScript SDK for the v2 API.
### 3. Build
```bash
npm run build
```
### 4. Install the plugin into OpenClaw
For local source development, install the current directory as a linked plugin:
```bash
openclaw plugins install -l .
```
### 5. Configure the plugin
If you use the `agentmemory/openclaw-memory` container image, the entrypoint already generates this configuration for you, so you usually do not need to edit it manually.
If you use `scripts/install-openclaw-plugin-v2.sh`, the script writes this configuration by default as well, and version-gates the `hooks.*` policy fields automatically. You only need to edit the OpenClaw config yourself when running with `WRITE_OPENCLAW_CONFIG=0` or when installing manually.
> ⚠️ **Important — `hooks.*` is version-gated.**
> The `hooks.allowPromptInjection` / `hooks.allowConversationAccess` fields were added to the gateway schema in **OpenClaw `2026.4.24`**. Earlier versions (including `2026.4.23`) use a strict zod schema and will **refuse to start** if these fields are present. **Pick the example that matches your installed OpenClaw version.** Run `openclaw --version` to check.
**Example A — OpenClaw `>= 2026.4.24` (recommended):**
```jsonc
{
"plugins": {
"slots": {
"memory": "memory-tencentdb-client"
},
"entries": {
"memory-tencentdb-client": {
"enabled": true,
// hooks.* is REQUIRED on >= 2026.4.24 for non-bundled plugins.
// Without allowConversationAccess=true, L0 capture is silently blocked.
"hooks": {
"allowPromptInjection": true,
"allowConversationAccess": true
},
"config": {
"server": {
"url": "http://127.0.0.1:8420",
"apiKey": "local",
"instanceId": "default"
},
"recall": {
"maxResults": 5,
"includePersona": true,
"includeSceneNav": true
},
"capture": {
"enabled": true
}
}
}
}
}
}
```
**Example B — OpenClaw `< 2026.4.24` (e.g. `2026.4.23`): omit the `hooks` block entirely.**
```jsonc
{
"plugins": {
"slots": {
"memory": "memory-tencentdb-client"
},
"entries": {
"memory-tencentdb-client": {
"enabled": true,
// ⚠️ DO NOT add a "hooks" block here. The strict schema in 2026.4.23 and
// earlier rejects allowPromptInjection / allowConversationAccess and the
// gateway will fail to start. Upgrade OpenClaw to use those fields.
"config": {
"server": {
"url": "http://127.0.0.1:8420",
"apiKey": "local",
"instanceId": "default"
},
"recall": {
"maxResults": 5,
"includePersona": true,
"includeSceneNav": true
},
"capture": {
"enabled": true
}
}
}
}
}
}
```
If the Gateway is protected by `TDAI_GATEWAY_API_KEY`, set `server.apiKey` to that value. `server.instanceId` is sent as `x-tdai-service-id`; standalone mode uses `default`.
`hooks.allowPromptInjection` allows `before_prompt_build` to inject recalled context. `hooks.allowConversationAccess` allows `agent_end` to read the raw conversation and write L0. **Version compatibility:**
- **OpenClaw `>= 2026.4.24`** — these fields are recognised by the gateway schema. `allowConversationAccess` must be set to `true` for non-bundled plugins (otherwise conversation hooks are silently blocked and L0 capture stops working). `allowPromptInjection` defaults to allowed when unset; we still write `true` to lock the intent.
- **OpenClaw `< 2026.4.24`** (including 4.23 and earlier) — the gateway uses a strict zod schema that **rejects** these fields. **Do not include the `hooks.*` block at all**, otherwise the gateway will fail to start. The installer script auto-detects the version and skips the block on older hosts.
### 6. Enable the plugin in OpenClaw
The plugin manifest ID is `memory-tencentdb-client`. The container image already wires it as the memory slot. If you install it manually, make sure OpenClaw loads the plugin and selects it for the memory slot. After changing the config, restart the Gateway:
```bash
openclaw gateway restart
```
## Adapter Responsibilities
| Area | Implementation | Description |
|---|---|---|
| Capture | `src/hooks/capture.ts` | Writes completed turns to L0 through `addConversation()` |
| Recall | `src/hooks/recall.ts` | Searches memories before prompt construction and injects concise context |
| L1 tool | `src/tools/memory-search.ts` | Lets the Agent actively search structured memories |
| L0 tool | `src/tools/conversation-search.ts` | Lets the Agent search raw conversation history |
| L2/L3 read tool | `src/tools/read-cos.ts` | Lets the Agent read scene/core artifacts when needed |
## Configuration
| Field | Default | Description |
|---|---|---|
| `server.url` | `http://127.0.0.1:8420` | Memory Gateway URL |
| `server.apiKey` | `""` | Bearer token sent by the SDK. Use `local` for default standalone mode. |
| `server.instanceId` | `default` | Memory space ID, sent as `x-tdai-service-id` |
| `recall.maxResults` | `5` | Max L1 memories injected per turn |
| `recall.includePersona` | `true` | Whether to include L3 core/profile context |
| `recall.includeSceneNav` | `true` | Whether to include L2 scene navigation |
| `capture.enabled` | `true` | Whether to auto-capture completed turns |
| `hooks.allowPromptInjection` | `true` | Allows prompt-build context injection. Defaults to enabled when unset; we write `true` explicitly to lock the intent. **Only write this field on OpenClaw `>= 2026.4.24`** — older gateways reject it with a strict schema. |
| `hooks.allowConversationAccess` | `true` | Allows `agent_end` / `llm_input` / `llm_output` to read raw conversation content for L0 writes. **Required (`true`) for non-bundled plugins** — without it, conversation hooks are silently blocked at registration and L0 capture stops working. **Only write this field on OpenClaw `>= 2026.4.24`** — older gateways reject it with a strict schema. |
## Files
```text
openclaw-plugin/
├── openclaw.plugin.json # plugin manifest
├── package.json # dependencies and build script
├── index.ts # OpenClaw entrypoint
├── src/hooks/capture.ts # L0 capture hook
├── src/hooks/recall.ts # recall hook
├── src/tools/ # Agent-callable tools
└── src/format.ts # prompt formatting helpers
```
## Using This as an Adapter Template
When adapting another Agent framework, copy the same pattern:
1. Capture completed user/assistant turns and call `addConversation()`.
2. Before the next prompt, call `searchAtomic()`, `readCore()`, and optionally `listScenarios()`.
3. Inject only concise, labeled memory context into the Agent prompt.
4. Expose active tools for L1 search, L0 conversation search, and L2 scene read.
5. Keep the adapter stateless; the Memory Gateway owns storage and asynchronous L1/L2/L3 processing.
## Notes
This plugin is a client-side adapter only. It should not start a Memory Gateway subprocess and should not implement memory extraction logic locally. For the standalone Gateway startup and SDK examples, see the root README.
openclaw-plugin/README_CN.md
+223
View File
@@ -0,0 +1,223 @@
# OpenClaw 适配参考：TencentDB Agent Memory
简体中文 · [English](./README.md)
该目录是一个 Agent 框架接入 TencentDB Agent Memory v2 API 的参考实现。它是 OpenClaw 的客户端插件：插件本身不做抽取、索引、场景归纳或画像生成，而是连接一个已经运行的 Memory Gateway，通过 TypeScript SDK 完成对话捕获、记忆召回和工具暴露。
在 standalone 本地模式下，推荐的 Memory Gateway 地址是 `http://127.0.0.1:8420`。默认约定是 `apiKey = "local"`、`serviceId = "default"`。如果 Gateway 显式启用了 `TDAI_GATEWAY_API_KEY`，则 `server.apiKey` 应使用同一个值。
## 架构
```text
OpenClaw runtime
└─ memory-tencentdb-client plugin
├─ hooks/capture.ts agent_end -> addConversation (L0)
├─ hooks/recall.ts before_prompt_build -> search + prompt 注入
├─ tools/memory-search.ts tdai_memory_search -> searchAtomic (L1)
├─ tools/conversation-search.ts tdai_conversation_search -> searchConversation (L0)
└─ tools/read-cos.ts tdai_read_cos -> 读取 L2/L3 文件
│
▼
@tencentdb-agent-memory/memory-sdk-ts
│ HTTP v2 API
▼
TencentDB Agent Memory Gateway（standalone :8420 或远端服务）
```
## 快速开始
推荐从仓库根目录执行自动安装脚本：
```bash
bash scripts/install-openclaw-plugin-v2.sh
```
脚本会检查/安装 OpenClaw CLI，安装插件依赖，构建插件，并通过 `openclaw plugins install -l` 将当前 `openclaw-plugin` 目录安装到 OpenClaw。默认还会更新 `~/.openclaw/openclaw.json`：设置 `plugins.slots.memory = "memory-tencentdb-client"`，启用插件，写入 standalone 默认的 `server`、`recall`、`capture` 配置。脚本会自动探测 OpenClaw 版本：**2026.4.24+** 会同时写入 `hooks.allowPromptInjection` / `hooks.allowConversationAccess`（non-bundled 插件需要 `allowConversationAccess=true` 才能采集对话）；**更老的版本会自动跳过**这两个字段——因为 2026.4.24 之前的 gateway 使用 strict zod schema，遇到这两个字段会启动失败。
可通过环境变量覆盖默认值：`OPENCLAW_CONFIG_FILE`、`TDAI_MEMORY_ENDPOINT`、`TDAI_MEMORY_API_KEY`、`TDAI_MEMORY_INSTANCE_ID`、`TDAI_MEMORY_RECALL_MAX_RESULTS`、`TDAI_MEMORY_CAPTURE_ENABLED`。如果只想安装插件、不修改 OpenClaw 配置，可设置 `WRITE_OPENCLAW_CONFIG=0`。
如需手动安装，可按下面步骤执行。
### 1. 安装 OpenClaw
如果你还没有安装 OpenClaw，请先安装 OpenClaw CLI：
```bash
curl -fsSL https://get.openclaw.dev | bash
```
确认命令可用：
```bash
openclaw --version
```
### 2. 安装依赖
```bash
cd openclaw-plugin
npm install
```
插件依赖 `@tencentdb-agent-memory/memory-sdk-ts`，也就是 v2 API 的 TypeScript SDK。
### 3. 构建
```bash
npm run build
```
### 4. 安装插件到 OpenClaw
开发或本地源码调试时，推荐以软链方式安装当前目录：
```bash
openclaw plugins install -l .
```
### 5. 配置插件
如果使用 `agentmemory/openclaw-memory` 容器镜像，镜像启动脚本已经自动完成下面这段配置，通常不需要手动修改。
如果使用 `scripts/install-openclaw-plugin-v2.sh`，脚本默认也会自动写入这段配置，并会自动按 OpenClaw 版本对 `hooks.*` 字段做版本门控。只有在 `WRITE_OPENCLAW_CONFIG=0` 或手动安装插件时，才需要自己编辑 OpenClaw 配置文件。
> ⚠️ **重要 —— `hooks.*` 与 OpenClaw 版本强相关。**
> `hooks.allowPromptInjection` / `hooks.allowConversationAccess` 是 **OpenClaw `2026.4.24`** 起才被 gateway schema 接受的字段。更老的版本（含 `2026.4.23`）使用 strict zod schema，**包含这两个字段会让 gateway 启动失败**。请按你本机 `openclaw --version` 显示的版本，**选择对应的示例**复制：
**示例 A —— OpenClaw `>= 2026.4.24`（推荐）：**
```jsonc
{
"plugins": {
"slots": {
"memory": "memory-tencentdb-client"
},
"entries": {
"memory-tencentdb-client": {
"enabled": true,
// 2026.4.24+ 上，non-bundled 插件必须显式 hooks.*。
// 缺少 allowConversationAccess=true 会让 L0 capture 被静默拦截。
"hooks": {
"allowPromptInjection": true,
"allowConversationAccess": true
},
"config": {
"server": {
"url": "http://127.0.0.1:8420",
"apiKey": "local",
"instanceId": "default"
},
"recall": {
"maxResults": 5,
"includePersona": true,
"includeSceneNav": true
},
"capture": {
"enabled": true
}
}
}
}
}
}
```
**示例 B —— OpenClaw `< 2026.4.24`（如 `2026.4.23`）：完全省略 `hooks` 块。**
```jsonc
{
"plugins": {
"slots": {
"memory": "memory-tencentdb-client"
},
"entries": {
"memory-tencentdb-client": {
"enabled": true,
// ⚠️ 不要添加 "hooks" 块。2026.4.23 及更早的 strict schema 会拒绝
// allowPromptInjection / allowConversationAccess，gateway 启动失败。
// 如需使用这两个字段，请升级 OpenClaw。
"config": {
"server": {
"url": "http://127.0.0.1:8420",
"apiKey": "local",
"instanceId": "default"
},
"recall": {
"maxResults": 5,
"includePersona": true,
"includeSceneNav": true
},
"capture": {
"enabled": true
}
}
}
}
}
}
```
如果 Gateway 启用了 `TDAI_GATEWAY_API_KEY`，请将 `server.apiKey` 设置成同一个值。`server.instanceId` 会作为 `x-tdai-service-id` 发送；standalone 模式默认使用 `default`。
`hooks.allowPromptInjection` 允许 `before_prompt_build` 注入召回上下文；`hooks.allowConversationAccess` 允许 `agent_end` 读取原始对话并写入 L0。**版本兼容性：**
- **OpenClaw `>= 2026.4.24`** —— 这两个字段才被 gateway schema 识别。non-bundled 插件必须显式 `allowConversationAccess=true`，否则会话钩子在注册阶段被静默拦截、L0 capture 不会落库；`allowPromptInjection` 未设置时默认放行，我们仍写 `true` 以锁定意图。
- **OpenClaw `< 2026.4.24`**（含 4.23 及更早）—— gateway 使用 strict zod schema，**会拒绝**这两个字段。**配置中不要包含 `hooks.*` 块**，否则 gateway 启动失败。安装脚本会自动按版本探测、在老版本上跳过该块。
### 6. 在 OpenClaw 中启用
插件 ID 是 `memory-tencentdb-client`。容器镜像中已经将它接入 memory slot。如果手动安装，请确保 OpenClaw 能加载该插件，并在 memory slot 中选中它。修改配置后可重启 Gateway：
```bash
openclaw gateway restart
```
## 适配职责
| 模块 | 实现文件 | 说明 |
|---|---|---|
| 对话捕获 | `src/hooks/capture.ts` | 对话结束后调用 `addConversation()` 写入 L0 |
| 记忆召回 | `src/hooks/recall.ts` | 构建 prompt 前搜索记忆并注入简洁上下文 |
| L1 工具 | `src/tools/memory-search.ts` | 让 Agent 主动搜索结构化记忆 |
| L0 工具 | `src/tools/conversation-search.ts` | 让 Agent 主动搜索历史对话 |
| L2/L3 读取工具 | `src/tools/read-cos.ts` | 让 Agent 在需要时读取场景或画像文件 |
## 配置项
| 字段 | 默认值 | 说明 |
|---|---|---|
| `server.url` | `http://127.0.0.1:8420` | Memory Gateway 地址 |
| `server.apiKey` | `""` | SDK 发送的 Bearer token。standalone 默认可用 `local`。 |
| `server.instanceId` | `default` | Memory 空间 ID，通过 `x-tdai-service-id` 发送 |
| `recall.maxResults` | `5` | 每轮最多注入多少条 L1 记忆 |
| `recall.includePersona` | `true` | 是否注入 L3 core/profile 上下文 |
| `recall.includeSceneNav` | `true` | 是否注入 L2 场景导航 |
| `capture.enabled` | `true` | 是否自动捕获完整对话轮次 |
| `hooks.allowPromptInjection` | `true` | 允许插件在 prompt 构建阶段注入召回上下文。未设置时默认放行，我们显式写 `true` 是为了锁定意图。**仅在 OpenClaw `>= 2026.4.24` 时写入此字段**——更老的 gateway 会用 strict schema 拒绝它。 |
| `hooks.allowConversationAccess` | `true` | 允许插件在 `agent_end` / `llm_input` / `llm_output` 读取原始对话用于 L0 写入。**non-bundled 插件必须显式 `true`**——否则会话钩子会在注册阶段被静默拦截，L0 capture 不会落库。**仅在 OpenClaw `>= 2026.4.24` 时写入此字段**——更老的 gateway 会用 strict schema 拒绝它。 |
## 文件结构
```text
openclaw-plugin/
├── openclaw.plugin.json # 插件清单
├── package.json # 依赖与构建脚本
├── index.ts # OpenClaw 入口
├── src/hooks/capture.ts # L0 捕获 hook
├── src/hooks/recall.ts # 召回 hook
├── src/tools/ # Agent 可调用工具
└── src/format.ts # prompt 格式化辅助
```
## 作为其它 Agent 的适配模板
如果你要适配其它 Agent 框架，可以复用同样模式：
1. 在一轮用户/助手对话完成后调用 `addConversation()`。
2. 在下一轮 prompt 构建前调用 `searchAtomic()`、`readCore()`，必要时调用 `listScenarios()`。
3. 只向 Agent prompt 注入简洁、带标签的记忆上下文。
4. 暴露 L1 搜索、L0 对话搜索、L2 场景读取等主动工具。
5. Adapter 保持无状态；存储、异步 L1/L2/L3 处理都交给 Memory Gateway。
## 注意
该插件只是客户端 adapter，不应在插件内启动 Memory Gateway 子进程，也不应在本地实现记忆抽取逻辑。standalone Gateway 启动方式与 SDK 示例见根目录 README。
openclaw-plugin/docs/architecture.md
+170
View File
@@ -0,0 +1,170 @@
# TencentDB Agent Memory Client — OpenClaw 记忆插件（客户端接入版）
> 创建: 2026-05-17 | 状态: 开发中
> 插件 ID: `memory-tencentdb-client`
> 显示名称: Memory TencentDB (Client)
## 1. 背景
服务化改造完成后，四层记忆数据（L0 对话/L1 原子/L2 场景/L3 画像）全部托管在远端 Gateway：
- **数据存储**: TCVDB (向量) + COS (文件) + Redis (状态)
- **Pipeline**: Gateway Worker 自动完成 L1→L2→L3 抽取
- **API**: 15 个 v2 REST 端点覆盖全部 CRUD + Search
**原插件（memory-tencentdb）**是"全栈"架构：本地 SQLite/VDB + 本地 Pipeline + 本地 Embedding + OpenClaw Hooks + CLI，~15000 行。
**新插件（memory-tencentdb-client）**是纯客户端：只注册 OpenClaw hooks + tools，所有数据操作通过 `@tencentdb-agent-memory/memory-sdk-ts` 委托给远端 Gateway。
## 2. 三层架构
```
┌───────────────────────────────────────────────────────┐
│ OpenClaw Plugin (memory-tencentdb-client) │ 框架适配层
│ hooks (recall/capture) + tools + prompt 注入 │ 只依赖 SDK，不碰 HTTP/存储
│ └─ import { MemoryClient, MemoryFileReader } from SDK│
├───────────────────────────────────────────────────────┤
│ @tencentdb-agent-memory/memory-sdk-ts (独立包) │ 通用 SDK 层
│ MemoryClient (14 API) + MemoryFileReader (STS 直读) │ 零框架依赖，纯 fetch
│ 以后 Dify / AutoGen / LangChain 也用这个 │
├───────────────────────────────────────────────────────┤
│ Gateway v2 API │ 远端服务
│ VDB + COS + Redis + Pipeline Worker │
└───────────────────────────────────────────────────────┘
```
## 3. 插件职责（只做框架适配层）
| 功能 | Hook/Tool | 实现 |
|------|-----------|------|
| **对话捕获** | `agent_end` hook | SDK `client.addConversation()` |
| **记忆召回** | `before_prompt_build` hook | 并行: `client.searchAtomic()` + `client.readCore()` + `client.listScenarios()` |
| **标签清理** | `before_message_write` hook | 剥离 `<relevant-memories>` 标签 |
| **L1 搜索** | `tdai_memory_search` tool | SDK `client.searchAtomic()` |
| **L0 搜索** | `tdai_conversation_search` tool | SDK `client.searchConversation()` |
| **文件读取** | `tdai_read_cos` tool | SDK `MemoryFileReader.read()` (STS 直读对象存储) |
| **Prompt 注入** | recall 内部 | 格式化: Persona + L1 记忆 + Scene Navigation + 工具引导 |
### 不做的事
- ❌ 不启动 VectorStore / SQLite / TCVDB
- ❌ 不启动 EmbeddingService
- ❌ 不启动 Pipeline / Timer / Worker
- ❌ 不做 L1/L2/L3 抽取
- ❌ 不管 COS 存储后端
- ❌ 不管 Redis 状态
- ❌ 不做本地 Checkpoint
## 4. 配置项
```jsonc
{
// Gateway 连接
"gateway.url": "http://127.0.0.1:8420",
"gateway.apiKey": "",
"gateway.instanceId": "default",
// 召回
"recall.maxResults": 5,
"recall.includePersona": true,
"recall.includeSceneNav": true,
// 捕获
"capture.enabled": true
}
```
## 5. 文件结构
```
memory-tencentdb-client/
├── openclaw.plugin.json # 插件清单
├── package.json # deps: { "@tencentdb-agent-memory/memory-sdk-ts": "^0.1.0-beta.1" }
├── index.ts # 入口：初始化 SDK + 注册 hooks/tools
├── src/
│ ├── hooks/
│ │ ├── recall.ts # before_prompt_build → SDK 召回 → prompt 注入
│ │ └── capture.ts # agent_end → SDK addConversation
│ ├── tools/
│ │ ├── memory-search.ts # tdai_memory_search → SDK searchAtomic
│ │ ├── conversation-search.ts # → SDK searchConversation
│ │ └── read-cos.ts # tdai_read_cos → SDK MemoryFileReader.read
│ └── format.ts # 召回结果格式化 + 工具引导注入
├── tests/
│ └── sdk-cos.ts # SDK COS 直读手动测试
├── .gitignore
└── README.md
```
## 6. SDK 依赖策略
```jsonc
"dependencies": {
// caret + 预发版语义：自动跟到 0.1.0-beta.* 系列最新
// 以及 0.1.x 系列正式版（发布后）
"@tencentdb-agent-memory/memory-sdk-ts": "^0.1.0-beta.1"
}
```
SDK 已发布到 npm registry，由 `npm install` 自动拉取，不再走 vendor / 本地 tgz。
SDK 保持独立包，不绑定任何框架，以后出 Dify 插件、Python 版等都复用。
## 7. read_cos 工具设计
### COS 直读（STS）
- SDK 的 `MemoryFileReader` 通过 Gateway `/v2/cos/secret` 获取 STS 临时凭证
- 凭证自动缓存，过期前 2 分钟刷新
- 直接 GET COS 对象（COS V5 签名），不经 Gateway 代理中转
### AI 如何知道可以调 read_cos
1. **Persona 末尾的 Scene Navigation**：
```
## 🗺️ Scene Navigation
### Path: scene_blocks/职业发展与技术实践.md
**热度**: 3 | Summary: 后端工程师，Go + TypeScript...
```
AI 看到路径后主动调 `tdai_read_cos` 读取详情。
2. **工具引导（format.ts 注入）**：
```
<memory-tools-guide>
- tdai_memory_search: 搜索结构化记忆
- tdai_conversation_search: 搜索原始对话
- tdai_read_cos: 读取场景文件（使用 Scene Navigation 中的路径）
</memory-tools-guide>
```
3. **工具 description**：
```
"Read a file from cloud storage. Use paths from Scene Navigation
(e.g. 'scene_blocks/xxx.md') or 'persona.md'."
```
## 8. 关键设计决策
### Q1: session_id 怎么确定？
直接使用 OpenClaw 框架传入的 `ctx.sessionKey`（hook context 自带），与原插件行为一致。不需要自己生成或拼接。
### Q2: 离线/断连降级？
第一版不做——Gateway 不可达时 hook 返回空（不注入记忆），capture 失败记 warn。后续可加本地 fallback。
### Q3: 和原插件冲突吗？
插件 ID 不同（`memory-tencentdb-client` vs `memory-tencentdb`），不冲突。但同时启用会重复捕获/注入，建议只启用一个。
## 9. 实现步骤
| # | 任务 | 预计 |
|---|------|------|
| 1 | `package.json` + `openclaw.plugin.json` + `.gitignore` + `README.md` | 15 min |
| 2 | `index.ts` — 初始化 SDK Client/MemoryFileReader + 注册 hooks/tools | 30 min |
| 3 | `hooks/capture.ts` — agent_end → addConversation | 20 min |
| 4 | `hooks/recall.ts` + `format.ts` — 并行召回 + prompt 格式化 | 45 min |
| 5 | `tools/*.ts` — 3 个工具转发 | 30 min |
| 6 | SDK 测试脚本 (`tests/sdk-cos.ts`) | 15 min |
| 7 | 本地联调测试 | 30 min |
**总计**: ~3 小时
openclaw-plugin/index.ts
+252
View File
@@ -0,0 +1,252 @@
/**
* memory-tencentdb-client — OpenClaw 记忆插件（客户端接入版）
*
* 通过 @tencentdb-agent-memory/memory-sdk-ts 连接远端 memory server，
* 提供四层记忆的自动捕获、召回和工具调用能力。
*
* 本插件不包含任何数据处理逻辑（无 VDB/Embedding/Pipeline），
* 所有操作委托给远端 server。
*/
import { MemoryClient } from "@tencentdb-agent-memory/memory-sdk-ts";
import { performRecall } from "./src/hooks/recall.js";
import { performCapture } from "./src/hooks/capture.js";
import { handleMemorySearch } from "./src/tools/memory-search.js";
import { handleConversationSearch } from "./src/tools/conversation-search.js";
import { handleReadCos } from "./src/tools/read-cos.js";
const TAG = "[memory-client]";
// ── Config types (matches openclaw.plugin.json configSchema) ──────────
interface ServerConfig {
url?: string;
apiKey?: string;
instanceId?: string;
}
interface RecallConfig {
maxResults?: number;
includePersona?: boolean;
includeSceneNav?: boolean;
}
interface CaptureConfig {
enabled?: boolean;
}
interface PluginConfig {
server?: ServerConfig;
recall?: RecallConfig;
capture?: CaptureConfig;
}
// Matches OpenClaw plugin register() signature: export default function register(api)
export default function register(api: any) {
// ── Read config (nested objects per configSchema) ──────────────────
const cfg = (api.pluginConfig ?? {}) as PluginConfig;
const server = cfg.server ?? {};
const recall = cfg.recall ?? {};
const capture = cfg.capture ?? {};
const serverUrl = server.url || "http://127.0.0.1:8420";
const apiKey = server.apiKey || "sk-xxxx";
const instanceId = server.instanceId || "default";
const recallMaxResults = recall.maxResults ?? 5;
const includePersona = recall.includePersona !== false;
const includeSceneNav = recall.includeSceneNav !== false;
const captureEnabled = capture.enabled !== false;
// ── Initialize SDK ──
// NOTE: pass config (not a raw Transport) so client.readFile can lazily
// build its internal MemoryFileReader for STS-signed reads.
const client = new MemoryClient({
endpoint: serverUrl,
apiKey,
serviceId: instanceId,
});
api.logger.info?.(
`${TAG} Initialized: server=${serverUrl}, instance=${instanceId}, ` +
`recall(persona=${includePersona},sceneNav=${includeSceneNav},max=${recallMaxResults}), ` +
`capture=${captureEnabled}`,
);
// ── Register Tools (same pattern as extensions/memory-tencentdb/index.ts) ──
api.registerTool(
{
name: "tdai_memory_search",
label: "Memory Search",
description:
"Search structured memories (L1). Returns relevant memory fragments about " +
"user preferences, past events, rules, and facts.",
parameters: {
type: "object",
properties: {
query: { type: "string", description: "Search query text (natural language)." },
limit: { type: "number", description: "Max results to return (default: 5)." },
type: { type: "string", description: "Filter by memory type." },
},
required: ["query"],
},
async execute(_toolCallId: string, params: Record<string, unknown>) {
return handleMemorySearch(client, params as any, api.logger);
},
},
{ name: "tdai_memory_search" },
);
api.registerTool(
{
name: "tdai_conversation_search",
label: "Conversation Search",
description:
"Search raw conversation history (L0). Returns original messages with timestamps.",
parameters: {
type: "object",
properties: {
query: { type: "string", description: "Search query text." },
limit: { type: "number", description: "Max results (default: 5)." },
session_key: { type: "string", description: "Filter by session ID." },
},
required: ["query"],
},
async execute(_toolCallId: string, params: Record<string, unknown>) {
return handleConversationSearch(client, params as any, api.logger);
},
},
{ name: "tdai_conversation_search" },
);
api.registerTool(
{
name: "tdai_read_cos",
label: "Read COS File",
description:
"Read a file from cloud storage. Use paths from Scene Navigation " +
"(e.g. 'scene_blocks/xxx.md') or 'persona.md'.",
parameters: {
type: "object",
properties: {
path: { type: "string", description: "File path (relative key)." },
},
required: ["path"],
},
async execute(_toolCallId: string, params: Record<string, unknown>) {
return handleReadCos(client, params as any, api.logger);
},
},
{ name: "tdai_read_cos" },
);
// ── Register Hooks (api.on pattern, same as memory-tencentdb) ──
// Per-session caches:
// - pendingOriginalPrompts: clean user prompt + messageCount captured at
// before_prompt_build, used at agent_end to (a) replace polluted user
// message and (b) position-slice this turn's new messages.
// - sessionCursors: max timestamp of last captured batch — used as a
// fallback when the position slice cannot be determined.
const pendingOriginalPrompts = new Map<string, { text: string; messageCount: number }>();
const sessionCursors = new Map<string, number>();
api.on("before_prompt_build", async (event: any, ctx: any) => {
const sessionKey = ctx?.sessionKey;
if (!sessionKey) return;
const userText = event?.prompt;
if (!userText) return;
// Cache original prompt for agent_end (only if capture is enabled — it is
// the only consumer; recall doesn't need this data).
if (captureEnabled) {
const messageCount = Array.isArray(event?.messages) ? event.messages.length : 0;
pendingOriginalPrompts.set(sessionKey, { text: userText, messageCount });
}
try {
const result = await performRecall(client, {
query: userText,
maxResults: recallMaxResults,
includePersona,
includeSceneNav,
}, api.logger);
// OpenClaw consumes the *return value* of before_prompt_build,
// not mutations on the event object. Map our RecallResult to the
// PluginHookBeforePromptBuildResult shape.
const out: { prependContext?: string; appendSystemContext?: string } = {};
if (result.prependContext) out.prependContext = result.prependContext;
if (result.appendSystemContext) out.appendSystemContext = result.appendSystemContext;
return out;
} catch (err) {
api.logger.warn(`${TAG} [recall] Failed: ${err instanceof Error ? err.message : String(err)}`);
}
});
if (captureEnabled) {
api.logger.info?.(`${TAG} Registering agent_end hook for auto-capture`);
api.on("agent_end", async (event: any, ctx: any) => {
const startMs = Date.now();
const sessionKey = ctx?.sessionKey;
const messages = (event?.messages ?? []) as unknown[];
api.logger.debug?.(
`${TAG} [agent_end] hook triggered: success=${event?.success}, ` +
`messages=${messages.length}, sessionKey=${sessionKey ?? "(none)"}`,
);
// Skip on agent failure — partial / errored turns shouldn't pollute L0.
if (event?.success === false) {
api.logger.info(`${TAG} [agent_end] agent did not succeed, skip capture`);
return;
}
if (!sessionKey) {
api.logger.warn(`${TAG} [agent_end] no sessionKey in ctx, skip capture`);
return;
}
if (messages.length === 0) {
api.logger.debug?.(`${TAG} [agent_end] event.messages is empty, skip capture`);
return;
}
const cached = pendingOriginalPrompts.get(sessionKey);
// Don't delete on read — keep until we successfully send (in case of retry),
// or let it be overwritten on next before_prompt_build.
try {
const result = await performCapture(
client,
{
sessionKey,
sessionId: ctx?.sessionId,
rawMessages: messages,
originalUserText: cached?.text,
originalUserMessageCount: cached?.messageCount,
afterTimestamp: sessionCursors.get(sessionKey),
},
api.logger,
);
if (result.maxTimestamp) {
sessionCursors.set(sessionKey, result.maxTimestamp);
}
// Cached prompt has been used — clear it so a stale value doesn't
// bleed into the next turn (e.g. after agent restart).
pendingOriginalPrompts.delete(sessionKey);
const elapsed = Date.now() - startMs;
api.logger.info(
`${TAG} [agent_end] capture done in ${elapsed}ms ` +
`(captured=${result.capturedCount}, serverTotal=${result.serverTotalCount ?? "?"})`,
);
} catch (err) {
const elapsed = Date.now() - startMs;
api.logger.warn(
`${TAG} [capture] Failed after ${elapsed}ms: ` +
(err instanceof Error ? err.message : String(err)),
);
}
});
} else {
api.logger.info?.(`${TAG} capture disabled by config`);
}
}
openclaw-plugin/openclaw.plugin.json
+75
View File
@@ -0,0 +1,75 @@
{
"id": "memory-tencentdb-client",
"name": "Memory TencentDB (Client)",
"version": "0.1.0",
"description": "长期记忆插件（客户端接入版）— 通过远端 memory server 提供四层记忆能力",
"author": "TDAI Team",
"activation": {
"onStartup": true
},
"contracts": {
"tools": [
"tdai_memory_search",
"tdai_conversation_search",
"tdai_read_cos"
]
},
"configSchema": {
"type": "object",
"properties": {
"server": {
"type": "object",
"description": "远端 memory server 连接配置",
"properties": {
"url": {
"type": "string",
"default": "http://127.0.0.1:8420",
"description": "Memory server URL"
},
"apiKey": {
"type": "string",
"default": "sk-xxxx",
"description": "API Key for server authentication"
},
"instanceId": {
"type": "string",
"default": "default",
"description": "Memory instance id（HTTP header x-tdai-service-id）"
}
}
},
"recall": {
"type": "object",
"description": "记忆召回设置",
"properties": {
"maxResults": {
"type": "number",
"default": 5,
"description": "每轮最多注入多少条 L1 记忆"
},
"includePersona": {
"type": "boolean",
"default": true,
"description": "是否注入 L3 画像"
},
"includeSceneNav": {
"type": "boolean",
"default": true,
"description": "是否注入 L2 场景导航索引"
}
}
},
"capture": {
"type": "object",
"description": "对话捕获设置",
"properties": {
"enabled": {
"type": "boolean",
"default": true,
"description": "是否启用自动对话捕获 (L0)"
}
}
}
}
}
}
openclaw-plugin/package.json
+35
View File
@@ -0,0 +1,35 @@
{
"name": "@tencentdb-agent-memory/openclaw-plugin",
"version": "1.0.0-beta.1",
"description": "OpenClaw memory plugin — client mode for TencentDB Agent Memory Gateway",
"type": "module",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"openclaw": {
"extensions": [
"./dist/index.js"
]
},
"scripts": {
"clean": "rm -rf dist *.tgz",
"build": "tsc",
"prepack": "npm run clean && npm install --no-save --no-audit --no-fund && npm run build",
"test:sdk": "node --env-file=.env --import tsx tests/sdk-cos.ts"
},
"dependencies": {
"@tencentdb-agent-memory/memory-sdk-ts": "^1.0.0"
},
"devDependencies": {
"typescript": "^5.5.0"
},
"engines": {
"node": ">=22.0.0"
},
"files": [
"dist/",
"src/",
"index.ts",
"openclaw.plugin.json",
"README.md"
]
}
openclaw-plugin/src/format.ts
+109
View File
@@ -0,0 +1,109 @@
/**
* Format recall results into prompt context.
*
* Output structure (mirrors original memory-tencentdb plugin):
* - prependContext: dynamic L1 memories (changes per turn, injected before user message)
* - appendSystemContext: stable content (Persona + Scene Nav + tools guide, appended to system prompt)
*/
import type { RecallResult } from "./hooks/recall.js";
interface L1Item {
id: string;
content: string;
type: string;
score?: number;
}
interface SceneEntry {
path: string;
created_at?: string;
updated_at?: string;
}
// ── Memory Tools Guide ──
const MEMORY_TOOLS_GUIDE = `<memory-tools-guide>
## 记忆工具调用指南
当上方注入的记忆片段不足以回答用户问题时，可主动调用以下工具获取更多信息：
- **tdai_memory_search**：搜索结构化记忆（L1），适用于回忆用户偏好、历史事件、规则等。
- **tdai_conversation_search**：搜索原始对话（L0），适用于查找具体消息原文、时间线、上下文细节。
- **tdai_read_cos**：读取场景文件详情（使用下方 Scene Navigation 中的路径，如 \`scene_blocks/xxx.md\`）。
### ⚠️ 调用次数限制
每轮对话中，tdai_memory_search 和 tdai_conversation_search **合计最多调用 3 次**。
- 首次搜索无结果时，可换关键词或换工具重试，但总调用次数不要超过 3 次。
- 若 3 次搜索后仍无结果，说明该信息不在记忆中，请直接根据已有信息回复用户。
</memory-tools-guide>`;
/**
* Format L1 memories as prependContext.
*/
function formatL1Memories(items: L1Item[]): string | undefined {
if (items.length === 0) return undefined;
const lines: string[] = [
"<relevant-memories>",
"",
];
for (const item of items) {
const typeTag = item.type ? `[${item.type}]` : "";
lines.push(`- ${typeTag} ${item.content}`);
}
lines.push("");
lines.push("</relevant-memories>");
return lines.join("\n");
}
/**
* Format stable system context: Persona + Scene Navigation + Tools Guide.
*/
function formatSystemContext(
persona: string | null,
scenes: SceneEntry[],
): string | undefined {
const parts: string[] = [];
// Persona (L3)
if (persona) {
parts.push("<user-persona>");
parts.push(persona);
parts.push("</user-persona>");
}
// Scene Navigation (L2 index) — only if not already in persona
if (scenes.length > 0 && (!persona || !persona.includes("Scene Navigation"))) {
parts.push("");
parts.push("## 🗺️ Scene Navigation");
parts.push("*以下是当前场景记忆索引，可使用 tdai_read_cos 读取详细内容。*");
parts.push("");
for (const scene of scenes) {
parts.push(`- \`${scene.path}\``);
}
}
// Tools guide (always append)
parts.push("");
parts.push(MEMORY_TOOLS_GUIDE);
const result = parts.join("\n").trim();
return result || undefined;
}
/**
* Main format function: produce RecallResult for prompt injection.
*/
export function formatRecallResult(
l1Items: L1Item[],
persona: string | null,
scenes: SceneEntry[],
): RecallResult {
return {
prependContext: formatL1Memories(l1Items),
appendSystemContext: formatSystemContext(persona, scenes),
};
}
openclaw-plugin/src/hooks/capture.ts
+235
View File
@@ -0,0 +1,235 @@
/**
* Capture hook (client mode):
* 1. Extract user/assistant messages from the agent_end raw message array
* 2. Apply position slice + (optional) timestamp cursor to keep only this turn
* 3. Replace the polluted user message with the cached original prompt
* 4. Sanitize text + strip code blocks (assistant) + filter noise
* 5. POST the cleaned messages to the gateway via SDK addConversation
*
* Mirrors the structural cleanup in extensions/memory-tencentdb/src/core/conversation/l0-recorder.ts
* (recordConversation), but does not write any local JSONL — the server is
* authoritative for L0 storage.
*/
import type { MemoryClient } from "@tencentdb-agent-memory/memory-sdk-ts";
import { sanitizeText, stripCodeBlocks, shouldCaptureL0 } from "../sanitize.js";
const TAG = "[memory-client][capture]";
interface Logger {
debug?: (msg: string) => void;
info: (msg: string) => void;
warn: (msg: string) => void;
error: (msg: string) => void;
}
interface ExtractedMessage {
role: "user" | "assistant";
content: string;
timestamp: number;
}
export interface CaptureContext {
/** sessionKey from agent_end ctx — used as conversation key on the server. */
sessionKey: string;
/** sessionId from agent_end ctx — passed to addConversation when present. */
sessionId?: string;
/** Raw event.messages array (full session history at agent_end time). */
rawMessages: unknown[];
/** Clean original user prompt (cached at before_prompt_build, pre-pollution). */
originalUserText?: string;
/**
* Number of messages in the session at before_prompt_build time.
* Used to position-slice rawMessages so we only re-send messages added in this turn.
*/
originalUserMessageCount?: number;
/**
* Epoch ms cursor: only messages with timestamp > this are sent.
* Used as a fallback when position slice is unavailable.
*/
afterTimestamp?: number;
}
export interface CaptureResult {
/** Number of messages actually sent to the gateway (post-filter). */
capturedCount: number;
/** Server-reported total count after this batch. */
serverTotalCount?: number;
/** Max timestamp among captured messages — caller should advance its cursor to this. */
maxTimestamp?: number;
}
/**
* Run a single agent_end capture.
*
* Returns immediately with `capturedCount: 0` if there is nothing to send.
* Errors from the gateway are thrown — caller should wrap in try/catch.
*/
export async function performCapture(
client: MemoryClient,
ctx: CaptureContext,
logger?: Logger,
): Promise<CaptureResult> {
const { sessionKey, sessionId, rawMessages, originalUserText, originalUserMessageCount, afterTimestamp } = ctx;
// ── Step 1. Position slice ──
// Only consider messages added AFTER before_prompt_build, i.e. this turn's input.
const usePositionSlice =
originalUserMessageCount != null &&
originalUserMessageCount > 0 &&
originalUserMessageCount <= rawMessages.length;
const slicedMessages = usePositionSlice
? rawMessages.slice(originalUserMessageCount)
: rawMessages;
if (usePositionSlice) {
logger?.debug?.(
`${TAG} Position slice: ${rawMessages.length} raw → ${slicedMessages.length} new ` +
`(sliceStart=${originalUserMessageCount})`,
);
}
// ── Step 2. Extract user/assistant messages ──
const allExtracted = extractUserAssistantMessages(slicedMessages);
logger?.debug?.(
`${TAG} Extracted ${allExtracted.length} user/assistant messages from ${slicedMessages.length} raw`,
);
// ── Step 3. Timestamp cursor (fallback when position slice unavailable) ──
const cursor = afterTimestamp ?? 0;
const filteredByTime = cursor !== 0
? allExtracted.filter((m) => m.timestamp > cursor)
: allExtracted;
if (cursor > 0) {
logger?.debug?.(
`${TAG} Timestamp filter: ${allExtracted.length} → ${filteredByTime.length} (cursor=${cursor})`,
);
}
if (filteredByTime.length === 0) {
logger?.debug?.(`${TAG} No new messages to capture`);
return { capturedCount: 0 };
}
// ── Step 4. Replace polluted user message with cached original ──
// The framework appends the user's message AFTER before_prompt_build and
// injects prependContext into it. Without this swap, the captured user
// text would contain the recall blob, causing a feedback loop.
if (originalUserText) {
const targetRaw = usePositionSlice
? (slicedMessages[0] as Record<string, unknown> | undefined)
: (originalUserMessageCount != null && originalUserMessageCount >= 0 && originalUserMessageCount < rawMessages.length)
? (rawMessages[originalUserMessageCount] as Record<string, unknown> | undefined)
: undefined;
const targetTs = typeof targetRaw?.timestamp === "number" ? targetRaw.timestamp : undefined;
if (targetTs != null) {
let replaced = false;
for (let i = 0; i < filteredByTime.length; i++) {
if (filteredByTime[i].role === "user" && filteredByTime[i].timestamp === targetTs) {
logger?.debug?.(
`${TAG} Replacing polluted user message (ts=${targetTs}, ` +
`${filteredByTime[i].content.length}→${originalUserText.length} chars)`,
);
filteredByTime[i] = { ...filteredByTime[i], content: originalUserText };
replaced = true;
break;
}
}
if (!replaced) {
logger?.warn?.(`${TAG} Could not match cached prompt to any extracted user message — relying on sanitizeText()`);
}
}
}
// ── Step 5. Sanitize + strip code + filter ──
const cleaned = filteredByTime
.map((m) => {
let content = sanitizeText(m.content);
if (m.role === "assistant") content = stripCodeBlocks(content);
return { role: m.role, content, timestamp: m.timestamp };
})
.filter((m) => shouldCaptureL0(m.content));
logger?.debug?.(
`${TAG} After sanitize+filter: ${cleaned.length} messages (from ${filteredByTime.length})`,
);
if (cleaned.length === 0) {
logger?.info(`${TAG} All messages filtered out, skipping POST`);
return { capturedCount: 0 };
}
// ── Step 6. POST to gateway ──
const result = await client.addConversation({
session_id: sessionId ?? sessionKey,
messages: cleaned.map((m) => ({
role: m.role,
content: m.content,
timestamp: new Date(m.timestamp).toISOString(),
})),
});
const maxTimestamp = Math.max(...cleaned.map((m) => m.timestamp));
logger?.info(
`${TAG} Captured ${cleaned.length} message(s) (server total=${result.total_count}, ` +
`sessionKey=${sessionKey.slice(0, 32)}${sessionKey.length > 32 ? "…" : ""})`,
);
return {
capturedCount: cleaned.length,
serverTotalCount: result.total_count,
maxTimestamp,
};
}
/**
* Extract user/assistant entries from the framework's raw message array.
*
* Handles both content shapes the framework may produce:
* - `content: string`
* - `content: Array<{ type: "text", text: string } | ...>`
*
* Strips inline base64 image data URIs (replaces with `[image]`) so they do
* not bloat the request payload or pollute downstream FTS / embeddings.
*/
function extractUserAssistantMessages(messages: unknown[]): ExtractedMessage[] {
const result: ExtractedMessage[] = [];
for (const msg of messages) {
if (!msg || typeof msg !== "object") continue;
const m = msg as Record<string, unknown>;
const role = m.role as string | undefined;
if (role !== "user" && role !== "assistant") continue;
let content: string | undefined;
if (typeof m.content === "string") {
content = m.content;
} else if (Array.isArray(m.content)) {
const parts: string[] = [];
for (const part of m.content) {
if (part && typeof part === "object" && (part as Record<string, unknown>).type === "text") {
const text = (part as Record<string, unknown>).text;
if (typeof text === "string") parts.push(text);
}
}
content = parts.join("\n");
}
if (content && /data:image\/[a-z+]+;base64,/i.test(content)) {
content = content.replace(/data:image\/[a-z+]+;base64,[A-Za-z0-9+/=]+/gi, "[image]");
}
if (content && content.trim()) {
const ts = typeof m.timestamp === "number" ? m.timestamp : Date.now();
result.push({
role: role as "user" | "assistant",
content: content.trim(),
timestamp: ts,
});
}
}
return result;
}
openclaw-plugin/src/hooks/recall.ts
+55
View File
@@ -0,0 +1,55 @@
/**
* Recall hook: search memories from Gateway + format prompt injection.
*/
import type { MemoryClient } from "@tencentdb-agent-memory/memory-sdk-ts";
import { formatRecallResult } from "../format.js";
const TAG = "[memory-client][recall]";
interface Logger {
debug?: (msg: string) => void;
info: (msg: string) => void;
warn: (msg: string) => void;
error: (msg: string) => void;
}
export interface RecallOptions {
query: string;
maxResults: number;
includePersona: boolean;
includeSceneNav: boolean;
}
export interface RecallResult {
prependContext?: string;
appendSystemContext?: string;
}
export async function performRecall(
client: MemoryClient,
opts: RecallOptions,
logger?: Logger,
): Promise<RecallResult> {
const startMs = Date.now();
// Parallel requests: L1 search + L3 persona + L2 scenario list
const [searchResult, persona, scenarios] = await Promise.allSettled([
client.searchAtomic({ query: opts.query, limit: opts.maxResults }),
opts.includePersona ? client.readCore() : Promise.resolve(null),
opts.includeSceneNav ? client.listScenarios({}) : Promise.resolve(null),
]);
// Extract results (graceful on failures)
const l1Items = searchResult.status === "fulfilled" ? (searchResult.value?.items ?? []) : [];
const personaContent = persona.status === "fulfilled" && persona.value ? persona.value.content : null;
const sceneEntries = scenarios.status === "fulfilled" && scenarios.value ? (scenarios.value.entries ?? []) : [];
const elapsedMs = Date.now() - startMs;
logger?.info(
`${TAG} Recall complete (${elapsedMs}ms): L1=${l1Items.length}, ` +
`persona=${personaContent ? "yes" : "no"}, scenes=${sceneEntries.length}`,
);
return formatRecallResult(l1Items, personaContent, sceneEntries);
}
openclaw-plugin/src/sanitize.ts
+88
View File
@@ -0,0 +1,88 @@
/**
* Text sanitization for L0 capture (client-side).
* Mirrors the cleaning logic in extensions/memory-tencentdb/src/utils/sanitize.ts
* — kept in sync to ensure the client and server filter the same noise.
*/
/** Strip injected memory tags + framework metadata blocks + media markers. */
export function sanitizeText(text: string): string {
let cleaned = text;
// Remove injected memory context tags (prevent feedback loops on re-capture)
cleaned = cleaned.replace(/<relevant-memories>[\s\S]*?<\/relevant-memories>/g, "");
cleaned = cleaned.replace(/<user-persona>[\s\S]*?<\/user-persona>/g, "");
cleaned = cleaned.replace(/<relevant-scenes>[\s\S]*?<\/relevant-scenes>/g, "");
cleaned = cleaned.replace(/<scene-navigation>[\s\S]*?<\/scene-navigation>/g, "");
cleaned = cleaned.replace(/<memory-tools-guide>[\s\S]*?<\/memory-tools-guide>/g, "");
// Offload-injected task context blocks
cleaned = cleaned.replace(/<current_task_context>[\s\S]*?<\/current_task_context>/g, "");
cleaned = cleaned.replace(/<history_task_context[\s\S]*?<\/history_task_context>/g, "");
// Framework-injected inbound metadata blocks (label + ```json ... ```)
cleaned = cleaned.replace(
/(?:Conversation info|Sender|Thread starter|Replied message|Forwarded message context|Chat history since last reply)\s*\(untrusted[\s\S]*?\):\s*```json\s*[\s\S]*?```/g,
"",
);
// Legacy conversation metadata JSON blocks
cleaned = cleaned.replace(/```json\s*\{[\s\S]*?"session[\s\S]*?\}\s*```/g, "");
// Reply directive tags: [[reply_to_current]]
cleaned = cleaned.replace(/\[\[reply_to[^\]]*\]\]\s*/g, "");
// Skill-selection wrappers: ¥¥[ ... ]¥¥
cleaned = cleaned.replace(/¥¥\[[\s\S]*?\]¥¥/g, "");
// Line-leading timestamps: [Tue 2026-03-24 03:48 UTC] / GMT+8 / GMT+5:30
cleaned = cleaned.replace(/^\[[\w\d\-:+ ]+\]\s*/gm, "");
// Gateway media-attachment markers
cleaned = cleaned.replace(/\[media attached:[^\]]*\]\s*/g, "");
// Gateway image-reply instructions
cleaned = cleaned.replace(
/To send an image back,[\s\S]*?(?:Keep caption in the text body\.)\s*/g,
"",
);
// System exec blocks: "System: [timestamp] Exec completed ..."
cleaned = cleaned.replace(/^System:\s*\[[\s\S]*?$/gm, "");
// Inline base64 image data URIs
cleaned = cleaned.replace(/data:image\/[a-z+]+;base64,[A-Za-z0-9+/=]+/gi, "");
// Null chars + collapse whitespace
cleaned = cleaned.replace(/\0/g, "").replace(/\n{3,}/g, "\n\n").trim();
return cleaned;
}
/**
* Strip fenced code blocks from assistant replies before L0 capture.
* Only applied to role=assistant — keeps explanatory text but drops noisy code.
*/
export function stripCodeBlocks(text: string): string {
return text.replace(/```[^\n]*\n[\s\S]*?```/g, "").replace(/\n{3,}/g, "\n\n").trim();
}
/**
* L0 capture filter — permissive. Only drops messages that are structurally
* useless (empty, framework bootstrap noise, slash commands).
*/
export function shouldCaptureL0(text: string): boolean {
if (!text || !text.trim()) return false;
if (isFrameworkNoise(text)) return false;
if (text.startsWith("/")) return false;
return true;
}
function isFrameworkNoise(text: string): boolean {
const t = text.trim();
if (t === "(session bootstrap)") return true;
if (t.startsWith("A new session was started via")) return true;
if (/^✅\s*New session started/.test(t)) return true;
if (t.startsWith("Pre-compaction memory flush")) return true;
if (/^NO_REPLY\s*$/.test(t)) return true;
return false;
}
openclaw-plugin/src/tools/conversation-search.ts
+57
View File
@@ -0,0 +1,57 @@
/**
* tdai_conversation_search tool — delegates to SDK searchConversation.
*/
import type { MemoryClient } from "@tencentdb-agent-memory/memory-sdk-ts";
interface Logger {
debug?: (msg: string) => void;
warn: (msg: string) => void;
}
export async function handleConversationSearch(
client: MemoryClient,
params: { query: string; limit?: number; session_key?: string },
logger?: Logger,
) {
const { query, limit = 5, session_key } = params;
if (!query?.trim()) {
return { content: [{ type: "text" as const, text: "Query cannot be empty." }] };
}
try {
logger?.debug?.(`[conversation-search] query="${query}", limit=${limit}, session=${session_key ?? "(all)"}`);
const result = await client.searchConversation({
query,
limit,
session_id: session_key,
});
const messages = result.messages ?? [];
logger?.debug?.(`[conversation-search] ✅ ${messages.length} results`);
if (messages.length === 0) {
return { content: [{ type: "text" as const, text: "No matching conversation messages found." }] };
}
const lines: string[] = [`Found ${messages.length} matching message(s):`, ""];
for (const msg of messages) {
const scoreStr = msg.score != null ? ` (score: ${msg.score.toFixed(3)})` : "";
const dateStr = msg.timestamp ? ` [${msg.timestamp}]` : "";
lines.push(`---`);
lines.push(`**[${msg.role}]**${dateStr}${scoreStr}`);
lines.push("");
lines.push(msg.content);
lines.push("");
}
return {
content: [{ type: "text" as const, text: lines.join("\n") }],
details: { count: messages.length },
};
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
logger?.warn(`[conversation-search] Failed: ${msg}`);
return { content: [{ type: "text" as const, text: `Conversation search failed: ${msg}` }] };
}
}
openclaw-plugin/src/tools/memory-search.ts
+50
View File
@@ -0,0 +1,50 @@
/**
* tdai_memory_search tool — delegates to SDK searchAtomic.
*/
import type { MemoryClient } from "@tencentdb-agent-memory/memory-sdk-ts";
interface Logger {
debug?: (msg: string) => void;
warn: (msg: string) => void;
}
export async function handleMemorySearch(
client: MemoryClient,
params: { query: string; limit?: number; type?: string },
logger?: Logger,
) {
const { query, limit = 5, type } = params;
if (!query?.trim()) {
return { content: [{ type: "text" as const, text: "Query cannot be empty." }] };
}
try {
logger?.debug?.(`[memory-search] query="${query}", limit=${limit}, type=${type ?? "(all)"}`);
const result = await client.searchAtomic({ query, limit, type });
const items = result.items ?? [];
logger?.debug?.(`[memory-search] ✅ ${items.length} results`);
if (items.length === 0) {
return { content: [{ type: "text" as const, text: "No matching memories found." }] };
}
const lines: string[] = [`Found ${items.length} matching memories:`, ""];
for (const item of items) {
const scoreStr = item.score != null ? ` (score: ${item.score.toFixed(3)})` : "";
lines.push(`- **[${item.type}]**${scoreStr}`);
lines.push(` ${item.content}`);
lines.push("");
}
return {
content: [{ type: "text" as const, text: lines.join("\n") }],
details: { count: items.length },
};
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
logger?.warn(`[memory-search] Failed: ${msg}`);
return { content: [{ type: "text" as const, text: `Memory search failed: ${msg}` }] };
}
}
openclaw-plugin/src/tools/read-cos.ts
+39
View File
@@ -0,0 +1,39 @@
/**
* tdai_read_cos tool — reads memory pipeline artifacts (persona.md,
* scene_blocks/*.md, ...) by relative path via the SDK's `client.readFile`.
*/
import type { MemoryClient } from "@tencentdb-agent-memory/memory-sdk-ts";
interface Logger {
debug?: (msg: string) => void;
warn: (msg: string) => void;
}
export async function handleReadCos(
client: MemoryClient,
params: { path: string },
logger?: Logger,
) {
const { path } = params;
if (!path?.trim()) {
return { content: [{ type: "text" as const, text: "Path cannot be empty." }] };
}
// Security: reject path traversal
if (path.includes("..") || path.startsWith("/")) {
return { content: [{ type: "text" as const, text: `Invalid path: "${path}"` }] };
}
try {
logger?.debug?.(`[read-cos] read: "${path}"`);
const content = await client.readFile(path);
logger?.debug?.(`[read-cos] ✅ "${path}" (${content.length} chars)`);
return { content: [{ type: "text" as const, text: content }] };
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
logger?.warn(`[read-cos] Failed to read "${path}": ${msg}`);
return { content: [{ type: "text" as const, text: `Failed to read file: ${msg}` }] };
}
}
openclaw-plugin/tests/sdk-cos.ts
+96
View File
@@ -0,0 +1,96 @@
/**
* SDK 直读测试脚本
*
* 验证 @tencentdb-agent-memory/memory-sdk-ts 的 MemoryFileReader 能否正确：
* 1. 获取 STS 临时凭证（通过 Gateway /v2/cos/secret）
* 2. 直读 persona.md
* 3. 直读 scene_blocks/*.md
*
* 前置条件：
* - Gateway 已启动
* - 后端存储上有数据（先跑过 E2E pipeline 测试）
*
* Usage:
* E2E_ENDPOINT=http://127.0.0.1:8420 \
* E2E_API_KEY=test-key-e2e \
* E2E_SERVICE_ID=tdai-mem-dev001 \
* npx tsx tests/sdk-cos.ts
*/
import { MemoryClient, HttpTransport, createMemoryFileReader } from "@tencentdb-agent-memory/memory-sdk-ts";
const ENDPOINT = process.env.E2E_ENDPOINT || "http://127.0.0.1:8420";
const API_KEY = process.env.E2E_API_KEY || "test-key-e2e";
const SERVICE_ID = process.env.E2E_SERVICE_ID || "tdai-mem-dev001";
async function main() {
console.log(`\n🧪 SDK 直读测试`);
console.log(` Endpoint: ${ENDPOINT}`);
console.log(` ServiceId: ${SERVICE_ID}`);
// 1. 初始化 SDK
const transport = new HttpTransport({
endpoint: ENDPOINT,
apiKey: API_KEY,
serviceId: SERVICE_ID,
});
const client = new MemoryClient(transport);
const fileReader = createMemoryFileReader({
endpoint: ENDPOINT,
apiKey: API_KEY,
serviceId: SERVICE_ID,
});
console.log(`\n── Step 1: 通过 Gateway API 列举 scenario 文件`);
const ls = await client.listScenarios({});
console.log(` 找到 ${ls.entries.length} 个文件:`);
for (const e of ls.entries) {
console.log(` - ${e.path}`);
}
if (ls.entries.length === 0) {
console.log(`\n⚠️ 后端存储上没有 scenario 文件，请先跑 E2E pipeline 测试生成数据。`);
process.exit(1);
}
// 2. 直读 persona.md
console.log(`\n── Step 2: 直读 persona.md`);
try {
const persona = await fileReader.read("persona.md");
console.log(` ✅ 读取成功 (${persona.length} chars)`);
console.log(` 内容前 200 字: ${persona.slice(0, 200)}...`);
} catch (err) {
console.log(` ❌ 读取失败: ${err instanceof Error ? err.message : String(err)}`);
}
// 3. 直读第一个 scene block
const firstScene = ls.entries[0];
const cosPath = `scene_blocks/${firstScene.path}`;
console.log(`\n── Step 3: 直读 ${cosPath}`);
try {
const content = await fileReader.read(cosPath);
console.log(` ✅ 读取成功 (${content.length} chars)`);
console.log(` 内容前 200 字: ${content.slice(0, 200)}...`);
} catch (err) {
console.log(` ❌ 读取失败: ${err instanceof Error ? err.message : String(err)}`);
}
// 4. 对比：通过 Gateway API 读同一个文件
console.log(`\n── Step 4: 对比 — Gateway API 读同一个文件`);
try {
const apiResult = await client.readScenario({ path: firstScene.path });
console.log(` ✅ Gateway API 读取成功 (${apiResult.content.length} chars)`);
console.log(` 内容前 200 字: ${apiResult.content.slice(0, 200)}...`);
} catch (err) {
console.log(` ❌ Gateway API 读取失败: ${err instanceof Error ? err.message : String(err)}`);
}
console.log(`\n${"═".repeat(50)}`);
console.log(` ✅ SDK 直读测试完成`);
console.log(`${"═".repeat(50)}\n`);
}
main().catch((err) => {
console.error("Fatal:", err);
process.exit(1);
});
openclaw-plugin/tsconfig.json
+18
View File
@@ -0,0 +1,18 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"declaration": true,
"outDir": "./dist",
"rootDir": ".",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"lib": ["ES2022", "DOM"],
"types": ["node"]
},
"include": ["index.ts", "src/**/*.ts"],
"exclude": ["node_modules", "dist", "tests"]
}
openclaw.plugin.json
+164
View File
@@ -0,0 +1,164 @@
{
"id": "memory-tencentdb",
"name": "Memory (TencentDB)",
"description": "Four-layer memory system — auto-captures, structures, and profiles conversational knowledge",
"commandAliases": ["memory-tdai"],
"activation": {
"onStartup": true
},
"contracts": {
"tools": ["tdai_memory_search", "tdai_conversation_search"]
},
"configSchema": {
"type": "object",
"additionalProperties": true,
"properties": {
"storeBackend": {
"type": "string",
"enum": ["sqlite", "tcvdb"],
"default": "sqlite",
"description": "存储后端：sqlite（本地 SQLite + sqlite-vec）或 tcvdb（腾讯云向量数据库）"
},
"capture": {
"type": "object",
"description": "对话自动捕获设置 (L0)",
"properties": {
"enabled": { "type": "boolean", "default": true, "description": "是否启用自动对话捕获" },
"excludeAgents": { "type": "array", "items": { "type": "string" }, "default": [], "description": "排除的 Agent glob 模式列表（如 bench-judge-*），匹配的 agent 不会被捕获、召回或参与 pipeline 调度" },
"l0l1RetentionDays": { "type": "number", "default": 0, "description": "L0/L1 本地文件保留天数（单位天）。0=不清理；非0时默认必须>=3。若为1或2，需显式开启 allowAggressiveCleanup" },
"allowAggressiveCleanup": { "type": "boolean", "default": false, "description": "是否允许高风险清理配置（l0l1RetentionDays=1或2）" },
"cleanTime": { "type": "string", "default": "03:00", "description": "每日清理执行时间（HH:mm）" }
}
},
"extraction": {
"type": "object",
"description": "记忆提取设置 (L1)",
"properties": {
"enabled": { "type": "boolean", "default": true, "description": "是否启用后台提取" },
"enableDedup": { "type": "boolean", "default": true, "description": "L1 智能去重（基于向量相似度或关键词进行冲突检测）" },
"maxMemoriesPerSession": { "type": "number", "default": 20, "description": "单次 L1 提取每 session 最大记忆条数" },
"model": { "type": "string", "description": "提取使用模型 (格式: provider/model)，未填写时使用openclaw默认模型" }
}
},
"persona": {
"type": "object",
"description": "场景归纳与用户画像设置 (L2/L3)",
"properties": {
"triggerEveryN": { "type": "number", "default": 50, "description": "每 N 条新记忆触发画像生成" },
"maxScenes": { "type": "number", "default": 15, "description": "最大场景数" },
"backupCount": { "type": "number", "default": 3, "description": "画像备份保留数量" },
"sceneBackupCount": { "type": "number", "default": 10, "description": "场景块备份保留数量" },
"model": { "type": "string", "description": "画像生成模型 (格式: provider/model)，未填写时使用openclaw默认模型" }
}
},
"pipeline": {
"type": "object",
"description": "L1→L2→L3 管线调度设置",
"properties": {
"everyNConversations": { "type": "number", "default": 5, "description": "每 N 轮对话触发 L1 批处理" },
"enableWarmup": { "type": "boolean", "default": true, "description": "Warm-up 模式：新 session 从 1 轮触发开始，每次 L1 后翻倍（1→2→4→...→everyN），加速早期记忆提取" },
"l1IdleTimeoutSeconds": { "type": "number", "default": 600, "description": "L1 空闲超时（秒）：用户停止对话后多久触发 L1 批处理" },
"l2DelayAfterL1Seconds": { "type": "number", "default": 10, "description": "L1 完成后延迟多久触发 L2（秒）" },
"l2MinIntervalSeconds": { "type": "number", "default": 900, "description": "同一 session 两次 L2 抽取的最小间隔（秒）" },
"l2MaxIntervalSeconds": { "type": "number", "default": 3600, "description": "同一活跃 session 的 L2 最大轮询间隔（秒）" },
"sessionActiveWindowHours": { "type": "number", "default": 24, "description": "session 活跃窗口（小时），超过此时间不活跃的 session 停止 L2 轮询" }
}
},
"recall": {
"type": "object",
"description": "记忆召回设置",
"properties": {
"enabled": { "type": "boolean", "default": true, "description": "是否启用自动召回" },
"maxResults": { "type": "number", "default": 5, "description": "召回最大结果数" },
"maxCharsPerMemory": { "type": "number", "default": 0, "description": "单条 L1 记忆注入的最大字符数；填 0 表示不限制" },
"maxTotalRecallChars": { "type": "number", "default": 0, "description": "本轮 auto-recall 注入的 L1 记忆总字符预算；填 0 表示不限制" },
"scoreThreshold": { "type": "number", "default": 0.3, "description": "最低分数阈值" },
"strategy": { "type": "string", "enum": ["embedding", "keyword", "hybrid"], "default": "hybrid", "description": "搜索策略：keyword(关键词)、embedding(向量)、hybrid(混合RRF融合，推荐)" },
"timeoutMs": { "type": "number", "default": 5000, "description": "召回整体超时（毫秒），超时后跳过记忆注入并打印警告日志" }
}
},
"embedding": {
"type": "object",
"description": "向量搜索 (Embedding) 配置",
"properties": {
"enabled": { "type": "boolean", "default": true, "description": "是否启用向量搜索（若 provider=none，则实际会被禁用）" },
"provider": { "type": "string", "default": "none", "description": "Embedding 服务提供者：填写兼容 OpenAI API 的远端服务名称（如 openai、deepseek 等）；不填或填 none 则禁用向量搜索" },
"proxyUrl": { "type": "string", "description": "本地代理地址（仅 provider=qclaw 时必填）。配置后 embedding 请求将通过该代理转发，原始 baseUrl 作为 Remote-URL 头传递" },
"baseUrl": { "type": "string", "description": "API Base URL（必填）：填写对应 provider 的 API 地址" },
"apiKey": { "type": "string", "description": "API Key（必填）" },
"model": { "type": "string", "description": "模型名称（必填）" },
"dimensions": { "type": "number", "description": "向量维度（必填，需与所选模型匹配）" },
"sendDimensions": { "type": "boolean", "default": true, "description": "是否在请求体中携带 dimensions 字段。默认 true（兼容 OpenAI text-embedding-3-* 的 Matryoshka 截断）。当目标服务为 BGE-M3 等不支持自定义维度的固定维度模型时，请设为 false，否则会被服务端以 HTTP 400 拒绝（'does not support matryoshka representation'）。" },
"conflictRecallTopK": { "type": "number", "default": 5, "description": "冲突检测时召回 Top-K 数" },
"maxInputChars": { "type": "number", "default": 5000, "description": "Embedding 输入文本最大字符数，超出时截断并打印警告日志（默认 5000，适合大多数模型的 token 上限）" },
"timeoutMs": { "type": "number", "default": 10000, "description": "单次 embedding API 调用超时（毫秒），超时后该次请求中止且不重试" },
"recallTimeoutMs": { "type": "number", "description": "recall 路径 embedding 超时（毫秒），覆盖 timeoutMs。用户等待中，建议设短一些（如 3000）" },
"captureTimeoutMs": { "type": "number", "description": "capture 路径 embedding 超时（毫秒），覆盖 timeoutMs。后台运行，可设长一些（如 15000）" }
}
},
"tcvdb": {
"type": "object",
"description": "腾讯云向量数据库配置（仅 storeBackend=tcvdb 时生效）",
"properties": {
"url": { "type": "string", "description": "实例 URL（必填，如 http://10.0.1.1:8100）" },
"username": { "type": "string", "default": "root", "description": "账户名" },
"apiKey": { "type": "string", "description": "API Key（必填）" },
"database": { "type": "string", "description": "数据库名（未填写时默认根据实例 ID 自动生成）" },
"alias": { "type": "string", "description": "用户友好别名（可选，用于 database.json 中识别）" },
"embeddingModel": { "type": "string", "default": "bge-large-zh", "description": "服务端 embedding 模型" },
"timeout": { "type": "number", "default": 10000, "description": "请求超时（毫秒）" },
"caPemPath": { "type": "string", "description": "CA 证书 PEM 文件路径（HTTPS 连接时使用）" }
}
},
"bm25": {
"type": "object",
"description": "BM25 稀疏向量编码设置（主要用于 tcvdb 后端）",
"properties": {
"enabled": { "type": "boolean", "default": true, "description": "是否启用 BM25 稀疏向量编码" },
"language": { "type": "string", "enum": ["zh", "en"], "default": "zh", "description": "分词语言：zh（中文）或 en（英文）" }
}
},
"report": {
"type": "object",
"description": "指标上报设置",
"properties": {
"enabled": { "type": "boolean", "default": false, "description": "是否启用指标上报（通过 Gateway 日志输出结构化 METRIC JSON）" },
"type": { "type": "string", "default": "local", "description": "上报方式：local 表示通过 logger 输出结构化 JSON 日志" }
}
},
"llm": {
"type": "object",
"description": "独立 LLM 配置 — 开启后 L1/L2/L3 提取绕过 OpenClaw 的内置模型，改用指定的 OpenAI-compatible API 直接调用。默认关闭（使用 OpenClaw 宿主模型）。",
"properties": {
"enabled": { "type": "boolean", "default": false, "description": "是否启用独立 LLM 模式。关闭时使用 OpenClaw 宿主模型。" },
"baseUrl": { "type": "string", "default": "https://api.openai.com/v1", "description": "OpenAI-compatible API 地址" },
"apiKey": { "type": "string", "description": "API Key" },
"model": { "type": "string", "default": "gpt-4o", "description": "模型名称（如 gpt-4o, deepseek-v3, claude-sonnet-4-6）" },
"maxTokens": { "type": "number", "default": 4096, "description": "最大输出 token 数" },
"timeoutMs": { "type": "number", "default": 120000, "description": "请求超时（毫秒）" }
}
},
"offload": {
"type": "object",
"description": "Context Offload 设置 — 多层上下文压缩系统（独立开关，默认关闭）",
"properties": {
"enabled": { "type": "boolean", "default": false, "description": "是否启用 Context Offload（默认关闭，不影响 Memory 功能）" },
"model": { "type": "string", "description": "Offload 使用的 LLM 模型（格式: provider/model），未填写时使用 openclaw 默认模型" },
"temperature": { "type": "number", "default": 0.2, "description": "LLM 温度参数" },
"forceTriggerThreshold": { "type": "number", "default": 4, "description": "累积多少个 tool pair 后强制触发 L1" },
"dataDir": { "type": "string", "description": "自定义数据目录（绝对路径），默认 ~/.openclaw/context-offload" },
"defaultContextWindow": { "type": "number", "default": 200000, "description": "默认上下文窗口大小" },
"maxPairsPerBatch": { "type": "number", "default": 20, "description": "L1 每批最大 tool pair 数" },
"l2NullThreshold": { "type": "number", "default": 4, "description": "offload.jsonl 中 node_id=null 数量达到此阈值时触发 L2" },
"l2TimeoutSeconds": { "type": "number", "default": 300, "description": "距上次 L2 超过此秒数时触发 L2" },
"mildOffloadRatio": { "type": "number", "default": 0.5, "description": "温和压缩触发比例（占 context window）" },
"aggressiveCompressRatio": { "type": "number", "default": 0.85, "description": "激进压缩触发比例" },
"mmdMaxTokenRatio": { "type": "number", "default": 0.2, "description": "MMD 注入 token 预算比例" },
"backendUrl": { "type": "string", "description": "后端服务 URL（如 https://offload-api.example.com），配置后 L1/L1.5/L2/L4 走后端" },
"backendApiKey": { "type": "string", "description": "后端 API 认证 token" },
"backendTimeoutMs": { "type": "number", "default": 10000, "description": "后端调用超时（毫秒）" }
}
}
}
}
}
package.json
+155
View File
@@ -0,0 +1,155 @@
{
"name": "@tencentdb-agent-memory/memory-tencentdb",
"version": "1.0.0-beta.1",
"description": "Four-layer local memory system plugin for OpenClaw — auto-captures, structures, and profiles conversational knowledge using local LLM + SQLite vector search (L0→L1→L2→L3 pipeline)",
"type": "module",
"main": "./dist/index.mjs",
"bin": {
"migrate-sqlite-to-tcvdb": "./bin/migrate-sqlite-to-tcvdb.mjs",
"export-tencent-vdb": "./bin/export-tencent-vdb.mjs",
"read-local-memory": "./bin/read-local-memory.mjs"
},
"exports": {
".": {
"import": "./dist/index.mjs",
"default": "./dist/index.mjs"
}
},
"scripts": {
"build": "npm run build:plugin && npm run build:scripts",
"build:plugin": "tsdown",
"build:scripts": "npm run build:migrate-sqlite-to-vdb && npm run build:export-tencent-vdb && npm run build:read-local-memory && npm run build:seed-v2",
"prepack": "npm run build",
"build:migrate-sqlite-to-vdb": "tsc -p scripts/migrate-sqlite-to-tcvdb/tsconfig.json --noEmitOnError false",
"migrate-sqlite-to-tcvdb": "node ./bin/migrate-sqlite-to-tcvdb.mjs",
"build:export-tencent-vdb": "tsc --project scripts/export-tencent-vdb/tsconfig.json",
"export-tencent-vdb": "node ./bin/export-tencent-vdb.mjs",
"build:read-local-memory": "tsc --project scripts/read-local-memory/tsconfig.json",
"read-local-memory": "node ./bin/read-local-memory.mjs",
"build:seed-v2": "tsc --project scripts/seed-v2/tsconfig.json",
"seed-v2": "node ./bin/seed-v2.mjs",
"test": "vitest run",
"test:oss": "vitest run -c vitest.oss.config.ts",
"test:watch": "vitest",
"test:coverage": "vitest run --coverage",
"postinstall": "bash scripts/openclaw-after-tool-call-messages.patch.sh 2>/dev/null || true"
},
"files": [
"dist/",
"bin/migrate-sqlite-to-tcvdb.mjs",
"bin/export-tencent-vdb.mjs",
"bin/read-local-memory.mjs",
"index.ts",
"scripts/migrate-sqlite-to-tcvdb/dist/",
"scripts/export-tencent-vdb/dist/",
"scripts/read-local-memory/dist/",
"scripts/memory-tencentdb-ctl.sh",
"scripts/install_hermes_memory_tencentdb.sh",
"scripts/README.memory-tencentdb-ctl.md",
"src",
"scripts/openclaw-after-tool-call-messages.patch.sh",
"scripts/setup-offload.sh",
"hermes-plugin/",
"openclaw.plugin.json",
"README.md",
"CHANGELOG.md",
"LICENSE",
"!src/integrations/",
"!src/**/*.test.ts",
"!src/**/*.spec.ts",
"!src/**/__tests__/"
],
"keywords": [
"openclaw",
"openclaw-plugin",
"memory",
"ai-memory",
"long-term-memory",
"vector-search",
"sqlite-vec",
"llm",
"conversation",
"persona",
"scene-extraction",
"embedding"
],
"author": "TencentDB Agent Memory Team",
"license": "MIT",
"engines": {
"node": ">=22.16.0"
},
"dependencies": {
"@ai-sdk/openai": "^3.0.53",
"@node-rs/jieba": "^2.0.1",
"@opentelemetry/api": "^1.9.0",
"@opentelemetry/api-logs": "^0.218.0",
"@opentelemetry/exporter-logs-otlp-http": "^0.218.0",
"@opentelemetry/exporter-trace-otlp-http": "^0.218.0",
"@opentelemetry/resources": "^2.7.1",
"@opentelemetry/sdk-logs": "^0.218.0",
"@opentelemetry/sdk-node": "^0.218.0",
"@opentelemetry/sdk-trace-base": "^2.7.1",
"@opentelemetry/semantic-conventions": "^1.41.1",
"@tencentdb-agent-memory/tcvdb-text": "^0.1.1",
"ai": "^6.0.164",
"crc-32": "^1.2.2",
"dayjs": "^1.11.20",
"js-tiktoken": "^1.0.18",
"json5": "^2.2.3",
"sqlite-vec": "0.1.7-alpha.2",
"tsx": "^4.21.0",
"undici": "^8.1.0",
"yaml": "^2.8.3",
"zod": "^4.4.3"
},
"optionalDependencies": {
"@clickhouse/client": "^1.18.5",
"@opentelemetry/context-async-hooks": "^2.7.1",
"@opentelemetry/exporter-logs-otlp-grpc": "^0.218.0",
"@opentelemetry/exporter-trace-otlp-grpc": "^0.218.0",
"cos-nodejs-sdk-v5": "^2.15.4",
"ioredis": "^5.10.1",
"kafkajs": "^2.2.4",
"opik": "^1.0.0"
},
"peerDependencies": {
"node-llama-cpp": "^3.16.2",
"openclaw": ">=2026.3.7"
},
"peerDependenciesMeta": {
"openclaw": {
"optional": true
},
"node-llama-cpp": {
"optional": true
}
},
"openclaw": {
"extensions": [
"./index.ts"
],
"compat": {
"pluginApi": ">=2026.3.13",
"minGatewayVersion": ">=2026.3.13"
},
"build": {
"openclawVersion": "2026.3.13",
"pluginSdkVersion": "2026.3.13"
},
"bundle": {
"stageRuntimeDependencies": true
}
},
"devDependencies": {
"@kubb/cli": "^4.37.7",
"@kubb/core": "^4.37.7",
"@kubb/plugin-oas": "^4.37.7",
"@kubb/plugin-ts": "^4.37.7",
"@kubb/plugin-zod": "^4.37.7",
"@types/node": "^25.5.2",
"@vitest/coverage-v8": "^4.1.2",
"tsdown": "^0.21.10",
"typescript": "^6.0.2",
"vitest": "^4.1.2"
}
}
scripts/README.memory-tencentdb-ctl.md
+342
View File
@@ -0,0 +1,342 @@
# memory-tencentdb-ctl.sh — memory_tencentdb运维脚本
> 配合 [`install_hermes_memory_tencentdb.sh`](./install_hermes_memory_tencentdb.sh) 使用。
> 先跑安装脚本部署好插件 + Node 依赖，之后日常的启停 / 配置全部通过 `memory-tencentdb-ctl.sh` 完成。
## 1. 运行模式
脚本有两种模式，**默认是独立模式**，完全不触碰 `~/.hermes`：
| 模式 | 激活方式 | 做什么 | 不做什么 |
|---|---|---|---|
| `standalone`（默认） | 无需任何参数 | 启停 Gateway；写 `$TDAI_DATA_DIR/tdai-gateway.json`；日志落 `$TDAI_DATA_DIR/logs/` | 不写 `$HERMES_HOME/env.d/`，不改 `$HERMES_HOME/config.yaml`，不读 hermes 相关 env |
| `hermes` | 命令行追加 `--hermes`，或环境 `MEMORY_TENCENTDB_MODE=hermes` | standalone 的全部 + `config llm` 同步写 `$HERMES_HOME/env.d/memory-tencentdb-llm.sh`；日志落 `$HERMES_HOME/logs/memory_tencentdb/`；开放 `enable-hermes-memory` 子命令 | — |
> **为什么 hermes 模式要多写 env 文件？**
> 因为 hermes 进程会托管式地把 Gateway 以子进程拉起来（supervisor 用 `os.environ.copy()` 传环境），此时 Gateway 读不到 `tdai-gateway.json` 所在的 shell 环境，必须通过 `$HERMES_HOME/env.d/*.sh` 让 hermes 自身 `source` 才能把凭据传进去。独立模式下 Gateway 自己读 JSON，无此需要。
## 2. 路径
> **路径变量约定**：本节及后续示例统一用 `$HERMES_HOME` 指代 hermes 的家目录，**默认 `~/.hermes`**，但你可以通过环境变量覆盖（例如 `export HERMES_HOME=/srv/hermes`），脚本和 hermes 自身都遵守这个变量。
>
> 自 0.4.x 起，所有 tdai 相关数据/代码默认收纳到统一根目录 `$MEMORY_TENCENTDB_ROOT`（默认 `~/.memory-tencentdb`）之下：
>
> - `$TDAI_INSTALL_DIR` 默认 `$MEMORY_TENCENTDB_ROOT/tdai-memory-openclaw-plugin`（即 `~/.memory-tencentdb/tdai-memory-openclaw-plugin`）
> - `$TDAI_DATA_DIR` 默认 `$MEMORY_TENCENTDB_ROOT/memory-tdai`（即 `~/.memory-tencentdb/memory-tdai`）
>
> 下文出现这些变量时**先 `export` 一份覆盖**，再跑命令即可全局生效。
> 旧版本使用 `~/tdai-memory-openclaw-plugin` 与 `~/memory-tdai`；`install_hermes_memory_tencentdb.sh` 在升级时会自动迁移这两个旧目录到新位置。
| 路径 | standalone | hermes | 作用 |
|---|---|---|---|
| `$TDAI_INSTALL_DIR` | ✅ | ✅ | 插件源码 + `node_modules` + `src/gateway/server.ts` |
| `$TDAI_DATA_DIR/tdai-gateway.json` | ✅ | ✅ | Gateway 主配置：`llm` / `memory.embedding` / `memory.tcvdb` / `memory.storeBackend`，权限 `0600` |
| `$TDAI_DATA_DIR/logs/` | ✅ 日志 | — | `gateway.stdout.log` / `gateway.stderr.log` / `gateway.pid` |
| `$HERMES_HOME/logs/memory_tencentdb/` | — | ✅ 日志 | 同上，换目录 |
| `$HERMES_HOME/env.d/memory-tencentdb-llm.sh` | — | ✅ | hermes 启动前 `source`，给 supervisor 托管的 Gateway 子进程注入 LLM 凭据 |
| `$HERMES_HOME/config.yaml` | — | ✅ | `enable-hermes-memory` 修改其 `memory.provider` |
| Gateway 监听 | `127.0.0.1:8420` | `127.0.0.1:8420` | 可被 `MEMORY_TENCENTDB_GATEWAY_HOST/PORT` 覆盖 |
所有路径都能用同名环境变量覆盖（再次列出便于对照）：`MEMORY_TENCENTDB_ROOT`（默认 `~/.memory-tencentdb`）、`TDAI_INSTALL_DIR`（默认 `$MEMORY_TENCENTDB_ROOT/tdai-memory-openclaw-plugin`）、`TDAI_DATA_DIR`（默认 `$MEMORY_TENCENTDB_ROOT/memory-tdai`）、`HERMES_HOME`（默认 `~/.hermes`）、`MEMORY_TENCENTDB_LOG_DIR`、`MEMORY_TENCENTDB_GATEWAY_HOST/PORT`。
依赖：`bash`、`python3`、`node >= 22`、`npx`、`lsof` 或 `ss`。
## 3. 安装 & 调用
脚本随 npm 包发布到 `node_modules/.../scripts/` 下，但**没有**注册为 `bin` 命令。要用全局命令名调用，必须自己做一次软链。
### 3.1 从 npm 包里直接运行（无需任何配置）
```bash
npm install @tencentdb-agent-memory/memory-tencentdb
# 项目内安装，路径可由 npm root 动态算出
"$(npm root)/@tencentdb-agent-memory/memory-tencentdb/scripts/memory-tencentdb-ctl.sh" --help
# 全局安装则用 npm root -g
"$(npm root -g)/@tencentdb-agent-memory/memory-tencentdb/scripts/memory-tencentdb-ctl.sh" --help
```
`npm root` / `npm root -g` 会在所有包管理器（npm / pnpm / yarn）和不同的 prefix 配置下返回正确目录，避免硬编码 `node_modules/` 路径。适合一次性、临时使用的场景。
### 3.2 软链到 PATH（推荐给运维 / 长期使用）
不论脚本来源（git clone 出来的仓库 / `npm install` 装的包 / 自定义部署目录），先把脚本路径**算出来存到一个变量里**，再统一做软链。这样无需关心你把仓库放在 `~/code/`、`/opt/`、还是别的什么地方。
```bash
# 第一步：定位 memory-tencentdb-ctl.sh 的真实路径（任选一种来源）
# (a) 从 git 仓库（在仓库根目录或任意子目录里执行）
SCRIPT="$(git -C "$(git rev-parse --show-toplevel)" ls-files | \
grep -E 'scripts/memory-tencentdb-ctl\.sh$' | head -1)"
SCRIPT="$(git rev-parse --show-toplevel)/$SCRIPT"
# (b) 从已 npm 全局安装的包
SCRIPT="$(npm root -g)/@tencentdb-agent-memory/memory-tencentdb/scripts/memory-tencentdb-ctl.sh"
# (c) 从项目本地的 node_modules
SCRIPT="$(npm root)/@tencentdb-agent-memory/memory-tencentdb/scripts/memory-tencentdb-ctl.sh"
# (d) 完全手写绝对路径（如部署到非标准位置）
SCRIPT="/opt/tdai/scripts/memory-tencentdb-ctl.sh"
# 第二步：验证路径正确，然后软链
test -f "$SCRIPT" && echo "ok: $SCRIPT" || { echo "not found"; exit 1; }
chmod +x "$SCRIPT"
sudo ln -sf "$SCRIPT" /usr/local/bin/memory-tencentdb-ctl
# 同样的办法把 install_hermes_memory_tencentdb.sh 链接成 install-memory-tencentdb（可选）
INSTALL_SCRIPT="$(dirname "$SCRIPT")/install_hermes_memory_tencentdb.sh"
test -f "$INSTALL_SCRIPT" && {
chmod +x "$INSTALL_SCRIPT"
sudo ln -sf "$INSTALL_SCRIPT" /usr/local/bin/install-memory-tencentdb
}
```
之后直接 `memory-tencentdb-ctl …` / `install-memory-tencentdb …`。
> **为什么不直接用 `npm bin` 注册？** 这两个脚本是运维工具而不是包的核心 API，主仓库希望用户**显式**完成 PATH 注册（避免无意中污染全局命令空间，并避免 npm 卸载时静默移除运维入口）。
## 4. 生命周期管理（两种模式通用）
```bash
memory-tencentdb-ctl start # 若 :8420 已占用会直接返回；否则后台 spawn，等待 /health 通过
memory-tencentdb-ctl stop # 先 SIGTERM，5s 内未退则 SIGKILL
memory-tencentdb-ctl restart
memory-tencentdb-ctl status # 打印模式、端口、data/log 路径、进程状态
memory-tencentdb-ctl health # GET /health，纯 python3 实现，不要求 curl
memory-tencentdb-ctl logs # tail -f stdout + stderr
memory-tencentdb-ctl logs err 500 # 只看 stderr 最近 500 行
```
启动命令解析顺序：
1. 环境变量 `MEMORY_TENCENTDB_GATEWAY_CMD`（`install_hermes_memory_tencentdb.sh` 写入 `/etc/profile.d/memory-tencentdb-env.sh` 的那条）。
2. 回退到 `sh -c 'cd $TDAI_INSTALL_DIR && exec npx tsx src/gateway/server.ts'`。
启动时会自动 `source` 的环境文件：
- 两种模式：`/etc/profile.d/memory-tencentdb-env.sh`
- 仅 hermes 模式：`/etc/profile.d/hermes-env.sh` 以及 `$HERMES_HOME/env.d/*.sh`
## 5. 配置 LLM / Embedding / VDB
三类凭据统一落到 `$TDAI_DATA_DIR/tdai-gateway.json`（`0600`，原子写）。**`config llm` 在 `--hermes` 模式下会额外写一份 env 文件**；Embedding / VDB 从不写 env。
### 5.1 LLM
```bash
# standalone 模式：只写 tdai-gateway.json
memory-tencentdb-ctl config llm \
--api-key "sk-xxxxxxxxxxxx" \
--base-url "https://api.openai.com/v1" \
--model "gpt-4o" \
--restart
# hermes 模式：tdai-gateway.json + $HERMES_HOME/env.d/memory-tencentdb-llm.sh
memory-tencentdb-ctl --hermes config llm \
--api-key "sk-xxxxxxxxxxxx" \
--base-url "https://api.openai.com/v1" \
--model "gpt-4o" \
--restart
```
- JSON 写入点：`$.llm.{baseUrl, apiKey, model}`。
- env 文件写入（仅 `--hermes`）：`TDAI_LLM_*` 及 `MEMORY_TENCENTDB_LLM_*` 别名（Python provider 的 `get_config_schema()` 会读后者）。
### 5.2 Embedding
默认关闭（`provider=none`）。启用远端 OpenAI 兼容服务：
```bash
memory-tencentdb-ctl config embedding \
--provider openai \
--api-key "sk-xxxx" \
--base-url "https://api.openai.com/v1" \
--model "text-embedding-3-small" \
--dimensions 1536 \
--restart
# 关闭 embedding（退化为 BM25/关键词召回）
memory-tencentdb-ctl config embedding --provider none --restart
```
- JSON 写入点：`$.memory.embedding.{provider, baseUrl, apiKey, model, dimensions, enabled, proxyUrl?}`。
- `qclaw` provider 额外要求 `--proxy-url`。
- 校验规则与 `src/config.ts` 的 `parseConfig()` 对齐：`dimensions` 为正整数，非 `none` 必须带 `apiKey/baseUrl/model/dimensions`；缺项直接报错不写半残 JSON。
### 5.3 VectorDB（Tencent Cloud VDB / tcvdb）
```bash
memory-tencentdb-ctl config vdb \
--url "http://xxx-vdb.tencentclb.com:8100" \
--username root \
--api-key "YOUR-VDB-API-KEY" \
--database "openclaw_memory" \
--alias "primary" \
--embedding-model "bge-large-zh" \
--ca-pem "/etc/ssl/vdb-ca.pem" \
--restart
```
- JSON 写入点：`$.memory.tcvdb.{url, username, apiKey, database, alias?, caPemPath?, embeddingModel?}`。
- 默认同时把 `$.memory.storeBackend` 切到 `"tcvdb"`；只想预埋配置先不切，加 `--no-set-backend`。
- `--ca-pem` 只写路径不复制文件；脚本会校验可读性。
### 5.4 退回本地 SQLite（关闭 VDB 后端）
```bash
# 默认：保留 memory.tcvdb 凭据（方便随时再切回去），仅把 storeBackend 改回 sqlite
memory-tencentdb-ctl config vdb-off --restart
# 同时把腾讯云 VDB 的 url / apiKey / database 等凭据从 JSON 中清掉
memory-tencentdb-ctl config vdb-off --purge-creds --restart
```
- JSON 写入点：把 `$.memory.storeBackend` 设为 `"sqlite"`；`--purge-creds` 时额外删除整段 `$.memory.tcvdb`。
- `$.llm` / `$.memory.embedding` 等其它顶级段**完全保留**，hermes 侧 `memory.provider` **不动**（仍是 `memory_tencentdb`，只是它内部存储退回 sqlite）。
- 配置文件不存在时给出 `warn` 并写入仅含 `{"memory":{"storeBackend":"sqlite"}}` 的最小配置。
- 与 `config vdb` 完全镜像：可与 `--dry-run` / `--restart` 组合使用。
### 5.5 查看当前配置
```bash
memory-tencentdb-ctl config show
```
- 打印 `tdai-gateway.json`，`apiKey`/`password`/`token` 字段自动脱敏为 `<redacted:NN chars>`。
- hermes 模式下额外打印 `$HERMES_HOME/env.d/memory-tencentdb-*.sh`（API key 也会脱敏），可直接贴工单。
## 6. 打通 hermes（仅 `--hermes` 模式）
```bash
memory-tencentdb-ctl --hermes enable-hermes-memory
```
幂等：把 `$HERMES_HOME/config.yaml` 的 `memory:` 段的 `provider:` 改成 `memory_tencentdb`（不存在则新增整段）。改完后重启 hermes：
```bash
source "$HERMES_HOME/env.d/memory-tencentdb-llm.sh"
pkill -f hermes-agent || true
hermes
```
> **关于写入策略**：脚本采用"格式保真"双路径，**永不重写整个 YAML**：
>
> 1. **首选**：检测到 [`ruamel.yaml`](https://yaml.readthedocs.io/) 时走 round-trip，完整保留注释、键序、引号、缩进风格（推荐 `pip install --user ruamel.yaml` 享受最佳保真度，**非必装**）；
> 2. **降级**：未安装 ruamel 时走最小化原位行编辑——只重写 `provider:` 那一行，缩进直接从同段已有兄弟键的前缀**逐字符拷贝**（零猜测、零格式破坏）；
> 3. 若 `memory:` 段不存在，则在文件末尾追加最小段，缩进从文档其它顶级段的子键拓印。
>
> 实测对真实 `~/.hermes/config.yaml` 做 byte-for-byte diff，除 `provider` 值本身外其余字节完全一致。
非 hermes 模式下调用该命令会直接报错退出。
> 想反过来"保留 hermes provider 不变，仅让 TDAI 内部存储退回 sqlite"，用 §5.4 的 `config vdb-off` 即可（不需要也**不要**改 hermes 的 `memory.provider`）。
## 7. 典型使用流程
### 场景 A：Gateway 独立部署（不使用 hermes）
```bash
# 1) 安装
# INSTALL_SCRIPT 的取值方式见上文 3.2 节（git rev-parse / npm root / 手填均可）
# 例如从 git 仓库根：INSTALL_SCRIPT="$(git rev-parse --show-toplevel)/scripts/install_hermes_memory_tencentdb.sh"
# 从 npm 全局： INSTALL_SCRIPT="$(npm root -g)/@tencentdb-agent-memory/memory-tencentdb/scripts/install_hermes_memory_tencentdb.sh"
bash "$INSTALL_SCRIPT"
# 2) 只配 Gateway 所需凭据
memory-tencentdb-ctl config llm --api-key "sk-..." --base-url "https://api.openai.com/v1" --model gpt-4o
memory-tencentdb-ctl config embedding --provider openai --api-key "sk-..." --base-url "https://api.openai.com/v1" \
--model text-embedding-3-small --dimensions 1536
memory-tencentdb-ctl config vdb --url "http://xxx:8100" --api-key "..." --database openclaw_memory
# 3) 启动 + 自检
memory-tencentdb-ctl start
memory-tencentdb-ctl status
memory-tencentdb-ctl health # 预期: {"status":"ok",...}
```
### 场景 B：集成 hermes
```bash
# 1) 安装（与场景 A 相同；INSTALL_SCRIPT 由上文 3.2 节算出）
bash "$INSTALL_SCRIPT"
# 2) 全程加 --hermes（或 export MEMORY_TENCENTDB_MODE=hermes 一次）
memory-tencentdb-ctl --hermes config llm --api-key "sk-..." --base-url "https://api.openai.com/v1" --model gpt-4o
memory-tencentdb-ctl --hermes config embedding --provider openai --api-key "sk-..." \
--base-url "https://api.openai.com/v1" \
--model text-embedding-3-small --dimensions 1536
memory-tencentdb-ctl --hermes config vdb --url "http://xxx:8100" --api-key "..." --database openclaw_memory
# 3) 启动 Gateway（通常由 hermes supervisor 托管，这里是手动兜底）
memory-tencentdb-ctl --hermes start
memory-tencentdb-ctl --hermes status
# 4) 在 hermes config 里启用 provider，并重启 hermes
memory-tencentdb-ctl --hermes enable-hermes-memory
source "$HERMES_HOME/env.d/memory-tencentdb-llm.sh"
pkill -f hermes-agent ; hermes
```
如果嫌 `--hermes` 每次都要带，可以：
```bash
export MEMORY_TENCENTDB_MODE=hermes
```
之后所有调用自动切到 hermes 模式，命令行无需再加 `--hermes`。
### 场景 C：临时把 TDAI 的存储退回 sqlite（保留 hermes 集成）
适用于 VDB 不可达 / 排障 / 离线开发等场景：希望 hermes 端 `memory.provider` 仍是 `memory_tencentdb`，但让 Gateway 改用本地 SQLite 落盘。
```bash
# (A) 默认：保留 memory.tcvdb 凭据，仅把 storeBackend 切回 sqlite
memory-tencentdb-ctl config vdb-off --restart
# (B) 排障结束想切回 vdb：当前需要重新跑一次 config vdb（必填项需重新提供，
# 即使 JSON 里凭据还在；脚本是基于必填校验的"重新声明"语义，不是 toggle）
memory-tencentdb-ctl config vdb \
--url "http://xxx-vdb.tencentclb.com:8100" \
--api-key "<你的 KEY>" \
--database "openclaw_memory" \
--restart
# (C) 彻底放弃 vdb：清掉凭据
memory-tencentdb-ctl config vdb-off --purge-creds --restart
```
> 之所以 (B) 不提供"零参数 vdb-on"，是因为原 `config vdb` 子命令把 `--url/--api-key/--database` 设为强校验，避免用户拼装出半残配置；如果你希望把"已存的凭据重新激活"也做成单命令，告诉维护者补一个 `config vdb-on` 即可（实现方式与 `vdb-off` 完全镜像）。
> **不要**为了"切回 sqlite"去改 `~/.hermes/config.yaml` 的 `memory.provider`！hermes 看到的依然是 `memory_tencentdb` provider，存储后端切换是 Gateway 内部的事，对 hermes 完全透明。
## 8. 全局选项 & 调试技巧
- 所有写操作支持 `--dry-run`（放在命令最前），会打印将要写入的内容但不落盘：
```bash
memory-tencentdb-ctl --dry-run config llm --api-key k --base-url https://x --model m
```
- 敏感文件权限一律 `0600`；`env.d/memory-tencentdb-llm.sh` 含明文 API key，**不要** commit。
- 启动失败：`memory-tencentdb-ctl logs err 200` 查看 stderr；手动前台跑一遍更容易看到报错：
```bash
cd "$TDAI_INSTALL_DIR" && npx tsx src/gateway/server.ts
```
- 端口冲突：`MEMORY_TENCENTDB_GATEWAY_PORT=18420 memory-tencentdb-ctl restart`。
- 验证 hermes 有没有吃到新 env（hermes 模式）：
```bash
tr '\0' '\n' < /proc/$(pgrep -n hermes-agent)/environ | grep -E 'TDAI_|MEMORY_TENCENTDB_'
```
## 9. 退出码
| 码 | 含义 |
|---|---|
| 0 | 成功 |
| 1 | 参数错误 / 业务校验失败（如 `--base-url` 非 http(s)；在 standalone 下调 hermes 专属命令） |
| 2 | 写盘失败（磁盘满、权限不足等） |
| 127 | 依赖缺失（`python3` / `node` / `npx`） |
---
要封装成 systemd unit，基于 `memory-tencentdb-ctl start` / `memory-tencentdb-ctl stop` 写 `Type=forking` 的 service 即可（Gateway 是无状态 HTTP sidecar，不依赖 systemd readiness 协议）。
scripts/bench_timer_scanner.py
+122
View File
@@ -0,0 +1,122 @@
#!/usr/bin/env python3
"""Benchmark: current per-instance ZSET vs Scheme D (16-shard global ZSET) timer scanning."""
import redis
import time
r = redis.Redis(host="127.0.0.1", port=6379, decode_responses=True)
r.flushdb()
INSTANCE_COUNT = 100000
SHARD_COUNT = 16
now = int(time.time() * 1000)
expired = now - 1000
SEP = "=" * 60
# ============================================================
# Phase 1: Write 100K per-instance ZSETs (current design)
# ============================================================
print(f"Phase 1: Writing {INSTANCE_COUNT} per-instance ZSETs...")
t0 = time.time()
pipe = r.pipeline()
for i in range(INSTANCE_COUNT):
inst = f"inst-{i:06d}"
key = "{" + f"tdai_memory:{inst}" + "}:timers"
pipe.zadd(key, {"sess_001:L1_idle": expired})
if i % 5000 == 4999:
pipe.execute()
pipe = r.pipeline()
pipe.execute()
write_time_old = time.time() - t0
print(f" Done: {write_time_old:.2f}s, keys={r.dbsize()}")
# ============================================================
# Phase 2: Simulate current Scanner (iterate all instances)
# ============================================================
print(f"\nPhase 2: Current scanner - iterate {INSTANCE_COUNT} instances...")
start = time.time()
total_expired_old = 0
for i in range(INSTANCE_COUNT):
inst = f"inst-{i:06d}"
key = "{" + f"tdai_memory:{inst}" + "}:timers"
members = r.zrangebyscore(key, "-inf", now)
if members:
r.zremrangebyscore(key, "-inf", now)
total_expired_old += len(members)
old_elapsed = time.time() - start
old_status = "TIMEOUT" if old_elapsed > 2 else "OK"
print(f" Expired found: {total_expired_old}")
print(f" Scan time: {old_elapsed:.2f}s [{old_status}] (target: <2s)")
print(f" Redis ops: ~{INSTANCE_COUNT * 2} (2 per instance)")
# ============================================================
# Phase 3: Write 16-shard ZSETs (Scheme D)
# ============================================================
print(f"\nPhase 3: Writing {INSTANCE_COUNT} timers into {SHARD_COUNT} shards...")
r.flushdb()
def shard_hash(s):
h = 5381
for c in s:
h = ((h << 5) + h + ord(c)) & 0xFFFFFFFF
return h % SHARD_COUNT
t0 = time.time()
pipe = r.pipeline()
for i in range(INSTANCE_COUNT):
inst = f"inst-{i:06d}"
shard = shard_hash(inst)
key = f"tdai_memory:timers:shard_{shard}"
member = f"{inst}:sess_001:L1_idle"
pipe.zadd(key, {member: expired})
if i % 5000 == 4999:
pipe.execute()
pipe = r.pipeline()
pipe.execute()
write_time_new = time.time() - t0
print(f" Done: {write_time_new:.2f}s, keys={r.dbsize()}")
print(f" Shard distribution:")
for s in range(SHARD_COUNT):
cnt = r.zcard(f"tdai_memory:timers:shard_{s}")
print(f" shard_{s:02d}: {cnt} members")
# ============================================================
# Phase 4: Scheme D scanner (scan 16 shards with Lua atomic claim)
# ============================================================
print(f"\nPhase 4: Scheme D scanner - scan {SHARD_COUNT} shards (Lua atomic)...")
LUA_CLAIM = """
local members = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', ARGV[1], 'LIMIT', 0, 50000)
if #members > 0 then
redis.call('ZREM', KEYS[1], unpack(members))
return members
end
return {}
"""
start = time.time()
total_expired_new = 0
for shard in range(SHARD_COUNT):
key = f"tdai_memory:timers:shard_{shard}"
members = r.eval(LUA_CLAIM, 1, key, now)
total_expired_new += len(members)
new_elapsed = time.time() - start
new_status = "TIMEOUT" if new_elapsed > 2 else "OK"
print(f" Expired found: {total_expired_new}")
print(f" Scan time: {new_elapsed:.4f}s [{new_status}] (target: <2s)")
print(f" Redis ops: {SHARD_COUNT} (1 EVAL per shard)")
# ============================================================
# Summary
# ============================================================
print(f"\n{SEP}")
print(f" BENCHMARK: {INSTANCE_COUNT} instances, 1 timer each")
print(f"{SEP}")
print(f" Current (per-instance ZSET): {old_elapsed:.2f}s / scan [{old_status}]")
print(f" Scheme D (16 shards): {new_elapsed:.4f}s / scan [{new_status}]")
if new_elapsed > 0:
print(f" Speedup: {old_elapsed/new_elapsed:.0f}x faster")
print(f" Max shard size: {max(r.zcard(f'tdai_memory:timers:shard_{s}') for s in range(SHARD_COUNT))} (all consumed)")
print(f"{SEP}")
# Cleanup
r.flushdb()
print("\nRedis flushed. Done.")
scripts/bugfix-20260423/BUGFIX-20260423-SOP.md
+88
View File
@@ -0,0 +1,88 @@
# Bugfix-20260423 打镜像 SOP
> **适用版本**: OpenClaw 2026.4.23
> **修复问题**: Issue #73806 — Zod schema `.strict()` 拒绝 `hooks.allowConversationAccess`，导致非捆绑插件无法注册会话钩子
> **脚本位置**: `scripts/bugfix-20260423.sh`
---
## 步骤一：停止 Gateway
```bash
openclaw gateway stop
```
确认已停止：
```bash
ps aux | grep gateway
```
确保没有 `openclaw-gateway` 进程在运行。
---
## 步骤二：执行 Patch
```bash
cd /path/to/memory-tdai/scripts
bash bugfix-20260423.sh
```
---
## 步骤三：验证
### 3.1 验证 openclaw.json 配置
```bash
cat ~/.openclaw/openclaw.json | python3 -m json.tool | grep allowConversationAccess
```
预期输出：
```
"allowConversationAccess": true
```
确认在 `plugins.entries.memory-tencentdb.hooks` 下。
### 3.2 验证 Zod Schema dist 文件
先定位 OpenClaw 安装目录（路径因环境而异，以下仅为示例）：
```bash
# 方式一：通过 which 自动定位
OC_DIR=$(node -e "const p=require('path'),f=require('fs'); \
const bin=require('child_process').execSync('which openclaw',{encoding:'utf8'}).trim(); \
let d=p.dirname(f.realpathSync(bin)); \
while(d!=p.dirname(d)){if(f.existsSync(p.join(d,'package.json'))){console.log(d);break;}d=p.dirname(d);}")
echo "$OC_DIR"
# 方式二：手动指定（示例路径，请根据实际环境替换）
# OC_DIR=~/.local/share/pnpm/global/5/.pnpm/openclaw@2026.4.23_@napi-rs+canvas@0.1.100/node_modules/openclaw
```
然后检查 `zod-schema-BhKK4qYw.js`：
```bash
cat "$OC_DIR/dist/zod-schema-BhKK4qYw.js" | grep allowConversationAccess -n
```
验证要点：
1. `allowConversationAccess` 已出现在输出中
2. **只出现了一次**（只有一行匹配）
3. 所在行的上下文形如：`allowPromptInjection:z.boolean().optional(),allowConversationAccess:z.boolean().optional()}).strict().optional()`
<!-- TODO: 贴验证截图 -->
---
## 验证通过后
两项验证均通过即可重新启动 Gateway：
```bash
openclaw gateway run
```
scripts/bugfix-20260423/bugfix-20260423-full.sh
+298
View File
@@ -0,0 +1,298 @@
#!/usr/bin/env bash
# ═══════════════════════════════════════════════════════════════════
# bugfix-20260423.sh — OC 2026.4.23 allowConversationAccess 修复
# ═══════════════════════════════════════════════════════════════════
# Issue #73806: OC 2026.4.23 的 Zod schema 使用 .strict() 拒绝
# hooks.allowConversationAccess 字段，导致非捆绑插件无法注册会话钩子
# (llm_input, llm_output, agent_end)。PR #71221 在 4.24 修复。
#
# 本脚本做两件事（均幂等，可安全重复执行）：
# 1. Patch dist JS: 给 hooks zod schema 注入 allowConversationAccess 字段
# 2. 写 openclaw.json: 设置 hooks.allowConversationAccess = true
#
# 版本限制：仅在 OC 2026.4.23 上执行 Part 1，其他版本安全跳过 dist patch。
# Part 2 (配置写入) 不限版本，始终确保配置存在。
#
# 用法：
# bash bugfix-20260423.sh [/path/to/openclaw]
#
# 环境变量：
# OPENCLAW_DIR — 覆盖 openclaw 安装路径（优先于参数）
# OPENCLAW_JSON — 覆盖配置文件路径（默认 ~/.openclaw/openclaw.json）
# DEBUG=1 — 开启调试输出
# ═══════════════════════════════════════════════════════════════════
set -euo pipefail
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m'
info() { echo -e "${CYAN}[INFO]${NC} $*"; }
ok() { echo -e "${GREEN}[OK]${NC} $*"; }
warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
fail() { echo -e "${RED}[FAIL]${NC} $*" >&2; exit 1; }
debug() { [[ "${DEBUG:-}" == "1" ]] && echo -e "${CYAN}[DEBUG]${NC} $*" || true; }
PLUGIN_ID="memory-tencentdb"
OPENCLAW_JSON="${OPENCLAW_JSON:-${HOME}/.openclaw/openclaw.json}"
# ═══════════════════════════════════════════════════════════════════
# Part 1: Patch dist JS (仅 2026.4.23)
# ═══════════════════════════════════════════════════════════════════
_resolve_openclaw_dir() {
# 参数 > 环境变量 > 自动检测
if [[ -n "${1:-}" ]]; then
echo "$1"; return 0
fi
if [[ -n "${OPENCLAW_DIR:-}" && -d "${OPENCLAW_DIR}" ]]; then
echo "$OPENCLAW_DIR"; return 0
fi
# 自动定位
node -e "
const {dirname, join} = require('path');
const {realpathSync, existsSync, readFileSync, statSync} = require('fs');
function walkUp(start) {
let dir = statSync(start).isDirectory() ? start : dirname(start);
for (let i = 0; i < 10; i++) {
const pj = join(dir, 'package.json');
if (existsSync(pj)) {
try { if (JSON.parse(readFileSync(pj,'utf8')).name==='openclaw') { console.log(dir); process.exit(0); } } catch {}
}
const parent = dirname(dir);
if (parent === dir) break;
dir = parent;
}
return null;
}
try {
const {execSync} = require('child_process');
const bin = execSync('which openclaw',{encoding:'utf8'}).trim();
const real = realpathSync(bin);
const found = walkUp(real);
if (found) { console.log(found); process.exit(0); }
const content = readFileSync(bin,'utf8');
const m = content.match(/['\"]([^'\"]*openclaw[^'\"]*\\.(?:js|mjs))['\"]/) ||
content.match(/['\"]([^'\"]*openclaw[^'\"]*)['\"].*node/);
if (m) { const f = walkUp(realpathSync(m[1])); if (f) { console.log(f); process.exit(0); } }
} catch {}
const searchDirs = [
join(process.env.HOME||'/root','.local/share/pnpm'),
join(process.env.HOME||'/root','.local/node/lib/node_modules'),
'/usr/local/lib/node_modules','/usr/lib/node_modules',
];
for (const base of searchDirs) {
if (!existsSync(base)) continue;
try {
const {execSync:e2} = require('child_process');
const out = e2('find '+JSON.stringify(base)+' -maxdepth 8 -name package.json -path \"*/openclaw/package.json\" 2>/dev/null',{encoding:'utf8',timeout:5000}).trim();
for (const line of out.split('\\n')) {
if (!line) continue;
try { if (JSON.parse(readFileSync(line,'utf8')).name==='openclaw') { console.log(dirname(line)); process.exit(0); } } catch {}
}
} catch {}
}
process.exit(1);
" 2>/dev/null
}
patch_dist_js() {
local oc_dir
oc_dir="$(_resolve_openclaw_dir "${1:-}")" || {
warn "[Part 1] 找不到 OpenClaw 安装目录，跳过 dist patch"
return 0
}
local dist_dir="$oc_dir/dist"
[[ -d "$dist_dir" ]] || { warn "[Part 1] dist 目录不存在: $dist_dir，跳过"; return 0; }
local version
version=$(grep -oP '"version"\s*:\s*"\K[^"]+' "$oc_dir/package.json" 2>/dev/null || echo "unknown")
info "[Part 1] OpenClaw 版本: $version"
# 版本门控：仅 2026.4.23
if [[ ! "$version" =~ ^2026\.4\.23($|[-\.]) ]]; then
ok "[Part 1] 版本 $version 不需要 schema patch，跳过"
return 0
fi
# 精确定位：hooks zod schema 的唯一特征
local -a candidates
mapfile -t candidates < <(
grep -rl 'allowPromptInjection' "$dist_dir" --include='*.js' 2>/dev/null | while read -r _f; do
if perl -0777 -ne 'exit(0) if /allowPromptInjection\s*:\s*[a-zA-Z_\$][a-zA-Z0-9_\$]*\s*\.\s*boolean\s*\(\s*\)\s*\.\s*optional\s*\(\s*\)\s*[,\s]*\}\s*\)\s*\.\s*strict\s*\(\s*\)/; exit(1)' "$_f" 2>/dev/null; then
echo "$_f"
fi
done
)
if [[ ${#candidates[@]} -eq 0 ]]; then
warn "[Part 1] 未找到 hooks zod schema 目标文件，跳过"
return 0
elif [[ ${#candidates[@]} -gt 1 ]]; then
warn "[Part 1] 发现 ${#candidates[@]} 个匹配文件（预期 1），安全起见跳过"
return 0
fi
local target="${candidates[0]}"
local relpath="${target#$dist_dir/}"
debug "[Part 1] 目标: $relpath"
# 幂等：目标文件中已包含 allowConversationAccess → 跳过
if grep -q 'allowConversationAccess' "$target" 2>/dev/null; then
ok "[Part 1] allowConversationAccess 已存在于 $relpath，跳过"
return 0
fi
# 备份
[[ -f "${target}.pre-aca-patch.bak" ]] || cp "$target" "${target}.pre-aca-patch.bak"
# 注入：用精确变量名匹配，避免贪婪回溯
perl -0777 -i -pe '
s/(allowPromptInjection\s*:\s*[a-zA-Z_\$][a-zA-Z0-9_\$]*\s*\.\s*boolean\s*\(\s*\)\s*\.\s*optional\s*\(\s*\))(\s*\}\s*\)\s*\.\s*strict\s*\(\s*\))/$1,allowConversationAccess:z.boolean().optional()$2/
' "$target"
# 验证
if grep -q 'allowConversationAccess' "$target" 2>/dev/null; then
ok "[Part 1] $relpath — patch 成功"
else
warn "[Part 1] patch 验证失败，恢复备份"
cp "${target}.pre-aca-patch.bak" "$target"
return 1
fi
}
# ═══════════════════════════════════════════════════════════════════
# Part 2: 写入 openclaw.json (不限版本，始终确保配置存在)
# ═══════════════════════════════════════════════════════════════════
patch_config_json() {
if [[ ! -f "$OPENCLAW_JSON" ]]; then
warn "[Part 2] openclaw.json 不存在: $OPENCLAW_JSON，跳过"
return 0
fi
# 幂等检测
local exists
exists=$(python3 -c "
import json
try:
with open('$OPENCLAW_JSON') as f:
cfg = json.load(f)
val = cfg.get('plugins',{}).get('entries',{}).get('$PLUGIN_ID',{}).get('hooks',{}).get('allowConversationAccess')
print('yes' if val is True else 'no')
except Exception:
print('no')
" 2>/dev/null || echo "no")
if [[ "$exists" == "yes" ]]; then
ok "[Part 2] hooks.allowConversationAccess 已存在，跳过"
return 0
fi
# 写入
python3 -c "
import json
with open('$OPENCLAW_JSON') as f:
cfg = json.load(f)
entry = cfg.setdefault('plugins', {}).setdefault('entries', {}).setdefault('$PLUGIN_ID', {})
hooks = entry.setdefault('hooks', {})
hooks['allowConversationAccess'] = True
with open('$OPENCLAW_JSON', 'w') as f:
json.dump(cfg, f, indent=2, ensure_ascii=False)
f.write('\n')
"
ok "[Part 2] hooks.allowConversationAccess = true 已写入"
}
# ═══════════════════════════════════════════════════════════════════
# 主入口
# ═══════════════════════════════════════════════════════════════════
echo ""
echo -e "${CYAN}═══════════════════════════════════════════════════════${NC}"
echo -e "${CYAN} bugfix-20260423: allowConversationAccess 一键修复${NC}"
echo -e "${CYAN} Issue #73806 | 适用版本: OC 2026.4.23${NC}"
echo -e "${CYAN}═══════════════════════════════════════════════════════${NC}"
echo ""
# ── Step 1: 停止 Gateway ──
info "[Step 1] 停止 Gateway..."
openclaw gateway stop 2>/dev/null || true
sleep 10
# 确认已停止
if ps aux | grep -v grep | grep -q 'openclaw-gateway'; then
warn "检测到 openclaw-gateway 仍在运行，尝试强制停止..."
pkill -9 -f 'openclaw-gateway' 2>/dev/null || true
sleep 3
fi
if ps aux | grep -v grep | grep -q 'openclaw-gateway'; then
fail "[Step 1] 无法停止 openclaw-gateway，请手动处理后重试"
fi
ok "[Step 1] Gateway 已停止"
echo ""
# ── Step 2: 执行 Patch ──
info "[Step 2] 执行 Patch..."
if ! patch_dist_js "${1:-}"; then
fail "[Step 2] dist JS patch 失败"
fi
if ! patch_config_json; then
fail "[Step 2] openclaw.json 写入失败"
fi
echo ""
# ── Step 3: 验证 ──
info "[Step 3] 验证结果..."
echo ""
# 3.1 验证 openclaw.json
info " [3.1] 检查 openclaw.json"
if grep -q '"allowConversationAccess"' "$OPENCLAW_JSON" 2>/dev/null; then
_json_line=$(grep -n 'allowConversationAccess' "$OPENCLAW_JSON")
ok " openclaw.json: allowConversationAccess ✓"
echo -e " ${CYAN}${_json_line}${NC}"
else
fail "[Step 3.1] openclaw.json 中未找到 allowConversationAccess"
fi
echo ""
# 3.2 验证 dist JS — 只检查 zod-schema-BhKK4qYw.js
info " [3.2] 检查 zod-schema-BhKK4qYw.js"
_oc_dir="$(_resolve_openclaw_dir "${1:-}" 2>/dev/null)" || _oc_dir=""
_zod_file="$_oc_dir/dist/zod-schema-BhKK4qYw.js"
if [[ -n "$_oc_dir" && -f "$_zod_file" ]]; then
_match_count=$(grep -c 'allowConversationAccess' "$_zod_file" 2>/dev/null || echo "0")
if [[ "$_match_count" -eq 0 ]]; then
fail "[Step 3.2] zod-schema-BhKK4qYw.js 中未找到 allowConversationAccess"
elif [[ "$_match_count" -eq 1 ]]; then
ok " zod-schema-BhKK4qYw.js: allowConversationAccess 出现 1 次 ✓"
echo -e " ${CYAN}匹配行:${NC}"
grep -n 'allowConversationAccess' "$_zod_file" | head -1 | sed 's/^/ /'
else
fail "[Step 3.2] zod-schema-BhKK4qYw.js 中 allowConversationAccess 出现 $_match_count 次（预期 1 次），可能重复注入"
fi
else
fail "[Step 3.2] 文件不存在: $_zod_file"
fi
echo ""
# ── Step 4: 重启 Gateway ──
info "[Step 4] 重启 Gateway..."
openclaw gateway start
sleep 10
if ps aux | grep -v grep | grep -q 'openclaw-gateway'; then
ok "[Step 4] Gateway 已启动 (pid=$(pgrep -f 'openclaw-gateway' | head -1))"
else
fail "[Step 4] Gateway 启动失败，请检查日志"
fi
echo ""
echo -e "${GREEN}═══════════════════════════════════════════════════════${NC}"
echo -e "${GREEN} ✅ bugfix-20260423 修复完成，Gateway 已重启${NC}"
echo -e "${GREEN}═══════════════════════════════════════════════════════${NC}"
echo ""
scripts/bugfix-20260423/bugfix-20260423.sh
+218
View File
@@ -0,0 +1,218 @@
#!/usr/bin/env bash
# ═══════════════════════════════════════════════════════════════════
# bugfix-20260423.sh — OC 2026.4.23 allowConversationAccess 修复
# ═══════════════════════════════════════════════════════════════════
# Issue #73806: OC 2026.4.23 的 Zod schema 使用 .strict() 拒绝
# hooks.allowConversationAccess 字段，导致非捆绑插件无法注册会话钩子
# (llm_input, llm_output, agent_end)。PR #71221 在 4.24 修复。
#
# 本脚本做两件事（均幂等，可安全重复执行）：
# 1. Patch dist JS: 给 hooks zod schema 注入 allowConversationAccess 字段
# 2. 写 openclaw.json: 设置 hooks.allowConversationAccess = true
#
# 版本限制：仅在 OC 2026.4.23 上执行 Part 1，其他版本安全跳过 dist patch。
# Part 2 (配置写入) 不限版本，始终确保配置存在。
#
# 用法：
# bash bugfix-20260423.sh [/path/to/openclaw]
#
# 环境变量：
# OPENCLAW_DIR — 覆盖 openclaw 安装路径（优先于参数）
# OPENCLAW_JSON — 覆盖配置文件路径（默认 ~/.openclaw/openclaw.json）
# DEBUG=1 — 开启调试输出
# ═══════════════════════════════════════════════════════════════════
set -euo pipefail
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m'
info() { echo -e "${CYAN}[INFO]${NC} $*"; }
ok() { echo -e "${GREEN}[OK]${NC} $*"; }
warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
fail() { echo -e "${RED}[FAIL]${NC} $*" >&2; exit 1; }
debug() { [[ "${DEBUG:-}" == "1" ]] && echo -e "${CYAN}[DEBUG]${NC} $*" || true; }
PLUGIN_ID="memory-tencentdb"
OPENCLAW_JSON="${OPENCLAW_JSON:-${HOME}/.openclaw/openclaw.json}"
# ═══════════════════════════════════════════════════════════════════
# Part 1: Patch dist JS (仅 2026.4.23)
# ═══════════════════════════════════════════════════════════════════
_resolve_openclaw_dir() {
# 参数 > 环境变量 > 自动检测
if [[ -n "${1:-}" ]]; then
echo "$1"; return 0
fi
if [[ -n "${OPENCLAW_DIR:-}" && -d "${OPENCLAW_DIR}" ]]; then
echo "$OPENCLAW_DIR"; return 0
fi
# 自动定位
node -e "
const {dirname, join} = require('path');
const {realpathSync, existsSync, readFileSync, statSync} = require('fs');
function walkUp(start) {
let dir = statSync(start).isDirectory() ? start : dirname(start);
for (let i = 0; i < 10; i++) {
const pj = join(dir, 'package.json');
if (existsSync(pj)) {
try { if (JSON.parse(readFileSync(pj,'utf8')).name==='openclaw') { console.log(dir); process.exit(0); } } catch {}
}
const parent = dirname(dir);
if (parent === dir) break;
dir = parent;
}
return null;
}
try {
const {execSync} = require('child_process');
const bin = execSync('which openclaw',{encoding:'utf8'}).trim();
const real = realpathSync(bin);
const found = walkUp(real);
if (found) { console.log(found); process.exit(0); }
const content = readFileSync(bin,'utf8');
const m = content.match(/['\"]([^'\"]*openclaw[^'\"]*\\.(?:js|mjs))['\"]/) ||
content.match(/['\"]([^'\"]*openclaw[^'\"]*)['\"].*node/);
if (m) { const f = walkUp(realpathSync(m[1])); if (f) { console.log(f); process.exit(0); } }
} catch {}
const searchDirs = [
join(process.env.HOME||'/root','.local/share/pnpm'),
join(process.env.HOME||'/root','.local/node/lib/node_modules'),
'/usr/local/lib/node_modules','/usr/lib/node_modules',
];
for (const base of searchDirs) {
if (!existsSync(base)) continue;
try {
const {execSync:e2} = require('child_process');
const out = e2('find '+JSON.stringify(base)+' -maxdepth 8 -name package.json -path \"*/openclaw/package.json\" 2>/dev/null',{encoding:'utf8',timeout:5000}).trim();
for (const line of out.split('\\n')) {
if (!line) continue;
try { if (JSON.parse(readFileSync(line,'utf8')).name==='openclaw') { console.log(dirname(line)); process.exit(0); } } catch {}
}
} catch {}
}
process.exit(1);
" 2>/dev/null
}
patch_dist_js() {
local oc_dir
oc_dir="$(_resolve_openclaw_dir "${1:-}")" || {
warn "[Part 1] 找不到 OpenClaw 安装目录，跳过 dist patch"
return 0
}
local dist_dir="$oc_dir/dist"
[[ -d "$dist_dir" ]] || { warn "[Part 1] dist 目录不存在: $dist_dir，跳过"; return 0; }
local version
version=$(grep -oP '"version"\s*:\s*"\K[^"]+' "$oc_dir/package.json" 2>/dev/null || echo "unknown")
info "[Part 1] OpenClaw 版本: $version"
# 版本门控：仅 2026.4.23
if [[ ! "$version" =~ ^2026\.4\.23($|[-\.]) ]]; then
ok "[Part 1] 版本 $version 不需要 schema patch，跳过"
return 0
fi
# 精确定位：hooks zod schema 的唯一特征
local -a candidates
mapfile -t candidates < <(
grep -rl 'allowPromptInjection' "$dist_dir" --include='*.js' 2>/dev/null | while read -r _f; do
if perl -0777 -ne 'exit(0) if /allowPromptInjection\s*:\s*[a-zA-Z_\$][a-zA-Z0-9_\$]*\s*\.\s*boolean\s*\(\s*\)\s*\.\s*optional\s*\(\s*\)\s*[,\s]*\}\s*\)\s*\.\s*strict\s*\(\s*\)/; exit(1)' "$_f" 2>/dev/null; then
echo "$_f"
fi
done
)
if [[ ${#candidates[@]} -eq 0 ]]; then
warn "[Part 1] 未找到 hooks zod schema 目标文件，跳过"
return 0
elif [[ ${#candidates[@]} -gt 1 ]]; then
warn "[Part 1] 发现 ${#candidates[@]} 个匹配文件（预期 1），安全起见跳过"
return 0
fi
local target="${candidates[0]}"
local relpath="${target#$dist_dir/}"
debug "[Part 1] 目标: $relpath"
# 幂等：目标文件中已包含 allowConversationAccess → 跳过
if grep -q 'allowConversationAccess' "$target" 2>/dev/null; then
ok "[Part 1] allowConversationAccess 已存在于 $relpath，跳过"
return 0
fi
# 备份
[[ -f "${target}.pre-aca-patch.bak" ]] || cp "$target" "${target}.pre-aca-patch.bak"
# 注入：用精确变量名匹配，避免贪婪回溯
perl -0777 -i -pe '
s/(allowPromptInjection\s*:\s*[a-zA-Z_\$][a-zA-Z0-9_\$]*\s*\.\s*boolean\s*\(\s*\)\s*\.\s*optional\s*\(\s*\))(\s*\}\s*\)\s*\.\s*strict\s*\(\s*\))/$1,allowConversationAccess:z.boolean().optional()$2/
' "$target"
# 验证
if grep -q 'allowConversationAccess' "$target" 2>/dev/null; then
ok "[Part 1] $relpath — patch 成功"
else
warn "[Part 1] patch 验证失败，恢复备份"
cp "${target}.pre-aca-patch.bak" "$target"
return 1
fi
}
# ═══════════════════════════════════════════════════════════════════
# Part 2: 写入 openclaw.json (不限版本，始终确保配置存在)
# ═══════════════════════════════════════════════════════════════════
patch_config_json() {
if [[ ! -f "$OPENCLAW_JSON" ]]; then
warn "[Part 2] openclaw.json 不存在: $OPENCLAW_JSON，跳过"
return 0
fi
# 幂等检测
local exists
exists=$(python3 -c "
import json
try:
with open('$OPENCLAW_JSON') as f:
cfg = json.load(f)
val = cfg.get('plugins',{}).get('entries',{}).get('$PLUGIN_ID',{}).get('hooks',{}).get('allowConversationAccess')
print('yes' if val is True else 'no')
except Exception:
print('no')
" 2>/dev/null || echo "no")
if [[ "$exists" == "yes" ]]; then
ok "[Part 2] hooks.allowConversationAccess 已存在，跳过"
return 0
fi
# 写入
python3 -c "
import json
with open('$OPENCLAW_JSON') as f:
cfg = json.load(f)
entry = cfg.setdefault('plugins', {}).setdefault('entries', {}).setdefault('$PLUGIN_ID', {})
hooks = entry.setdefault('hooks', {})
hooks['allowConversationAccess'] = True
with open('$OPENCLAW_JSON', 'w') as f:
json.dump(cfg, f, indent=2, ensure_ascii=False)
f.write('\n')
"
ok "[Part 2] hooks.allowConversationAccess = true 已写入"
}
# ═══════════════════════════════════════════════════════════════════
# 主入口
# ═══════════════════════════════════════════════════════════════════
info "── bugfix-20260423: allowConversationAccess ──"
patch_dist_js "${1:-}"
patch_config_json
echo ""
ok "bugfix-20260423 完成"
scripts/check_sts_permissions.py
+240
View File
@@ -0,0 +1,240 @@
#!/usr/bin/env python3
"""
STS 临时密钥权限排查脚本
从 Shark 获取 COS 凭证，逐项测试各种 COS 操作权限。
用法: python3 check_sts_permissions.py
依赖: pip install cos-python-sdk-v5
"""
import json
import re
import sys
import urllib.request
import traceback
SHARK_URL = "http://tdai.gateway.cd.test.polaris:8000/meta/GetMemoryPlusCosConfig"
INSTANCE_ID = "mem-rkgqhd5z"
def fetch_cos_config():
"""从 Shark 获取 COS 配置"""
req = urllib.request.Request(
SHARK_URL,
data=b"{}",
headers={"Content-Type": "application/json"},
method="POST",
)
resp = urllib.request.urlopen(req, timeout=10)
payload = json.loads(resp.read())
print("=" * 60)
print("Shark 原始返回:")
print(json.dumps(payload, indent=2, ensure_ascii=False))
print("=" * 60)
if payload.get("code") != 0:
print(f"Shark 返回错误: code={payload.get('code')}, message={payload.get('message')}")
sys.exit(1)
return payload["data"]
def parse_cos_url(cos_url):
"""解析 COS URL 提取 bucket 和 region"""
host = cos_url.replace("https://", "").replace("http://", "").rstrip("/")
m = re.match(r"^(.+?)\.cos(?:-internal)?\.(.+?)\.(?:myqcloud\.com|tencentcos\.cn)$", host)
if not m:
raise ValueError(f"无法解析 COS URL: {cos_url}")
return m.group(1), m.group(2)
def test_permissions(data):
"""逐项测试 COS 操作权限"""
try:
from qcloud_cos import CosConfig, CosS3Client
except ImportError:
print("\n请先安装 COS SDK: pip install cos-python-sdk-v5")
sys.exit(1)
cos_url = data["CosUrl"]
secret_id = data["TmpSecretId"]
secret_key = data["TmpSecretKey"]
token = data["TmpToken"]
path_prefix = data["PathPrefix"]
expiration = data.get("ExpirationTime", "未知")
bucket, region = parse_cos_url(cos_url)
print(f"\n凭证信息:")
print(f" CosUrl: {cos_url}")
print(f" Bucket: {bucket}")
print(f" Region: {region}")
print(f" TmpSecretId: {secret_id[:10]}...{secret_id[-4:]}")
print(f" TmpSecretKey: {secret_key[:6]}...{secret_key[-4:]}")
print(f" TmpToken: {'有' if token else '无'} (长度={len(token) if token else 0})")
print(f" PathPrefix: {path_prefix}")
print(f" Expiration: {expiration}")
# 实际 prefix（按 memory service 逻辑拼接）
prefix = f"{path_prefix.rstrip('/')}/{INSTANCE_ID}/"
print(f" 实际 Key 前缀: {prefix}")
# 同时测试外网域名（开发机可能无法访问 cos-internal）
# 如果 CosUrl 是内网域名，先用外网试
use_public = "cos-internal" in cos_url or "tencentcos.cn" in cos_url
if use_public:
print(f"\n 注意: CosUrl 为内网域名，将同时用外网域名测试")
config = CosConfig(
Region=region,
SecretId=secret_id,
SecretKey=secret_key,
Token=token,
)
client = CosS3Client(config)
test_key = f"{prefix}__permission_test__.txt"
results = {}
# ── Test 1: GetService (列出所有 Bucket) ──
print(f"\n{'─' * 60}")
print("Test 1: GetService (列出所有 Bucket)")
try:
resp = client.list_buckets()
buckets = resp.get("Buckets", {}).get("Bucket", [])
results["GetService"] = f"OK ({len(buckets)} buckets)"
print(f" ✅ OK - 可见 {len(buckets)} 个 Bucket")
for b in buckets[:5]:
print(f" - {b.get('Name', '?')} ({b.get('Location', '?')})")
if len(buckets) > 5:
print(f" ... 及其他 {len(buckets) - 5} 个")
except Exception as e:
results["GetService"] = f"DENIED ({e})"
print(f" ❌ DENIED - {e}")
# ── Test 2: HeadBucket (检查 Bucket 是否存在/可访问) ──
print(f"\nTest 2: HeadBucket (Bucket={bucket})")
try:
client.head_bucket(Bucket=bucket)
results["HeadBucket"] = "OK"
print(f" ✅ OK - Bucket 存在且可访问")
except Exception as e:
results["HeadBucket"] = f"DENIED ({e})"
print(f" ❌ DENIED - {e}")
# ── Test 3: GetBucket / ListObjects (在 prefix 下列出对象) ──
print(f"\nTest 3: ListObjects (Prefix={prefix})")
try:
resp = client.list_objects(Bucket=bucket, Prefix=prefix, MaxKeys=10)
contents = resp.get("Contents", [])
results["ListObjects"] = f"OK ({len(contents)} objects)"
print(f" ✅ OK - 列出 {len(contents)} 个对象")
for c in contents[:5]:
print(f" - {c['Key']} ({c['Size']} bytes)")
except Exception as e:
results["ListObjects"] = f"DENIED ({e})"
print(f" ❌ DENIED - {e}")
# ── Test 3b: ListObjects 在 PathPrefix 下（不带 instanceId）──
raw_prefix = path_prefix.rstrip("/") + "/"
print(f"\nTest 3b: ListObjects (Prefix={raw_prefix}, 不带 instanceId)")
try:
resp = client.list_objects(Bucket=bucket, Prefix=raw_prefix, MaxKeys=10)
contents = resp.get("Contents", [])
results["ListObjects(raw)"] = f"OK ({len(contents)} objects)"
print(f" ✅ OK - 列出 {len(contents)} 个对象")
for c in contents[:5]:
print(f" - {c['Key']} ({c['Size']} bytes)")
except Exception as e:
results["ListObjects(raw)"] = f"DENIED ({e})"
print(f" ❌ DENIED - {e}")
# ── Test 3c: ListObjects 在 bucket 根目录 ──
print(f"\nTest 3c: ListObjects (Prefix='', Bucket 根目录)")
try:
resp = client.list_objects(Bucket=bucket, Prefix="", MaxKeys=10)
contents = resp.get("Contents", [])
results["ListObjects(root)"] = f"OK ({len(contents)} objects)"
print(f" ✅ OK - 列出 {len(contents)} 个对象")
for c in contents[:5]:
print(f" - {c['Key']} ({c['Size']} bytes)")
except Exception as e:
results["ListObjects(root)"] = f"DENIED ({e})"
print(f" ❌ DENIED - {e}")
# ── Test 4: PutObject (写入测试文件) ──
print(f"\nTest 4: PutObject (Key={test_key})")
try:
resp = client.put_object(
Bucket=bucket,
Body=b"permission test",
Key=test_key,
ContentType="text/plain",
)
results["PutObject"] = f"OK (ETag={resp.get('ETag', '?')})"
print(f" ✅ OK - ETag={resp.get('ETag', '?')}")
except Exception as e:
results["PutObject"] = f"DENIED ({e})"
print(f" ❌ DENIED - {e}")
# ── Test 5: GetObject (读取刚写入的文件) ──
print(f"\nTest 5: GetObject (Key={test_key})")
try:
resp = client.get_object(Bucket=bucket, Key=test_key)
body = resp["Body"].get_raw_stream().read()
results["GetObject"] = f"OK ({len(body)} bytes)"
print(f" ✅ OK - 读到 {len(body)} 字节: {body[:50]}")
except Exception as e:
results["GetObject"] = f"DENIED ({e})"
print(f" ❌ DENIED - {e}")
# ── Test 6: HeadObject (获取文件元数据) ──
print(f"\nTest 6: HeadObject (Key={test_key})")
try:
resp = client.head_object(Bucket=bucket, Key=test_key)
results["HeadObject"] = f"OK (size={resp.get('Content-Length', '?')})"
print(f" ✅ OK - Content-Length={resp.get('Content-Length', '?')}")
except Exception as e:
results["HeadObject"] = f"DENIED ({e})"
print(f" ❌ DENIED - {e}")
# ── Test 7: DeleteObject (删除测试文件) ──
print(f"\nTest 7: DeleteObject (Key={test_key})")
try:
client.delete_object(Bucket=bucket, Key=test_key)
results["DeleteObject"] = "OK"
print(f" ✅ OK - 删除成功")
except Exception as e:
results["DeleteObject"] = f"DENIED ({e})"
print(f" ❌ DENIED - {e}")
# ── 汇总 ──
print(f"\n{'=' * 60}")
print("权限汇总:")
print(f"{'=' * 60}")
for op, status in results.items():
icon = "✅" if status.startswith("OK") else "❌"
print(f" {icon} {op:25s} → {status}")
print(f"{'=' * 60}")
ok_count = sum(1 for s in results.values() if s.startswith("OK"))
total = len(results)
print(f"\n结论: {ok_count}/{total} 项操作有权限")
if ok_count == 0:
print("⚠️ STS 临时密钥无任何 COS 操作权限！需要 Shark 团队修复 STS Policy。")
print(" 所需权限: cos:GetObject, cos:PutObject, cos:DeleteObject, cos:GetBucket, cos:HeadObject")
elif ok_count < total:
denied = [op for op, s in results.items() if not s.startswith("OK")]
print(f"⚠️ 部分操作无权限: {', '.join(denied)}")
else:
print("✅ STS 临时密钥拥有完整的 COS 操作权限，write_scenario 应该可以正常工作。")
if __name__ == "__main__":
print("STS 临时密钥权限排查")
print(f"Shark 地址: {SHARK_URL}")
print(f"实例 ID: {INSTANCE_ID}")
data = fetch_cos_config()
test_permissions(data)
scripts/export-diagnostic.sh
Executable
+227
View File
@@ -0,0 +1,227 @@
#!/usr/bin/env bash
# OpenClaw + memory-tencentdb（原 memory-tdai）诊断数据导出脚本
# 注：插件已更名为 memory-tencentdb，但数据目录始终为 memory-tdai（代码硬编码）
# 用法: bash export-diagnostic.sh [输出目录]
# 默认输出到 ~/Downloads/openclaw-diagnostic-<timestamp>/
set -euo pipefail
# ── 参数 ──
OUTPUT_BASE="${1:-$HOME/Downloads}"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
EXPORT_DIR="${OUTPUT_BASE}/openclaw-diagnostic-${TIMESTAMP}"
ARCHIVE_PATH="${EXPORT_DIR}.tar.gz"
# ── OpenClaw 工作目录探测 ──
if [ -n "${OPENCLAW_STATE_DIR:-}" ]; then
STATE_DIR="$OPENCLAW_STATE_DIR"
elif [ -d "$HOME/.openclaw" ]; then
STATE_DIR="$HOME/.openclaw"
elif [ -d "$HOME/.clawdbot" ]; then
STATE_DIR="$HOME/.clawdbot"
else
echo "❌ 未找到 OpenClaw 工作目录 (~/.openclaw 或 ~/.clawdbot)"
exit 1
fi
echo "📂 OpenClaw 工作目录: $STATE_DIR"
echo "📦 导出目录: $EXPORT_DIR"
mkdir -p "$EXPORT_DIR"
# ── 1. 收集环境信息 ──
echo "🔍 收集环境信息..."
{
echo "=== 导出时间 ==="
date -Iseconds 2>/dev/null || date
echo ""
echo "=== 系统信息 ==="
echo "OS: $(uname -a)"
echo "Node: $(node --version 2>/dev/null || echo 'not found')"
echo "pnpm: $(pnpm --version 2>/dev/null || echo 'not found')"
echo ""
echo "=== OpenClaw 版本 ==="
openclaw --version 2>/dev/null || pnpm openclaw --version 2>/dev/null || echo "(unknown)"
echo ""
echo "=== 工作目录 ==="
echo "STATE_DIR: $STATE_DIR"
echo ""
echo "=== 目录结构 ==="
ls -la "$STATE_DIR/" 2>/dev/null || echo "(empty)"
echo ""
echo "=== memory-tdai 目录结构 ==="
ls -laR "$STATE_DIR/memory-tdai/" 2>/dev/null || echo "(not found)"
echo ""
echo "=== 磁盘占用 ==="
du -sh "$STATE_DIR/memory-tdai/"* 2>/dev/null || echo "(not found)"
} > "$EXPORT_DIR/env-info.txt" 2>&1
# ── 2. 收集 OpenClaw 日志 ──
echo "📋 收集 OpenClaw 日志..."
mkdir -p "$EXPORT_DIR/logs"
# 网关日志 (~/.openclaw/logs/)
if [ -d "$STATE_DIR/logs" ]; then
cp -r "$STATE_DIR/logs/" "$EXPORT_DIR/logs/gateway-logs/" 2>/dev/null || true
fi
# 滚动日志 (/tmp/openclaw/)
TMP_LOG_DIR="/tmp/openclaw"
if [ -d "$TMP_LOG_DIR" ]; then
mkdir -p "$EXPORT_DIR/logs/rolling-logs"
# 只取最近 3 个日志文件
ls -t "$TMP_LOG_DIR"/openclaw-*.log 2>/dev/null | head -3 | while read -r f; do
# 每个文件只取最后 5000 行，避免过大
tail -5000 "$f" > "$EXPORT_DIR/logs/rolling-logs/$(basename "$f")" 2>/dev/null || true
done
fi
# ── 3. 收集记忆插件数据 ──
# 注：数据目录名为 memory-tdai（历史原因，插件更名为 memory-tencentdb 后未改目录名）
echo "🧠 收集记忆插件数据..."
MEMORY_DIR="$STATE_DIR/memory-tdai"
if [ -d "$MEMORY_DIR" ]; then
mkdir -p "$EXPORT_DIR/memory-tdai"
# L0 对话记录 (JSONL)
if [ -d "$MEMORY_DIR/conversations" ]; then
cp -r "$MEMORY_DIR/conversations/" "$EXPORT_DIR/memory-tdai/conversations/" 2>/dev/null || true
fi
# L1 结构化记忆 (JSONL)
if [ -d "$MEMORY_DIR/records" ]; then
cp -r "$MEMORY_DIR/records/" "$EXPORT_DIR/memory-tdai/records/" 2>/dev/null || true
fi
# L2 场景文件 (Markdown)
if [ -d "$MEMORY_DIR/scene_blocks" ]; then
cp -r "$MEMORY_DIR/scene_blocks/" "$EXPORT_DIR/memory-tdai/scene_blocks/" 2>/dev/null || true
fi
# L3 用户画像
[ -f "$MEMORY_DIR/persona.md" ] && cp "$MEMORY_DIR/persona.md" "$EXPORT_DIR/memory-tdai/" 2>/dev/null || true
# checkpoint + scene_index
if [ -d "$MEMORY_DIR/.metadata" ]; then
cp -r "$MEMORY_DIR/.metadata/" "$EXPORT_DIR/memory-tdai/.metadata/" 2>/dev/null || true
fi
# SQLite 数据库（用于检查向量/FTS索引状态）
[ -f "$MEMORY_DIR/vectors.db" ] && cp "$MEMORY_DIR/vectors.db" "$EXPORT_DIR/memory-tdai/" 2>/dev/null || true
# 备份目录（可选，可能较大）
if [ -d "$MEMORY_DIR/.backup" ]; then
cp -r "$MEMORY_DIR/.backup/" "$EXPORT_DIR/memory-tdai/.backup/" 2>/dev/null || true
fi
else
echo " ⚠️ 未找到 memory-tdai 数据目录（memory-tencentdb 插件的数据也存储在此目录）"
fi
# ── 4. 收集 OpenClaw 配置（脱敏） ──
echo "🔧 收集 OpenClaw 配置（已脱敏）..."
CONFIG_FILE="$STATE_DIR/openclaw.json"
if [ -f "$CONFIG_FILE" ]; then
# 使用 node 脱敏处理配置
node -e "
const fs = require('fs');
const JSON5 = (() => { try { return require('json5'); } catch { return JSON; } })();
const raw = fs.readFileSync('$CONFIG_FILE', 'utf-8');
let cfg;
try { cfg = JSON5.parse(raw); } catch { cfg = JSON.parse(raw); }
// 递归脱敏函数
function redact(obj, path) {
if (!obj || typeof obj !== 'object') return obj;
if (Array.isArray(obj)) return obj.map((v, i) => redact(v, path + '[' + i + ']'));
const result = {};
for (const [k, v] of Object.entries(obj)) {
const fullPath = path ? path + '.' + k : k;
// 脱敏规则：API key、token、password、secret 类字段
if (/api_?key|token|password|secret|credential/i.test(k) && typeof v === 'string') {
result[k] = v.length > 0 ? '***REDACTED(' + v.length + 'chars)***' : '';
}
// 脱敏 SecretRef 对象
else if (v && typeof v === 'object' && v.source && v.id && v.provider) {
result[k] = { source: v.source, provider: v.provider, id: '***REDACTED***' };
}
// 整体跳过的顶层敏感块
else if (['models', 'secrets', 'channels', 'env'].includes(k) && !path) {
result[k] = '***REDACTED_SECTION(use openclaw config get ' + k + ' to inspect)***';
}
// gateway.auth 内的 token/password
else if (path === 'gateway.auth' && /token|password/i.test(k)) {
result[k] = typeof v === 'string' ? '***REDACTED***' : v;
}
else {
result[k] = redact(v, fullPath);
}
}
return result;
}
const redacted = redact(cfg, '');
// plugins 已经过递归 redact()，其中 apiKey/token/password/secret 等字段
// 会被自动脱敏，同时保留 provider/model/enabled 等排查所需的非敏感配置
fs.writeFileSync('$EXPORT_DIR/openclaw-config-redacted.json', JSON.stringify(redacted, null, 2));
console.log(' ✅ 配置已脱敏导出');
" 2>&1 || {
echo " ⚠️ Node 脱敏失败，使用 grep 粗略脱敏"
# 粗略脱敏：删除包含敏感关键字的行
grep -v -iE '(api.?key|token|password|secret|credential).*:.*"[^"]{8,}"' "$CONFIG_FILE" \
| sed -E 's/"(models|secrets|channels|env)"\s*:\s*\{[^}]*\}/"__REDACTED_SECTION__"/g' \
> "$EXPORT_DIR/openclaw-config-redacted.json" 2>/dev/null || true
}
else
echo " ⚠️ 未找到配置文件"
fi
# ── 5. 收集插件安装信息 ──
echo "🔌 收集插件安装信息..."
if [ -d "$STATE_DIR/extensions" ]; then
{
echo "=== 已安装插件 ==="
ls -la "$STATE_DIR/extensions/" 2>/dev/null
echo ""
for ext_dir in "$STATE_DIR/extensions"/*/; do
[ -d "$ext_dir" ] || continue
pkg="$ext_dir/node_modules/openclaw/package.json"
plugin_pkg="$ext_dir/package.json"
echo "--- $(basename "$ext_dir") ---"
if [ -f "$plugin_pkg" ]; then
node -e "const p=require('$plugin_pkg'); console.log('name:', p.name, 'version:', p.version)" 2>/dev/null || true
fi
done
} > "$EXPORT_DIR/plugins-info.txt" 2>&1
fi
# ── 6. 打包 ──
echo "📦 打包中..."
cd "$(dirname "$EXPORT_DIR")"
tar -czf "$ARCHIVE_PATH" "$(basename "$EXPORT_DIR")"
# 计算大小
ARCHIVE_SIZE=$(du -sh "$ARCHIVE_PATH" | cut -f1)
echo ""
echo "═══════════════════════════════════════════════════"
echo " ✅ 诊断数据导出完成"
echo "═══════════════════════════════════════════════════"
echo ""
echo " 📦 压缩包: $ARCHIVE_PATH"
echo " 📏 大小: $ARCHIVE_SIZE"
echo ""
echo " 包含内容:"
echo " - env-info.txt — 环境信息、目录结构"
echo " - logs/ — OpenClaw 网关日志 + 滚动日志"
echo " - memory-tdai/ — 记忆插件全量数据 (L0~L3 + SQLite)"
echo " - openclaw-config-redacted.json — 脱敏后的配置文件"
echo " - plugins-info.txt — 插件安装信息"
echo ""
echo " ⚠️ 安全提示:"
echo " - 配置文件已自动脱敏（API Key、Token、Password 等已移除）"
echo " - models/secrets/channels/env 等敏感配置块已整体替换"
echo " - 记忆数据中可能包含用户对话内容，请确认后再发送"
echo ""
echo " 📤 请手动检查后发送给研发团队"
echo "═══════════════════════════════════════════════════"
scripts/export-tencent-vdb/export-tencent-vdb.ts
+634
View File
@@ -0,0 +1,634 @@
#!/usr/bin/env node
/**
* 腾讯云 VDB (Tencent VectorDB) 数据导出脚本
*
* 连接腾讯云向量数据库实例，查询指定数据库下 collection 的文档，导出为 .jsonl 文件。
* 仅支持腾讯云向量数据库（Tencent VectorDB），不支持其他厂商的向量数据库。
*
* 所有连接参数通过 CLI 传入，无需 .env 文件。
*
* 用法：
* node ./bin/export-tencent-vdb.mjs --url <地址> --username <用户名> --api-key <密钥> --database <库名>
* node ./bin/export-tencent-vdb.mjs --url <地址> --username <用户名> --api-key <密钥> --database <库名> --probe
* node ./bin/export-tencent-vdb.mjs --url <地址> --username <用户名> --api-key <密钥> --database <库名> -c <collection> -o /tmp/backup
*
* 输出：
* 默认输出到当前工作目录下的 ./vdb-export-YYYY-MM-DD/，可通过 -o 指定。
* <outputDir>/
* ├── <collection>.jsonl — 每行一个 JSON 文档
* ├── schemas.json — 导出的 collection 表结构（索引、embedding 配置等）
* └── export-meta.json — 导出元信息
*
* 导出字段说明：
* 默认行为：导出所有字段，但跳过 vector（稠密向量，1024维浮点数组，体积大）。
* 加 --include-vectors：导出全部字段，包括 vector，不跳过任何内容。
* 注：sparse_vector（BM25 稀疏向量）始终导出，不受此开关影响。
*
* 依赖：Node.js >= 18（内置 fetch）
*/
import fs from "node:fs";
import path from "node:path";
// ============================================================
// CLI 参数解析（含 VDB 连接信息）
// ============================================================
interface VDBConfig {
url: string;
username: string;
apiKey: string;
database: string;
timeout: number;
}
interface CliArgs {
// 连接参数
url?: string;
username?: string;
apiKey?: string;
database?: string;
timeout: number;
// 导出参数
output: string;
collection?: string;
filter?: string;
limit?: number;
offset: number;
includeVectors: boolean;
probe: boolean;
help: boolean;
}
const PAGE_SIZE = 100;
function parseArgs(): CliArgs {
const args = process.argv.slice(2);
const result: CliArgs = {
timeout: 30000,
output: `./vdb-export-${new Date().toISOString().slice(0, 10)}`,
offset: 0,
includeVectors: false,
probe: false,
help: false,
};
for (let i = 0; i < args.length; i++) {
switch (args[i]) {
case "--url":
result.url = args[++i];
break;
case "--username":
result.username = args[++i];
break;
case "--api-key":
result.apiKey = args[++i];
break;
case "--database":
result.database = args[++i];
break;
case "--timeout":
result.timeout = parseInt(args[++i], 10) || 30000;
break;
case "--output":
case "-o":
result.output = args[++i];
break;
case "--collection":
case "-c":
result.collection = args[++i];
break;
case "--filter":
case "-f":
result.filter = args[++i];
break;
case "--limit":
case "-l": {
const v = parseInt(args[++i], 10);
if (isNaN(v) || v < 1) {
console.error(`❌ --limit 必须 >= 1，收到: ${args[i]}`);
process.exit(1);
}
result.limit = v;
break;
}
case "--offset": {
const v = parseInt(args[++i], 10);
if (isNaN(v) || v < 0) {
console.error(`❌ --offset 必须 >= 0，收到: ${args[i]}`);
process.exit(1);
}
result.offset = v;
break;
}
case "--include-vectors":
result.includeVectors = true;
break;
case "--probe":
result.probe = true;
break;
case "--help":
case "-h":
result.help = true;
break;
}
}
return result;
}
function validateConfig(args: CliArgs): VDBConfig {
const missing: string[] = [];
if (!args.url) missing.push("--url");
if (!args.username) missing.push("--username");
if (!args.apiKey) missing.push("--api-key");
if (!args.database) missing.push("--database");
if (missing.length > 0) {
console.error("❌ 缺少必填参数：");
for (const k of missing) {
console.error(` - ${k}`);
}
console.error();
console.error("示例：");
console.error();
console.error(' node ./bin/export-tencent-vdb.mjs \\');
console.error(' --url "http://your-vdb-host:8100" \\');
console.error(' --username "root" \\');
console.error(' --api-key "your-api-key" \\');
console.error(' --database "your-database"');
console.error();
console.error("使用 --help 查看完整参数说明。");
process.exit(1);
}
return {
url: args.url!,
username: args.username!,
apiKey: args.apiKey!,
database: args.database!,
timeout: args.timeout,
};
}
function printHelp(): void {
console.log(`
腾讯云 VDB (Tencent VectorDB) 数据导出脚本
用法：
node ./bin/export-tencent-vdb.mjs [连接参数] [选项]
连接参数（必填）：
--url <地址> VDB 实例 HTTP 地址（如 http://your-vdb-host:8100）
--username <用户名> 认证用户名（如 root）
--api-key <密钥> 认证密钥
--database <库名> 数据库名称
选项：
--timeout <毫秒> 单次请求超时（默认: 30000）
-o, --output <目录> 输出目录（默认: ./vdb-export-YYYY-MM-DD）
-c, --collection <全名> 只导出指定 collection（全名匹配，不指定则导出所有）
-f, --filter <表达式> VDB Filter 过滤条件（如 'agent_id = "xxx"'）
-l, --limit <数量> 最多导出多少条（不指定则导出全部）
--offset <偏移> 从第几条开始（默认: 0），须为分页大小的整数倍
--include-vectors 保留 vector 稠密向量字段（默认跳过）
--probe 仅测试连通性，列出 collection 信息后退出
-h, --help 显示帮助
输出：
<outputDir>/
├── <collection全名>.jsonl 每行一个 JSON 文档
├── schemas.json 表结构
└── export-meta.json 导出元信息
导出字段说明：
默认跳过 vector（稠密向量），保留 sparse_vector（BM25）。
加 --include-vectors 导出全部字段。
示例：
# 测试连通性
node ./bin/export-tencent-vdb.mjs \\
--url "http://gz-vdb-xxx:8100" --username root --api-key "xxx" --database mydb \\
--probe
# 全量导出
node ./bin/export-tencent-vdb.mjs \\
--url "http://gz-vdb-xxx:8100" --username root --api-key "xxx" --database mydb
# 导出指定 collection 到指定目录
node ./bin/export-tencent-vdb.mjs \\
--url "http://gz-vdb-xxx:8100" --username root --api-key "xxx" --database mydb \\
-c mydb_l0_conversations -o /tmp/backup
# 带过滤条件
node ./bin/export-tencent-vdb.mjs \\
--url "http://gz-vdb-xxx:8100" --username root --api-key "xxx" --database mydb \\
-f 'role = "user"'
`);
}
// ============================================================
// VDB HTTP Client
// ============================================================
class VDBClient {
private baseUrl: string;
private authHeader: string;
private database: string;
private timeout: number;
constructor(cfg: VDBConfig) {
this.baseUrl = cfg.url.replace(/\/$/, "");
this.authHeader = `Bearer account=${cfg.username}&api_key=${cfg.apiKey}`;
this.database = cfg.database;
this.timeout = cfg.timeout;
}
async request<T>(apiPath: string, body: Record<string, unknown>): Promise<T> {
const url = `${this.baseUrl}${apiPath}`;
const resp = await fetch(url, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: this.authHeader,
},
body: JSON.stringify(body),
signal: AbortSignal.timeout(this.timeout),
});
if (!resp.ok) {
const text = await resp.text().catch(() => "(unable to read body)");
throw new Error(`VDB API error: HTTP ${resp.status} — ${text.slice(0, 500)}`);
}
const json = (await resp.json()) as { code: number; msg: string } & T;
if (json.code !== 0) {
throw new Error(`VDB API error [${apiPath}]: code=${json.code}, msg=${json.msg}`);
}
return json;
}
async listCollections(): Promise<
Array<{ collection: string; documentCount: number }>
> {
const result = await this.request<{
collections: Array<{
collection: string;
documentCount: number;
[key: string]: unknown;
}>;
}>("/collection/list", {
database: this.database,
});
return (result.collections || []).map((c) => ({
collection: c.collection,
documentCount: c.documentCount ?? 0,
}));
}
async queryDocuments(
collection: string,
options: {
limit: number;
offset: number;
filter?: string;
retrieveVector?: boolean;
},
): Promise<{
documents: Array<Record<string, unknown>>;
count: number;
}> {
const query: Record<string, unknown> = {
limit: options.limit,
offset: options.offset,
};
if (options.filter) {
query.filter = options.filter;
}
if (options.retrieveVector) {
query.retrieveVector = true;
}
const result = await this.request<{
documents: Array<Record<string, unknown>>;
count: number;
}>("/document/query", {
database: this.database,
collection,
readConsistency: "strongConsistency",
query,
});
return {
documents: result.documents || [],
count: result.count ?? 0,
};
}
async describeCollection(collection: string): Promise<Record<string, unknown>> {
const result = await this.request<{
collection: Record<string, unknown>;
}>("/collection/describe", {
database: this.database,
collection,
});
return result.collection || {};
}
}
// ============================================================
// 导出逻辑
// ============================================================
interface ExportOptions {
filter?: string;
limit?: number;
offset: number;
includeVectors: boolean;
expectedTotal?: number;
}
async function exportCollection(
client: VDBClient,
collection: string,
outputDir: string,
options: ExportOptions,
): Promise<{ docCount: number; filePath: string }> {
const filePath = path.join(outputDir, `${collection}.jsonl`);
const writeStream = fs.createWriteStream(filePath, { encoding: "utf-8" });
const isRangeMode = options.limit !== undefined;
const maxDocs = options.limit ?? Infinity;
const pageSize = isRangeMode ? Math.min(options.limit!, PAGE_SIZE) : PAGE_SIZE;
let currentOffset = options.offset;
let totalExported = 0;
let hasMore = true;
console.log(` 📦 ${collection}`);
if (options.expectedTotal !== undefined) {
console.log(` 文档总数: ${options.expectedTotal}`);
}
if (options.filter) {
console.log(` 过滤条件: ${options.filter}`);
}
if (isRangeMode) {
console.log(` 导出范围: offset=${options.offset}, limit=${options.limit}`);
}
while (hasMore && totalExported < maxDocs) {
const remaining = maxDocs - totalExported;
const thisPageSize = Math.min(pageSize, remaining);
try {
const result = await client.queryDocuments(collection, {
limit: thisPageSize,
offset: currentOffset,
filter: options.filter,
retrieveVector: options.includeVectors,
});
const docs = result.documents;
if (!docs || docs.length === 0) {
hasMore = false;
break;
}
for (const doc of docs) {
const exportDoc = { ...doc };
if (!options.includeVectors) {
delete exportDoc.vector;
}
writeStream.write(JSON.stringify(exportDoc) + "\n");
}
totalExported += docs.length;
currentOffset += docs.length;
if (options.expectedTotal !== undefined && !isRangeMode) {
const pct = Math.min(
100,
Math.round((totalExported / options.expectedTotal) * 100),
);
process.stdout.write(
`\r 进度: ${totalExported}/${options.expectedTotal} (${pct}%)`,
);
} else {
process.stdout.write(`\r 已导出: ${totalExported} 条`);
}
if (docs.length < thisPageSize) {
hasMore = false;
}
} catch (err) {
console.error(
`\n ❌ 查询失败 (offset=${currentOffset}): ${err instanceof Error ? err.message : String(err)}`,
);
hasMore = false;
}
}
writeStream.end();
await new Promise<void>((resolve) => writeStream.on("finish", resolve));
console.log(
`\n ✅ 完成: ${totalExported} 条 → ${path.basename(filePath)}`,
);
return { docCount: totalExported, filePath };
}
// ============================================================
// Main
// ============================================================
async function main(): Promise<void> {
const args = parseArgs();
if (args.help) {
printHelp();
process.exit(0);
}
const config = validateConfig(args);
console.log("╔═══════════════════════════════════════════════════╗");
console.log("║ 腾讯云 VDB (Tencent VectorDB) 数据导出工具 ║");
console.log("╚═══════════════════════════════════════════════════╝");
console.log();
console.log(`📌 VDB 地址: ${config.url}`);
console.log(`📌 数据库: ${config.database}`);
console.log(`📌 输出目录: ${args.output}`);
if (args.collection) {
console.log(`📌 指定导出: ${args.collection}`);
}
if (args.filter) {
console.log(`📌 过滤条件: ${args.filter}`);
}
if (args.limit !== undefined) {
console.log(`📌 导出上限: ${args.limit} 条`);
}
if (args.offset > 0) {
console.log(`📌 起始偏移: ${args.offset}`);
}
if (args.includeVectors) {
console.log(`📌 包含向量: 是`);
}
console.log();
fs.mkdirSync(args.output, { recursive: true });
const client = new VDBClient(config);
let allCollections: Array<{ collection: string; documentCount: number }>;
try {
allCollections = await client.listCollections();
} catch (err) {
console.error(
`❌ 列出 collection 失败: ${err instanceof Error ? err.message : String(err)}`,
);
process.exit(1);
}
let targetCollections: Array<{ collection: string; documentCount: number }>;
if (args.collection) {
const found = allCollections.find((c) => c.collection === args.collection);
if (!found) {
console.error(
`❌ Collection "${args.collection}" 不存在。可用的 collection：`,
);
for (const c of allCollections) {
console.error(` - ${c.collection} (${c.documentCount} 条)`);
}
process.exit(1);
}
targetCollections = [found];
} else {
targetCollections = allCollections;
console.log(
`🔍 找到 ${targetCollections.length} 个 collection：`,
);
for (const c of targetCollections) {
console.log(` - ${c.collection} (${c.documentCount} 条)`);
}
}
if (targetCollections.length === 0) {
console.log("⚠️ 数据库中没有 collection，无数据可导出");
process.exit(0);
}
// --probe 模式：只测试连通性，列出信息后退出
if (args.probe) {
console.log();
console.log("✅ 连通性测试通过");
console.log();
console.log(` VDB 地址: ${config.url}`);
console.log(` 数据库: ${config.database}`);
console.log(` Collection: ${targetCollections.length} 个`);
const totalDocs = targetCollections.reduce((s, c) => s + c.documentCount, 0);
console.log(` 总文档数: ${totalDocs}`);
console.log();
for (const c of targetCollections) {
console.log(` - ${c.collection} (${c.documentCount} 条)`);
}
console.log();
process.exit(0);
}
console.log();
// 获取并保存表结构
const schemas: Record<string, Record<string, unknown>> = {};
console.log("📐 获取表结构...");
for (const col of targetCollections) {
try {
const schema = await client.describeCollection(col.collection);
schemas[col.collection] = schema;
const indexCount = Array.isArray(schema.indexes) ? schema.indexes.length : 0;
const emb = schema.embedding as Record<string, unknown> | undefined;
const embInfo = emb ? `embedding=${emb.field}→${emb.model}` : "无 embedding";
console.log(` ✅ ${col.collection} (${indexCount} 个索引, ${embInfo})`);
} catch (err) {
console.error(
` ⚠️ ${col.collection} 表结构获取失败: ${err instanceof Error ? err.message : String(err)}`,
);
}
}
console.log();
const schemaPath = path.join(args.output, "schemas.json");
fs.writeFileSync(schemaPath, JSON.stringify(schemas, null, 2) + "\n");
const exportResults: Array<{
collection: string;
docCount: number;
filePath: string;
}> = [];
for (const col of targetCollections) {
try {
const result = await exportCollection(client, col.collection, args.output, {
filter: args.filter,
limit: args.limit,
offset: args.offset,
includeVectors: args.includeVectors,
expectedTotal: col.documentCount,
});
exportResults.push({ collection: col.collection, ...result });
} catch (err) {
console.error(
`❌ 导出 ${col.collection} 失败: ${err instanceof Error ? err.message : String(err)}`,
);
exportResults.push({
collection: col.collection,
docCount: 0,
filePath: "",
});
}
console.log();
}
const meta = {
exportedAt: new Date().toISOString(),
vdbUrl: config.url,
database: config.database,
filter: args.filter ?? null,
offset: args.offset,
limit: args.limit ?? null,
includeVectors: args.includeVectors,
collections: exportResults.map((r) => ({
collection: r.collection,
documentCount: r.docCount,
file: r.filePath ? path.basename(r.filePath) : null,
})),
totalDocuments: exportResults.reduce((sum, r) => sum + r.docCount, 0),
};
const metaPath = path.join(args.output, "export-meta.json");
fs.writeFileSync(metaPath, JSON.stringify(meta, null, 2) + "\n");
console.log("═══════════════════════════════════════════════════");
console.log(" ✅ 导出完成");
console.log("═══════════════════════════════════════════════════");
console.log();
console.log(` 📁 输出目录: ${args.output}`);
console.log(` 📊 总文档数: ${meta.totalDocuments}`);
for (const r of exportResults) {
const status = r.docCount > 0 ? "✅" : "⚠️";
console.log(
` ${status} ${r.collection}: ${r.docCount} 条`,
);
}
console.log(` 📋 元信息: ${path.basename(metaPath)}`);
console.log(` 📐 表结构: ${path.basename(schemaPath)}`);
console.log();
}
main().catch((err) => {
console.error(
`\n❌ 导出失败: ${err instanceof Error ? err.message : String(err)}`,
);
process.exit(1);
});
scripts/export-tencent-vdb/tsconfig.json
+19
View File
@@ -0,0 +1,19 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "Node16",
"moduleResolution": "Node16",
"outDir": "./dist",
"rootDir": ".",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"types": ["node"],
"declaration": false,
"sourceMap": false
},
"include": ["export-tencent-vdb.ts"],
"exclude": ["dist", "node_modules", "docs"]
}
scripts/install-hermes-plugin-v2.sh
Executable
+163
View File
@@ -0,0 +1,163 @@
#!/usr/bin/env bash
set -euo pipefail
log() { printf '[install-hermes-plugin-v2] %s\n' "$*" >&2; }
fail() { printf '[install-hermes-plugin-v2][ERROR] %s\n' "$*" >&2; exit 1; }
need_cmd() { command -v "$1" >/dev/null 2>&1 || fail "missing command: $1"; }
# Target user/home detection follows the legacy Hermes installer convention:
# 1. INSTALL_AS_USER, 2. SUDO_USER, 3. current user.
USERNAME="${INSTALL_AS_USER:-${SUDO_USER:-$(whoami)}}"
USER_HOME="$(eval echo "~$USERNAME")"
SDK_WHEEL_URL="${SDK_WHEEL_URL:-https://cnb.cool/tencent/cloud/nosql/nosql-utilities/-/commit-assets/download/cc74bd6dbc931727da9ab6907b5ab1a07d7afd9d/tencentdb_agent_memory_sdk_python-0.1.0-py3-none-any.whl}"
SDK_WHEEL_NAME="tencentdb_agent_memory_sdk_python-0.1.0-py3-none-any.whl"
FORCE="${FORCE:-0}"
ALLOW_SYSTEM_PYTHON="${ALLOW_SYSTEM_PYTHON:-0}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
PROVIDER_SRC="${HERMES_PROVIDER_SRC:-$REPO_ROOT/hermes-plugin/memory/memory_tencentdb_v2}"
HERMES_HOME="${HERMES_HOME:-$USER_HOME/.hermes}"
HERMES_AGENT_DIR="${HERMES_AGENT_DIR:-$HERMES_HOME/hermes-agent}"
HERMES_VENV_DIR="${HERMES_VENV_DIR:-$HERMES_AGENT_DIR/venv}"
HERMES_CONFIG="${HERMES_CONFIG:-$HERMES_HOME/config.yaml}"
HERMES_ENV="${HERMES_ENV:-$HERMES_HOME/.env}"
HERMES_MEMORY_PLUGIN_DIR="${HERMES_MEMORY_PLUGIN_DIR:-$HERMES_AGENT_DIR/plugins/memory}"
PROVIDER_TARGET="$HERMES_MEMORY_PLUGIN_DIR/memory_tencentdb_v2"
# Install the SDK into the Python environment that Hermes actually uses.
# Selection order:
# 1. PYTHON_BIN, if explicitly provided
# 2. HERMES_VENV_DIR/bin/python, if present
# 3. Python interpreter from the installed `hermes` command shebang, if discoverable
# 4. system python3 only when ALLOW_SYSTEM_PYTHON=1
if [[ -n "${PYTHON_BIN:-}" ]]; then
:
elif [[ -x "$HERMES_VENV_DIR/bin/python" ]]; then
PYTHON_BIN="$HERMES_VENV_DIR/bin/python"
elif command -v hermes >/dev/null 2>&1; then
HERMES_BIN="$(command -v hermes)"
HERMES_SHEBANG="$(head -n 1 "$HERMES_BIN" 2>/dev/null || true)"
if [[ "$HERMES_SHEBANG" == '#!'*python* ]]; then
HERMES_SHEBANG="${HERMES_SHEBANG#'#!'}"
read -r HERMES_SHEBANG_CMD HERMES_SHEBANG_ARG _ <<<"$HERMES_SHEBANG"
if [[ "$(basename "$HERMES_SHEBANG_CMD")" == "env" && -n "${HERMES_SHEBANG_ARG:-}" ]]; then
PYTHON_BIN="$(command -v "$HERMES_SHEBANG_ARG" || true)"
elif [[ -x "$HERMES_SHEBANG_CMD" ]]; then
PYTHON_BIN="$HERMES_SHEBANG_CMD"
fi
fi
fi
if [[ -z "${PYTHON_BIN:-}" ]]; then
if [[ "$ALLOW_SYSTEM_PYTHON" == "1" ]]; then
PYTHON_BIN="python3"
else
fail "Hermes Python not found. Set PYTHON_BIN=/path/to/hermes/python or HERMES_VENV_DIR=/path/to/venv. Refusing to use system python3 by default to avoid externally-managed-environment installs."
fi
fi
TDAI_MEMORY_ENDPOINT="${TDAI_MEMORY_ENDPOINT:-http://127.0.0.1:8420}"
TDAI_MEMORY_API_KEY="${TDAI_MEMORY_API_KEY:-local}"
TDAI_MEMORY_SERVICE_ID="${TDAI_MEMORY_SERVICE_ID:-default}"
WRITE_HERMES_ENV="${WRITE_HERMES_ENV:-1}"
need_cmd curl
need_cmd "$PYTHON_BIN"
if [[ ! -d "$PROVIDER_SRC" ]]; then
fail "Hermes provider directory not found: $PROVIDER_SRC"
fi
if [[ ! -d "$HERMES_AGENT_DIR" ]]; then
log "WARN: Hermes agent dir not found: $HERMES_AGENT_DIR"
log " Set HERMES_AGENT_DIR if Hermes is installed elsewhere."
fi
log "Downloading Python SDK wheel"
TMP_DIR="$(mktemp -d)"
cleanup() { rm -rf "$TMP_DIR"; }
trap cleanup EXIT
curl -fL -o "$TMP_DIR/$SDK_WHEEL_NAME" "$SDK_WHEEL_URL"
log "Installing Python SDK with $PYTHON_BIN"
"$PYTHON_BIN" -m pip install "$TMP_DIR/$SDK_WHEEL_NAME"
log "Installing Hermes provider"
mkdir -p "$HERMES_MEMORY_PLUGIN_DIR"
if [[ -e "$PROVIDER_TARGET" || -L "$PROVIDER_TARGET" ]]; then
if [[ "$FORCE" == "1" ]]; then
rm -rf "$PROVIDER_TARGET"
else
fail "target already exists: $PROVIDER_TARGET (set FORCE=1 to overwrite)"
fi
fi
ln -s "$PROVIDER_SRC" "$PROVIDER_TARGET"
log "Provider linked: $PROVIDER_TARGET -> $PROVIDER_SRC"
log "Checking Hermes config"
if [[ -f "$HERMES_CONFIG" ]]; then
if sed -n '/^memory:/,/^[[:alpha:]_][[:alnum:]_]*:/p' "$HERMES_CONFIG" | grep -q 'provider: memory_tencentdb_v2'; then
log "memory.provider already set to memory_tencentdb_v2"
else
log "Provider installed but NOT enabled by default. Add/edit in $HERMES_CONFIG:"
cat >&2 <<'EOF'
memory:
provider: memory_tencentdb_v2
EOF
fi
else
log "WARN: $HERMES_CONFIG not found; create it or run Hermes installer first."
log " To enable the provider, add:"
cat >&2 <<'EOF'
memory:
provider: memory_tencentdb_v2
EOF
fi
_update_env() {
local key="$1"
local value="$2"
local file="$3"
mkdir -p "$(dirname "$file")"
touch "$file"
local tmp
tmp="$(mktemp)"
grep -v -E "^(# *)?${key}=" "$file" > "$tmp" || true
local escaped="$value"
escaped="${escaped//\\/\\\\}"
escaped="${escaped//\"/\\\"}"
printf '%s="%s"\n' "$key" "$escaped" >> "$tmp"
mv "$tmp" "$file"
}
if [[ "$WRITE_HERMES_ENV" == "1" ]]; then
log "Writing Memory SDK env vars to $HERMES_ENV"
_update_env "TDAI_MEMORY_ENDPOINT" "$TDAI_MEMORY_ENDPOINT" "$HERMES_ENV"
_update_env "TDAI_MEMORY_API_KEY" "$TDAI_MEMORY_API_KEY" "$HERMES_ENV"
_update_env "TDAI_MEMORY_SERVICE_ID" "$TDAI_MEMORY_SERVICE_ID" "$HERMES_ENV"
fi
cat >&2 <<EOF
[install-hermes-plugin-v2] Done.
Provider installed at:
$PROVIDER_TARGET
SDK env file:
$HERMES_ENV
If not already enabled, add this to $HERMES_CONFIG:
memory:
provider: memory_tencentdb_v2
Standalone Gateway env:
TDAI_MEMORY_ENDPOINT="$TDAI_MEMORY_ENDPOINT"
TDAI_MEMORY_API_KEY="$TDAI_MEMORY_API_KEY"
TDAI_MEMORY_SERVICE_ID="$TDAI_MEMORY_SERVICE_ID"
EOF
scripts/install-openclaw-plugin-v2.sh
Executable
+187
View File
@@ -0,0 +1,187 @@
#!/usr/bin/env bash
set -euo pipefail
log() { printf '[install-openclaw-plugin-v2] %s\n' "$*" >&2; }
fail() { printf '[install-openclaw-plugin-v2][ERROR] %s\n' "$*" >&2; exit 1; }
need_cmd() { command -v "$1" >/dev/null 2>&1 || fail "missing command: $1"; }
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
PLUGIN_DIR="${OPENCLAW_PLUGIN_DIR:-$REPO_ROOT/openclaw-plugin}"
INSTALL_OPENCLAW="${INSTALL_OPENCLAW:-1}"
OPENCLAW_INSTALL_FLAGS="${OPENCLAW_INSTALL_FLAGS:-}"
WRITE_OPENCLAW_CONFIG="${WRITE_OPENCLAW_CONFIG:-1}"
OPENCLAW_CONFIG_FILE="${OPENCLAW_CONFIG_FILE:-$HOME/.openclaw/openclaw.json}"
MEMORY_PLUGIN_ID="memory-tencentdb-client"
TDAI_MEMORY_ENDPOINT="${TDAI_MEMORY_ENDPOINT:-http://127.0.0.1:8420}"
TDAI_MEMORY_API_KEY="${TDAI_MEMORY_API_KEY:-local}"
TDAI_MEMORY_INSTANCE_ID="${TDAI_MEMORY_INSTANCE_ID:-${TDAI_MEMORY_SERVICE_ID:-default}}"
TDAI_MEMORY_RECALL_MAX_RESULTS="${TDAI_MEMORY_RECALL_MAX_RESULTS:-5}"
TDAI_MEMORY_INCLUDE_PERSONA="${TDAI_MEMORY_INCLUDE_PERSONA:-true}"
TDAI_MEMORY_INCLUDE_SCENE_NAV="${TDAI_MEMORY_INCLUDE_SCENE_NAV:-true}"
TDAI_MEMORY_CAPTURE_ENABLED="${TDAI_MEMORY_CAPTURE_ENABLED:-true}"
TDAI_MEMORY_ALLOW_PROMPT_INJECTION="${TDAI_MEMORY_ALLOW_PROMPT_INJECTION:-true}"
TDAI_MEMORY_ALLOW_CONVERSATION_ACCESS="${TDAI_MEMORY_ALLOW_CONVERSATION_ACCESS:-true}"
need_cmd npm
need_cmd node
need_cmd curl
if [[ ! -d "$PLUGIN_DIR" ]]; then
fail "OpenClaw plugin directory not found: $PLUGIN_DIR"
fi
if ! command -v openclaw >/dev/null 2>&1; then
if [[ "$INSTALL_OPENCLAW" == "1" ]]; then
log "OpenClaw CLI not found; installing from https://get.openclaw.dev"
curl -fsSL https://get.openclaw.dev | bash
export PATH="$HOME/.local/bin:$HOME/bin:/usr/local/bin:$PATH"
else
fail "OpenClaw CLI not found. Install it first or run with INSTALL_OPENCLAW=1."
fi
fi
command -v openclaw >/dev/null 2>&1 || fail "OpenClaw CLI still not found after install; please check PATH"
OPENCLAW_VERSION_RAW="$(openclaw --version 2>/dev/null || printf 'unknown')"
log "OpenClaw version: $OPENCLAW_VERSION_RAW"
# Hook policy fields (allowPromptInjection / allowConversationAccess) were
# accepted by the gateway zod schema starting at 2026.4.24 (PR #71221).
# 2026.4.23 and earlier use `.strict()` and will REFUSE to start the gateway
# if these fields are present. We therefore version-gate the config write to
# match `src/utils/ensure-hook-policy.ts::HOOK_POLICY_MIN_VERSION`.
HOOK_POLICY_MIN="2026.4.24"
WRITE_HOOK_POLICY=0
if [[ "$OPENCLAW_VERSION_RAW" =~ ([0-9]+)\.([0-9]+)\.([0-9]+) ]]; then
v_major="${BASH_REMATCH[1]}"
v_minor="${BASH_REMATCH[2]}"
v_patch="${BASH_REMATCH[3]}"
IFS=. read -r m_major m_minor m_patch <<<"$HOOK_POLICY_MIN"
if (( v_major > m_major )) \
|| { (( v_major == m_major )) && (( v_minor > m_minor )); } \
|| { (( v_major == m_major )) && (( v_minor == m_minor )) && (( v_patch >= m_patch )); }; then
WRITE_HOOK_POLICY=1
fi
fi
if [[ "$WRITE_HOOK_POLICY" == "1" ]]; then
log "Detected OpenClaw >= $HOOK_POLICY_MIN — will write hooks.allowPromptInjection / allowConversationAccess"
else
log "Detected OpenClaw < $HOOK_POLICY_MIN (or version unparseable) — will SKIP hooks.* policy fields (older gateway schemas reject them)"
fi
log "Installing plugin dependencies"
(cd "$PLUGIN_DIR" && npm install)
log "Building plugin"
(cd "$PLUGIN_DIR" && npm run build)
log "Installing linked plugin into OpenClaw: $PLUGIN_DIR"
# shellcheck disable=SC2086
openclaw plugins install -l "$PLUGIN_DIR" $OPENCLAW_INSTALL_FLAGS
if [[ "$WRITE_OPENCLAW_CONFIG" == "1" ]]; then
log "Updating OpenClaw config: $OPENCLAW_CONFIG_FILE"
mkdir -p "$(dirname "$OPENCLAW_CONFIG_FILE")"
if [[ -f "$OPENCLAW_CONFIG_FILE" ]]; then
cp "$OPENCLAW_CONFIG_FILE" "$OPENCLAW_CONFIG_FILE.bak.$(date +%Y%m%d%H%M%S)"
fi
OPENCLAW_CONFIG_FILE="$OPENCLAW_CONFIG_FILE" \
MEMORY_PLUGIN_ID="$MEMORY_PLUGIN_ID" \
TDAI_MEMORY_ENDPOINT="$TDAI_MEMORY_ENDPOINT" \
TDAI_MEMORY_API_KEY="$TDAI_MEMORY_API_KEY" \
TDAI_MEMORY_INSTANCE_ID="$TDAI_MEMORY_INSTANCE_ID" \
TDAI_MEMORY_RECALL_MAX_RESULTS="$TDAI_MEMORY_RECALL_MAX_RESULTS" \
TDAI_MEMORY_INCLUDE_PERSONA="$TDAI_MEMORY_INCLUDE_PERSONA" \
TDAI_MEMORY_INCLUDE_SCENE_NAV="$TDAI_MEMORY_INCLUDE_SCENE_NAV" \
TDAI_MEMORY_CAPTURE_ENABLED="$TDAI_MEMORY_CAPTURE_ENABLED" \
TDAI_MEMORY_ALLOW_PROMPT_INJECTION="$TDAI_MEMORY_ALLOW_PROMPT_INJECTION" \
TDAI_MEMORY_ALLOW_CONVERSATION_ACCESS="$TDAI_MEMORY_ALLOW_CONVERSATION_ACCESS" \
WRITE_HOOK_POLICY="$WRITE_HOOK_POLICY" \
node <<'NODE'
const fs = require('node:fs');
const file = process.env.OPENCLAW_CONFIG_FILE;
const pluginId = process.env.MEMORY_PLUGIN_ID;
function parseBool(value, fallback) {
if (value === undefined || value === '') return fallback;
return !['0', 'false', 'no', 'off'].includes(String(value).toLowerCase());
}
function parsePositiveInt(value, fallback) {
const parsed = Number.parseInt(String(value ?? ''), 10);
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
}
let config = {};
if (fs.existsSync(file)) {
const raw = fs.readFileSync(file, 'utf8').trim();
if (raw) config = JSON.parse(raw);
}
config.plugins ??= {};
config.plugins.slots ??= {};
config.plugins.entries ??= {};
config.plugins.slots.memory = pluginId;
const entry = config.plugins.entries[pluginId] ?? {};
entry.enabled = true;
// Hook policy fields (allowPromptInjection / allowConversationAccess) are
// only safe to write on OpenClaw >= 2026.4.24. The shell wrapper sets
// WRITE_HOOK_POLICY=1 after parsing `openclaw --version`; on older gateways
// it stays 0 and we *remove* any stale fields to keep the gateway bootable.
const writeHookPolicy = process.env.WRITE_HOOK_POLICY === '1';
if (writeHookPolicy) {
entry.hooks ??= {};
entry.hooks.allowPromptInjection = parseBool(process.env.TDAI_MEMORY_ALLOW_PROMPT_INJECTION, entry.hooks.allowPromptInjection ?? true);
entry.hooks.allowConversationAccess = parseBool(process.env.TDAI_MEMORY_ALLOW_CONVERSATION_ACCESS, entry.hooks.allowConversationAccess ?? true);
} else if (entry.hooks && (entry.hooks.allowPromptInjection !== undefined || entry.hooks.allowConversationAccess !== undefined)) {
// Older gateways (zod .strict()) reject these fields; strip them defensively.
delete entry.hooks.allowPromptInjection;
delete entry.hooks.allowConversationAccess;
if (Object.keys(entry.hooks).length === 0) delete entry.hooks;
}
entry.config ??= {};
entry.config.server ??= {};
entry.config.recall ??= {};
entry.config.capture ??= {};
entry.config.server.url = process.env.TDAI_MEMORY_ENDPOINT || entry.config.server.url || 'http://127.0.0.1:8420';
entry.config.server.apiKey = process.env.TDAI_MEMORY_API_KEY || entry.config.server.apiKey || 'local';
entry.config.server.instanceId = process.env.TDAI_MEMORY_INSTANCE_ID || entry.config.server.instanceId || 'default';
entry.config.recall.maxResults = parsePositiveInt(process.env.TDAI_MEMORY_RECALL_MAX_RESULTS, entry.config.recall.maxResults ?? 5);
entry.config.recall.includePersona = parseBool(process.env.TDAI_MEMORY_INCLUDE_PERSONA, entry.config.recall.includePersona ?? true);
entry.config.recall.includeSceneNav = parseBool(process.env.TDAI_MEMORY_INCLUDE_SCENE_NAV, entry.config.recall.includeSceneNav ?? true);
entry.config.capture.enabled = parseBool(process.env.TDAI_MEMORY_CAPTURE_ENABLED, entry.config.capture.enabled ?? true);
config.plugins.entries[pluginId] = entry;
fs.writeFileSync(file, `${JSON.stringify(config, null, 2)}\n`, { mode: 0o600 });
NODE
else
log "Skipping OpenClaw config update because WRITE_OPENCLAW_CONFIG=$WRITE_OPENCLAW_CONFIG"
fi
if [[ "$WRITE_HOOK_POLICY" == "1" ]]; then
HOOK_SUMMARY=" plugins.entries[\"$MEMORY_PLUGIN_ID\"].hooks.allowPromptInjection = $TDAI_MEMORY_ALLOW_PROMPT_INJECTION
plugins.entries[\"$MEMORY_PLUGIN_ID\"].hooks.allowConversationAccess = $TDAI_MEMORY_ALLOW_CONVERSATION_ACCESS"
else
HOOK_SUMMARY=" plugins.entries[\"$MEMORY_PLUGIN_ID\"].hooks.* = (skipped — OpenClaw < $HOOK_POLICY_MIN does not accept these fields)"
fi
cat >&2 <<EOF
[install-openclaw-plugin-v2] Done.
Memory plugin configured:
plugins.slots.memory = "$MEMORY_PLUGIN_ID"
plugins.entries["$MEMORY_PLUGIN_ID"].enabled = true
$HOOK_SUMMARY
server.url = "$TDAI_MEMORY_ENDPOINT"
server.apiKey = "$TDAI_MEMORY_API_KEY"
server.instanceId = "$TDAI_MEMORY_INSTANCE_ID"
Next steps:
1. Ensure Memory Gateway is running at $TDAI_MEMORY_ENDPOINT.
2. Restart OpenClaw Gateway if needed:
openclaw gateway restart
EOF
scripts/install_hermes_memory_tencentdb.sh
Executable
+368
View File
@@ -0,0 +1,368 @@
#!/bin/bash
#
# install_memory_tencentdb.sh
#
# 在 install_hermes_ubuntu.sh 之后执行，用于：
# 1. 通过 npm 下载 @tencentdb-agent-memory/memory-tencentdb@latest 到
# $MEMORY_TENCENTDB_ROOT/tdai-memory-openclaw-plugin（默认 ~/.memory-tencentdb/tdai-memory-openclaw-plugin）
# 2. 安装 Gateway 的 Node.js 依赖（npm install）
# 3. 配置 hermes config.yaml 使用 memory_tencentdb 记忆提供者
# 4. 设置 Gateway 自动启动环境变量
#
# 路径约定（全部位于 ~/.memory-tencentdb/ 之下，可通过环境变量覆盖）：
# $MEMORY_TENCENTDB_ROOT 默认 ~/.memory-tencentdb
# $TDAI_INSTALL_DIR 默认 $MEMORY_TENCENTDB_ROOT/tdai-memory-openclaw-plugin
# $TDAI_DATA_DIR 默认 $MEMORY_TENCENTDB_ROOT/memory-tdai
#
# 旧版本（<= 0.3.x）使用 ~/tdai-memory-openclaw-plugin 与 ~/memory-tdai；
# 本脚本会在执行前自动迁移这两个旧目录到新位置（见 Step 0）。
#
# 使用方式：
# 以目标用户身份执行（推荐）：
# su - <username> -c "bash ~/install_memory_tencentdb.sh"
# # 或直接以该用户登录后执行
# bash ~/install_memory_tencentdb.sh
#
# 以 root 身份执行（镜像构建场景）：
# bash ~/install_memory_tencentdb.sh
# # root 会自动 su 切换到目标用户执行，完成后修复权限
#
# 前置条件：
# - install_hermes_ubuntu.sh 已执行完成（hermes-agent 已安装）
# - Node.js >= 22 已安装
set -e
# 动态获取目标安装用户及其 HOME 目录。
# 优先级：
# 1. 显式 ``INSTALL_AS_USER`` 环境变量（管理员脚本场景：root 跑安装但
# 想为另一个用户配置）
# 2. ``SUDO_USER``（被 ``sudo`` 调用时，切回原用户而不是 root）
# 3. ``whoami`` —— 当前 EUID 对应的用户
#
# 注意：当 root 直接 ssh 登录跑（非 sudo）时，前两个都不会被设置，
# ``whoami`` 返回 ``root``。下面的 ``id -u`` == 0 分支会识别这种"目标
# 就是 root"的情况、跳过 ``su - root`` 递归。
USERNAME="${INSTALL_AS_USER:-${SUDO_USER:-$(whoami)}}"
USER_HOME=$(eval echo ~$USERNAME)
# npm 包名
NPM_PACKAGE="@tencentdb-agent-memory/memory-tencentdb@latest"
# Hermes 路径
HERMES_HOME="$USER_HOME/.hermes"
# HERMES_AGENT_DIR（fix: issue #18）
# 用户通过环境变量传什么就用什么；未设置时 fallback 到传统路径。
# 如果目录不存在，后续前置检查会统一报错。
HERMES_AGENT_DIR="${HERMES_AGENT_DIR:-$HERMES_HOME/hermes-agent}"
HERMES_CONFIG="$HERMES_HOME/config.yaml"
# memory-tencentdb 统一根目录（所有 tdai 相关数据/代码都收纳在此）
# 可通过环境变量 MEMORY_TENCENTDB_ROOT 覆盖
MEMORY_TENCENTDB_ROOT="${MEMORY_TENCENTDB_ROOT:-$USER_HOME/.memory-tencentdb}"
# tdai 解压目标目录（位于统一根目录下）
# 可通过环境变量 TDAI_INSTALL_DIR 覆盖
TDAI_INSTALL_DIR="${TDAI_INSTALL_DIR:-$MEMORY_TENCENTDB_ROOT/tdai-memory-openclaw-plugin}"
# tdai 数据目录（Gateway baseDir，位于统一根目录下）
# 可通过环境变量 TDAI_DATA_DIR 覆盖
TDAI_DATA_DIR="${TDAI_DATA_DIR:-$MEMORY_TENCENTDB_ROOT/memory-tdai}"
# 旧路径（仅用于自动迁移）
LEGACY_INSTALL_DIR="$USER_HOME/tdai-memory-openclaw-plugin"
LEGACY_DATA_DIR="$USER_HOME/memory-tdai"
# ==================== root → 自动切换到目标用户 ====================
# 与 install_hermes_ubuntu.sh 保持一致：如果以 root 执行且目标用户不是
# root，自动 su 切到目标用户运行实际安装逻辑。
#
# 如果当前是 root 且目标用户也是 root（``USERNAME=root``，例如直接 ssh
# 登录 root 跑安装），跳过 ``su - root`` —— 否则会无限递归（``su - root``
# 进入的仍是 root，又走到这个分支，再次 su，永远停不下来）。见 issue #20。
if [ "$(id -u)" -eq 0 ] && [ "$USERNAME" != "root" ]; then
echo "[memory-tencentdb] Running as root, switching to $USERNAME for installation..."
# 验证前置条件
if [ ! -d "$HERMES_AGENT_DIR" ]; then
echo "[ERROR] Hermes agent not found at $HERMES_AGENT_DIR"
echo "[ERROR] Please run install_hermes_ubuntu.sh first."
exit 1
fi
# 切换到目标用户执行
TEMP_SCRIPT=$(mktemp /tmp/memory-tencentdb-install-XXXXXX.sh)
cp "${BASH_SOURCE[0]}" "$TEMP_SCRIPT"
chmod 755 "$TEMP_SCRIPT"
su - $USERNAME -c "bash $TEMP_SCRIPT" </dev/null
# 修复权限
echo "[memory-tencentdb] Fixing permissions..."
chown -R $USERNAME:$USERNAME "$USER_HOME"
rm -f "$TEMP_SCRIPT"
echo "[memory-tencentdb] Installation completed successfully"
exit 0
elif [ "$(id -u)" -eq 0 ]; then
# 当前是 root 且目标用户也是 root：直接以 root 跑后续安装逻辑，
# 不再走 ``su -`` 切换（避免 #20 的递归）。
echo "[memory-tencentdb] Running as root; target user is also root — installing in place."
fi
# ==================== 用户阶段（核心安装逻辑） ====================
echo "[memory-tencentdb] Installing memory-tencentdb plugin (user: $(whoami))..."
# 验证前置条件
if [ ! -d "$HERMES_AGENT_DIR" ]; then
echo "[ERROR] Hermes agent not found at $HERMES_AGENT_DIR"
echo "[ERROR] Please run install_hermes_ubuntu.sh first."
exit 1
fi
# 加载 hermes 环境（PATH 中需要 node/npm）
if [ -f /etc/profile.d/hermes-env.sh ]; then
source /etc/profile.d/hermes-env.sh
fi
# 确保统一根目录存在
mkdir -p "$MEMORY_TENCENTDB_ROOT"
# ---------- Step 0: 自动迁移旧路径（向后兼容） ----------
#
# 历史版本把 tdai 解压到 ~/tdai-memory-openclaw-plugin、数据放在 ~/memory-tdai。
# 现在统一收纳到 ~/.memory-tencentdb/ 之下，这里做一次性自动迁移。
# 已经在新位置时跳过；新旧都存在时打印警告并保留新位置不动。
migrate_legacy_dir() {
local legacy="$1"
local target="$2"
local label="$3"
if [ ! -e "$legacy" ]; then
return 0
fi
if [ -L "$legacy" ]; then
# 旧位置是 symlink，直接清掉
echo "[memory-tencentdb] Removing legacy symlink: $legacy"
rm -f "$legacy"
return 0
fi
if [ -e "$target" ]; then
echo "[memory-tencentdb] WARN: legacy $label dir exists at $legacy but new location $target also exists." >&2
echo "[memory-tencentdb] WARN: keeping new location; please review and remove $legacy manually if obsolete." >&2
return 0
fi
echo "[memory-tencentdb] Migrating legacy $label dir: $legacy -> $target"
mkdir -p "$(dirname "$target")"
mv "$legacy" "$target"
}
migrate_legacy_dir "$LEGACY_INSTALL_DIR" "$TDAI_INSTALL_DIR" "install"
migrate_legacy_dir "$LEGACY_DATA_DIR" "$TDAI_DATA_DIR" "data"
# ---------- Step 1: 通过 npm 下载包并提取到 $TDAI_INSTALL_DIR ----------
echo "[memory-tencentdb] Step 1: Downloading $NPM_PACKAGE via npm..."
# 清理旧安装
rm -rf "$TDAI_INSTALL_DIR"
# 使用临时目录通过 npm install 下载包
TEMP_DOWNLOAD=$(mktemp -d /tmp/memory-tencentdb-download-XXXXXX)
cd "$TEMP_DOWNLOAD"
npm init -y --silent > /dev/null 2>&1
npm install "$NPM_PACKAGE" --omit=dev 2>&1 | tail -5
# 包安装后位于 node_modules/@tencentdb-agent-memory/memory-tencentdb
PACK_DIR="$TEMP_DOWNLOAD/node_modules/@tencentdb-agent-memory/memory-tencentdb"
if [ ! -d "$PACK_DIR" ]; then
echo "[ERROR] Downloaded package directory not found at $PACK_DIR"
rm -rf "$TEMP_DOWNLOAD"
exit 1
fi
# 将包内容移动到目标安装目录
mkdir -p "$(dirname "$TDAI_INSTALL_DIR")"
cp -r "$PACK_DIR" "$TDAI_INSTALL_DIR"
echo "[memory-tencentdb] Package downloaded and extracted to $TDAI_INSTALL_DIR"
# ---------- Step 2: 安装 Gateway Node.js 依赖 ----------
echo "[memory-tencentdb] Step 2: Installing Gateway dependencies..."
cd "$TDAI_INSTALL_DIR"
echo "[memory-tencentdb] Running npm install (this may take a while)..."
npm install --omit=dev 2>&1 | tail -5
# 安装 tsx（Gateway 启动需要），优先本地安装
if ! npx tsx --version &>/dev/null; then
npm install tsx 2>&1 | tail -2
fi
echo "[memory-tencentdb] Gateway dependencies installed"
# ---------- Step 2.5: 将插件链接到 hermes 插件目录 ----------
echo "[memory-tencentdb] Step 2.5: Linking plugin into hermes plugins directory..."
HERMES_PLUGIN_DIR="$HERMES_AGENT_DIR/plugins/memory/memory_tencentdb"
PLUGIN_SRC_DIR="$TDAI_INSTALL_DIR/hermes-plugin/memory/memory_tencentdb"
# 移除旧链接/目录
rm -rf "$HERMES_PLUGIN_DIR"
# 创建 symlink 使 hermes 能发现插件
ln -sf "$PLUGIN_SRC_DIR" "$HERMES_PLUGIN_DIR"
echo "[memory-tencentdb] Plugin linked: $HERMES_PLUGIN_DIR -> $PLUGIN_SRC_DIR"
# ---------- Step 3: 提示用户手动开启 memory_tencentdb（不自动修改 config） ----------
echo "[memory-tencentdb] Step 3: Checking hermes config..."
# 插件已链接到 hermes 插件目录，但默认不自动启用，仅提示
if [ -f "$HERMES_CONFIG" ]; then
if sed -n '/^memory:/,/^[a-zA-Z]/p' "$HERMES_CONFIG" | grep -q "provider: memory_tencentdb"; then
echo "[memory-tencentdb] memory.provider already set to memory_tencentdb"
else
echo "[memory-tencentdb] Plugin installed but NOT enabled by default."
echo "[memory-tencentdb] To enable tdai memory, add/edit in $HERMES_CONFIG:"
echo ""
echo " memory:"
echo " provider: memory_tencentdb"
echo ""
fi
else
echo "[memory-tencentdb] WARN: $HERMES_CONFIG not found, please run install_hermes_ubuntu.sh first"
fi
# ---------- Step 4: 配置 Gateway 环境变量 ----------
echo "[memory-tencentdb] Step 4: Setting up Gateway environment..."
# 构建 Gateway 启动命令
# 使用 sh -c 包裹，先 cd 到插件目录再启动 Gateway（ESM 解析需要）
#
# 解析 node 绝对路径写入 GATEWAY_CMD（fix: issue #19）
# 当 Hermes 或独立 Gateway 以 systemd service 运行时，systemd 不会
# source 任何 user shell rc 文件，nvm/asdf 注入的 PATH 不存在。
# 用 `command -v node` 在 install 时解析绝对路径，并改用 Node 原生
# `--import tsx/esm`（Node >= 20.6 stable）替代 `npx tsx`，
# 让最终命令完全不依赖运行时 PATH。
NODE_BIN="$(command -v node || true)"
if [ -z "$NODE_BIN" ]; then
echo "[ERROR] 'node' not found in PATH; cannot generate Gateway start command." >&2
echo "[ERROR] If you installed Node via nvm/asdf, source the loader script first:" >&2
echo "[ERROR] source ~/.bashrc # or 'nvm use <version>'" >&2
exit 1
fi
echo "[memory-tencentdb] Resolved node: $NODE_BIN"
GATEWAY_CMD="sh -c 'cd $TDAI_INSTALL_DIR && exec \"$NODE_BIN\" --import tsx/esm src/gateway/server.ts'"
# ── 4a: /etc/profile.d（SSH 交互式登录场景） ──
# 写入 /etc/profile.d 持久化环境变量，供 SSH 手动执行 `hermes` 时使用。
# 注意：LLM 相关变量（API key、model 等）需要用户后续手动配置
ENVFILE="/etc/profile.d/memory-tencentdb-env.sh"
cat << ENVEOF | sudo tee "$ENVFILE" > /dev/null
# memory-tencentdb Gateway 环境变量
export MEMORY_TENCENTDB_GATEWAY_CMD="$GATEWAY_CMD"
export MEMORY_TENCENTDB_GATEWAY_HOST="127.0.0.1"
export MEMORY_TENCENTDB_GATEWAY_PORT="8420"
# LLM 配置（按需修改）
# export MEMORY_TENCENTDB_LLM_API_KEY="sk-..."
# export MEMORY_TENCENTDB_LLM_BASE_URL="https://api.openai.com/v1"
# export MEMORY_TENCENTDB_LLM_MODEL="gpt-4o"
ENVEOF
echo "[memory-tencentdb] Environment variables written to $ENVFILE"
# ── 4b: ~/.hermes/.env（systemd service 场景） ──
# hermes-gateway 通过 systemd user service 启动时不会 source /etc/profile.d/*.sh，
# 但 hermes 的 run.py 启动时会 load_dotenv("~/.hermes/.env")。
# 因此必须将关键变量同步写入 .env，否则 systemd 场景下 Gateway 无法自动启动。
HERMES_ENV="$HERMES_HOME/.env"
_append_or_update_env() {
local key="$1"
local value="$2"
local file="$3"
if [ ! -f "$file" ]; then
touch "$file"
fi
# 移除已有的同名变量行（含注释掉的和带引号的），再追加
sed -i "/^${key}=/d" "$file"
sed -i "/^# *${key}=/d" "$file"
# python-dotenv 要求含空格/引号/特殊字符的值用双引号包裹
echo "${key}=\"${value}\"" >> "$file"
}
_append_or_update_env "MEMORY_TENCENTDB_GATEWAY_CMD" "$GATEWAY_CMD" "$HERMES_ENV"
_append_or_update_env "MEMORY_TENCENTDB_GATEWAY_HOST" "127.0.0.1" "$HERMES_ENV"
_append_or_update_env "MEMORY_TENCENTDB_GATEWAY_PORT" "8420" "$HERMES_ENV"
echo "[memory-tencentdb] Gateway env vars also written to $HERMES_ENV (for systemd service)"
# ---------- 清理 ----------
rm -rf "$TEMP_DOWNLOAD"
# ---------- 验证安装 ----------
echo ""
echo "=========================================="
echo "[memory-tencentdb] Installation Summary"
echo "=========================================="
echo " Root dir: $MEMORY_TENCENTDB_ROOT"
echo " tdai source: $TDAI_INSTALL_DIR"
echo " tdai data dir: $TDAI_DATA_DIR"
echo " Hermes config: $HERMES_CONFIG"
echo " Env file: $ENVFILE"
echo ""
echo " Installed files in tdai dir:"
ls -la "$TDAI_INSTALL_DIR"/ 2>/dev/null | head -20 || echo " (none)"
echo ""
# 验证 hermes 插件文件存在（在解压目录中）
PLUGIN_SRC="$TDAI_INSTALL_DIR/hermes-plugin/memory/memory_tencentdb"
MISSING=0
for f in __init__.py plugin.yaml client.py supervisor.py; do
if [ ! -f "$PLUGIN_SRC/$f" ]; then
echo " [WARN] Missing: $PLUGIN_SRC/$f"
MISSING=1
fi
done
if [ "$MISSING" -eq 0 ]; then
echo " [OK] All hermes plugin files present"
fi
# 验证 Gateway 入口存在
if [ -f "$TDAI_INSTALL_DIR/src/gateway/server.ts" ]; then
echo " [OK] Gateway entry point found"
else
echo " [WARN] Gateway server.ts not found at $TDAI_INSTALL_DIR/src/gateway/server.ts"
fi
# 验证 node_modules 已安装
if [ -d "$TDAI_INSTALL_DIR/node_modules" ]; then
echo " [OK] Gateway node_modules installed"
else
echo " [WARN] Gateway node_modules not found"
fi
echo ""
echo "[memory-tencentdb] Done!"
echo ""
echo " NOTE: Before using the memory plugin, configure LLM credentials in ~/.hermes/.env:"
echo " MEMORY_TENCENTDB_LLM_API_KEY=your-api-key"
echo " MEMORY_TENCENTDB_LLM_BASE_URL=https://api.openai.com/v1"
echo " MEMORY_TENCENTDB_LLM_MODEL=gpt-4o"
echo ""
echo " (For systemd-managed hermes-gateway, ~/.hermes/.env is the authoritative config."
echo " /etc/profile.d/ is only used for interactive SSH sessions.)"
echo ""
scripts/memory-tencentdb-ctl.sh
Executable
+1030
View File
File diff suppressed because it is too large Load Diff
scripts/migrate-sqlite-to-tcvdb/README.md
+239
View File
@@ -0,0 +1,239 @@
# SQLite → 腾讯云向量数据库迁移工具
离线迁移工具，用于将 memory-tdai 的数据从本地 SQLite 存储迁移到腾讯云向量数据库（TCVDB）。
## 前置条件
- Node.js >= 22.16.0
- 插件已通过 `openclaw plugins install` 安装
- 迁移脚本已编译（见下方）
## 编译
迁移脚本使用 TypeScript 编写，运行前需要先编译：
```bash
npm run build:migrate-sqlite-to-vdb
```
编译产物输出到 `scripts/migrate-sqlite-to-tcvdb/dist/`，可直接用 Node 运行。
## 使用方法
```bash
# 预检模式（仅查看源数据，不执行写入）
npm run migrate:sqlite-to-tcvdb -- \
--plugin-data-dir ~/.openclaw/memory-tdai \
--openclaw-config-path ~/.openclaw/openclaw.json \
--tcvdb-url http://127.0.0.1:80 \
--tcvdb-username root \
--tcvdb-api-key-env TCVDB_API_KEY \
--tcvdb-database agent_memory_prod \
--tcvdb-embedding-model bge-large-zh \
--dry-run
# 正式迁移
npm run migrate:sqlite-to-tcvdb -- \
--plugin-data-dir ~/.openclaw/memory-tdai \
--openclaw-config-path ~/.openclaw/openclaw.json \
--tcvdb-url http://127.0.0.1:80 \
--tcvdb-username root \
--tcvdb-api-key-env TCVDB_API_KEY \
--tcvdb-database agent_memory_prod \
--tcvdb-embedding-model bge-large-zh \
--yes
```
### 更多示例
```bash
# 直接传入 API Key（不通过环境变量）
npm run migrate:sqlite-to-tcvdb -- \
--plugin-data-dir ~/.openclaw/memory-tdai \
--openclaw-config-path ~/.openclaw/openclaw.json \
--tcvdb-url http://127.0.0.1:80 \
--tcvdb-username root \
--tcvdb-api-key 'your-api-key-here' \
--tcvdb-database agent_memory_prod \
--tcvdb-embedding-model bge-large-zh \
--yes
```
```bash
# 指定自定义 SQLite 路径（数据库不在默认 vectors.db 位置时）
npm run migrate:sqlite-to-tcvdb -- \
--plugin-data-dir ~/.openclaw/memory-tdai \
--sqlite-path /backup/2026-04/vectors-snapshot.db \
--openclaw-config-path ~/.openclaw/openclaw.json \
--tcvdb-url http://127.0.0.1:80 \
--tcvdb-username root \
--tcvdb-api-key-env TCVDB_API_KEY \
--tcvdb-database agent_memory_prod \
--tcvdb-embedding-model bge-large-zh \
--yes
```
```bash
# 只迁移 L1 记忆层（跳过 L0 原始消息和 Profile）
npm run migrate:sqlite-to-tcvdb -- \
--plugin-data-dir ~/.openclaw/memory-tdai \
--openclaw-config-path ~/.openclaw/openclaw.json \
--tcvdb-url http://127.0.0.1:80 \
--tcvdb-username root \
--tcvdb-api-key-env TCVDB_API_KEY \
--tcvdb-database agent_memory_prod \
--tcvdb-embedding-model bge-large-zh \
--layers l1 \
--yes
```
```bash
# 只迁移 L0 和 L1（不迁移 Profile）
npm run migrate:sqlite-to-tcvdb -- \
--plugin-data-dir ~/.openclaw/memory-tdai \
--openclaw-config-path ~/.openclaw/openclaw.json \
--tcvdb-url http://127.0.0.1:80 \
--tcvdb-username root \
--tcvdb-api-key-env TCVDB_API_KEY \
--tcvdb-database agent_memory_prod \
--tcvdb-embedding-model bge-large-zh \
--layers l0,l1 \
--yes
```
```bash
# 英文语料场景：使用英文 BM25 分词
npm run migrate:sqlite-to-tcvdb -- \
--plugin-data-dir ~/.openclaw/memory-tdai \
--openclaw-config-path ~/.openclaw/openclaw.json \
--tcvdb-url http://127.0.0.1:80 \
--tcvdb-username root \
--tcvdb-api-key-env TCVDB_API_KEY \
--tcvdb-database agent_memory_prod \
--tcvdb-embedding-model bge-large-en-v1.5 \
--bm25-language en \
--yes
```
```bash
# 禁用 BM25 稀疏向量（仅使用密集向量检索）
npm run migrate:sqlite-to-tcvdb -- \
--plugin-data-dir ~/.openclaw/memory-tdai \
--openclaw-config-path ~/.openclaw/openclaw.json \
--tcvdb-url http://127.0.0.1:80 \
--tcvdb-username root \
--tcvdb-api-key-env TCVDB_API_KEY \
--tcvdb-database agent_memory_prod \
--tcvdb-embedding-model bge-large-zh \
--no-bm25-enabled \
--yes
```
```bash
# 仅迁移数据，不自动更新 openclaw.json 和 manifest（手动管理配置）
npm run migrate:sqlite-to-tcvdb -- \
--plugin-data-dir ~/.openclaw/memory-tdai \
--openclaw-config-path ~/.openclaw/openclaw.json \
--tcvdb-url http://127.0.0.1:80 \
--tcvdb-username root \
--tcvdb-api-key-env TCVDB_API_KEY \
--tcvdb-database agent_memory_prod \
--tcvdb-embedding-model bge-large-zh \
--no-apply-config \
--no-rewrite-manifest \
--yes
```
```bash
# 追加迁移：允许目标库已有数据，跳过非空检查
npm run migrate:sqlite-to-tcvdb -- \
--plugin-data-dir ~/.openclaw/memory-tdai \
--openclaw-config-path ~/.openclaw/openclaw.json \
--tcvdb-url http://127.0.0.1:80 \
--tcvdb-username root \
--tcvdb-api-key-env TCVDB_API_KEY \
--tcvdb-database agent_memory_prod \
--tcvdb-embedding-model bge-large-zh \
--no-fail-if-target-nonempty \
--no-verify-counts \
--yes
```
```bash
# 输出迁移摘要到 JSON 文件（适合 CI/自动化流水线）
npm run migrate:sqlite-to-tcvdb -- \
--plugin-data-dir ~/.openclaw/memory-tdai \
--openclaw-config-path ~/.openclaw/openclaw.json \
--tcvdb-url http://127.0.0.1:80 \
--tcvdb-username root \
--tcvdb-api-key-env TCVDB_API_KEY \
--tcvdb-database agent_memory_prod \
--tcvdb-embedding-model bge-large-zh \
--summary-json-path ./migration-report.json \
--job-id "migrate-2026-04-13" \
--yes
```
```bash
# 设置自定义超时和别名
npm run migrate:sqlite-to-tcvdb -- \
--plugin-data-dir ~/.openclaw/memory-tdai \
--openclaw-config-path ~/.openclaw/openclaw.json \
--tcvdb-url http://10.0.1.50:80 \
--tcvdb-username admin \
--tcvdb-api-key-env TCVDB_API_KEY \
--tcvdb-database agent_memory_prod \
--tcvdb-embedding-model bge-large-zh \
--tcvdb-alias "生产环境-主库" \
--tcvdb-timeout-ms 30000 \
--yes
```
## 参数说明
| 参数 | 必填 | 默认值 | 说明 |
|---|---|---|---|
| `--plugin-data-dir` | 是 | — | 插件数据目录路径 |
| `--openclaw-config-path` | 是 | — | `openclaw.json` 配置文件路径 |
| `--sqlite-path` | 否 | `<plugin-data-dir>/vectors.db` | SQLite 数据库文件路径（默认取数据目录下的 `vectors.db`） |
| `--plugin-id` | 否 | `memory-tencentdb` | 写入配置时使用的插件 ID |
| `--tcvdb-url` | 是 | — | TCVDB 服务地址 |
| `--tcvdb-username` | 是 | — | TCVDB 用户名 |
| `--tcvdb-api-key` | * | — | TCVDB API 密钥（明文） |
| `--tcvdb-api-key-env` | * | — | 包含 API 密钥的环境变量名 |
| `--tcvdb-database` | 是 | — | TCVDB 数据库名 |
| `--tcvdb-embedding-model` | 是 | — | Embedding 模型名称 |
| `--tcvdb-alias` | 否 | `""` | 用户自定义别名 |
| `--tcvdb-timeout-ms` | 否 | `10000` | 请求超时时间（毫秒） |
| `--layers` | 否 | `l0,l1,l2,l3` | 要迁移的层（逗号分隔） |
| `--dry-run` | 否 | `false` | 仅预览，不执行写入 |
| `--yes` | 否 | `false` | 跳过交互确认 |
| `--apply-config` | 否 | `true` | 迁移后更新 openclaw.json |
| `--config-backup` | 否 | `true` | 写入配置前先备份原配置文件 |
| `--rewrite-manifest` | 否 | `true` | 将 manifest.json 更新为 tcvdb |
| `--fail-if-target-nonempty` | 否 | `true` | 目标库非空时中止 |
| `--verify-counts` | 否 | `true` | 迁移后校验记录数 |
| `--summary-json-path` | 否 | — | 将迁移摘要写入此文件 |
| `--job-id` | 否 | — | 迁移任务 ID（用于追踪） |
| `--bm25-enabled` | 否 | `true` | 启用 BM25 稀疏向量 |
| `--bm25-language` | 否 | `zh` | BM25 语言（`zh` 或 `en`） |
\* `--tcvdb-api-key` 和 `--tcvdb-api-key-env` 二选一，必须提供其中一个。
## 目录结构
```
scripts/migrate-sqlite-to-tcvdb/
├── cli-entry.ts # CLI 入口
├── sqlite-to-tcvdb.ts # 迁移核心逻辑（参数解析、预检、数据迁移）
├── config-write.ts # OpenClaw 配置更新（JSON5，自包含）
├── manifest-write.ts # Manifest 重写
├── *.test.ts # 就近放置的测试文件
├── tsconfig.json # 迁移脚本编译配置
├── dist/ # 编译产物（已 gitignore）
└── README.md # 本文件
bin/migrate-sqlite-to-tcvdb.mjs # 极薄 bin 包装 → dist/
```
迁移脚本通过 `../../src/` 引用存储实现（VectorStore、TcvdbMemoryStore 等），但**不依赖 `openclaw/plugin-sdk`**。配置写回直接使用 `json5`。
scripts/migrate-sqlite-to-tcvdb/cli-entry.ts
+12
View File
@@ -0,0 +1,12 @@
import { runMigrationCli } from "./sqlite-to-tcvdb.js";
const TAG = "[memory-tdai][migrate-cli]";
try {
const summary = await runMigrationCli(process.argv.slice(2));
process.stdout.write(`${JSON.stringify(summary, null, 2)}\n`);
} catch (err) {
const message = err instanceof Error ? err.message : String(err);
process.stderr.write(`${TAG} ${message}\n`);
process.exitCode = 1;
}
scripts/migrate-sqlite-to-tcvdb/config-write.ts
+129
View File
@@ -0,0 +1,129 @@
import fs from "node:fs/promises";
import JSON5 from "json5";
export type Bm25Language = "zh" | "en";
export interface MigrationPluginConfigTarget {
url: string;
username: string;
apiKey: string;
database: string;
alias: string;
embeddingModel: string;
timeout: number;
bm25Enabled: boolean;
bm25Language: Bm25Language;
}
export interface WriteMigrationPluginConfigParams {
configPath: string;
pluginId: string;
tcvdb: {
url: string;
username: string;
apiKey: string;
database: string;
alias: string;
embeddingModel: string;
timeout: number;
};
bm25: {
enabled: boolean;
language: Bm25Language;
};
}
interface ConfigWriteAdapterDeps {
fs?: Pick<typeof fs, "readFile" | "writeFile" | "mkdir">;
parseConfig?: (raw: string) => unknown;
}
function asRecord(value: unknown): Record<string, unknown> {
return value !== null && typeof value === "object" && !Array.isArray(value)
? { ...(value as Record<string, unknown>) }
: {};
}
export function buildMigrationPluginConfigPatch(target: MigrationPluginConfigTarget): Record<string, unknown> {
return {
storeBackend: "tcvdb",
tcvdb: {
url: target.url,
username: target.username,
apiKey: target.apiKey,
database: target.database,
alias: target.alias,
embeddingModel: target.embeddingModel,
timeout: target.timeout,
},
bm25: {
enabled: target.bm25Enabled,
language: target.bm25Language,
},
};
}
function applyPluginConfigPatch(
sourceConfig: Record<string, unknown>,
pluginId: string,
patch: Record<string, unknown>,
): Record<string, unknown> {
const nextConfig = asRecord(sourceConfig);
const plugins = asRecord(nextConfig.plugins);
const entries = asRecord(plugins.entries);
const targetEntry = asRecord(entries[pluginId]);
const targetPluginConfig = asRecord(targetEntry.config);
const patchTcvdb = asRecord(patch.tcvdb);
const patchBm25 = asRecord(patch.bm25);
const mergedPluginConfig: Record<string, unknown> = {
...targetPluginConfig,
...patch,
tcvdb: {
...asRecord(targetPluginConfig.tcvdb),
...patchTcvdb,
},
bm25: {
...asRecord(targetPluginConfig.bm25),
...patchBm25,
},
};
entries[pluginId] = {
...targetEntry,
config: mergedPluginConfig,
};
plugins.entries = entries;
nextConfig.plugins = plugins;
return nextConfig;
}
export async function writeMigrationPluginConfig(
params: WriteMigrationPluginConfigParams,
deps: ConfigWriteAdapterDeps = {},
): Promise<void> {
const fsImpl = deps.fs ?? fs;
const parseConfig = deps.parseConfig ?? ((raw: string) => JSON5.parse(raw));
let parsed: unknown;
try {
parsed = parseConfig(await fsImpl.readFile(params.configPath, "utf-8"));
} catch {
throw new Error(
`Config migration writer only supports single-file JSON/JSON5: ${params.configPath}`,
);
}
const nextConfig = applyPluginConfigPatch(
asRecord(parsed),
params.pluginId,
buildMigrationPluginConfigPatch({
...params.tcvdb,
bm25Enabled: params.bm25.enabled,
bm25Language: params.bm25.language,
}),
);
await fsImpl.writeFile(params.configPath, `${JSON.stringify(nextConfig, null, 2)}\n`, "utf-8");
}
scripts/migrate-sqlite-to-tcvdb/manifest-write.ts
+69
View File
@@ -0,0 +1,69 @@
import fs from "node:fs/promises";
import path from "node:path";
import {
buildStoreInfo,
manifestPath,
readManifest,
writeManifest,
} from "../../src/utils/manifest.js";
export interface RewriteMigrationManifestParams {
dataDir: string;
tcvdbUrl: string;
tcvdbDatabase: string;
tcvdbAlias?: string;
backupExisting?: boolean;
now?: () => string;
}
export interface RewriteMigrationManifestResult {
created: boolean;
updated: boolean;
backupPath?: string;
}
const DEFAULT_BACKUP_FILE = "manifest.json.migrate.bak";
export async function rewriteMigrationManifest(
params: RewriteMigrationManifestParams,
): Promise<RewriteMigrationManifestResult> {
const existing = readManifest(params.dataDir);
const nextStore = buildStoreInfo({
type: "tcvdb",
tcvdbUrl: params.tcvdbUrl,
tcvdbDatabase: params.tcvdbDatabase,
tcvdbAlias: params.tcvdbAlias,
});
if (!existing) {
writeManifest(params.dataDir, {
version: 1,
createdAt: params.now?.() ?? new Date().toISOString(),
store: nextStore,
seed: null,
});
return {
created: true,
updated: false,
backupPath: undefined,
};
}
let backupPath: string | undefined;
if (params.backupExisting !== false) {
backupPath = path.join(path.dirname(manifestPath(params.dataDir)), DEFAULT_BACKUP_FILE);
await fs.mkdir(path.dirname(backupPath), { recursive: true });
await fs.copyFile(manifestPath(params.dataDir), backupPath);
}
writeManifest(params.dataDir, {
...existing,
store: nextStore,
});
return {
created: false,
updated: true,
backupPath,
};
}
scripts/migrate-sqlite-to-tcvdb/node-llama-cpp.d.ts
Vendored
+11
View File
@@ -0,0 +1,11 @@
/**
* Type stub for node-llama-cpp — the migration script does not use local
* embedding but TypeScript still resolves the module through transitive
* imports (sqlite.ts → embedding.ts → import("node-llama-cpp")).
*
* This stub satisfies the compiler without requiring the actual package.
*/
declare module "node-llama-cpp" {
const _: any;
export = _;
}
scripts/migrate-sqlite-to-tcvdb/sqlite-to-tcvdb.ts
+849
View File
@@ -0,0 +1,849 @@
import fs from "node:fs/promises";
import path from "node:path";
import { parseArgs } from "node:util";
import type { MemoryRecord } from "../../src/core/record/l1-writer.js";
import { listLocalProfiles } from "../../src/core/profile/profile-sync.js";
import { createBM25Encoder } from "../../src/core/store/bm25-local.js";
import { VectorStore, type L0RecordRow } from "../../src/core/store/sqlite.js";
import { TcvdbMemoryStore } from "../../src/core/store/tcvdb.js";
import type { L0Record, L1RecordRow, ProfileRecord, ProfileSyncRecord, StoreInitResult } from "../../src/core/store/types.js";
import { readManifest } from "../../src/utils/manifest.js";
import {
rewriteMigrationManifest as rewriteMigrationManifestDefault,
type RewriteMigrationManifestResult,
} from "./manifest-write.js";
import {
writeMigrationPluginConfig as writeMigrationPluginConfigDefault,
} from "./config-write.js";
export const DEFAULT_MIGRATION_PLUGIN_ID = "memory-tencentdb";
export const ALL_MIGRATION_LAYERS = ["l0", "l1", "l2", "l3"] as const;
const TAG = "[memory-tdai][migrate]";
function log(message: string): void {
process.stderr.write(`${TAG} ${message}\n`);
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export type MigrationLayer = (typeof ALL_MIGRATION_LAYERS)[number];
export type Bm25Language = "zh" | "en";
export interface ResolvedMigrationCliOptions {
pluginDataDir: string;
sqlitePath: string;
openclawConfigPath: string;
pluginId: string;
layers: MigrationLayer[];
applyConfig: boolean;
configBackup: boolean;
rewriteManifest: boolean;
failIfTargetNonempty: boolean;
verifyCounts: boolean;
dryRun: boolean;
yes: boolean;
bm25Enabled: boolean;
bm25Language: Bm25Language;
summaryJsonPath?: string;
jobId?: string;
tcvdb: {
url: string;
username: string;
apiKey: string;
database: string;
alias: string;
embeddingModel: string;
timeout: number;
caPemPath?: string;
};
}
export interface MigrationPreflightSummary {
pluginId: string;
dryRun: boolean;
layers: MigrationLayer[];
paths: {
pluginDataDir: string;
sqlitePath: string;
openclawConfigPath: string;
};
source: {
l0Count: number;
l1Count: number;
profileCount: number;
manifestExists: boolean;
manifestStoreType: "sqlite" | "tcvdb" | null;
};
target: {
url: string;
username: string;
database: string;
alias: string;
embeddingModel: string;
timeout: number;
bm25Enabled: boolean;
bm25Language: Bm25Language;
};
options: {
applyConfig: boolean;
configBackup: boolean;
rewriteManifest: boolean;
failIfTargetNonempty: boolean;
verifyCounts: boolean;
yes: boolean;
};
migration?: {
l0Migrated: number;
l1Migrated: number;
profileMigrated: number;
targetL0Count: number;
targetL1Count: number;
targetProfileCount: number;
configWritten: boolean;
manifestWritten: boolean;
manifestBackupPath?: string;
};
}
export const DEFAULT_MIGRATION_PAGE_SIZE = 50;
export interface MigrationTargetStore {
init(providerInfo?: unknown): Promise<StoreInitResult> | StoreInitResult;
isDegraded(): boolean;
close(): void;
upsertL1(record: MemoryRecord, embedding?: Float32Array): Promise<boolean> | boolean;
upsertL0(record: L0Record, embedding?: Float32Array): Promise<boolean> | boolean;
upsertL1Batch?(records: MemoryRecord[]): Promise<number>;
upsertL0Batch?(records: L0Record[]): Promise<number>;
countL1(): Promise<number> | number;
countL0(): Promise<number> | number;
pullProfiles?(): Promise<ProfileRecord[]>;
syncProfiles?(records: ProfileSyncRecord[]): Promise<void>;
}
export interface RunMigrationCliDeps {
createTargetStore?: (options: ResolvedMigrationCliOptions) => MigrationTargetStore;
writeMigrationPluginConfig?: typeof writeMigrationPluginConfigDefault;
rewriteMigrationManifest?: (params: {
dataDir: string;
tcvdbUrl: string;
tcvdbDatabase: string;
tcvdbAlias?: string;
}) => Promise<RewriteMigrationManifestResult>;
verifyDelayMs?: number;
}
function getRequiredString(values: Record<string, string | boolean | undefined>, key: string): string {
const value = values[key];
if (typeof value !== "string" || value.trim() === "") {
throw new Error(`Missing required option --${key}`);
}
return value.trim();
}
function getOptionalString(values: Record<string, string | boolean | undefined>, key: string): string | undefined {
const value = values[key];
if (typeof value !== "string") return undefined;
const trimmed = value.trim();
return trimmed === "" ? undefined : trimmed;
}
function resolveBooleanOption(
values: Record<string, string | boolean | undefined>,
key: string,
defaultValue: boolean,
): boolean {
const value = values[key];
return typeof value === "boolean" ? value : defaultValue;
}
function resolveTcvdbApiKey(values: Record<string, string | boolean | undefined>): string {
const directApiKey = getOptionalString(values, "tcvdb-api-key");
const apiKeyEnvName = getOptionalString(values, "tcvdb-api-key-env");
if (directApiKey && apiKeyEnvName) {
throw new Error("Provide either --tcvdb-api-key or --tcvdb-api-key-env, not both");
}
if (directApiKey) {
return directApiKey;
}
if (!apiKeyEnvName) {
throw new Error("Missing required TCVDB API key input: use --tcvdb-api-key or --tcvdb-api-key-env");
}
const envValue = process.env[apiKeyEnvName]?.trim();
if (!envValue) {
throw new Error(`Environment variable ${apiKeyEnvName} is empty or not set`);
}
return envValue;
}
function parseLayers(rawLayers: string | undefined): MigrationLayer[] {
if (!rawLayers) return [...ALL_MIGRATION_LAYERS];
const values = rawLayers
.split(",")
.map((layer) => layer.trim())
.filter(Boolean);
if (values.length === 0) {
throw new Error("--layers must include at least one layer");
}
const uniqueLayers = [...new Set(values)];
const invalid = uniqueLayers.filter(
(layer): layer is string => !ALL_MIGRATION_LAYERS.includes(layer as MigrationLayer),
);
if (invalid.length > 0) {
throw new Error(`Unsupported layer(s): ${invalid.join(", ")}`);
}
return uniqueLayers as MigrationLayer[];
}
function parseBm25Language(rawLanguage: string | undefined): Bm25Language {
if (!rawLanguage) return "zh";
if (rawLanguage === "zh" || rawLanguage === "en") {
return rawLanguage;
}
throw new Error(`Unsupported --bm25-language value: ${rawLanguage}`);
}
function parseTimeout(rawValue: string | undefined): number {
if (!rawValue) return 10000;
const timeout = Number(rawValue);
if (!Number.isFinite(timeout) || timeout <= 0) {
throw new Error(`Invalid --tcvdb-timeout-ms value: ${rawValue}`);
}
return timeout;
}
/**
* Build a "nothing to migrate" summary — used when source data dir or sqlite
* file doesn't exist (e.g. fresh deployment that hasn't captured any data yet).
*/
function buildEmptySummary(options: ResolvedMigrationCliOptions): MigrationPreflightSummary {
return {
pluginId: options.pluginId,
dryRun: options.dryRun,
layers: options.layers,
paths: {
pluginDataDir: options.pluginDataDir,
sqlitePath: options.sqlitePath,
openclawConfigPath: options.openclawConfigPath,
},
source: {
l0Count: 0,
l1Count: 0,
profileCount: 0,
manifestExists: false,
manifestStoreType: null,
},
target: {
url: options.tcvdb.url,
username: options.tcvdb.username,
database: options.tcvdb.database,
alias: options.tcvdb.alias,
embeddingModel: options.tcvdb.embeddingModel,
timeout: options.tcvdb.timeout,
bm25Enabled: options.bm25Enabled,
bm25Language: options.bm25Language,
},
options: {
applyConfig: options.applyConfig,
configBackup: options.configBackup,
rewriteManifest: options.rewriteManifest,
failIfTargetNonempty: options.failIfTargetNonempty,
verifyCounts: options.verifyCounts,
yes: options.yes,
},
};
}
async function ensureReadablePath(filePath: string, label: string): Promise<void> {
try {
await fs.access(filePath);
} catch {
throw new Error(`${label} does not exist or is not accessible: ${filePath}`);
}
}
async function ensureReadableDirectory(dirPath: string, label: string): Promise<void> {
const stat = await fs.stat(dirPath).catch(() => null);
if (!stat?.isDirectory()) {
throw new Error(`${label} is not a directory: ${dirPath}`);
}
}
function safeParseMetadata(raw: string): Record<string, unknown> {
if (!raw) return {};
try {
const parsed = JSON.parse(raw);
return parsed && typeof parsed === "object" && !Array.isArray(parsed)
? (parsed as Record<string, unknown>)
: {};
} catch {
return {};
}
}
function compactTimestamps(row: L1RecordRow): string[] {
return [...new Set([row.timestamp_str, row.timestamp_start, row.timestamp_end].filter(Boolean))];
}
function mapL1RowToMemoryRecord(row: L1RecordRow): MemoryRecord {
const timestamps = compactTimestamps(row);
const fallbackIso = row.updated_time || row.created_time || row.timestamp_end || row.timestamp_str || new Date(0).toISOString();
return {
id: row.record_id,
content: row.content,
type: row.type as MemoryRecord["type"],
priority: row.priority,
scene_name: row.scene_name,
source_message_ids: [],
metadata: safeParseMetadata(row.metadata_json),
timestamps,
createdAt: row.created_time || fallbackIso,
updatedAt: row.updated_time || row.created_time || fallbackIso,
sessionKey: row.session_key || "",
sessionId: row.session_id || "",
};
}
function mapL0RowToRecord(row: L0RecordRow): L0Record {
return {
id: row.record_id,
sessionKey: row.session_key,
sessionId: row.session_id || "",
role: row.role,
messageText: row.message_text,
recordedAt: row.recorded_at || "",
timestamp: row.timestamp ?? 0,
};
}
function createTargetStoreDefault(options: ResolvedMigrationCliOptions): MigrationTargetStore {
const bm25Encoder = createBM25Encoder(
{
enabled: options.bm25Enabled,
language: options.bm25Language,
},
);
return new TcvdbMemoryStore({
url: options.tcvdb.url,
username: options.tcvdb.username,
apiKey: options.tcvdb.apiKey,
database: options.tcvdb.database,
embeddingModel: options.tcvdb.embeddingModel,
timeout: options.tcvdb.timeout,
caPemPath: options.tcvdb.caPemPath,
bm25Encoder,
});
}
async function ensureTargetIsEmpty(
options: ResolvedMigrationCliOptions,
targetStore: MigrationTargetStore,
): Promise<void> {
if (!options.failIfTargetNonempty) return;
const [existingL1, existingL0, existingProfiles] = await Promise.all([
Promise.resolve(targetStore.countL1()),
Promise.resolve(targetStore.countL0()),
targetStore.pullProfiles ? targetStore.pullProfiles().then((records) => records.length) : Promise.resolve(0),
]);
if (existingL1 > 0 || existingL0 > 0 || existingProfiles > 0) {
throw new Error(
`Target store is not empty (L1=${existingL1}, L0=${existingL0}, profiles=${existingProfiles})`,
);
}
}
async function migrateL1Records(sourceStore: VectorStore, targetStore: MigrationTargetStore, pageSize: number): Promise<number> {
let migrated = 0;
let cursor = "";
const useBatch = typeof targetStore.upsertL1Batch === "function";
// eslint-disable-next-line no-constant-condition
while (true) {
const rows = sourceStore.queryL1RecordsCursor(cursor, pageSize);
if (rows.length === 0) break;
const records = rows.map(mapL1RowToMemoryRecord);
if (useBatch) {
const count = await targetStore.upsertL1Batch!(records);
if (count === 0) {
throw new Error(`Failed to batch migrate L1 records (cursor=${cursor}, page=${rows.length})`);
}
} else {
for (const record of records) {
const ok = await Promise.resolve(targetStore.upsertL1(record));
if (!ok) throw new Error(`Failed to migrate L1 record ${record.id}`);
}
}
migrated += rows.length;
cursor = rows[rows.length - 1].record_id;
log(`L1: 已迁移 ${migrated} 条...`);
if (rows.length < pageSize) break;
}
log(`L1: 迁移完成，共 ${migrated} 条`);
return migrated;
}
async function migrateL0Records(sourceStore: VectorStore, targetStore: MigrationTargetStore, pageSize: number): Promise<number> {
let migrated = 0;
let cursor = "";
const useBatch = typeof targetStore.upsertL0Batch === "function";
// eslint-disable-next-line no-constant-condition
while (true) {
const rows = sourceStore.queryL0RecordsCursor(cursor, pageSize);
if (rows.length === 0) break;
const records = rows.map(mapL0RowToRecord);
if (useBatch) {
const count = await targetStore.upsertL0Batch!(records);
if (count === 0) {
throw new Error(`Failed to batch migrate L0 records (cursor=${cursor}, page=${rows.length})`);
}
} else {
for (const record of records) {
const ok = await Promise.resolve(targetStore.upsertL0(record));
if (!ok) throw new Error(`Failed to migrate L0 record ${record.id}`);
}
}
migrated += rows.length;
cursor = rows[rows.length - 1].record_id;
log(`L0: 已迁移 ${migrated} 条...`);
if (rows.length < pageSize) break;
}
log(`L0: 迁移完成，共 ${migrated} 条`);
return migrated;
}
async function migrateProfiles(
pluginDataDir: string,
targetStore: MigrationTargetStore,
): Promise<number> {
const profiles = await listLocalProfiles(pluginDataDir);
if (profiles.length === 0) {
log("Profiles: 无本地 profile，跳过");
return 0;
}
if (!targetStore.syncProfiles) {
throw new Error("Target store does not support profile sync");
}
log(`Profiles: 发现 ${profiles.length} 个本地 profile，开始同步...`);
await targetStore.syncProfiles(profiles);
log(`Profiles: 同步完成，共 ${profiles.length} 个`);
return profiles.length;
}
async function verifyMigratedCounts(
summary: MigrationPreflightSummary,
targetStore: MigrationTargetStore,
delayMs: number,
): Promise<{ l1Count: number; l0Count: number; profileCount: number }> {
if (delayMs > 0) {
log(`等待 ${Math.round(delayMs / 1000)} 秒让远端数据落盘...`);
await sleep(delayMs);
}
log("开始校验迁移数量...");
const [l1Count, l0Count, profileCount] = await Promise.all([
Promise.resolve(targetStore.countL1()),
Promise.resolve(targetStore.countL0()),
targetStore.pullProfiles ? targetStore.pullProfiles().then((records) => records.length) : Promise.resolve(0),
]);
log(`校验结果: L1=${l1Count}/${summary.source.l1Count}, L0=${l0Count}/${summary.source.l0Count}, Profiles=${profileCount}/${summary.source.profileCount}`);
if (l1Count !== summary.source.l1Count) {
throw new Error(`L1 count verification failed: source=${summary.source.l1Count}, target=${l1Count}`);
}
if (l0Count !== summary.source.l0Count) {
throw new Error(`L0 count verification failed: source=${summary.source.l0Count}, target=${l0Count}`);
}
if (profileCount !== summary.source.profileCount) {
throw new Error(
`Profile count verification failed: source=${summary.source.profileCount}, target=${profileCount}`,
);
}
log("校验通过");
return { l1Count, l0Count, profileCount };
}
async function writeSummaryJson(
summaryJsonPath: string | undefined,
summary: MigrationPreflightSummary,
): Promise<void> {
if (!summaryJsonPath) return;
await fs.mkdir(path.dirname(summaryJsonPath), { recursive: true });
await fs.writeFile(summaryJsonPath, `${JSON.stringify(summary, null, 2)}\n`, "utf-8");
}
function printUsageAndExit(): never {
const text = `
SQLite → 腾讯云向量数据库迁移工具
Usage:
migrate-sqlite-to-tcvdb [options]
Required:
--plugin-data-dir <path> 插件数据目录路径
--openclaw-config-path <path> openclaw.json 配置文件路径
--tcvdb-url <url> TCVDB 服务地址
--tcvdb-username <name> TCVDB 用户名
--tcvdb-database <name> TCVDB 数据库名
--tcvdb-embedding-model <name> Embedding 模型名称
--tcvdb-api-key <key> TCVDB API 密钥（明文，与 --tcvdb-api-key-env 二选一）
--tcvdb-api-key-env <var> 包含 API 密钥的环境变量名
Optional:
--sqlite-path <path> SQLite 数据库路径（默认: <plugin-data-dir>/vectors.db）
--plugin-id <id> 写入配置时使用的插件 ID（默认: memory-tencentdb）
--layers <l0,l1,l2,l3> 要迁移的层，逗号分隔（默认: l0,l1,l2,l3）
--tcvdb-alias <name> 用户自定义别名
--tcvdb-timeout-ms <ms> 请求超时（默认: 10000）
--tcvdb-ca-pem <path> CA 证书 PEM 文件路径（HTTPS 连接时使用）
--bm25-language <zh|en> BM25 分词语言（默认: zh）
--summary-json-path <path> 将迁移摘要写入此文件
--job-id <id> 迁移任务 ID（用于追踪）
Flags:
--dry-run 仅预览，不执行写入
--yes 跳过交互确认
--no-apply-config 不自动更新 openclaw.json
--no-config-backup 写入配置前不备份
--no-rewrite-manifest 不更新 manifest.json
--no-fail-if-target-nonempty 目标库非空时不中止
--no-verify-counts 迁移后不校验记录数
--no-bm25-enabled 禁用 BM25 稀疏向量
-h, --help 显示此帮助信息
Examples:
# 预检模式
migrate-sqlite-to-tcvdb \\
--plugin-data-dir ~/.openclaw/memory-tdai \\
--openclaw-config-path ~/.openclaw/openclaw.json \\
--tcvdb-url http://127.0.0.1:80 --tcvdb-username root \\
--tcvdb-api-key-env TCVDB_API_KEY \\
--tcvdb-database agent_memory_prod \\
--tcvdb-embedding-model bge-large-zh \\
--dry-run
# 正式迁移
migrate-sqlite-to-tcvdb \\
--plugin-data-dir ~/.openclaw/memory-tdai \\
--openclaw-config-path ~/.openclaw/openclaw.json \\
--tcvdb-url http://127.0.0.1:80 --tcvdb-username root \\
--tcvdb-api-key-env TCVDB_API_KEY \\
--tcvdb-database agent_memory_prod \\
--tcvdb-embedding-model bge-large-zh \\
--yes
`.trimStart();
process.stdout.write(text);
process.exit(0);
}
export function resolveMigrationCliOptions(argv: string[]): ResolvedMigrationCliOptions {
const { values } = parseArgs({
args: argv,
strict: true,
allowPositionals: false,
allowNegative: true,
options: {
help: { type: "boolean", short: "h" },
"plugin-data-dir": { type: "string" },
"sqlite-path": { type: "string" },
"openclaw-config-path": { type: "string" },
"plugin-id": { type: "string" },
layers: { type: "string" },
"apply-config": { type: "boolean" },
"config-backup": { type: "boolean" },
"rewrite-manifest": { type: "boolean" },
"fail-if-target-nonempty": { type: "boolean" },
"verify-counts": { type: "boolean" },
"dry-run": { type: "boolean" },
yes: { type: "boolean" },
"bm25-enabled": { type: "boolean" },
"bm25-language": { type: "string" },
"summary-json-path": { type: "string" },
"job-id": { type: "string" },
"tcvdb-url": { type: "string" },
"tcvdb-username": { type: "string" },
"tcvdb-api-key": { type: "string" },
"tcvdb-api-key-env": { type: "string" },
"tcvdb-database": { type: "string" },
"tcvdb-alias": { type: "string" },
"tcvdb-embedding-model": { type: "string" },
"tcvdb-timeout-ms": { type: "string" },
"tcvdb-ca-pem": { type: "string" },
},
});
if (values.help) {
printUsageAndExit();
}
const pluginDataDir = path.resolve(getRequiredString(values, "plugin-data-dir"));
const sqlitePath = path.resolve(
getOptionalString(values, "sqlite-path") ?? path.join(pluginDataDir, "vectors.db"),
);
const summaryJsonPath = getOptionalString(values, "summary-json-path");
return {
pluginDataDir,
sqlitePath,
openclawConfigPath: path.resolve(getRequiredString(values, "openclaw-config-path")),
pluginId: getOptionalString(values, "plugin-id") ?? DEFAULT_MIGRATION_PLUGIN_ID,
layers: parseLayers(getOptionalString(values, "layers")),
applyConfig: resolveBooleanOption(values, "apply-config", true),
configBackup: resolveBooleanOption(values, "config-backup", true),
rewriteManifest: resolveBooleanOption(values, "rewrite-manifest", true),
failIfTargetNonempty: resolveBooleanOption(values, "fail-if-target-nonempty", true),
verifyCounts: resolveBooleanOption(values, "verify-counts", true),
dryRun: resolveBooleanOption(values, "dry-run", false),
yes: resolveBooleanOption(values, "yes", false),
bm25Enabled: resolveBooleanOption(values, "bm25-enabled", true),
bm25Language: parseBm25Language(getOptionalString(values, "bm25-language")),
summaryJsonPath: summaryJsonPath ? path.resolve(summaryJsonPath) : undefined,
jobId: getOptionalString(values, "job-id"),
tcvdb: {
url: getRequiredString(values, "tcvdb-url"),
username: getRequiredString(values, "tcvdb-username"),
apiKey: resolveTcvdbApiKey(values),
database: getRequiredString(values, "tcvdb-database"),
alias: getOptionalString(values, "tcvdb-alias") ?? "",
embeddingModel: getRequiredString(values, "tcvdb-embedding-model"),
timeout: parseTimeout(getOptionalString(values, "tcvdb-timeout-ms")),
caPemPath: getOptionalString(values, "tcvdb-ca-pem"),
},
};
}
export async function collectMigrationPreflight(
options: ResolvedMigrationCliOptions,
): Promise<MigrationPreflightSummary> {
// ── 优雅处理"数据目录 / sqlite 不存在"的场景 ──
// 如果源数据路径不存在（全新部署、尚未有任何 capture），不报错——
// 返回"全零"summary，让调用方判断是否跳过迁移。
const dirExists = await fs.stat(options.pluginDataDir).then(s => s.isDirectory()).catch(() => false);
if (!dirExists) {
log(`plugin data directory 不存在，无需迁移: ${options.pluginDataDir}`);
return buildEmptySummary(options);
}
const sqliteExists = await fs.access(options.sqlitePath).then(() => true).catch(() => false);
if (!sqliteExists) {
log(`sqlite database 不存在，无需迁移: ${options.sqlitePath}`);
return buildEmptySummary(options);
}
await ensureReadablePath(options.openclawConfigPath, "OpenClaw config file");
const store = new VectorStore(options.sqlitePath, 0);
const initResult = store.init();
if (store.isDegraded()) {
store.close();
throw new Error(`Failed to open sqlite store for migration preflight: ${initResult.reason ?? "unknown error"}`);
}
try {
const [profiles, manifest] = await Promise.all([
listLocalProfiles(options.pluginDataDir),
Promise.resolve(readManifest(options.pluginDataDir)),
]);
return {
pluginId: options.pluginId,
dryRun: options.dryRun,
layers: options.layers,
paths: {
pluginDataDir: options.pluginDataDir,
sqlitePath: options.sqlitePath,
openclawConfigPath: options.openclawConfigPath,
},
source: {
l0Count: store.countL0(),
l1Count: store.countL1(),
profileCount: profiles.length,
manifestExists: manifest !== null,
manifestStoreType: manifest?.store.type ?? null,
},
target: {
url: options.tcvdb.url,
username: options.tcvdb.username,
database: options.tcvdb.database,
alias: options.tcvdb.alias,
embeddingModel: options.tcvdb.embeddingModel,
timeout: options.tcvdb.timeout,
bm25Enabled: options.bm25Enabled,
bm25Language: options.bm25Language,
},
options: {
applyConfig: options.applyConfig,
configBackup: options.configBackup,
rewriteManifest: options.rewriteManifest,
failIfTargetNonempty: options.failIfTargetNonempty,
verifyCounts: options.verifyCounts,
yes: options.yes,
},
};
} finally {
store.close();
}
}
export async function runMigrationCli(
argv: string[],
deps: RunMigrationCliDeps = {},
): Promise<MigrationPreflightSummary> {
const options = resolveMigrationCliOptions(argv);
log("开始预检...");
const summary = await collectMigrationPreflight(options);
log(`预检完成: 源数据 L1=${summary.source.l1Count}, L0=${summary.source.l0Count}, Profiles=${summary.source.profileCount}`);
log(`目标: ${summary.target.url} / ${summary.target.database}`);
const hasSourceData = summary.source.l0Count > 0 || summary.source.l1Count > 0 || summary.source.profileCount > 0;
if (!hasSourceData) {
log("源数据为空，跳过数据迁移。");
}
if (options.dryRun) {
log("预检模式 (dry-run)，不执行写入");
await writeSummaryJson(options.summaryJsonPath, summary);
return summary;
}
const createTargetStore = deps.createTargetStore ?? createTargetStoreDefault;
const writeMigrationPluginConfig = deps.writeMigrationPluginConfig ?? writeMigrationPluginConfigDefault;
const rewriteMigrationManifest = deps.rewriteMigrationManifest ?? (async (params) =>
await rewriteMigrationManifestDefault(params));
const migration = {
l1Migrated: 0,
l0Migrated: 0,
profileMigrated: 0,
targetL1Count: 0,
targetL0Count: 0,
targetProfileCount: 0,
configWritten: false,
manifestWritten: false,
manifestBackupPath: undefined as string | undefined,
};
if (hasSourceData) {
log("打开 SQLite 源库...");
const sourceStore = new VectorStore(options.sqlitePath, 0);
const sourceInitResult = sourceStore.init();
if (sourceStore.isDegraded()) {
sourceStore.close();
throw new Error(`Failed to reopen sqlite source store: ${sourceInitResult.reason ?? "unknown error"}`);
}
log("初始化目标库...");
const targetStore = createTargetStore(options);
try {
await Promise.resolve(targetStore.init());
if (targetStore.isDegraded()) {
throw new Error("Target store entered degraded mode during initialization");
}
await ensureTargetIsEmpty(options, targetStore);
const pageSize = DEFAULT_MIGRATION_PAGE_SIZE;
log(`分页大小: ${pageSize} 条/批`);
migration.l1Migrated = options.layers.includes("l1") ? await migrateL1Records(sourceStore, targetStore, pageSize) : 0;
migration.l0Migrated = options.layers.includes("l0") ? await migrateL0Records(sourceStore, targetStore, pageSize) : 0;
migration.profileMigrated = options.layers.includes("l2") || options.layers.includes("l3")
? await migrateProfiles(options.pluginDataDir, targetStore)
: 0;
const verifyDelayMs = deps.verifyDelayMs ?? 10_000;
const verifiedCounts = options.verifyCounts
? await verifyMigratedCounts(summary, targetStore, verifyDelayMs)
: {
l1Count: await Promise.resolve(targetStore.countL1()),
l0Count: await Promise.resolve(targetStore.countL0()),
profileCount: targetStore.pullProfiles ? (await targetStore.pullProfiles()).length : 0,
};
migration.targetL1Count = verifiedCounts.l1Count;
migration.targetL0Count = verifiedCounts.l0Count;
migration.targetProfileCount = verifiedCounts.profileCount;
} finally {
sourceStore.close();
targetStore.close();
}
}
if (options.applyConfig) {
log(`写入配置到 ${options.openclawConfigPath} ...`);
await writeMigrationPluginConfig({
configPath: options.openclawConfigPath,
pluginId: options.pluginId,
tcvdb: {
url: options.tcvdb.url,
username: options.tcvdb.username,
apiKey: options.tcvdb.apiKey,
database: options.tcvdb.database,
alias: options.tcvdb.alias,
embeddingModel: options.tcvdb.embeddingModel,
timeout: options.tcvdb.timeout,
},
bm25: {
enabled: options.bm25Enabled,
language: options.bm25Language,
},
});
migration.configWritten = true;
log("配置写入完成");
}
if (options.rewriteManifest) {
log("更新 manifest...");
const manifestResult = await rewriteMigrationManifest({
dataDir: options.pluginDataDir,
tcvdbUrl: options.tcvdb.url,
tcvdbDatabase: options.tcvdb.database,
tcvdbAlias: options.tcvdb.alias || undefined,
});
migration.manifestWritten = manifestResult.created || manifestResult.updated;
migration.manifestBackupPath = manifestResult.backupPath;
log(`Manifest ${manifestResult.created ? "创建" : "更新"}完成${manifestResult.backupPath ? `，备份: ${manifestResult.backupPath}` : ""}`);
}
summary.migration = {
l1Migrated: migration.l1Migrated,
l0Migrated: migration.l0Migrated,
profileMigrated: migration.profileMigrated,
targetL1Count: migration.targetL1Count,
targetL0Count: migration.targetL0Count,
targetProfileCount: migration.targetProfileCount,
configWritten: migration.configWritten,
manifestWritten: migration.manifestWritten,
manifestBackupPath: migration.manifestBackupPath,
};
await writeSummaryJson(options.summaryJsonPath, summary);
log("迁移全部完成!");
return summary;
}
scripts/migrate-sqlite-to-tcvdb/tsconfig.json
+25
View File
@@ -0,0 +1,25 @@
{
"compilerOptions": {
"target": "es2023",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"declaration": false,
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"types": ["node"],
"paths": {
"node-llama-cpp": ["./node-llama-cpp.d.ts"]
},
"outDir": "dist",
"rootDir": "../.."
},
"files": [
"cli-entry.ts",
"sqlite-to-tcvdb.ts",
"config-write.ts",
"manifest-write.ts"
]
}
scripts/openclaw-after-tool-call-messages.patch.sh
+317
View File
@@ -0,0 +1,317 @@
#!/usr/bin/env bash
# ═══════════════════════════════════════════════════════════════════
# OpenClaw Patch: after_tool_call hook 注入 session messages
# ═══════════════════════════════════════════════════════════════════
# 用途：在 after_tool_call hookEvent 中注入 ctx.params.session?.messages，
# 使 context-offload 等插件能在工具调用后访问完整历史消息列表。
#
# 兼容策略（按优先级尝试，首个成功即停止）：
# 策略 1: AST-like — 搜索 hookEvent 对象中 durationMs 字段，在其后追加 messages
# 策略 2: 旧版 dispatch-*.js — 针对早期版本文件布局
# 策略 3: runAfterToolCall 锚点 — 在 hookEvent 关闭大括号前插入
# 策略 4: 通用 fallback — 基于 after_tool_call + durationMs 的宽松匹配
#
# 用法：
# bash openclaw-after-tool-call-messages.patch.sh
# bash openclaw-after-tool-call-messages.patch.sh /custom/path/to/openclaw
#
# 幂等：已 patch 过的文件会自动跳过，可安全重复执行。
# ═══════════════════════════════════════════════════════════════════
set -euo pipefail
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m'
info() { echo -e "${CYAN}[INFO]${NC} $*"; }
ok() { echo -e "${GREEN}[OK]${NC} $*"; }
warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
fail() { echo -e "${RED}[FAIL]${NC} $*"; exit 1; }
debug() { [[ "${DEBUG:-}" == "1" ]] && echo -e "${CYAN}[DEBUG]${NC} $*" || true; }
# ─── 定位 OpenClaw 安装目录 ──────────────────────────────────────
# Uses Node.js require.resolve to locate the package root — handles
# nvm, pnpm, npm, yarn, volta, and any other layout automatically.
_node_resolve_openclaw() {
node -e "
const {dirname, join} = require('path');
const {realpathSync, existsSync, readFileSync, statSync} = require('fs');
// Helper: walk up from a file/dir to find the openclaw package root
function walkUp(start) {
let dir = statSync(start).isDirectory() ? start : dirname(start);
for (let i = 0; i < 10; i++) {
const pj = join(dir, 'package.json');
if (existsSync(pj)) {
try {
const pkg = JSON.parse(readFileSync(pj, 'utf8'));
if (pkg.name === 'openclaw') return dir;
} catch {}
}
const parent = dirname(dir);
if (parent === dir) break;
dir = parent;
}
return null;
}
// Strategy 1: which openclaw → realpath → walk up
try {
const {execSync} = require('child_process');
const bin = execSync('which openclaw', {encoding:'utf8'}).trim();
const real = realpathSync(bin);
const found = walkUp(real);
if (found) { console.log(found); process.exit(0); }
// pnpm uses shell shims: the bin file is a script, not a symlink.
// Read the shim content to extract the real entry point path.
const content = readFileSync(bin, 'utf8');
// pnpm shim contains a line like: exec node \"/path/.../openclaw/dist/cli.js\"
// or: require(\"/path/.../openclaw/dist/cli.js\")
const m = content.match(/['\"]([^'\"]*openclaw[^'\"]*\\.(?:js|mjs))['\"]/) ||
content.match(/['\"]([^'\"]*openclaw[^'\"]*)['\"].*node/);
if (m) {
const shimTarget = realpathSync(m[1]);
const found2 = walkUp(shimTarget);
if (found2) { console.log(found2); process.exit(0); }
}
} catch {}
// Strategy 2: search common pnpm/npm global paths
const {execSync: exec2} = require('child_process');
const searchDirs = [
join(process.env.HOME || '/root', '.local/share/pnpm'),
join(process.env.HOME || '/root', '.local/node/lib/node_modules'),
'/usr/local/lib/node_modules',
'/usr/lib/node_modules',
];
for (const base of searchDirs) {
if (!existsSync(base)) continue;
try {
const out = exec2(
'find ' + JSON.stringify(base) + ' -maxdepth 8 -name package.json -path \"*/openclaw/package.json\" 2>/dev/null',
{encoding:'utf8', timeout: 5000}
).trim();
for (const line of out.split('\\n')) {
if (!line) continue;
try {
const pkg = JSON.parse(readFileSync(line, 'utf8'));
if (pkg.name === 'openclaw') { console.log(dirname(line)); process.exit(0); }
} catch {}
}
} catch {}
}
process.exit(1);
" 2>/dev/null
}
if [[ -n "${1:-}" ]]; then
OPENCLAW_DIR="$1"
elif OPENCLAW_DIR="$(_node_resolve_openclaw)"; then
debug "Node.js resolved openclaw → $OPENCLAW_DIR"
else
fail "找不到 OpenClaw 安装目录。请手动指定：\n bash $0 /path/to/openclaw"
fi
DIST_DIR="$OPENCLAW_DIR/dist"
if [[ ! -d "$DIST_DIR" ]]; then
fail "dist 目录不存在: $DIST_DIR"
fi
info "OpenClaw 目录: $OPENCLAW_DIR"
# ─── 检测 OpenClaw 版本 ──────────────────────────────────────────
VERSION=$(grep -oP '"version"\s*:\s*"\K[^"]+' "$OPENCLAW_DIR/package.json" 2>/dev/null || echo "unknown")
info "检测到 OpenClaw 版本: $VERSION"
# ─── 已 patch 检测 ───────────────────────────────────────────────
# 核心标记：hookEvent 内部 durationMs 之后紧跟 messages 注入
# 支持多种缩进格式（tab / 空格 / 混合）
INJECTION_CODE='messages: ctx.params.session?.messages'
INJECTION_CODE_ALT='messages:ctx.params.session?.messages'
is_already_patched() {
local f="$1"
# 方法 1: 精确检测 — durationMs 后面紧跟 messages 注入（允许任意空白）
if perl -0777 -ne 'exit(0) if /durationMs[,\s]*\n\s*messages\s*:\s*ctx\.params\.session\?\s*\.messages/; exit(1)' "$f" 2>/dev/null; then
return 0
fi
# 方法 2: 上下文检测 — after_tool_call hookEvent 对象内部（durationMs 附近）有 messages 注入
# 注意: 不能用全文件检测，因为 before_compaction 也有 messages: ctx.params.session.messages
if perl -0777 -ne 'exit(0) if /(?:hookEvent|hook_event)\s*=\s*\{[\s\S]{0,500}durationMs[\s\S]{0,100}messages\s*:\s*ctx\.params\.session/; exit(1)' "$f" 2>/dev/null; then
return 0
fi
return 1
}
verify_patch() {
local f="$1"
is_already_patched "$f"
}
# ─── 备份工具 ────────────────────────────────────────────────────
backup_file() {
local f="$1"
local bak="${f}.pre-offload-patch.bak"
if [[ ! -f "$bak" ]]; then
cp "$f" "$bak"
debug "备份: $bak"
fi
}
# ─── 查找所有候选文件 ────────────────────────────────────────────
# 收集所有包含 after_tool_call 的 JS 文件（不限子目录深度）
mapfile -t CANDIDATE_FILES < <(grep -rl 'after_tool_call' "$DIST_DIR" --include='*.js' 2>/dev/null || true)
if [[ ${#CANDIDATE_FILES[@]} -eq 0 ]]; then
warn "在 $DIST_DIR 下未找到包含 after_tool_call 的 JS 文件"
fi
info "找到 ${#CANDIDATE_FILES[@]} 个候选文件"
PATCHED=0
SKIPPED=0
FAILED=0
# ─── 对每个候选文件尝试多种策略 ──────────────────────────────────
for f in "${CANDIDATE_FILES[@]}"; do
fname="$(basename "$f")"
relpath="${f#$DIST_DIR/}"
# 已 patch → 跳过
if is_already_patched "$f"; then
warn "$relpath — 已经 patch 过，跳过"
((SKIPPED++)) || true
continue
fi
# 确认文件中有 durationMs（hookEvent 的标志字段）
if ! grep -q 'durationMs' "$f" 2>/dev/null; then
debug "$relpath — 不含 durationMs，非 patch 目标，跳过"
continue
fi
# 确认 durationMs 附近有 after_tool_call 上下文（避免误匹配 before_compaction 等）
if ! perl -0777 -ne 'exit(0) if /after_tool_call[\s\S]{0,2000}durationMs/; exit(1)' "$f" 2>/dev/null; then
debug "$relpath — durationMs 不在 after_tool_call 上下文中，跳过"
continue
fi
backup_file "$f"
applied=false
# ── 策略 1: hookEvent 对象中 durationMs 是最后一个字段 ────────
# 匹配: durationMs<换行><空白>};<换行><空白>hookRunnerAfter
# 或: durationMs<换行><空白>};<换行><空白>await ...hookRunner...afterToolCall
# 缩进用 \s+ 宽松匹配
if [[ "$applied" == "false" ]]; then
if perl -0777 -ne 'exit(0) if /durationMs\s*\n(\s*)\};\s*\n\s*(hookRunnerAfter|await\s+\S*hookRunner\S*\.runAfterToolCall|hookRunner\S*\.runAfterToolCall)/; exit(1)' "$f" 2>/dev/null; then
debug "$relpath — 命中策略1 (hookRunnerAfter 锚点)"
perl -0777 -i -pe 's/(durationMs)\s*\n(\s*\};\s*\n\s*(?:hookRunnerAfter|await\s+\S*hookRunner\S*\.runAfterToolCall|hookRunner\S*\.runAfterToolCall))/$1,\n\t\t\tmessages: ctx.params.session?.messages\n$2/' "$f"
if verify_patch "$f"; then
ok "[策略1] $relpath — patch 成功"
((PATCHED++)) || true
applied=true
fi
fi
fi
# ── 策略 2: 旧版 dispatch-*.js — durationMs 行末独占 ─────────
if [[ "$applied" == "false" ]]; then
if echo "$relpath" | grep -qP 'dispatch-.*\.js' 2>/dev/null; then
# 匹配行末独占的 durationMs（前面是空白）
if grep -qP '^\s+durationMs\s*$' "$f" 2>/dev/null; then
debug "$relpath — 命中策略2 (旧版 dispatch)"
sed -i -E 's/^(\s+)(durationMs)\s*$/\1\2,\n\1messages: ctx.params.session?.messages/' "$f"
if verify_patch "$f"; then
ok "[策略2] $relpath — patch 成功"
((PATCHED++)) || true
applied=true
fi
fi
fi
fi
# ── 策略 3: durationMs 后跟 }; 但无 hookRunnerAfter 锚点 ────
# 匹配: durationMs<换行><空白>}; （hookEvent 闭合）
# 通过上下文（附近有 after_tool_call）确认是正确的对象
if [[ "$applied" == "false" ]]; then
# 找到包含 durationMs 且附近(±20行)有 after_tool_call 的代码区域
if perl -0777 -ne 'exit(0) if /after_tool_call[\s\S]{0,800}durationMs\s*\n(\s*)\};/; exit(1)' "$f" 2>/dev/null; then
debug "$relpath — 命中策略3 (durationMs→}; 邻近 after_tool_call)"
# 只替换 after_tool_call 上下文附近的 durationMs → };
perl -0777 -i -pe 's/(after_tool_call[\s\S]{0,800}durationMs)\s*\n(\s*\};)/$1,\n\t\t\tmessages: ctx.params.session?.messages\n$2/' "$f"
if verify_patch "$f"; then
ok "[策略3] $relpath — patch 成功"
((PATCHED++)) || true
applied=true
fi
fi
fi
# ── 策略 4: 通用 fallback — hookEvent 赋值中的 durationMs ────
# 匹配形如: const hookEvent = { ... durationMs ... }
# 或: hookEvent = { ... durationMs ... }
# 用 perl 找到包含 after_tool_call 和 durationMs 的对象字面量，在 durationMs 后插入
if [[ "$applied" == "false" ]]; then
# 极宽松: 在文件中找到 "durationMs" 后面最近的 "}" 或 "};" ，在中间插入
# 但限制在 after_tool_call 关键字附近 2000 字符内
if perl -0777 -ne 'exit(0) if /after_tool_call[\s\S]{0,2000}?(?:hookEvent|hook_event)[\s\S]{0,500}?durationMs/; exit(1)' "$f" 2>/dev/null; then
debug "$relpath — 命中策略4 (通用 fallback)"
# 在 durationMs 后追加 (仅首次匹配)
perl -0777 -i -pe '
my $done = 0;
s/(after_tool_call[\s\S]{0,2000}?(?:hookEvent|hook_event)[\s\S]{0,500}?durationMs)\s*\n(\s*)(\};)/
if (!$done) { $done = 1; "$1,\n$2\tmessages: ctx.params.session?.messages\n$2$3" }
else { "$1\n$2$3" }
/ge;
' "$f"
if verify_patch "$f"; then
ok "[策略4] $relpath — patch 成功"
((PATCHED++)) || true
applied=true
else
warn "[策略4] $relpath — patch 后验证失败，恢复备份"
cp "${f}.pre-offload-patch.bak" "$f"
fi
fi
fi
# ── 无策略命中 ───────────────────────────────────────────────
if [[ "$applied" == "false" ]]; then
debug "$relpath — 无策略命中"
((FAILED++)) || true
fi
done
# ─── 结果报告 ────────────────────────────────────────────────────
echo ""
echo -e "${GREEN}═══════════════════════════════════════════════════════${NC}"
echo -e "${GREEN} Patch 完成 (OpenClaw $VERSION)${NC}"
echo -e "${GREEN}═══════════════════════════════════════════════════════${NC}"
echo ""
echo -e " 成功: ${GREEN}${PATCHED}${NC} 跳过: ${YELLOW}${SKIPPED}${NC} 失败: ${RED}${FAILED}${NC}"
echo ""
if [[ $PATCHED -gt 0 ]]; then
echo -e " ${CYAN}重启 OpenClaw 后生效。${NC}"
echo -e " ${CYAN}备份文件: *.pre-offload-patch.bak${NC}"
elif [[ $SKIPPED -gt 0 && $FAILED -eq 0 ]]; then
echo -e " ${YELLOW}所有目标文件已 patch，无需重复操作。${NC}"
elif [[ $FAILED -gt 0 ]]; then
echo -e " ${RED}部分文件未能 patch。可能需要手动检查或更新 patch 脚本。${NC}"
echo -e " ${RED}提示：设置 DEBUG=1 运行以查看详细匹配过程：${NC}"
echo -e " ${RED} DEBUG=1 bash $0 $OPENCLAW_DIR${NC}"
else
echo -e " ${RED}未找到匹配的目标文件，请确认 OpenClaw 版本。${NC}"
echo -e " ${RED}提示：设置 DEBUG=1 运行以查看详细匹配过程：${NC}"
echo -e " ${RED} DEBUG=1 bash $0 $OPENCLAW_DIR${NC}"
fi
echo ""
# ─── 退出码 ───────────────────────────────────────────────────────
# 0: 成功（至少有一个文件 patch 成功或已跳过）
# 1: 失败（无任何 patch 成功且无已跳过的文件）
if [[ $PATCHED -gt 0 || $SKIPPED -gt 0 ]]; then
exit 0
else
exit 1
fi
scripts/read-local-memory/read-local-memory.ts
+1057
View File
File diff suppressed because it is too large Load Diff
scripts/read-local-memory/tsconfig.json
+19
View File
@@ -0,0 +1,19 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "Node16",
"moduleResolution": "Node16",
"outDir": "./dist",
"rootDir": ".",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"types": ["node"],
"declaration": false,
"sourceMap": false
},
"include": ["read-local-memory.ts"],
"exclude": ["dist", "node_modules"]
}
scripts/seed-v2/README.md
+229
View File
@@ -0,0 +1,229 @@
# seed-v2
> 通过 v2 API 把历史对话灌入 memory-tencentdb gateway。
## 是什么
`seed-v2` 是一个**纯 HTTP 客户端**，把一批历史对话喂给已经在跑的 gateway：
```
fixture.json
│
│ HTTP POST /v2/conversation/add (一轮一次)
▼
gateway (standalone or service)
│
│ 自动调度 L0 → L1 → L2 → L3
▼
本地 vectors.db / records / scene_blocks / persona.md
```
客户端**不 import 任何 plugin 内部模块**，只用 fetch + `/v2/conversation/add` + `/v2/pipeline/status`。
跟老版 `src/cli/commands/seed.ts`（v1，import 核心 runtime 直跑）相比：
- ✅ 同一份代码同时适用于 standalone 和 service 模式
- ✅ 不会和已经在跑的 gateway 抢同一份 SQLite / state
- ✅ 通过 status 接口节流，串行写入 → 等抽取完 → 再写下一批
- ✅ 调度状态由 gateway 侧 `StatefulPipelineManager` 统一管，跟生产路径完全一致
## 调用方式
### 方式 1：通过 npm bin（推荐）
```bash
# 1. 把 gateway 起起来（standalone 用法）
cd extensions/memory-tencentdb/__tests__/standalone && ./start.sh
# 2. 灌数据
cd ../..
npm run seed-v2 -- --input ./scripts/seed-v2/fixtures/minimal.json
# 3. 完事停 gateway
./__tests__/standalone/stop.sh
```
### 方式 2：直接命令
```bash
node ./bin/seed-v2.mjs --input fixture.json --endpoint http://127.0.0.1:18420
```
### 方式 3：一键 wrapper（含起停 + 落盘断言）
```bash
./scripts/seed-v2/seed-v2.sh # 默认 fixture
./scripts/seed-v2/seed-v2.sh path/to/my-fixture.json # 自定 fixture
./scripts/seed-v2/seed-v2.sh --keep # 跑完不停 gateway
./scripts/seed-v2/seed-v2.sh --no-start # gateway 已在跑
```
## CLI 参数
| 参数 | 短 | env | 默认 | 说明 |
| --- | --- | --- | --- | --- |
| `--input <file>` | `-i` | — | **必填** | fixture JSON 路径 |
| `--endpoint <url>` | `-e` | `SEED_ENDPOINT` | `http://127.0.0.1:18420` | gateway 地址 |
| `--api-key <key>` | — | `SEED_API_KEY` | `standalone-e2e` | Bearer key |
| `--service-id <id>` | `-s` | `SEED_SERVICE_ID` | `default` | `x-tdai-service-id` |
| `--every-n <n>` | `-n` | `SEED_EVERY_N` | 5 | 每 N 轮等一次 L1 idle |
| `--poll-ms <ms>` | — | `SEED_POLL_MS` | 500 | status 轮询间隔 |
| `--stable-rounds <n>` | — | `SEED_STABLE_ROUNDS` | 2 | 连续 N 次 idle 才认为 stable |
| `--max-wait-ms <ms>` | — | `SEED_MAX_WAIT_MS` | 600000 | 每批 wait（L1 only）超时（默认 10 分钟）|
| `--final-max-wait-ms <ms>` | — | `SEED_FINAL_MAX_WAIT_MS` | 600000 | 最终 wait（L1+L2+L3 全 idle）超时（默认 10 分钟）|
| `--no-final-wait` | — | — | false | 写完最后一批不等 cascade |
| `--dry-run` | — | — | false | 只打印计划不实际请求 |
| `--session-key <key>` | — | `SEED_FALLBACK_SESSION_KEY` | — | fallback 给缺 sessionKey 的 session |
| `--strict-round-role` | — | `SEED_STRICT_ROUND_ROLE=1` | false | 每 round 必须含 user+assistant |
| `--no-auto-fill-timestamps` | — | `SEED_AUTO_FILL_TIMESTAMPS=0` | false | 缺 ts 时不自动 fill |
| `--quiet` | `-q` | `SEED_VERBOSE=0` | false | 静默 |
| `--help` | `-h` | — | — | 帮助 |
## Fixture 校验（与 v1 seed 完全对齐）
启动时会做 6 层校验（任一失败 → exit 2，不会浪费时间灌一半才挂）：
| 层 | 检查 |
| --- | --- |
| 1. file | 存在 + 非空 + 合法 JSON |
| 2. top-level | Format A `{ sessions: [...] }` 或 Format B `[...]` |
| 3. session | sessionKey 必须非空字符串 + conversations 必须 2D array |
| 4. round | 必须非空 array；`--strict-round-role` 时还要求 user + assistant 同时存在 |
| 5. message | role/content 非空字符串 + timestamp 类型校验（int 或 ISO string）|
| 6. timestamp consistency | **全有 / 全无 / mixed** 三态：mixed 拒绝；全无时按 `--no-auto-fill-timestamps` 决定行为 |
错误输出示例（mixed timestamps）：
```
$ seed-v2 --input mixed.json
[seed-v2] Seed input validation failed (1 error(s)):
[timestamp_consistency] Timestamp consistency check failed: some messages have timestamps while others do not. ...
exit code: 2
```
## Timestamp 三态
| 状态 | 行为 |
| --- | --- |
| **全有** | 保留原值，`hasTimestamps=true` |
| **全无 + auto-fill 开**（默认）| `fillTimestamps()` 给全局递增 epoch ms（`Date.now()` 起，每条 +100ms）—— 跨 session 也单调，避免 L0 capture cursor 误过滤 |
| **全无 + `--no-auto-fill-timestamps`** | timestamp=0；写入时 client 用 `Date.now()` 兜底 |
| **mixed** | exit 2，明确报错 |
## Fixture 格式
兼容老 seed Format A（`{ sessions: [...] }`）和 Format B（顶层数组）：
```jsonc
{
"sessions": [
{
"sessionKey": "user-001",
"sessionId": "user-001", // 可选，缺省 = sessionKey
"conversations": [ // rounds
[ // round 0：一组消息
{ "role": "user", "content": "..." },
{ "role": "assistant", "content": "..." }
],
[ /* round 1 ... */ ]
]
}
]
}
```
参考最小示例：[`fixtures/minimal.json`](./fixtures/minimal.json)（2 session × 6 round = 24 messages）。
## 阻塞节奏（重要）
完全对齐老 `src/core/seed/seed-runtime.ts:executeSeed`：
```
for session, round:
POST /v2/conversation/add { session_id, messages } # 立刻返回
if (round + 1) % every_n == 0:
waitForL1Idle (stable 2 polls) # 只等 L1，L2/L3 后台异步跑
end-of-session: waitForL1Idle # 末尾再等一次
final: waitForL1Idle # 写完所有 round 后再等 L1 drain
```
`/v2/pipeline/status` 按 L 分桶返回 `{ l1: {idle,…}, l2: {…}, l3: {…} }`。seed-v2 **只看 `data.l1.idle`**：
- `l1.idle = (l1.queued === 0 && l1.running === 0)` — 跟老 v1 `seed-runtime.waitForL1Idle` 完全对齐
- L2/L3 抽取慢（cascade 一次 LLM 工具链可能 3-5 分钟）→ **不会**阻塞 seed 的下一批 dispatch
- L2 LLM timeout / lock-conflict drop 等异常情况 → **不会**让 `l1.idle` 误判 false（之前用单标量 `busy` 时会被这两个 bug 污染）
详见 [`docs/plans/server/14-pipeline-status-api.md`](../../docs/plans/server/14-pipeline-status-api.md)。
## Pipeline 调参（服务端 yaml）
灌入速度由**两侧**共同决定：seed 客户端的 wait 节奏 + gateway 流水线的触发节奏。客户端参数已在上面 [CLI 参数](#cli-参数) 表里，下面是**服务端**对应的 `tdai-gateway.standalone.yaml` 配置项（标星号的最常调）：
```yaml
memory:
capture:
enabled: true # L0 capture 开关，必须 true 否则没数据进流水线
extraction:
enabled: true
enableDedup: true # L1 抽取后去重（vector recall topK=5）；关掉提速但可能产生重复 record
maxMemoriesPerSession: 20 # ★ 单 session L1 抽取的 atomic memory 数上限
persona:
triggerEveryN: 50 # ★ 每抽取 N 条 L1 record 触发 1 次 L3 persona-gen
maxScenes: 15 # L2 scene 数上限（达到后停止 CREATE 新 scene）
pipeline:
everyNConversations: 5 # ★★★ 每 N 轮对话触发 1 次 L1 抽取（必须跟 seed 客户端 --every-n 对齐）
enableWarmup: true # 冷启动前几次更快触发 L1（让 persona.md 早点形成骨架）
l1IdleTimeoutSeconds: 600 # ★ session 空闲 N 秒后强制触发 L1（兜底，防 conversation < everyN 时 L1 永不抽）
l2DelayAfterL1Seconds: 90 # ★★ L1 完成后延迟 N 秒触发 L2（避免每条 L1 都触发一次 L2，太碎）
l2MinIntervalSeconds: 900 # ★★ 同 session 两次 L2 之间最小间隔（节流，防 L2 LLM 调用过频）
l2MaxIntervalSeconds: 3600 # ★ 同 session L2 最大间隔（兜底，无论间隔多久必跑一次）
```
### 怎么调：常见场景
| 场景 | 怎么调 |
| --- | --- |
| **seed 灌入想最快**（接受 L1 粒度粗）| `everyNConversations: 20`（每 20 轮才抽一次 L1）+ seed `--every-n 20` 对齐 |
| **seed 灌入想 L1 立即抽**（每轮都抽，最细粒度但贵）| `everyNConversations: 1` + seed `--every-n 1` |
| **fixture 是短对话**（< everyN 轮）触发不到 L1 | 缩 `l1IdleTimeoutSeconds: 60`（1分钟无新消息就触发兜底）|
| **L2 cascade 太频繁**（cost 高）| 拉大 `l2MinIntervalSeconds: 1800`（30 分钟才一次）|
| **L2 cascade 太稀疏** | 缩 `l2DelayAfterL1Seconds: 30` + `l2MinIntervalSeconds: 300` |
| **测试/调试想看快速反馈** | `triggerEveryN: 10` + `everyNConversations: 2` + `l1IdleTimeoutSeconds: 30` |
### seed 客户端 vs 服务端的对应
| seed 参数 | 服务端 yaml 对应 | 是否需对齐 |
| --- | --- | --- |
| `--every-n` (`SEED_EVERY_N`) | `pipeline.everyNConversations` | ⚠️ **建议一致**（客户端 wait 间隔 = 服务端触发间隔，灌入流畅）|
| `--max-wait-ms` | `pipeline.l1IdleTimeoutSeconds` | wait 时长应 ≥ idle timeout（否则会 max-wait-reach 而不是 stable-idle）|
| `--final-max-wait-ms` | 综合 L2 慢 task 时长 | ≥ p95(L2 LLM duration) × maxScenes/(同时抽的 scene 数)（默认 10 分钟够大多数场景）|
| `--stable-rounds` | — | 客户端独有，控制连续多少次 idle 才认定真稳定 |
### 调参对 seed 总耗时的影响（粗估）
| 配置 | persona-0（10 sessions × 366 msgs）耗时 |
| --- | --- |
| 默认（`everyNConversations=5`，`l2DelayAfterL1=90s`，`l2MinInterval=900s`）| ~90 min（实测 persona-0）|
| 降本：`everyNConversations=20`，`l2MinInterval=3600s` | ~30-50 min（L1 抽得粗，L2 抽得少）|
| 极速：`--no-final-wait` + 仅 L0 模式（`extraction.enabled=false`）| ~1-2 min（只灌 L0，不抽取）|
> ⚠️ **改 yaml 后必须重启 gateway**（`scripts/seed-v2/seed-v2.sh` 自动起停；如果用 `start.sh` 手动起的需 `stop.sh && start.sh`）。
## 退出码
| code | 含义 |
| --- | --- |
| 0 | 全部成功 |
| 1 | seed 失败（写入或 wait 异常）|
| 2 | 前置条件不满足（fixture 缺失 / gateway 不通 / 配置错误）|
## 构建 + 发布
```bash
npm run build:seed-v2 # tsc 编译到 scripts/seed-v2/dist/
npm run build:scripts # 构建所有 scripts（含 seed-v2）
```
`bin/seed-v2.mjs` 是薄启动器：dist 存在就跑 dist，否则 fallback 到 `tsx` 跑源码。开发期不需要先 build。
scripts/seed-v2/fixtures/minimal.json
+64
View File
@@ -0,0 +1,64 @@
{
"sessions": [
{
"sessionKey": "seed-alice",
"sessionId": "seed-alice",
"conversations": [
[
{ "role": "user", "content": "我叫 Alice，最近在学 Rust 语言，主要在做嵌入式开发。" },
{ "role": "assistant", "content": "了解，Rust 在嵌入式领域优势明显。你目前用的是哪块板子？" }
],
[
{ "role": "user", "content": "我用的是 STM32H7 系列。我特别喜欢 Rust 的 ownership 模型，感觉对资源管理很有帮助。" },
{ "role": "assistant", "content": "STM32H7 性能不错。Rust 的所有权模型确实能在编译期就排除很多资源泄漏问题。" }
],
[
{ "role": "user", "content": "我打算做一个低功耗的传感器节点，用 LoRa 通信。" },
{ "role": "assistant", "content": "LoRa 适合低功耗场景。你考虑用 RTIC 还是 embassy 这种异步框架吗？" }
],
[
{ "role": "user", "content": "我倾向于 embassy，async/await 写起来更顺。" },
{ "role": "assistant", "content": "embassy 在嵌入式 async 场景里生态成熟，executor 也很轻量。" }
],
[
{ "role": "user", "content": "我每天大概能写 4 小时代码，主要在晚上 8 点到 12 点。" },
{ "role": "assistant", "content": "晚上是连续编码的好时段。建议早上 review 一下前一天的代码再开始新模块。" }
],
[
{ "role": "user", "content": "对了，我之前用的语言是 C 和 Python，从 C 转 Rust 一开始很难适应借用检查器。" },
{ "role": "assistant", "content": "从 C 转 Rust 确实需要思维转换，多用 cargo clippy 帮你养成习惯。" }
]
]
},
{
"sessionKey": "seed-bob",
"sessionId": "seed-bob",
"conversations": [
[
{ "role": "user", "content": "Hi，我是 Bob，做数据分析的，平时主要用 Python 和 SQL。" },
{ "role": "assistant", "content": "Python + SQL 是数据分析的黄金组合。你做哪类业务的数据？" }
],
[
{ "role": "user", "content": "电商，主要看用户行为漏斗和复购率。最近在搞 RFM 模型。" },
{ "role": "assistant", "content": "RFM 是经典模型。你打算用它做用户分群再做精准营销吗？" }
],
[
{ "role": "user", "content": "对，分群之后想接 A/B 测试，看不同分群对优惠券敏感度。" },
{ "role": "assistant", "content": "好思路。建议把 RFM 评分时间窗对齐 A/B 实验周期，避免季节性偏差。" }
],
[
{ "role": "user", "content": "我团队有 3 个人，一个 BI、一个数据科学家，加我自己。" },
{ "role": "assistant", "content": "团队结构合理。BI 负责报表，数据科学家做模型，你协调和落地。" }
],
[
{ "role": "user", "content": "我们用 ClickHouse 存日志，每天大概 5 亿条记录。" },
{ "role": "assistant", "content": "ClickHouse 处理这种规模没压力。注意 partition 设计和 MergeTree 引擎选型。" }
],
[
{ "role": "user", "content": "对了，我下个月要去新加坡出差一周参加 data conf。" },
{ "role": "assistant", "content": "祝出差顺利！data conf 是个不错的拓展视野的场合。" }
]
]
}
]
}
scripts/seed-v2/fixtures/mixed-timestamps.json
+13
View File
@@ -0,0 +1,13 @@
{
"sessions": [
{
"sessionKey": "mixed-bob",
"conversations": [
[
{ "role": "user", "content": "测试 mixed timestamp 是否被检测", "timestamp": 1700000000000 },
{ "role": "assistant", "content": "这一条故意没有 timestamp" }
]
]
}
]
}
scripts/seed-v2/fixtures/no-timestamps.json
+17
View File
@@ -0,0 +1,17 @@
{
"sessions": [
{
"sessionKey": "no-ts-alice",
"conversations": [
[
{ "role": "user", "content": "我叫 Alice，最近在学 Python 数据分析。" },
{ "role": "assistant", "content": "Python 数据分析通常涉及 pandas、numpy 这些库。你目前学到哪一步？" }
],
[
{ "role": "user", "content": "我刚学完 pandas 基础，现在在做股票数据的处理。" },
{ "role": "assistant", "content": "股票数据典型的就是时间序列分析，建议熟悉一下 datetime 索引。" }
]
]
}
]
}
scripts/seed-v2/input-types.ts
+85
View File
@@ -0,0 +1,85 @@
/**
* Type definitions for seed-v2 fixture input.
*
* ⚠️ 这份文件是从 `src/core/seed/types.ts` 拷来的精简版（去掉了 v1 CLI 专属的
* `SeedCommandOptions` / `SeedProgress` / `SeedSummary`）。
*
* 老 v1 seed (`src/cli/commands/seed.ts` + `/seed` v1 endpoint) 计划废弃后，
* 本文件成为唯一真理之源。在那之前，如果在 `src/core/seed/types.ts` 改了
* Raw* / Normalized* / ValidationError，请同步到这里。
*/
// ============================
// Raw input types (before validation)
// ============================
export interface RawMessage {
role: string;
content: string;
/** Epoch ms (number) or ISO 8601 string. */
timestamp?: number | string;
}
export interface RawSession {
sessionKey: string;
sessionId?: string;
conversations: RawMessage[][];
}
/** Format A: `{ sessions: [...] }` */
export interface FormatA {
sessions: RawSession[];
}
/** Format B: `[...]` (top-level array of sessions) */
export type FormatB = RawSession[];
// ============================
// Normalized types (after validation)
// ============================
export interface NormalizedMessage {
role: string;
content: string;
/** Epoch ms — always present after normalization. 0 means "to be filled later". */
timestamp: number;
}
export interface NormalizedRound {
messages: NormalizedMessage[];
}
export interface NormalizedSession {
sessionKey: string;
sessionId: string;
rounds: NormalizedRound[];
sourceIndex: number;
}
export interface NormalizedInput {
sessions: NormalizedSession[];
totalRounds: number;
totalMessages: number;
hasTimestamps: boolean;
}
// ============================
// Validation
// ============================
export type ValidationStage =
| "file"
| "top_level"
| "session"
| "round"
| "message"
| "timestamp_consistency";
export interface ValidationError {
stage: ValidationStage;
sourceIndex?: number;
sessionKey?: string;
roundIndex?: number;
messageIndex?: number;
message: string;
}
scripts/seed-v2/input-validate.ts
+408
View File
@@ -0,0 +1,408 @@
/**
* Input loading, validation, normalization, and timestamp handling for seed-v2.
*
* ⚠️ 这份文件是从 `src/core/seed/input.ts` 拷来的精简版（去掉了 v1 CLI 专属的
* `loadAndValidateInput`，因为它依赖 `SeedCommandOptions` 类型且会做交互式
* timestamp 确认 —— seed-v2 是 HTTP 工具，不交互）。
*
* 老 v1 seed (`src/cli/commands/seed.ts` + `/seed` v1 endpoint) 计划废弃后，
* 本文件成为唯一真理之源。在那之前，如果在 `src/core/seed/input.ts` 改了
* `validateAndNormalizeRaw` / `fillTimestamps` / `SeedValidationError`，请
* 同步到这里。
*
* 6 层校验 (file → top-level → session → round → message → timestamp consistency)
* 中的 file 层在 seed-v2 自己读 fixture 时做（`loadFixtureFile`），不在本文件里。
*/
import crypto from "node:crypto";
import type {
RawSession,
FormatA,
ValidationError,
NormalizedInput,
NormalizedSession,
NormalizedRound,
NormalizedMessage,
} from "./input-types.js";
// ============================
// Public API
// ============================
/**
* Validate and normalize seed input from an already-parsed JSON object.
*
* Throws `SeedValidationError` on validation failures.
*
* Timestamp policy:
* - All messages have timestamp → `hasTimestamps=true`, normalized as-is
* - All messages missing → if `autoFillTimestamps` (default true), fill with
* globally-monotonic Date.now() + 100ms increments; else leave at 0
* - Mixed (some have, some missing) → throw error
*/
export function validateAndNormalizeRaw(
raw: unknown,
opts?: { sessionKey?: string; strictRoundRole?: boolean; autoFillTimestamps?: boolean },
): NormalizedInput {
const strictRoundRole = opts?.strictRoundRole ?? false;
const autoFillTimestamps = opts?.autoFillTimestamps ?? true;
// Layer 2: Top-level — detect Format A vs B
const sessions = extractSessions(raw);
// Layers 3-5: session / round / message validation
const errors: ValidationError[] = [];
validateSessions(sessions, strictRoundRole, errors);
if (errors.length > 0) {
throw new SeedValidationError(errors);
}
// Layer 6: Timestamp consistency (all-have / all-missing / mixed → error)
const tsResult = checkTimestampConsistency(sessions);
if (tsResult.status === "mixed") {
throw new SeedValidationError([{
stage: "timestamp_consistency",
message:
"Timestamp consistency check failed: some messages have timestamps while others do not. " +
"All messages must either have timestamps or none must have timestamps.",
}]);
}
// Normalize
const normalized = normalizeSessions(sessions, opts?.sessionKey);
const input: NormalizedInput = {
sessions: normalized.sessions,
totalRounds: normalized.totalRounds,
totalMessages: normalized.totalMessages,
hasTimestamps: tsResult.status === "all_present",
};
// Auto-fill timestamps when none present (HTTP tool, no interactive prompt)
if (tsResult.status === "all_missing" && autoFillTimestamps) {
fillTimestamps(input);
}
return input;
}
/**
* Fill timestamps for all messages when the input has no timestamps.
*
* Uses a single monotonically increasing counter across ALL sessions to
* guarantee global timestamp ordering — critical when multiple sessions
* share the same sessionKey, since L0 capture cursor is advanced per-session
* and would filter out later sessions whose timestamps fall below the cursor
* if ordering were not globally monotonic.
*/
export function fillTimestamps(input: NormalizedInput): void {
let currentTs = Date.now();
for (const session of input.sessions) {
for (const round of session.rounds) {
for (let i = 0; i < round.messages.length; i++) {
round.messages[i]!.timestamp = currentTs;
currentTs += 100; // 100ms gap per message — matches v1 fillTimestamps
}
}
}
input.hasTimestamps = true;
}
// ============================
// Validation error class
// ============================
export class SeedValidationError extends Error {
public readonly errors: ValidationError[];
constructor(errors: ValidationError[]) {
const summary = errors.map((e) => formatValidationError(e)).join("\n");
super(`Seed input validation failed (${errors.length} error(s)):\n${summary}`);
this.name = "SeedValidationError";
this.errors = errors;
}
}
function formatValidationError(e: ValidationError): string {
const parts: string[] = [` [${e.stage}]`];
if (e.sourceIndex != null) parts.push(`session[${e.sourceIndex}]`);
if (e.sessionKey) parts.push(`key="${e.sessionKey}"`);
if (e.roundIndex != null) parts.push(`round[${e.roundIndex}]`);
if (e.messageIndex != null) parts.push(`msg[${e.messageIndex}]`);
parts.push(e.message);
return parts.join(" ");
}
// ============================
// Layer 2: Top-level format detection
// ============================
function extractSessions(raw: unknown): RawSession[] {
// Format A: { sessions: [...] }
if (
raw != null &&
typeof raw === "object" &&
!Array.isArray(raw) &&
"sessions" in raw
) {
const obj = raw as FormatA;
if (!Array.isArray(obj.sessions)) {
throw new SeedValidationError([{
stage: "top_level",
message: 'Format A detected but "sessions" is not an array.',
}]);
}
return obj.sessions;
}
// Format B: [...]
if (Array.isArray(raw)) {
return raw as RawSession[];
}
throw new SeedValidationError([{
stage: "top_level",
message:
"Unrecognized input format. Expected either:\n" +
' Format A: { "sessions": [...] }\n' +
" Format B: [ { sessionKey, conversations }, ... ]",
}]);
}
// ============================
// Layers 3-5: session / round / message validation
// ============================
function validateSessions(
sessions: RawSession[],
strictRoundRole: boolean,
errors: ValidationError[],
): void {
if (sessions.length === 0) {
errors.push({
stage: "session",
message: "No sessions found in input.",
});
return;
}
for (let si = 0; si < sessions.length; si++) {
const session = sessions[si]!;
// Layer 3: session validation
if (!session.sessionKey || typeof session.sessionKey !== "string" || session.sessionKey.trim() === "") {
errors.push({
stage: "session",
sourceIndex: si,
message: '"sessionKey" is required and must be a non-empty string.',
});
}
if (!Array.isArray(session.conversations)) {
errors.push({
stage: "session",
sourceIndex: si,
sessionKey: session.sessionKey,
message: '"conversations" must be a two-dimensional array (array of rounds).',
});
continue;
}
for (let ri = 0; ri < session.conversations.length; ri++) {
const round = session.conversations[ri];
// Layer 4: round validation
if (!Array.isArray(round)) {
errors.push({
stage: "round",
sourceIndex: si,
sessionKey: session.sessionKey,
roundIndex: ri,
message: "Round must be an array of messages.",
});
continue;
}
if (round.length === 0) {
errors.push({
stage: "round",
sourceIndex: si,
sessionKey: session.sessionKey,
roundIndex: ri,
message: "Round must be a non-empty array.",
});
continue;
}
// Strict round-role: each round must have at least one user and one assistant
if (strictRoundRole) {
const roles = new Set(round.map((m) => m.role));
if (!roles.has("user")) {
errors.push({
stage: "round",
sourceIndex: si,
sessionKey: session.sessionKey,
roundIndex: ri,
message: '--strict-round-role: round must contain at least one "user" message.',
});
}
if (!roles.has("assistant")) {
errors.push({
stage: "round",
sourceIndex: si,
sessionKey: session.sessionKey,
roundIndex: ri,
message: '--strict-round-role: round must contain at least one "assistant" message.',
});
}
}
// Layer 5: message validation
for (let mi = 0; mi < round.length; mi++) {
const msg = round[mi]!;
if (!msg.role || typeof msg.role !== "string") {
errors.push({
stage: "message",
sourceIndex: si,
sessionKey: session.sessionKey,
roundIndex: ri,
messageIndex: mi,
message: '"role" is required and must be a non-empty string.',
});
}
if (!msg.content || typeof msg.content !== "string" || msg.content.trim() === "") {
errors.push({
stage: "message",
sourceIndex: si,
sessionKey: session.sessionKey,
roundIndex: ri,
messageIndex: mi,
message: '"content" is required and must be a non-empty string.',
});
}
if (msg.timestamp !== undefined) {
if (typeof msg.timestamp === "number") {
if (!Number.isInteger(msg.timestamp)) {
errors.push({
stage: "message",
sourceIndex: si,
sessionKey: session.sessionKey,
roundIndex: ri,
messageIndex: mi,
message: '"timestamp" must be an integer (epoch milliseconds). Negative values allowed for pre-1970 dates.',
});
}
} else if (typeof msg.timestamp === "string") {
if (Number.isNaN(new Date(msg.timestamp).getTime())) {
errors.push({
stage: "message",
sourceIndex: si,
sessionKey: session.sessionKey,
roundIndex: ri,
messageIndex: mi,
message: `"timestamp" string is not a valid ISO 8601 date: "${msg.timestamp}".`,
});
}
} else {
errors.push({
stage: "message",
sourceIndex: si,
sessionKey: session.sessionKey,
roundIndex: ri,
messageIndex: mi,
message: '"timestamp" must be a number (epoch ms) or an ISO 8601 string.',
});
}
}
}
}
}
}
// ============================
// Layer 6: Timestamp consistency
// ============================
interface TimestampCheckResult {
status: "all_present" | "all_missing" | "mixed";
}
function checkTimestampConsistency(sessions: RawSession[]): TimestampCheckResult {
let hasTs = false;
let missingTs = false;
for (const session of sessions) {
if (!Array.isArray(session.conversations)) continue;
for (const round of session.conversations) {
if (!Array.isArray(round)) continue;
for (const msg of round) {
if (msg.timestamp !== undefined && msg.timestamp !== null) {
hasTs = true;
} else {
missingTs = true;
}
if (hasTs && missingTs) {
return { status: "mixed" };
}
}
}
}
if (hasTs && !missingTs) return { status: "all_present" };
if (!hasTs && missingTs) return { status: "all_missing" };
// No messages at all — treat as all_missing (will be caught by session validation)
return { status: "all_missing" };
}
// ============================
// Normalization
// ============================
function normalizeSessions(
sessions: RawSession[],
fallbackSessionKey?: string,
): { sessions: NormalizedSession[]; totalRounds: number; totalMessages: number } {
const normalized: NormalizedSession[] = [];
let totalRounds = 0;
let totalMessages = 0;
for (let si = 0; si < sessions.length; si++) {
const raw = sessions[si]!;
const sessionKey = raw.sessionKey || fallbackSessionKey || "seed-user";
const sessionId = raw.sessionId || crypto.randomUUID();
const rounds: NormalizedRound[] = [];
for (const rawRound of raw.conversations) {
if (!Array.isArray(rawRound)) continue;
const messages: NormalizedMessage[] = rawRound.map((msg) => ({
role: msg.role,
content: msg.content,
// Normalize timestamp: ISO string → epoch ms; number → pass-through; missing → 0 (filled later)
timestamp: msg.timestamp == null
? 0
: typeof msg.timestamp === "string"
? new Date(msg.timestamp).getTime()
: msg.timestamp,
}));
rounds.push({ messages });
totalMessages += messages.length;
}
totalRounds += rounds.length;
normalized.push({
sessionKey,
sessionId,
rounds,
sourceIndex: si,
});
}
return { sessions: normalized, totalRounds, totalMessages };
}
scripts/seed-v2/seed-v2.sh
Executable
+210
View File
@@ -0,0 +1,210 @@
#!/usr/bin/env bash
# ============================================================
# seed-v2.sh — 起 standalone gateway → 跑 seed-v2 → 验落盘 → 停
#
# 这是给"想完整自动化跑一次"的用户准备的便利 wrapper。
# 不想自动起停 gateway 的话，直接 `npm run seed-v2 -- --input <file>` 即可。
#
# 用法:
# ./seed-v2.sh # 默认用 fixtures/minimal.json
# ./seed-v2.sh path/to/fixture.json # 用指定 fixture
# ./seed-v2.sh --keep # 跑完不停 gateway（调试）
# ./seed-v2.sh --no-start # 假设 gateway 已在跑（不 reset、不 start、不 stop）
# ./seed-v2.sh --no-reset # 不清空数据目录（增量灌）
#
# 透传给 seed-v2.ts 的环境变量:
# SEED_ENDPOINT / SEED_API_KEY / SEED_SERVICE_ID
# SEED_EVERY_N / SEED_POLL_MS / SEED_STABLE_ROUNDS / SEED_MAX_WAIT_MS
# SEED_VERBOSE 1=详细 / 0=安静
#
# 退出码:
# 0 成功 + 落盘断言全部通过
# 1 seed 或断言失败
# 2 前置条件不满足
# ============================================================
set -euo pipefail
# Compute our own dir BEFORE sourcing env.sh (which overwrites SCRIPT_DIR).
SEED_V2_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PLUGIN_ROOT="$(cd "$SEED_V2_DIR/../.." && pwd)"
STANDALONE_HELPERS="$PLUGIN_ROOT/__tests__/standalone"
# 复用 standalone 测试套件里的 env / start / stop 工具（保持一份真理之源，
# 不在这里重复实现 gateway 起停）
if [ ! -f "$STANDALONE_HELPERS/env.sh" ]; then
echo "✗ standalone helpers not found at: $STANDALONE_HELPERS" >&2
echo " this wrapper depends on env.sh / start.sh / stop.sh from there." >&2
exit 2
fi
# shellcheck disable=SC1091
source "$STANDALONE_HELPERS/env.sh"
# ============================================================
# 参数
# ============================================================
KEEP_RUNNING=0
NO_START=0
DO_RESET=1
FIXTURE="$SEED_V2_DIR/fixtures/minimal.json"
while [ $# -gt 0 ]; do
case "$1" in
--keep) KEEP_RUNNING=1 ;;
--no-start) NO_START=1; KEEP_RUNNING=1; DO_RESET=0 ;;
--no-reset) DO_RESET=0 ;;
-h|--help)
sed -n '2,25p' "$0" | sed 's/^# \?//'
exit 0
;;
-*)
log_err "unknown flag: $1"
exit 2
;;
*)
FIXTURE="$1"
;;
esac
shift
done
[ -f "$FIXTURE" ] || { log_err "fixture not found: $FIXTURE"; exit 2; }
# ============================================================
# Reset 数据目录（必须在 start.sh 之前）
# ============================================================
if [ "$DO_RESET" -eq 1 ]; then
if echo "$STANDALONE_DATA_DIR" | grep -q "/\.codebuddy/"; then
log_info "resetting data dir: $STANDALONE_DATA_DIR"
rm -rf "$STANDALONE_DATA_DIR"
else
log_err "refuse to reset $STANDALONE_DATA_DIR (must be under .codebuddy/)"
exit 2
fi
fi
# ============================================================
# 起 gateway
# ============================================================
if [ "$NO_START" -eq 0 ]; then
log_info "starting gateway..."
if ! "$STANDALONE_HELPERS/start.sh"; then
log_err "start.sh failed"
exit 2
fi
fi
if ! gateway_alive; then
log_err "gateway not reachable at $STANDALONE_BASE"
exit 2
fi
# ============================================================
# trap：保证不论结果如何，都按 KEEP_RUNNING 决定要不要停
# ============================================================
cleanup() {
local rc=$?
if [ "$KEEP_RUNNING" -eq 0 ]; then
log_info "stopping gateway"
"$STANDALONE_HELPERS/stop.sh" >/dev/null 2>&1 || true
else
log_info "gateway kept at $STANDALONE_BASE (use $STANDALONE_HELPERS/stop.sh to stop)"
fi
exit $rc
}
trap cleanup EXIT INT TERM
# ============================================================
# 跑 seed-v2（通过 npm bin）
# ============================================================
log_info "running seed-v2 with fixture: $FIXTURE"
echo
cd "$PLUGIN_ROOT"
SEED_ENDPOINT="${SEED_ENDPOINT:-$STANDALONE_BASE}" \
SEED_API_KEY="${SEED_API_KEY:-standalone-e2e}" \
SEED_SERVICE_ID="${SEED_SERVICE_ID:-default}" \
SEED_EVERY_N="${SEED_EVERY_N:-5}" \
SEED_POLL_MS="${SEED_POLL_MS:-500}" \
SEED_STABLE_ROUNDS="${SEED_STABLE_ROUNDS:-2}" \
SEED_MAX_WAIT_MS="${SEED_MAX_WAIT_MS:-600000}" \
SEED_VERBOSE="${SEED_VERBOSE:-1}" \
node "$PLUGIN_ROOT/bin/seed-v2.mjs" --input "$FIXTURE"
SEED_RC=$?
# ============================================================
# 落盘断言
# ============================================================
echo
log_info "=== Post-seed assertions ==="
ASSERT_PASS=0
ASSERT_FAIL=0
assert() {
local name="$1"
local cond="$2"
if eval "$cond" >/dev/null 2>&1; then
log_ok "$name"
ASSERT_PASS=$((ASSERT_PASS+1))
else
log_err "$name"
ASSERT_FAIL=$((ASSERT_FAIL+1))
fi
}
# 1. L0: JSONL 镜像（standalone 特有）
TODAY="$(date +%Y-%m-%d)"
JSONL_PATH="$STANDALONE_DATA_DIR/conversations/$TODAY.jsonl"
assert "L0: JSONL mirror exists" "[ -f \"$JSONL_PATH\" ]"
if [ -f "$JSONL_PATH" ]; then
JSONL_LINES=$(wc -l < "$JSONL_PATH")
log_info " JSONL line count: $JSONL_LINES"
fi
# 2. L0: SQLite (vectors.db)
DB_PATH="$STANDALONE_DATA_DIR/vectors.db"
assert "L0: SQLite vectors.db exists" "[ -f \"$DB_PATH\" ]"
SQLITE_BIN="$(command -v sqlite3 || true)"
if [ -z "$SQLITE_BIN" ] && [ -x "$HOME/miniconda3/bin/sqlite3" ]; then
SQLITE_BIN="$HOME/miniconda3/bin/sqlite3"
fi
if [ -f "$DB_PATH" ] && [ -n "$SQLITE_BIN" ]; then
L0_COUNT="$("$SQLITE_BIN" "$DB_PATH" "SELECT COUNT(*) FROM l0_conversations" 2>/dev/null || echo 0)"
log_info " SQLite l0_conversations count: $L0_COUNT"
assert "L0: SQLite has records" "[ \"$L0_COUNT\" -gt 0 ]"
else
log_info "L0: sqlite3 binary not found, skipping SQLite assertion"
fi
# 3. L1: records JSONL — 仅当 LLM key 配置时会有产出
RECORDS_DIR="$STANDALONE_DATA_DIR/records"
if [ -d "$RECORDS_DIR" ]; then
RECORD_FILES="$(find "$RECORDS_DIR" -name "*.jsonl" -size +0 2>/dev/null | wc -l)"
if [ "$RECORD_FILES" -gt 0 ]; then
log_ok "L1: records/ has $RECORD_FILES non-empty jsonl file(s) [LLM key configured]"
ASSERT_PASS=$((ASSERT_PASS+1))
else
log_info "L1: records/ exists but empty (LLM key likely not configured — skipped, not a failure)"
fi
else
log_info "L1: records/ not created (LLM key likely not configured — skipped, not a failure)"
fi
# 4. Pipeline status: busy=false at the end
STATUS_RESP="$(curl -sS -X POST "$STANDALONE_BASE/v2/pipeline/status" \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer standalone-e2e' \
-H 'x-tdai-service-id: default' \
-d '{}')"
assert "Pipeline status: busy=false at completion" \
"echo '$STATUS_RESP' | python3 -c 'import sys,json; assert json.loads(sys.stdin.read())[\"data\"][\"busy\"] is False'"
# 汇总
echo
if [ "$ASSERT_FAIL" -gt 0 ]; then
log_err "Post-seed assertions: $ASSERT_PASS passed, $ASSERT_FAIL failed"
exit 1
fi
log_ok "Post-seed assertions: $ASSERT_PASS/$ASSERT_PASS passed"
exit $SEED_RC
scripts/seed-v2/seed-v2.ts
+655
View File
@@ -0,0 +1,655 @@
#!/usr/bin/env npx tsx
/**
* seed-v2 — 通过 v2 API 把历史对话灌入 memory-tencentdb gateway。
*
* 与老 `src/cli/commands/seed.ts`（v1，import 核心 runtime 直跑）相比：
* - 纯 HTTP 客户端，不 import 任何 plugin 内部模块
* - 通过 `/v2/conversation/add` 写 L0 + 让 gateway 自动调度 L1/L2/L3
* - 通过轮询 `/v2/pipeline/status` 的 `busy` 标量做节流
* - 既适用于 standalone，也适用于 service 模式（同一份代码）
*
* 阻塞节奏（对齐老 `seed-runtime.executeSeed`）:
* - per-round 写入 conversation/add 后立即继续
* - 每 N 轮（--every-n / SEED_EVERY_N）等一次 busy=false（stable polls）
* - 每个 session 末尾再等一次
* - 全部跑完后做 final drain
*
* @example
* # 用 npm script 入口
* npm run seed-v2 -- --input ./fixtures/minimal.json
*
* # 或直接 tsx
* npx tsx scripts/seed-v2/seed-v2.ts --input fixture.json --endpoint http://127.0.0.1:18420
*
* # 关键参数
* --input <file> 必选，fixture JSON 路径
* --endpoint <url> gateway 地址（默认 http://127.0.0.1:18420）
* --service-id <id> x-tdai-service-id（默认 default）
* --api-key <key> Bearer key（默认 standalone-e2e）
* --every-n <n> 每 N 轮 wait 一次 busy=false（默认 5）
* --max-wait-ms <ms> 单次 wait 最长时间（默认 600000=10 分钟）
* --no-final-wait 写完最后一批不等 cascade，立即退出
* --dry-run 只打印计划不实际请求
* --quiet 静默模式
*/
import { readFileSync, existsSync } from "node:fs";
import { resolve as resolvePath } from "node:path";
import { parseArgs } from "node:util";
import dayjs from "dayjs";
import { validateAndNormalizeRaw, fillTimestamps, SeedValidationError } from "./input-validate.js";
import type { NormalizedInput } from "./input-types.js";
// ============================
// Logger (timestamped console wrappers)
// ============================
/**
* Full ISO 8601 local-timezone timestamp, e.g. "2026-05-21T15:03:00.123+08:00".
* Aligned with the standalone gateway's console logger so seed-v2 lines and
* gateway lines can be merged/sorted by timestamp during incident review.
*/
function ts(): string {
return dayjs().format("YYYY-MM-DDTHH:mm:ss.SSSZ");
}
const log = (msg: string): void => { console.log(`${ts()} ${msg}`); };
const warn = (msg: string): void => { console.warn(`${ts()} ${msg}`); };
const err = (msg: string): void => { console.error(`${ts()} ${msg}`); };
// Truncate long content to a single-line preview safe for logs.
function preview(s: string, max = 80): string {
const oneLine = s.replace(/\s+/g, " ").trim();
return oneLine.length <= max ? oneLine : `${oneLine.slice(0, max)}...`;
}
// ============================
// CLI args + env fallback
// ============================
interface CliOptions {
input: string;
endpoint: string;
apiKey: string;
serviceId: string;
everyN: number;
pollMs: number;
stableRounds: number;
maxWaitMs: number; // per every-N / per-session-tail wait timeout (L1 only)
finalMaxWaitMs: number; // final all-layer drain wait timeout (L1+L2+L3)
finalWait: boolean;
dryRun: boolean;
quiet: boolean;
// Input validation / normalization options (parity with old seed CLI)
sessionKey?: string; // fallback for sessions missing sessionKey
strictRoundRole: boolean; // each round must have user + assistant
autoFillTimestamps: boolean; // when all-missing, fill globally-monotonic ts
}
function parseCli(argv: string[]): CliOptions {
const { values } = parseArgs({
args: argv,
options: {
input: { type: "string", short: "i" },
endpoint: { type: "string", short: "e" },
"api-key": { type: "string" },
"service-id": { type: "string", short: "s" },
"every-n": { type: "string", short: "n" },
"poll-ms": { type: "string" },
"stable-rounds": { type: "string" },
"max-wait-ms": { type: "string" },
"final-max-wait-ms": { type: "string" },
"no-final-wait": { type: "boolean" },
"dry-run": { type: "boolean" },
"session-key": { type: "string" },
"strict-round-role": { type: "boolean" },
"no-auto-fill-timestamps": { type: "boolean" },
quiet: { type: "boolean", short: "q" },
help: { type: "boolean", short: "h" },
},
allowPositionals: false,
strict: false,
});
if (values.help) {
printHelp();
process.exit(0);
}
const input = (values.input as string | undefined) ?? process.argv[2];
if (!input) {
console.error("error: --input <fixture.json> is required");
printHelp();
process.exit(2);
}
const intOpt = (key: string, env: string, def: number): number => {
const raw = (values[key] as string | undefined) ?? process.env[env];
if (raw == null || raw === "") return def;
const n = parseInt(raw, 10);
return Number.isFinite(n) ? n : def;
};
return {
input,
endpoint: (values.endpoint as string) ?? process.env.SEED_ENDPOINT ?? "http://127.0.0.1:18420",
apiKey: (values["api-key"] as string) ?? process.env.SEED_API_KEY ?? "standalone-e2e",
serviceId: (values["service-id"] as string) ?? process.env.SEED_SERVICE_ID ?? "default",
everyN: intOpt("every-n", "SEED_EVERY_N", 5),
pollMs: intOpt("poll-ms", "SEED_POLL_MS", 500),
stableRounds: intOpt("stable-rounds", "SEED_STABLE_ROUNDS", 2),
maxWaitMs: intOpt("max-wait-ms", "SEED_MAX_WAIT_MS", 180_000),
finalMaxWaitMs: intOpt("final-max-wait-ms", "SEED_FINAL_MAX_WAIT_MS", 600_000),
finalWait: !values["no-final-wait"],
dryRun: Boolean(values["dry-run"]),
quiet: Boolean(values.quiet) || process.env.SEED_VERBOSE === "0",
sessionKey: (values["session-key"] as string | undefined) ?? process.env.SEED_FALLBACK_SESSION_KEY,
strictRoundRole: Boolean(values["strict-round-role"]) || process.env.SEED_STRICT_ROUND_ROLE === "1",
autoFillTimestamps: !values["no-auto-fill-timestamps"] && process.env.SEED_AUTO_FILL_TIMESTAMPS !== "0",
};
}
function printHelp(): void {
console.log(`
seed-v2 — feed historical conversations into memory-tencentdb via v2 API
Usage:
seed-v2 --input <fixture.json> [options]
Required:
-i, --input <file> Fixture JSON file (Format A: { sessions: [...] })
Options:
-e, --endpoint <url> Gateway URL (default: http://127.0.0.1:18420)
--api-key <key> Bearer key (default: standalone-e2e)
-s, --service-id <id> x-tdai-service-id (default: default)
-n, --every-n <n> Wait for busy=false every N rounds per session (default: 5)
--poll-ms <ms> Status poll interval (default: 500)
--stable-rounds <n> Consecutive idle polls before considered stable (default: 2)
--max-wait-ms <ms> Per-batch wait timeout, L1 only (default: 180000 = 3 min)
--final-max-wait-ms <ms> Final drain wait timeout, ALL layers L1+L2+L3 (default: 600000 = 10 min)
--no-final-wait Skip final cascade drain wait
--dry-run Print plan, no requests
--session-key <key> Fallback sessionKey for sessions missing one
--strict-round-role Each round must contain user + assistant
--no-auto-fill-timestamps Don't auto-fill missing timestamps (will error if all missing)
-q, --quiet Quiet mode (env SEED_VERBOSE=0)
-h, --help Show this help
Fixture JSON (Format A):
{
"sessions": [
{
"sessionKey": "user-001",
"sessionId": "user-001",
"conversations": [
[
{ "role": "user", "content": "..." },
{ "role": "assistant", "content": "..." }
]
]
}
]
}
Exit codes:
0 success
1 seed or wait failed
2 fixture / config error
`.trim());
}
// ============================
// HTTP helper
// ============================
interface ApiResponse<T = unknown> {
code: number;
message: string;
request_id: string;
data?: T;
}
interface ApiCallResult<T = unknown> {
body: ApiResponse<T>;
httpStatus: number;
durationMs: number;
}
class SeedClient {
private callSeq = 0;
constructor(private opts: CliOptions) {}
async post<T = unknown>(path: string, body: unknown): Promise<ApiCallResult<T>> {
this.callSeq++;
const url = `${this.opts.endpoint}/v2/${path}`;
const startedAt = Date.now();
const resp = await fetch(url, {
method: "POST",
headers: {
"Content-Type": "application/json",
"Authorization": `Bearer ${this.opts.apiKey}`,
"x-tdai-service-id": this.opts.serviceId,
},
body: JSON.stringify(body),
});
const durationMs = Date.now() - startedAt;
const text = await resp.text();
let parsed: ApiResponse<T>;
try {
parsed = JSON.parse(text) as ApiResponse<T>;
} catch {
throw new Error(`non-JSON response from /v2/${path}: ${resp.status} ${text.slice(0, 200)}`);
}
return { body: parsed, httpStatus: resp.status, durationMs };
}
}
// ============================
// Fixture loading + validation (delegates to input-validate.ts)
// ============================
/**
* Load fixture from disk → parse JSON → validate → normalize. Returns the
* NormalizedInput shape used downstream (sessions[].rounds[].messages[]),
* with timestamps either preserved, mixed (rejected), or auto-filled to
* globally-monotonic epoch ms (parity with v1 fillTimestamps).
*
* On validation failure throws SeedValidationError; caller should print and exit 2.
*/
function loadAndValidate(opts: CliOptions): NormalizedInput {
const abs = resolvePath(opts.input);
if (!existsSync(abs)) {
throw new SeedValidationError([{ stage: "file", message: `Fixture not found: ${abs}` }]);
}
const text = readFileSync(abs, "utf8").trim();
if (!text) {
throw new SeedValidationError([{ stage: "file", message: `Fixture is empty: ${abs}` }]);
}
let raw: unknown;
try {
raw = JSON.parse(text);
} catch (err) {
throw new SeedValidationError([{
stage: "file",
message: `JSON parse error: ${err instanceof Error ? err.message : String(err)}`,
}]);
}
return validateAndNormalizeRaw(raw, {
sessionKey: opts.sessionKey,
strictRoundRole: opts.strictRoundRole,
autoFillTimestamps: opts.autoFillTimestamps,
});
}
// ============================
// Status polling
// ============================
interface LayerStatus {
queued: number;
running: number;
queued_sessions: string[];
running_sessions: string[];
idle: boolean;
}
interface StatusData {
l1: LayerStatus;
l2: LayerStatus;
l3: LayerStatus;
}
async function pollStatus(client: SeedClient): Promise<StatusData> {
const r = await client.post<StatusData>("pipeline/status", {});
if (r.body.code !== 0 || !r.body.data) {
throw new Error(`/v2/pipeline/status failed: HTTP ${r.httpStatus} code=${r.body.code} msg=${r.body.message}`);
}
return r.body.data;
}
/**
* Wait until the requested layers are idle for `stableRounds` consecutive polls.
*
* @param waitMode "l1" → only wait for L1 to drain (mid-batch / per-session);
* L2/L3 cascade may still run in background.
* "all" → wait for L1+L2+L3 all idle (final drain after all
* rounds dispatched).
*
* On timeout: log a `[wait] done reason=timeout` line and return (do not
* throw). Data is already on disk and the gateway keeps cooking; callers
* decide whether the next batch can proceed.
*
* Logging:
* - `[wait] enter` once at the start (label, mode, budget).
* - `[poll]` per status snapshot, throttled: first poll, OR queue/running
* signature changed, OR 5s heartbeat. Long stable waits stay quiet,
* stalls keep showing progress.
* - `[wait] done` once at exit with reason=stable|timeout, elapsed, polls.
*/
async function waitForBusyFalse(
client: SeedClient,
opts: CliOptions,
label: string,
waitMode: "l1" | "all",
maxWaitMs: number,
): Promise<void> {
const startTime = Date.now();
let consecutiveIdle = 0;
let lastSnapshot: StatusData | null = null;
let lastLogAtMs = 0;
let pollCount = 0;
const HEARTBEAT_INTERVAL_MS = 5000;
log(`[seed-v2] [wait] enter label="${label}" mode=${waitMode} maxWaitMs=${maxWaitMs} stableRounds=${opts.stableRounds} pollMs=${opts.pollMs}`);
// Format a compact one-line snapshot for logging.
const fmt = (s: StatusData): string => {
const layer = (name: string, l: LayerStatus) => {
const extras: string[] = [];
if (l.running_sessions.length > 0) extras.push(`running=${JSON.stringify(l.running_sessions)}`);
if (l.queued_sessions.length > 0) extras.push(`queued=${JSON.stringify(l.queued_sessions)}`);
const tail = extras.length > 0 ? ` ${extras.join(" ")}` : "";
return `${name}(q=${l.queued} r=${l.running} idle=${l.idle}${tail})`;
};
return [layer("L1", s.l1), layer("L2", s.l2), layer("L3", s.l3)].join(" ");
};
// Stable signature for change-detection (only fields that matter for idle).
const sig = (s: StatusData): string =>
`${s.l1.queued}/${s.l1.running}|${s.l2.queued}/${s.l2.running}|${s.l3.queued}/${s.l3.running}`;
let lastSig = "";
while (true) {
const elapsed = Date.now() - startTime;
if (elapsed > maxWaitMs) {
const last = lastSnapshot ? fmt(lastSnapshot) : "no-snapshot";
warn(
`[seed-v2] [wait] done label="${label}" mode=${waitMode} elapsed=${(elapsed / 1000).toFixed(1)}s ` +
`polls=${pollCount} reason=timeout last=${last}`,
);
return;
}
const s = await pollStatus(client);
pollCount++;
lastSnapshot = s;
const isIdle = waitMode === "l1"
? s.l1.idle
: (s.l1.idle && s.l2.idle && s.l3.idle);
if (isIdle) consecutiveIdle++;
else consecutiveIdle = 0;
// Decide whether to log this poll.
const now = Date.now();
const currentSig = sig(s);
const changed = currentSig !== lastSig;
const heartbeat = now - lastLogAtMs >= HEARTBEAT_INTERVAL_MS;
const isFirstPoll = lastLogAtMs === 0;
const willBeStable = isIdle && consecutiveIdle >= opts.stableRounds;
if (!opts.quiet && (isFirstPoll || changed || heartbeat || willBeStable)) {
const mark = isIdle ? "idle" : "busy";
const stable = willBeStable ? " ✓ STABLE" : "";
log(
`[seed-v2] [poll] ${label} t+${(elapsed / 1000).toFixed(1)}s ${mark} ${fmt(s)} ` +
`consecutiveIdle=${consecutiveIdle}${stable}`,
);
lastLogAtMs = now;
lastSig = currentSig;
}
if (willBeStable) {
log(
`[seed-v2] [wait] done label="${label}" mode=${waitMode} elapsed=${(elapsed / 1000).toFixed(1)}s ` +
`polls=${pollCount} reason=stable`,
);
return;
}
await new Promise((r) => setTimeout(r, opts.pollMs));
}
}
// ============================
// Conversation/add per round
// ============================
interface AddRoundContext {
sessionIndex: number; // 1-based among non-empty sessions
totalSessions: number;
sessionKey: string;
sessionId: string;
roundIndex: number; // 1-based within this session
totalRoundsInSession: number;
cumRounds: number; // 1-based across whole fixture
totalRounds: number;
cumMessagesAfter: number; // includes this round (post-success)
totalMessages: number;
}
async function addRound(
client: SeedClient,
ctx: AddRoundContext,
messages: { role: string; content: string; timestamp: number }[],
quiet: boolean,
): Promise<number> {
// After validateAndNormalizeRaw + fillTimestamps, every message has a positive
// epoch-ms `timestamp` (or 0 if user explicitly opted out via
// --no-auto-fill-timestamps). v2 conversation/add schema requires ISO 8601 string.
const payload = messages.map((m) => ({
role: m.role,
content: m.content,
timestamp: new Date(m.timestamp || Date.now()).toISOString(),
}));
const roles = payload.map((p) => p.role).join(",");
const firstTs = payload[0]?.timestamp ?? "n/a";
const lastTs = payload[payload.length - 1]?.timestamp ?? "n/a";
if (!quiet) {
log(
`[seed-v2] [post] session ${ctx.sessionIndex}/${ctx.totalSessions} round ${ctx.roundIndex}/${ctx.totalRoundsInSession} ` +
`(${ctx.sessionKey}) → POST /v2/conversation/add session_id="${ctx.sessionId}" ` +
`msgs=${payload.length} roles=[${roles}] firstTs=${firstTs} lastTs=${lastTs}`,
);
for (let i = 0; i < payload.length; i++) {
const m = payload[i]!;
log(`[seed-v2] [post] ${m.role}[${i}]: "${preview(m.content)}"`);
}
}
const r = await client.post<{ accepted_ids: string[]; total_count: number }>("conversation/add", {
session_id: ctx.sessionId,
messages: payload,
});
const ok = r.body.code === 0 && r.body.data;
if (!ok) {
err(
`[seed-v2] [post] session ${ctx.sessionIndex}/${ctx.totalSessions} round ${ctx.roundIndex}/${ctx.totalRoundsInSession} ` +
`(${ctx.sessionKey}) ✗ HTTP ${r.httpStatus} code=${r.body.code} msg="${r.body.message}" in ${r.durationMs}ms`,
);
throw new Error(`/v2/conversation/add failed for ${ctx.sessionId}: HTTP ${r.httpStatus} code=${r.body.code} msg=${r.body.message}`);
}
const accepted = r.body.data!.accepted_ids.length;
if (!quiet) {
log(
`[seed-v2] [post] session ${ctx.sessionIndex}/${ctx.totalSessions} round ${ctx.roundIndex}/${ctx.totalRoundsInSession} ` +
`(${ctx.sessionKey}) ✓ HTTP ${r.httpStatus} code=0 accepted=${accepted} in ${r.durationMs}ms ` +
`(cum: rounds=${ctx.cumRounds}/${ctx.totalRounds} msgs=${ctx.cumMessagesAfter}/${ctx.totalMessages})`,
);
}
return accepted;
}
// ============================
// Main
// ============================
async function main(): Promise<void> {
const opts = parseCli(process.argv.slice(2));
const client = new SeedClient(opts);
log(
`[seed-v2] endpoint=${opts.endpoint} serviceId=${opts.serviceId} apiKey=${opts.apiKey ? "***" : "(none)"}`,
);
log(
`[seed-v2] cadence: everyN=${opts.everyN} stableRounds=${opts.stableRounds} pollMs=${opts.pollMs} ` +
`maxWaitMs=${opts.maxWaitMs}(L1-only) finalMaxWaitMs=${opts.finalMaxWaitMs}(all-layers) finalWait=${opts.finalWait}`,
);
// Load + validate + normalize fixture (Layer 2-6 of v1 seed: format detect /
// session+round+message validation / timestamp consistency check / auto-fill).
let normalized: NormalizedInput;
try {
normalized = loadAndValidate(opts);
} catch (e) {
if (e instanceof SeedValidationError) {
err(`[seed-v2] ${e.message}`);
} else {
err(`[seed-v2] fixture load failed: ${e instanceof Error ? e.message : String(e)}`);
}
process.exit(2);
}
const sessions = normalized.sessions;
const { totalRounds, totalMessages, hasTimestamps } = normalized;
log(
`[seed-v2] fixture=${opts.input} sessions=${sessions.length} rounds=${totalRounds} ` +
`messages=${totalMessages} hasTimestamps=${hasTimestamps}` +
(!hasTimestamps && opts.autoFillTimestamps ? " (auto-filled)" : "") +
` autoFill=${opts.autoFillTimestamps} strictRoundRole=${opts.strictRoundRole}`,
);
// Session manifest preview (first 3 + count).
const previewSessions = sessions.slice(0, 3)
.map((s) => `${s.sessionKey}(${s.rounds.length}r)`).join(" ");
const moreSessions = sessions.length > 3 ? ` ... (+${sessions.length - 3} more)` : "";
log(`[seed-v2] sessions: ${previewSessions}${moreSessions}`);
if (opts.dryRun) {
log("[seed-v2] DRY RUN — exiting without making any requests");
for (const s of sessions) {
log(`[seed-v2] would add: sessionKey=${s.sessionKey} sessionId=${s.sessionId} rounds=${s.rounds.length}`);
}
process.exit(0);
}
// Pre-flight: status接口 must respond
let preflight: StatusData;
try {
preflight = await pollStatus(client);
} catch (e) {
err(`[seed-v2] pre-flight /v2/pipeline/status failed: ${(e as Error).message}`);
err(`[seed-v2] gateway up at ${opts.endpoint}? deployMode=standalone? stateBackend configured?`);
process.exit(2);
}
log(
`[seed-v2] pre-flight ok l1.idle=${preflight.l1.idle} ` +
`(l1 q=${preflight.l1.queued} r=${preflight.l1.running}, ` +
`l2 q=${preflight.l2.queued} r=${preflight.l2.running}, ` +
`l3 q=${preflight.l3.queued} r=${preflight.l3.running})`,
);
// Per-session, per-round seeding (timestamps now come from normalized input,
// not synthesized per-call — global monotonic ordering already guaranteed
// by validateAndNormalizeRaw + fillTimestamps).
const startMs = Date.now();
let roundsDone = 0;
let messagesDone = 0;
for (let si = 0; si < sessions.length; si++) {
const session = sessions[si]!;
const sessionKey = session.sessionKey;
const sessionId = session.sessionId; // always present after normalization
if (session.rounds.length === 0) {
warn(`[seed-v2] session ${si + 1}/${sessions.length} key="${sessionKey}" has no rounds, skipping`);
continue;
}
const sessionStartMs = Date.now();
let sessionMsgs = 0;
log(
`[seed-v2] session ${si + 1}/${sessions.length} START key="${sessionKey}" id="${sessionId}" ` +
`rounds=${session.rounds.length}`,
);
for (let ri = 0; ri < session.rounds.length; ri++) {
const round = session.rounds[ri]!;
if (round.messages.length === 0) continue;
const accepted = await addRound(
client,
{
sessionIndex: si + 1,
totalSessions: sessions.length,
sessionKey,
sessionId,
roundIndex: ri + 1,
totalRoundsInSession: session.rounds.length,
cumRounds: roundsDone + 1,
totalRounds,
cumMessagesAfter: messagesDone + round.messages.length,
totalMessages,
},
round.messages,
opts.quiet,
);
roundsDone++;
messagesDone += accepted;
sessionMsgs += accepted;
// Old seed-runtime parity: every N rounds in this session, wait for L1 idle.
const roundInSession = ri + 1;
if (roundInSession % opts.everyN === 0) {
await waitForBusyFalse(client, opts, `every-${opts.everyN}@${sessionKey}#${roundInSession}`, "l1", opts.maxWaitMs);
}
}
// Per-session tail wait (matches old seed-runtime per-session waitForL1Idle).
await waitForBusyFalse(client, opts, `session-tail@${sessionKey}`, "l1", opts.maxWaitMs);
const sessionDur = (Date.now() - sessionStartMs) / 1000;
log(
`[seed-v2] session ${si + 1}/${sessions.length} END key="${sessionKey}" ` +
`rounds=${session.rounds.length} msgs=${sessionMsgs} duration=${sessionDur.toFixed(1)}s`,
);
}
// Final wait: drain any cascade L2/L3 work (best-effort).
if (opts.finalWait) {
log(
`[seed-v2] all rounds dispatched, final drain — waiting for L1+L2+L3 all idle ` +
`(max ${(opts.finalMaxWaitMs / 1000).toFixed(0)}s)...`,
);
await waitForBusyFalse(client, opts, "final", "all", opts.finalMaxWaitMs);
} else {
log(`[seed-v2] all rounds dispatched, --no-final-wait set, exiting immediately`);
}
const durationMs = Date.now() - startMs;
log(
`[seed-v2] done sessions=${sessions.length} rounds=${roundsDone}/${totalRounds} ` +
`msgs=${messagesDone}/${totalMessages} duration=${(durationMs / 1000).toFixed(1)}s`,
);
// Final status snapshot for downstream verification.
const final = await pollStatus(client);
const allIdle = final.l1.idle && final.l2.idle && final.l3.idle;
log(
`[seed-v2] final status: l1.idle=${final.l1.idle} l2.idle=${final.l2.idle} l3.idle=${final.l3.idle} ` +
`(l1 q=${final.l1.queued} r=${final.l1.running}, ` +
`l2 q=${final.l2.queued} r=${final.l2.running}, ` +
`l3 q=${final.l3.queued} r=${final.l3.running})` +
(allIdle ? " ✓ all clean" : " ⚠ residual work — gateway will keep cooking after exit"),
);
}
main().catch((e) => {
err(`[seed-v2] FATAL: ${e instanceof Error ? e.stack : String(e)}`);
process.exit(1);
});
scripts/seed-v2/tsconfig.json
+19
View File
@@ -0,0 +1,19 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "Node16",
"moduleResolution": "Node16",
"outDir": "./dist",
"rootDir": ".",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"types": ["node"],
"declaration": false,
"sourceMap": false
},
"include": ["seed-v2.ts", "input-types.ts", "input-validate.ts"],
"exclude": ["dist", "node_modules"]
}
scripts/setup-offload.sh
Executable
+340
View File
@@ -0,0 +1,340 @@
#!/usr/bin/env bash
# ═══════════════════════════════════════════════════════════════════
# setup-offload.sh — Offload 功能一键开启/关闭
# ═══════════════════════════════════════════════════════════════════
#
# 用法:
# bash setup-offload.sh --enable --user-id <userId> --backend-url <url> [--backend-api-key <key>]
# bash setup-offload.sh --disable
# bash setup-offload.sh --status
#
# 开启流程:
# 1. 前置检查（openclaw.json 存在、openclaw 已安装）
# 2. Patch 校验 & 执行（after_tool_call messages 注入）— 失败则终止
# 3. 设置 plugins.slots.contextEngine
# 4. 设置 offload.enabled + backendUrl + userId [+ backendApiKey]
# 5. 设置 compaction.mode = safeguard
#
# 关闭流程:
# 1. 设置 offload.enabled = false
# 2. 删除 plugins.slots.contextEngine（清理 slot 占用）
# ═══════════════════════════════════════════════════════════════════
set -euo pipefail
# ── 颜色 ──
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
info() { echo -e "${CYAN}[INFO]${NC} $*"; }
ok() { echo -e "${GREEN}[OK]${NC} $*"; }
warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
fail() { echo -e "${RED}[FAIL]${NC} $*" >&2; exit 1; }
# ── 常量 ──
OPENCLAW_JSON="${HOME}/.openclaw/openclaw.json"
PLUGIN_ID="memory-tencentdb"
CONTEXT_ENGINE_ID="memory-tencentdb"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PATCH_SCRIPT="${SCRIPT_DIR}/openclaw-after-tool-call-messages.patch.sh"
# ── 参数解析 ──
MODE=""
USER_ID=""
BACKEND_URL=""
BACKEND_API_KEY=""
while [[ $# -gt 0 ]]; do
case "$1" in
--enable) MODE="enable"; shift ;;
--disable) MODE="disable"; shift ;;
--status) MODE="status"; shift ;;
--user-id) USER_ID="$2"; shift 2 ;;
--backend-url) BACKEND_URL="$2"; shift 2 ;;
--backend-api-key) BACKEND_API_KEY="$2"; shift 2 ;;
-h|--help)
echo "用法:"
echo " bash setup-offload.sh --enable --user-id <userId> --backend-url <url> [--backend-api-key <key>]"
echo " bash setup-offload.sh --disable"
echo " bash setup-offload.sh --status"
echo ""
echo "参数:"
echo " --user-id (必填) 用户 ID"
echo " --backend-url (必填) offload 后端地址，如 http://1.2.3.4:8080"
echo " --backend-api-key (可选) 后端 API 认证 token"
exit 0
;;
*) fail "未知参数: $1" ;;
esac
done
[[ -z "$MODE" ]] && fail "请指定模式: --enable / --disable / --status"
# ═══════════════════════════════════════════════════════════════════
# 公共函数
# ═══════════════════════════════════════════════════════════════════
check_openclaw_json() {
if [[ ! -f "$OPENCLAW_JSON" ]]; then
fail "openclaw.json 不存在: $OPENCLAW_JSON"
fi
# 验证 JSON 格式
python3 -c "import json; json.load(open('$OPENCLAW_JSON'))" 2>/dev/null \
|| fail "openclaw.json 格式错误"
}
backup_config() {
local bak="${OPENCLAW_JSON}.bak.$(date +%Y%m%d_%H%M%S)"
cp "$OPENCLAW_JSON" "$bak"
info "配置已备份: $bak"
}
# ═══════════════════════════════════════════════════════════════════
# --status: 显示当前配置状态
# ═══════════════════════════════════════════════════════════════════
show_status() {
check_openclaw_json
python3 -c "
import json
with open('$OPENCLAW_JSON') as f:
cfg = json.load(f)
# Context Engine Slot
slot = cfg.get('plugins', {}).get('slots', {}).get('contextEngine', '(未设置)')
print(f' Context Engine Slot: {slot}')
# Offload config
offload = cfg.get('plugins', {}).get('entries', {}).get('$PLUGIN_ID', {}).get('config', {}).get('offload', {})
enabled = offload.get('enabled', False)
backend = offload.get('backendUrl', '(未设置)')
user_id = offload.get('userId', '(未设置)')
api_key = offload.get('backendApiKey', '')
timeout = offload.get('backendTimeoutMs', '(默认)')
mild = offload.get('mildOffloadRatio', '(默认 0.5)')
agg = offload.get('aggressiveCompressRatio', '(默认 0.85)')
status_icon = '✅ 已开启' if enabled else '❌ 已关闭'
api_key_display = f'{api_key[:8]}...' if api_key and len(api_key) > 8 else (api_key or '(未设置)')
print(f' Offload 状态: {status_icon}')
print(f' Backend URL: {backend}')
print(f' Backend Key: {api_key_display}')
print(f' User ID: {user_id}')
print(f' Timeout: {timeout}ms')
print(f' Mild Ratio: {mild}')
print(f' Aggressive: {agg}')
# Compaction mode
compaction = cfg.get('agents', {}).get('defaults', {}).get('compaction', {}).get('mode', '(未设置)')
print(f' Compaction: {compaction}')
"
}
# ═══════════════════════════════════════════════════════════════════
# --enable: 开启 offload
# ═══════════════════════════════════════════════════════════════════
enable_offload() {
# 参数校验
[[ -z "$USER_ID" ]] && fail "缺少 --user-id 参数"
[[ -z "$BACKEND_URL" ]] && fail "缺少 --backend-url 参数"
# URL 格式基本校验
if [[ ! "$BACKEND_URL" =~ ^https?:// ]]; then
fail "backendUrl 格式错误，应以 http:// 或 https:// 开头: $BACKEND_URL"
fi
check_openclaw_json
backup_config
echo ""
info "${BOLD}[1/4] Patch 校验${NC}"
# ── Step 1: 调用 patch 脚本（幂等，已 patch 过会跳过） ──
# patch 脚本自带精确的幂等检测，通过退出码判断结果：
# 0 = 成功（新 patch 或已跳过）
# 1 = 失败（无法 patch）
if [[ -f "$PATCH_SCRIPT" ]]; then
info "执行 patch 脚本..."
local patch_exit=0
local patch_output
patch_output=$(bash "$PATCH_SCRIPT" 2>&1) || patch_exit=$?
# 显示 patch 脚本输出（缩进）
while IFS= read -r line; do
echo " $line"
done <<< "$patch_output"
if [[ $patch_exit -eq 0 ]]; then
ok "Patch 校验通过"
else
echo ""
echo -e "${RED}═══════════════════════════════════════════════════${NC}"
echo -e "${RED} ❌ Patch 失败（退出码: $patch_exit）${NC}"
echo -e "${RED}═══════════════════════════════════════════════════${NC}"
echo ""
echo -e " ${RED}after_tool_call hook 无法获取 session messages，${NC}"
echo -e " ${RED}offload L1/L3 压缩将无法正常工作。${NC}"
echo ""
echo -e " ${CYAN}排查步骤:${NC}"
echo -e " 1. DEBUG=1 bash $PATCH_SCRIPT"
echo -e " 2. 检查 openclaw 版本是否兼容"
echo ""
exit 2
fi
else
echo -e "${RED}[FAIL]${NC} Patch 脚本不存在: $PATCH_SCRIPT" >&2
echo -e " ${RED}offload 功能依赖此 patch，终止开启流程。${NC}" >&2
exit 2
fi
# ── Step 2: 设置 context engine slot ──
echo ""
info "${BOLD}[2/4] 设置 Context Engine Slot${NC}"
python3 -c "
import json
with open('$OPENCLAW_JSON') as f:
cfg = json.load(f)
# 确保 plugins.slots 存在
cfg.setdefault('plugins', {}).setdefault('slots', {})
cfg['plugins']['slots']['contextEngine'] = '$CONTEXT_ENGINE_ID'
print(' slots.contextEngine = $CONTEXT_ENGINE_ID')
with open('$OPENCLAW_JSON', 'w') as f:
json.dump(cfg, f, indent=2, ensure_ascii=False)
f.write('\n')
"
ok "Context Engine Slot 已设置"
# ── Step 3: 设置 offload 配置 ──
echo ""
info "${BOLD}[3/4] 设置 Offload 配置${NC}"
python3 -c "
import json
with open('$OPENCLAW_JSON') as f:
cfg = json.load(f)
# 确保路径存在
entry = cfg.setdefault('plugins', {}).setdefault('entries', {}).setdefault('$PLUGIN_ID', {})
config = entry.setdefault('config', {})
offload = config.setdefault('offload', {})
# 设置必要配置
offload['enabled'] = True
offload['backendUrl'] = '$BACKEND_URL'
offload['userId'] = '$USER_ID'
offload.setdefault('backendTimeoutMs', 120000)
api_key = '$BACKEND_API_KEY'
if api_key:
offload['backendApiKey'] = api_key
print(f' offload.backendApiKey = {api_key[:8]}...' if len(api_key) > 8 else f' offload.backendApiKey = {api_key}')
elif 'backendApiKey' in offload:
del offload['backendApiKey']
print(' offload.enabled = true')
print(' offload.backendUrl = $BACKEND_URL')
print(' offload.userId = $USER_ID')
print(f' offload.backendTimeoutMs = {offload[\"backendTimeoutMs\"]}')
with open('$OPENCLAW_JSON', 'w') as f:
json.dump(cfg, f, indent=2, ensure_ascii=False)
f.write('\n')
"
ok "Offload 配置已设置"
# ── Step 4: 设置 compaction mode ──
echo ""
info "${BOLD}[4/4] 设置 Compaction Mode${NC}"
python3 -c "
import json
with open('$OPENCLAW_JSON') as f:
cfg = json.load(f)
defaults = cfg.setdefault('agents', {}).setdefault('defaults', {})
compaction = defaults.setdefault('compaction', {})
old_mode = compaction.get('mode', '(未设置)')
compaction['mode'] = 'safeguard'
print(f' compaction.mode: {old_mode} → safeguard')
with open('$OPENCLAW_JSON', 'w') as f:
json.dump(cfg, f, indent=2, ensure_ascii=False)
f.write('\n')
"
ok "Compaction mode 已设置为 safeguard"
# ── 完成 ──
echo ""
echo -e "${GREEN}═══════════════════════════════════════════════════${NC}"
echo -e "${GREEN} ✅ Offload 已开启${NC}"
echo -e "${GREEN}═══════════════════════════════════════════════════${NC}"
echo ""
show_status
echo ""
echo -e " ${CYAN}提示: 需要重启 gateway 才能生效${NC}"
echo -e " ${CYAN} bash install-plugin.sh --restart${NC}"
}
# ═══════════════════════════════════════════════════════════════════
# --disable: 关闭 offload
# ═══════════════════════════════════════════════════════════════════
disable_offload() {
check_openclaw_json
backup_config
python3 -c "
import json
with open('$OPENCLAW_JSON') as f:
cfg = json.load(f)
# 关闭 offload.enabled（用 setdefault 确保路径存在，修改能回写到 cfg）
entry = cfg.setdefault('plugins', {}).setdefault('entries', {}).setdefault('$PLUGIN_ID', {})
config = entry.setdefault('config', {})
offload = config.setdefault('offload', {})
offload['enabled'] = False
print(' offload.enabled = false')
# 移除 contextEngine slot
plugins = cfg.get('plugins', {})
slots = plugins.get('slots', {})
if 'contextEngine' in slots:
del slots['contextEngine']
print(' slots.contextEngine → 已删除')
# 如果 slots 变空了，也一并移除 slots 键
if not slots and 'slots' in plugins:
del plugins['slots']
print(' plugins.slots → 已清理（空对象）')
else:
print(' slots.contextEngine → 无需删除（不存在）')
with open('$OPENCLAW_JSON', 'w') as f:
json.dump(cfg, f, indent=2, ensure_ascii=False)
f.write('\n')
"
echo ""
echo -e "${YELLOW}═══════════════════════════════════════════════════${NC}"
echo -e "${YELLOW} ❌ Offload 已关闭${NC}"
echo -e "${YELLOW}═══════════════════════════════════════════════════${NC}"
echo ""
echo -e " ${CYAN}提示: 需要重启 gateway 才能生效${NC}"
echo -e " ${CYAN} bash install-plugin.sh --restart${NC}"
}
# ═══════════════════════════════════════════════════════════════════
# 主入口
# ═══════════════════════════════════════════════════════════════════
case "$MODE" in
enable) enable_offload ;;
disable) disable_offload ;;
status)
echo ""
info "${BOLD}Offload 配置状态${NC}"
show_status
;;
esac
scripts/stress-test-v2.sh
+325
View File
@@ -0,0 +1,325 @@
#!/usr/bin/env bash
# ============================================================
# V2 API 全接口压测脚本
# 测试所有12个 v2 端点的可用性、并发处理、Redis 状态正确性
# ============================================================
set -euo pipefail
HOST="${1:-http://localhost:8420}"
API_KEY="${2:-b9WmawnJFpb9vn0XWKDKSxF5Eaf5SeXdIHaRpShmSbgg}"
SERVICE_ID="${3:-mem-j4wjesud}"
CONCURRENCY="${4:-10}"
ROUNDS="${5:-5}"
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
PASS=0
FAIL=0
ERRORS=""
log_ok() { echo -e "${GREEN}[PASS]${NC} $1"; PASS=$((PASS+1)); }
log_fail() { echo -e "${RED}[FAIL]${NC} $1"; FAIL=$((FAIL+1)); ERRORS="${ERRORS}\n - $1"; }
log_info() { echo -e "${YELLOW}[INFO]${NC} $1"; }
call_api() {
local endpoint="$1"
local body="$2"
local desc="$3"
local resp
resp=$(curl -s -w "\n%{http_code}" -X POST "${HOST}${endpoint}" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${API_KEY}" \
-H "x-tdai-service-id: ${SERVICE_ID}" \
-d "${body}" 2>/dev/null)
local http_code
http_code=$(echo "$resp" | tail -1)
local body_resp
body_resp=$(echo "$resp" | sed '$d')
local code
code=$(echo "$body_resp" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('code',''))" 2>/dev/null || echo "parse_error")
if [[ "$http_code" == "200" && "$code" == "0" ]]; then
log_ok "$desc (HTTP=$http_code, code=$code)"
echo "$body_resp"
return 0
else
log_fail "$desc (HTTP=$http_code, code=$code, body=${body_resp:0:200})"
return 1
fi
}
# ============================================================
echo "=========================================="
echo " V2 API 全接口压测"
echo " Host: $HOST"
echo " ServiceID: $SERVICE_ID"
echo " Concurrency: $CONCURRENCY"
echo " Rounds: $ROUNDS"
echo "=========================================="
# 0. Health check
log_info "Step 0: Health check"
HEALTH=$(curl -s "${HOST}/health" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['status'])" 2>/dev/null)
if [[ "$HEALTH" == "ok" ]]; then
log_ok "Health check"
else
log_fail "Health check (status=$HEALTH)"
exit 1
fi
# ============================================================
# 1. Conversation Add (L0 写入) - 并发压测
# ============================================================
log_info "Step 1: POST /v2/conversation/add — $CONCURRENCY concurrent × $ROUNDS rounds"
TS=$(date +%s)
for round in $(seq 1 $ROUNDS); do
pids=()
for i in $(seq 1 $CONCURRENCY); do
SESSION="stress-${TS}-r${round}-c${i}"
BODY=$(cat <<EOF
{
"instance_id": "${SERVICE_ID}",
"session_id": "${SESSION}",
"messages": [
{"role": "user", "content": "压测消息 round=${round} thread=${i} 我喜欢在周末去公园跑步"},
{"role": "assistant", "content": "跑步是很好的运动习惯！你通常跑多久？"},
{"role": "user", "content": "一般跑30分钟左右，然后做拉伸"},
{"role": "assistant", "content": "30分钟加拉伸是很健康的安排"}
]
}
EOF
)
curl -s -o /dev/null -w "" -X POST "${HOST}/v2/conversation/add" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${API_KEY}" \
-H "x-tdai-service-id: ${SERVICE_ID}" \
-d "${BODY}" &
pids+=($!)
done
# Wait for all concurrent requests
fail_count=0
for pid in "${pids[@]}"; do
wait "$pid" || fail_count=$((fail_count+1))
done
if [[ $fail_count -eq 0 ]]; then
log_ok "Round $round: $CONCURRENCY concurrent conversation/add OK"
else
log_fail "Round $round: $fail_count/$CONCURRENCY conversation/add failed"
fi
done
# ============================================================
# 2. Conversation Query (L0 查询)
# ============================================================
log_info "Step 2: POST /v2/conversation/query"
SESSION_Q="stress-${TS}-r1-c1"
call_api "/v2/conversation/query" \
"{\"instance_id\":\"${SERVICE_ID}\",\"session_id\":\"${SESSION_Q}\",\"limit\":10}" \
"conversation/query session=${SESSION_Q}" > /dev/null || true
# ============================================================
# 3. Conversation Search (L0 搜索)
# ============================================================
log_info "Step 3: POST /v2/conversation/search"
call_api "/v2/conversation/search" \
"{\"instance_id\":\"${SERVICE_ID}\",\"query\":\"跑步\",\"limit\":5}" \
"conversation/search query=跑步" > /dev/null || true
# ============================================================
# 4. Conversation Delete (L0 删除)
# ============================================================
log_info "Step 4: POST /v2/conversation/delete"
call_api "/v2/conversation/delete" \
"{\"instance_id\":\"${SERVICE_ID}\",\"session_id\":\"stress-${TS}-r${ROUNDS}-c${CONCURRENCY}\",\"ids\":[\"nonexistent-id\"]}" \
"conversation/delete (nonexistent OK)" > /dev/null || true
# ============================================================
# 5. Atomic Add (L1 写入)
# ============================================================
log_info "Step 5: POST /v2/atomic/add — concurrent"
pids=()
for i in $(seq 1 $CONCURRENCY); do
BODY=$(cat <<EOF
{
"instance_id": "${SERVICE_ID}",
"session_id": "stress-atomic-${TS}",
"memories": [
{"text": "用户喜欢在周末跑步 (thread=$i)", "type": "episodic", "priority": 0.8},
{"text": "用户每次跑30分钟 (thread=$i)", "type": "episodic", "priority": 0.6}
]
}
EOF
)
curl -s -o /dev/null -w "" -X POST "${HOST}/v2/atomic/add" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${API_KEY}" \
-H "x-tdai-service-id: ${SERVICE_ID}" \
-d "${BODY}" &
pids+=($!)
done
fail_count=0
for pid in "${pids[@]}"; do
wait "$pid" || fail_count=$((fail_count+1))
done
if [[ $fail_count -eq 0 ]]; then
log_ok "atomic/add: $CONCURRENCY concurrent OK"
else
log_fail "atomic/add: $fail_count/$CONCURRENCY failed"
fi
# ============================================================
# 6. Atomic Query (L1 查询)
# ============================================================
log_info "Step 6: POST /v2/atomic/query"
call_api "/v2/atomic/query" \
"{\"instance_id\":\"${SERVICE_ID}\",\"session_id\":\"stress-atomic-${TS}\",\"limit\":20}" \
"atomic/query" > /dev/null || true
# ============================================================
# 7. Atomic Search (L1 搜索)
# ============================================================
log_info "Step 7: POST /v2/atomic/search"
call_api "/v2/atomic/search" \
"{\"instance_id\":\"${SERVICE_ID}\",\"query\":\"跑步\",\"limit\":5}" \
"atomic/search query=跑步" > /dev/null || true
# ============================================================
# 8. Atomic Delete (L1 删除)
# ============================================================
log_info "Step 8: POST /v2/atomic/delete"
call_api "/v2/atomic/delete" \
"{\"instance_id\":\"${SERVICE_ID}\",\"ids\":[\"nonexistent-l1-id\"]}" \
"atomic/delete (nonexistent OK)" > /dev/null || true
# ============================================================
# 9. Scenario List (L2 列表)
# ============================================================
log_info "Step 9: POST /v2/scenario/ls"
call_api "/v2/scenario/ls" \
"{\"instance_id\":\"${SERVICE_ID}\"}" \
"scenario/ls" > /dev/null || true
# ============================================================
# 10. Scenario Read (L2 读取)
# ============================================================
log_info "Step 10: POST /v2/scenario/read"
call_api "/v2/scenario/read" \
"{\"instance_id\":\"${SERVICE_ID}\",\"path\":\"failover-test.md\"}" \
"scenario/read path=failover-test.md" > /dev/null || true
# ============================================================
# 11. Scenario Write (L2 写入)
# ============================================================
log_info "Step 11: POST /v2/scenario/write"
call_api "/v2/scenario/write" \
"{\"instance_id\":\"${SERVICE_ID}\",\"path\":\"stress-test-${TS}.md\",\"content\":\"# 压测场景\\n\\n这是压测写入的场景文件。\\n\\n## 用户偏好\\n- 周末跑步\"}" \
"scenario/write" > /dev/null || true
# ============================================================
# 12. Scenario Delete (L2 删除)
# ============================================================
log_info "Step 12: POST /v2/scenario/rm"
call_api "/v2/scenario/rm" \
"{\"instance_id\":\"${SERVICE_ID}\",\"path\":\"stress-test-${TS}.md\"}" \
"scenario/rm (cleanup)" > /dev/null || true
# ============================================================
# 13. Persona Read (L3 读取)
# ============================================================
log_info "Step 13: POST /v2/persona/read"
call_api "/v2/persona/read" \
"{\"instance_id\":\"${SERVICE_ID}\"}" \
"persona/read" > /dev/null || true
# ============================================================
# 14. Persona Write (L3 写入)
# ============================================================
log_info "Step 14: POST /v2/persona/write"
call_api "/v2/persona/write" \
"{\"instance_id\":\"${SERVICE_ID}\",\"content\":\"# 用户画像\\n\\n## 基本信息\\n- 喜欢跑步\\n- 周末锻炼\\n\"}" \
"persona/write" > /dev/null || true
# ============================================================
# 15. 高并发混合读写压测
# ============================================================
log_info "Step 15: Mixed concurrent read/write stress ($((CONCURRENCY*2)) requests)"
pids=()
for i in $(seq 1 $CONCURRENCY); do
# Write
curl -s -o /dev/null -X POST "${HOST}/v2/conversation/add" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${API_KEY}" \
-H "x-tdai-service-id: ${SERVICE_ID}" \
-d "{\"instance_id\":\"${SERVICE_ID}\",\"session_id\":\"mix-${TS}-${i}\",\"messages\":[{\"role\":\"user\",\"content\":\"混合压测消息 $i\"}]}" &
pids+=($!)
# Read
curl -s -o /dev/null -X POST "${HOST}/v2/conversation/query" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${API_KEY}" \
-H "x-tdai-service-id: ${SERVICE_ID}" \
-d "{\"instance_id\":\"${SERVICE_ID}\",\"session_id\":\"mix-${TS}-${i}\",\"limit\":5}" &
pids+=($!)
done
fail_count=0
for pid in "${pids[@]}"; do
wait "$pid" || fail_count=$((fail_count+1))
done
if [[ $fail_count -eq 0 ]]; then
log_ok "Mixed stress: $((CONCURRENCY*2)) concurrent OK"
else
log_fail "Mixed stress: $fail_count/$((CONCURRENCY*2)) failed"
fi
# ============================================================
# 16. Redis 状态校验
# ============================================================
log_info "Step 16: Health check + Redis state validation"
HEALTH_RESP=$(curl -s "${HOST}/health")
WORKER_TASKS=$(echo "$HEALTH_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); s=d['services']; print(f\"consumed={s['pipelineWorker']['tasksConsumed']} completed={s['pipelineWorker']['tasksCompleted']} failed={s['pipelineWorker']['tasksFailed']} deadLetter={s['pipelineWorker']['tasksDeadLettered']}\")" 2>/dev/null)
SCANNER=$(echo "$HEALTH_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); s=d['services']['timerScanner']; print(f\"scans={s['scansCompleted']} enqueued={s['tasksEnqueued']} errors={s['scanErrors']}\")" 2>/dev/null)
STATE=$(echo "$HEALTH_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['services']['stateBackend'])" 2>/dev/null)
echo " Worker: $WORKER_TASKS"
echo " Scanner: $SCANNER"
echo " Redis: $STATE"
if [[ "$STATE" == "connected" ]]; then
log_ok "Redis state: connected"
else
log_fail "Redis state: $STATE"
fi
TASK_FAILED=$(echo "$HEALTH_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['services']['pipelineWorker']['tasksFailed'])" 2>/dev/null)
if [[ "$TASK_FAILED" == "0" ]]; then
log_ok "No pipeline task failures"
else
log_fail "Pipeline task failures: $TASK_FAILED"
fi
# ============================================================
# Summary
# ============================================================
echo ""
echo "=========================================="
echo " 压测结果汇总"
echo "=========================================="
echo -e " ${GREEN}PASS: $PASS${NC}"
echo -e " ${RED}FAIL: $FAIL${NC}"
if [[ $FAIL -gt 0 ]]; then
echo -e " 失败详情:${ERRORS}"
fi
echo "=========================================="
TOTAL_MSGS=$((CONCURRENCY * ROUNDS * 4 + CONCURRENCY * 1))
echo " 总写入消息数: ~$TOTAL_MSGS (L0) + $((CONCURRENCY*2)) (L1)"
echo "=========================================="
exit $FAIL
scripts/sync-hermes-llm-to-tdai.sh
Executable
+233
View File
@@ -0,0 +1,233 @@
#!/usr/bin/env bash
#
# sync-hermes-llm-to-tdai.sh — 读取系统 Hermes 模型配置，同步到 TDAI Gateway
#
# 前置条件：
# 1. memory-tencentdb-ctl.sh 存在
# 2. ~/.hermes/config.yaml 存在且包含模型配置
#
# 用法：
# bash sync-hermes-llm-to-tdai.sh [--restart] [--dry-run] [--hermes]
#
# 参数语义：
# --restart 仅当 CTL 脚本的 `status` 显示 Gateway 当前为 RUNNING 时，
# 才透传给 `memory-tencentdb-ctl.sh config llm` 执行重启；
# 若当前未运行，则本脚本只同步配置，不拉起新进程。
# --hermes 以 hermes 集成模式调用 CTL（影响 status 判定与配置落点）
# --standalone 以 standalone 模式调用 CTL
# --dry-run 透传给 CTL，预演写配置但不真正落盘
#
# 可选环境变量：
# HERMES_HOME hermes 主目录（默认 ~/.hermes）
# MEMORY_TENCENTDB_ROOT TDAI 统一根目录（默认 ~/.memory-tencentdb）
#
set -euo pipefail
SCRIPT_NAME="sync-hermes-llm-to-tdai"
USER_HOME="${HOME:-$(eval echo "~$(whoami)")}"
# ============================================================
# 路径
# ============================================================
HERMES_HOME="${HERMES_HOME:-$USER_HOME/.hermes}"
HERMES_CONFIG="$HERMES_HOME/config.yaml"
MEMORY_TENCENTDB_ROOT="${MEMORY_TENCENTDB_ROOT:-$USER_HOME/.memory-tencentdb}"
TDAI_INSTALL_DIR="${TDAI_INSTALL_DIR:-$MEMORY_TENCENTDB_ROOT/tdai-memory-openclaw-plugin}"
CTL_SCRIPT="$TDAI_INSTALL_DIR/scripts/memory-tencentdb-ctl.sh"
# ============================================================
# 通用 helpers
# ============================================================
log() { printf '[%s] %s\n' "$SCRIPT_NAME" "$*"; }
warn() { printf '[%s:warn] %s\n' "$SCRIPT_NAME" "$*" >&2; }
die() { printf '[%s:error] %s\n' "$SCRIPT_NAME" "$*" >&2; exit "${2:-1}"; }
should_restart_gateway() {
local ctl_mode_args=()
case "${SYNC_MODE:-standalone}" in
hermes) ctl_mode_args+=("--hermes") ;;
*) ctl_mode_args+=("--standalone") ;;
esac
local status_out
status_out="$(bash "$CTL_SCRIPT" "${ctl_mode_args[@]}" status 2>&1 || true)"
if printf '%s\n' "$status_out" | grep -Eq 'state[[:space:]]*:[[:space:]]*RUNNING'; then
log "通过 ctl status 检测到 Gateway 正在运行，允许执行 --restart"
return 0
fi
warn "ctl status 未显示 Gateway 处于 RUNNING；本次仅同步配置，不执行 --restart，也不会拉起新进程。"
return 1
}
# ============================================================
# Step 1: 检查 memory-tencentdb-ctl.sh 是否存在
# ============================================================
if [[ ! -f "$CTL_SCRIPT" ]]; then
# 也尝试旧路径
_LEGACY_CTL="$USER_HOME/tdai-memory-openclaw-plugin/scripts/memory-tencentdb-ctl.sh"
if [[ -f "$_LEGACY_CTL" ]]; then
CTL_SCRIPT="$_LEGACY_CTL"
warn "使用旧路径: $CTL_SCRIPT"
else
die "memory-tencentdb-ctl.sh 不存在，TDAI 未安装。退出。" 1
fi
fi
log "找到 ctl 脚本: $CTL_SCRIPT"
# ============================================================
# Step 2: 检查 hermes config.yaml
# ============================================================
if [[ ! -f "$HERMES_CONFIG" ]]; then
die "Hermes 配置文件不存在: $HERMES_CONFIG" 1
fi
log "读取 Hermes 配置: $HERMES_CONFIG"
# ============================================================
# Step 3: 从 hermes config.yaml 提取模型配置
# ============================================================
need_cmd() {
command -v "$1" >/dev/null 2>&1 || die "必需命令未找到: $1" 127
}
need_cmd python3
# 使用 python3 安全解析 YAML，提取 model 段的 default / base_url / api_key
read_hermes_model_config() {
python3 - "$HERMES_CONFIG" <<'PYEOF'
import sys, json
config_path = sys.argv[1]
# 尝试使用 PyYAML；如果不存在则手动逐行解析
try:
import yaml
with open(config_path, "r", encoding="utf-8") as f:
cfg = yaml.safe_load(f) or {}
model_section = cfg.get("model", {})
result = {
"model": model_section.get("default", ""),
"base_url": model_section.get("base_url", ""),
"api_key": model_section.get("api_key", ""),
}
print(json.dumps(result))
except ImportError:
# PyYAML 不可用，用简单逐行解析提取 model 段
import re
with open(config_path, "r", encoding="utf-8") as f:
lines = f.readlines()
in_model = False
result = {"model": "", "base_url": "", "api_key": ""}
for line in lines:
stripped = line.rstrip("\n")
# 检测 model: 顶层段
if re.match(r"^model:\s*$", stripped) or re.match(r"^model:\s*#", stripped):
in_model = True
continue
# 进入新的顶层段则退出
if in_model and re.match(r"^[A-Za-z_]", stripped):
break
if in_model:
m = re.match(r"^\s+default:\s*(.+)", stripped)
if m:
result["model"] = m.group(1).strip().strip("'\"")
m = re.match(r"^\s+base_url:\s*(.+)", stripped)
if m:
result["base_url"] = m.group(1).strip().strip("'\"")
m = re.match(r"^\s+api_key:\s*(.+)", stripped)
if m:
result["api_key"] = m.group(1).strip().strip("'\"")
print(json.dumps(result))
PYEOF
}
MODEL_JSON="$(read_hermes_model_config)"
HERMES_MODEL="$(printf '%s' "$MODEL_JSON" | python3 -c 'import json,sys; print(json.load(sys.stdin)["model"])')"
HERMES_BASE_URL="$(printf '%s' "$MODEL_JSON" | python3 -c 'import json,sys; print(json.load(sys.stdin)["base_url"])')"
HERMES_API_KEY="$(printf '%s' "$MODEL_JSON" | python3 -c 'import json,sys; print(json.load(sys.stdin)["api_key"])')"
# 校验必需字段
if [[ -z "$HERMES_MODEL" ]]; then
die "Hermes 配置中 model.default 为空" 1
fi
if [[ -z "$HERMES_BASE_URL" ]]; then
die "Hermes 配置中 model.base_url 为空" 1
fi
if [[ -z "$HERMES_API_KEY" ]]; then
die "Hermes 配置中 model.api_key 为空" 1
fi
log "Hermes 模型配置:"
log " model = $HERMES_MODEL"
log " base_url = $HERMES_BASE_URL"
log " api_key = <${#HERMES_API_KEY} chars>"
# ============================================================
# Step 4: 透传参数（--restart / --dry-run / --hermes）
# ============================================================
CTL_EXTRA_ARGS=()
SYNC_MODE="standalone"
REQUEST_RESTART=0
for arg in "$@"; do
case "$arg" in
--restart)
REQUEST_RESTART=1
;;
--hermes)
SYNC_MODE="hermes"
CTL_EXTRA_ARGS+=("$arg")
;;
--standalone)
SYNC_MODE="standalone"
CTL_EXTRA_ARGS+=("$arg")
;;
--dry-run)
CTL_EXTRA_ARGS+=("$arg")
;;
*)
warn "忽略未知参数: $arg"
;;
esac
done
if [[ $REQUEST_RESTART -eq 1 ]]; then
# 新语义：--restart 只在 Gateway 当前已运行时才生效。
# 这里复用 CTL 的 status 输出作为唯一事实来源，避免本脚本
# 自己维护另一套“端口监听/健康检查”判定逻辑。
if should_restart_gateway; then
CTL_EXTRA_ARGS+=("--restart")
else
log "已按新逻辑跳过 --restart。"
fi
fi
# ============================================================
# Step 5: 调用 memory-tencentdb-ctl.sh config llm 同步配置
# ============================================================
log "调用 memory-tencentdb-ctl.sh config llm 同步配置到 TDAI ..."
bash "$CTL_SCRIPT" "${CTL_EXTRA_ARGS[@]+"${CTL_EXTRA_ARGS[@]}"}" \
config llm \
--api-key "$HERMES_API_KEY" \
--base-url "$HERMES_BASE_URL" \
--model "$HERMES_MODEL"
log "同步完成。"
log ""
log "如需立即重启 Gateway 使配置生效，请重新运行并追加 --restart："
log " bash $0 --restart"
scripts/verify-cold-start-loop.sh
+61
View File
@@ -0,0 +1,61 @@
#!/bin/bash
# ============================================================
# 验证 1: 冷启动循环 Bug 复现
#
# 条件: 手动构造 checkpoint 状态 → last_persona_at=0,
# 但 persona.md 存在 + scene_blocks 存在
# 预期: L3 反复触发但每次都跳过
# ============================================================
set -e
EP="https://tdai.apigateway.cd.test.polaris"
AUTH="Authorization: Bearer DQfp9PnHn+iKwON8+ipBfOCXx1ISlfXxSWWENu095ZIp"
SID="X-TDAI-Service-ID: mem-rkgqhd5z"
CT="Content-Type: application/json"
SESSION="coldstart-verify-$(date +%s)"
echo "============================================================"
echo "验证 1: 冷启动循环复现"
echo "Session: $SESSION"
echo "============================================================"
echo
# Step 1: 写入足够多的对话触发 L1 → L2 → L3 pipeline
echo "--- Step 1: 写入对话数据触发 pipeline ---"
for i in $(seq 1 10); do
curl -sk -X POST "$EP/v2/conversation/add" -H "$CT" -H "$AUTH" -H "$SID" \
-d "{\"session_id\":\"$SESSION\",\"messages\":[{\"role\":\"user\",\"content\":\"第${i}轮对话：测试冷启动循环\",\"timestamp\":\"2026-05-18T11:${i}:00Z\"},{\"role\":\"assistant\",\"content\":\"收到第${i}轮\",\"timestamp\":\"2026-05-18T11:${i}:05Z\"}]}" > /dev/null
echo " 写入第 $i 轮对话"
done
echo
# Step 2: 写入 persona 和 scenario (模拟已有数据)
echo "--- Step 2: 写入 persona + scenario ---"
curl -sk -X POST "$EP/v2/persona/write" -H "$CT" -H "$AUTH" -H "$SID" \
-d '{"content": "# Persona\n\n这是测试persona，模拟已存在的状态。"}'
echo
curl -sk -X POST "$EP/v2/scenario/write" -H "$CT" -H "$AUTH" -H "$SID" \
-d '{"path": "test-coldstart.md", "content": "# Test Scene\n\n场景文件存在。"}'
echo
echo
# Step 3: 等待 pipeline 触发
echo "--- Step 3: 等待 60s 让 pipeline 执行 ---"
echo " (观察容器日志看是否出现 'Trigger P2 (cold start)' 循环)"
echo " 在另一个终端执行:"
echo " sudo docker logs -f memory-test-fixed 2>&1 | grep -E 'Trigger P2|cold start|Persona generation skipped|last_persona_at'"
echo
sleep 5
# Step 4: 检查结果
echo "--- Step 4: 读取 persona 验证 ---"
curl -sk -X POST "$EP/v2/persona/read" -H "$CT" -H "$AUTH" -H "$SID" -d '{}'
echo
echo
echo "============================================================"
echo "如果日志中反复出现:"
echo " [trigger] Trigger P2 (cold start): scenes_processed=N, total_processed=0"
echo " [L3] Persona generation skipped (no changes)"
echo "则 Bug 复现成功。"
echo "============================================================"

Some files were not shown because too many files have changed in this diff Show More