mirror of
https://github.com/TencentCloud/TencentDB-Agent-Memory
synced 2026-07-26 04:04:30 +00:00
feat: release v1.0.0-beta.1
This commit is contained in:
chrishuan
@@ -0,0 +1,342 @@
|
||||
# memory-tencentdb-ctl.sh — memory_tencentdb运维脚本
|
||||
|
||||
> 配合 [`install_hermes_memory_tencentdb.sh`](./install_hermes_memory_tencentdb.sh) 使用。
|
||||
> 先跑安装脚本部署好插件 + Node 依赖，之后日常的启停 / 配置全部通过 `memory-tencentdb-ctl.sh` 完成。
|
||||
|
||||
## 1. 运行模式
|
||||
|
||||
脚本有两种模式，**默认是独立模式**，完全不触碰 `~/.hermes`：
|
||||
|
||||
| 模式 | 激活方式 | 做什么 | 不做什么 |
|
||||
|---|---|---|---|
|
||||
| `standalone`（默认） | 无需任何参数 | 启停 Gateway；写 `$TDAI_DATA_DIR/tdai-gateway.json`；日志落 `$TDAI_DATA_DIR/logs/` | 不写 `$HERMES_HOME/env.d/`，不改 `$HERMES_HOME/config.yaml`，不读 hermes 相关 env |
|
||||
| `hermes` | 命令行追加 `--hermes`，或环境 `MEMORY_TENCENTDB_MODE=hermes` | standalone 的全部 + `config llm` 同步写 `$HERMES_HOME/env.d/memory-tencentdb-llm.sh`；日志落 `$HERMES_HOME/logs/memory_tencentdb/`；开放 `enable-hermes-memory` 子命令 | — |
|
||||
|
||||
> **为什么 hermes 模式要多写 env 文件？**
|
||||
> 因为 hermes 进程会托管式地把 Gateway 以子进程拉起来（supervisor 用 `os.environ.copy()` 传环境），此时 Gateway 读不到 `tdai-gateway.json` 所在的 shell 环境，必须通过 `$HERMES_HOME/env.d/*.sh` 让 hermes 自身 `source` 才能把凭据传进去。独立模式下 Gateway 自己读 JSON，无此需要。
|
||||
|
||||
## 2. 路径
|
||||
|
||||
> **路径变量约定**：本节及后续示例统一用 `$HERMES_HOME` 指代 hermes 的家目录，**默认 `~/.hermes`**，但你可以通过环境变量覆盖（例如 `export HERMES_HOME=/srv/hermes`），脚本和 hermes 自身都遵守这个变量。
|
||||
>
|
||||
> 自 0.4.x 起，所有 tdai 相关数据/代码默认收纳到统一根目录 `$MEMORY_TENCENTDB_ROOT`（默认 `~/.memory-tencentdb`）之下：
|
||||
>
|
||||
> - `$TDAI_INSTALL_DIR` 默认 `$MEMORY_TENCENTDB_ROOT/tdai-memory-openclaw-plugin`（即 `~/.memory-tencentdb/tdai-memory-openclaw-plugin`）
|
||||
> - `$TDAI_DATA_DIR` 默认 `$MEMORY_TENCENTDB_ROOT/memory-tdai`（即 `~/.memory-tencentdb/memory-tdai`）
|
||||
>
|
||||
> 下文出现这些变量时**先 `export` 一份覆盖**，再跑命令即可全局生效。
|
||||
> 旧版本使用 `~/tdai-memory-openclaw-plugin` 与 `~/memory-tdai`；`install_hermes_memory_tencentdb.sh` 在升级时会自动迁移这两个旧目录到新位置。
|
||||
|
||||
| 路径 | standalone | hermes | 作用 |
|
||||
|---|---|---|---|
|
||||
| `$TDAI_INSTALL_DIR` | ✅ | ✅ | 插件源码 + `node_modules` + `src/gateway/server.ts` |
|
||||
| `$TDAI_DATA_DIR/tdai-gateway.json` | ✅ | ✅ | Gateway 主配置：`llm` / `memory.embedding` / `memory.tcvdb` / `memory.storeBackend`，权限 `0600` |
|
||||
| `$TDAI_DATA_DIR/logs/` | ✅ 日志 | — | `gateway.stdout.log` / `gateway.stderr.log` / `gateway.pid` |
|
||||
| `$HERMES_HOME/logs/memory_tencentdb/` | — | ✅ 日志 | 同上，换目录 |
|
||||
| `$HERMES_HOME/env.d/memory-tencentdb-llm.sh` | — | ✅ | hermes 启动前 `source`，给 supervisor 托管的 Gateway 子进程注入 LLM 凭据 |
|
||||
| `$HERMES_HOME/config.yaml` | — | ✅ | `enable-hermes-memory` 修改其 `memory.provider` |
|
||||
| Gateway 监听 | `127.0.0.1:8420` | `127.0.0.1:8420` | 可被 `MEMORY_TENCENTDB_GATEWAY_HOST/PORT` 覆盖 |
|
||||
|
||||
所有路径都能用同名环境变量覆盖（再次列出便于对照）：`MEMORY_TENCENTDB_ROOT`（默认 `~/.memory-tencentdb`）、`TDAI_INSTALL_DIR`（默认 `$MEMORY_TENCENTDB_ROOT/tdai-memory-openclaw-plugin`）、`TDAI_DATA_DIR`（默认 `$MEMORY_TENCENTDB_ROOT/memory-tdai`）、`HERMES_HOME`（默认 `~/.hermes`）、`MEMORY_TENCENTDB_LOG_DIR`、`MEMORY_TENCENTDB_GATEWAY_HOST/PORT`。
|
||||
|
||||
依赖：`bash`、`python3`、`node >= 22`、`npx`、`lsof` 或 `ss`。
|
||||
|
||||
## 3. 安装 & 调用
|
||||
|
||||
脚本随 npm 包发布到 `node_modules/.../scripts/` 下，但**没有**注册为 `bin` 命令。要用全局命令名调用，必须自己做一次软链。
|
||||
|
||||
### 3.1 从 npm 包里直接运行（无需任何配置）
|
||||
|
||||
```bash
|
||||
npm install @tencentdb-agent-memory/memory-tencentdb
|
||||
|
||||
# 项目内安装，路径可由 npm root 动态算出
|
||||
"$(npm root)/@tencentdb-agent-memory/memory-tencentdb/scripts/memory-tencentdb-ctl.sh" --help
|
||||
|
||||
# 全局安装则用 npm root -g
|
||||
"$(npm root -g)/@tencentdb-agent-memory/memory-tencentdb/scripts/memory-tencentdb-ctl.sh" --help
|
||||
```
|
||||
|
||||
`npm root` / `npm root -g` 会在所有包管理器（npm / pnpm / yarn）和不同的 prefix 配置下返回正确目录，避免硬编码 `node_modules/` 路径。适合一次性、临时使用的场景。
|
||||
|
||||
### 3.2 软链到 PATH（推荐给运维 / 长期使用）
|
||||
|
||||
不论脚本来源（git clone 出来的仓库 / `npm install` 装的包 / 自定义部署目录），先把脚本路径**算出来存到一个变量里**，再统一做软链。这样无需关心你把仓库放在 `~/code/`、`/opt/`、还是别的什么地方。
|
||||
|
||||
```bash
|
||||
# 第一步：定位 memory-tencentdb-ctl.sh 的真实路径（任选一种来源）
|
||||
|
||||
# (a) 从 git 仓库（在仓库根目录或任意子目录里执行）
|
||||
SCRIPT="$(git -C "$(git rev-parse --show-toplevel)" ls-files | \
|
||||
grep -E 'scripts/memory-tencentdb-ctl\.sh$' | head -1)"
|
||||
SCRIPT="$(git rev-parse --show-toplevel)/$SCRIPT"
|
||||
|
||||
# (b) 从已 npm 全局安装的包
|
||||
SCRIPT="$(npm root -g)/@tencentdb-agent-memory/memory-tencentdb/scripts/memory-tencentdb-ctl.sh"
|
||||
|
||||
# (c) 从项目本地的 node_modules
|
||||
SCRIPT="$(npm root)/@tencentdb-agent-memory/memory-tencentdb/scripts/memory-tencentdb-ctl.sh"
|
||||
|
||||
# (d) 完全手写绝对路径（如部署到非标准位置）
|
||||
SCRIPT="/opt/tdai/scripts/memory-tencentdb-ctl.sh"
|
||||
|
||||
# 第二步：验证路径正确，然后软链
|
||||
test -f "$SCRIPT" && echo "ok: $SCRIPT" || { echo "not found"; exit 1; }
|
||||
chmod +x "$SCRIPT"
|
||||
sudo ln -sf "$SCRIPT" /usr/local/bin/memory-tencentdb-ctl
|
||||
|
||||
# 同样的办法把 install_hermes_memory_tencentdb.sh 链接成 install-memory-tencentdb（可选）
|
||||
INSTALL_SCRIPT="$(dirname "$SCRIPT")/install_hermes_memory_tencentdb.sh"
|
||||
test -f "$INSTALL_SCRIPT" && {
|
||||
chmod +x "$INSTALL_SCRIPT"
|
||||
sudo ln -sf "$INSTALL_SCRIPT" /usr/local/bin/install-memory-tencentdb
|
||||
}
|
||||
```
|
||||
|
||||
之后直接 `memory-tencentdb-ctl …` / `install-memory-tencentdb …`。
|
||||
|
||||
> **为什么不直接用 `npm bin` 注册？** 这两个脚本是运维工具而不是包的核心 API，主仓库希望用户**显式**完成 PATH 注册（避免无意中污染全局命令空间，并避免 npm 卸载时静默移除运维入口）。
|
||||
|
||||
## 4. 生命周期管理（两种模式通用）
|
||||
|
||||
```bash
|
||||
memory-tencentdb-ctl start # 若 :8420 已占用会直接返回；否则后台 spawn，等待 /health 通过
|
||||
memory-tencentdb-ctl stop # 先 SIGTERM，5s 内未退则 SIGKILL
|
||||
memory-tencentdb-ctl restart
|
||||
memory-tencentdb-ctl status # 打印模式、端口、data/log 路径、进程状态
|
||||
memory-tencentdb-ctl health # GET /health，纯 python3 实现，不要求 curl
|
||||
memory-tencentdb-ctl logs # tail -f stdout + stderr
|
||||
memory-tencentdb-ctl logs err 500 # 只看 stderr 最近 500 行
|
||||
```
|
||||
|
||||
启动命令解析顺序：
|
||||
|
||||
1. 环境变量 `MEMORY_TENCENTDB_GATEWAY_CMD`（`install_hermes_memory_tencentdb.sh` 写入 `/etc/profile.d/memory-tencentdb-env.sh` 的那条）。
|
||||
2. 回退到 `sh -c 'cd $TDAI_INSTALL_DIR && exec npx tsx src/gateway/server.ts'`。
|
||||
|
||||
启动时会自动 `source` 的环境文件：
|
||||
|
||||
- 两种模式：`/etc/profile.d/memory-tencentdb-env.sh`
|
||||
- 仅 hermes 模式：`/etc/profile.d/hermes-env.sh` 以及 `$HERMES_HOME/env.d/*.sh`
|
||||
|
||||
## 5. 配置 LLM / Embedding / VDB
|
||||
|
||||
三类凭据统一落到 `$TDAI_DATA_DIR/tdai-gateway.json`（`0600`，原子写）。**`config llm` 在 `--hermes` 模式下会额外写一份 env 文件**；Embedding / VDB 从不写 env。
|
||||
|
||||
### 5.1 LLM
|
||||
|
||||
```bash
|
||||
# standalone 模式：只写 tdai-gateway.json
|
||||
memory-tencentdb-ctl config llm \
|
||||
--api-key "sk-xxxxxxxxxxxx" \
|
||||
--base-url "https://api.openai.com/v1" \
|
||||
--model "gpt-4o" \
|
||||
--restart
|
||||
|
||||
# hermes 模式：tdai-gateway.json + $HERMES_HOME/env.d/memory-tencentdb-llm.sh
|
||||
memory-tencentdb-ctl --hermes config llm \
|
||||
--api-key "sk-xxxxxxxxxxxx" \
|
||||
--base-url "https://api.openai.com/v1" \
|
||||
--model "gpt-4o" \
|
||||
--restart
|
||||
```
|
||||
|
||||
- JSON 写入点：`$.llm.{baseUrl, apiKey, model}`。
|
||||
- env 文件写入（仅 `--hermes`）：`TDAI_LLM_*` 及 `MEMORY_TENCENTDB_LLM_*` 别名（Python provider 的 `get_config_schema()` 会读后者）。
|
||||
|
||||
### 5.2 Embedding
|
||||
|
||||
默认关闭（`provider=none`）。启用远端 OpenAI 兼容服务：
|
||||
|
||||
```bash
|
||||
memory-tencentdb-ctl config embedding \
|
||||
--provider openai \
|
||||
--api-key "sk-xxxx" \
|
||||
--base-url "https://api.openai.com/v1" \
|
||||
--model "text-embedding-3-small" \
|
||||
--dimensions 1536 \
|
||||
--restart
|
||||
|
||||
# 关闭 embedding（退化为 BM25/关键词召回）
|
||||
memory-tencentdb-ctl config embedding --provider none --restart
|
||||
```
|
||||
|
||||
- JSON 写入点：`$.memory.embedding.{provider, baseUrl, apiKey, model, dimensions, enabled, proxyUrl?}`。
|
||||
- `qclaw` provider 额外要求 `--proxy-url`。
|
||||
- 校验规则与 `src/config.ts` 的 `parseConfig()` 对齐：`dimensions` 为正整数，非 `none` 必须带 `apiKey/baseUrl/model/dimensions`；缺项直接报错不写半残 JSON。
|
||||
|
||||
### 5.3 VectorDB（Tencent Cloud VDB / tcvdb）
|
||||
|
||||
```bash
|
||||
memory-tencentdb-ctl config vdb \
|
||||
--url "http://xxx-vdb.tencentclb.com:8100" \
|
||||
--username root \
|
||||
--api-key "YOUR-VDB-API-KEY" \
|
||||
--database "openclaw_memory" \
|
||||
--alias "primary" \
|
||||
--embedding-model "bge-large-zh" \
|
||||
--ca-pem "/etc/ssl/vdb-ca.pem" \
|
||||
--restart
|
||||
```
|
||||
|
||||
- JSON 写入点：`$.memory.tcvdb.{url, username, apiKey, database, alias?, caPemPath?, embeddingModel?}`。
|
||||
- 默认同时把 `$.memory.storeBackend` 切到 `"tcvdb"`；只想预埋配置先不切，加 `--no-set-backend`。
|
||||
- `--ca-pem` 只写路径不复制文件；脚本会校验可读性。
|
||||
|
||||
### 5.4 退回本地 SQLite（关闭 VDB 后端）
|
||||
|
||||
```bash
|
||||
# 默认：保留 memory.tcvdb 凭据（方便随时再切回去），仅把 storeBackend 改回 sqlite
|
||||
memory-tencentdb-ctl config vdb-off --restart
|
||||
|
||||
# 同时把腾讯云 VDB 的 url / apiKey / database 等凭据从 JSON 中清掉
|
||||
memory-tencentdb-ctl config vdb-off --purge-creds --restart
|
||||
```
|
||||
|
||||
- JSON 写入点：把 `$.memory.storeBackend` 设为 `"sqlite"`；`--purge-creds` 时额外删除整段 `$.memory.tcvdb`。
|
||||
- `$.llm` / `$.memory.embedding` 等其它顶级段**完全保留**，hermes 侧 `memory.provider` **不动**（仍是 `memory_tencentdb`，只是它内部存储退回 sqlite）。
|
||||
- 配置文件不存在时给出 `warn` 并写入仅含 `{"memory":{"storeBackend":"sqlite"}}` 的最小配置。
|
||||
- 与 `config vdb` 完全镜像：可与 `--dry-run` / `--restart` 组合使用。
|
||||
|
||||
### 5.5 查看当前配置
|
||||
|
||||
```bash
|
||||
memory-tencentdb-ctl config show
|
||||
```
|
||||
|
||||
- 打印 `tdai-gateway.json`，`apiKey`/`password`/`token` 字段自动脱敏为 `<redacted:NN chars>`。
|
||||
- hermes 模式下额外打印 `$HERMES_HOME/env.d/memory-tencentdb-*.sh`（API key 也会脱敏），可直接贴工单。
|
||||
|
||||
## 6. 打通 hermes（仅 `--hermes` 模式）
|
||||
|
||||
```bash
|
||||
memory-tencentdb-ctl --hermes enable-hermes-memory
|
||||
```
|
||||
|
||||
幂等：把 `$HERMES_HOME/config.yaml` 的 `memory:` 段的 `provider:` 改成 `memory_tencentdb`（不存在则新增整段）。改完后重启 hermes：
|
||||
|
||||
```bash
|
||||
source "$HERMES_HOME/env.d/memory-tencentdb-llm.sh"
|
||||
pkill -f hermes-agent || true
|
||||
hermes
|
||||
```
|
||||
|
||||
> **关于写入策略**：脚本采用"格式保真"双路径，**永不重写整个 YAML**：
|
||||
>
|
||||
> 1. **首选**：检测到 [`ruamel.yaml`](https://yaml.readthedocs.io/) 时走 round-trip，完整保留注释、键序、引号、缩进风格（推荐 `pip install --user ruamel.yaml` 享受最佳保真度，**非必装**）；
|
||||
> 2. **降级**：未安装 ruamel 时走最小化原位行编辑——只重写 `provider:` 那一行，缩进直接从同段已有兄弟键的前缀**逐字符拷贝**（零猜测、零格式破坏）；
|
||||
> 3. 若 `memory:` 段不存在，则在文件末尾追加最小段，缩进从文档其它顶级段的子键拓印。
|
||||
>
|
||||
> 实测对真实 `~/.hermes/config.yaml` 做 byte-for-byte diff，除 `provider` 值本身外其余字节完全一致。
|
||||
|
||||
非 hermes 模式下调用该命令会直接报错退出。
|
||||
|
||||
> 想反过来"保留 hermes provider 不变，仅让 TDAI 内部存储退回 sqlite"，用 §5.4 的 `config vdb-off` 即可（不需要也**不要**改 hermes 的 `memory.provider`）。
|
||||
|
||||
## 7. 典型使用流程
|
||||
|
||||
### 场景 A：Gateway 独立部署（不使用 hermes）
|
||||
|
||||
```bash
|
||||
# 1) 安装
|
||||
# INSTALL_SCRIPT 的取值方式见上文 3.2 节（git rev-parse / npm root / 手填均可）
|
||||
# 例如从 git 仓库根：INSTALL_SCRIPT="$(git rev-parse --show-toplevel)/scripts/install_hermes_memory_tencentdb.sh"
|
||||
# 从 npm 全局： INSTALL_SCRIPT="$(npm root -g)/@tencentdb-agent-memory/memory-tencentdb/scripts/install_hermes_memory_tencentdb.sh"
|
||||
bash "$INSTALL_SCRIPT"
|
||||
|
||||
# 2) 只配 Gateway 所需凭据
|
||||
memory-tencentdb-ctl config llm --api-key "sk-..." --base-url "https://api.openai.com/v1" --model gpt-4o
|
||||
memory-tencentdb-ctl config embedding --provider openai --api-key "sk-..." --base-url "https://api.openai.com/v1" \
|
||||
--model text-embedding-3-small --dimensions 1536
|
||||
memory-tencentdb-ctl config vdb --url "http://xxx:8100" --api-key "..." --database openclaw_memory
|
||||
|
||||
# 3) 启动 + 自检
|
||||
memory-tencentdb-ctl start
|
||||
memory-tencentdb-ctl status
|
||||
memory-tencentdb-ctl health # 预期: {"status":"ok",...}
|
||||
```
|
||||
|
||||
### 场景 B：集成 hermes
|
||||
|
||||
```bash
|
||||
# 1) 安装（与场景 A 相同；INSTALL_SCRIPT 由上文 3.2 节算出）
|
||||
bash "$INSTALL_SCRIPT"
|
||||
|
||||
# 2) 全程加 --hermes（或 export MEMORY_TENCENTDB_MODE=hermes 一次）
|
||||
memory-tencentdb-ctl --hermes config llm --api-key "sk-..." --base-url "https://api.openai.com/v1" --model gpt-4o
|
||||
memory-tencentdb-ctl --hermes config embedding --provider openai --api-key "sk-..." \
|
||||
--base-url "https://api.openai.com/v1" \
|
||||
--model text-embedding-3-small --dimensions 1536
|
||||
memory-tencentdb-ctl --hermes config vdb --url "http://xxx:8100" --api-key "..." --database openclaw_memory
|
||||
|
||||
# 3) 启动 Gateway（通常由 hermes supervisor 托管，这里是手动兜底）
|
||||
memory-tencentdb-ctl --hermes start
|
||||
memory-tencentdb-ctl --hermes status
|
||||
|
||||
# 4) 在 hermes config 里启用 provider，并重启 hermes
|
||||
memory-tencentdb-ctl --hermes enable-hermes-memory
|
||||
source "$HERMES_HOME/env.d/memory-tencentdb-llm.sh"
|
||||
pkill -f hermes-agent ; hermes
|
||||
```
|
||||
|
||||
如果嫌 `--hermes` 每次都要带，可以：
|
||||
|
||||
```bash
|
||||
export MEMORY_TENCENTDB_MODE=hermes
|
||||
```
|
||||
|
||||
之后所有调用自动切到 hermes 模式，命令行无需再加 `--hermes`。
|
||||
|
||||
### 场景 C：临时把 TDAI 的存储退回 sqlite（保留 hermes 集成）
|
||||
|
||||
适用于 VDB 不可达 / 排障 / 离线开发等场景：希望 hermes 端 `memory.provider` 仍是 `memory_tencentdb`，但让 Gateway 改用本地 SQLite 落盘。
|
||||
|
||||
```bash
|
||||
# (A) 默认：保留 memory.tcvdb 凭据，仅把 storeBackend 切回 sqlite
|
||||
memory-tencentdb-ctl config vdb-off --restart
|
||||
|
||||
# (B) 排障结束想切回 vdb：当前需要重新跑一次 config vdb（必填项需重新提供，
|
||||
# 即使 JSON 里凭据还在；脚本是基于必填校验的"重新声明"语义，不是 toggle）
|
||||
memory-tencentdb-ctl config vdb \
|
||||
--url "http://xxx-vdb.tencentclb.com:8100" \
|
||||
--api-key "<你的 KEY>" \
|
||||
--database "openclaw_memory" \
|
||||
--restart
|
||||
|
||||
# (C) 彻底放弃 vdb：清掉凭据
|
||||
memory-tencentdb-ctl config vdb-off --purge-creds --restart
|
||||
```
|
||||
|
||||
> 之所以 (B) 不提供"零参数 vdb-on"，是因为原 `config vdb` 子命令把 `--url/--api-key/--database` 设为强校验，避免用户拼装出半残配置；如果你希望把"已存的凭据重新激活"也做成单命令，告诉维护者补一个 `config vdb-on` 即可（实现方式与 `vdb-off` 完全镜像）。
|
||||
|
||||
> **不要**为了"切回 sqlite"去改 `~/.hermes/config.yaml` 的 `memory.provider`！hermes 看到的依然是 `memory_tencentdb` provider，存储后端切换是 Gateway 内部的事，对 hermes 完全透明。
|
||||
|
||||
## 8. 全局选项 & 调试技巧
|
||||
|
||||
- 所有写操作支持 `--dry-run`（放在命令最前），会打印将要写入的内容但不落盘：
|
||||
```bash
|
||||
memory-tencentdb-ctl --dry-run config llm --api-key k --base-url https://x --model m
|
||||
```
|
||||
- 敏感文件权限一律 `0600`；`env.d/memory-tencentdb-llm.sh` 含明文 API key，**不要** commit。
|
||||
- 启动失败：`memory-tencentdb-ctl logs err 200` 查看 stderr；手动前台跑一遍更容易看到报错：
|
||||
```bash
|
||||
cd "$TDAI_INSTALL_DIR" && npx tsx src/gateway/server.ts
|
||||
```
|
||||
- 端口冲突：`MEMORY_TENCENTDB_GATEWAY_PORT=18420 memory-tencentdb-ctl restart`。
|
||||
- 验证 hermes 有没有吃到新 env（hermes 模式）：
|
||||
```bash
|
||||
tr '\0' '\n' < /proc/$(pgrep -n hermes-agent)/environ | grep -E 'TDAI_|MEMORY_TENCENTDB_'
|
||||
```
|
||||
|
||||
## 9. 退出码
|
||||
|
||||
| 码 | 含义 |
|
||||
|---|---|
|
||||
| 0 | 成功 |
|
||||
| 1 | 参数错误 / 业务校验失败（如 `--base-url` 非 http(s)；在 standalone 下调 hermes 专属命令） |
|
||||
| 2 | 写盘失败（磁盘满、权限不足等） |
|
||||
| 127 | 依赖缺失（`python3` / `node` / `npx`） |
|
||||
|
||||
---
|
||||
|
||||
要封装成 systemd unit，基于 `memory-tencentdb-ctl start` / `memory-tencentdb-ctl stop` 写 `Type=forking` 的 service 即可（Gateway 是无状态 HTTP sidecar，不依赖 systemd readiness 协议）。
|
||||
@@ -0,0 +1,122 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Benchmark: current per-instance ZSET vs Scheme D (16-shard global ZSET) timer scanning."""
|
||||
import redis
|
||||
import time
|
||||
|
||||
r = redis.Redis(host="127.0.0.1", port=6379, decode_responses=True)
|
||||
r.flushdb()
|
||||
|
||||
INSTANCE_COUNT = 100000
|
||||
SHARD_COUNT = 16
|
||||
now = int(time.time() * 1000)
|
||||
expired = now - 1000
|
||||
|
||||
SEP = "=" * 60
|
||||
|
||||
# ============================================================
|
||||
# Phase 1: Write 100K per-instance ZSETs (current design)
|
||||
# ============================================================
|
||||
print(f"Phase 1: Writing {INSTANCE_COUNT} per-instance ZSETs...")
|
||||
t0 = time.time()
|
||||
pipe = r.pipeline()
|
||||
for i in range(INSTANCE_COUNT):
|
||||
inst = f"inst-{i:06d}"
|
||||
key = "{" + f"tdai_memory:{inst}" + "}:timers"
|
||||
pipe.zadd(key, {"sess_001:L1_idle": expired})
|
||||
if i % 5000 == 4999:
|
||||
pipe.execute()
|
||||
pipe = r.pipeline()
|
||||
pipe.execute()
|
||||
write_time_old = time.time() - t0
|
||||
print(f" Done: {write_time_old:.2f}s, keys={r.dbsize()}")
|
||||
|
||||
# ============================================================
|
||||
# Phase 2: Simulate current Scanner (iterate all instances)
|
||||
# ============================================================
|
||||
print(f"\nPhase 2: Current scanner - iterate {INSTANCE_COUNT} instances...")
|
||||
start = time.time()
|
||||
total_expired_old = 0
|
||||
for i in range(INSTANCE_COUNT):
|
||||
inst = f"inst-{i:06d}"
|
||||
key = "{" + f"tdai_memory:{inst}" + "}:timers"
|
||||
members = r.zrangebyscore(key, "-inf", now)
|
||||
if members:
|
||||
r.zremrangebyscore(key, "-inf", now)
|
||||
total_expired_old += len(members)
|
||||
old_elapsed = time.time() - start
|
||||
old_status = "TIMEOUT" if old_elapsed > 2 else "OK"
|
||||
print(f" Expired found: {total_expired_old}")
|
||||
print(f" Scan time: {old_elapsed:.2f}s [{old_status}] (target: <2s)")
|
||||
print(f" Redis ops: ~{INSTANCE_COUNT * 2} (2 per instance)")
|
||||
|
||||
# ============================================================
|
||||
# Phase 3: Write 16-shard ZSETs (Scheme D)
|
||||
# ============================================================
|
||||
print(f"\nPhase 3: Writing {INSTANCE_COUNT} timers into {SHARD_COUNT} shards...")
|
||||
r.flushdb()
|
||||
|
||||
def shard_hash(s):
|
||||
h = 5381
|
||||
for c in s:
|
||||
h = ((h << 5) + h + ord(c)) & 0xFFFFFFFF
|
||||
return h % SHARD_COUNT
|
||||
|
||||
t0 = time.time()
|
||||
pipe = r.pipeline()
|
||||
for i in range(INSTANCE_COUNT):
|
||||
inst = f"inst-{i:06d}"
|
||||
shard = shard_hash(inst)
|
||||
key = f"tdai_memory:timers:shard_{shard}"
|
||||
member = f"{inst}:sess_001:L1_idle"
|
||||
pipe.zadd(key, {member: expired})
|
||||
if i % 5000 == 4999:
|
||||
pipe.execute()
|
||||
pipe = r.pipeline()
|
||||
pipe.execute()
|
||||
write_time_new = time.time() - t0
|
||||
print(f" Done: {write_time_new:.2f}s, keys={r.dbsize()}")
|
||||
print(f" Shard distribution:")
|
||||
for s in range(SHARD_COUNT):
|
||||
cnt = r.zcard(f"tdai_memory:timers:shard_{s}")
|
||||
print(f" shard_{s:02d}: {cnt} members")
|
||||
|
||||
# ============================================================
|
||||
# Phase 4: Scheme D scanner (scan 16 shards with Lua atomic claim)
|
||||
# ============================================================
|
||||
print(f"\nPhase 4: Scheme D scanner - scan {SHARD_COUNT} shards (Lua atomic)...")
|
||||
LUA_CLAIM = """
|
||||
local members = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', ARGV[1], 'LIMIT', 0, 50000)
|
||||
if #members > 0 then
|
||||
redis.call('ZREM', KEYS[1], unpack(members))
|
||||
return members
|
||||
end
|
||||
return {}
|
||||
"""
|
||||
start = time.time()
|
||||
total_expired_new = 0
|
||||
for shard in range(SHARD_COUNT):
|
||||
key = f"tdai_memory:timers:shard_{shard}"
|
||||
members = r.eval(LUA_CLAIM, 1, key, now)
|
||||
total_expired_new += len(members)
|
||||
new_elapsed = time.time() - start
|
||||
new_status = "TIMEOUT" if new_elapsed > 2 else "OK"
|
||||
print(f" Expired found: {total_expired_new}")
|
||||
print(f" Scan time: {new_elapsed:.4f}s [{new_status}] (target: <2s)")
|
||||
print(f" Redis ops: {SHARD_COUNT} (1 EVAL per shard)")
|
||||
|
||||
# ============================================================
|
||||
# Summary
|
||||
# ============================================================
|
||||
print(f"\n{SEP}")
|
||||
print(f" BENCHMARK: {INSTANCE_COUNT} instances, 1 timer each")
|
||||
print(f"{SEP}")
|
||||
print(f" Current (per-instance ZSET): {old_elapsed:.2f}s / scan [{old_status}]")
|
||||
print(f" Scheme D (16 shards): {new_elapsed:.4f}s / scan [{new_status}]")
|
||||
if new_elapsed > 0:
|
||||
print(f" Speedup: {old_elapsed/new_elapsed:.0f}x faster")
|
||||
print(f" Max shard size: {max(r.zcard(f'tdai_memory:timers:shard_{s}') for s in range(SHARD_COUNT))} (all consumed)")
|
||||
print(f"{SEP}")
|
||||
|
||||
# Cleanup
|
||||
r.flushdb()
|
||||
print("\nRedis flushed. Done.")
|
||||
@@ -0,0 +1,88 @@
|
||||
# Bugfix-20260423 打镜像 SOP
|
||||
|
||||
> **适用版本**: OpenClaw 2026.4.23
|
||||
> **修复问题**: Issue #73806 — Zod schema `.strict()` 拒绝 `hooks.allowConversationAccess`，导致非捆绑插件无法注册会话钩子
|
||||
> **脚本位置**: `scripts/bugfix-20260423.sh`
|
||||
|
||||
---
|
||||
|
||||
## 步骤一：停止 Gateway
|
||||
|
||||
```bash
|
||||
openclaw gateway stop
|
||||
```
|
||||
|
||||
确认已停止：
|
||||
|
||||
```bash
|
||||
ps aux | grep gateway
|
||||
```
|
||||
|
||||
确保没有 `openclaw-gateway` 进程在运行。
|
||||
|
||||
---
|
||||
|
||||
## 步骤二：执行 Patch
|
||||
|
||||
```bash
|
||||
cd /path/to/memory-tdai/scripts
|
||||
bash bugfix-20260423.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 步骤三：验证
|
||||
|
||||
### 3.1 验证 openclaw.json 配置
|
||||
|
||||
```bash
|
||||
cat ~/.openclaw/openclaw.json | python3 -m json.tool | grep allowConversationAccess
|
||||
```
|
||||
|
||||
预期输出：
|
||||
|
||||
```
|
||||
"allowConversationAccess": true
|
||||
```
|
||||
|
||||
确认在 `plugins.entries.memory-tencentdb.hooks` 下。
|
||||
|
||||
### 3.2 验证 Zod Schema dist 文件
|
||||
|
||||
先定位 OpenClaw 安装目录（路径因环境而异，以下仅为示例）：
|
||||
|
||||
```bash
|
||||
# 方式一：通过 which 自动定位
|
||||
OC_DIR=$(node -e "const p=require('path'),f=require('fs'); \
|
||||
const bin=require('child_process').execSync('which openclaw',{encoding:'utf8'}).trim(); \
|
||||
let d=p.dirname(f.realpathSync(bin)); \
|
||||
while(d!=p.dirname(d)){if(f.existsSync(p.join(d,'package.json'))){console.log(d);break;}d=p.dirname(d);}")
|
||||
echo "$OC_DIR"
|
||||
|
||||
# 方式二：手动指定（示例路径，请根据实际环境替换）
|
||||
# OC_DIR=~/.local/share/pnpm/global/5/.pnpm/openclaw@2026.4.23_@napi-rs+canvas@0.1.100/node_modules/openclaw
|
||||
```
|
||||
|
||||
然后检查 `zod-schema-BhKK4qYw.js`：
|
||||
|
||||
```bash
|
||||
cat "$OC_DIR/dist/zod-schema-BhKK4qYw.js" | grep allowConversationAccess -n
|
||||
```
|
||||
|
||||
验证要点：
|
||||
|
||||
1. `allowConversationAccess` 已出现在输出中
|
||||
2. **只出现了一次**（只有一行匹配）
|
||||
3. 所在行的上下文形如：`allowPromptInjection:z.boolean().optional(),allowConversationAccess:z.boolean().optional()}).strict().optional()`
|
||||
|
||||
<!-- TODO: 贴验证截图 -->
|
||||
|
||||
---
|
||||
|
||||
## 验证通过后
|
||||
|
||||
两项验证均通过即可重新启动 Gateway：
|
||||
|
||||
```bash
|
||||
openclaw gateway run
|
||||
```
|
||||
@@ -0,0 +1,298 @@
|
||||
#!/usr/bin/env bash
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# bugfix-20260423.sh — OC 2026.4.23 allowConversationAccess 修复
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# Issue #73806: OC 2026.4.23 的 Zod schema 使用 .strict() 拒绝
|
||||
# hooks.allowConversationAccess 字段，导致非捆绑插件无法注册会话钩子
|
||||
# (llm_input, llm_output, agent_end)。PR #71221 在 4.24 修复。
|
||||
#
|
||||
# 本脚本做两件事（均幂等，可安全重复执行）：
|
||||
# 1. Patch dist JS: 给 hooks zod schema 注入 allowConversationAccess 字段
|
||||
# 2. 写 openclaw.json: 设置 hooks.allowConversationAccess = true
|
||||
#
|
||||
# 版本限制：仅在 OC 2026.4.23 上执行 Part 1，其他版本安全跳过 dist patch。
|
||||
# Part 2 (配置写入) 不限版本，始终确保配置存在。
|
||||
#
|
||||
# 用法：
|
||||
# bash bugfix-20260423.sh [/path/to/openclaw]
|
||||
#
|
||||
# 环境变量：
|
||||
# OPENCLAW_DIR — 覆盖 openclaw 安装路径（优先于参数）
|
||||
# OPENCLAW_JSON — 覆盖配置文件路径（默认 ~/.openclaw/openclaw.json）
|
||||
# DEBUG=1 — 开启调试输出
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
set -euo pipefail
|
||||
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m'
|
||||
info() { echo -e "${CYAN}[INFO]${NC} $*"; }
|
||||
ok() { echo -e "${GREEN}[OK]${NC} $*"; }
|
||||
warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
|
||||
fail() { echo -e "${RED}[FAIL]${NC} $*" >&2; exit 1; }
|
||||
debug() { [[ "${DEBUG:-}" == "1" ]] && echo -e "${CYAN}[DEBUG]${NC} $*" || true; }
|
||||
|
||||
PLUGIN_ID="memory-tencentdb"
|
||||
OPENCLAW_JSON="${OPENCLAW_JSON:-${HOME}/.openclaw/openclaw.json}"
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# Part 1: Patch dist JS (仅 2026.4.23)
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
_resolve_openclaw_dir() {
|
||||
# 参数 > 环境变量 > 自动检测
|
||||
if [[ -n "${1:-}" ]]; then
|
||||
echo "$1"; return 0
|
||||
fi
|
||||
if [[ -n "${OPENCLAW_DIR:-}" && -d "${OPENCLAW_DIR}" ]]; then
|
||||
echo "$OPENCLAW_DIR"; return 0
|
||||
fi
|
||||
# 自动定位
|
||||
node -e "
|
||||
const {dirname, join} = require('path');
|
||||
const {realpathSync, existsSync, readFileSync, statSync} = require('fs');
|
||||
function walkUp(start) {
|
||||
let dir = statSync(start).isDirectory() ? start : dirname(start);
|
||||
for (let i = 0; i < 10; i++) {
|
||||
const pj = join(dir, 'package.json');
|
||||
if (existsSync(pj)) {
|
||||
try { if (JSON.parse(readFileSync(pj,'utf8')).name==='openclaw') { console.log(dir); process.exit(0); } } catch {}
|
||||
}
|
||||
const parent = dirname(dir);
|
||||
if (parent === dir) break;
|
||||
dir = parent;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
const {execSync} = require('child_process');
|
||||
const bin = execSync('which openclaw',{encoding:'utf8'}).trim();
|
||||
const real = realpathSync(bin);
|
||||
const found = walkUp(real);
|
||||
if (found) { console.log(found); process.exit(0); }
|
||||
const content = readFileSync(bin,'utf8');
|
||||
const m = content.match(/['\"]([^'\"]*openclaw[^'\"]*\\.(?:js|mjs))['\"]/) ||
|
||||
content.match(/['\"]([^'\"]*openclaw[^'\"]*)['\"].*node/);
|
||||
if (m) { const f = walkUp(realpathSync(m[1])); if (f) { console.log(f); process.exit(0); } }
|
||||
} catch {}
|
||||
const searchDirs = [
|
||||
join(process.env.HOME||'/root','.local/share/pnpm'),
|
||||
join(process.env.HOME||'/root','.local/node/lib/node_modules'),
|
||||
'/usr/local/lib/node_modules','/usr/lib/node_modules',
|
||||
];
|
||||
for (const base of searchDirs) {
|
||||
if (!existsSync(base)) continue;
|
||||
try {
|
||||
const {execSync:e2} = require('child_process');
|
||||
const out = e2('find '+JSON.stringify(base)+' -maxdepth 8 -name package.json -path \"*/openclaw/package.json\" 2>/dev/null',{encoding:'utf8',timeout:5000}).trim();
|
||||
for (const line of out.split('\\n')) {
|
||||
if (!line) continue;
|
||||
try { if (JSON.parse(readFileSync(line,'utf8')).name==='openclaw') { console.log(dirname(line)); process.exit(0); } } catch {}
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
process.exit(1);
|
||||
" 2>/dev/null
|
||||
}
|
||||
|
||||
patch_dist_js() {
|
||||
local oc_dir
|
||||
oc_dir="$(_resolve_openclaw_dir "${1:-}")" || {
|
||||
warn "[Part 1] 找不到 OpenClaw 安装目录，跳过 dist patch"
|
||||
return 0
|
||||
}
|
||||
|
||||
local dist_dir="$oc_dir/dist"
|
||||
[[ -d "$dist_dir" ]] || { warn "[Part 1] dist 目录不存在: $dist_dir，跳过"; return 0; }
|
||||
|
||||
local version
|
||||
version=$(grep -oP '"version"\s*:\s*"\K[^"]+' "$oc_dir/package.json" 2>/dev/null || echo "unknown")
|
||||
info "[Part 1] OpenClaw 版本: $version"
|
||||
|
||||
# 版本门控：仅 2026.4.23
|
||||
if [[ ! "$version" =~ ^2026\.4\.23($|[-\.]) ]]; then
|
||||
ok "[Part 1] 版本 $version 不需要 schema patch，跳过"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# 精确定位：hooks zod schema 的唯一特征
|
||||
local -a candidates
|
||||
mapfile -t candidates < <(
|
||||
grep -rl 'allowPromptInjection' "$dist_dir" --include='*.js' 2>/dev/null | while read -r _f; do
|
||||
if perl -0777 -ne 'exit(0) if /allowPromptInjection\s*:\s*[a-zA-Z_\$][a-zA-Z0-9_\$]*\s*\.\s*boolean\s*\(\s*\)\s*\.\s*optional\s*\(\s*\)\s*[,\s]*\}\s*\)\s*\.\s*strict\s*\(\s*\)/; exit(1)' "$_f" 2>/dev/null; then
|
||||
echo "$_f"
|
||||
fi
|
||||
done
|
||||
)
|
||||
|
||||
if [[ ${#candidates[@]} -eq 0 ]]; then
|
||||
warn "[Part 1] 未找到 hooks zod schema 目标文件，跳过"
|
||||
return 0
|
||||
elif [[ ${#candidates[@]} -gt 1 ]]; then
|
||||
warn "[Part 1] 发现 ${#candidates[@]} 个匹配文件（预期 1），安全起见跳过"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local target="${candidates[0]}"
|
||||
local relpath="${target#$dist_dir/}"
|
||||
debug "[Part 1] 目标: $relpath"
|
||||
|
||||
# 幂等：目标文件中已包含 allowConversationAccess → 跳过
|
||||
if grep -q 'allowConversationAccess' "$target" 2>/dev/null; then
|
||||
ok "[Part 1] allowConversationAccess 已存在于 $relpath，跳过"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# 备份
|
||||
[[ -f "${target}.pre-aca-patch.bak" ]] || cp "$target" "${target}.pre-aca-patch.bak"
|
||||
|
||||
# 注入：用精确变量名匹配，避免贪婪回溯
|
||||
perl -0777 -i -pe '
|
||||
s/(allowPromptInjection\s*:\s*[a-zA-Z_\$][a-zA-Z0-9_\$]*\s*\.\s*boolean\s*\(\s*\)\s*\.\s*optional\s*\(\s*\))(\s*\}\s*\)\s*\.\s*strict\s*\(\s*\))/$1,allowConversationAccess:z.boolean().optional()$2/
|
||||
' "$target"
|
||||
|
||||
# 验证
|
||||
if grep -q 'allowConversationAccess' "$target" 2>/dev/null; then
|
||||
ok "[Part 1] $relpath — patch 成功"
|
||||
else
|
||||
warn "[Part 1] patch 验证失败，恢复备份"
|
||||
cp "${target}.pre-aca-patch.bak" "$target"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# Part 2: 写入 openclaw.json (不限版本，始终确保配置存在)
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
patch_config_json() {
|
||||
if [[ ! -f "$OPENCLAW_JSON" ]]; then
|
||||
warn "[Part 2] openclaw.json 不存在: $OPENCLAW_JSON，跳过"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# 幂等检测
|
||||
local exists
|
||||
exists=$(python3 -c "
|
||||
import json
|
||||
try:
|
||||
with open('$OPENCLAW_JSON') as f:
|
||||
cfg = json.load(f)
|
||||
val = cfg.get('plugins',{}).get('entries',{}).get('$PLUGIN_ID',{}).get('hooks',{}).get('allowConversationAccess')
|
||||
print('yes' if val is True else 'no')
|
||||
except Exception:
|
||||
print('no')
|
||||
" 2>/dev/null || echo "no")
|
||||
|
||||
if [[ "$exists" == "yes" ]]; then
|
||||
ok "[Part 2] hooks.allowConversationAccess 已存在，跳过"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# 写入
|
||||
python3 -c "
|
||||
import json
|
||||
|
||||
with open('$OPENCLAW_JSON') as f:
|
||||
cfg = json.load(f)
|
||||
|
||||
entry = cfg.setdefault('plugins', {}).setdefault('entries', {}).setdefault('$PLUGIN_ID', {})
|
||||
hooks = entry.setdefault('hooks', {})
|
||||
hooks['allowConversationAccess'] = True
|
||||
|
||||
with open('$OPENCLAW_JSON', 'w') as f:
|
||||
json.dump(cfg, f, indent=2, ensure_ascii=False)
|
||||
f.write('\n')
|
||||
"
|
||||
ok "[Part 2] hooks.allowConversationAccess = true 已写入"
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# 主入口
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
echo ""
|
||||
echo -e "${CYAN}═══════════════════════════════════════════════════════${NC}"
|
||||
echo -e "${CYAN} bugfix-20260423: allowConversationAccess 一键修复${NC}"
|
||||
echo -e "${CYAN} Issue #73806 | 适用版本: OC 2026.4.23${NC}"
|
||||
echo -e "${CYAN}═══════════════════════════════════════════════════════${NC}"
|
||||
echo ""
|
||||
|
||||
# ── Step 1: 停止 Gateway ──
|
||||
info "[Step 1] 停止 Gateway..."
|
||||
openclaw gateway stop 2>/dev/null || true
|
||||
sleep 10
|
||||
|
||||
# 确认已停止
|
||||
if ps aux | grep -v grep | grep -q 'openclaw-gateway'; then
|
||||
warn "检测到 openclaw-gateway 仍在运行，尝试强制停止..."
|
||||
pkill -9 -f 'openclaw-gateway' 2>/dev/null || true
|
||||
sleep 3
|
||||
fi
|
||||
|
||||
if ps aux | grep -v grep | grep -q 'openclaw-gateway'; then
|
||||
fail "[Step 1] 无法停止 openclaw-gateway，请手动处理后重试"
|
||||
fi
|
||||
ok "[Step 1] Gateway 已停止"
|
||||
echo ""
|
||||
|
||||
# ── Step 2: 执行 Patch ──
|
||||
info "[Step 2] 执行 Patch..."
|
||||
if ! patch_dist_js "${1:-}"; then
|
||||
fail "[Step 2] dist JS patch 失败"
|
||||
fi
|
||||
if ! patch_config_json; then
|
||||
fail "[Step 2] openclaw.json 写入失败"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# ── Step 3: 验证 ──
|
||||
info "[Step 3] 验证结果..."
|
||||
echo ""
|
||||
|
||||
# 3.1 验证 openclaw.json
|
||||
info " [3.1] 检查 openclaw.json"
|
||||
if grep -q '"allowConversationAccess"' "$OPENCLAW_JSON" 2>/dev/null; then
|
||||
_json_line=$(grep -n 'allowConversationAccess' "$OPENCLAW_JSON")
|
||||
ok " openclaw.json: allowConversationAccess ✓"
|
||||
echo -e " ${CYAN}${_json_line}${NC}"
|
||||
else
|
||||
fail "[Step 3.1] openclaw.json 中未找到 allowConversationAccess"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 3.2 验证 dist JS — 只检查 zod-schema-BhKK4qYw.js
|
||||
info " [3.2] 检查 zod-schema-BhKK4qYw.js"
|
||||
_oc_dir="$(_resolve_openclaw_dir "${1:-}" 2>/dev/null)" || _oc_dir=""
|
||||
_zod_file="$_oc_dir/dist/zod-schema-BhKK4qYw.js"
|
||||
if [[ -n "$_oc_dir" && -f "$_zod_file" ]]; then
|
||||
_match_count=$(grep -c 'allowConversationAccess' "$_zod_file" 2>/dev/null || echo "0")
|
||||
|
||||
if [[ "$_match_count" -eq 0 ]]; then
|
||||
fail "[Step 3.2] zod-schema-BhKK4qYw.js 中未找到 allowConversationAccess"
|
||||
elif [[ "$_match_count" -eq 1 ]]; then
|
||||
ok " zod-schema-BhKK4qYw.js: allowConversationAccess 出现 1 次 ✓"
|
||||
echo -e " ${CYAN}匹配行:${NC}"
|
||||
grep -n 'allowConversationAccess' "$_zod_file" | head -1 | sed 's/^/ /'
|
||||
else
|
||||
fail "[Step 3.2] zod-schema-BhKK4qYw.js 中 allowConversationAccess 出现 $_match_count 次（预期 1 次），可能重复注入"
|
||||
fi
|
||||
else
|
||||
fail "[Step 3.2] 文件不存在: $_zod_file"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# ── Step 4: 重启 Gateway ──
|
||||
info "[Step 4] 重启 Gateway..."
|
||||
openclaw gateway start
|
||||
sleep 10
|
||||
|
||||
if ps aux | grep -v grep | grep -q 'openclaw-gateway'; then
|
||||
ok "[Step 4] Gateway 已启动 (pid=$(pgrep -f 'openclaw-gateway' | head -1))"
|
||||
else
|
||||
fail "[Step 4] Gateway 启动失败，请检查日志"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}═══════════════════════════════════════════════════════${NC}"
|
||||
echo -e "${GREEN} ✅ bugfix-20260423 修复完成，Gateway 已重启${NC}"
|
||||
echo -e "${GREEN}═══════════════════════════════════════════════════════${NC}"
|
||||
echo ""
|
||||
@@ -0,0 +1,218 @@
|
||||
#!/usr/bin/env bash
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# bugfix-20260423.sh — OC 2026.4.23 allowConversationAccess 修复
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# Issue #73806: OC 2026.4.23 的 Zod schema 使用 .strict() 拒绝
|
||||
# hooks.allowConversationAccess 字段，导致非捆绑插件无法注册会话钩子
|
||||
# (llm_input, llm_output, agent_end)。PR #71221 在 4.24 修复。
|
||||
#
|
||||
# 本脚本做两件事（均幂等，可安全重复执行）：
|
||||
# 1. Patch dist JS: 给 hooks zod schema 注入 allowConversationAccess 字段
|
||||
# 2. 写 openclaw.json: 设置 hooks.allowConversationAccess = true
|
||||
#
|
||||
# 版本限制：仅在 OC 2026.4.23 上执行 Part 1，其他版本安全跳过 dist patch。
|
||||
# Part 2 (配置写入) 不限版本，始终确保配置存在。
|
||||
#
|
||||
# 用法：
|
||||
# bash bugfix-20260423.sh [/path/to/openclaw]
|
||||
#
|
||||
# 环境变量：
|
||||
# OPENCLAW_DIR — 覆盖 openclaw 安装路径（优先于参数）
|
||||
# OPENCLAW_JSON — 覆盖配置文件路径（默认 ~/.openclaw/openclaw.json）
|
||||
# DEBUG=1 — 开启调试输出
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
set -euo pipefail
|
||||
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m'
|
||||
info() { echo -e "${CYAN}[INFO]${NC} $*"; }
|
||||
ok() { echo -e "${GREEN}[OK]${NC} $*"; }
|
||||
warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
|
||||
fail() { echo -e "${RED}[FAIL]${NC} $*" >&2; exit 1; }
|
||||
debug() { [[ "${DEBUG:-}" == "1" ]] && echo -e "${CYAN}[DEBUG]${NC} $*" || true; }
|
||||
|
||||
PLUGIN_ID="memory-tencentdb"
|
||||
OPENCLAW_JSON="${OPENCLAW_JSON:-${HOME}/.openclaw/openclaw.json}"
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# Part 1: Patch dist JS (仅 2026.4.23)
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
_resolve_openclaw_dir() {
|
||||
# 参数 > 环境变量 > 自动检测
|
||||
if [[ -n "${1:-}" ]]; then
|
||||
echo "$1"; return 0
|
||||
fi
|
||||
if [[ -n "${OPENCLAW_DIR:-}" && -d "${OPENCLAW_DIR}" ]]; then
|
||||
echo "$OPENCLAW_DIR"; return 0
|
||||
fi
|
||||
# 自动定位
|
||||
node -e "
|
||||
const {dirname, join} = require('path');
|
||||
const {realpathSync, existsSync, readFileSync, statSync} = require('fs');
|
||||
function walkUp(start) {
|
||||
let dir = statSync(start).isDirectory() ? start : dirname(start);
|
||||
for (let i = 0; i < 10; i++) {
|
||||
const pj = join(dir, 'package.json');
|
||||
if (existsSync(pj)) {
|
||||
try { if (JSON.parse(readFileSync(pj,'utf8')).name==='openclaw') { console.log(dir); process.exit(0); } } catch {}
|
||||
}
|
||||
const parent = dirname(dir);
|
||||
if (parent === dir) break;
|
||||
dir = parent;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
const {execSync} = require('child_process');
|
||||
const bin = execSync('which openclaw',{encoding:'utf8'}).trim();
|
||||
const real = realpathSync(bin);
|
||||
const found = walkUp(real);
|
||||
if (found) { console.log(found); process.exit(0); }
|
||||
const content = readFileSync(bin,'utf8');
|
||||
const m = content.match(/['\"]([^'\"]*openclaw[^'\"]*\\.(?:js|mjs))['\"]/) ||
|
||||
content.match(/['\"]([^'\"]*openclaw[^'\"]*)['\"].*node/);
|
||||
if (m) { const f = walkUp(realpathSync(m[1])); if (f) { console.log(f); process.exit(0); } }
|
||||
} catch {}
|
||||
const searchDirs = [
|
||||
join(process.env.HOME||'/root','.local/share/pnpm'),
|
||||
join(process.env.HOME||'/root','.local/node/lib/node_modules'),
|
||||
'/usr/local/lib/node_modules','/usr/lib/node_modules',
|
||||
];
|
||||
for (const base of searchDirs) {
|
||||
if (!existsSync(base)) continue;
|
||||
try {
|
||||
const {execSync:e2} = require('child_process');
|
||||
const out = e2('find '+JSON.stringify(base)+' -maxdepth 8 -name package.json -path \"*/openclaw/package.json\" 2>/dev/null',{encoding:'utf8',timeout:5000}).trim();
|
||||
for (const line of out.split('\\n')) {
|
||||
if (!line) continue;
|
||||
try { if (JSON.parse(readFileSync(line,'utf8')).name==='openclaw') { console.log(dirname(line)); process.exit(0); } } catch {}
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
process.exit(1);
|
||||
" 2>/dev/null
|
||||
}
|
||||
|
||||
patch_dist_js() {
|
||||
local oc_dir
|
||||
oc_dir="$(_resolve_openclaw_dir "${1:-}")" || {
|
||||
warn "[Part 1] 找不到 OpenClaw 安装目录，跳过 dist patch"
|
||||
return 0
|
||||
}
|
||||
|
||||
local dist_dir="$oc_dir/dist"
|
||||
[[ -d "$dist_dir" ]] || { warn "[Part 1] dist 目录不存在: $dist_dir，跳过"; return 0; }
|
||||
|
||||
local version
|
||||
version=$(grep -oP '"version"\s*:\s*"\K[^"]+' "$oc_dir/package.json" 2>/dev/null || echo "unknown")
|
||||
info "[Part 1] OpenClaw 版本: $version"
|
||||
|
||||
# 版本门控：仅 2026.4.23
|
||||
if [[ ! "$version" =~ ^2026\.4\.23($|[-\.]) ]]; then
|
||||
ok "[Part 1] 版本 $version 不需要 schema patch，跳过"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# 精确定位：hooks zod schema 的唯一特征
|
||||
local -a candidates
|
||||
mapfile -t candidates < <(
|
||||
grep -rl 'allowPromptInjection' "$dist_dir" --include='*.js' 2>/dev/null | while read -r _f; do
|
||||
if perl -0777 -ne 'exit(0) if /allowPromptInjection\s*:\s*[a-zA-Z_\$][a-zA-Z0-9_\$]*\s*\.\s*boolean\s*\(\s*\)\s*\.\s*optional\s*\(\s*\)\s*[,\s]*\}\s*\)\s*\.\s*strict\s*\(\s*\)/; exit(1)' "$_f" 2>/dev/null; then
|
||||
echo "$_f"
|
||||
fi
|
||||
done
|
||||
)
|
||||
|
||||
if [[ ${#candidates[@]} -eq 0 ]]; then
|
||||
warn "[Part 1] 未找到 hooks zod schema 目标文件，跳过"
|
||||
return 0
|
||||
elif [[ ${#candidates[@]} -gt 1 ]]; then
|
||||
warn "[Part 1] 发现 ${#candidates[@]} 个匹配文件（预期 1），安全起见跳过"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local target="${candidates[0]}"
|
||||
local relpath="${target#$dist_dir/}"
|
||||
debug "[Part 1] 目标: $relpath"
|
||||
|
||||
# 幂等：目标文件中已包含 allowConversationAccess → 跳过
|
||||
if grep -q 'allowConversationAccess' "$target" 2>/dev/null; then
|
||||
ok "[Part 1] allowConversationAccess 已存在于 $relpath，跳过"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# 备份
|
||||
[[ -f "${target}.pre-aca-patch.bak" ]] || cp "$target" "${target}.pre-aca-patch.bak"
|
||||
|
||||
# 注入：用精确变量名匹配，避免贪婪回溯
|
||||
perl -0777 -i -pe '
|
||||
s/(allowPromptInjection\s*:\s*[a-zA-Z_\$][a-zA-Z0-9_\$]*\s*\.\s*boolean\s*\(\s*\)\s*\.\s*optional\s*\(\s*\))(\s*\}\s*\)\s*\.\s*strict\s*\(\s*\))/$1,allowConversationAccess:z.boolean().optional()$2/
|
||||
' "$target"
|
||||
|
||||
# 验证
|
||||
if grep -q 'allowConversationAccess' "$target" 2>/dev/null; then
|
||||
ok "[Part 1] $relpath — patch 成功"
|
||||
else
|
||||
warn "[Part 1] patch 验证失败，恢复备份"
|
||||
cp "${target}.pre-aca-patch.bak" "$target"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# Part 2: 写入 openclaw.json (不限版本，始终确保配置存在)
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
patch_config_json() {
|
||||
if [[ ! -f "$OPENCLAW_JSON" ]]; then
|
||||
warn "[Part 2] openclaw.json 不存在: $OPENCLAW_JSON，跳过"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# 幂等检测
|
||||
local exists
|
||||
exists=$(python3 -c "
|
||||
import json
|
||||
try:
|
||||
with open('$OPENCLAW_JSON') as f:
|
||||
cfg = json.load(f)
|
||||
val = cfg.get('plugins',{}).get('entries',{}).get('$PLUGIN_ID',{}).get('hooks',{}).get('allowConversationAccess')
|
||||
print('yes' if val is True else 'no')
|
||||
except Exception:
|
||||
print('no')
|
||||
" 2>/dev/null || echo "no")
|
||||
|
||||
if [[ "$exists" == "yes" ]]; then
|
||||
ok "[Part 2] hooks.allowConversationAccess 已存在，跳过"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# 写入
|
||||
python3 -c "
|
||||
import json
|
||||
|
||||
with open('$OPENCLAW_JSON') as f:
|
||||
cfg = json.load(f)
|
||||
|
||||
entry = cfg.setdefault('plugins', {}).setdefault('entries', {}).setdefault('$PLUGIN_ID', {})
|
||||
hooks = entry.setdefault('hooks', {})
|
||||
hooks['allowConversationAccess'] = True
|
||||
|
||||
with open('$OPENCLAW_JSON', 'w') as f:
|
||||
json.dump(cfg, f, indent=2, ensure_ascii=False)
|
||||
f.write('\n')
|
||||
"
|
||||
ok "[Part 2] hooks.allowConversationAccess = true 已写入"
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# 主入口
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
info "── bugfix-20260423: allowConversationAccess ──"
|
||||
|
||||
patch_dist_js "${1:-}"
|
||||
patch_config_json
|
||||
|
||||
echo ""
|
||||
ok "bugfix-20260423 完成"
|
||||
@@ -0,0 +1,240 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
STS 临时密钥权限排查脚本
|
||||
从 Shark 获取 COS 凭证，逐项测试各种 COS 操作权限。
|
||||
|
||||
用法: python3 check_sts_permissions.py
|
||||
依赖: pip install cos-python-sdk-v5
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import urllib.request
|
||||
import traceback
|
||||
|
||||
SHARK_URL = "http://tdai.gateway.cd.test.polaris:8000/meta/GetMemoryPlusCosConfig"
|
||||
INSTANCE_ID = "mem-rkgqhd5z"
|
||||
|
||||
|
||||
def fetch_cos_config():
|
||||
"""从 Shark 获取 COS 配置"""
|
||||
req = urllib.request.Request(
|
||||
SHARK_URL,
|
||||
data=b"{}",
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
resp = urllib.request.urlopen(req, timeout=10)
|
||||
payload = json.loads(resp.read())
|
||||
print("=" * 60)
|
||||
print("Shark 原始返回:")
|
||||
print(json.dumps(payload, indent=2, ensure_ascii=False))
|
||||
print("=" * 60)
|
||||
|
||||
if payload.get("code") != 0:
|
||||
print(f"Shark 返回错误: code={payload.get('code')}, message={payload.get('message')}")
|
||||
sys.exit(1)
|
||||
|
||||
return payload["data"]
|
||||
|
||||
|
||||
def parse_cos_url(cos_url):
|
||||
"""解析 COS URL 提取 bucket 和 region"""
|
||||
host = cos_url.replace("https://", "").replace("http://", "").rstrip("/")
|
||||
m = re.match(r"^(.+?)\.cos(?:-internal)?\.(.+?)\.(?:myqcloud\.com|tencentcos\.cn)$", host)
|
||||
if not m:
|
||||
raise ValueError(f"无法解析 COS URL: {cos_url}")
|
||||
return m.group(1), m.group(2)
|
||||
|
||||
|
||||
def test_permissions(data):
|
||||
"""逐项测试 COS 操作权限"""
|
||||
try:
|
||||
from qcloud_cos import CosConfig, CosS3Client
|
||||
except ImportError:
|
||||
print("\n请先安装 COS SDK: pip install cos-python-sdk-v5")
|
||||
sys.exit(1)
|
||||
|
||||
cos_url = data["CosUrl"]
|
||||
secret_id = data["TmpSecretId"]
|
||||
secret_key = data["TmpSecretKey"]
|
||||
token = data["TmpToken"]
|
||||
path_prefix = data["PathPrefix"]
|
||||
expiration = data.get("ExpirationTime", "未知")
|
||||
|
||||
bucket, region = parse_cos_url(cos_url)
|
||||
|
||||
print(f"\n凭证信息:")
|
||||
print(f" CosUrl: {cos_url}")
|
||||
print(f" Bucket: {bucket}")
|
||||
print(f" Region: {region}")
|
||||
print(f" TmpSecretId: {secret_id[:10]}...{secret_id[-4:]}")
|
||||
print(f" TmpSecretKey: {secret_key[:6]}...{secret_key[-4:]}")
|
||||
print(f" TmpToken: {'有' if token else '无'} (长度={len(token) if token else 0})")
|
||||
print(f" PathPrefix: {path_prefix}")
|
||||
print(f" Expiration: {expiration}")
|
||||
|
||||
# 实际 prefix（按 memory service 逻辑拼接）
|
||||
prefix = f"{path_prefix.rstrip('/')}/{INSTANCE_ID}/"
|
||||
print(f" 实际 Key 前缀: {prefix}")
|
||||
|
||||
# 同时测试外网域名（开发机可能无法访问 cos-internal）
|
||||
# 如果 CosUrl 是内网域名，先用外网试
|
||||
use_public = "cos-internal" in cos_url or "tencentcos.cn" in cos_url
|
||||
if use_public:
|
||||
print(f"\n 注意: CosUrl 为内网域名，将同时用外网域名测试")
|
||||
|
||||
config = CosConfig(
|
||||
Region=region,
|
||||
SecretId=secret_id,
|
||||
SecretKey=secret_key,
|
||||
Token=token,
|
||||
)
|
||||
client = CosS3Client(config)
|
||||
|
||||
test_key = f"{prefix}__permission_test__.txt"
|
||||
results = {}
|
||||
|
||||
# ── Test 1: GetService (列出所有 Bucket) ──
|
||||
print(f"\n{'─' * 60}")
|
||||
print("Test 1: GetService (列出所有 Bucket)")
|
||||
try:
|
||||
resp = client.list_buckets()
|
||||
buckets = resp.get("Buckets", {}).get("Bucket", [])
|
||||
results["GetService"] = f"OK ({len(buckets)} buckets)"
|
||||
print(f" ✅ OK - 可见 {len(buckets)} 个 Bucket")
|
||||
for b in buckets[:5]:
|
||||
print(f" - {b.get('Name', '?')} ({b.get('Location', '?')})")
|
||||
if len(buckets) > 5:
|
||||
print(f" ... 及其他 {len(buckets) - 5} 个")
|
||||
except Exception as e:
|
||||
results["GetService"] = f"DENIED ({e})"
|
||||
print(f" ❌ DENIED - {e}")
|
||||
|
||||
# ── Test 2: HeadBucket (检查 Bucket 是否存在/可访问) ──
|
||||
print(f"\nTest 2: HeadBucket (Bucket={bucket})")
|
||||
try:
|
||||
client.head_bucket(Bucket=bucket)
|
||||
results["HeadBucket"] = "OK"
|
||||
print(f" ✅ OK - Bucket 存在且可访问")
|
||||
except Exception as e:
|
||||
results["HeadBucket"] = f"DENIED ({e})"
|
||||
print(f" ❌ DENIED - {e}")
|
||||
|
||||
# ── Test 3: GetBucket / ListObjects (在 prefix 下列出对象) ──
|
||||
print(f"\nTest 3: ListObjects (Prefix={prefix})")
|
||||
try:
|
||||
resp = client.list_objects(Bucket=bucket, Prefix=prefix, MaxKeys=10)
|
||||
contents = resp.get("Contents", [])
|
||||
results["ListObjects"] = f"OK ({len(contents)} objects)"
|
||||
print(f" ✅ OK - 列出 {len(contents)} 个对象")
|
||||
for c in contents[:5]:
|
||||
print(f" - {c['Key']} ({c['Size']} bytes)")
|
||||
except Exception as e:
|
||||
results["ListObjects"] = f"DENIED ({e})"
|
||||
print(f" ❌ DENIED - {e}")
|
||||
|
||||
# ── Test 3b: ListObjects 在 PathPrefix 下（不带 instanceId）──
|
||||
raw_prefix = path_prefix.rstrip("/") + "/"
|
||||
print(f"\nTest 3b: ListObjects (Prefix={raw_prefix}, 不带 instanceId)")
|
||||
try:
|
||||
resp = client.list_objects(Bucket=bucket, Prefix=raw_prefix, MaxKeys=10)
|
||||
contents = resp.get("Contents", [])
|
||||
results["ListObjects(raw)"] = f"OK ({len(contents)} objects)"
|
||||
print(f" ✅ OK - 列出 {len(contents)} 个对象")
|
||||
for c in contents[:5]:
|
||||
print(f" - {c['Key']} ({c['Size']} bytes)")
|
||||
except Exception as e:
|
||||
results["ListObjects(raw)"] = f"DENIED ({e})"
|
||||
print(f" ❌ DENIED - {e}")
|
||||
|
||||
# ── Test 3c: ListObjects 在 bucket 根目录 ──
|
||||
print(f"\nTest 3c: ListObjects (Prefix='', Bucket 根目录)")
|
||||
try:
|
||||
resp = client.list_objects(Bucket=bucket, Prefix="", MaxKeys=10)
|
||||
contents = resp.get("Contents", [])
|
||||
results["ListObjects(root)"] = f"OK ({len(contents)} objects)"
|
||||
print(f" ✅ OK - 列出 {len(contents)} 个对象")
|
||||
for c in contents[:5]:
|
||||
print(f" - {c['Key']} ({c['Size']} bytes)")
|
||||
except Exception as e:
|
||||
results["ListObjects(root)"] = f"DENIED ({e})"
|
||||
print(f" ❌ DENIED - {e}")
|
||||
|
||||
# ── Test 4: PutObject (写入测试文件) ──
|
||||
print(f"\nTest 4: PutObject (Key={test_key})")
|
||||
try:
|
||||
resp = client.put_object(
|
||||
Bucket=bucket,
|
||||
Body=b"permission test",
|
||||
Key=test_key,
|
||||
ContentType="text/plain",
|
||||
)
|
||||
results["PutObject"] = f"OK (ETag={resp.get('ETag', '?')})"
|
||||
print(f" ✅ OK - ETag={resp.get('ETag', '?')}")
|
||||
except Exception as e:
|
||||
results["PutObject"] = f"DENIED ({e})"
|
||||
print(f" ❌ DENIED - {e}")
|
||||
|
||||
# ── Test 5: GetObject (读取刚写入的文件) ──
|
||||
print(f"\nTest 5: GetObject (Key={test_key})")
|
||||
try:
|
||||
resp = client.get_object(Bucket=bucket, Key=test_key)
|
||||
body = resp["Body"].get_raw_stream().read()
|
||||
results["GetObject"] = f"OK ({len(body)} bytes)"
|
||||
print(f" ✅ OK - 读到 {len(body)} 字节: {body[:50]}")
|
||||
except Exception as e:
|
||||
results["GetObject"] = f"DENIED ({e})"
|
||||
print(f" ❌ DENIED - {e}")
|
||||
|
||||
# ── Test 6: HeadObject (获取文件元数据) ──
|
||||
print(f"\nTest 6: HeadObject (Key={test_key})")
|
||||
try:
|
||||
resp = client.head_object(Bucket=bucket, Key=test_key)
|
||||
results["HeadObject"] = f"OK (size={resp.get('Content-Length', '?')})"
|
||||
print(f" ✅ OK - Content-Length={resp.get('Content-Length', '?')}")
|
||||
except Exception as e:
|
||||
results["HeadObject"] = f"DENIED ({e})"
|
||||
print(f" ❌ DENIED - {e}")
|
||||
|
||||
# ── Test 7: DeleteObject (删除测试文件) ──
|
||||
print(f"\nTest 7: DeleteObject (Key={test_key})")
|
||||
try:
|
||||
client.delete_object(Bucket=bucket, Key=test_key)
|
||||
results["DeleteObject"] = "OK"
|
||||
print(f" ✅ OK - 删除成功")
|
||||
except Exception as e:
|
||||
results["DeleteObject"] = f"DENIED ({e})"
|
||||
print(f" ❌ DENIED - {e}")
|
||||
|
||||
# ── 汇总 ──
|
||||
print(f"\n{'=' * 60}")
|
||||
print("权限汇总:")
|
||||
print(f"{'=' * 60}")
|
||||
for op, status in results.items():
|
||||
icon = "✅" if status.startswith("OK") else "❌"
|
||||
print(f" {icon} {op:25s} → {status}")
|
||||
print(f"{'=' * 60}")
|
||||
|
||||
ok_count = sum(1 for s in results.values() if s.startswith("OK"))
|
||||
total = len(results)
|
||||
print(f"\n结论: {ok_count}/{total} 项操作有权限")
|
||||
|
||||
if ok_count == 0:
|
||||
print("⚠️ STS 临时密钥无任何 COS 操作权限！需要 Shark 团队修复 STS Policy。")
|
||||
print(" 所需权限: cos:GetObject, cos:PutObject, cos:DeleteObject, cos:GetBucket, cos:HeadObject")
|
||||
elif ok_count < total:
|
||||
denied = [op for op, s in results.items() if not s.startswith("OK")]
|
||||
print(f"⚠️ 部分操作无权限: {', '.join(denied)}")
|
||||
else:
|
||||
print("✅ STS 临时密钥拥有完整的 COS 操作权限，write_scenario 应该可以正常工作。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("STS 临时密钥权限排查")
|
||||
print(f"Shark 地址: {SHARK_URL}")
|
||||
print(f"实例 ID: {INSTANCE_ID}")
|
||||
|
||||
data = fetch_cos_config()
|
||||
test_permissions(data)
|
||||
Executable
+227
@@ -0,0 +1,227 @@
|
||||
#!/usr/bin/env bash
|
||||
# OpenClaw + memory-tencentdb（原 memory-tdai）诊断数据导出脚本
|
||||
# 注：插件已更名为 memory-tencentdb，但数据目录始终为 memory-tdai（代码硬编码）
|
||||
# 用法: bash export-diagnostic.sh [输出目录]
|
||||
# 默认输出到 ~/Downloads/openclaw-diagnostic-<timestamp>/
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# ── 参数 ──
|
||||
OUTPUT_BASE="${1:-$HOME/Downloads}"
|
||||
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
|
||||
EXPORT_DIR="${OUTPUT_BASE}/openclaw-diagnostic-${TIMESTAMP}"
|
||||
ARCHIVE_PATH="${EXPORT_DIR}.tar.gz"
|
||||
|
||||
# ── OpenClaw 工作目录探测 ──
|
||||
if [ -n "${OPENCLAW_STATE_DIR:-}" ]; then
|
||||
STATE_DIR="$OPENCLAW_STATE_DIR"
|
||||
elif [ -d "$HOME/.openclaw" ]; then
|
||||
STATE_DIR="$HOME/.openclaw"
|
||||
elif [ -d "$HOME/.clawdbot" ]; then
|
||||
STATE_DIR="$HOME/.clawdbot"
|
||||
else
|
||||
echo "❌ 未找到 OpenClaw 工作目录 (~/.openclaw 或 ~/.clawdbot)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "📂 OpenClaw 工作目录: $STATE_DIR"
|
||||
echo "📦 导出目录: $EXPORT_DIR"
|
||||
|
||||
mkdir -p "$EXPORT_DIR"
|
||||
|
||||
# ── 1. 收集环境信息 ──
|
||||
echo "🔍 收集环境信息..."
|
||||
{
|
||||
echo "=== 导出时间 ==="
|
||||
date -Iseconds 2>/dev/null || date
|
||||
echo ""
|
||||
echo "=== 系统信息 ==="
|
||||
echo "OS: $(uname -a)"
|
||||
echo "Node: $(node --version 2>/dev/null || echo 'not found')"
|
||||
echo "pnpm: $(pnpm --version 2>/dev/null || echo 'not found')"
|
||||
echo ""
|
||||
echo "=== OpenClaw 版本 ==="
|
||||
openclaw --version 2>/dev/null || pnpm openclaw --version 2>/dev/null || echo "(unknown)"
|
||||
echo ""
|
||||
echo "=== 工作目录 ==="
|
||||
echo "STATE_DIR: $STATE_DIR"
|
||||
echo ""
|
||||
echo "=== 目录结构 ==="
|
||||
ls -la "$STATE_DIR/" 2>/dev/null || echo "(empty)"
|
||||
echo ""
|
||||
echo "=== memory-tdai 目录结构 ==="
|
||||
ls -laR "$STATE_DIR/memory-tdai/" 2>/dev/null || echo "(not found)"
|
||||
echo ""
|
||||
echo "=== 磁盘占用 ==="
|
||||
du -sh "$STATE_DIR/memory-tdai/"* 2>/dev/null || echo "(not found)"
|
||||
} > "$EXPORT_DIR/env-info.txt" 2>&1
|
||||
|
||||
# ── 2. 收集 OpenClaw 日志 ──
|
||||
echo "📋 收集 OpenClaw 日志..."
|
||||
mkdir -p "$EXPORT_DIR/logs"
|
||||
|
||||
# 网关日志 (~/.openclaw/logs/)
|
||||
if [ -d "$STATE_DIR/logs" ]; then
|
||||
cp -r "$STATE_DIR/logs/" "$EXPORT_DIR/logs/gateway-logs/" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# 滚动日志 (/tmp/openclaw/)
|
||||
TMP_LOG_DIR="/tmp/openclaw"
|
||||
if [ -d "$TMP_LOG_DIR" ]; then
|
||||
mkdir -p "$EXPORT_DIR/logs/rolling-logs"
|
||||
# 只取最近 3 个日志文件
|
||||
ls -t "$TMP_LOG_DIR"/openclaw-*.log 2>/dev/null | head -3 | while read -r f; do
|
||||
# 每个文件只取最后 5000 行，避免过大
|
||||
tail -5000 "$f" > "$EXPORT_DIR/logs/rolling-logs/$(basename "$f")" 2>/dev/null || true
|
||||
done
|
||||
fi
|
||||
|
||||
# ── 3. 收集记忆插件数据 ──
|
||||
# 注：数据目录名为 memory-tdai（历史原因，插件更名为 memory-tencentdb 后未改目录名）
|
||||
echo "🧠 收集记忆插件数据..."
|
||||
MEMORY_DIR="$STATE_DIR/memory-tdai"
|
||||
if [ -d "$MEMORY_DIR" ]; then
|
||||
mkdir -p "$EXPORT_DIR/memory-tdai"
|
||||
|
||||
# L0 对话记录 (JSONL)
|
||||
if [ -d "$MEMORY_DIR/conversations" ]; then
|
||||
cp -r "$MEMORY_DIR/conversations/" "$EXPORT_DIR/memory-tdai/conversations/" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# L1 结构化记忆 (JSONL)
|
||||
if [ -d "$MEMORY_DIR/records" ]; then
|
||||
cp -r "$MEMORY_DIR/records/" "$EXPORT_DIR/memory-tdai/records/" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# L2 场景文件 (Markdown)
|
||||
if [ -d "$MEMORY_DIR/scene_blocks" ]; then
|
||||
cp -r "$MEMORY_DIR/scene_blocks/" "$EXPORT_DIR/memory-tdai/scene_blocks/" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# L3 用户画像
|
||||
[ -f "$MEMORY_DIR/persona.md" ] && cp "$MEMORY_DIR/persona.md" "$EXPORT_DIR/memory-tdai/" 2>/dev/null || true
|
||||
|
||||
# checkpoint + scene_index
|
||||
if [ -d "$MEMORY_DIR/.metadata" ]; then
|
||||
cp -r "$MEMORY_DIR/.metadata/" "$EXPORT_DIR/memory-tdai/.metadata/" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# SQLite 数据库（用于检查向量/FTS索引状态）
|
||||
[ -f "$MEMORY_DIR/vectors.db" ] && cp "$MEMORY_DIR/vectors.db" "$EXPORT_DIR/memory-tdai/" 2>/dev/null || true
|
||||
|
||||
# 备份目录（可选，可能较大）
|
||||
if [ -d "$MEMORY_DIR/.backup" ]; then
|
||||
cp -r "$MEMORY_DIR/.backup/" "$EXPORT_DIR/memory-tdai/.backup/" 2>/dev/null || true
|
||||
fi
|
||||
else
|
||||
echo " ⚠️ 未找到 memory-tdai 数据目录（memory-tencentdb 插件的数据也存储在此目录）"
|
||||
fi
|
||||
|
||||
# ── 4. 收集 OpenClaw 配置（脱敏） ──
|
||||
echo "🔧 收集 OpenClaw 配置（已脱敏）..."
|
||||
CONFIG_FILE="$STATE_DIR/openclaw.json"
|
||||
if [ -f "$CONFIG_FILE" ]; then
|
||||
# 使用 node 脱敏处理配置
|
||||
node -e "
|
||||
const fs = require('fs');
|
||||
const JSON5 = (() => { try { return require('json5'); } catch { return JSON; } })();
|
||||
const raw = fs.readFileSync('$CONFIG_FILE', 'utf-8');
|
||||
let cfg;
|
||||
try { cfg = JSON5.parse(raw); } catch { cfg = JSON.parse(raw); }
|
||||
|
||||
// 递归脱敏函数
|
||||
function redact(obj, path) {
|
||||
if (!obj || typeof obj !== 'object') return obj;
|
||||
if (Array.isArray(obj)) return obj.map((v, i) => redact(v, path + '[' + i + ']'));
|
||||
const result = {};
|
||||
for (const [k, v] of Object.entries(obj)) {
|
||||
const fullPath = path ? path + '.' + k : k;
|
||||
// 脱敏规则：API key、token、password、secret 类字段
|
||||
if (/api_?key|token|password|secret|credential/i.test(k) && typeof v === 'string') {
|
||||
result[k] = v.length > 0 ? '***REDACTED(' + v.length + 'chars)***' : '';
|
||||
}
|
||||
// 脱敏 SecretRef 对象
|
||||
else if (v && typeof v === 'object' && v.source && v.id && v.provider) {
|
||||
result[k] = { source: v.source, provider: v.provider, id: '***REDACTED***' };
|
||||
}
|
||||
// 整体跳过的顶层敏感块
|
||||
else if (['models', 'secrets', 'channels', 'env'].includes(k) && !path) {
|
||||
result[k] = '***REDACTED_SECTION(use openclaw config get ' + k + ' to inspect)***';
|
||||
}
|
||||
// gateway.auth 内的 token/password
|
||||
else if (path === 'gateway.auth' && /token|password/i.test(k)) {
|
||||
result[k] = typeof v === 'string' ? '***REDACTED***' : v;
|
||||
}
|
||||
else {
|
||||
result[k] = redact(v, fullPath);
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
const redacted = redact(cfg, '');
|
||||
// plugins 已经过递归 redact()，其中 apiKey/token/password/secret 等字段
|
||||
// 会被自动脱敏，同时保留 provider/model/enabled 等排查所需的非敏感配置
|
||||
|
||||
fs.writeFileSync('$EXPORT_DIR/openclaw-config-redacted.json', JSON.stringify(redacted, null, 2));
|
||||
console.log(' ✅ 配置已脱敏导出');
|
||||
" 2>&1 || {
|
||||
echo " ⚠️ Node 脱敏失败，使用 grep 粗略脱敏"
|
||||
# 粗略脱敏：删除包含敏感关键字的行
|
||||
grep -v -iE '(api.?key|token|password|secret|credential).*:.*"[^"]{8,}"' "$CONFIG_FILE" \
|
||||
| sed -E 's/"(models|secrets|channels|env)"\s*:\s*\{[^}]*\}/"__REDACTED_SECTION__"/g' \
|
||||
> "$EXPORT_DIR/openclaw-config-redacted.json" 2>/dev/null || true
|
||||
}
|
||||
else
|
||||
echo " ⚠️ 未找到配置文件"
|
||||
fi
|
||||
|
||||
# ── 5. 收集插件安装信息 ──
|
||||
echo "🔌 收集插件安装信息..."
|
||||
if [ -d "$STATE_DIR/extensions" ]; then
|
||||
{
|
||||
echo "=== 已安装插件 ==="
|
||||
ls -la "$STATE_DIR/extensions/" 2>/dev/null
|
||||
echo ""
|
||||
for ext_dir in "$STATE_DIR/extensions"/*/; do
|
||||
[ -d "$ext_dir" ] || continue
|
||||
pkg="$ext_dir/node_modules/openclaw/package.json"
|
||||
plugin_pkg="$ext_dir/package.json"
|
||||
echo "--- $(basename "$ext_dir") ---"
|
||||
if [ -f "$plugin_pkg" ]; then
|
||||
node -e "const p=require('$plugin_pkg'); console.log('name:', p.name, 'version:', p.version)" 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
} > "$EXPORT_DIR/plugins-info.txt" 2>&1
|
||||
fi
|
||||
|
||||
# ── 6. 打包 ──
|
||||
echo "📦 打包中..."
|
||||
cd "$(dirname "$EXPORT_DIR")"
|
||||
tar -czf "$ARCHIVE_PATH" "$(basename "$EXPORT_DIR")"
|
||||
|
||||
# 计算大小
|
||||
ARCHIVE_SIZE=$(du -sh "$ARCHIVE_PATH" | cut -f1)
|
||||
|
||||
echo ""
|
||||
echo "═══════════════════════════════════════════════════"
|
||||
echo " ✅ 诊断数据导出完成"
|
||||
echo "═══════════════════════════════════════════════════"
|
||||
echo ""
|
||||
echo " 📦 压缩包: $ARCHIVE_PATH"
|
||||
echo " 📏 大小: $ARCHIVE_SIZE"
|
||||
echo ""
|
||||
echo " 包含内容:"
|
||||
echo " - env-info.txt — 环境信息、目录结构"
|
||||
echo " - logs/ — OpenClaw 网关日志 + 滚动日志"
|
||||
echo " - memory-tdai/ — 记忆插件全量数据 (L0~L3 + SQLite)"
|
||||
echo " - openclaw-config-redacted.json — 脱敏后的配置文件"
|
||||
echo " - plugins-info.txt — 插件安装信息"
|
||||
echo ""
|
||||
echo " ⚠️ 安全提示:"
|
||||
echo " - 配置文件已自动脱敏（API Key、Token、Password 等已移除）"
|
||||
echo " - models/secrets/channels/env 等敏感配置块已整体替换"
|
||||
echo " - 记忆数据中可能包含用户对话内容，请确认后再发送"
|
||||
echo ""
|
||||
echo " 📤 请手动检查后发送给研发团队"
|
||||
echo "═══════════════════════════════════════════════════"
|
||||
@@ -0,0 +1,634 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* 腾讯云 VDB (Tencent VectorDB) 数据导出脚本
|
||||
*
|
||||
* 连接腾讯云向量数据库实例，查询指定数据库下 collection 的文档，导出为 .jsonl 文件。
|
||||
* 仅支持腾讯云向量数据库（Tencent VectorDB），不支持其他厂商的向量数据库。
|
||||
*
|
||||
* 所有连接参数通过 CLI 传入，无需 .env 文件。
|
||||
*
|
||||
* 用法：
|
||||
* node ./bin/export-tencent-vdb.mjs --url <地址> --username <用户名> --api-key <密钥> --database <库名>
|
||||
* node ./bin/export-tencent-vdb.mjs --url <地址> --username <用户名> --api-key <密钥> --database <库名> --probe
|
||||
* node ./bin/export-tencent-vdb.mjs --url <地址> --username <用户名> --api-key <密钥> --database <库名> -c <collection> -o /tmp/backup
|
||||
*
|
||||
* 输出：
|
||||
* 默认输出到当前工作目录下的 ./vdb-export-YYYY-MM-DD/，可通过 -o 指定。
|
||||
* <outputDir>/
|
||||
* ├── <collection>.jsonl — 每行一个 JSON 文档
|
||||
* ├── schemas.json — 导出的 collection 表结构（索引、embedding 配置等）
|
||||
* └── export-meta.json — 导出元信息
|
||||
*
|
||||
* 导出字段说明：
|
||||
* 默认行为：导出所有字段，但跳过 vector（稠密向量，1024维浮点数组，体积大）。
|
||||
* 加 --include-vectors：导出全部字段，包括 vector，不跳过任何内容。
|
||||
* 注：sparse_vector（BM25 稀疏向量）始终导出，不受此开关影响。
|
||||
*
|
||||
* 依赖：Node.js >= 18（内置 fetch）
|
||||
*/
|
||||
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
// ============================================================
|
||||
// CLI 参数解析（含 VDB 连接信息）
|
||||
// ============================================================
|
||||
|
||||
interface VDBConfig {
|
||||
url: string;
|
||||
username: string;
|
||||
apiKey: string;
|
||||
database: string;
|
||||
timeout: number;
|
||||
}
|
||||
|
||||
interface CliArgs {
|
||||
// 连接参数
|
||||
url?: string;
|
||||
username?: string;
|
||||
apiKey?: string;
|
||||
database?: string;
|
||||
timeout: number;
|
||||
// 导出参数
|
||||
output: string;
|
||||
collection?: string;
|
||||
filter?: string;
|
||||
limit?: number;
|
||||
offset: number;
|
||||
includeVectors: boolean;
|
||||
probe: boolean;
|
||||
help: boolean;
|
||||
}
|
||||
|
||||
const PAGE_SIZE = 100;
|
||||
|
||||
function parseArgs(): CliArgs {
|
||||
const args = process.argv.slice(2);
|
||||
const result: CliArgs = {
|
||||
timeout: 30000,
|
||||
output: `./vdb-export-${new Date().toISOString().slice(0, 10)}`,
|
||||
offset: 0,
|
||||
includeVectors: false,
|
||||
probe: false,
|
||||
help: false,
|
||||
};
|
||||
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
switch (args[i]) {
|
||||
case "--url":
|
||||
result.url = args[++i];
|
||||
break;
|
||||
case "--username":
|
||||
result.username = args[++i];
|
||||
break;
|
||||
case "--api-key":
|
||||
result.apiKey = args[++i];
|
||||
break;
|
||||
case "--database":
|
||||
result.database = args[++i];
|
||||
break;
|
||||
case "--timeout":
|
||||
result.timeout = parseInt(args[++i], 10) || 30000;
|
||||
break;
|
||||
case "--output":
|
||||
case "-o":
|
||||
result.output = args[++i];
|
||||
break;
|
||||
case "--collection":
|
||||
case "-c":
|
||||
result.collection = args[++i];
|
||||
break;
|
||||
case "--filter":
|
||||
case "-f":
|
||||
result.filter = args[++i];
|
||||
break;
|
||||
case "--limit":
|
||||
case "-l": {
|
||||
const v = parseInt(args[++i], 10);
|
||||
if (isNaN(v) || v < 1) {
|
||||
console.error(`❌ --limit 必须 >= 1，收到: ${args[i]}`);
|
||||
process.exit(1);
|
||||
}
|
||||
result.limit = v;
|
||||
break;
|
||||
}
|
||||
case "--offset": {
|
||||
const v = parseInt(args[++i], 10);
|
||||
if (isNaN(v) || v < 0) {
|
||||
console.error(`❌ --offset 必须 >= 0，收到: ${args[i]}`);
|
||||
process.exit(1);
|
||||
}
|
||||
result.offset = v;
|
||||
break;
|
||||
}
|
||||
case "--include-vectors":
|
||||
result.includeVectors = true;
|
||||
break;
|
||||
case "--probe":
|
||||
result.probe = true;
|
||||
break;
|
||||
case "--help":
|
||||
case "-h":
|
||||
result.help = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
function validateConfig(args: CliArgs): VDBConfig {
|
||||
const missing: string[] = [];
|
||||
if (!args.url) missing.push("--url");
|
||||
if (!args.username) missing.push("--username");
|
||||
if (!args.apiKey) missing.push("--api-key");
|
||||
if (!args.database) missing.push("--database");
|
||||
|
||||
if (missing.length > 0) {
|
||||
console.error("❌ 缺少必填参数：");
|
||||
for (const k of missing) {
|
||||
console.error(` - ${k}`);
|
||||
}
|
||||
console.error();
|
||||
console.error("示例：");
|
||||
console.error();
|
||||
console.error(' node ./bin/export-tencent-vdb.mjs \\');
|
||||
console.error(' --url "http://your-vdb-host:8100" \\');
|
||||
console.error(' --username "root" \\');
|
||||
console.error(' --api-key "your-api-key" \\');
|
||||
console.error(' --database "your-database"');
|
||||
console.error();
|
||||
console.error("使用 --help 查看完整参数说明。");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return {
|
||||
url: args.url!,
|
||||
username: args.username!,
|
||||
apiKey: args.apiKey!,
|
||||
database: args.database!,
|
||||
timeout: args.timeout,
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
console.log(`
|
||||
腾讯云 VDB (Tencent VectorDB) 数据导出脚本
|
||||
|
||||
用法：
|
||||
node ./bin/export-tencent-vdb.mjs [连接参数] [选项]
|
||||
|
||||
连接参数（必填）：
|
||||
--url <地址> VDB 实例 HTTP 地址（如 http://your-vdb-host:8100）
|
||||
--username <用户名> 认证用户名（如 root）
|
||||
--api-key <密钥> 认证密钥
|
||||
--database <库名> 数据库名称
|
||||
|
||||
选项：
|
||||
--timeout <毫秒> 单次请求超时（默认: 30000）
|
||||
-o, --output <目录> 输出目录（默认: ./vdb-export-YYYY-MM-DD）
|
||||
-c, --collection <全名> 只导出指定 collection（全名匹配，不指定则导出所有）
|
||||
-f, --filter <表达式> VDB Filter 过滤条件（如 'agent_id = "xxx"'）
|
||||
-l, --limit <数量> 最多导出多少条（不指定则导出全部）
|
||||
--offset <偏移> 从第几条开始（默认: 0），须为分页大小的整数倍
|
||||
--include-vectors 保留 vector 稠密向量字段（默认跳过）
|
||||
--probe 仅测试连通性，列出 collection 信息后退出
|
||||
-h, --help 显示帮助
|
||||
|
||||
输出：
|
||||
<outputDir>/
|
||||
├── <collection全名>.jsonl 每行一个 JSON 文档
|
||||
├── schemas.json 表结构
|
||||
└── export-meta.json 导出元信息
|
||||
|
||||
导出字段说明：
|
||||
默认跳过 vector（稠密向量），保留 sparse_vector（BM25）。
|
||||
加 --include-vectors 导出全部字段。
|
||||
|
||||
示例：
|
||||
# 测试连通性
|
||||
node ./bin/export-tencent-vdb.mjs \\
|
||||
--url "http://gz-vdb-xxx:8100" --username root --api-key "xxx" --database mydb \\
|
||||
--probe
|
||||
|
||||
# 全量导出
|
||||
node ./bin/export-tencent-vdb.mjs \\
|
||||
--url "http://gz-vdb-xxx:8100" --username root --api-key "xxx" --database mydb
|
||||
|
||||
# 导出指定 collection 到指定目录
|
||||
node ./bin/export-tencent-vdb.mjs \\
|
||||
--url "http://gz-vdb-xxx:8100" --username root --api-key "xxx" --database mydb \\
|
||||
-c mydb_l0_conversations -o /tmp/backup
|
||||
|
||||
# 带过滤条件
|
||||
node ./bin/export-tencent-vdb.mjs \\
|
||||
--url "http://gz-vdb-xxx:8100" --username root --api-key "xxx" --database mydb \\
|
||||
-f 'role = "user"'
|
||||
`);
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// VDB HTTP Client
|
||||
// ============================================================
|
||||
|
||||
class VDBClient {
|
||||
private baseUrl: string;
|
||||
private authHeader: string;
|
||||
private database: string;
|
||||
private timeout: number;
|
||||
|
||||
constructor(cfg: VDBConfig) {
|
||||
this.baseUrl = cfg.url.replace(/\/$/, "");
|
||||
this.authHeader = `Bearer account=${cfg.username}&api_key=${cfg.apiKey}`;
|
||||
this.database = cfg.database;
|
||||
this.timeout = cfg.timeout;
|
||||
}
|
||||
|
||||
async request<T>(apiPath: string, body: Record<string, unknown>): Promise<T> {
|
||||
const url = `${this.baseUrl}${apiPath}`;
|
||||
|
||||
const resp = await fetch(url, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: this.authHeader,
|
||||
},
|
||||
body: JSON.stringify(body),
|
||||
signal: AbortSignal.timeout(this.timeout),
|
||||
});
|
||||
|
||||
if (!resp.ok) {
|
||||
const text = await resp.text().catch(() => "(unable to read body)");
|
||||
throw new Error(`VDB API error: HTTP ${resp.status} — ${text.slice(0, 500)}`);
|
||||
}
|
||||
|
||||
const json = (await resp.json()) as { code: number; msg: string } & T;
|
||||
if (json.code !== 0) {
|
||||
throw new Error(`VDB API error [${apiPath}]: code=${json.code}, msg=${json.msg}`);
|
||||
}
|
||||
return json;
|
||||
}
|
||||
|
||||
async listCollections(): Promise<
|
||||
Array<{ collection: string; documentCount: number }>
|
||||
> {
|
||||
const result = await this.request<{
|
||||
collections: Array<{
|
||||
collection: string;
|
||||
documentCount: number;
|
||||
[key: string]: unknown;
|
||||
}>;
|
||||
}>("/collection/list", {
|
||||
database: this.database,
|
||||
});
|
||||
return (result.collections || []).map((c) => ({
|
||||
collection: c.collection,
|
||||
documentCount: c.documentCount ?? 0,
|
||||
}));
|
||||
}
|
||||
|
||||
async queryDocuments(
|
||||
collection: string,
|
||||
options: {
|
||||
limit: number;
|
||||
offset: number;
|
||||
filter?: string;
|
||||
retrieveVector?: boolean;
|
||||
},
|
||||
): Promise<{
|
||||
documents: Array<Record<string, unknown>>;
|
||||
count: number;
|
||||
}> {
|
||||
const query: Record<string, unknown> = {
|
||||
limit: options.limit,
|
||||
offset: options.offset,
|
||||
};
|
||||
if (options.filter) {
|
||||
query.filter = options.filter;
|
||||
}
|
||||
if (options.retrieveVector) {
|
||||
query.retrieveVector = true;
|
||||
}
|
||||
|
||||
const result = await this.request<{
|
||||
documents: Array<Record<string, unknown>>;
|
||||
count: number;
|
||||
}>("/document/query", {
|
||||
database: this.database,
|
||||
collection,
|
||||
readConsistency: "strongConsistency",
|
||||
query,
|
||||
});
|
||||
|
||||
return {
|
||||
documents: result.documents || [],
|
||||
count: result.count ?? 0,
|
||||
};
|
||||
}
|
||||
|
||||
async describeCollection(collection: string): Promise<Record<string, unknown>> {
|
||||
const result = await this.request<{
|
||||
collection: Record<string, unknown>;
|
||||
}>("/collection/describe", {
|
||||
database: this.database,
|
||||
collection,
|
||||
});
|
||||
return result.collection || {};
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 导出逻辑
|
||||
// ============================================================
|
||||
|
||||
interface ExportOptions {
|
||||
filter?: string;
|
||||
limit?: number;
|
||||
offset: number;
|
||||
includeVectors: boolean;
|
||||
expectedTotal?: number;
|
||||
}
|
||||
|
||||
async function exportCollection(
|
||||
client: VDBClient,
|
||||
collection: string,
|
||||
outputDir: string,
|
||||
options: ExportOptions,
|
||||
): Promise<{ docCount: number; filePath: string }> {
|
||||
const filePath = path.join(outputDir, `${collection}.jsonl`);
|
||||
const writeStream = fs.createWriteStream(filePath, { encoding: "utf-8" });
|
||||
|
||||
const isRangeMode = options.limit !== undefined;
|
||||
const maxDocs = options.limit ?? Infinity;
|
||||
const pageSize = isRangeMode ? Math.min(options.limit!, PAGE_SIZE) : PAGE_SIZE;
|
||||
|
||||
let currentOffset = options.offset;
|
||||
let totalExported = 0;
|
||||
let hasMore = true;
|
||||
|
||||
console.log(` 📦 ${collection}`);
|
||||
if (options.expectedTotal !== undefined) {
|
||||
console.log(` 文档总数: ${options.expectedTotal}`);
|
||||
}
|
||||
if (options.filter) {
|
||||
console.log(` 过滤条件: ${options.filter}`);
|
||||
}
|
||||
if (isRangeMode) {
|
||||
console.log(` 导出范围: offset=${options.offset}, limit=${options.limit}`);
|
||||
}
|
||||
|
||||
while (hasMore && totalExported < maxDocs) {
|
||||
const remaining = maxDocs - totalExported;
|
||||
const thisPageSize = Math.min(pageSize, remaining);
|
||||
|
||||
try {
|
||||
const result = await client.queryDocuments(collection, {
|
||||
limit: thisPageSize,
|
||||
offset: currentOffset,
|
||||
filter: options.filter,
|
||||
retrieveVector: options.includeVectors,
|
||||
});
|
||||
|
||||
const docs = result.documents;
|
||||
if (!docs || docs.length === 0) {
|
||||
hasMore = false;
|
||||
break;
|
||||
}
|
||||
|
||||
for (const doc of docs) {
|
||||
const exportDoc = { ...doc };
|
||||
if (!options.includeVectors) {
|
||||
delete exportDoc.vector;
|
||||
}
|
||||
writeStream.write(JSON.stringify(exportDoc) + "\n");
|
||||
}
|
||||
|
||||
totalExported += docs.length;
|
||||
currentOffset += docs.length;
|
||||
|
||||
if (options.expectedTotal !== undefined && !isRangeMode) {
|
||||
const pct = Math.min(
|
||||
100,
|
||||
Math.round((totalExported / options.expectedTotal) * 100),
|
||||
);
|
||||
process.stdout.write(
|
||||
`\r 进度: ${totalExported}/${options.expectedTotal} (${pct}%)`,
|
||||
);
|
||||
} else {
|
||||
process.stdout.write(`\r 已导出: ${totalExported} 条`);
|
||||
}
|
||||
|
||||
if (docs.length < thisPageSize) {
|
||||
hasMore = false;
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(
|
||||
`\n ❌ 查询失败 (offset=${currentOffset}): ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
hasMore = false;
|
||||
}
|
||||
}
|
||||
|
||||
writeStream.end();
|
||||
await new Promise<void>((resolve) => writeStream.on("finish", resolve));
|
||||
|
||||
console.log(
|
||||
`\n ✅ 完成: ${totalExported} 条 → ${path.basename(filePath)}`,
|
||||
);
|
||||
|
||||
return { docCount: totalExported, filePath };
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// Main
|
||||
// ============================================================
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const args = parseArgs();
|
||||
|
||||
if (args.help) {
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const config = validateConfig(args);
|
||||
|
||||
console.log("╔═══════════════════════════════════════════════════╗");
|
||||
console.log("║ 腾讯云 VDB (Tencent VectorDB) 数据导出工具 ║");
|
||||
console.log("╚═══════════════════════════════════════════════════╝");
|
||||
console.log();
|
||||
console.log(`📌 VDB 地址: ${config.url}`);
|
||||
console.log(`📌 数据库: ${config.database}`);
|
||||
console.log(`📌 输出目录: ${args.output}`);
|
||||
if (args.collection) {
|
||||
console.log(`📌 指定导出: ${args.collection}`);
|
||||
}
|
||||
if (args.filter) {
|
||||
console.log(`📌 过滤条件: ${args.filter}`);
|
||||
}
|
||||
if (args.limit !== undefined) {
|
||||
console.log(`📌 导出上限: ${args.limit} 条`);
|
||||
}
|
||||
if (args.offset > 0) {
|
||||
console.log(`📌 起始偏移: ${args.offset}`);
|
||||
}
|
||||
if (args.includeVectors) {
|
||||
console.log(`📌 包含向量: 是`);
|
||||
}
|
||||
console.log();
|
||||
|
||||
fs.mkdirSync(args.output, { recursive: true });
|
||||
|
||||
const client = new VDBClient(config);
|
||||
|
||||
let allCollections: Array<{ collection: string; documentCount: number }>;
|
||||
try {
|
||||
allCollections = await client.listCollections();
|
||||
} catch (err) {
|
||||
console.error(
|
||||
`❌ 列出 collection 失败: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
let targetCollections: Array<{ collection: string; documentCount: number }>;
|
||||
if (args.collection) {
|
||||
const found = allCollections.find((c) => c.collection === args.collection);
|
||||
if (!found) {
|
||||
console.error(
|
||||
`❌ Collection "${args.collection}" 不存在。可用的 collection：`,
|
||||
);
|
||||
for (const c of allCollections) {
|
||||
console.error(` - ${c.collection} (${c.documentCount} 条)`);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
targetCollections = [found];
|
||||
} else {
|
||||
targetCollections = allCollections;
|
||||
console.log(
|
||||
`🔍 找到 ${targetCollections.length} 个 collection：`,
|
||||
);
|
||||
for (const c of targetCollections) {
|
||||
console.log(` - ${c.collection} (${c.documentCount} 条)`);
|
||||
}
|
||||
}
|
||||
|
||||
if (targetCollections.length === 0) {
|
||||
console.log("⚠️ 数据库中没有 collection，无数据可导出");
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
// --probe 模式：只测试连通性，列出信息后退出
|
||||
if (args.probe) {
|
||||
console.log();
|
||||
console.log("✅ 连通性测试通过");
|
||||
console.log();
|
||||
console.log(` VDB 地址: ${config.url}`);
|
||||
console.log(` 数据库: ${config.database}`);
|
||||
console.log(` Collection: ${targetCollections.length} 个`);
|
||||
const totalDocs = targetCollections.reduce((s, c) => s + c.documentCount, 0);
|
||||
console.log(` 总文档数: ${totalDocs}`);
|
||||
console.log();
|
||||
for (const c of targetCollections) {
|
||||
console.log(` - ${c.collection} (${c.documentCount} 条)`);
|
||||
}
|
||||
console.log();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
console.log();
|
||||
|
||||
// 获取并保存表结构
|
||||
const schemas: Record<string, Record<string, unknown>> = {};
|
||||
console.log("📐 获取表结构...");
|
||||
for (const col of targetCollections) {
|
||||
try {
|
||||
const schema = await client.describeCollection(col.collection);
|
||||
schemas[col.collection] = schema;
|
||||
const indexCount = Array.isArray(schema.indexes) ? schema.indexes.length : 0;
|
||||
const emb = schema.embedding as Record<string, unknown> | undefined;
|
||||
const embInfo = emb ? `embedding=${emb.field}→${emb.model}` : "无 embedding";
|
||||
console.log(` ✅ ${col.collection} (${indexCount} 个索引, ${embInfo})`);
|
||||
} catch (err) {
|
||||
console.error(
|
||||
` ⚠️ ${col.collection} 表结构获取失败: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
console.log();
|
||||
|
||||
const schemaPath = path.join(args.output, "schemas.json");
|
||||
fs.writeFileSync(schemaPath, JSON.stringify(schemas, null, 2) + "\n");
|
||||
|
||||
const exportResults: Array<{
|
||||
collection: string;
|
||||
docCount: number;
|
||||
filePath: string;
|
||||
}> = [];
|
||||
|
||||
for (const col of targetCollections) {
|
||||
try {
|
||||
const result = await exportCollection(client, col.collection, args.output, {
|
||||
filter: args.filter,
|
||||
limit: args.limit,
|
||||
offset: args.offset,
|
||||
includeVectors: args.includeVectors,
|
||||
expectedTotal: col.documentCount,
|
||||
});
|
||||
exportResults.push({ collection: col.collection, ...result });
|
||||
} catch (err) {
|
||||
console.error(
|
||||
`❌ 导出 ${col.collection} 失败: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
exportResults.push({
|
||||
collection: col.collection,
|
||||
docCount: 0,
|
||||
filePath: "",
|
||||
});
|
||||
}
|
||||
console.log();
|
||||
}
|
||||
|
||||
const meta = {
|
||||
exportedAt: new Date().toISOString(),
|
||||
vdbUrl: config.url,
|
||||
database: config.database,
|
||||
filter: args.filter ?? null,
|
||||
offset: args.offset,
|
||||
limit: args.limit ?? null,
|
||||
includeVectors: args.includeVectors,
|
||||
collections: exportResults.map((r) => ({
|
||||
collection: r.collection,
|
||||
documentCount: r.docCount,
|
||||
file: r.filePath ? path.basename(r.filePath) : null,
|
||||
})),
|
||||
totalDocuments: exportResults.reduce((sum, r) => sum + r.docCount, 0),
|
||||
};
|
||||
|
||||
const metaPath = path.join(args.output, "export-meta.json");
|
||||
fs.writeFileSync(metaPath, JSON.stringify(meta, null, 2) + "\n");
|
||||
|
||||
console.log("═══════════════════════════════════════════════════");
|
||||
console.log(" ✅ 导出完成");
|
||||
console.log("═══════════════════════════════════════════════════");
|
||||
console.log();
|
||||
console.log(` 📁 输出目录: ${args.output}`);
|
||||
console.log(` 📊 总文档数: ${meta.totalDocuments}`);
|
||||
for (const r of exportResults) {
|
||||
const status = r.docCount > 0 ? "✅" : "⚠️";
|
||||
console.log(
|
||||
` ${status} ${r.collection}: ${r.docCount} 条`,
|
||||
);
|
||||
}
|
||||
console.log(` 📋 元信息: ${path.basename(metaPath)}`);
|
||||
console.log(` 📐 表结构: ${path.basename(schemaPath)}`);
|
||||
console.log();
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(
|
||||
`\n❌ 导出失败: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "Node16",
|
||||
"moduleResolution": "Node16",
|
||||
"outDir": "./dist",
|
||||
"rootDir": ".",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true,
|
||||
"types": ["node"],
|
||||
"declaration": false,
|
||||
"sourceMap": false
|
||||
},
|
||||
"include": ["export-tencent-vdb.ts"],
|
||||
"exclude": ["dist", "node_modules", "docs"]
|
||||
}
|
||||
Executable
+163
@@ -0,0 +1,163 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
log() { printf '[install-hermes-plugin-v2] %s\n' "$*" >&2; }
|
||||
fail() { printf '[install-hermes-plugin-v2][ERROR] %s\n' "$*" >&2; exit 1; }
|
||||
need_cmd() { command -v "$1" >/dev/null 2>&1 || fail "missing command: $1"; }
|
||||
|
||||
# Target user/home detection follows the legacy Hermes installer convention:
|
||||
# 1. INSTALL_AS_USER, 2. SUDO_USER, 3. current user.
|
||||
USERNAME="${INSTALL_AS_USER:-${SUDO_USER:-$(whoami)}}"
|
||||
USER_HOME="$(eval echo "~$USERNAME")"
|
||||
|
||||
SDK_WHEEL_URL="${SDK_WHEEL_URL:-https://cnb.cool/tencent/cloud/nosql/nosql-utilities/-/commit-assets/download/cc74bd6dbc931727da9ab6907b5ab1a07d7afd9d/tencentdb_agent_memory_sdk_python-0.1.0-py3-none-any.whl}"
|
||||
SDK_WHEEL_NAME="tencentdb_agent_memory_sdk_python-0.1.0-py3-none-any.whl"
|
||||
FORCE="${FORCE:-0}"
|
||||
ALLOW_SYSTEM_PYTHON="${ALLOW_SYSTEM_PYTHON:-0}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
PROVIDER_SRC="${HERMES_PROVIDER_SRC:-$REPO_ROOT/hermes-plugin/memory/memory_tencentdb_v2}"
|
||||
|
||||
HERMES_HOME="${HERMES_HOME:-$USER_HOME/.hermes}"
|
||||
HERMES_AGENT_DIR="${HERMES_AGENT_DIR:-$HERMES_HOME/hermes-agent}"
|
||||
HERMES_VENV_DIR="${HERMES_VENV_DIR:-$HERMES_AGENT_DIR/venv}"
|
||||
HERMES_CONFIG="${HERMES_CONFIG:-$HERMES_HOME/config.yaml}"
|
||||
HERMES_ENV="${HERMES_ENV:-$HERMES_HOME/.env}"
|
||||
HERMES_MEMORY_PLUGIN_DIR="${HERMES_MEMORY_PLUGIN_DIR:-$HERMES_AGENT_DIR/plugins/memory}"
|
||||
PROVIDER_TARGET="$HERMES_MEMORY_PLUGIN_DIR/memory_tencentdb_v2"
|
||||
|
||||
# Install the SDK into the Python environment that Hermes actually uses.
|
||||
# Selection order:
|
||||
# 1. PYTHON_BIN, if explicitly provided
|
||||
# 2. HERMES_VENV_DIR/bin/python, if present
|
||||
# 3. Python interpreter from the installed `hermes` command shebang, if discoverable
|
||||
# 4. system python3 only when ALLOW_SYSTEM_PYTHON=1
|
||||
if [[ -n "${PYTHON_BIN:-}" ]]; then
|
||||
:
|
||||
elif [[ -x "$HERMES_VENV_DIR/bin/python" ]]; then
|
||||
PYTHON_BIN="$HERMES_VENV_DIR/bin/python"
|
||||
elif command -v hermes >/dev/null 2>&1; then
|
||||
HERMES_BIN="$(command -v hermes)"
|
||||
HERMES_SHEBANG="$(head -n 1 "$HERMES_BIN" 2>/dev/null || true)"
|
||||
if [[ "$HERMES_SHEBANG" == '#!'*python* ]]; then
|
||||
HERMES_SHEBANG="${HERMES_SHEBANG#'#!'}"
|
||||
read -r HERMES_SHEBANG_CMD HERMES_SHEBANG_ARG _ <<<"$HERMES_SHEBANG"
|
||||
if [[ "$(basename "$HERMES_SHEBANG_CMD")" == "env" && -n "${HERMES_SHEBANG_ARG:-}" ]]; then
|
||||
PYTHON_BIN="$(command -v "$HERMES_SHEBANG_ARG" || true)"
|
||||
elif [[ -x "$HERMES_SHEBANG_CMD" ]]; then
|
||||
PYTHON_BIN="$HERMES_SHEBANG_CMD"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ -z "${PYTHON_BIN:-}" ]]; then
|
||||
if [[ "$ALLOW_SYSTEM_PYTHON" == "1" ]]; then
|
||||
PYTHON_BIN="python3"
|
||||
else
|
||||
fail "Hermes Python not found. Set PYTHON_BIN=/path/to/hermes/python or HERMES_VENV_DIR=/path/to/venv. Refusing to use system python3 by default to avoid externally-managed-environment installs."
|
||||
fi
|
||||
fi
|
||||
|
||||
TDAI_MEMORY_ENDPOINT="${TDAI_MEMORY_ENDPOINT:-http://127.0.0.1:8420}"
|
||||
TDAI_MEMORY_API_KEY="${TDAI_MEMORY_API_KEY:-local}"
|
||||
TDAI_MEMORY_SERVICE_ID="${TDAI_MEMORY_SERVICE_ID:-default}"
|
||||
WRITE_HERMES_ENV="${WRITE_HERMES_ENV:-1}"
|
||||
|
||||
need_cmd curl
|
||||
need_cmd "$PYTHON_BIN"
|
||||
|
||||
if [[ ! -d "$PROVIDER_SRC" ]]; then
|
||||
fail "Hermes provider directory not found: $PROVIDER_SRC"
|
||||
fi
|
||||
|
||||
if [[ ! -d "$HERMES_AGENT_DIR" ]]; then
|
||||
log "WARN: Hermes agent dir not found: $HERMES_AGENT_DIR"
|
||||
log " Set HERMES_AGENT_DIR if Hermes is installed elsewhere."
|
||||
fi
|
||||
|
||||
log "Downloading Python SDK wheel"
|
||||
TMP_DIR="$(mktemp -d)"
|
||||
cleanup() { rm -rf "$TMP_DIR"; }
|
||||
trap cleanup EXIT
|
||||
curl -fL -o "$TMP_DIR/$SDK_WHEEL_NAME" "$SDK_WHEEL_URL"
|
||||
|
||||
log "Installing Python SDK with $PYTHON_BIN"
|
||||
"$PYTHON_BIN" -m pip install "$TMP_DIR/$SDK_WHEEL_NAME"
|
||||
|
||||
log "Installing Hermes provider"
|
||||
mkdir -p "$HERMES_MEMORY_PLUGIN_DIR"
|
||||
if [[ -e "$PROVIDER_TARGET" || -L "$PROVIDER_TARGET" ]]; then
|
||||
if [[ "$FORCE" == "1" ]]; then
|
||||
rm -rf "$PROVIDER_TARGET"
|
||||
else
|
||||
fail "target already exists: $PROVIDER_TARGET (set FORCE=1 to overwrite)"
|
||||
fi
|
||||
fi
|
||||
ln -s "$PROVIDER_SRC" "$PROVIDER_TARGET"
|
||||
log "Provider linked: $PROVIDER_TARGET -> $PROVIDER_SRC"
|
||||
|
||||
log "Checking Hermes config"
|
||||
if [[ -f "$HERMES_CONFIG" ]]; then
|
||||
if sed -n '/^memory:/,/^[[:alpha:]_][[:alnum:]_]*:/p' "$HERMES_CONFIG" | grep -q 'provider: memory_tencentdb_v2'; then
|
||||
log "memory.provider already set to memory_tencentdb_v2"
|
||||
else
|
||||
log "Provider installed but NOT enabled by default. Add/edit in $HERMES_CONFIG:"
|
||||
cat >&2 <<'EOF'
|
||||
|
||||
memory:
|
||||
provider: memory_tencentdb_v2
|
||||
EOF
|
||||
fi
|
||||
else
|
||||
log "WARN: $HERMES_CONFIG not found; create it or run Hermes installer first."
|
||||
log " To enable the provider, add:"
|
||||
cat >&2 <<'EOF'
|
||||
|
||||
memory:
|
||||
provider: memory_tencentdb_v2
|
||||
EOF
|
||||
fi
|
||||
|
||||
_update_env() {
|
||||
local key="$1"
|
||||
local value="$2"
|
||||
local file="$3"
|
||||
mkdir -p "$(dirname "$file")"
|
||||
touch "$file"
|
||||
local tmp
|
||||
tmp="$(mktemp)"
|
||||
grep -v -E "^(# *)?${key}=" "$file" > "$tmp" || true
|
||||
local escaped="$value"
|
||||
escaped="${escaped//\\/\\\\}"
|
||||
escaped="${escaped//\"/\\\"}"
|
||||
printf '%s="%s"\n' "$key" "$escaped" >> "$tmp"
|
||||
mv "$tmp" "$file"
|
||||
}
|
||||
|
||||
if [[ "$WRITE_HERMES_ENV" == "1" ]]; then
|
||||
log "Writing Memory SDK env vars to $HERMES_ENV"
|
||||
_update_env "TDAI_MEMORY_ENDPOINT" "$TDAI_MEMORY_ENDPOINT" "$HERMES_ENV"
|
||||
_update_env "TDAI_MEMORY_API_KEY" "$TDAI_MEMORY_API_KEY" "$HERMES_ENV"
|
||||
_update_env "TDAI_MEMORY_SERVICE_ID" "$TDAI_MEMORY_SERVICE_ID" "$HERMES_ENV"
|
||||
fi
|
||||
|
||||
cat >&2 <<EOF
|
||||
|
||||
[install-hermes-plugin-v2] Done.
|
||||
Provider installed at:
|
||||
$PROVIDER_TARGET
|
||||
|
||||
SDK env file:
|
||||
$HERMES_ENV
|
||||
|
||||
If not already enabled, add this to $HERMES_CONFIG:
|
||||
|
||||
memory:
|
||||
provider: memory_tencentdb_v2
|
||||
|
||||
Standalone Gateway env:
|
||||
TDAI_MEMORY_ENDPOINT="$TDAI_MEMORY_ENDPOINT"
|
||||
TDAI_MEMORY_API_KEY="$TDAI_MEMORY_API_KEY"
|
||||
TDAI_MEMORY_SERVICE_ID="$TDAI_MEMORY_SERVICE_ID"
|
||||
EOF
|
||||
Executable
+187
@@ -0,0 +1,187 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
log() { printf '[install-openclaw-plugin-v2] %s\n' "$*" >&2; }
|
||||
fail() { printf '[install-openclaw-plugin-v2][ERROR] %s\n' "$*" >&2; exit 1; }
|
||||
need_cmd() { command -v "$1" >/dev/null 2>&1 || fail "missing command: $1"; }
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
PLUGIN_DIR="${OPENCLAW_PLUGIN_DIR:-$REPO_ROOT/openclaw-plugin}"
|
||||
INSTALL_OPENCLAW="${INSTALL_OPENCLAW:-1}"
|
||||
OPENCLAW_INSTALL_FLAGS="${OPENCLAW_INSTALL_FLAGS:-}"
|
||||
WRITE_OPENCLAW_CONFIG="${WRITE_OPENCLAW_CONFIG:-1}"
|
||||
OPENCLAW_CONFIG_FILE="${OPENCLAW_CONFIG_FILE:-$HOME/.openclaw/openclaw.json}"
|
||||
MEMORY_PLUGIN_ID="memory-tencentdb-client"
|
||||
TDAI_MEMORY_ENDPOINT="${TDAI_MEMORY_ENDPOINT:-http://127.0.0.1:8420}"
|
||||
TDAI_MEMORY_API_KEY="${TDAI_MEMORY_API_KEY:-local}"
|
||||
TDAI_MEMORY_INSTANCE_ID="${TDAI_MEMORY_INSTANCE_ID:-${TDAI_MEMORY_SERVICE_ID:-default}}"
|
||||
TDAI_MEMORY_RECALL_MAX_RESULTS="${TDAI_MEMORY_RECALL_MAX_RESULTS:-5}"
|
||||
TDAI_MEMORY_INCLUDE_PERSONA="${TDAI_MEMORY_INCLUDE_PERSONA:-true}"
|
||||
TDAI_MEMORY_INCLUDE_SCENE_NAV="${TDAI_MEMORY_INCLUDE_SCENE_NAV:-true}"
|
||||
TDAI_MEMORY_CAPTURE_ENABLED="${TDAI_MEMORY_CAPTURE_ENABLED:-true}"
|
||||
TDAI_MEMORY_ALLOW_PROMPT_INJECTION="${TDAI_MEMORY_ALLOW_PROMPT_INJECTION:-true}"
|
||||
TDAI_MEMORY_ALLOW_CONVERSATION_ACCESS="${TDAI_MEMORY_ALLOW_CONVERSATION_ACCESS:-true}"
|
||||
|
||||
need_cmd npm
|
||||
need_cmd node
|
||||
need_cmd curl
|
||||
|
||||
if [[ ! -d "$PLUGIN_DIR" ]]; then
|
||||
fail "OpenClaw plugin directory not found: $PLUGIN_DIR"
|
||||
fi
|
||||
|
||||
if ! command -v openclaw >/dev/null 2>&1; then
|
||||
if [[ "$INSTALL_OPENCLAW" == "1" ]]; then
|
||||
log "OpenClaw CLI not found; installing from https://get.openclaw.dev"
|
||||
curl -fsSL https://get.openclaw.dev | bash
|
||||
export PATH="$HOME/.local/bin:$HOME/bin:/usr/local/bin:$PATH"
|
||||
else
|
||||
fail "OpenClaw CLI not found. Install it first or run with INSTALL_OPENCLAW=1."
|
||||
fi
|
||||
fi
|
||||
|
||||
command -v openclaw >/dev/null 2>&1 || fail "OpenClaw CLI still not found after install; please check PATH"
|
||||
OPENCLAW_VERSION_RAW="$(openclaw --version 2>/dev/null || printf 'unknown')"
|
||||
log "OpenClaw version: $OPENCLAW_VERSION_RAW"
|
||||
|
||||
# Hook policy fields (allowPromptInjection / allowConversationAccess) were
|
||||
# accepted by the gateway zod schema starting at 2026.4.24 (PR #71221).
|
||||
# 2026.4.23 and earlier use `.strict()` and will REFUSE to start the gateway
|
||||
# if these fields are present. We therefore version-gate the config write to
|
||||
# match `src/utils/ensure-hook-policy.ts::HOOK_POLICY_MIN_VERSION`.
|
||||
HOOK_POLICY_MIN="2026.4.24"
|
||||
WRITE_HOOK_POLICY=0
|
||||
if [[ "$OPENCLAW_VERSION_RAW" =~ ([0-9]+)\.([0-9]+)\.([0-9]+) ]]; then
|
||||
v_major="${BASH_REMATCH[1]}"
|
||||
v_minor="${BASH_REMATCH[2]}"
|
||||
v_patch="${BASH_REMATCH[3]}"
|
||||
IFS=. read -r m_major m_minor m_patch <<<"$HOOK_POLICY_MIN"
|
||||
if (( v_major > m_major )) \
|
||||
|| { (( v_major == m_major )) && (( v_minor > m_minor )); } \
|
||||
|| { (( v_major == m_major )) && (( v_minor == m_minor )) && (( v_patch >= m_patch )); }; then
|
||||
WRITE_HOOK_POLICY=1
|
||||
fi
|
||||
fi
|
||||
if [[ "$WRITE_HOOK_POLICY" == "1" ]]; then
|
||||
log "Detected OpenClaw >= $HOOK_POLICY_MIN — will write hooks.allowPromptInjection / allowConversationAccess"
|
||||
else
|
||||
log "Detected OpenClaw < $HOOK_POLICY_MIN (or version unparseable) — will SKIP hooks.* policy fields (older gateway schemas reject them)"
|
||||
fi
|
||||
|
||||
log "Installing plugin dependencies"
|
||||
(cd "$PLUGIN_DIR" && npm install)
|
||||
|
||||
log "Building plugin"
|
||||
(cd "$PLUGIN_DIR" && npm run build)
|
||||
|
||||
log "Installing linked plugin into OpenClaw: $PLUGIN_DIR"
|
||||
# shellcheck disable=SC2086
|
||||
openclaw plugins install -l "$PLUGIN_DIR" $OPENCLAW_INSTALL_FLAGS
|
||||
|
||||
if [[ "$WRITE_OPENCLAW_CONFIG" == "1" ]]; then
|
||||
log "Updating OpenClaw config: $OPENCLAW_CONFIG_FILE"
|
||||
mkdir -p "$(dirname "$OPENCLAW_CONFIG_FILE")"
|
||||
if [[ -f "$OPENCLAW_CONFIG_FILE" ]]; then
|
||||
cp "$OPENCLAW_CONFIG_FILE" "$OPENCLAW_CONFIG_FILE.bak.$(date +%Y%m%d%H%M%S)"
|
||||
fi
|
||||
|
||||
OPENCLAW_CONFIG_FILE="$OPENCLAW_CONFIG_FILE" \
|
||||
MEMORY_PLUGIN_ID="$MEMORY_PLUGIN_ID" \
|
||||
TDAI_MEMORY_ENDPOINT="$TDAI_MEMORY_ENDPOINT" \
|
||||
TDAI_MEMORY_API_KEY="$TDAI_MEMORY_API_KEY" \
|
||||
TDAI_MEMORY_INSTANCE_ID="$TDAI_MEMORY_INSTANCE_ID" \
|
||||
TDAI_MEMORY_RECALL_MAX_RESULTS="$TDAI_MEMORY_RECALL_MAX_RESULTS" \
|
||||
TDAI_MEMORY_INCLUDE_PERSONA="$TDAI_MEMORY_INCLUDE_PERSONA" \
|
||||
TDAI_MEMORY_INCLUDE_SCENE_NAV="$TDAI_MEMORY_INCLUDE_SCENE_NAV" \
|
||||
TDAI_MEMORY_CAPTURE_ENABLED="$TDAI_MEMORY_CAPTURE_ENABLED" \
|
||||
TDAI_MEMORY_ALLOW_PROMPT_INJECTION="$TDAI_MEMORY_ALLOW_PROMPT_INJECTION" \
|
||||
TDAI_MEMORY_ALLOW_CONVERSATION_ACCESS="$TDAI_MEMORY_ALLOW_CONVERSATION_ACCESS" \
|
||||
WRITE_HOOK_POLICY="$WRITE_HOOK_POLICY" \
|
||||
node <<'NODE'
|
||||
const fs = require('node:fs');
|
||||
|
||||
const file = process.env.OPENCLAW_CONFIG_FILE;
|
||||
const pluginId = process.env.MEMORY_PLUGIN_ID;
|
||||
|
||||
function parseBool(value, fallback) {
|
||||
if (value === undefined || value === '') return fallback;
|
||||
return !['0', 'false', 'no', 'off'].includes(String(value).toLowerCase());
|
||||
}
|
||||
|
||||
function parsePositiveInt(value, fallback) {
|
||||
const parsed = Number.parseInt(String(value ?? ''), 10);
|
||||
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
|
||||
}
|
||||
|
||||
let config = {};
|
||||
if (fs.existsSync(file)) {
|
||||
const raw = fs.readFileSync(file, 'utf8').trim();
|
||||
if (raw) config = JSON.parse(raw);
|
||||
}
|
||||
|
||||
config.plugins ??= {};
|
||||
config.plugins.slots ??= {};
|
||||
config.plugins.entries ??= {};
|
||||
config.plugins.slots.memory = pluginId;
|
||||
|
||||
const entry = config.plugins.entries[pluginId] ?? {};
|
||||
entry.enabled = true;
|
||||
// Hook policy fields (allowPromptInjection / allowConversationAccess) are
|
||||
// only safe to write on OpenClaw >= 2026.4.24. The shell wrapper sets
|
||||
// WRITE_HOOK_POLICY=1 after parsing `openclaw --version`; on older gateways
|
||||
// it stays 0 and we *remove* any stale fields to keep the gateway bootable.
|
||||
const writeHookPolicy = process.env.WRITE_HOOK_POLICY === '1';
|
||||
if (writeHookPolicy) {
|
||||
entry.hooks ??= {};
|
||||
entry.hooks.allowPromptInjection = parseBool(process.env.TDAI_MEMORY_ALLOW_PROMPT_INJECTION, entry.hooks.allowPromptInjection ?? true);
|
||||
entry.hooks.allowConversationAccess = parseBool(process.env.TDAI_MEMORY_ALLOW_CONVERSATION_ACCESS, entry.hooks.allowConversationAccess ?? true);
|
||||
} else if (entry.hooks && (entry.hooks.allowPromptInjection !== undefined || entry.hooks.allowConversationAccess !== undefined)) {
|
||||
// Older gateways (zod .strict()) reject these fields; strip them defensively.
|
||||
delete entry.hooks.allowPromptInjection;
|
||||
delete entry.hooks.allowConversationAccess;
|
||||
if (Object.keys(entry.hooks).length === 0) delete entry.hooks;
|
||||
}
|
||||
entry.config ??= {};
|
||||
entry.config.server ??= {};
|
||||
entry.config.recall ??= {};
|
||||
entry.config.capture ??= {};
|
||||
|
||||
entry.config.server.url = process.env.TDAI_MEMORY_ENDPOINT || entry.config.server.url || 'http://127.0.0.1:8420';
|
||||
entry.config.server.apiKey = process.env.TDAI_MEMORY_API_KEY || entry.config.server.apiKey || 'local';
|
||||
entry.config.server.instanceId = process.env.TDAI_MEMORY_INSTANCE_ID || entry.config.server.instanceId || 'default';
|
||||
entry.config.recall.maxResults = parsePositiveInt(process.env.TDAI_MEMORY_RECALL_MAX_RESULTS, entry.config.recall.maxResults ?? 5);
|
||||
entry.config.recall.includePersona = parseBool(process.env.TDAI_MEMORY_INCLUDE_PERSONA, entry.config.recall.includePersona ?? true);
|
||||
entry.config.recall.includeSceneNav = parseBool(process.env.TDAI_MEMORY_INCLUDE_SCENE_NAV, entry.config.recall.includeSceneNav ?? true);
|
||||
entry.config.capture.enabled = parseBool(process.env.TDAI_MEMORY_CAPTURE_ENABLED, entry.config.capture.enabled ?? true);
|
||||
|
||||
config.plugins.entries[pluginId] = entry;
|
||||
fs.writeFileSync(file, `${JSON.stringify(config, null, 2)}\n`, { mode: 0o600 });
|
||||
NODE
|
||||
else
|
||||
log "Skipping OpenClaw config update because WRITE_OPENCLAW_CONFIG=$WRITE_OPENCLAW_CONFIG"
|
||||
fi
|
||||
|
||||
if [[ "$WRITE_HOOK_POLICY" == "1" ]]; then
|
||||
HOOK_SUMMARY=" plugins.entries[\"$MEMORY_PLUGIN_ID\"].hooks.allowPromptInjection = $TDAI_MEMORY_ALLOW_PROMPT_INJECTION
|
||||
plugins.entries[\"$MEMORY_PLUGIN_ID\"].hooks.allowConversationAccess = $TDAI_MEMORY_ALLOW_CONVERSATION_ACCESS"
|
||||
else
|
||||
HOOK_SUMMARY=" plugins.entries[\"$MEMORY_PLUGIN_ID\"].hooks.* = (skipped — OpenClaw < $HOOK_POLICY_MIN does not accept these fields)"
|
||||
fi
|
||||
|
||||
cat >&2 <<EOF
|
||||
|
||||
[install-openclaw-plugin-v2] Done.
|
||||
Memory plugin configured:
|
||||
plugins.slots.memory = "$MEMORY_PLUGIN_ID"
|
||||
plugins.entries["$MEMORY_PLUGIN_ID"].enabled = true
|
||||
$HOOK_SUMMARY
|
||||
server.url = "$TDAI_MEMORY_ENDPOINT"
|
||||
server.apiKey = "$TDAI_MEMORY_API_KEY"
|
||||
server.instanceId = "$TDAI_MEMORY_INSTANCE_ID"
|
||||
|
||||
Next steps:
|
||||
1. Ensure Memory Gateway is running at $TDAI_MEMORY_ENDPOINT.
|
||||
2. Restart OpenClaw Gateway if needed:
|
||||
openclaw gateway restart
|
||||
EOF
|
||||
Executable
+368
@@ -0,0 +1,368 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# install_memory_tencentdb.sh
|
||||
#
|
||||
# 在 install_hermes_ubuntu.sh 之后执行，用于：
|
||||
# 1. 通过 npm 下载 @tencentdb-agent-memory/memory-tencentdb@latest 到
|
||||
# $MEMORY_TENCENTDB_ROOT/tdai-memory-openclaw-plugin（默认 ~/.memory-tencentdb/tdai-memory-openclaw-plugin）
|
||||
# 2. 安装 Gateway 的 Node.js 依赖（npm install）
|
||||
# 3. 配置 hermes config.yaml 使用 memory_tencentdb 记忆提供者
|
||||
# 4. 设置 Gateway 自动启动环境变量
|
||||
#
|
||||
# 路径约定（全部位于 ~/.memory-tencentdb/ 之下，可通过环境变量覆盖）：
|
||||
# $MEMORY_TENCENTDB_ROOT 默认 ~/.memory-tencentdb
|
||||
# $TDAI_INSTALL_DIR 默认 $MEMORY_TENCENTDB_ROOT/tdai-memory-openclaw-plugin
|
||||
# $TDAI_DATA_DIR 默认 $MEMORY_TENCENTDB_ROOT/memory-tdai
|
||||
#
|
||||
# 旧版本（<= 0.3.x）使用 ~/tdai-memory-openclaw-plugin 与 ~/memory-tdai；
|
||||
# 本脚本会在执行前自动迁移这两个旧目录到新位置（见 Step 0）。
|
||||
#
|
||||
# 使用方式：
|
||||
# 以目标用户身份执行（推荐）：
|
||||
# su - <username> -c "bash ~/install_memory_tencentdb.sh"
|
||||
# # 或直接以该用户登录后执行
|
||||
# bash ~/install_memory_tencentdb.sh
|
||||
#
|
||||
# 以 root 身份执行（镜像构建场景）：
|
||||
# bash ~/install_memory_tencentdb.sh
|
||||
# # root 会自动 su 切换到目标用户执行，完成后修复权限
|
||||
#
|
||||
# 前置条件：
|
||||
# - install_hermes_ubuntu.sh 已执行完成（hermes-agent 已安装）
|
||||
# - Node.js >= 22 已安装
|
||||
|
||||
set -e
|
||||
|
||||
# 动态获取目标安装用户及其 HOME 目录。
|
||||
# 优先级：
|
||||
# 1. 显式 ``INSTALL_AS_USER`` 环境变量（管理员脚本场景：root 跑安装但
|
||||
# 想为另一个用户配置）
|
||||
# 2. ``SUDO_USER``（被 ``sudo`` 调用时，切回原用户而不是 root）
|
||||
# 3. ``whoami`` —— 当前 EUID 对应的用户
|
||||
#
|
||||
# 注意：当 root 直接 ssh 登录跑（非 sudo）时，前两个都不会被设置，
|
||||
# ``whoami`` 返回 ``root``。下面的 ``id -u`` == 0 分支会识别这种"目标
|
||||
# 就是 root"的情况、跳过 ``su - root`` 递归。
|
||||
USERNAME="${INSTALL_AS_USER:-${SUDO_USER:-$(whoami)}}"
|
||||
USER_HOME=$(eval echo ~$USERNAME)
|
||||
|
||||
# npm 包名
|
||||
NPM_PACKAGE="@tencentdb-agent-memory/memory-tencentdb@latest"
|
||||
|
||||
# Hermes 路径
|
||||
HERMES_HOME="$USER_HOME/.hermes"
|
||||
# HERMES_AGENT_DIR（fix: issue #18）
|
||||
# 用户通过环境变量传什么就用什么；未设置时 fallback 到传统路径。
|
||||
# 如果目录不存在，后续前置检查会统一报错。
|
||||
HERMES_AGENT_DIR="${HERMES_AGENT_DIR:-$HERMES_HOME/hermes-agent}"
|
||||
HERMES_CONFIG="$HERMES_HOME/config.yaml"
|
||||
|
||||
# memory-tencentdb 统一根目录（所有 tdai 相关数据/代码都收纳在此）
|
||||
# 可通过环境变量 MEMORY_TENCENTDB_ROOT 覆盖
|
||||
MEMORY_TENCENTDB_ROOT="${MEMORY_TENCENTDB_ROOT:-$USER_HOME/.memory-tencentdb}"
|
||||
|
||||
# tdai 解压目标目录（位于统一根目录下）
|
||||
# 可通过环境变量 TDAI_INSTALL_DIR 覆盖
|
||||
TDAI_INSTALL_DIR="${TDAI_INSTALL_DIR:-$MEMORY_TENCENTDB_ROOT/tdai-memory-openclaw-plugin}"
|
||||
|
||||
# tdai 数据目录（Gateway baseDir，位于统一根目录下）
|
||||
# 可通过环境变量 TDAI_DATA_DIR 覆盖
|
||||
TDAI_DATA_DIR="${TDAI_DATA_DIR:-$MEMORY_TENCENTDB_ROOT/memory-tdai}"
|
||||
|
||||
# 旧路径（仅用于自动迁移）
|
||||
LEGACY_INSTALL_DIR="$USER_HOME/tdai-memory-openclaw-plugin"
|
||||
LEGACY_DATA_DIR="$USER_HOME/memory-tdai"
|
||||
|
||||
# ==================== root → 自动切换到目标用户 ====================
|
||||
# 与 install_hermes_ubuntu.sh 保持一致：如果以 root 执行且目标用户不是
|
||||
# root，自动 su 切到目标用户运行实际安装逻辑。
|
||||
#
|
||||
# 如果当前是 root 且目标用户也是 root（``USERNAME=root``，例如直接 ssh
|
||||
# 登录 root 跑安装），跳过 ``su - root`` —— 否则会无限递归（``su - root``
|
||||
# 进入的仍是 root，又走到这个分支，再次 su，永远停不下来）。见 issue #20。
|
||||
|
||||
if [ "$(id -u)" -eq 0 ] && [ "$USERNAME" != "root" ]; then
|
||||
echo "[memory-tencentdb] Running as root, switching to $USERNAME for installation..."
|
||||
|
||||
# 验证前置条件
|
||||
if [ ! -d "$HERMES_AGENT_DIR" ]; then
|
||||
echo "[ERROR] Hermes agent not found at $HERMES_AGENT_DIR"
|
||||
echo "[ERROR] Please run install_hermes_ubuntu.sh first."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 切换到目标用户执行
|
||||
TEMP_SCRIPT=$(mktemp /tmp/memory-tencentdb-install-XXXXXX.sh)
|
||||
cp "${BASH_SOURCE[0]}" "$TEMP_SCRIPT"
|
||||
chmod 755 "$TEMP_SCRIPT"
|
||||
su - $USERNAME -c "bash $TEMP_SCRIPT" </dev/null
|
||||
|
||||
# 修复权限
|
||||
echo "[memory-tencentdb] Fixing permissions..."
|
||||
chown -R $USERNAME:$USERNAME "$USER_HOME"
|
||||
|
||||
rm -f "$TEMP_SCRIPT"
|
||||
echo "[memory-tencentdb] Installation completed successfully"
|
||||
exit 0
|
||||
elif [ "$(id -u)" -eq 0 ]; then
|
||||
# 当前是 root 且目标用户也是 root：直接以 root 跑后续安装逻辑，
|
||||
# 不再走 ``su -`` 切换（避免 #20 的递归）。
|
||||
echo "[memory-tencentdb] Running as root; target user is also root — installing in place."
|
||||
fi
|
||||
|
||||
# ==================== 用户阶段（核心安装逻辑） ====================
|
||||
|
||||
echo "[memory-tencentdb] Installing memory-tencentdb plugin (user: $(whoami))..."
|
||||
|
||||
# 验证前置条件
|
||||
if [ ! -d "$HERMES_AGENT_DIR" ]; then
|
||||
echo "[ERROR] Hermes agent not found at $HERMES_AGENT_DIR"
|
||||
echo "[ERROR] Please run install_hermes_ubuntu.sh first."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 加载 hermes 环境（PATH 中需要 node/npm）
|
||||
if [ -f /etc/profile.d/hermes-env.sh ]; then
|
||||
source /etc/profile.d/hermes-env.sh
|
||||
fi
|
||||
|
||||
# 确保统一根目录存在
|
||||
mkdir -p "$MEMORY_TENCENTDB_ROOT"
|
||||
|
||||
# ---------- Step 0: 自动迁移旧路径（向后兼容） ----------
|
||||
#
|
||||
# 历史版本把 tdai 解压到 ~/tdai-memory-openclaw-plugin、数据放在 ~/memory-tdai。
|
||||
# 现在统一收纳到 ~/.memory-tencentdb/ 之下，这里做一次性自动迁移。
|
||||
# 已经在新位置时跳过；新旧都存在时打印警告并保留新位置不动。
|
||||
|
||||
migrate_legacy_dir() {
|
||||
local legacy="$1"
|
||||
local target="$2"
|
||||
local label="$3"
|
||||
if [ ! -e "$legacy" ]; then
|
||||
return 0
|
||||
fi
|
||||
if [ -L "$legacy" ]; then
|
||||
# 旧位置是 symlink，直接清掉
|
||||
echo "[memory-tencentdb] Removing legacy symlink: $legacy"
|
||||
rm -f "$legacy"
|
||||
return 0
|
||||
fi
|
||||
if [ -e "$target" ]; then
|
||||
echo "[memory-tencentdb] WARN: legacy $label dir exists at $legacy but new location $target also exists." >&2
|
||||
echo "[memory-tencentdb] WARN: keeping new location; please review and remove $legacy manually if obsolete." >&2
|
||||
return 0
|
||||
fi
|
||||
echo "[memory-tencentdb] Migrating legacy $label dir: $legacy -> $target"
|
||||
mkdir -p "$(dirname "$target")"
|
||||
mv "$legacy" "$target"
|
||||
}
|
||||
|
||||
migrate_legacy_dir "$LEGACY_INSTALL_DIR" "$TDAI_INSTALL_DIR" "install"
|
||||
migrate_legacy_dir "$LEGACY_DATA_DIR" "$TDAI_DATA_DIR" "data"
|
||||
|
||||
# ---------- Step 1: 通过 npm 下载包并提取到 $TDAI_INSTALL_DIR ----------
|
||||
|
||||
echo "[memory-tencentdb] Step 1: Downloading $NPM_PACKAGE via npm..."
|
||||
|
||||
# 清理旧安装
|
||||
rm -rf "$TDAI_INSTALL_DIR"
|
||||
|
||||
# 使用临时目录通过 npm install 下载包
|
||||
TEMP_DOWNLOAD=$(mktemp -d /tmp/memory-tencentdb-download-XXXXXX)
|
||||
cd "$TEMP_DOWNLOAD"
|
||||
npm init -y --silent > /dev/null 2>&1
|
||||
npm install "$NPM_PACKAGE" --omit=dev 2>&1 | tail -5
|
||||
|
||||
# 包安装后位于 node_modules/@tencentdb-agent-memory/memory-tencentdb
|
||||
PACK_DIR="$TEMP_DOWNLOAD/node_modules/@tencentdb-agent-memory/memory-tencentdb"
|
||||
|
||||
if [ ! -d "$PACK_DIR" ]; then
|
||||
echo "[ERROR] Downloaded package directory not found at $PACK_DIR"
|
||||
rm -rf "$TEMP_DOWNLOAD"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 将包内容移动到目标安装目录
|
||||
mkdir -p "$(dirname "$TDAI_INSTALL_DIR")"
|
||||
cp -r "$PACK_DIR" "$TDAI_INSTALL_DIR"
|
||||
|
||||
echo "[memory-tencentdb] Package downloaded and extracted to $TDAI_INSTALL_DIR"
|
||||
|
||||
# ---------- Step 2: 安装 Gateway Node.js 依赖 ----------
|
||||
|
||||
echo "[memory-tencentdb] Step 2: Installing Gateway dependencies..."
|
||||
|
||||
cd "$TDAI_INSTALL_DIR"
|
||||
|
||||
echo "[memory-tencentdb] Running npm install (this may take a while)..."
|
||||
npm install --omit=dev 2>&1 | tail -5
|
||||
|
||||
# 安装 tsx（Gateway 启动需要），优先本地安装
|
||||
if ! npx tsx --version &>/dev/null; then
|
||||
npm install tsx 2>&1 | tail -2
|
||||
fi
|
||||
|
||||
echo "[memory-tencentdb] Gateway dependencies installed"
|
||||
|
||||
# ---------- Step 2.5: 将插件链接到 hermes 插件目录 ----------
|
||||
|
||||
echo "[memory-tencentdb] Step 2.5: Linking plugin into hermes plugins directory..."
|
||||
|
||||
HERMES_PLUGIN_DIR="$HERMES_AGENT_DIR/plugins/memory/memory_tencentdb"
|
||||
PLUGIN_SRC_DIR="$TDAI_INSTALL_DIR/hermes-plugin/memory/memory_tencentdb"
|
||||
|
||||
# 移除旧链接/目录
|
||||
rm -rf "$HERMES_PLUGIN_DIR"
|
||||
|
||||
# 创建 symlink 使 hermes 能发现插件
|
||||
ln -sf "$PLUGIN_SRC_DIR" "$HERMES_PLUGIN_DIR"
|
||||
|
||||
echo "[memory-tencentdb] Plugin linked: $HERMES_PLUGIN_DIR -> $PLUGIN_SRC_DIR"
|
||||
|
||||
# ---------- Step 3: 提示用户手动开启 memory_tencentdb（不自动修改 config） ----------
|
||||
|
||||
echo "[memory-tencentdb] Step 3: Checking hermes config..."
|
||||
|
||||
# 插件已链接到 hermes 插件目录，但默认不自动启用，仅提示
|
||||
if [ -f "$HERMES_CONFIG" ]; then
|
||||
if sed -n '/^memory:/,/^[a-zA-Z]/p' "$HERMES_CONFIG" | grep -q "provider: memory_tencentdb"; then
|
||||
echo "[memory-tencentdb] memory.provider already set to memory_tencentdb"
|
||||
else
|
||||
echo "[memory-tencentdb] Plugin installed but NOT enabled by default."
|
||||
echo "[memory-tencentdb] To enable tdai memory, add/edit in $HERMES_CONFIG:"
|
||||
echo ""
|
||||
echo " memory:"
|
||||
echo " provider: memory_tencentdb"
|
||||
echo ""
|
||||
fi
|
||||
else
|
||||
echo "[memory-tencentdb] WARN: $HERMES_CONFIG not found, please run install_hermes_ubuntu.sh first"
|
||||
fi
|
||||
|
||||
# ---------- Step 4: 配置 Gateway 环境变量 ----------
|
||||
|
||||
echo "[memory-tencentdb] Step 4: Setting up Gateway environment..."
|
||||
|
||||
# 构建 Gateway 启动命令
|
||||
# 使用 sh -c 包裹，先 cd 到插件目录再启动 Gateway（ESM 解析需要）
|
||||
#
|
||||
# 解析 node 绝对路径写入 GATEWAY_CMD（fix: issue #19）
|
||||
# 当 Hermes 或独立 Gateway 以 systemd service 运行时，systemd 不会
|
||||
# source 任何 user shell rc 文件，nvm/asdf 注入的 PATH 不存在。
|
||||
# 用 `command -v node` 在 install 时解析绝对路径，并改用 Node 原生
|
||||
# `--import tsx/esm`（Node >= 20.6 stable）替代 `npx tsx`，
|
||||
# 让最终命令完全不依赖运行时 PATH。
|
||||
NODE_BIN="$(command -v node || true)"
|
||||
if [ -z "$NODE_BIN" ]; then
|
||||
echo "[ERROR] 'node' not found in PATH; cannot generate Gateway start command." >&2
|
||||
echo "[ERROR] If you installed Node via nvm/asdf, source the loader script first:" >&2
|
||||
echo "[ERROR] source ~/.bashrc # or 'nvm use <version>'" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "[memory-tencentdb] Resolved node: $NODE_BIN"
|
||||
|
||||
GATEWAY_CMD="sh -c 'cd $TDAI_INSTALL_DIR && exec \"$NODE_BIN\" --import tsx/esm src/gateway/server.ts'"
|
||||
|
||||
# ── 4a: /etc/profile.d（SSH 交互式登录场景） ──
|
||||
# 写入 /etc/profile.d 持久化环境变量，供 SSH 手动执行 `hermes` 时使用。
|
||||
# 注意：LLM 相关变量（API key、model 等）需要用户后续手动配置
|
||||
ENVFILE="/etc/profile.d/memory-tencentdb-env.sh"
|
||||
cat << ENVEOF | sudo tee "$ENVFILE" > /dev/null
|
||||
# memory-tencentdb Gateway 环境变量
|
||||
export MEMORY_TENCENTDB_GATEWAY_CMD="$GATEWAY_CMD"
|
||||
export MEMORY_TENCENTDB_GATEWAY_HOST="127.0.0.1"
|
||||
export MEMORY_TENCENTDB_GATEWAY_PORT="8420"
|
||||
# LLM 配置（按需修改）
|
||||
# export MEMORY_TENCENTDB_LLM_API_KEY="sk-..."
|
||||
# export MEMORY_TENCENTDB_LLM_BASE_URL="https://api.openai.com/v1"
|
||||
# export MEMORY_TENCENTDB_LLM_MODEL="gpt-4o"
|
||||
ENVEOF
|
||||
|
||||
echo "[memory-tencentdb] Environment variables written to $ENVFILE"
|
||||
|
||||
# ── 4b: ~/.hermes/.env（systemd service 场景） ──
|
||||
# hermes-gateway 通过 systemd user service 启动时不会 source /etc/profile.d/*.sh，
|
||||
# 但 hermes 的 run.py 启动时会 load_dotenv("~/.hermes/.env")。
|
||||
# 因此必须将关键变量同步写入 .env，否则 systemd 场景下 Gateway 无法自动启动。
|
||||
HERMES_ENV="$HERMES_HOME/.env"
|
||||
|
||||
_append_or_update_env() {
|
||||
local key="$1"
|
||||
local value="$2"
|
||||
local file="$3"
|
||||
if [ ! -f "$file" ]; then
|
||||
touch "$file"
|
||||
fi
|
||||
# 移除已有的同名变量行（含注释掉的和带引号的），再追加
|
||||
sed -i "/^${key}=/d" "$file"
|
||||
sed -i "/^# *${key}=/d" "$file"
|
||||
# python-dotenv 要求含空格/引号/特殊字符的值用双引号包裹
|
||||
echo "${key}=\"${value}\"" >> "$file"
|
||||
}
|
||||
|
||||
_append_or_update_env "MEMORY_TENCENTDB_GATEWAY_CMD" "$GATEWAY_CMD" "$HERMES_ENV"
|
||||
_append_or_update_env "MEMORY_TENCENTDB_GATEWAY_HOST" "127.0.0.1" "$HERMES_ENV"
|
||||
_append_or_update_env "MEMORY_TENCENTDB_GATEWAY_PORT" "8420" "$HERMES_ENV"
|
||||
|
||||
echo "[memory-tencentdb] Gateway env vars also written to $HERMES_ENV (for systemd service)"
|
||||
|
||||
# ---------- 清理 ----------
|
||||
|
||||
rm -rf "$TEMP_DOWNLOAD"
|
||||
|
||||
# ---------- 验证安装 ----------
|
||||
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo "[memory-tencentdb] Installation Summary"
|
||||
echo "=========================================="
|
||||
echo " Root dir: $MEMORY_TENCENTDB_ROOT"
|
||||
echo " tdai source: $TDAI_INSTALL_DIR"
|
||||
echo " tdai data dir: $TDAI_DATA_DIR"
|
||||
echo " Hermes config: $HERMES_CONFIG"
|
||||
echo " Env file: $ENVFILE"
|
||||
echo ""
|
||||
echo " Installed files in tdai dir:"
|
||||
ls -la "$TDAI_INSTALL_DIR"/ 2>/dev/null | head -20 || echo " (none)"
|
||||
echo ""
|
||||
|
||||
# 验证 hermes 插件文件存在（在解压目录中）
|
||||
PLUGIN_SRC="$TDAI_INSTALL_DIR/hermes-plugin/memory/memory_tencentdb"
|
||||
MISSING=0
|
||||
for f in __init__.py plugin.yaml client.py supervisor.py; do
|
||||
if [ ! -f "$PLUGIN_SRC/$f" ]; then
|
||||
echo " [WARN] Missing: $PLUGIN_SRC/$f"
|
||||
MISSING=1
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$MISSING" -eq 0 ]; then
|
||||
echo " [OK] All hermes plugin files present"
|
||||
fi
|
||||
|
||||
# 验证 Gateway 入口存在
|
||||
if [ -f "$TDAI_INSTALL_DIR/src/gateway/server.ts" ]; then
|
||||
echo " [OK] Gateway entry point found"
|
||||
else
|
||||
echo " [WARN] Gateway server.ts not found at $TDAI_INSTALL_DIR/src/gateway/server.ts"
|
||||
fi
|
||||
|
||||
# 验证 node_modules 已安装
|
||||
if [ -d "$TDAI_INSTALL_DIR/node_modules" ]; then
|
||||
echo " [OK] Gateway node_modules installed"
|
||||
else
|
||||
echo " [WARN] Gateway node_modules not found"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "[memory-tencentdb] Done!"
|
||||
echo ""
|
||||
echo " NOTE: Before using the memory plugin, configure LLM credentials in ~/.hermes/.env:"
|
||||
echo " MEMORY_TENCENTDB_LLM_API_KEY=your-api-key"
|
||||
echo " MEMORY_TENCENTDB_LLM_BASE_URL=https://api.openai.com/v1"
|
||||
echo " MEMORY_TENCENTDB_LLM_MODEL=gpt-4o"
|
||||
echo ""
|
||||
echo " (For systemd-managed hermes-gateway, ~/.hermes/.env is the authoritative config."
|
||||
echo " /etc/profile.d/ is only used for interactive SSH sessions.)"
|
||||
echo ""
|
||||
Executable
+1030
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,239 @@
|
||||
# SQLite → 腾讯云向量数据库迁移工具
|
||||
|
||||
离线迁移工具，用于将 memory-tdai 的数据从本地 SQLite 存储迁移到腾讯云向量数据库（TCVDB）。
|
||||
|
||||
## 前置条件
|
||||
|
||||
- Node.js >= 22.16.0
|
||||
- 插件已通过 `openclaw plugins install` 安装
|
||||
- 迁移脚本已编译（见下方）
|
||||
|
||||
## 编译
|
||||
|
||||
迁移脚本使用 TypeScript 编写，运行前需要先编译：
|
||||
|
||||
```bash
|
||||
npm run build:migrate-sqlite-to-vdb
|
||||
```
|
||||
|
||||
编译产物输出到 `scripts/migrate-sqlite-to-tcvdb/dist/`，可直接用 Node 运行。
|
||||
|
||||
## 使用方法
|
||||
|
||||
```bash
|
||||
# 预检模式（仅查看源数据，不执行写入）
|
||||
npm run migrate:sqlite-to-tcvdb -- \
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \
|
||||
--tcvdb-url http://127.0.0.1:80 \
|
||||
--tcvdb-username root \
|
||||
--tcvdb-api-key-env TCVDB_API_KEY \
|
||||
--tcvdb-database agent_memory_prod \
|
||||
--tcvdb-embedding-model bge-large-zh \
|
||||
--dry-run
|
||||
|
||||
# 正式迁移
|
||||
npm run migrate:sqlite-to-tcvdb -- \
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \
|
||||
--tcvdb-url http://127.0.0.1:80 \
|
||||
--tcvdb-username root \
|
||||
--tcvdb-api-key-env TCVDB_API_KEY \
|
||||
--tcvdb-database agent_memory_prod \
|
||||
--tcvdb-embedding-model bge-large-zh \
|
||||
--yes
|
||||
```
|
||||
|
||||
### 更多示例
|
||||
|
||||
```bash
|
||||
# 直接传入 API Key（不通过环境变量）
|
||||
npm run migrate:sqlite-to-tcvdb -- \
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \
|
||||
--tcvdb-url http://127.0.0.1:80 \
|
||||
--tcvdb-username root \
|
||||
--tcvdb-api-key 'your-api-key-here' \
|
||||
--tcvdb-database agent_memory_prod \
|
||||
--tcvdb-embedding-model bge-large-zh \
|
||||
--yes
|
||||
```
|
||||
|
||||
```bash
|
||||
# 指定自定义 SQLite 路径（数据库不在默认 vectors.db 位置时）
|
||||
npm run migrate:sqlite-to-tcvdb -- \
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \
|
||||
--sqlite-path /backup/2026-04/vectors-snapshot.db \
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \
|
||||
--tcvdb-url http://127.0.0.1:80 \
|
||||
--tcvdb-username root \
|
||||
--tcvdb-api-key-env TCVDB_API_KEY \
|
||||
--tcvdb-database agent_memory_prod \
|
||||
--tcvdb-embedding-model bge-large-zh \
|
||||
--yes
|
||||
```
|
||||
|
||||
```bash
|
||||
# 只迁移 L1 记忆层（跳过 L0 原始消息和 Profile）
|
||||
npm run migrate:sqlite-to-tcvdb -- \
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \
|
||||
--tcvdb-url http://127.0.0.1:80 \
|
||||
--tcvdb-username root \
|
||||
--tcvdb-api-key-env TCVDB_API_KEY \
|
||||
--tcvdb-database agent_memory_prod \
|
||||
--tcvdb-embedding-model bge-large-zh \
|
||||
--layers l1 \
|
||||
--yes
|
||||
```
|
||||
|
||||
```bash
|
||||
# 只迁移 L0 和 L1（不迁移 Profile）
|
||||
npm run migrate:sqlite-to-tcvdb -- \
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \
|
||||
--tcvdb-url http://127.0.0.1:80 \
|
||||
--tcvdb-username root \
|
||||
--tcvdb-api-key-env TCVDB_API_KEY \
|
||||
--tcvdb-database agent_memory_prod \
|
||||
--tcvdb-embedding-model bge-large-zh \
|
||||
--layers l0,l1 \
|
||||
--yes
|
||||
```
|
||||
|
||||
```bash
|
||||
# 英文语料场景：使用英文 BM25 分词
|
||||
npm run migrate:sqlite-to-tcvdb -- \
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \
|
||||
--tcvdb-url http://127.0.0.1:80 \
|
||||
--tcvdb-username root \
|
||||
--tcvdb-api-key-env TCVDB_API_KEY \
|
||||
--tcvdb-database agent_memory_prod \
|
||||
--tcvdb-embedding-model bge-large-en-v1.5 \
|
||||
--bm25-language en \
|
||||
--yes
|
||||
```
|
||||
|
||||
```bash
|
||||
# 禁用 BM25 稀疏向量（仅使用密集向量检索）
|
||||
npm run migrate:sqlite-to-tcvdb -- \
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \
|
||||
--tcvdb-url http://127.0.0.1:80 \
|
||||
--tcvdb-username root \
|
||||
--tcvdb-api-key-env TCVDB_API_KEY \
|
||||
--tcvdb-database agent_memory_prod \
|
||||
--tcvdb-embedding-model bge-large-zh \
|
||||
--no-bm25-enabled \
|
||||
--yes
|
||||
```
|
||||
|
||||
```bash
|
||||
# 仅迁移数据，不自动更新 openclaw.json 和 manifest（手动管理配置）
|
||||
npm run migrate:sqlite-to-tcvdb -- \
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \
|
||||
--tcvdb-url http://127.0.0.1:80 \
|
||||
--tcvdb-username root \
|
||||
--tcvdb-api-key-env TCVDB_API_KEY \
|
||||
--tcvdb-database agent_memory_prod \
|
||||
--tcvdb-embedding-model bge-large-zh \
|
||||
--no-apply-config \
|
||||
--no-rewrite-manifest \
|
||||
--yes
|
||||
```
|
||||
|
||||
```bash
|
||||
# 追加迁移：允许目标库已有数据，跳过非空检查
|
||||
npm run migrate:sqlite-to-tcvdb -- \
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \
|
||||
--tcvdb-url http://127.0.0.1:80 \
|
||||
--tcvdb-username root \
|
||||
--tcvdb-api-key-env TCVDB_API_KEY \
|
||||
--tcvdb-database agent_memory_prod \
|
||||
--tcvdb-embedding-model bge-large-zh \
|
||||
--no-fail-if-target-nonempty \
|
||||
--no-verify-counts \
|
||||
--yes
|
||||
```
|
||||
|
||||
```bash
|
||||
# 输出迁移摘要到 JSON 文件（适合 CI/自动化流水线）
|
||||
npm run migrate:sqlite-to-tcvdb -- \
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \
|
||||
--tcvdb-url http://127.0.0.1:80 \
|
||||
--tcvdb-username root \
|
||||
--tcvdb-api-key-env TCVDB_API_KEY \
|
||||
--tcvdb-database agent_memory_prod \
|
||||
--tcvdb-embedding-model bge-large-zh \
|
||||
--summary-json-path ./migration-report.json \
|
||||
--job-id "migrate-2026-04-13" \
|
||||
--yes
|
||||
```
|
||||
|
||||
```bash
|
||||
# 设置自定义超时和别名
|
||||
npm run migrate:sqlite-to-tcvdb -- \
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \
|
||||
--tcvdb-url http://10.0.1.50:80 \
|
||||
--tcvdb-username admin \
|
||||
--tcvdb-api-key-env TCVDB_API_KEY \
|
||||
--tcvdb-database agent_memory_prod \
|
||||
--tcvdb-embedding-model bge-large-zh \
|
||||
--tcvdb-alias "生产环境-主库" \
|
||||
--tcvdb-timeout-ms 30000 \
|
||||
--yes
|
||||
```
|
||||
|
||||
## 参数说明
|
||||
|
||||
| 参数 | 必填 | 默认值 | 说明 |
|
||||
|---|---|---|---|
|
||||
| `--plugin-data-dir` | 是 | — | 插件数据目录路径 |
|
||||
| `--openclaw-config-path` | 是 | — | `openclaw.json` 配置文件路径 |
|
||||
| `--sqlite-path` | 否 | `<plugin-data-dir>/vectors.db` | SQLite 数据库文件路径（默认取数据目录下的 `vectors.db`） |
|
||||
| `--plugin-id` | 否 | `memory-tencentdb` | 写入配置时使用的插件 ID |
|
||||
| `--tcvdb-url` | 是 | — | TCVDB 服务地址 |
|
||||
| `--tcvdb-username` | 是 | — | TCVDB 用户名 |
|
||||
| `--tcvdb-api-key` | * | — | TCVDB API 密钥（明文） |
|
||||
| `--tcvdb-api-key-env` | * | — | 包含 API 密钥的环境变量名 |
|
||||
| `--tcvdb-database` | 是 | — | TCVDB 数据库名 |
|
||||
| `--tcvdb-embedding-model` | 是 | — | Embedding 模型名称 |
|
||||
| `--tcvdb-alias` | 否 | `""` | 用户自定义别名 |
|
||||
| `--tcvdb-timeout-ms` | 否 | `10000` | 请求超时时间（毫秒） |
|
||||
| `--layers` | 否 | `l0,l1,l2,l3` | 要迁移的层（逗号分隔） |
|
||||
| `--dry-run` | 否 | `false` | 仅预览，不执行写入 |
|
||||
| `--yes` | 否 | `false` | 跳过交互确认 |
|
||||
| `--apply-config` | 否 | `true` | 迁移后更新 openclaw.json |
|
||||
| `--config-backup` | 否 | `true` | 写入配置前先备份原配置文件 |
|
||||
| `--rewrite-manifest` | 否 | `true` | 将 manifest.json 更新为 tcvdb |
|
||||
| `--fail-if-target-nonempty` | 否 | `true` | 目标库非空时中止 |
|
||||
| `--verify-counts` | 否 | `true` | 迁移后校验记录数 |
|
||||
| `--summary-json-path` | 否 | — | 将迁移摘要写入此文件 |
|
||||
| `--job-id` | 否 | — | 迁移任务 ID（用于追踪） |
|
||||
| `--bm25-enabled` | 否 | `true` | 启用 BM25 稀疏向量 |
|
||||
| `--bm25-language` | 否 | `zh` | BM25 语言（`zh` 或 `en`） |
|
||||
|
||||
\* `--tcvdb-api-key` 和 `--tcvdb-api-key-env` 二选一，必须提供其中一个。
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
scripts/migrate-sqlite-to-tcvdb/
|
||||
├── cli-entry.ts # CLI 入口
|
||||
├── sqlite-to-tcvdb.ts # 迁移核心逻辑（参数解析、预检、数据迁移）
|
||||
├── config-write.ts # OpenClaw 配置更新（JSON5，自包含）
|
||||
├── manifest-write.ts # Manifest 重写
|
||||
├── *.test.ts # 就近放置的测试文件
|
||||
├── tsconfig.json # 迁移脚本编译配置
|
||||
├── dist/ # 编译产物（已 gitignore）
|
||||
└── README.md # 本文件
|
||||
|
||||
bin/migrate-sqlite-to-tcvdb.mjs # 极薄 bin 包装 → dist/
|
||||
```
|
||||
|
||||
迁移脚本通过 `../../src/` 引用存储实现（VectorStore、TcvdbMemoryStore 等），但**不依赖 `openclaw/plugin-sdk`**。配置写回直接使用 `json5`。
|
||||
@@ -0,0 +1,12 @@
|
||||
import { runMigrationCli } from "./sqlite-to-tcvdb.js";
|
||||
|
||||
const TAG = "[memory-tdai][migrate-cli]";
|
||||
|
||||
try {
|
||||
const summary = await runMigrationCli(process.argv.slice(2));
|
||||
process.stdout.write(`${JSON.stringify(summary, null, 2)}\n`);
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`${TAG} ${message}\n`);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
import fs from "node:fs/promises";
|
||||
import JSON5 from "json5";
|
||||
|
||||
export type Bm25Language = "zh" | "en";
|
||||
|
||||
export interface MigrationPluginConfigTarget {
|
||||
url: string;
|
||||
username: string;
|
||||
apiKey: string;
|
||||
database: string;
|
||||
alias: string;
|
||||
embeddingModel: string;
|
||||
timeout: number;
|
||||
bm25Enabled: boolean;
|
||||
bm25Language: Bm25Language;
|
||||
}
|
||||
|
||||
export interface WriteMigrationPluginConfigParams {
|
||||
configPath: string;
|
||||
pluginId: string;
|
||||
tcvdb: {
|
||||
url: string;
|
||||
username: string;
|
||||
apiKey: string;
|
||||
database: string;
|
||||
alias: string;
|
||||
embeddingModel: string;
|
||||
timeout: number;
|
||||
};
|
||||
bm25: {
|
||||
enabled: boolean;
|
||||
language: Bm25Language;
|
||||
};
|
||||
}
|
||||
|
||||
interface ConfigWriteAdapterDeps {
|
||||
fs?: Pick<typeof fs, "readFile" | "writeFile" | "mkdir">;
|
||||
parseConfig?: (raw: string) => unknown;
|
||||
}
|
||||
|
||||
function asRecord(value: unknown): Record<string, unknown> {
|
||||
return value !== null && typeof value === "object" && !Array.isArray(value)
|
||||
? { ...(value as Record<string, unknown>) }
|
||||
: {};
|
||||
}
|
||||
|
||||
export function buildMigrationPluginConfigPatch(target: MigrationPluginConfigTarget): Record<string, unknown> {
|
||||
return {
|
||||
storeBackend: "tcvdb",
|
||||
tcvdb: {
|
||||
url: target.url,
|
||||
username: target.username,
|
||||
apiKey: target.apiKey,
|
||||
database: target.database,
|
||||
alias: target.alias,
|
||||
embeddingModel: target.embeddingModel,
|
||||
timeout: target.timeout,
|
||||
},
|
||||
bm25: {
|
||||
enabled: target.bm25Enabled,
|
||||
language: target.bm25Language,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function applyPluginConfigPatch(
|
||||
sourceConfig: Record<string, unknown>,
|
||||
pluginId: string,
|
||||
patch: Record<string, unknown>,
|
||||
): Record<string, unknown> {
|
||||
const nextConfig = asRecord(sourceConfig);
|
||||
const plugins = asRecord(nextConfig.plugins);
|
||||
const entries = asRecord(plugins.entries);
|
||||
const targetEntry = asRecord(entries[pluginId]);
|
||||
const targetPluginConfig = asRecord(targetEntry.config);
|
||||
const patchTcvdb = asRecord(patch.tcvdb);
|
||||
const patchBm25 = asRecord(patch.bm25);
|
||||
|
||||
const mergedPluginConfig: Record<string, unknown> = {
|
||||
...targetPluginConfig,
|
||||
...patch,
|
||||
tcvdb: {
|
||||
...asRecord(targetPluginConfig.tcvdb),
|
||||
...patchTcvdb,
|
||||
},
|
||||
bm25: {
|
||||
...asRecord(targetPluginConfig.bm25),
|
||||
...patchBm25,
|
||||
},
|
||||
};
|
||||
|
||||
entries[pluginId] = {
|
||||
...targetEntry,
|
||||
config: mergedPluginConfig,
|
||||
};
|
||||
|
||||
plugins.entries = entries;
|
||||
nextConfig.plugins = plugins;
|
||||
return nextConfig;
|
||||
}
|
||||
|
||||
export async function writeMigrationPluginConfig(
|
||||
params: WriteMigrationPluginConfigParams,
|
||||
deps: ConfigWriteAdapterDeps = {},
|
||||
): Promise<void> {
|
||||
const fsImpl = deps.fs ?? fs;
|
||||
const parseConfig = deps.parseConfig ?? ((raw: string) => JSON5.parse(raw));
|
||||
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = parseConfig(await fsImpl.readFile(params.configPath, "utf-8"));
|
||||
} catch {
|
||||
throw new Error(
|
||||
`Config migration writer only supports single-file JSON/JSON5: ${params.configPath}`,
|
||||
);
|
||||
}
|
||||
|
||||
const nextConfig = applyPluginConfigPatch(
|
||||
asRecord(parsed),
|
||||
params.pluginId,
|
||||
buildMigrationPluginConfigPatch({
|
||||
...params.tcvdb,
|
||||
bm25Enabled: params.bm25.enabled,
|
||||
bm25Language: params.bm25.language,
|
||||
}),
|
||||
);
|
||||
|
||||
await fsImpl.writeFile(params.configPath, `${JSON.stringify(nextConfig, null, 2)}\n`, "utf-8");
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import {
|
||||
buildStoreInfo,
|
||||
manifestPath,
|
||||
readManifest,
|
||||
writeManifest,
|
||||
} from "../../src/utils/manifest.js";
|
||||
|
||||
export interface RewriteMigrationManifestParams {
|
||||
dataDir: string;
|
||||
tcvdbUrl: string;
|
||||
tcvdbDatabase: string;
|
||||
tcvdbAlias?: string;
|
||||
backupExisting?: boolean;
|
||||
now?: () => string;
|
||||
}
|
||||
|
||||
export interface RewriteMigrationManifestResult {
|
||||
created: boolean;
|
||||
updated: boolean;
|
||||
backupPath?: string;
|
||||
}
|
||||
|
||||
const DEFAULT_BACKUP_FILE = "manifest.json.migrate.bak";
|
||||
|
||||
export async function rewriteMigrationManifest(
|
||||
params: RewriteMigrationManifestParams,
|
||||
): Promise<RewriteMigrationManifestResult> {
|
||||
const existing = readManifest(params.dataDir);
|
||||
const nextStore = buildStoreInfo({
|
||||
type: "tcvdb",
|
||||
tcvdbUrl: params.tcvdbUrl,
|
||||
tcvdbDatabase: params.tcvdbDatabase,
|
||||
tcvdbAlias: params.tcvdbAlias,
|
||||
});
|
||||
|
||||
if (!existing) {
|
||||
writeManifest(params.dataDir, {
|
||||
version: 1,
|
||||
createdAt: params.now?.() ?? new Date().toISOString(),
|
||||
store: nextStore,
|
||||
seed: null,
|
||||
});
|
||||
return {
|
||||
created: true,
|
||||
updated: false,
|
||||
backupPath: undefined,
|
||||
};
|
||||
}
|
||||
|
||||
let backupPath: string | undefined;
|
||||
if (params.backupExisting !== false) {
|
||||
backupPath = path.join(path.dirname(manifestPath(params.dataDir)), DEFAULT_BACKUP_FILE);
|
||||
await fs.mkdir(path.dirname(backupPath), { recursive: true });
|
||||
await fs.copyFile(manifestPath(params.dataDir), backupPath);
|
||||
}
|
||||
|
||||
writeManifest(params.dataDir, {
|
||||
...existing,
|
||||
store: nextStore,
|
||||
});
|
||||
|
||||
return {
|
||||
created: false,
|
||||
updated: true,
|
||||
backupPath,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
/**
|
||||
* Type stub for node-llama-cpp — the migration script does not use local
|
||||
* embedding but TypeScript still resolves the module through transitive
|
||||
* imports (sqlite.ts → embedding.ts → import("node-llama-cpp")).
|
||||
*
|
||||
* This stub satisfies the compiler without requiring the actual package.
|
||||
*/
|
||||
declare module "node-llama-cpp" {
|
||||
const _: any;
|
||||
export = _;
|
||||
}
|
||||
@@ -0,0 +1,849 @@
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { parseArgs } from "node:util";
|
||||
import type { MemoryRecord } from "../../src/core/record/l1-writer.js";
|
||||
import { listLocalProfiles } from "../../src/core/profile/profile-sync.js";
|
||||
import { createBM25Encoder } from "../../src/core/store/bm25-local.js";
|
||||
import { VectorStore, type L0RecordRow } from "../../src/core/store/sqlite.js";
|
||||
import { TcvdbMemoryStore } from "../../src/core/store/tcvdb.js";
|
||||
import type { L0Record, L1RecordRow, ProfileRecord, ProfileSyncRecord, StoreInitResult } from "../../src/core/store/types.js";
|
||||
import { readManifest } from "../../src/utils/manifest.js";
|
||||
import {
|
||||
rewriteMigrationManifest as rewriteMigrationManifestDefault,
|
||||
type RewriteMigrationManifestResult,
|
||||
} from "./manifest-write.js";
|
||||
import {
|
||||
writeMigrationPluginConfig as writeMigrationPluginConfigDefault,
|
||||
} from "./config-write.js";
|
||||
|
||||
export const DEFAULT_MIGRATION_PLUGIN_ID = "memory-tencentdb";
|
||||
export const ALL_MIGRATION_LAYERS = ["l0", "l1", "l2", "l3"] as const;
|
||||
|
||||
const TAG = "[memory-tdai][migrate]";
|
||||
|
||||
function log(message: string): void {
|
||||
process.stderr.write(`${TAG} ${message}\n`);
|
||||
}
|
||||
|
||||
function sleep(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
export type MigrationLayer = (typeof ALL_MIGRATION_LAYERS)[number];
|
||||
export type Bm25Language = "zh" | "en";
|
||||
|
||||
export interface ResolvedMigrationCliOptions {
|
||||
pluginDataDir: string;
|
||||
sqlitePath: string;
|
||||
openclawConfigPath: string;
|
||||
pluginId: string;
|
||||
layers: MigrationLayer[];
|
||||
applyConfig: boolean;
|
||||
configBackup: boolean;
|
||||
rewriteManifest: boolean;
|
||||
failIfTargetNonempty: boolean;
|
||||
verifyCounts: boolean;
|
||||
dryRun: boolean;
|
||||
yes: boolean;
|
||||
bm25Enabled: boolean;
|
||||
bm25Language: Bm25Language;
|
||||
summaryJsonPath?: string;
|
||||
jobId?: string;
|
||||
tcvdb: {
|
||||
url: string;
|
||||
username: string;
|
||||
apiKey: string;
|
||||
database: string;
|
||||
alias: string;
|
||||
embeddingModel: string;
|
||||
timeout: number;
|
||||
caPemPath?: string;
|
||||
};
|
||||
}
|
||||
|
||||
export interface MigrationPreflightSummary {
|
||||
pluginId: string;
|
||||
dryRun: boolean;
|
||||
layers: MigrationLayer[];
|
||||
paths: {
|
||||
pluginDataDir: string;
|
||||
sqlitePath: string;
|
||||
openclawConfigPath: string;
|
||||
};
|
||||
source: {
|
||||
l0Count: number;
|
||||
l1Count: number;
|
||||
profileCount: number;
|
||||
manifestExists: boolean;
|
||||
manifestStoreType: "sqlite" | "tcvdb" | null;
|
||||
};
|
||||
target: {
|
||||
url: string;
|
||||
username: string;
|
||||
database: string;
|
||||
alias: string;
|
||||
embeddingModel: string;
|
||||
timeout: number;
|
||||
bm25Enabled: boolean;
|
||||
bm25Language: Bm25Language;
|
||||
};
|
||||
options: {
|
||||
applyConfig: boolean;
|
||||
configBackup: boolean;
|
||||
rewriteManifest: boolean;
|
||||
failIfTargetNonempty: boolean;
|
||||
verifyCounts: boolean;
|
||||
yes: boolean;
|
||||
};
|
||||
migration?: {
|
||||
l0Migrated: number;
|
||||
l1Migrated: number;
|
||||
profileMigrated: number;
|
||||
targetL0Count: number;
|
||||
targetL1Count: number;
|
||||
targetProfileCount: number;
|
||||
configWritten: boolean;
|
||||
manifestWritten: boolean;
|
||||
manifestBackupPath?: string;
|
||||
};
|
||||
}
|
||||
|
||||
export const DEFAULT_MIGRATION_PAGE_SIZE = 50;
|
||||
|
||||
export interface MigrationTargetStore {
|
||||
init(providerInfo?: unknown): Promise<StoreInitResult> | StoreInitResult;
|
||||
isDegraded(): boolean;
|
||||
close(): void;
|
||||
upsertL1(record: MemoryRecord, embedding?: Float32Array): Promise<boolean> | boolean;
|
||||
upsertL0(record: L0Record, embedding?: Float32Array): Promise<boolean> | boolean;
|
||||
upsertL1Batch?(records: MemoryRecord[]): Promise<number>;
|
||||
upsertL0Batch?(records: L0Record[]): Promise<number>;
|
||||
countL1(): Promise<number> | number;
|
||||
countL0(): Promise<number> | number;
|
||||
pullProfiles?(): Promise<ProfileRecord[]>;
|
||||
syncProfiles?(records: ProfileSyncRecord[]): Promise<void>;
|
||||
}
|
||||
|
||||
export interface RunMigrationCliDeps {
|
||||
createTargetStore?: (options: ResolvedMigrationCliOptions) => MigrationTargetStore;
|
||||
writeMigrationPluginConfig?: typeof writeMigrationPluginConfigDefault;
|
||||
rewriteMigrationManifest?: (params: {
|
||||
dataDir: string;
|
||||
tcvdbUrl: string;
|
||||
tcvdbDatabase: string;
|
||||
tcvdbAlias?: string;
|
||||
}) => Promise<RewriteMigrationManifestResult>;
|
||||
verifyDelayMs?: number;
|
||||
}
|
||||
|
||||
function getRequiredString(values: Record<string, string | boolean | undefined>, key: string): string {
|
||||
const value = values[key];
|
||||
if (typeof value !== "string" || value.trim() === "") {
|
||||
throw new Error(`Missing required option --${key}`);
|
||||
}
|
||||
return value.trim();
|
||||
}
|
||||
|
||||
function getOptionalString(values: Record<string, string | boolean | undefined>, key: string): string | undefined {
|
||||
const value = values[key];
|
||||
if (typeof value !== "string") return undefined;
|
||||
const trimmed = value.trim();
|
||||
return trimmed === "" ? undefined : trimmed;
|
||||
}
|
||||
|
||||
function resolveBooleanOption(
|
||||
values: Record<string, string | boolean | undefined>,
|
||||
key: string,
|
||||
defaultValue: boolean,
|
||||
): boolean {
|
||||
const value = values[key];
|
||||
return typeof value === "boolean" ? value : defaultValue;
|
||||
}
|
||||
|
||||
function resolveTcvdbApiKey(values: Record<string, string | boolean | undefined>): string {
|
||||
const directApiKey = getOptionalString(values, "tcvdb-api-key");
|
||||
const apiKeyEnvName = getOptionalString(values, "tcvdb-api-key-env");
|
||||
|
||||
if (directApiKey && apiKeyEnvName) {
|
||||
throw new Error("Provide either --tcvdb-api-key or --tcvdb-api-key-env, not both");
|
||||
}
|
||||
if (directApiKey) {
|
||||
return directApiKey;
|
||||
}
|
||||
if (!apiKeyEnvName) {
|
||||
throw new Error("Missing required TCVDB API key input: use --tcvdb-api-key or --tcvdb-api-key-env");
|
||||
}
|
||||
|
||||
const envValue = process.env[apiKeyEnvName]?.trim();
|
||||
if (!envValue) {
|
||||
throw new Error(`Environment variable ${apiKeyEnvName} is empty or not set`);
|
||||
}
|
||||
return envValue;
|
||||
}
|
||||
|
||||
function parseLayers(rawLayers: string | undefined): MigrationLayer[] {
|
||||
if (!rawLayers) return [...ALL_MIGRATION_LAYERS];
|
||||
|
||||
const values = rawLayers
|
||||
.split(",")
|
||||
.map((layer) => layer.trim())
|
||||
.filter(Boolean);
|
||||
|
||||
if (values.length === 0) {
|
||||
throw new Error("--layers must include at least one layer");
|
||||
}
|
||||
|
||||
const uniqueLayers = [...new Set(values)];
|
||||
const invalid = uniqueLayers.filter(
|
||||
(layer): layer is string => !ALL_MIGRATION_LAYERS.includes(layer as MigrationLayer),
|
||||
);
|
||||
if (invalid.length > 0) {
|
||||
throw new Error(`Unsupported layer(s): ${invalid.join(", ")}`);
|
||||
}
|
||||
|
||||
return uniqueLayers as MigrationLayer[];
|
||||
}
|
||||
|
||||
function parseBm25Language(rawLanguage: string | undefined): Bm25Language {
|
||||
if (!rawLanguage) return "zh";
|
||||
if (rawLanguage === "zh" || rawLanguage === "en") {
|
||||
return rawLanguage;
|
||||
}
|
||||
throw new Error(`Unsupported --bm25-language value: ${rawLanguage}`);
|
||||
}
|
||||
|
||||
function parseTimeout(rawValue: string | undefined): number {
|
||||
if (!rawValue) return 10000;
|
||||
const timeout = Number(rawValue);
|
||||
if (!Number.isFinite(timeout) || timeout <= 0) {
|
||||
throw new Error(`Invalid --tcvdb-timeout-ms value: ${rawValue}`);
|
||||
}
|
||||
return timeout;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a "nothing to migrate" summary — used when source data dir or sqlite
|
||||
* file doesn't exist (e.g. fresh deployment that hasn't captured any data yet).
|
||||
*/
|
||||
function buildEmptySummary(options: ResolvedMigrationCliOptions): MigrationPreflightSummary {
|
||||
return {
|
||||
pluginId: options.pluginId,
|
||||
dryRun: options.dryRun,
|
||||
layers: options.layers,
|
||||
paths: {
|
||||
pluginDataDir: options.pluginDataDir,
|
||||
sqlitePath: options.sqlitePath,
|
||||
openclawConfigPath: options.openclawConfigPath,
|
||||
},
|
||||
source: {
|
||||
l0Count: 0,
|
||||
l1Count: 0,
|
||||
profileCount: 0,
|
||||
manifestExists: false,
|
||||
manifestStoreType: null,
|
||||
},
|
||||
target: {
|
||||
url: options.tcvdb.url,
|
||||
username: options.tcvdb.username,
|
||||
database: options.tcvdb.database,
|
||||
alias: options.tcvdb.alias,
|
||||
embeddingModel: options.tcvdb.embeddingModel,
|
||||
timeout: options.tcvdb.timeout,
|
||||
bm25Enabled: options.bm25Enabled,
|
||||
bm25Language: options.bm25Language,
|
||||
},
|
||||
options: {
|
||||
applyConfig: options.applyConfig,
|
||||
configBackup: options.configBackup,
|
||||
rewriteManifest: options.rewriteManifest,
|
||||
failIfTargetNonempty: options.failIfTargetNonempty,
|
||||
verifyCounts: options.verifyCounts,
|
||||
yes: options.yes,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function ensureReadablePath(filePath: string, label: string): Promise<void> {
|
||||
try {
|
||||
await fs.access(filePath);
|
||||
} catch {
|
||||
throw new Error(`${label} does not exist or is not accessible: ${filePath}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function ensureReadableDirectory(dirPath: string, label: string): Promise<void> {
|
||||
const stat = await fs.stat(dirPath).catch(() => null);
|
||||
if (!stat?.isDirectory()) {
|
||||
throw new Error(`${label} is not a directory: ${dirPath}`);
|
||||
}
|
||||
}
|
||||
|
||||
function safeParseMetadata(raw: string): Record<string, unknown> {
|
||||
if (!raw) return {};
|
||||
try {
|
||||
const parsed = JSON.parse(raw);
|
||||
return parsed && typeof parsed === "object" && !Array.isArray(parsed)
|
||||
? (parsed as Record<string, unknown>)
|
||||
: {};
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
function compactTimestamps(row: L1RecordRow): string[] {
|
||||
return [...new Set([row.timestamp_str, row.timestamp_start, row.timestamp_end].filter(Boolean))];
|
||||
}
|
||||
|
||||
function mapL1RowToMemoryRecord(row: L1RecordRow): MemoryRecord {
|
||||
const timestamps = compactTimestamps(row);
|
||||
const fallbackIso = row.updated_time || row.created_time || row.timestamp_end || row.timestamp_str || new Date(0).toISOString();
|
||||
return {
|
||||
id: row.record_id,
|
||||
content: row.content,
|
||||
type: row.type as MemoryRecord["type"],
|
||||
priority: row.priority,
|
||||
scene_name: row.scene_name,
|
||||
source_message_ids: [],
|
||||
metadata: safeParseMetadata(row.metadata_json),
|
||||
timestamps,
|
||||
createdAt: row.created_time || fallbackIso,
|
||||
updatedAt: row.updated_time || row.created_time || fallbackIso,
|
||||
sessionKey: row.session_key || "",
|
||||
sessionId: row.session_id || "",
|
||||
};
|
||||
}
|
||||
|
||||
function mapL0RowToRecord(row: L0RecordRow): L0Record {
|
||||
return {
|
||||
id: row.record_id,
|
||||
sessionKey: row.session_key,
|
||||
sessionId: row.session_id || "",
|
||||
role: row.role,
|
||||
messageText: row.message_text,
|
||||
recordedAt: row.recorded_at || "",
|
||||
timestamp: row.timestamp ?? 0,
|
||||
};
|
||||
}
|
||||
|
||||
function createTargetStoreDefault(options: ResolvedMigrationCliOptions): MigrationTargetStore {
|
||||
const bm25Encoder = createBM25Encoder(
|
||||
{
|
||||
enabled: options.bm25Enabled,
|
||||
language: options.bm25Language,
|
||||
},
|
||||
);
|
||||
|
||||
return new TcvdbMemoryStore({
|
||||
url: options.tcvdb.url,
|
||||
username: options.tcvdb.username,
|
||||
apiKey: options.tcvdb.apiKey,
|
||||
database: options.tcvdb.database,
|
||||
embeddingModel: options.tcvdb.embeddingModel,
|
||||
timeout: options.tcvdb.timeout,
|
||||
caPemPath: options.tcvdb.caPemPath,
|
||||
bm25Encoder,
|
||||
});
|
||||
}
|
||||
|
||||
async function ensureTargetIsEmpty(
|
||||
options: ResolvedMigrationCliOptions,
|
||||
targetStore: MigrationTargetStore,
|
||||
): Promise<void> {
|
||||
if (!options.failIfTargetNonempty) return;
|
||||
|
||||
const [existingL1, existingL0, existingProfiles] = await Promise.all([
|
||||
Promise.resolve(targetStore.countL1()),
|
||||
Promise.resolve(targetStore.countL0()),
|
||||
targetStore.pullProfiles ? targetStore.pullProfiles().then((records) => records.length) : Promise.resolve(0),
|
||||
]);
|
||||
|
||||
if (existingL1 > 0 || existingL0 > 0 || existingProfiles > 0) {
|
||||
throw new Error(
|
||||
`Target store is not empty (L1=${existingL1}, L0=${existingL0}, profiles=${existingProfiles})`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function migrateL1Records(sourceStore: VectorStore, targetStore: MigrationTargetStore, pageSize: number): Promise<number> {
|
||||
let migrated = 0;
|
||||
let cursor = "";
|
||||
const useBatch = typeof targetStore.upsertL1Batch === "function";
|
||||
|
||||
// eslint-disable-next-line no-constant-condition
|
||||
while (true) {
|
||||
const rows = sourceStore.queryL1RecordsCursor(cursor, pageSize);
|
||||
if (rows.length === 0) break;
|
||||
|
||||
const records = rows.map(mapL1RowToMemoryRecord);
|
||||
|
||||
if (useBatch) {
|
||||
const count = await targetStore.upsertL1Batch!(records);
|
||||
if (count === 0) {
|
||||
throw new Error(`Failed to batch migrate L1 records (cursor=${cursor}, page=${rows.length})`);
|
||||
}
|
||||
} else {
|
||||
for (const record of records) {
|
||||
const ok = await Promise.resolve(targetStore.upsertL1(record));
|
||||
if (!ok) throw new Error(`Failed to migrate L1 record ${record.id}`);
|
||||
}
|
||||
}
|
||||
|
||||
migrated += rows.length;
|
||||
cursor = rows[rows.length - 1].record_id;
|
||||
log(`L1: 已迁移 ${migrated} 条...`);
|
||||
|
||||
if (rows.length < pageSize) break;
|
||||
}
|
||||
|
||||
log(`L1: 迁移完成，共 ${migrated} 条`);
|
||||
return migrated;
|
||||
}
|
||||
|
||||
async function migrateL0Records(sourceStore: VectorStore, targetStore: MigrationTargetStore, pageSize: number): Promise<number> {
|
||||
let migrated = 0;
|
||||
let cursor = "";
|
||||
const useBatch = typeof targetStore.upsertL0Batch === "function";
|
||||
|
||||
// eslint-disable-next-line no-constant-condition
|
||||
while (true) {
|
||||
const rows = sourceStore.queryL0RecordsCursor(cursor, pageSize);
|
||||
if (rows.length === 0) break;
|
||||
|
||||
const records = rows.map(mapL0RowToRecord);
|
||||
|
||||
if (useBatch) {
|
||||
const count = await targetStore.upsertL0Batch!(records);
|
||||
if (count === 0) {
|
||||
throw new Error(`Failed to batch migrate L0 records (cursor=${cursor}, page=${rows.length})`);
|
||||
}
|
||||
} else {
|
||||
for (const record of records) {
|
||||
const ok = await Promise.resolve(targetStore.upsertL0(record));
|
||||
if (!ok) throw new Error(`Failed to migrate L0 record ${record.id}`);
|
||||
}
|
||||
}
|
||||
|
||||
migrated += rows.length;
|
||||
cursor = rows[rows.length - 1].record_id;
|
||||
log(`L0: 已迁移 ${migrated} 条...`);
|
||||
|
||||
if (rows.length < pageSize) break;
|
||||
}
|
||||
|
||||
log(`L0: 迁移完成，共 ${migrated} 条`);
|
||||
return migrated;
|
||||
}
|
||||
|
||||
async function migrateProfiles(
|
||||
pluginDataDir: string,
|
||||
targetStore: MigrationTargetStore,
|
||||
): Promise<number> {
|
||||
const profiles = await listLocalProfiles(pluginDataDir);
|
||||
if (profiles.length === 0) {
|
||||
log("Profiles: 无本地 profile，跳过");
|
||||
return 0;
|
||||
}
|
||||
if (!targetStore.syncProfiles) {
|
||||
throw new Error("Target store does not support profile sync");
|
||||
}
|
||||
log(`Profiles: 发现 ${profiles.length} 个本地 profile，开始同步...`);
|
||||
await targetStore.syncProfiles(profiles);
|
||||
log(`Profiles: 同步完成，共 ${profiles.length} 个`);
|
||||
return profiles.length;
|
||||
}
|
||||
|
||||
async function verifyMigratedCounts(
|
||||
summary: MigrationPreflightSummary,
|
||||
targetStore: MigrationTargetStore,
|
||||
delayMs: number,
|
||||
): Promise<{ l1Count: number; l0Count: number; profileCount: number }> {
|
||||
if (delayMs > 0) {
|
||||
log(`等待 ${Math.round(delayMs / 1000)} 秒让远端数据落盘...`);
|
||||
await sleep(delayMs);
|
||||
}
|
||||
log("开始校验迁移数量...");
|
||||
|
||||
const [l1Count, l0Count, profileCount] = await Promise.all([
|
||||
Promise.resolve(targetStore.countL1()),
|
||||
Promise.resolve(targetStore.countL0()),
|
||||
targetStore.pullProfiles ? targetStore.pullProfiles().then((records) => records.length) : Promise.resolve(0),
|
||||
]);
|
||||
|
||||
log(`校验结果: L1=${l1Count}/${summary.source.l1Count}, L0=${l0Count}/${summary.source.l0Count}, Profiles=${profileCount}/${summary.source.profileCount}`);
|
||||
|
||||
if (l1Count !== summary.source.l1Count) {
|
||||
throw new Error(`L1 count verification failed: source=${summary.source.l1Count}, target=${l1Count}`);
|
||||
}
|
||||
if (l0Count !== summary.source.l0Count) {
|
||||
throw new Error(`L0 count verification failed: source=${summary.source.l0Count}, target=${l0Count}`);
|
||||
}
|
||||
if (profileCount !== summary.source.profileCount) {
|
||||
throw new Error(
|
||||
`Profile count verification failed: source=${summary.source.profileCount}, target=${profileCount}`,
|
||||
);
|
||||
}
|
||||
|
||||
log("校验通过");
|
||||
return { l1Count, l0Count, profileCount };
|
||||
}
|
||||
|
||||
async function writeSummaryJson(
|
||||
summaryJsonPath: string | undefined,
|
||||
summary: MigrationPreflightSummary,
|
||||
): Promise<void> {
|
||||
if (!summaryJsonPath) return;
|
||||
await fs.mkdir(path.dirname(summaryJsonPath), { recursive: true });
|
||||
await fs.writeFile(summaryJsonPath, `${JSON.stringify(summary, null, 2)}\n`, "utf-8");
|
||||
}
|
||||
|
||||
function printUsageAndExit(): never {
|
||||
const text = `
|
||||
SQLite → 腾讯云向量数据库迁移工具
|
||||
|
||||
Usage:
|
||||
migrate-sqlite-to-tcvdb [options]
|
||||
|
||||
Required:
|
||||
--plugin-data-dir <path> 插件数据目录路径
|
||||
--openclaw-config-path <path> openclaw.json 配置文件路径
|
||||
--tcvdb-url <url> TCVDB 服务地址
|
||||
--tcvdb-username <name> TCVDB 用户名
|
||||
--tcvdb-database <name> TCVDB 数据库名
|
||||
--tcvdb-embedding-model <name> Embedding 模型名称
|
||||
--tcvdb-api-key <key> TCVDB API 密钥（明文，与 --tcvdb-api-key-env 二选一）
|
||||
--tcvdb-api-key-env <var> 包含 API 密钥的环境变量名
|
||||
|
||||
Optional:
|
||||
--sqlite-path <path> SQLite 数据库路径（默认: <plugin-data-dir>/vectors.db）
|
||||
--plugin-id <id> 写入配置时使用的插件 ID（默认: memory-tencentdb）
|
||||
--layers <l0,l1,l2,l3> 要迁移的层，逗号分隔（默认: l0,l1,l2,l3）
|
||||
--tcvdb-alias <name> 用户自定义别名
|
||||
--tcvdb-timeout-ms <ms> 请求超时（默认: 10000）
|
||||
--tcvdb-ca-pem <path> CA 证书 PEM 文件路径（HTTPS 连接时使用）
|
||||
--bm25-language <zh|en> BM25 分词语言（默认: zh）
|
||||
--summary-json-path <path> 将迁移摘要写入此文件
|
||||
--job-id <id> 迁移任务 ID（用于追踪）
|
||||
|
||||
Flags:
|
||||
--dry-run 仅预览，不执行写入
|
||||
--yes 跳过交互确认
|
||||
--no-apply-config 不自动更新 openclaw.json
|
||||
--no-config-backup 写入配置前不备份
|
||||
--no-rewrite-manifest 不更新 manifest.json
|
||||
--no-fail-if-target-nonempty 目标库非空时不中止
|
||||
--no-verify-counts 迁移后不校验记录数
|
||||
--no-bm25-enabled 禁用 BM25 稀疏向量
|
||||
|
||||
-h, --help 显示此帮助信息
|
||||
|
||||
Examples:
|
||||
# 预检模式
|
||||
migrate-sqlite-to-tcvdb \\
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \\
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \\
|
||||
--tcvdb-url http://127.0.0.1:80 --tcvdb-username root \\
|
||||
--tcvdb-api-key-env TCVDB_API_KEY \\
|
||||
--tcvdb-database agent_memory_prod \\
|
||||
--tcvdb-embedding-model bge-large-zh \\
|
||||
--dry-run
|
||||
|
||||
# 正式迁移
|
||||
migrate-sqlite-to-tcvdb \\
|
||||
--plugin-data-dir ~/.openclaw/memory-tdai \\
|
||||
--openclaw-config-path ~/.openclaw/openclaw.json \\
|
||||
--tcvdb-url http://127.0.0.1:80 --tcvdb-username root \\
|
||||
--tcvdb-api-key-env TCVDB_API_KEY \\
|
||||
--tcvdb-database agent_memory_prod \\
|
||||
--tcvdb-embedding-model bge-large-zh \\
|
||||
--yes
|
||||
`.trimStart();
|
||||
process.stdout.write(text);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
export function resolveMigrationCliOptions(argv: string[]): ResolvedMigrationCliOptions {
|
||||
const { values } = parseArgs({
|
||||
args: argv,
|
||||
strict: true,
|
||||
allowPositionals: false,
|
||||
allowNegative: true,
|
||||
options: {
|
||||
help: { type: "boolean", short: "h" },
|
||||
"plugin-data-dir": { type: "string" },
|
||||
"sqlite-path": { type: "string" },
|
||||
"openclaw-config-path": { type: "string" },
|
||||
"plugin-id": { type: "string" },
|
||||
layers: { type: "string" },
|
||||
"apply-config": { type: "boolean" },
|
||||
"config-backup": { type: "boolean" },
|
||||
"rewrite-manifest": { type: "boolean" },
|
||||
"fail-if-target-nonempty": { type: "boolean" },
|
||||
"verify-counts": { type: "boolean" },
|
||||
"dry-run": { type: "boolean" },
|
||||
yes: { type: "boolean" },
|
||||
"bm25-enabled": { type: "boolean" },
|
||||
"bm25-language": { type: "string" },
|
||||
"summary-json-path": { type: "string" },
|
||||
"job-id": { type: "string" },
|
||||
"tcvdb-url": { type: "string" },
|
||||
"tcvdb-username": { type: "string" },
|
||||
"tcvdb-api-key": { type: "string" },
|
||||
"tcvdb-api-key-env": { type: "string" },
|
||||
"tcvdb-database": { type: "string" },
|
||||
"tcvdb-alias": { type: "string" },
|
||||
"tcvdb-embedding-model": { type: "string" },
|
||||
"tcvdb-timeout-ms": { type: "string" },
|
||||
"tcvdb-ca-pem": { type: "string" },
|
||||
},
|
||||
});
|
||||
|
||||
if (values.help) {
|
||||
printUsageAndExit();
|
||||
}
|
||||
|
||||
const pluginDataDir = path.resolve(getRequiredString(values, "plugin-data-dir"));
|
||||
const sqlitePath = path.resolve(
|
||||
getOptionalString(values, "sqlite-path") ?? path.join(pluginDataDir, "vectors.db"),
|
||||
);
|
||||
const summaryJsonPath = getOptionalString(values, "summary-json-path");
|
||||
|
||||
return {
|
||||
pluginDataDir,
|
||||
sqlitePath,
|
||||
openclawConfigPath: path.resolve(getRequiredString(values, "openclaw-config-path")),
|
||||
pluginId: getOptionalString(values, "plugin-id") ?? DEFAULT_MIGRATION_PLUGIN_ID,
|
||||
layers: parseLayers(getOptionalString(values, "layers")),
|
||||
applyConfig: resolveBooleanOption(values, "apply-config", true),
|
||||
configBackup: resolveBooleanOption(values, "config-backup", true),
|
||||
rewriteManifest: resolveBooleanOption(values, "rewrite-manifest", true),
|
||||
failIfTargetNonempty: resolveBooleanOption(values, "fail-if-target-nonempty", true),
|
||||
verifyCounts: resolveBooleanOption(values, "verify-counts", true),
|
||||
dryRun: resolveBooleanOption(values, "dry-run", false),
|
||||
yes: resolveBooleanOption(values, "yes", false),
|
||||
bm25Enabled: resolveBooleanOption(values, "bm25-enabled", true),
|
||||
bm25Language: parseBm25Language(getOptionalString(values, "bm25-language")),
|
||||
summaryJsonPath: summaryJsonPath ? path.resolve(summaryJsonPath) : undefined,
|
||||
jobId: getOptionalString(values, "job-id"),
|
||||
tcvdb: {
|
||||
url: getRequiredString(values, "tcvdb-url"),
|
||||
username: getRequiredString(values, "tcvdb-username"),
|
||||
apiKey: resolveTcvdbApiKey(values),
|
||||
database: getRequiredString(values, "tcvdb-database"),
|
||||
alias: getOptionalString(values, "tcvdb-alias") ?? "",
|
||||
embeddingModel: getRequiredString(values, "tcvdb-embedding-model"),
|
||||
timeout: parseTimeout(getOptionalString(values, "tcvdb-timeout-ms")),
|
||||
caPemPath: getOptionalString(values, "tcvdb-ca-pem"),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export async function collectMigrationPreflight(
|
||||
options: ResolvedMigrationCliOptions,
|
||||
): Promise<MigrationPreflightSummary> {
|
||||
// ── 优雅处理"数据目录 / sqlite 不存在"的场景 ──
|
||||
// 如果源数据路径不存在（全新部署、尚未有任何 capture），不报错——
|
||||
// 返回"全零"summary，让调用方判断是否跳过迁移。
|
||||
const dirExists = await fs.stat(options.pluginDataDir).then(s => s.isDirectory()).catch(() => false);
|
||||
if (!dirExists) {
|
||||
log(`plugin data directory 不存在，无需迁移: ${options.pluginDataDir}`);
|
||||
return buildEmptySummary(options);
|
||||
}
|
||||
const sqliteExists = await fs.access(options.sqlitePath).then(() => true).catch(() => false);
|
||||
if (!sqliteExists) {
|
||||
log(`sqlite database 不存在，无需迁移: ${options.sqlitePath}`);
|
||||
return buildEmptySummary(options);
|
||||
}
|
||||
await ensureReadablePath(options.openclawConfigPath, "OpenClaw config file");
|
||||
|
||||
const store = new VectorStore(options.sqlitePath, 0);
|
||||
const initResult = store.init();
|
||||
if (store.isDegraded()) {
|
||||
store.close();
|
||||
throw new Error(`Failed to open sqlite store for migration preflight: ${initResult.reason ?? "unknown error"}`);
|
||||
}
|
||||
|
||||
try {
|
||||
const [profiles, manifest] = await Promise.all([
|
||||
listLocalProfiles(options.pluginDataDir),
|
||||
Promise.resolve(readManifest(options.pluginDataDir)),
|
||||
]);
|
||||
|
||||
return {
|
||||
pluginId: options.pluginId,
|
||||
dryRun: options.dryRun,
|
||||
layers: options.layers,
|
||||
paths: {
|
||||
pluginDataDir: options.pluginDataDir,
|
||||
sqlitePath: options.sqlitePath,
|
||||
openclawConfigPath: options.openclawConfigPath,
|
||||
},
|
||||
source: {
|
||||
l0Count: store.countL0(),
|
||||
l1Count: store.countL1(),
|
||||
profileCount: profiles.length,
|
||||
manifestExists: manifest !== null,
|
||||
manifestStoreType: manifest?.store.type ?? null,
|
||||
},
|
||||
target: {
|
||||
url: options.tcvdb.url,
|
||||
username: options.tcvdb.username,
|
||||
database: options.tcvdb.database,
|
||||
alias: options.tcvdb.alias,
|
||||
embeddingModel: options.tcvdb.embeddingModel,
|
||||
timeout: options.tcvdb.timeout,
|
||||
bm25Enabled: options.bm25Enabled,
|
||||
bm25Language: options.bm25Language,
|
||||
},
|
||||
options: {
|
||||
applyConfig: options.applyConfig,
|
||||
configBackup: options.configBackup,
|
||||
rewriteManifest: options.rewriteManifest,
|
||||
failIfTargetNonempty: options.failIfTargetNonempty,
|
||||
verifyCounts: options.verifyCounts,
|
||||
yes: options.yes,
|
||||
},
|
||||
};
|
||||
} finally {
|
||||
store.close();
|
||||
}
|
||||
}
|
||||
|
||||
export async function runMigrationCli(
|
||||
argv: string[],
|
||||
deps: RunMigrationCliDeps = {},
|
||||
): Promise<MigrationPreflightSummary> {
|
||||
const options = resolveMigrationCliOptions(argv);
|
||||
log("开始预检...");
|
||||
const summary = await collectMigrationPreflight(options);
|
||||
log(`预检完成: 源数据 L1=${summary.source.l1Count}, L0=${summary.source.l0Count}, Profiles=${summary.source.profileCount}`);
|
||||
log(`目标: ${summary.target.url} / ${summary.target.database}`);
|
||||
|
||||
const hasSourceData = summary.source.l0Count > 0 || summary.source.l1Count > 0 || summary.source.profileCount > 0;
|
||||
|
||||
if (!hasSourceData) {
|
||||
log("源数据为空，跳过数据迁移。");
|
||||
}
|
||||
|
||||
if (options.dryRun) {
|
||||
log("预检模式 (dry-run)，不执行写入");
|
||||
await writeSummaryJson(options.summaryJsonPath, summary);
|
||||
return summary;
|
||||
}
|
||||
|
||||
const createTargetStore = deps.createTargetStore ?? createTargetStoreDefault;
|
||||
const writeMigrationPluginConfig = deps.writeMigrationPluginConfig ?? writeMigrationPluginConfigDefault;
|
||||
const rewriteMigrationManifest = deps.rewriteMigrationManifest ?? (async (params) =>
|
||||
await rewriteMigrationManifestDefault(params));
|
||||
|
||||
const migration = {
|
||||
l1Migrated: 0,
|
||||
l0Migrated: 0,
|
||||
profileMigrated: 0,
|
||||
targetL1Count: 0,
|
||||
targetL0Count: 0,
|
||||
targetProfileCount: 0,
|
||||
configWritten: false,
|
||||
manifestWritten: false,
|
||||
manifestBackupPath: undefined as string | undefined,
|
||||
};
|
||||
|
||||
if (hasSourceData) {
|
||||
log("打开 SQLite 源库...");
|
||||
const sourceStore = new VectorStore(options.sqlitePath, 0);
|
||||
const sourceInitResult = sourceStore.init();
|
||||
if (sourceStore.isDegraded()) {
|
||||
sourceStore.close();
|
||||
throw new Error(`Failed to reopen sqlite source store: ${sourceInitResult.reason ?? "unknown error"}`);
|
||||
}
|
||||
|
||||
log("初始化目标库...");
|
||||
const targetStore = createTargetStore(options);
|
||||
|
||||
try {
|
||||
await Promise.resolve(targetStore.init());
|
||||
if (targetStore.isDegraded()) {
|
||||
throw new Error("Target store entered degraded mode during initialization");
|
||||
}
|
||||
|
||||
await ensureTargetIsEmpty(options, targetStore);
|
||||
|
||||
const pageSize = DEFAULT_MIGRATION_PAGE_SIZE;
|
||||
log(`分页大小: ${pageSize} 条/批`);
|
||||
|
||||
migration.l1Migrated = options.layers.includes("l1") ? await migrateL1Records(sourceStore, targetStore, pageSize) : 0;
|
||||
migration.l0Migrated = options.layers.includes("l0") ? await migrateL0Records(sourceStore, targetStore, pageSize) : 0;
|
||||
migration.profileMigrated = options.layers.includes("l2") || options.layers.includes("l3")
|
||||
? await migrateProfiles(options.pluginDataDir, targetStore)
|
||||
: 0;
|
||||
|
||||
const verifyDelayMs = deps.verifyDelayMs ?? 10_000;
|
||||
|
||||
const verifiedCounts = options.verifyCounts
|
||||
? await verifyMigratedCounts(summary, targetStore, verifyDelayMs)
|
||||
: {
|
||||
l1Count: await Promise.resolve(targetStore.countL1()),
|
||||
l0Count: await Promise.resolve(targetStore.countL0()),
|
||||
profileCount: targetStore.pullProfiles ? (await targetStore.pullProfiles()).length : 0,
|
||||
};
|
||||
|
||||
migration.targetL1Count = verifiedCounts.l1Count;
|
||||
migration.targetL0Count = verifiedCounts.l0Count;
|
||||
migration.targetProfileCount = verifiedCounts.profileCount;
|
||||
} finally {
|
||||
sourceStore.close();
|
||||
targetStore.close();
|
||||
}
|
||||
}
|
||||
|
||||
if (options.applyConfig) {
|
||||
log(`写入配置到 ${options.openclawConfigPath} ...`);
|
||||
await writeMigrationPluginConfig({
|
||||
configPath: options.openclawConfigPath,
|
||||
pluginId: options.pluginId,
|
||||
tcvdb: {
|
||||
url: options.tcvdb.url,
|
||||
username: options.tcvdb.username,
|
||||
apiKey: options.tcvdb.apiKey,
|
||||
database: options.tcvdb.database,
|
||||
alias: options.tcvdb.alias,
|
||||
embeddingModel: options.tcvdb.embeddingModel,
|
||||
timeout: options.tcvdb.timeout,
|
||||
},
|
||||
bm25: {
|
||||
enabled: options.bm25Enabled,
|
||||
language: options.bm25Language,
|
||||
},
|
||||
});
|
||||
migration.configWritten = true;
|
||||
log("配置写入完成");
|
||||
}
|
||||
|
||||
if (options.rewriteManifest) {
|
||||
log("更新 manifest...");
|
||||
const manifestResult = await rewriteMigrationManifest({
|
||||
dataDir: options.pluginDataDir,
|
||||
tcvdbUrl: options.tcvdb.url,
|
||||
tcvdbDatabase: options.tcvdb.database,
|
||||
tcvdbAlias: options.tcvdb.alias || undefined,
|
||||
});
|
||||
migration.manifestWritten = manifestResult.created || manifestResult.updated;
|
||||
migration.manifestBackupPath = manifestResult.backupPath;
|
||||
log(`Manifest ${manifestResult.created ? "创建" : "更新"}完成${manifestResult.backupPath ? `，备份: ${manifestResult.backupPath}` : ""}`);
|
||||
}
|
||||
|
||||
summary.migration = {
|
||||
l1Migrated: migration.l1Migrated,
|
||||
l0Migrated: migration.l0Migrated,
|
||||
profileMigrated: migration.profileMigrated,
|
||||
targetL1Count: migration.targetL1Count,
|
||||
targetL0Count: migration.targetL0Count,
|
||||
targetProfileCount: migration.targetProfileCount,
|
||||
configWritten: migration.configWritten,
|
||||
manifestWritten: migration.manifestWritten,
|
||||
manifestBackupPath: migration.manifestBackupPath,
|
||||
};
|
||||
|
||||
await writeSummaryJson(options.summaryJsonPath, summary);
|
||||
log("迁移全部完成!");
|
||||
return summary;
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "es2023",
|
||||
"module": "NodeNext",
|
||||
"moduleResolution": "NodeNext",
|
||||
"declaration": false,
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true,
|
||||
"types": ["node"],
|
||||
"paths": {
|
||||
"node-llama-cpp": ["./node-llama-cpp.d.ts"]
|
||||
},
|
||||
"outDir": "dist",
|
||||
"rootDir": "../.."
|
||||
},
|
||||
"files": [
|
||||
"cli-entry.ts",
|
||||
"sqlite-to-tcvdb.ts",
|
||||
"config-write.ts",
|
||||
"manifest-write.ts"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,317 @@
|
||||
#!/usr/bin/env bash
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# OpenClaw Patch: after_tool_call hook 注入 session messages
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# 用途：在 after_tool_call hookEvent 中注入 ctx.params.session?.messages，
|
||||
# 使 context-offload 等插件能在工具调用后访问完整历史消息列表。
|
||||
#
|
||||
# 兼容策略（按优先级尝试，首个成功即停止）：
|
||||
# 策略 1: AST-like — 搜索 hookEvent 对象中 durationMs 字段，在其后追加 messages
|
||||
# 策略 2: 旧版 dispatch-*.js — 针对早期版本文件布局
|
||||
# 策略 3: runAfterToolCall 锚点 — 在 hookEvent 关闭大括号前插入
|
||||
# 策略 4: 通用 fallback — 基于 after_tool_call + durationMs 的宽松匹配
|
||||
#
|
||||
# 用法：
|
||||
# bash openclaw-after-tool-call-messages.patch.sh
|
||||
# bash openclaw-after-tool-call-messages.patch.sh /custom/path/to/openclaw
|
||||
#
|
||||
# 幂等：已 patch 过的文件会自动跳过，可安全重复执行。
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
set -euo pipefail
|
||||
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m'
|
||||
info() { echo -e "${CYAN}[INFO]${NC} $*"; }
|
||||
ok() { echo -e "${GREEN}[OK]${NC} $*"; }
|
||||
warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
|
||||
fail() { echo -e "${RED}[FAIL]${NC} $*"; exit 1; }
|
||||
debug() { [[ "${DEBUG:-}" == "1" ]] && echo -e "${CYAN}[DEBUG]${NC} $*" || true; }
|
||||
|
||||
# ─── 定位 OpenClaw 安装目录 ──────────────────────────────────────
|
||||
# Uses Node.js require.resolve to locate the package root — handles
|
||||
# nvm, pnpm, npm, yarn, volta, and any other layout automatically.
|
||||
_node_resolve_openclaw() {
|
||||
node -e "
|
||||
const {dirname, join} = require('path');
|
||||
const {realpathSync, existsSync, readFileSync, statSync} = require('fs');
|
||||
|
||||
// Helper: walk up from a file/dir to find the openclaw package root
|
||||
function walkUp(start) {
|
||||
let dir = statSync(start).isDirectory() ? start : dirname(start);
|
||||
for (let i = 0; i < 10; i++) {
|
||||
const pj = join(dir, 'package.json');
|
||||
if (existsSync(pj)) {
|
||||
try {
|
||||
const pkg = JSON.parse(readFileSync(pj, 'utf8'));
|
||||
if (pkg.name === 'openclaw') return dir;
|
||||
} catch {}
|
||||
}
|
||||
const parent = dirname(dir);
|
||||
if (parent === dir) break;
|
||||
dir = parent;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Strategy 1: which openclaw → realpath → walk up
|
||||
try {
|
||||
const {execSync} = require('child_process');
|
||||
const bin = execSync('which openclaw', {encoding:'utf8'}).trim();
|
||||
const real = realpathSync(bin);
|
||||
const found = walkUp(real);
|
||||
if (found) { console.log(found); process.exit(0); }
|
||||
|
||||
// pnpm uses shell shims: the bin file is a script, not a symlink.
|
||||
// Read the shim content to extract the real entry point path.
|
||||
const content = readFileSync(bin, 'utf8');
|
||||
// pnpm shim contains a line like: exec node \"/path/.../openclaw/dist/cli.js\"
|
||||
// or: require(\"/path/.../openclaw/dist/cli.js\")
|
||||
const m = content.match(/['\"]([^'\"]*openclaw[^'\"]*\\.(?:js|mjs))['\"]/) ||
|
||||
content.match(/['\"]([^'\"]*openclaw[^'\"]*)['\"].*node/);
|
||||
if (m) {
|
||||
const shimTarget = realpathSync(m[1]);
|
||||
const found2 = walkUp(shimTarget);
|
||||
if (found2) { console.log(found2); process.exit(0); }
|
||||
}
|
||||
} catch {}
|
||||
|
||||
// Strategy 2: search common pnpm/npm global paths
|
||||
const {execSync: exec2} = require('child_process');
|
||||
const searchDirs = [
|
||||
join(process.env.HOME || '/root', '.local/share/pnpm'),
|
||||
join(process.env.HOME || '/root', '.local/node/lib/node_modules'),
|
||||
'/usr/local/lib/node_modules',
|
||||
'/usr/lib/node_modules',
|
||||
];
|
||||
for (const base of searchDirs) {
|
||||
if (!existsSync(base)) continue;
|
||||
try {
|
||||
const out = exec2(
|
||||
'find ' + JSON.stringify(base) + ' -maxdepth 8 -name package.json -path \"*/openclaw/package.json\" 2>/dev/null',
|
||||
{encoding:'utf8', timeout: 5000}
|
||||
).trim();
|
||||
for (const line of out.split('\\n')) {
|
||||
if (!line) continue;
|
||||
try {
|
||||
const pkg = JSON.parse(readFileSync(line, 'utf8'));
|
||||
if (pkg.name === 'openclaw') { console.log(dirname(line)); process.exit(0); }
|
||||
} catch {}
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
|
||||
process.exit(1);
|
||||
" 2>/dev/null
|
||||
}
|
||||
|
||||
if [[ -n "${1:-}" ]]; then
|
||||
OPENCLAW_DIR="$1"
|
||||
elif OPENCLAW_DIR="$(_node_resolve_openclaw)"; then
|
||||
debug "Node.js resolved openclaw → $OPENCLAW_DIR"
|
||||
else
|
||||
fail "找不到 OpenClaw 安装目录。请手动指定：\n bash $0 /path/to/openclaw"
|
||||
fi
|
||||
|
||||
DIST_DIR="$OPENCLAW_DIR/dist"
|
||||
|
||||
if [[ ! -d "$DIST_DIR" ]]; then
|
||||
fail "dist 目录不存在: $DIST_DIR"
|
||||
fi
|
||||
|
||||
info "OpenClaw 目录: $OPENCLAW_DIR"
|
||||
|
||||
# ─── 检测 OpenClaw 版本 ──────────────────────────────────────────
|
||||
VERSION=$(grep -oP '"version"\s*:\s*"\K[^"]+' "$OPENCLAW_DIR/package.json" 2>/dev/null || echo "unknown")
|
||||
info "检测到 OpenClaw 版本: $VERSION"
|
||||
|
||||
# ─── 已 patch 检测 ───────────────────────────────────────────────
|
||||
# 核心标记：hookEvent 内部 durationMs 之后紧跟 messages 注入
|
||||
# 支持多种缩进格式（tab / 空格 / 混合）
|
||||
INJECTION_CODE='messages: ctx.params.session?.messages'
|
||||
INJECTION_CODE_ALT='messages:ctx.params.session?.messages'
|
||||
|
||||
is_already_patched() {
|
||||
local f="$1"
|
||||
# 方法 1: 精确检测 — durationMs 后面紧跟 messages 注入（允许任意空白）
|
||||
if perl -0777 -ne 'exit(0) if /durationMs[,\s]*\n\s*messages\s*:\s*ctx\.params\.session\?\s*\.messages/; exit(1)' "$f" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
# 方法 2: 上下文检测 — after_tool_call hookEvent 对象内部（durationMs 附近）有 messages 注入
|
||||
# 注意: 不能用全文件检测，因为 before_compaction 也有 messages: ctx.params.session.messages
|
||||
if perl -0777 -ne 'exit(0) if /(?:hookEvent|hook_event)\s*=\s*\{[\s\S]{0,500}durationMs[\s\S]{0,100}messages\s*:\s*ctx\.params\.session/; exit(1)' "$f" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
verify_patch() {
|
||||
local f="$1"
|
||||
is_already_patched "$f"
|
||||
}
|
||||
|
||||
# ─── 备份工具 ────────────────────────────────────────────────────
|
||||
backup_file() {
|
||||
local f="$1"
|
||||
local bak="${f}.pre-offload-patch.bak"
|
||||
if [[ ! -f "$bak" ]]; then
|
||||
cp "$f" "$bak"
|
||||
debug "备份: $bak"
|
||||
fi
|
||||
}
|
||||
|
||||
# ─── 查找所有候选文件 ────────────────────────────────────────────
|
||||
# 收集所有包含 after_tool_call 的 JS 文件（不限子目录深度）
|
||||
mapfile -t CANDIDATE_FILES < <(grep -rl 'after_tool_call' "$DIST_DIR" --include='*.js' 2>/dev/null || true)
|
||||
|
||||
if [[ ${#CANDIDATE_FILES[@]} -eq 0 ]]; then
|
||||
warn "在 $DIST_DIR 下未找到包含 after_tool_call 的 JS 文件"
|
||||
fi
|
||||
|
||||
info "找到 ${#CANDIDATE_FILES[@]} 个候选文件"
|
||||
|
||||
PATCHED=0
|
||||
SKIPPED=0
|
||||
FAILED=0
|
||||
|
||||
# ─── 对每个候选文件尝试多种策略 ──────────────────────────────────
|
||||
for f in "${CANDIDATE_FILES[@]}"; do
|
||||
fname="$(basename "$f")"
|
||||
relpath="${f#$DIST_DIR/}"
|
||||
|
||||
# 已 patch → 跳过
|
||||
if is_already_patched "$f"; then
|
||||
warn "$relpath — 已经 patch 过，跳过"
|
||||
((SKIPPED++)) || true
|
||||
continue
|
||||
fi
|
||||
|
||||
# 确认文件中有 durationMs（hookEvent 的标志字段）
|
||||
if ! grep -q 'durationMs' "$f" 2>/dev/null; then
|
||||
debug "$relpath — 不含 durationMs，非 patch 目标，跳过"
|
||||
continue
|
||||
fi
|
||||
|
||||
# 确认 durationMs 附近有 after_tool_call 上下文（避免误匹配 before_compaction 等）
|
||||
if ! perl -0777 -ne 'exit(0) if /after_tool_call[\s\S]{0,2000}durationMs/; exit(1)' "$f" 2>/dev/null; then
|
||||
debug "$relpath — durationMs 不在 after_tool_call 上下文中，跳过"
|
||||
continue
|
||||
fi
|
||||
|
||||
backup_file "$f"
|
||||
applied=false
|
||||
|
||||
# ── 策略 1: hookEvent 对象中 durationMs 是最后一个字段 ────────
|
||||
# 匹配: durationMs<换行><空白>};<换行><空白>hookRunnerAfter
|
||||
# 或: durationMs<换行><空白>};<换行><空白>await ...hookRunner...afterToolCall
|
||||
# 缩进用 \s+ 宽松匹配
|
||||
if [[ "$applied" == "false" ]]; then
|
||||
if perl -0777 -ne 'exit(0) if /durationMs\s*\n(\s*)\};\s*\n\s*(hookRunnerAfter|await\s+\S*hookRunner\S*\.runAfterToolCall|hookRunner\S*\.runAfterToolCall)/; exit(1)' "$f" 2>/dev/null; then
|
||||
debug "$relpath — 命中策略1 (hookRunnerAfter 锚点)"
|
||||
perl -0777 -i -pe 's/(durationMs)\s*\n(\s*\};\s*\n\s*(?:hookRunnerAfter|await\s+\S*hookRunner\S*\.runAfterToolCall|hookRunner\S*\.runAfterToolCall))/$1,\n\t\t\tmessages: ctx.params.session?.messages\n$2/' "$f"
|
||||
if verify_patch "$f"; then
|
||||
ok "[策略1] $relpath — patch 成功"
|
||||
((PATCHED++)) || true
|
||||
applied=true
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── 策略 2: 旧版 dispatch-*.js — durationMs 行末独占 ─────────
|
||||
if [[ "$applied" == "false" ]]; then
|
||||
if echo "$relpath" | grep -qP 'dispatch-.*\.js' 2>/dev/null; then
|
||||
# 匹配行末独占的 durationMs（前面是空白）
|
||||
if grep -qP '^\s+durationMs\s*$' "$f" 2>/dev/null; then
|
||||
debug "$relpath — 命中策略2 (旧版 dispatch)"
|
||||
sed -i -E 's/^(\s+)(durationMs)\s*$/\1\2,\n\1messages: ctx.params.session?.messages/' "$f"
|
||||
if verify_patch "$f"; then
|
||||
ok "[策略2] $relpath — patch 成功"
|
||||
((PATCHED++)) || true
|
||||
applied=true
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── 策略 3: durationMs 后跟 }; 但无 hookRunnerAfter 锚点 ────
|
||||
# 匹配: durationMs<换行><空白>}; （hookEvent 闭合）
|
||||
# 通过上下文（附近有 after_tool_call）确认是正确的对象
|
||||
if [[ "$applied" == "false" ]]; then
|
||||
# 找到包含 durationMs 且附近(±20行)有 after_tool_call 的代码区域
|
||||
if perl -0777 -ne 'exit(0) if /after_tool_call[\s\S]{0,800}durationMs\s*\n(\s*)\};/; exit(1)' "$f" 2>/dev/null; then
|
||||
debug "$relpath — 命中策略3 (durationMs→}; 邻近 after_tool_call)"
|
||||
# 只替换 after_tool_call 上下文附近的 durationMs → };
|
||||
perl -0777 -i -pe 's/(after_tool_call[\s\S]{0,800}durationMs)\s*\n(\s*\};)/$1,\n\t\t\tmessages: ctx.params.session?.messages\n$2/' "$f"
|
||||
if verify_patch "$f"; then
|
||||
ok "[策略3] $relpath — patch 成功"
|
||||
((PATCHED++)) || true
|
||||
applied=true
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── 策略 4: 通用 fallback — hookEvent 赋值中的 durationMs ────
|
||||
# 匹配形如: const hookEvent = { ... durationMs ... }
|
||||
# 或: hookEvent = { ... durationMs ... }
|
||||
# 用 perl 找到包含 after_tool_call 和 durationMs 的对象字面量，在 durationMs 后插入
|
||||
if [[ "$applied" == "false" ]]; then
|
||||
# 极宽松: 在文件中找到 "durationMs" 后面最近的 "}" 或 "};" ，在中间插入
|
||||
# 但限制在 after_tool_call 关键字附近 2000 字符内
|
||||
if perl -0777 -ne 'exit(0) if /after_tool_call[\s\S]{0,2000}?(?:hookEvent|hook_event)[\s\S]{0,500}?durationMs/; exit(1)' "$f" 2>/dev/null; then
|
||||
debug "$relpath — 命中策略4 (通用 fallback)"
|
||||
# 在 durationMs 后追加 (仅首次匹配)
|
||||
perl -0777 -i -pe '
|
||||
my $done = 0;
|
||||
s/(after_tool_call[\s\S]{0,2000}?(?:hookEvent|hook_event)[\s\S]{0,500}?durationMs)\s*\n(\s*)(\};)/
|
||||
if (!$done) { $done = 1; "$1,\n$2\tmessages: ctx.params.session?.messages\n$2$3" }
|
||||
else { "$1\n$2$3" }
|
||||
/ge;
|
||||
' "$f"
|
||||
if verify_patch "$f"; then
|
||||
ok "[策略4] $relpath — patch 成功"
|
||||
((PATCHED++)) || true
|
||||
applied=true
|
||||
else
|
||||
warn "[策略4] $relpath — patch 后验证失败，恢复备份"
|
||||
cp "${f}.pre-offload-patch.bak" "$f"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── 无策略命中 ───────────────────────────────────────────────
|
||||
if [[ "$applied" == "false" ]]; then
|
||||
debug "$relpath — 无策略命中"
|
||||
((FAILED++)) || true
|
||||
fi
|
||||
done
|
||||
|
||||
# ─── 结果报告 ────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo -e "${GREEN}═══════════════════════════════════════════════════════${NC}"
|
||||
echo -e "${GREEN} Patch 完成 (OpenClaw $VERSION)${NC}"
|
||||
echo -e "${GREEN}═══════════════════════════════════════════════════════${NC}"
|
||||
echo ""
|
||||
echo -e " 成功: ${GREEN}${PATCHED}${NC} 跳过: ${YELLOW}${SKIPPED}${NC} 失败: ${RED}${FAILED}${NC}"
|
||||
echo ""
|
||||
if [[ $PATCHED -gt 0 ]]; then
|
||||
echo -e " ${CYAN}重启 OpenClaw 后生效。${NC}"
|
||||
echo -e " ${CYAN}备份文件: *.pre-offload-patch.bak${NC}"
|
||||
elif [[ $SKIPPED -gt 0 && $FAILED -eq 0 ]]; then
|
||||
echo -e " ${YELLOW}所有目标文件已 patch，无需重复操作。${NC}"
|
||||
elif [[ $FAILED -gt 0 ]]; then
|
||||
echo -e " ${RED}部分文件未能 patch。可能需要手动检查或更新 patch 脚本。${NC}"
|
||||
echo -e " ${RED}提示：设置 DEBUG=1 运行以查看详细匹配过程：${NC}"
|
||||
echo -e " ${RED} DEBUG=1 bash $0 $OPENCLAW_DIR${NC}"
|
||||
else
|
||||
echo -e " ${RED}未找到匹配的目标文件，请确认 OpenClaw 版本。${NC}"
|
||||
echo -e " ${RED}提示：设置 DEBUG=1 运行以查看详细匹配过程：${NC}"
|
||||
echo -e " ${RED} DEBUG=1 bash $0 $OPENCLAW_DIR${NC}"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# ─── 退出码 ───────────────────────────────────────────────────────
|
||||
# 0: 成功（至少有一个文件 patch 成功或已跳过）
|
||||
# 1: 失败（无任何 patch 成功且无已跳过的文件）
|
||||
if [[ $PATCHED -gt 0 || $SKIPPED -gt 0 ]]; then
|
||||
exit 0
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "Node16",
|
||||
"moduleResolution": "Node16",
|
||||
"outDir": "./dist",
|
||||
"rootDir": ".",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true,
|
||||
"types": ["node"],
|
||||
"declaration": false,
|
||||
"sourceMap": false
|
||||
},
|
||||
"include": ["read-local-memory.ts"],
|
||||
"exclude": ["dist", "node_modules"]
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
# seed-v2
|
||||
|
||||
> 通过 v2 API 把历史对话灌入 memory-tencentdb gateway。
|
||||
|
||||
## 是什么
|
||||
|
||||
`seed-v2` 是一个**纯 HTTP 客户端**，把一批历史对话喂给已经在跑的 gateway：
|
||||
|
||||
```
|
||||
fixture.json
|
||||
│
|
||||
│ HTTP POST /v2/conversation/add (一轮一次)
|
||||
▼
|
||||
gateway (standalone or service)
|
||||
│
|
||||
│ 自动调度 L0 → L1 → L2 → L3
|
||||
▼
|
||||
本地 vectors.db / records / scene_blocks / persona.md
|
||||
```
|
||||
|
||||
客户端**不 import 任何 plugin 内部模块**，只用 fetch + `/v2/conversation/add` + `/v2/pipeline/status`。
|
||||
|
||||
跟老版 `src/cli/commands/seed.ts`（v1，import 核心 runtime 直跑）相比：
|
||||
- ✅ 同一份代码同时适用于 standalone 和 service 模式
|
||||
- ✅ 不会和已经在跑的 gateway 抢同一份 SQLite / state
|
||||
- ✅ 通过 status 接口节流，串行写入 → 等抽取完 → 再写下一批
|
||||
- ✅ 调度状态由 gateway 侧 `StatefulPipelineManager` 统一管，跟生产路径完全一致
|
||||
|
||||
## 调用方式
|
||||
|
||||
### 方式 1：通过 npm bin（推荐）
|
||||
|
||||
```bash
|
||||
# 1. 把 gateway 起起来（standalone 用法）
|
||||
cd extensions/memory-tencentdb/__tests__/standalone && ./start.sh
|
||||
|
||||
# 2. 灌数据
|
||||
cd ../..
|
||||
npm run seed-v2 -- --input ./scripts/seed-v2/fixtures/minimal.json
|
||||
|
||||
# 3. 完事停 gateway
|
||||
./__tests__/standalone/stop.sh
|
||||
```
|
||||
|
||||
### 方式 2：直接命令
|
||||
|
||||
```bash
|
||||
node ./bin/seed-v2.mjs --input fixture.json --endpoint http://127.0.0.1:18420
|
||||
```
|
||||
|
||||
### 方式 3：一键 wrapper（含起停 + 落盘断言）
|
||||
|
||||
```bash
|
||||
./scripts/seed-v2/seed-v2.sh # 默认 fixture
|
||||
./scripts/seed-v2/seed-v2.sh path/to/my-fixture.json # 自定 fixture
|
||||
./scripts/seed-v2/seed-v2.sh --keep # 跑完不停 gateway
|
||||
./scripts/seed-v2/seed-v2.sh --no-start # gateway 已在跑
|
||||
```
|
||||
|
||||
## CLI 参数
|
||||
|
||||
| 参数 | 短 | env | 默认 | 说明 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `--input <file>` | `-i` | — | **必填** | fixture JSON 路径 |
|
||||
| `--endpoint <url>` | `-e` | `SEED_ENDPOINT` | `http://127.0.0.1:18420` | gateway 地址 |
|
||||
| `--api-key <key>` | — | `SEED_API_KEY` | `standalone-e2e` | Bearer key |
|
||||
| `--service-id <id>` | `-s` | `SEED_SERVICE_ID` | `default` | `x-tdai-service-id` |
|
||||
| `--every-n <n>` | `-n` | `SEED_EVERY_N` | 5 | 每 N 轮等一次 L1 idle |
|
||||
| `--poll-ms <ms>` | — | `SEED_POLL_MS` | 500 | status 轮询间隔 |
|
||||
| `--stable-rounds <n>` | — | `SEED_STABLE_ROUNDS` | 2 | 连续 N 次 idle 才认为 stable |
|
||||
| `--max-wait-ms <ms>` | — | `SEED_MAX_WAIT_MS` | 600000 | 每批 wait（L1 only）超时（默认 10 分钟）|
|
||||
| `--final-max-wait-ms <ms>` | — | `SEED_FINAL_MAX_WAIT_MS` | 600000 | 最终 wait（L1+L2+L3 全 idle）超时（默认 10 分钟）|
|
||||
| `--no-final-wait` | — | — | false | 写完最后一批不等 cascade |
|
||||
| `--dry-run` | — | — | false | 只打印计划不实际请求 |
|
||||
| `--session-key <key>` | — | `SEED_FALLBACK_SESSION_KEY` | — | fallback 给缺 sessionKey 的 session |
|
||||
| `--strict-round-role` | — | `SEED_STRICT_ROUND_ROLE=1` | false | 每 round 必须含 user+assistant |
|
||||
| `--no-auto-fill-timestamps` | — | `SEED_AUTO_FILL_TIMESTAMPS=0` | false | 缺 ts 时不自动 fill |
|
||||
| `--quiet` | `-q` | `SEED_VERBOSE=0` | false | 静默 |
|
||||
| `--help` | `-h` | — | — | 帮助 |
|
||||
|
||||
## Fixture 校验（与 v1 seed 完全对齐）
|
||||
|
||||
启动时会做 6 层校验（任一失败 → exit 2，不会浪费时间灌一半才挂）：
|
||||
|
||||
| 层 | 检查 |
|
||||
| --- | --- |
|
||||
| 1. file | 存在 + 非空 + 合法 JSON |
|
||||
| 2. top-level | Format A `{ sessions: [...] }` 或 Format B `[...]` |
|
||||
| 3. session | sessionKey 必须非空字符串 + conversations 必须 2D array |
|
||||
| 4. round | 必须非空 array；`--strict-round-role` 时还要求 user + assistant 同时存在 |
|
||||
| 5. message | role/content 非空字符串 + timestamp 类型校验（int 或 ISO string）|
|
||||
| 6. timestamp consistency | **全有 / 全无 / mixed** 三态：mixed 拒绝；全无时按 `--no-auto-fill-timestamps` 决定行为 |
|
||||
|
||||
错误输出示例（mixed timestamps）：
|
||||
|
||||
```
|
||||
$ seed-v2 --input mixed.json
|
||||
[seed-v2] Seed input validation failed (1 error(s)):
|
||||
[timestamp_consistency] Timestamp consistency check failed: some messages have timestamps while others do not. ...
|
||||
exit code: 2
|
||||
```
|
||||
|
||||
## Timestamp 三态
|
||||
|
||||
| 状态 | 行为 |
|
||||
| --- | --- |
|
||||
| **全有** | 保留原值，`hasTimestamps=true` |
|
||||
| **全无 + auto-fill 开**（默认）| `fillTimestamps()` 给全局递增 epoch ms（`Date.now()` 起，每条 +100ms）—— 跨 session 也单调，避免 L0 capture cursor 误过滤 |
|
||||
| **全无 + `--no-auto-fill-timestamps`** | timestamp=0；写入时 client 用 `Date.now()` 兜底 |
|
||||
| **mixed** | exit 2，明确报错 |
|
||||
|
||||
## Fixture 格式
|
||||
|
||||
兼容老 seed Format A（`{ sessions: [...] }`）和 Format B（顶层数组）：
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"sessions": [
|
||||
{
|
||||
"sessionKey": "user-001",
|
||||
"sessionId": "user-001", // 可选，缺省 = sessionKey
|
||||
"conversations": [ // rounds
|
||||
[ // round 0：一组消息
|
||||
{ "role": "user", "content": "..." },
|
||||
{ "role": "assistant", "content": "..." }
|
||||
],
|
||||
[ /* round 1 ... */ ]
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
参考最小示例：[`fixtures/minimal.json`](./fixtures/minimal.json)（2 session × 6 round = 24 messages）。
|
||||
|
||||
## 阻塞节奏（重要）
|
||||
|
||||
完全对齐老 `src/core/seed/seed-runtime.ts:executeSeed`：
|
||||
|
||||
```
|
||||
for session, round:
|
||||
POST /v2/conversation/add { session_id, messages } # 立刻返回
|
||||
if (round + 1) % every_n == 0:
|
||||
waitForL1Idle (stable 2 polls) # 只等 L1，L2/L3 后台异步跑
|
||||
end-of-session: waitForL1Idle # 末尾再等一次
|
||||
final: waitForL1Idle # 写完所有 round 后再等 L1 drain
|
||||
```
|
||||
|
||||
`/v2/pipeline/status` 按 L 分桶返回 `{ l1: {idle,…}, l2: {…}, l3: {…} }`。seed-v2 **只看 `data.l1.idle`**：
|
||||
|
||||
- `l1.idle = (l1.queued === 0 && l1.running === 0)` — 跟老 v1 `seed-runtime.waitForL1Idle` 完全对齐
|
||||
- L2/L3 抽取慢（cascade 一次 LLM 工具链可能 3-5 分钟）→ **不会**阻塞 seed 的下一批 dispatch
|
||||
- L2 LLM timeout / lock-conflict drop 等异常情况 → **不会**让 `l1.idle` 误判 false（之前用单标量 `busy` 时会被这两个 bug 污染）
|
||||
|
||||
详见 [`docs/plans/server/14-pipeline-status-api.md`](../../docs/plans/server/14-pipeline-status-api.md)。
|
||||
|
||||
## Pipeline 调参（服务端 yaml）
|
||||
|
||||
灌入速度由**两侧**共同决定：seed 客户端的 wait 节奏 + gateway 流水线的触发节奏。客户端参数已在上面 [CLI 参数](#cli-参数) 表里，下面是**服务端**对应的 `tdai-gateway.standalone.yaml` 配置项（标星号的最常调）：
|
||||
|
||||
```yaml
|
||||
memory:
|
||||
capture:
|
||||
enabled: true # L0 capture 开关，必须 true 否则没数据进流水线
|
||||
|
||||
extraction:
|
||||
enabled: true
|
||||
enableDedup: true # L1 抽取后去重（vector recall topK=5）；关掉提速但可能产生重复 record
|
||||
maxMemoriesPerSession: 20 # ★ 单 session L1 抽取的 atomic memory 数上限
|
||||
|
||||
persona:
|
||||
triggerEveryN: 50 # ★ 每抽取 N 条 L1 record 触发 1 次 L3 persona-gen
|
||||
maxScenes: 15 # L2 scene 数上限（达到后停止 CREATE 新 scene）
|
||||
|
||||
pipeline:
|
||||
everyNConversations: 5 # ★★★ 每 N 轮对话触发 1 次 L1 抽取（必须跟 seed 客户端 --every-n 对齐）
|
||||
enableWarmup: true # 冷启动前几次更快触发 L1（让 persona.md 早点形成骨架）
|
||||
l1IdleTimeoutSeconds: 600 # ★ session 空闲 N 秒后强制触发 L1（兜底，防 conversation < everyN 时 L1 永不抽）
|
||||
l2DelayAfterL1Seconds: 90 # ★★ L1 完成后延迟 N 秒触发 L2（避免每条 L1 都触发一次 L2，太碎）
|
||||
l2MinIntervalSeconds: 900 # ★★ 同 session 两次 L2 之间最小间隔（节流，防 L2 LLM 调用过频）
|
||||
l2MaxIntervalSeconds: 3600 # ★ 同 session L2 最大间隔（兜底，无论间隔多久必跑一次）
|
||||
```
|
||||
|
||||
### 怎么调：常见场景
|
||||
|
||||
| 场景 | 怎么调 |
|
||||
| --- | --- |
|
||||
| **seed 灌入想最快**（接受 L1 粒度粗）| `everyNConversations: 20`（每 20 轮才抽一次 L1）+ seed `--every-n 20` 对齐 |
|
||||
| **seed 灌入想 L1 立即抽**（每轮都抽，最细粒度但贵）| `everyNConversations: 1` + seed `--every-n 1` |
|
||||
| **fixture 是短对话**（< everyN 轮）触发不到 L1 | 缩 `l1IdleTimeoutSeconds: 60`（1分钟无新消息就触发兜底）|
|
||||
| **L2 cascade 太频繁**（cost 高）| 拉大 `l2MinIntervalSeconds: 1800`（30 分钟才一次）|
|
||||
| **L2 cascade 太稀疏** | 缩 `l2DelayAfterL1Seconds: 30` + `l2MinIntervalSeconds: 300` |
|
||||
| **测试/调试想看快速反馈** | `triggerEveryN: 10` + `everyNConversations: 2` + `l1IdleTimeoutSeconds: 30` |
|
||||
|
||||
### seed 客户端 vs 服务端的对应
|
||||
|
||||
| seed 参数 | 服务端 yaml 对应 | 是否需对齐 |
|
||||
| --- | --- | --- |
|
||||
| `--every-n` (`SEED_EVERY_N`) | `pipeline.everyNConversations` | ⚠️ **建议一致**（客户端 wait 间隔 = 服务端触发间隔，灌入流畅）|
|
||||
| `--max-wait-ms` | `pipeline.l1IdleTimeoutSeconds` | wait 时长应 ≥ idle timeout（否则会 max-wait-reach 而不是 stable-idle）|
|
||||
| `--final-max-wait-ms` | 综合 L2 慢 task 时长 | ≥ p95(L2 LLM duration) × maxScenes/(同时抽的 scene 数)（默认 10 分钟够大多数场景）|
|
||||
| `--stable-rounds` | — | 客户端独有，控制连续多少次 idle 才认定真稳定 |
|
||||
|
||||
### 调参对 seed 总耗时的影响（粗估）
|
||||
|
||||
| 配置 | persona-0（10 sessions × 366 msgs）耗时 |
|
||||
| --- | --- |
|
||||
| 默认（`everyNConversations=5`，`l2DelayAfterL1=90s`，`l2MinInterval=900s`）| ~90 min（实测 persona-0）|
|
||||
| 降本：`everyNConversations=20`，`l2MinInterval=3600s` | ~30-50 min（L1 抽得粗，L2 抽得少）|
|
||||
| 极速：`--no-final-wait` + 仅 L0 模式（`extraction.enabled=false`）| ~1-2 min（只灌 L0，不抽取）|
|
||||
|
||||
> ⚠️ **改 yaml 后必须重启 gateway**（`scripts/seed-v2/seed-v2.sh` 自动起停；如果用 `start.sh` 手动起的需 `stop.sh && start.sh`）。
|
||||
|
||||
## 退出码
|
||||
|
||||
| code | 含义 |
|
||||
| --- | --- |
|
||||
| 0 | 全部成功 |
|
||||
| 1 | seed 失败（写入或 wait 异常）|
|
||||
| 2 | 前置条件不满足（fixture 缺失 / gateway 不通 / 配置错误）|
|
||||
|
||||
## 构建 + 发布
|
||||
|
||||
```bash
|
||||
npm run build:seed-v2 # tsc 编译到 scripts/seed-v2/dist/
|
||||
npm run build:scripts # 构建所有 scripts（含 seed-v2）
|
||||
```
|
||||
|
||||
`bin/seed-v2.mjs` 是薄启动器：dist 存在就跑 dist，否则 fallback 到 `tsx` 跑源码。开发期不需要先 build。
|
||||
@@ -0,0 +1,64 @@
|
||||
{
|
||||
"sessions": [
|
||||
{
|
||||
"sessionKey": "seed-alice",
|
||||
"sessionId": "seed-alice",
|
||||
"conversations": [
|
||||
[
|
||||
{ "role": "user", "content": "我叫 Alice，最近在学 Rust 语言，主要在做嵌入式开发。" },
|
||||
{ "role": "assistant", "content": "了解，Rust 在嵌入式领域优势明显。你目前用的是哪块板子？" }
|
||||
],
|
||||
[
|
||||
{ "role": "user", "content": "我用的是 STM32H7 系列。我特别喜欢 Rust 的 ownership 模型，感觉对资源管理很有帮助。" },
|
||||
{ "role": "assistant", "content": "STM32H7 性能不错。Rust 的所有权模型确实能在编译期就排除很多资源泄漏问题。" }
|
||||
],
|
||||
[
|
||||
{ "role": "user", "content": "我打算做一个低功耗的传感器节点，用 LoRa 通信。" },
|
||||
{ "role": "assistant", "content": "LoRa 适合低功耗场景。你考虑用 RTIC 还是 embassy 这种异步框架吗？" }
|
||||
],
|
||||
[
|
||||
{ "role": "user", "content": "我倾向于 embassy，async/await 写起来更顺。" },
|
||||
{ "role": "assistant", "content": "embassy 在嵌入式 async 场景里生态成熟，executor 也很轻量。" }
|
||||
],
|
||||
[
|
||||
{ "role": "user", "content": "我每天大概能写 4 小时代码，主要在晚上 8 点到 12 点。" },
|
||||
{ "role": "assistant", "content": "晚上是连续编码的好时段。建议早上 review 一下前一天的代码再开始新模块。" }
|
||||
],
|
||||
[
|
||||
{ "role": "user", "content": "对了，我之前用的语言是 C 和 Python，从 C 转 Rust 一开始很难适应借用检查器。" },
|
||||
{ "role": "assistant", "content": "从 C 转 Rust 确实需要思维转换，多用 cargo clippy 帮你养成习惯。" }
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"sessionKey": "seed-bob",
|
||||
"sessionId": "seed-bob",
|
||||
"conversations": [
|
||||
[
|
||||
{ "role": "user", "content": "Hi，我是 Bob，做数据分析的，平时主要用 Python 和 SQL。" },
|
||||
{ "role": "assistant", "content": "Python + SQL 是数据分析的黄金组合。你做哪类业务的数据？" }
|
||||
],
|
||||
[
|
||||
{ "role": "user", "content": "电商，主要看用户行为漏斗和复购率。最近在搞 RFM 模型。" },
|
||||
{ "role": "assistant", "content": "RFM 是经典模型。你打算用它做用户分群再做精准营销吗？" }
|
||||
],
|
||||
[
|
||||
{ "role": "user", "content": "对，分群之后想接 A/B 测试，看不同分群对优惠券敏感度。" },
|
||||
{ "role": "assistant", "content": "好思路。建议把 RFM 评分时间窗对齐 A/B 实验周期，避免季节性偏差。" }
|
||||
],
|
||||
[
|
||||
{ "role": "user", "content": "我团队有 3 个人，一个 BI、一个数据科学家，加我自己。" },
|
||||
{ "role": "assistant", "content": "团队结构合理。BI 负责报表，数据科学家做模型，你协调和落地。" }
|
||||
],
|
||||
[
|
||||
{ "role": "user", "content": "我们用 ClickHouse 存日志，每天大概 5 亿条记录。" },
|
||||
{ "role": "assistant", "content": "ClickHouse 处理这种规模没压力。注意 partition 设计和 MergeTree 引擎选型。" }
|
||||
],
|
||||
[
|
||||
{ "role": "user", "content": "对了，我下个月要去新加坡出差一周参加 data conf。" },
|
||||
{ "role": "assistant", "content": "祝出差顺利！data conf 是个不错的拓展视野的场合。" }
|
||||
]
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"sessions": [
|
||||
{
|
||||
"sessionKey": "mixed-bob",
|
||||
"conversations": [
|
||||
[
|
||||
{ "role": "user", "content": "测试 mixed timestamp 是否被检测", "timestamp": 1700000000000 },
|
||||
{ "role": "assistant", "content": "这一条故意没有 timestamp" }
|
||||
]
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"sessions": [
|
||||
{
|
||||
"sessionKey": "no-ts-alice",
|
||||
"conversations": [
|
||||
[
|
||||
{ "role": "user", "content": "我叫 Alice，最近在学 Python 数据分析。" },
|
||||
{ "role": "assistant", "content": "Python 数据分析通常涉及 pandas、numpy 这些库。你目前学到哪一步？" }
|
||||
],
|
||||
[
|
||||
{ "role": "user", "content": "我刚学完 pandas 基础，现在在做股票数据的处理。" },
|
||||
{ "role": "assistant", "content": "股票数据典型的就是时间序列分析，建议熟悉一下 datetime 索引。" }
|
||||
]
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
/**
|
||||
* Type definitions for seed-v2 fixture input.
|
||||
*
|
||||
* ⚠️ 这份文件是从 `src/core/seed/types.ts` 拷来的精简版（去掉了 v1 CLI 专属的
|
||||
* `SeedCommandOptions` / `SeedProgress` / `SeedSummary`）。
|
||||
*
|
||||
* 老 v1 seed (`src/cli/commands/seed.ts` + `/seed` v1 endpoint) 计划废弃后，
|
||||
* 本文件成为唯一真理之源。在那之前，如果在 `src/core/seed/types.ts` 改了
|
||||
* Raw* / Normalized* / ValidationError，请同步到这里。
|
||||
*/
|
||||
|
||||
// ============================
|
||||
// Raw input types (before validation)
|
||||
// ============================
|
||||
|
||||
export interface RawMessage {
|
||||
role: string;
|
||||
content: string;
|
||||
/** Epoch ms (number) or ISO 8601 string. */
|
||||
timestamp?: number | string;
|
||||
}
|
||||
|
||||
export interface RawSession {
|
||||
sessionKey: string;
|
||||
sessionId?: string;
|
||||
conversations: RawMessage[][];
|
||||
}
|
||||
|
||||
/** Format A: `{ sessions: [...] }` */
|
||||
export interface FormatA {
|
||||
sessions: RawSession[];
|
||||
}
|
||||
|
||||
/** Format B: `[...]` (top-level array of sessions) */
|
||||
export type FormatB = RawSession[];
|
||||
|
||||
// ============================
|
||||
// Normalized types (after validation)
|
||||
// ============================
|
||||
|
||||
export interface NormalizedMessage {
|
||||
role: string;
|
||||
content: string;
|
||||
/** Epoch ms — always present after normalization. 0 means "to be filled later". */
|
||||
timestamp: number;
|
||||
}
|
||||
|
||||
export interface NormalizedRound {
|
||||
messages: NormalizedMessage[];
|
||||
}
|
||||
|
||||
export interface NormalizedSession {
|
||||
sessionKey: string;
|
||||
sessionId: string;
|
||||
rounds: NormalizedRound[];
|
||||
sourceIndex: number;
|
||||
}
|
||||
|
||||
export interface NormalizedInput {
|
||||
sessions: NormalizedSession[];
|
||||
totalRounds: number;
|
||||
totalMessages: number;
|
||||
hasTimestamps: boolean;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Validation
|
||||
// ============================
|
||||
|
||||
export type ValidationStage =
|
||||
| "file"
|
||||
| "top_level"
|
||||
| "session"
|
||||
| "round"
|
||||
| "message"
|
||||
| "timestamp_consistency";
|
||||
|
||||
export interface ValidationError {
|
||||
stage: ValidationStage;
|
||||
sourceIndex?: number;
|
||||
sessionKey?: string;
|
||||
roundIndex?: number;
|
||||
messageIndex?: number;
|
||||
message: string;
|
||||
}
|
||||
@@ -0,0 +1,408 @@
|
||||
/**
|
||||
* Input loading, validation, normalization, and timestamp handling for seed-v2.
|
||||
*
|
||||
* ⚠️ 这份文件是从 `src/core/seed/input.ts` 拷来的精简版（去掉了 v1 CLI 专属的
|
||||
* `loadAndValidateInput`，因为它依赖 `SeedCommandOptions` 类型且会做交互式
|
||||
* timestamp 确认 —— seed-v2 是 HTTP 工具，不交互）。
|
||||
*
|
||||
* 老 v1 seed (`src/cli/commands/seed.ts` + `/seed` v1 endpoint) 计划废弃后，
|
||||
* 本文件成为唯一真理之源。在那之前，如果在 `src/core/seed/input.ts` 改了
|
||||
* `validateAndNormalizeRaw` / `fillTimestamps` / `SeedValidationError`，请
|
||||
* 同步到这里。
|
||||
*
|
||||
* 6 层校验 (file → top-level → session → round → message → timestamp consistency)
|
||||
* 中的 file 层在 seed-v2 自己读 fixture 时做（`loadFixtureFile`），不在本文件里。
|
||||
*/
|
||||
|
||||
import crypto from "node:crypto";
|
||||
import type {
|
||||
RawSession,
|
||||
FormatA,
|
||||
ValidationError,
|
||||
NormalizedInput,
|
||||
NormalizedSession,
|
||||
NormalizedRound,
|
||||
NormalizedMessage,
|
||||
} from "./input-types.js";
|
||||
|
||||
// ============================
|
||||
// Public API
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Validate and normalize seed input from an already-parsed JSON object.
|
||||
*
|
||||
* Throws `SeedValidationError` on validation failures.
|
||||
*
|
||||
* Timestamp policy:
|
||||
* - All messages have timestamp → `hasTimestamps=true`, normalized as-is
|
||||
* - All messages missing → if `autoFillTimestamps` (default true), fill with
|
||||
* globally-monotonic Date.now() + 100ms increments; else leave at 0
|
||||
* - Mixed (some have, some missing) → throw error
|
||||
*/
|
||||
export function validateAndNormalizeRaw(
|
||||
raw: unknown,
|
||||
opts?: { sessionKey?: string; strictRoundRole?: boolean; autoFillTimestamps?: boolean },
|
||||
): NormalizedInput {
|
||||
const strictRoundRole = opts?.strictRoundRole ?? false;
|
||||
const autoFillTimestamps = opts?.autoFillTimestamps ?? true;
|
||||
|
||||
// Layer 2: Top-level — detect Format A vs B
|
||||
const sessions = extractSessions(raw);
|
||||
|
||||
// Layers 3-5: session / round / message validation
|
||||
const errors: ValidationError[] = [];
|
||||
validateSessions(sessions, strictRoundRole, errors);
|
||||
|
||||
if (errors.length > 0) {
|
||||
throw new SeedValidationError(errors);
|
||||
}
|
||||
|
||||
// Layer 6: Timestamp consistency (all-have / all-missing / mixed → error)
|
||||
const tsResult = checkTimestampConsistency(sessions);
|
||||
if (tsResult.status === "mixed") {
|
||||
throw new SeedValidationError([{
|
||||
stage: "timestamp_consistency",
|
||||
message:
|
||||
"Timestamp consistency check failed: some messages have timestamps while others do not. " +
|
||||
"All messages must either have timestamps or none must have timestamps.",
|
||||
}]);
|
||||
}
|
||||
|
||||
// Normalize
|
||||
const normalized = normalizeSessions(sessions, opts?.sessionKey);
|
||||
|
||||
const input: NormalizedInput = {
|
||||
sessions: normalized.sessions,
|
||||
totalRounds: normalized.totalRounds,
|
||||
totalMessages: normalized.totalMessages,
|
||||
hasTimestamps: tsResult.status === "all_present",
|
||||
};
|
||||
|
||||
// Auto-fill timestamps when none present (HTTP tool, no interactive prompt)
|
||||
if (tsResult.status === "all_missing" && autoFillTimestamps) {
|
||||
fillTimestamps(input);
|
||||
}
|
||||
|
||||
return input;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fill timestamps for all messages when the input has no timestamps.
|
||||
*
|
||||
* Uses a single monotonically increasing counter across ALL sessions to
|
||||
* guarantee global timestamp ordering — critical when multiple sessions
|
||||
* share the same sessionKey, since L0 capture cursor is advanced per-session
|
||||
* and would filter out later sessions whose timestamps fall below the cursor
|
||||
* if ordering were not globally monotonic.
|
||||
*/
|
||||
export function fillTimestamps(input: NormalizedInput): void {
|
||||
let currentTs = Date.now();
|
||||
for (const session of input.sessions) {
|
||||
for (const round of session.rounds) {
|
||||
for (let i = 0; i < round.messages.length; i++) {
|
||||
round.messages[i]!.timestamp = currentTs;
|
||||
currentTs += 100; // 100ms gap per message — matches v1 fillTimestamps
|
||||
}
|
||||
}
|
||||
}
|
||||
input.hasTimestamps = true;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Validation error class
|
||||
// ============================
|
||||
|
||||
export class SeedValidationError extends Error {
|
||||
public readonly errors: ValidationError[];
|
||||
|
||||
constructor(errors: ValidationError[]) {
|
||||
const summary = errors.map((e) => formatValidationError(e)).join("\n");
|
||||
super(`Seed input validation failed (${errors.length} error(s)):\n${summary}`);
|
||||
this.name = "SeedValidationError";
|
||||
this.errors = errors;
|
||||
}
|
||||
}
|
||||
|
||||
function formatValidationError(e: ValidationError): string {
|
||||
const parts: string[] = [` [${e.stage}]`];
|
||||
if (e.sourceIndex != null) parts.push(`session[${e.sourceIndex}]`);
|
||||
if (e.sessionKey) parts.push(`key="${e.sessionKey}"`);
|
||||
if (e.roundIndex != null) parts.push(`round[${e.roundIndex}]`);
|
||||
if (e.messageIndex != null) parts.push(`msg[${e.messageIndex}]`);
|
||||
parts.push(e.message);
|
||||
return parts.join(" ");
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Layer 2: Top-level format detection
|
||||
// ============================
|
||||
|
||||
function extractSessions(raw: unknown): RawSession[] {
|
||||
// Format A: { sessions: [...] }
|
||||
if (
|
||||
raw != null &&
|
||||
typeof raw === "object" &&
|
||||
!Array.isArray(raw) &&
|
||||
"sessions" in raw
|
||||
) {
|
||||
const obj = raw as FormatA;
|
||||
if (!Array.isArray(obj.sessions)) {
|
||||
throw new SeedValidationError([{
|
||||
stage: "top_level",
|
||||
message: 'Format A detected but "sessions" is not an array.',
|
||||
}]);
|
||||
}
|
||||
return obj.sessions;
|
||||
}
|
||||
|
||||
// Format B: [...]
|
||||
if (Array.isArray(raw)) {
|
||||
return raw as RawSession[];
|
||||
}
|
||||
|
||||
throw new SeedValidationError([{
|
||||
stage: "top_level",
|
||||
message:
|
||||
"Unrecognized input format. Expected either:\n" +
|
||||
' Format A: { "sessions": [...] }\n' +
|
||||
" Format B: [ { sessionKey, conversations }, ... ]",
|
||||
}]);
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Layers 3-5: session / round / message validation
|
||||
// ============================
|
||||
|
||||
function validateSessions(
|
||||
sessions: RawSession[],
|
||||
strictRoundRole: boolean,
|
||||
errors: ValidationError[],
|
||||
): void {
|
||||
if (sessions.length === 0) {
|
||||
errors.push({
|
||||
stage: "session",
|
||||
message: "No sessions found in input.",
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
for (let si = 0; si < sessions.length; si++) {
|
||||
const session = sessions[si]!;
|
||||
|
||||
// Layer 3: session validation
|
||||
if (!session.sessionKey || typeof session.sessionKey !== "string" || session.sessionKey.trim() === "") {
|
||||
errors.push({
|
||||
stage: "session",
|
||||
sourceIndex: si,
|
||||
message: '"sessionKey" is required and must be a non-empty string.',
|
||||
});
|
||||
}
|
||||
|
||||
if (!Array.isArray(session.conversations)) {
|
||||
errors.push({
|
||||
stage: "session",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
message: '"conversations" must be a two-dimensional array (array of rounds).',
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
for (let ri = 0; ri < session.conversations.length; ri++) {
|
||||
const round = session.conversations[ri];
|
||||
|
||||
// Layer 4: round validation
|
||||
if (!Array.isArray(round)) {
|
||||
errors.push({
|
||||
stage: "round",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
message: "Round must be an array of messages.",
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
if (round.length === 0) {
|
||||
errors.push({
|
||||
stage: "round",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
message: "Round must be a non-empty array.",
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
// Strict round-role: each round must have at least one user and one assistant
|
||||
if (strictRoundRole) {
|
||||
const roles = new Set(round.map((m) => m.role));
|
||||
if (!roles.has("user")) {
|
||||
errors.push({
|
||||
stage: "round",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
message: '--strict-round-role: round must contain at least one "user" message.',
|
||||
});
|
||||
}
|
||||
if (!roles.has("assistant")) {
|
||||
errors.push({
|
||||
stage: "round",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
message: '--strict-round-role: round must contain at least one "assistant" message.',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Layer 5: message validation
|
||||
for (let mi = 0; mi < round.length; mi++) {
|
||||
const msg = round[mi]!;
|
||||
|
||||
if (!msg.role || typeof msg.role !== "string") {
|
||||
errors.push({
|
||||
stage: "message",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
messageIndex: mi,
|
||||
message: '"role" is required and must be a non-empty string.',
|
||||
});
|
||||
}
|
||||
|
||||
if (!msg.content || typeof msg.content !== "string" || msg.content.trim() === "") {
|
||||
errors.push({
|
||||
stage: "message",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
messageIndex: mi,
|
||||
message: '"content" is required and must be a non-empty string.',
|
||||
});
|
||||
}
|
||||
|
||||
if (msg.timestamp !== undefined) {
|
||||
if (typeof msg.timestamp === "number") {
|
||||
if (!Number.isInteger(msg.timestamp)) {
|
||||
errors.push({
|
||||
stage: "message",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
messageIndex: mi,
|
||||
message: '"timestamp" must be an integer (epoch milliseconds). Negative values allowed for pre-1970 dates.',
|
||||
});
|
||||
}
|
||||
} else if (typeof msg.timestamp === "string") {
|
||||
if (Number.isNaN(new Date(msg.timestamp).getTime())) {
|
||||
errors.push({
|
||||
stage: "message",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
messageIndex: mi,
|
||||
message: `"timestamp" string is not a valid ISO 8601 date: "${msg.timestamp}".`,
|
||||
});
|
||||
}
|
||||
} else {
|
||||
errors.push({
|
||||
stage: "message",
|
||||
sourceIndex: si,
|
||||
sessionKey: session.sessionKey,
|
||||
roundIndex: ri,
|
||||
messageIndex: mi,
|
||||
message: '"timestamp" must be a number (epoch ms) or an ISO 8601 string.',
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Layer 6: Timestamp consistency
|
||||
// ============================
|
||||
|
||||
interface TimestampCheckResult {
|
||||
status: "all_present" | "all_missing" | "mixed";
|
||||
}
|
||||
|
||||
function checkTimestampConsistency(sessions: RawSession[]): TimestampCheckResult {
|
||||
let hasTs = false;
|
||||
let missingTs = false;
|
||||
|
||||
for (const session of sessions) {
|
||||
if (!Array.isArray(session.conversations)) continue;
|
||||
for (const round of session.conversations) {
|
||||
if (!Array.isArray(round)) continue;
|
||||
for (const msg of round) {
|
||||
if (msg.timestamp !== undefined && msg.timestamp !== null) {
|
||||
hasTs = true;
|
||||
} else {
|
||||
missingTs = true;
|
||||
}
|
||||
if (hasTs && missingTs) {
|
||||
return { status: "mixed" };
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (hasTs && !missingTs) return { status: "all_present" };
|
||||
if (!hasTs && missingTs) return { status: "all_missing" };
|
||||
// No messages at all — treat as all_missing (will be caught by session validation)
|
||||
return { status: "all_missing" };
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Normalization
|
||||
// ============================
|
||||
|
||||
function normalizeSessions(
|
||||
sessions: RawSession[],
|
||||
fallbackSessionKey?: string,
|
||||
): { sessions: NormalizedSession[]; totalRounds: number; totalMessages: number } {
|
||||
const normalized: NormalizedSession[] = [];
|
||||
let totalRounds = 0;
|
||||
let totalMessages = 0;
|
||||
|
||||
for (let si = 0; si < sessions.length; si++) {
|
||||
const raw = sessions[si]!;
|
||||
|
||||
const sessionKey = raw.sessionKey || fallbackSessionKey || "seed-user";
|
||||
const sessionId = raw.sessionId || crypto.randomUUID();
|
||||
|
||||
const rounds: NormalizedRound[] = [];
|
||||
for (const rawRound of raw.conversations) {
|
||||
if (!Array.isArray(rawRound)) continue;
|
||||
|
||||
const messages: NormalizedMessage[] = rawRound.map((msg) => ({
|
||||
role: msg.role,
|
||||
content: msg.content,
|
||||
// Normalize timestamp: ISO string → epoch ms; number → pass-through; missing → 0 (filled later)
|
||||
timestamp: msg.timestamp == null
|
||||
? 0
|
||||
: typeof msg.timestamp === "string"
|
||||
? new Date(msg.timestamp).getTime()
|
||||
: msg.timestamp,
|
||||
}));
|
||||
|
||||
rounds.push({ messages });
|
||||
totalMessages += messages.length;
|
||||
}
|
||||
|
||||
totalRounds += rounds.length;
|
||||
normalized.push({
|
||||
sessionKey,
|
||||
sessionId,
|
||||
rounds,
|
||||
sourceIndex: si,
|
||||
});
|
||||
}
|
||||
|
||||
return { sessions: normalized, totalRounds, totalMessages };
|
||||
}
|
||||
Executable
+210
@@ -0,0 +1,210 @@
|
||||
#!/usr/bin/env bash
|
||||
# ============================================================
|
||||
# seed-v2.sh — 起 standalone gateway → 跑 seed-v2 → 验落盘 → 停
|
||||
#
|
||||
# 这是给"想完整自动化跑一次"的用户准备的便利 wrapper。
|
||||
# 不想自动起停 gateway 的话，直接 `npm run seed-v2 -- --input <file>` 即可。
|
||||
#
|
||||
# 用法:
|
||||
# ./seed-v2.sh # 默认用 fixtures/minimal.json
|
||||
# ./seed-v2.sh path/to/fixture.json # 用指定 fixture
|
||||
# ./seed-v2.sh --keep # 跑完不停 gateway（调试）
|
||||
# ./seed-v2.sh --no-start # 假设 gateway 已在跑（不 reset、不 start、不 stop）
|
||||
# ./seed-v2.sh --no-reset # 不清空数据目录（增量灌）
|
||||
#
|
||||
# 透传给 seed-v2.ts 的环境变量:
|
||||
# SEED_ENDPOINT / SEED_API_KEY / SEED_SERVICE_ID
|
||||
# SEED_EVERY_N / SEED_POLL_MS / SEED_STABLE_ROUNDS / SEED_MAX_WAIT_MS
|
||||
# SEED_VERBOSE 1=详细 / 0=安静
|
||||
#
|
||||
# 退出码:
|
||||
# 0 成功 + 落盘断言全部通过
|
||||
# 1 seed 或断言失败
|
||||
# 2 前置条件不满足
|
||||
# ============================================================
|
||||
set -euo pipefail
|
||||
|
||||
# Compute our own dir BEFORE sourcing env.sh (which overwrites SCRIPT_DIR).
|
||||
SEED_V2_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PLUGIN_ROOT="$(cd "$SEED_V2_DIR/../.." && pwd)"
|
||||
STANDALONE_HELPERS="$PLUGIN_ROOT/__tests__/standalone"
|
||||
|
||||
# 复用 standalone 测试套件里的 env / start / stop 工具（保持一份真理之源，
|
||||
# 不在这里重复实现 gateway 起停）
|
||||
if [ ! -f "$STANDALONE_HELPERS/env.sh" ]; then
|
||||
echo "✗ standalone helpers not found at: $STANDALONE_HELPERS" >&2
|
||||
echo " this wrapper depends on env.sh / start.sh / stop.sh from there." >&2
|
||||
exit 2
|
||||
fi
|
||||
# shellcheck disable=SC1091
|
||||
source "$STANDALONE_HELPERS/env.sh"
|
||||
|
||||
# ============================================================
|
||||
# 参数
|
||||
# ============================================================
|
||||
KEEP_RUNNING=0
|
||||
NO_START=0
|
||||
DO_RESET=1
|
||||
FIXTURE="$SEED_V2_DIR/fixtures/minimal.json"
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--keep) KEEP_RUNNING=1 ;;
|
||||
--no-start) NO_START=1; KEEP_RUNNING=1; DO_RESET=0 ;;
|
||||
--no-reset) DO_RESET=0 ;;
|
||||
-h|--help)
|
||||
sed -n '2,25p' "$0" | sed 's/^# \?//'
|
||||
exit 0
|
||||
;;
|
||||
-*)
|
||||
log_err "unknown flag: $1"
|
||||
exit 2
|
||||
;;
|
||||
*)
|
||||
FIXTURE="$1"
|
||||
;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
[ -f "$FIXTURE" ] || { log_err "fixture not found: $FIXTURE"; exit 2; }
|
||||
|
||||
# ============================================================
|
||||
# Reset 数据目录（必须在 start.sh 之前）
|
||||
# ============================================================
|
||||
if [ "$DO_RESET" -eq 1 ]; then
|
||||
if echo "$STANDALONE_DATA_DIR" | grep -q "/\.codebuddy/"; then
|
||||
log_info "resetting data dir: $STANDALONE_DATA_DIR"
|
||||
rm -rf "$STANDALONE_DATA_DIR"
|
||||
else
|
||||
log_err "refuse to reset $STANDALONE_DATA_DIR (must be under .codebuddy/)"
|
||||
exit 2
|
||||
fi
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# 起 gateway
|
||||
# ============================================================
|
||||
if [ "$NO_START" -eq 0 ]; then
|
||||
log_info "starting gateway..."
|
||||
if ! "$STANDALONE_HELPERS/start.sh"; then
|
||||
log_err "start.sh failed"
|
||||
exit 2
|
||||
fi
|
||||
fi
|
||||
|
||||
if ! gateway_alive; then
|
||||
log_err "gateway not reachable at $STANDALONE_BASE"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# trap：保证不论结果如何，都按 KEEP_RUNNING 决定要不要停
|
||||
# ============================================================
|
||||
cleanup() {
|
||||
local rc=$?
|
||||
if [ "$KEEP_RUNNING" -eq 0 ]; then
|
||||
log_info "stopping gateway"
|
||||
"$STANDALONE_HELPERS/stop.sh" >/dev/null 2>&1 || true
|
||||
else
|
||||
log_info "gateway kept at $STANDALONE_BASE (use $STANDALONE_HELPERS/stop.sh to stop)"
|
||||
fi
|
||||
exit $rc
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
# ============================================================
|
||||
# 跑 seed-v2（通过 npm bin）
|
||||
# ============================================================
|
||||
log_info "running seed-v2 with fixture: $FIXTURE"
|
||||
echo
|
||||
|
||||
cd "$PLUGIN_ROOT"
|
||||
SEED_ENDPOINT="${SEED_ENDPOINT:-$STANDALONE_BASE}" \
|
||||
SEED_API_KEY="${SEED_API_KEY:-standalone-e2e}" \
|
||||
SEED_SERVICE_ID="${SEED_SERVICE_ID:-default}" \
|
||||
SEED_EVERY_N="${SEED_EVERY_N:-5}" \
|
||||
SEED_POLL_MS="${SEED_POLL_MS:-500}" \
|
||||
SEED_STABLE_ROUNDS="${SEED_STABLE_ROUNDS:-2}" \
|
||||
SEED_MAX_WAIT_MS="${SEED_MAX_WAIT_MS:-600000}" \
|
||||
SEED_VERBOSE="${SEED_VERBOSE:-1}" \
|
||||
node "$PLUGIN_ROOT/bin/seed-v2.mjs" --input "$FIXTURE"
|
||||
|
||||
SEED_RC=$?
|
||||
|
||||
# ============================================================
|
||||
# 落盘断言
|
||||
# ============================================================
|
||||
echo
|
||||
log_info "=== Post-seed assertions ==="
|
||||
|
||||
ASSERT_PASS=0
|
||||
ASSERT_FAIL=0
|
||||
assert() {
|
||||
local name="$1"
|
||||
local cond="$2"
|
||||
if eval "$cond" >/dev/null 2>&1; then
|
||||
log_ok "$name"
|
||||
ASSERT_PASS=$((ASSERT_PASS+1))
|
||||
else
|
||||
log_err "$name"
|
||||
ASSERT_FAIL=$((ASSERT_FAIL+1))
|
||||
fi
|
||||
}
|
||||
|
||||
# 1. L0: JSONL 镜像（standalone 特有）
|
||||
TODAY="$(date +%Y-%m-%d)"
|
||||
JSONL_PATH="$STANDALONE_DATA_DIR/conversations/$TODAY.jsonl"
|
||||
assert "L0: JSONL mirror exists" "[ -f \"$JSONL_PATH\" ]"
|
||||
if [ -f "$JSONL_PATH" ]; then
|
||||
JSONL_LINES=$(wc -l < "$JSONL_PATH")
|
||||
log_info " JSONL line count: $JSONL_LINES"
|
||||
fi
|
||||
|
||||
# 2. L0: SQLite (vectors.db)
|
||||
DB_PATH="$STANDALONE_DATA_DIR/vectors.db"
|
||||
assert "L0: SQLite vectors.db exists" "[ -f \"$DB_PATH\" ]"
|
||||
SQLITE_BIN="$(command -v sqlite3 || true)"
|
||||
if [ -z "$SQLITE_BIN" ] && [ -x "$HOME/miniconda3/bin/sqlite3" ]; then
|
||||
SQLITE_BIN="$HOME/miniconda3/bin/sqlite3"
|
||||
fi
|
||||
if [ -f "$DB_PATH" ] && [ -n "$SQLITE_BIN" ]; then
|
||||
L0_COUNT="$("$SQLITE_BIN" "$DB_PATH" "SELECT COUNT(*) FROM l0_conversations" 2>/dev/null || echo 0)"
|
||||
log_info " SQLite l0_conversations count: $L0_COUNT"
|
||||
assert "L0: SQLite has records" "[ \"$L0_COUNT\" -gt 0 ]"
|
||||
else
|
||||
log_info "L0: sqlite3 binary not found, skipping SQLite assertion"
|
||||
fi
|
||||
|
||||
# 3. L1: records JSONL — 仅当 LLM key 配置时会有产出
|
||||
RECORDS_DIR="$STANDALONE_DATA_DIR/records"
|
||||
if [ -d "$RECORDS_DIR" ]; then
|
||||
RECORD_FILES="$(find "$RECORDS_DIR" -name "*.jsonl" -size +0 2>/dev/null | wc -l)"
|
||||
if [ "$RECORD_FILES" -gt 0 ]; then
|
||||
log_ok "L1: records/ has $RECORD_FILES non-empty jsonl file(s) [LLM key configured]"
|
||||
ASSERT_PASS=$((ASSERT_PASS+1))
|
||||
else
|
||||
log_info "L1: records/ exists but empty (LLM key likely not configured — skipped, not a failure)"
|
||||
fi
|
||||
else
|
||||
log_info "L1: records/ not created (LLM key likely not configured — skipped, not a failure)"
|
||||
fi
|
||||
|
||||
# 4. Pipeline status: busy=false at the end
|
||||
STATUS_RESP="$(curl -sS -X POST "$STANDALONE_BASE/v2/pipeline/status" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'Authorization: Bearer standalone-e2e' \
|
||||
-H 'x-tdai-service-id: default' \
|
||||
-d '{}')"
|
||||
assert "Pipeline status: busy=false at completion" \
|
||||
"echo '$STATUS_RESP' | python3 -c 'import sys,json; assert json.loads(sys.stdin.read())[\"data\"][\"busy\"] is False'"
|
||||
|
||||
# 汇总
|
||||
echo
|
||||
if [ "$ASSERT_FAIL" -gt 0 ]; then
|
||||
log_err "Post-seed assertions: $ASSERT_PASS passed, $ASSERT_FAIL failed"
|
||||
exit 1
|
||||
fi
|
||||
log_ok "Post-seed assertions: $ASSERT_PASS/$ASSERT_PASS passed"
|
||||
|
||||
exit $SEED_RC
|
||||
@@ -0,0 +1,655 @@
|
||||
#!/usr/bin/env npx tsx
|
||||
/**
|
||||
* seed-v2 — 通过 v2 API 把历史对话灌入 memory-tencentdb gateway。
|
||||
*
|
||||
* 与老 `src/cli/commands/seed.ts`（v1，import 核心 runtime 直跑）相比：
|
||||
* - 纯 HTTP 客户端，不 import 任何 plugin 内部模块
|
||||
* - 通过 `/v2/conversation/add` 写 L0 + 让 gateway 自动调度 L1/L2/L3
|
||||
* - 通过轮询 `/v2/pipeline/status` 的 `busy` 标量做节流
|
||||
* - 既适用于 standalone，也适用于 service 模式（同一份代码）
|
||||
*
|
||||
* 阻塞节奏（对齐老 `seed-runtime.executeSeed`）:
|
||||
* - per-round 写入 conversation/add 后立即继续
|
||||
* - 每 N 轮（--every-n / SEED_EVERY_N）等一次 busy=false（stable polls）
|
||||
* - 每个 session 末尾再等一次
|
||||
* - 全部跑完后做 final drain
|
||||
*
|
||||
* @example
|
||||
* # 用 npm script 入口
|
||||
* npm run seed-v2 -- --input ./fixtures/minimal.json
|
||||
*
|
||||
* # 或直接 tsx
|
||||
* npx tsx scripts/seed-v2/seed-v2.ts --input fixture.json --endpoint http://127.0.0.1:18420
|
||||
*
|
||||
* # 关键参数
|
||||
* --input <file> 必选，fixture JSON 路径
|
||||
* --endpoint <url> gateway 地址（默认 http://127.0.0.1:18420）
|
||||
* --service-id <id> x-tdai-service-id（默认 default）
|
||||
* --api-key <key> Bearer key（默认 standalone-e2e）
|
||||
* --every-n <n> 每 N 轮 wait 一次 busy=false（默认 5）
|
||||
* --max-wait-ms <ms> 单次 wait 最长时间（默认 600000=10 分钟）
|
||||
* --no-final-wait 写完最后一批不等 cascade，立即退出
|
||||
* --dry-run 只打印计划不实际请求
|
||||
* --quiet 静默模式
|
||||
*/
|
||||
|
||||
import { readFileSync, existsSync } from "node:fs";
|
||||
import { resolve as resolvePath } from "node:path";
|
||||
import { parseArgs } from "node:util";
|
||||
import dayjs from "dayjs";
|
||||
|
||||
import { validateAndNormalizeRaw, fillTimestamps, SeedValidationError } from "./input-validate.js";
|
||||
import type { NormalizedInput } from "./input-types.js";
|
||||
|
||||
// ============================
|
||||
// Logger (timestamped console wrappers)
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Full ISO 8601 local-timezone timestamp, e.g. "2026-05-21T15:03:00.123+08:00".
|
||||
* Aligned with the standalone gateway's console logger so seed-v2 lines and
|
||||
* gateway lines can be merged/sorted by timestamp during incident review.
|
||||
*/
|
||||
function ts(): string {
|
||||
return dayjs().format("YYYY-MM-DDTHH:mm:ss.SSSZ");
|
||||
}
|
||||
|
||||
const log = (msg: string): void => { console.log(`${ts()} ${msg}`); };
|
||||
const warn = (msg: string): void => { console.warn(`${ts()} ${msg}`); };
|
||||
const err = (msg: string): void => { console.error(`${ts()} ${msg}`); };
|
||||
|
||||
// Truncate long content to a single-line preview safe for logs.
|
||||
function preview(s: string, max = 80): string {
|
||||
const oneLine = s.replace(/\s+/g, " ").trim();
|
||||
return oneLine.length <= max ? oneLine : `${oneLine.slice(0, max)}...`;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// CLI args + env fallback
|
||||
// ============================
|
||||
|
||||
interface CliOptions {
|
||||
input: string;
|
||||
endpoint: string;
|
||||
apiKey: string;
|
||||
serviceId: string;
|
||||
everyN: number;
|
||||
pollMs: number;
|
||||
stableRounds: number;
|
||||
maxWaitMs: number; // per every-N / per-session-tail wait timeout (L1 only)
|
||||
finalMaxWaitMs: number; // final all-layer drain wait timeout (L1+L2+L3)
|
||||
finalWait: boolean;
|
||||
dryRun: boolean;
|
||||
quiet: boolean;
|
||||
// Input validation / normalization options (parity with old seed CLI)
|
||||
sessionKey?: string; // fallback for sessions missing sessionKey
|
||||
strictRoundRole: boolean; // each round must have user + assistant
|
||||
autoFillTimestamps: boolean; // when all-missing, fill globally-monotonic ts
|
||||
}
|
||||
|
||||
function parseCli(argv: string[]): CliOptions {
|
||||
const { values } = parseArgs({
|
||||
args: argv,
|
||||
options: {
|
||||
input: { type: "string", short: "i" },
|
||||
endpoint: { type: "string", short: "e" },
|
||||
"api-key": { type: "string" },
|
||||
"service-id": { type: "string", short: "s" },
|
||||
"every-n": { type: "string", short: "n" },
|
||||
"poll-ms": { type: "string" },
|
||||
"stable-rounds": { type: "string" },
|
||||
"max-wait-ms": { type: "string" },
|
||||
"final-max-wait-ms": { type: "string" },
|
||||
"no-final-wait": { type: "boolean" },
|
||||
"dry-run": { type: "boolean" },
|
||||
"session-key": { type: "string" },
|
||||
"strict-round-role": { type: "boolean" },
|
||||
"no-auto-fill-timestamps": { type: "boolean" },
|
||||
quiet: { type: "boolean", short: "q" },
|
||||
help: { type: "boolean", short: "h" },
|
||||
},
|
||||
allowPositionals: false,
|
||||
strict: false,
|
||||
});
|
||||
|
||||
if (values.help) {
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const input = (values.input as string | undefined) ?? process.argv[2];
|
||||
if (!input) {
|
||||
console.error("error: --input <fixture.json> is required");
|
||||
printHelp();
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const intOpt = (key: string, env: string, def: number): number => {
|
||||
const raw = (values[key] as string | undefined) ?? process.env[env];
|
||||
if (raw == null || raw === "") return def;
|
||||
const n = parseInt(raw, 10);
|
||||
return Number.isFinite(n) ? n : def;
|
||||
};
|
||||
|
||||
return {
|
||||
input,
|
||||
endpoint: (values.endpoint as string) ?? process.env.SEED_ENDPOINT ?? "http://127.0.0.1:18420",
|
||||
apiKey: (values["api-key"] as string) ?? process.env.SEED_API_KEY ?? "standalone-e2e",
|
||||
serviceId: (values["service-id"] as string) ?? process.env.SEED_SERVICE_ID ?? "default",
|
||||
everyN: intOpt("every-n", "SEED_EVERY_N", 5),
|
||||
pollMs: intOpt("poll-ms", "SEED_POLL_MS", 500),
|
||||
stableRounds: intOpt("stable-rounds", "SEED_STABLE_ROUNDS", 2),
|
||||
maxWaitMs: intOpt("max-wait-ms", "SEED_MAX_WAIT_MS", 180_000),
|
||||
finalMaxWaitMs: intOpt("final-max-wait-ms", "SEED_FINAL_MAX_WAIT_MS", 600_000),
|
||||
finalWait: !values["no-final-wait"],
|
||||
dryRun: Boolean(values["dry-run"]),
|
||||
quiet: Boolean(values.quiet) || process.env.SEED_VERBOSE === "0",
|
||||
sessionKey: (values["session-key"] as string | undefined) ?? process.env.SEED_FALLBACK_SESSION_KEY,
|
||||
strictRoundRole: Boolean(values["strict-round-role"]) || process.env.SEED_STRICT_ROUND_ROLE === "1",
|
||||
autoFillTimestamps: !values["no-auto-fill-timestamps"] && process.env.SEED_AUTO_FILL_TIMESTAMPS !== "0",
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
console.log(`
|
||||
seed-v2 — feed historical conversations into memory-tencentdb via v2 API
|
||||
|
||||
Usage:
|
||||
seed-v2 --input <fixture.json> [options]
|
||||
|
||||
Required:
|
||||
-i, --input <file> Fixture JSON file (Format A: { sessions: [...] })
|
||||
|
||||
Options:
|
||||
-e, --endpoint <url> Gateway URL (default: http://127.0.0.1:18420)
|
||||
--api-key <key> Bearer key (default: standalone-e2e)
|
||||
-s, --service-id <id> x-tdai-service-id (default: default)
|
||||
-n, --every-n <n> Wait for busy=false every N rounds per session (default: 5)
|
||||
--poll-ms <ms> Status poll interval (default: 500)
|
||||
--stable-rounds <n> Consecutive idle polls before considered stable (default: 2)
|
||||
--max-wait-ms <ms> Per-batch wait timeout, L1 only (default: 180000 = 3 min)
|
||||
--final-max-wait-ms <ms> Final drain wait timeout, ALL layers L1+L2+L3 (default: 600000 = 10 min)
|
||||
--no-final-wait Skip final cascade drain wait
|
||||
--dry-run Print plan, no requests
|
||||
--session-key <key> Fallback sessionKey for sessions missing one
|
||||
--strict-round-role Each round must contain user + assistant
|
||||
--no-auto-fill-timestamps Don't auto-fill missing timestamps (will error if all missing)
|
||||
-q, --quiet Quiet mode (env SEED_VERBOSE=0)
|
||||
-h, --help Show this help
|
||||
|
||||
Fixture JSON (Format A):
|
||||
{
|
||||
"sessions": [
|
||||
{
|
||||
"sessionKey": "user-001",
|
||||
"sessionId": "user-001",
|
||||
"conversations": [
|
||||
[
|
||||
{ "role": "user", "content": "..." },
|
||||
{ "role": "assistant", "content": "..." }
|
||||
]
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Exit codes:
|
||||
0 success
|
||||
1 seed or wait failed
|
||||
2 fixture / config error
|
||||
`.trim());
|
||||
}
|
||||
|
||||
// ============================
|
||||
// HTTP helper
|
||||
// ============================
|
||||
|
||||
interface ApiResponse<T = unknown> {
|
||||
code: number;
|
||||
message: string;
|
||||
request_id: string;
|
||||
data?: T;
|
||||
}
|
||||
|
||||
interface ApiCallResult<T = unknown> {
|
||||
body: ApiResponse<T>;
|
||||
httpStatus: number;
|
||||
durationMs: number;
|
||||
}
|
||||
|
||||
class SeedClient {
|
||||
private callSeq = 0;
|
||||
constructor(private opts: CliOptions) {}
|
||||
|
||||
async post<T = unknown>(path: string, body: unknown): Promise<ApiCallResult<T>> {
|
||||
this.callSeq++;
|
||||
const url = `${this.opts.endpoint}/v2/${path}`;
|
||||
const startedAt = Date.now();
|
||||
const resp = await fetch(url, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": `Bearer ${this.opts.apiKey}`,
|
||||
"x-tdai-service-id": this.opts.serviceId,
|
||||
},
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
const durationMs = Date.now() - startedAt;
|
||||
const text = await resp.text();
|
||||
let parsed: ApiResponse<T>;
|
||||
try {
|
||||
parsed = JSON.parse(text) as ApiResponse<T>;
|
||||
} catch {
|
||||
throw new Error(`non-JSON response from /v2/${path}: ${resp.status} ${text.slice(0, 200)}`);
|
||||
}
|
||||
return { body: parsed, httpStatus: resp.status, durationMs };
|
||||
}
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Fixture loading + validation (delegates to input-validate.ts)
|
||||
// ============================
|
||||
|
||||
/**
|
||||
* Load fixture from disk → parse JSON → validate → normalize. Returns the
|
||||
* NormalizedInput shape used downstream (sessions[].rounds[].messages[]),
|
||||
* with timestamps either preserved, mixed (rejected), or auto-filled to
|
||||
* globally-monotonic epoch ms (parity with v1 fillTimestamps).
|
||||
*
|
||||
* On validation failure throws SeedValidationError; caller should print and exit 2.
|
||||
*/
|
||||
function loadAndValidate(opts: CliOptions): NormalizedInput {
|
||||
const abs = resolvePath(opts.input);
|
||||
if (!existsSync(abs)) {
|
||||
throw new SeedValidationError([{ stage: "file", message: `Fixture not found: ${abs}` }]);
|
||||
}
|
||||
const text = readFileSync(abs, "utf8").trim();
|
||||
if (!text) {
|
||||
throw new SeedValidationError([{ stage: "file", message: `Fixture is empty: ${abs}` }]);
|
||||
}
|
||||
let raw: unknown;
|
||||
try {
|
||||
raw = JSON.parse(text);
|
||||
} catch (err) {
|
||||
throw new SeedValidationError([{
|
||||
stage: "file",
|
||||
message: `JSON parse error: ${err instanceof Error ? err.message : String(err)}`,
|
||||
}]);
|
||||
}
|
||||
return validateAndNormalizeRaw(raw, {
|
||||
sessionKey: opts.sessionKey,
|
||||
strictRoundRole: opts.strictRoundRole,
|
||||
autoFillTimestamps: opts.autoFillTimestamps,
|
||||
});
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Status polling
|
||||
// ============================
|
||||
|
||||
interface LayerStatus {
|
||||
queued: number;
|
||||
running: number;
|
||||
queued_sessions: string[];
|
||||
running_sessions: string[];
|
||||
idle: boolean;
|
||||
}
|
||||
|
||||
interface StatusData {
|
||||
l1: LayerStatus;
|
||||
l2: LayerStatus;
|
||||
l3: LayerStatus;
|
||||
}
|
||||
|
||||
async function pollStatus(client: SeedClient): Promise<StatusData> {
|
||||
const r = await client.post<StatusData>("pipeline/status", {});
|
||||
if (r.body.code !== 0 || !r.body.data) {
|
||||
throw new Error(`/v2/pipeline/status failed: HTTP ${r.httpStatus} code=${r.body.code} msg=${r.body.message}`);
|
||||
}
|
||||
return r.body.data;
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait until the requested layers are idle for `stableRounds` consecutive polls.
|
||||
*
|
||||
* @param waitMode "l1" → only wait for L1 to drain (mid-batch / per-session);
|
||||
* L2/L3 cascade may still run in background.
|
||||
* "all" → wait for L1+L2+L3 all idle (final drain after all
|
||||
* rounds dispatched).
|
||||
*
|
||||
* On timeout: log a `[wait] done reason=timeout` line and return (do not
|
||||
* throw). Data is already on disk and the gateway keeps cooking; callers
|
||||
* decide whether the next batch can proceed.
|
||||
*
|
||||
* Logging:
|
||||
* - `[wait] enter` once at the start (label, mode, budget).
|
||||
* - `[poll]` per status snapshot, throttled: first poll, OR queue/running
|
||||
* signature changed, OR 5s heartbeat. Long stable waits stay quiet,
|
||||
* stalls keep showing progress.
|
||||
* - `[wait] done` once at exit with reason=stable|timeout, elapsed, polls.
|
||||
*/
|
||||
async function waitForBusyFalse(
|
||||
client: SeedClient,
|
||||
opts: CliOptions,
|
||||
label: string,
|
||||
waitMode: "l1" | "all",
|
||||
maxWaitMs: number,
|
||||
): Promise<void> {
|
||||
const startTime = Date.now();
|
||||
let consecutiveIdle = 0;
|
||||
let lastSnapshot: StatusData | null = null;
|
||||
let lastLogAtMs = 0;
|
||||
let pollCount = 0;
|
||||
const HEARTBEAT_INTERVAL_MS = 5000;
|
||||
|
||||
log(`[seed-v2] [wait] enter label="${label}" mode=${waitMode} maxWaitMs=${maxWaitMs} stableRounds=${opts.stableRounds} pollMs=${opts.pollMs}`);
|
||||
|
||||
// Format a compact one-line snapshot for logging.
|
||||
const fmt = (s: StatusData): string => {
|
||||
const layer = (name: string, l: LayerStatus) => {
|
||||
const extras: string[] = [];
|
||||
if (l.running_sessions.length > 0) extras.push(`running=${JSON.stringify(l.running_sessions)}`);
|
||||
if (l.queued_sessions.length > 0) extras.push(`queued=${JSON.stringify(l.queued_sessions)}`);
|
||||
const tail = extras.length > 0 ? ` ${extras.join(" ")}` : "";
|
||||
return `${name}(q=${l.queued} r=${l.running} idle=${l.idle}${tail})`;
|
||||
};
|
||||
return [layer("L1", s.l1), layer("L2", s.l2), layer("L3", s.l3)].join(" ");
|
||||
};
|
||||
|
||||
// Stable signature for change-detection (only fields that matter for idle).
|
||||
const sig = (s: StatusData): string =>
|
||||
`${s.l1.queued}/${s.l1.running}|${s.l2.queued}/${s.l2.running}|${s.l3.queued}/${s.l3.running}`;
|
||||
let lastSig = "";
|
||||
|
||||
while (true) {
|
||||
const elapsed = Date.now() - startTime;
|
||||
if (elapsed > maxWaitMs) {
|
||||
const last = lastSnapshot ? fmt(lastSnapshot) : "no-snapshot";
|
||||
warn(
|
||||
`[seed-v2] [wait] done label="${label}" mode=${waitMode} elapsed=${(elapsed / 1000).toFixed(1)}s ` +
|
||||
`polls=${pollCount} reason=timeout last=${last}`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const s = await pollStatus(client);
|
||||
pollCount++;
|
||||
lastSnapshot = s;
|
||||
const isIdle = waitMode === "l1"
|
||||
? s.l1.idle
|
||||
: (s.l1.idle && s.l2.idle && s.l3.idle);
|
||||
|
||||
if (isIdle) consecutiveIdle++;
|
||||
else consecutiveIdle = 0;
|
||||
|
||||
// Decide whether to log this poll.
|
||||
const now = Date.now();
|
||||
const currentSig = sig(s);
|
||||
const changed = currentSig !== lastSig;
|
||||
const heartbeat = now - lastLogAtMs >= HEARTBEAT_INTERVAL_MS;
|
||||
const isFirstPoll = lastLogAtMs === 0;
|
||||
const willBeStable = isIdle && consecutiveIdle >= opts.stableRounds;
|
||||
|
||||
if (!opts.quiet && (isFirstPoll || changed || heartbeat || willBeStable)) {
|
||||
const mark = isIdle ? "idle" : "busy";
|
||||
const stable = willBeStable ? " ✓ STABLE" : "";
|
||||
log(
|
||||
`[seed-v2] [poll] ${label} t+${(elapsed / 1000).toFixed(1)}s ${mark} ${fmt(s)} ` +
|
||||
`consecutiveIdle=${consecutiveIdle}${stable}`,
|
||||
);
|
||||
lastLogAtMs = now;
|
||||
lastSig = currentSig;
|
||||
}
|
||||
|
||||
if (willBeStable) {
|
||||
log(
|
||||
`[seed-v2] [wait] done label="${label}" mode=${waitMode} elapsed=${(elapsed / 1000).toFixed(1)}s ` +
|
||||
`polls=${pollCount} reason=stable`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
await new Promise((r) => setTimeout(r, opts.pollMs));
|
||||
}
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Conversation/add per round
|
||||
// ============================
|
||||
|
||||
interface AddRoundContext {
|
||||
sessionIndex: number; // 1-based among non-empty sessions
|
||||
totalSessions: number;
|
||||
sessionKey: string;
|
||||
sessionId: string;
|
||||
roundIndex: number; // 1-based within this session
|
||||
totalRoundsInSession: number;
|
||||
cumRounds: number; // 1-based across whole fixture
|
||||
totalRounds: number;
|
||||
cumMessagesAfter: number; // includes this round (post-success)
|
||||
totalMessages: number;
|
||||
}
|
||||
|
||||
async function addRound(
|
||||
client: SeedClient,
|
||||
ctx: AddRoundContext,
|
||||
messages: { role: string; content: string; timestamp: number }[],
|
||||
quiet: boolean,
|
||||
): Promise<number> {
|
||||
// After validateAndNormalizeRaw + fillTimestamps, every message has a positive
|
||||
// epoch-ms `timestamp` (or 0 if user explicitly opted out via
|
||||
// --no-auto-fill-timestamps). v2 conversation/add schema requires ISO 8601 string.
|
||||
const payload = messages.map((m) => ({
|
||||
role: m.role,
|
||||
content: m.content,
|
||||
timestamp: new Date(m.timestamp || Date.now()).toISOString(),
|
||||
}));
|
||||
|
||||
const roles = payload.map((p) => p.role).join(",");
|
||||
const firstTs = payload[0]?.timestamp ?? "n/a";
|
||||
const lastTs = payload[payload.length - 1]?.timestamp ?? "n/a";
|
||||
|
||||
if (!quiet) {
|
||||
log(
|
||||
`[seed-v2] [post] session ${ctx.sessionIndex}/${ctx.totalSessions} round ${ctx.roundIndex}/${ctx.totalRoundsInSession} ` +
|
||||
`(${ctx.sessionKey}) → POST /v2/conversation/add session_id="${ctx.sessionId}" ` +
|
||||
`msgs=${payload.length} roles=[${roles}] firstTs=${firstTs} lastTs=${lastTs}`,
|
||||
);
|
||||
for (let i = 0; i < payload.length; i++) {
|
||||
const m = payload[i]!;
|
||||
log(`[seed-v2] [post] ${m.role}[${i}]: "${preview(m.content)}"`);
|
||||
}
|
||||
}
|
||||
|
||||
const r = await client.post<{ accepted_ids: string[]; total_count: number }>("conversation/add", {
|
||||
session_id: ctx.sessionId,
|
||||
messages: payload,
|
||||
});
|
||||
|
||||
const ok = r.body.code === 0 && r.body.data;
|
||||
if (!ok) {
|
||||
err(
|
||||
`[seed-v2] [post] session ${ctx.sessionIndex}/${ctx.totalSessions} round ${ctx.roundIndex}/${ctx.totalRoundsInSession} ` +
|
||||
`(${ctx.sessionKey}) ✗ HTTP ${r.httpStatus} code=${r.body.code} msg="${r.body.message}" in ${r.durationMs}ms`,
|
||||
);
|
||||
throw new Error(`/v2/conversation/add failed for ${ctx.sessionId}: HTTP ${r.httpStatus} code=${r.body.code} msg=${r.body.message}`);
|
||||
}
|
||||
|
||||
const accepted = r.body.data!.accepted_ids.length;
|
||||
if (!quiet) {
|
||||
log(
|
||||
`[seed-v2] [post] session ${ctx.sessionIndex}/${ctx.totalSessions} round ${ctx.roundIndex}/${ctx.totalRoundsInSession} ` +
|
||||
`(${ctx.sessionKey}) ✓ HTTP ${r.httpStatus} code=0 accepted=${accepted} in ${r.durationMs}ms ` +
|
||||
`(cum: rounds=${ctx.cumRounds}/${ctx.totalRounds} msgs=${ctx.cumMessagesAfter}/${ctx.totalMessages})`,
|
||||
);
|
||||
}
|
||||
return accepted;
|
||||
}
|
||||
|
||||
// ============================
|
||||
// Main
|
||||
// ============================
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const opts = parseCli(process.argv.slice(2));
|
||||
const client = new SeedClient(opts);
|
||||
|
||||
log(
|
||||
`[seed-v2] endpoint=${opts.endpoint} serviceId=${opts.serviceId} apiKey=${opts.apiKey ? "***" : "(none)"}`,
|
||||
);
|
||||
log(
|
||||
`[seed-v2] cadence: everyN=${opts.everyN} stableRounds=${opts.stableRounds} pollMs=${opts.pollMs} ` +
|
||||
`maxWaitMs=${opts.maxWaitMs}(L1-only) finalMaxWaitMs=${opts.finalMaxWaitMs}(all-layers) finalWait=${opts.finalWait}`,
|
||||
);
|
||||
|
||||
// Load + validate + normalize fixture (Layer 2-6 of v1 seed: format detect /
|
||||
// session+round+message validation / timestamp consistency check / auto-fill).
|
||||
let normalized: NormalizedInput;
|
||||
try {
|
||||
normalized = loadAndValidate(opts);
|
||||
} catch (e) {
|
||||
if (e instanceof SeedValidationError) {
|
||||
err(`[seed-v2] ${e.message}`);
|
||||
} else {
|
||||
err(`[seed-v2] fixture load failed: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
process.exit(2);
|
||||
}
|
||||
const sessions = normalized.sessions;
|
||||
const { totalRounds, totalMessages, hasTimestamps } = normalized;
|
||||
log(
|
||||
`[seed-v2] fixture=${opts.input} sessions=${sessions.length} rounds=${totalRounds} ` +
|
||||
`messages=${totalMessages} hasTimestamps=${hasTimestamps}` +
|
||||
(!hasTimestamps && opts.autoFillTimestamps ? " (auto-filled)" : "") +
|
||||
` autoFill=${opts.autoFillTimestamps} strictRoundRole=${opts.strictRoundRole}`,
|
||||
);
|
||||
|
||||
// Session manifest preview (first 3 + count).
|
||||
const previewSessions = sessions.slice(0, 3)
|
||||
.map((s) => `${s.sessionKey}(${s.rounds.length}r)`).join(" ");
|
||||
const moreSessions = sessions.length > 3 ? ` ... (+${sessions.length - 3} more)` : "";
|
||||
log(`[seed-v2] sessions: ${previewSessions}${moreSessions}`);
|
||||
|
||||
if (opts.dryRun) {
|
||||
log("[seed-v2] DRY RUN — exiting without making any requests");
|
||||
for (const s of sessions) {
|
||||
log(`[seed-v2] would add: sessionKey=${s.sessionKey} sessionId=${s.sessionId} rounds=${s.rounds.length}`);
|
||||
}
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
// Pre-flight: status接口 must respond
|
||||
let preflight: StatusData;
|
||||
try {
|
||||
preflight = await pollStatus(client);
|
||||
} catch (e) {
|
||||
err(`[seed-v2] pre-flight /v2/pipeline/status failed: ${(e as Error).message}`);
|
||||
err(`[seed-v2] gateway up at ${opts.endpoint}? deployMode=standalone? stateBackend configured?`);
|
||||
process.exit(2);
|
||||
}
|
||||
log(
|
||||
`[seed-v2] pre-flight ok l1.idle=${preflight.l1.idle} ` +
|
||||
`(l1 q=${preflight.l1.queued} r=${preflight.l1.running}, ` +
|
||||
`l2 q=${preflight.l2.queued} r=${preflight.l2.running}, ` +
|
||||
`l3 q=${preflight.l3.queued} r=${preflight.l3.running})`,
|
||||
);
|
||||
|
||||
// Per-session, per-round seeding (timestamps now come from normalized input,
|
||||
// not synthesized per-call — global monotonic ordering already guaranteed
|
||||
// by validateAndNormalizeRaw + fillTimestamps).
|
||||
const startMs = Date.now();
|
||||
let roundsDone = 0;
|
||||
let messagesDone = 0;
|
||||
|
||||
for (let si = 0; si < sessions.length; si++) {
|
||||
const session = sessions[si]!;
|
||||
const sessionKey = session.sessionKey;
|
||||
const sessionId = session.sessionId; // always present after normalization
|
||||
|
||||
if (session.rounds.length === 0) {
|
||||
warn(`[seed-v2] session ${si + 1}/${sessions.length} key="${sessionKey}" has no rounds, skipping`);
|
||||
continue;
|
||||
}
|
||||
|
||||
const sessionStartMs = Date.now();
|
||||
let sessionMsgs = 0;
|
||||
log(
|
||||
`[seed-v2] session ${si + 1}/${sessions.length} START key="${sessionKey}" id="${sessionId}" ` +
|
||||
`rounds=${session.rounds.length}`,
|
||||
);
|
||||
|
||||
for (let ri = 0; ri < session.rounds.length; ri++) {
|
||||
const round = session.rounds[ri]!;
|
||||
if (round.messages.length === 0) continue;
|
||||
|
||||
const accepted = await addRound(
|
||||
client,
|
||||
{
|
||||
sessionIndex: si + 1,
|
||||
totalSessions: sessions.length,
|
||||
sessionKey,
|
||||
sessionId,
|
||||
roundIndex: ri + 1,
|
||||
totalRoundsInSession: session.rounds.length,
|
||||
cumRounds: roundsDone + 1,
|
||||
totalRounds,
|
||||
cumMessagesAfter: messagesDone + round.messages.length,
|
||||
totalMessages,
|
||||
},
|
||||
round.messages,
|
||||
opts.quiet,
|
||||
);
|
||||
roundsDone++;
|
||||
messagesDone += accepted;
|
||||
sessionMsgs += accepted;
|
||||
|
||||
// Old seed-runtime parity: every N rounds in this session, wait for L1 idle.
|
||||
const roundInSession = ri + 1;
|
||||
if (roundInSession % opts.everyN === 0) {
|
||||
await waitForBusyFalse(client, opts, `every-${opts.everyN}@${sessionKey}#${roundInSession}`, "l1", opts.maxWaitMs);
|
||||
}
|
||||
}
|
||||
|
||||
// Per-session tail wait (matches old seed-runtime per-session waitForL1Idle).
|
||||
await waitForBusyFalse(client, opts, `session-tail@${sessionKey}`, "l1", opts.maxWaitMs);
|
||||
|
||||
const sessionDur = (Date.now() - sessionStartMs) / 1000;
|
||||
log(
|
||||
`[seed-v2] session ${si + 1}/${sessions.length} END key="${sessionKey}" ` +
|
||||
`rounds=${session.rounds.length} msgs=${sessionMsgs} duration=${sessionDur.toFixed(1)}s`,
|
||||
);
|
||||
}
|
||||
|
||||
// Final wait: drain any cascade L2/L3 work (best-effort).
|
||||
if (opts.finalWait) {
|
||||
log(
|
||||
`[seed-v2] all rounds dispatched, final drain — waiting for L1+L2+L3 all idle ` +
|
||||
`(max ${(opts.finalMaxWaitMs / 1000).toFixed(0)}s)...`,
|
||||
);
|
||||
await waitForBusyFalse(client, opts, "final", "all", opts.finalMaxWaitMs);
|
||||
} else {
|
||||
log(`[seed-v2] all rounds dispatched, --no-final-wait set, exiting immediately`);
|
||||
}
|
||||
|
||||
const durationMs = Date.now() - startMs;
|
||||
log(
|
||||
`[seed-v2] done sessions=${sessions.length} rounds=${roundsDone}/${totalRounds} ` +
|
||||
`msgs=${messagesDone}/${totalMessages} duration=${(durationMs / 1000).toFixed(1)}s`,
|
||||
);
|
||||
|
||||
// Final status snapshot for downstream verification.
|
||||
const final = await pollStatus(client);
|
||||
const allIdle = final.l1.idle && final.l2.idle && final.l3.idle;
|
||||
log(
|
||||
`[seed-v2] final status: l1.idle=${final.l1.idle} l2.idle=${final.l2.idle} l3.idle=${final.l3.idle} ` +
|
||||
`(l1 q=${final.l1.queued} r=${final.l1.running}, ` +
|
||||
`l2 q=${final.l2.queued} r=${final.l2.running}, ` +
|
||||
`l3 q=${final.l3.queued} r=${final.l3.running})` +
|
||||
(allIdle ? " ✓ all clean" : " ⚠ residual work — gateway will keep cooking after exit"),
|
||||
);
|
||||
}
|
||||
|
||||
main().catch((e) => {
|
||||
err(`[seed-v2] FATAL: ${e instanceof Error ? e.stack : String(e)}`);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "Node16",
|
||||
"moduleResolution": "Node16",
|
||||
"outDir": "./dist",
|
||||
"rootDir": ".",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true,
|
||||
"types": ["node"],
|
||||
"declaration": false,
|
||||
"sourceMap": false
|
||||
},
|
||||
"include": ["seed-v2.ts", "input-types.ts", "input-validate.ts"],
|
||||
"exclude": ["dist", "node_modules"]
|
||||
}
|
||||
Executable
+340
@@ -0,0 +1,340 @@
|
||||
#!/usr/bin/env bash
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# setup-offload.sh — Offload 功能一键开启/关闭
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
#
|
||||
# 用法:
|
||||
# bash setup-offload.sh --enable --user-id <userId> --backend-url <url> [--backend-api-key <key>]
|
||||
# bash setup-offload.sh --disable
|
||||
# bash setup-offload.sh --status
|
||||
#
|
||||
# 开启流程:
|
||||
# 1. 前置检查（openclaw.json 存在、openclaw 已安装）
|
||||
# 2. Patch 校验 & 执行（after_tool_call messages 注入）— 失败则终止
|
||||
# 3. 设置 plugins.slots.contextEngine
|
||||
# 4. 设置 offload.enabled + backendUrl + userId [+ backendApiKey]
|
||||
# 5. 设置 compaction.mode = safeguard
|
||||
#
|
||||
# 关闭流程:
|
||||
# 1. 设置 offload.enabled = false
|
||||
# 2. 删除 plugins.slots.contextEngine（清理 slot 占用）
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
set -euo pipefail
|
||||
|
||||
# ── 颜色 ──
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
|
||||
CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
|
||||
info() { echo -e "${CYAN}[INFO]${NC} $*"; }
|
||||
ok() { echo -e "${GREEN}[OK]${NC} $*"; }
|
||||
warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
|
||||
fail() { echo -e "${RED}[FAIL]${NC} $*" >&2; exit 1; }
|
||||
|
||||
# ── 常量 ──
|
||||
OPENCLAW_JSON="${HOME}/.openclaw/openclaw.json"
|
||||
PLUGIN_ID="memory-tencentdb"
|
||||
CONTEXT_ENGINE_ID="memory-tencentdb"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PATCH_SCRIPT="${SCRIPT_DIR}/openclaw-after-tool-call-messages.patch.sh"
|
||||
|
||||
# ── 参数解析 ──
|
||||
MODE=""
|
||||
USER_ID=""
|
||||
BACKEND_URL=""
|
||||
BACKEND_API_KEY=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--enable) MODE="enable"; shift ;;
|
||||
--disable) MODE="disable"; shift ;;
|
||||
--status) MODE="status"; shift ;;
|
||||
--user-id) USER_ID="$2"; shift 2 ;;
|
||||
--backend-url) BACKEND_URL="$2"; shift 2 ;;
|
||||
--backend-api-key) BACKEND_API_KEY="$2"; shift 2 ;;
|
||||
-h|--help)
|
||||
echo "用法:"
|
||||
echo " bash setup-offload.sh --enable --user-id <userId> --backend-url <url> [--backend-api-key <key>]"
|
||||
echo " bash setup-offload.sh --disable"
|
||||
echo " bash setup-offload.sh --status"
|
||||
echo ""
|
||||
echo "参数:"
|
||||
echo " --user-id (必填) 用户 ID"
|
||||
echo " --backend-url (必填) offload 后端地址，如 http://1.2.3.4:8080"
|
||||
echo " --backend-api-key (可选) 后端 API 认证 token"
|
||||
exit 0
|
||||
;;
|
||||
*) fail "未知参数: $1" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[[ -z "$MODE" ]] && fail "请指定模式: --enable / --disable / --status"
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# 公共函数
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
check_openclaw_json() {
|
||||
if [[ ! -f "$OPENCLAW_JSON" ]]; then
|
||||
fail "openclaw.json 不存在: $OPENCLAW_JSON"
|
||||
fi
|
||||
# 验证 JSON 格式
|
||||
python3 -c "import json; json.load(open('$OPENCLAW_JSON'))" 2>/dev/null \
|
||||
|| fail "openclaw.json 格式错误"
|
||||
}
|
||||
|
||||
backup_config() {
|
||||
local bak="${OPENCLAW_JSON}.bak.$(date +%Y%m%d_%H%M%S)"
|
||||
cp "$OPENCLAW_JSON" "$bak"
|
||||
info "配置已备份: $bak"
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# --status: 显示当前配置状态
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
show_status() {
|
||||
check_openclaw_json
|
||||
python3 -c "
|
||||
import json
|
||||
|
||||
with open('$OPENCLAW_JSON') as f:
|
||||
cfg = json.load(f)
|
||||
|
||||
# Context Engine Slot
|
||||
slot = cfg.get('plugins', {}).get('slots', {}).get('contextEngine', '(未设置)')
|
||||
print(f' Context Engine Slot: {slot}')
|
||||
|
||||
# Offload config
|
||||
offload = cfg.get('plugins', {}).get('entries', {}).get('$PLUGIN_ID', {}).get('config', {}).get('offload', {})
|
||||
enabled = offload.get('enabled', False)
|
||||
backend = offload.get('backendUrl', '(未设置)')
|
||||
user_id = offload.get('userId', '(未设置)')
|
||||
api_key = offload.get('backendApiKey', '')
|
||||
timeout = offload.get('backendTimeoutMs', '(默认)')
|
||||
mild = offload.get('mildOffloadRatio', '(默认 0.5)')
|
||||
agg = offload.get('aggressiveCompressRatio', '(默认 0.85)')
|
||||
|
||||
status_icon = '✅ 已开启' if enabled else '❌ 已关闭'
|
||||
api_key_display = f'{api_key[:8]}...' if api_key and len(api_key) > 8 else (api_key or '(未设置)')
|
||||
print(f' Offload 状态: {status_icon}')
|
||||
print(f' Backend URL: {backend}')
|
||||
print(f' Backend Key: {api_key_display}')
|
||||
print(f' User ID: {user_id}')
|
||||
print(f' Timeout: {timeout}ms')
|
||||
print(f' Mild Ratio: {mild}')
|
||||
print(f' Aggressive: {agg}')
|
||||
|
||||
# Compaction mode
|
||||
compaction = cfg.get('agents', {}).get('defaults', {}).get('compaction', {}).get('mode', '(未设置)')
|
||||
print(f' Compaction: {compaction}')
|
||||
"
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# --enable: 开启 offload
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
enable_offload() {
|
||||
# 参数校验
|
||||
[[ -z "$USER_ID" ]] && fail "缺少 --user-id 参数"
|
||||
[[ -z "$BACKEND_URL" ]] && fail "缺少 --backend-url 参数"
|
||||
|
||||
# URL 格式基本校验
|
||||
if [[ ! "$BACKEND_URL" =~ ^https?:// ]]; then
|
||||
fail "backendUrl 格式错误，应以 http:// 或 https:// 开头: $BACKEND_URL"
|
||||
fi
|
||||
|
||||
check_openclaw_json
|
||||
backup_config
|
||||
|
||||
echo ""
|
||||
info "${BOLD}[1/4] Patch 校验${NC}"
|
||||
|
||||
# ── Step 1: 调用 patch 脚本（幂等，已 patch 过会跳过） ──
|
||||
# patch 脚本自带精确的幂等检测，通过退出码判断结果：
|
||||
# 0 = 成功（新 patch 或已跳过）
|
||||
# 1 = 失败（无法 patch）
|
||||
if [[ -f "$PATCH_SCRIPT" ]]; then
|
||||
info "执行 patch 脚本..."
|
||||
local patch_exit=0
|
||||
local patch_output
|
||||
patch_output=$(bash "$PATCH_SCRIPT" 2>&1) || patch_exit=$?
|
||||
|
||||
# 显示 patch 脚本输出（缩进）
|
||||
while IFS= read -r line; do
|
||||
echo " $line"
|
||||
done <<< "$patch_output"
|
||||
|
||||
if [[ $patch_exit -eq 0 ]]; then
|
||||
ok "Patch 校验通过"
|
||||
else
|
||||
echo ""
|
||||
echo -e "${RED}═══════════════════════════════════════════════════${NC}"
|
||||
echo -e "${RED} ❌ Patch 失败（退出码: $patch_exit）${NC}"
|
||||
echo -e "${RED}═══════════════════════════════════════════════════${NC}"
|
||||
echo ""
|
||||
echo -e " ${RED}after_tool_call hook 无法获取 session messages，${NC}"
|
||||
echo -e " ${RED}offload L1/L3 压缩将无法正常工作。${NC}"
|
||||
echo ""
|
||||
echo -e " ${CYAN}排查步骤:${NC}"
|
||||
echo -e " 1. DEBUG=1 bash $PATCH_SCRIPT"
|
||||
echo -e " 2. 检查 openclaw 版本是否兼容"
|
||||
echo ""
|
||||
exit 2
|
||||
fi
|
||||
else
|
||||
echo -e "${RED}[FAIL]${NC} Patch 脚本不存在: $PATCH_SCRIPT" >&2
|
||||
echo -e " ${RED}offload 功能依赖此 patch，终止开启流程。${NC}" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# ── Step 2: 设置 context engine slot ──
|
||||
echo ""
|
||||
info "${BOLD}[2/4] 设置 Context Engine Slot${NC}"
|
||||
|
||||
python3 -c "
|
||||
import json
|
||||
|
||||
with open('$OPENCLAW_JSON') as f:
|
||||
cfg = json.load(f)
|
||||
|
||||
# 确保 plugins.slots 存在
|
||||
cfg.setdefault('plugins', {}).setdefault('slots', {})
|
||||
cfg['plugins']['slots']['contextEngine'] = '$CONTEXT_ENGINE_ID'
|
||||
print(' slots.contextEngine = $CONTEXT_ENGINE_ID')
|
||||
|
||||
with open('$OPENCLAW_JSON', 'w') as f:
|
||||
json.dump(cfg, f, indent=2, ensure_ascii=False)
|
||||
f.write('\n')
|
||||
"
|
||||
ok "Context Engine Slot 已设置"
|
||||
|
||||
# ── Step 3: 设置 offload 配置 ──
|
||||
echo ""
|
||||
info "${BOLD}[3/4] 设置 Offload 配置${NC}"
|
||||
|
||||
python3 -c "
|
||||
import json
|
||||
|
||||
with open('$OPENCLAW_JSON') as f:
|
||||
cfg = json.load(f)
|
||||
|
||||
# 确保路径存在
|
||||
entry = cfg.setdefault('plugins', {}).setdefault('entries', {}).setdefault('$PLUGIN_ID', {})
|
||||
config = entry.setdefault('config', {})
|
||||
offload = config.setdefault('offload', {})
|
||||
|
||||
# 设置必要配置
|
||||
offload['enabled'] = True
|
||||
offload['backendUrl'] = '$BACKEND_URL'
|
||||
offload['userId'] = '$USER_ID'
|
||||
offload.setdefault('backendTimeoutMs', 120000)
|
||||
|
||||
api_key = '$BACKEND_API_KEY'
|
||||
if api_key:
|
||||
offload['backendApiKey'] = api_key
|
||||
print(f' offload.backendApiKey = {api_key[:8]}...' if len(api_key) > 8 else f' offload.backendApiKey = {api_key}')
|
||||
elif 'backendApiKey' in offload:
|
||||
del offload['backendApiKey']
|
||||
|
||||
print(' offload.enabled = true')
|
||||
print(' offload.backendUrl = $BACKEND_URL')
|
||||
print(' offload.userId = $USER_ID')
|
||||
print(f' offload.backendTimeoutMs = {offload[\"backendTimeoutMs\"]}')
|
||||
|
||||
with open('$OPENCLAW_JSON', 'w') as f:
|
||||
json.dump(cfg, f, indent=2, ensure_ascii=False)
|
||||
f.write('\n')
|
||||
"
|
||||
ok "Offload 配置已设置"
|
||||
|
||||
# ── Step 4: 设置 compaction mode ──
|
||||
echo ""
|
||||
info "${BOLD}[4/4] 设置 Compaction Mode${NC}"
|
||||
|
||||
python3 -c "
|
||||
import json
|
||||
|
||||
with open('$OPENCLAW_JSON') as f:
|
||||
cfg = json.load(f)
|
||||
|
||||
defaults = cfg.setdefault('agents', {}).setdefault('defaults', {})
|
||||
compaction = defaults.setdefault('compaction', {})
|
||||
old_mode = compaction.get('mode', '(未设置)')
|
||||
compaction['mode'] = 'safeguard'
|
||||
print(f' compaction.mode: {old_mode} → safeguard')
|
||||
|
||||
with open('$OPENCLAW_JSON', 'w') as f:
|
||||
json.dump(cfg, f, indent=2, ensure_ascii=False)
|
||||
f.write('\n')
|
||||
"
|
||||
ok "Compaction mode 已设置为 safeguard"
|
||||
|
||||
# ── 完成 ──
|
||||
echo ""
|
||||
echo -e "${GREEN}═══════════════════════════════════════════════════${NC}"
|
||||
echo -e "${GREEN} ✅ Offload 已开启${NC}"
|
||||
echo -e "${GREEN}═══════════════════════════════════════════════════${NC}"
|
||||
echo ""
|
||||
show_status
|
||||
echo ""
|
||||
echo -e " ${CYAN}提示: 需要重启 gateway 才能生效${NC}"
|
||||
echo -e " ${CYAN} bash install-plugin.sh --restart${NC}"
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# --disable: 关闭 offload
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
disable_offload() {
|
||||
check_openclaw_json
|
||||
backup_config
|
||||
|
||||
python3 -c "
|
||||
import json
|
||||
|
||||
with open('$OPENCLAW_JSON') as f:
|
||||
cfg = json.load(f)
|
||||
|
||||
# 关闭 offload.enabled（用 setdefault 确保路径存在，修改能回写到 cfg）
|
||||
entry = cfg.setdefault('plugins', {}).setdefault('entries', {}).setdefault('$PLUGIN_ID', {})
|
||||
config = entry.setdefault('config', {})
|
||||
offload = config.setdefault('offload', {})
|
||||
offload['enabled'] = False
|
||||
print(' offload.enabled = false')
|
||||
|
||||
# 移除 contextEngine slot
|
||||
plugins = cfg.get('plugins', {})
|
||||
slots = plugins.get('slots', {})
|
||||
if 'contextEngine' in slots:
|
||||
del slots['contextEngine']
|
||||
print(' slots.contextEngine → 已删除')
|
||||
# 如果 slots 变空了，也一并移除 slots 键
|
||||
if not slots and 'slots' in plugins:
|
||||
del plugins['slots']
|
||||
print(' plugins.slots → 已清理（空对象）')
|
||||
else:
|
||||
print(' slots.contextEngine → 无需删除（不存在）')
|
||||
|
||||
with open('$OPENCLAW_JSON', 'w') as f:
|
||||
json.dump(cfg, f, indent=2, ensure_ascii=False)
|
||||
f.write('\n')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo -e "${YELLOW}═══════════════════════════════════════════════════${NC}"
|
||||
echo -e "${YELLOW} ❌ Offload 已关闭${NC}"
|
||||
echo -e "${YELLOW}═══════════════════════════════════════════════════${NC}"
|
||||
echo ""
|
||||
echo -e " ${CYAN}提示: 需要重启 gateway 才能生效${NC}"
|
||||
echo -e " ${CYAN} bash install-plugin.sh --restart${NC}"
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# 主入口
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
case "$MODE" in
|
||||
enable) enable_offload ;;
|
||||
disable) disable_offload ;;
|
||||
status)
|
||||
echo ""
|
||||
info "${BOLD}Offload 配置状态${NC}"
|
||||
show_status
|
||||
;;
|
||||
esac
|
||||
@@ -0,0 +1,325 @@
|
||||
#!/usr/bin/env bash
|
||||
# ============================================================
|
||||
# V2 API 全接口压测脚本
|
||||
# 测试所有12个 v2 端点的可用性、并发处理、Redis 状态正确性
|
||||
# ============================================================
|
||||
set -euo pipefail
|
||||
|
||||
HOST="${1:-http://localhost:8420}"
|
||||
API_KEY="${2:-b9WmawnJFpb9vn0XWKDKSxF5Eaf5SeXdIHaRpShmSbgg}"
|
||||
SERVICE_ID="${3:-mem-j4wjesud}"
|
||||
CONCURRENCY="${4:-10}"
|
||||
ROUNDS="${5:-5}"
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m'
|
||||
|
||||
PASS=0
|
||||
FAIL=0
|
||||
ERRORS=""
|
||||
|
||||
log_ok() { echo -e "${GREEN}[PASS]${NC} $1"; PASS=$((PASS+1)); }
|
||||
log_fail() { echo -e "${RED}[FAIL]${NC} $1"; FAIL=$((FAIL+1)); ERRORS="${ERRORS}\n - $1"; }
|
||||
log_info() { echo -e "${YELLOW}[INFO]${NC} $1"; }
|
||||
|
||||
call_api() {
|
||||
local endpoint="$1"
|
||||
local body="$2"
|
||||
local desc="$3"
|
||||
|
||||
local resp
|
||||
resp=$(curl -s -w "\n%{http_code}" -X POST "${HOST}${endpoint}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${API_KEY}" \
|
||||
-H "x-tdai-service-id: ${SERVICE_ID}" \
|
||||
-d "${body}" 2>/dev/null)
|
||||
|
||||
local http_code
|
||||
http_code=$(echo "$resp" | tail -1)
|
||||
local body_resp
|
||||
body_resp=$(echo "$resp" | sed '$d')
|
||||
|
||||
local code
|
||||
code=$(echo "$body_resp" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('code',''))" 2>/dev/null || echo "parse_error")
|
||||
|
||||
if [[ "$http_code" == "200" && "$code" == "0" ]]; then
|
||||
log_ok "$desc (HTTP=$http_code, code=$code)"
|
||||
echo "$body_resp"
|
||||
return 0
|
||||
else
|
||||
log_fail "$desc (HTTP=$http_code, code=$code, body=${body_resp:0:200})"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ============================================================
|
||||
echo "=========================================="
|
||||
echo " V2 API 全接口压测"
|
||||
echo " Host: $HOST"
|
||||
echo " ServiceID: $SERVICE_ID"
|
||||
echo " Concurrency: $CONCURRENCY"
|
||||
echo " Rounds: $ROUNDS"
|
||||
echo "=========================================="
|
||||
|
||||
# 0. Health check
|
||||
log_info "Step 0: Health check"
|
||||
HEALTH=$(curl -s "${HOST}/health" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['status'])" 2>/dev/null)
|
||||
if [[ "$HEALTH" == "ok" ]]; then
|
||||
log_ok "Health check"
|
||||
else
|
||||
log_fail "Health check (status=$HEALTH)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# 1. Conversation Add (L0 写入) - 并发压测
|
||||
# ============================================================
|
||||
log_info "Step 1: POST /v2/conversation/add — $CONCURRENCY concurrent × $ROUNDS rounds"
|
||||
TS=$(date +%s)
|
||||
|
||||
for round in $(seq 1 $ROUNDS); do
|
||||
pids=()
|
||||
for i in $(seq 1 $CONCURRENCY); do
|
||||
SESSION="stress-${TS}-r${round}-c${i}"
|
||||
BODY=$(cat <<EOF
|
||||
{
|
||||
"instance_id": "${SERVICE_ID}",
|
||||
"session_id": "${SESSION}",
|
||||
"messages": [
|
||||
{"role": "user", "content": "压测消息 round=${round} thread=${i} 我喜欢在周末去公园跑步"},
|
||||
{"role": "assistant", "content": "跑步是很好的运动习惯！你通常跑多久？"},
|
||||
{"role": "user", "content": "一般跑30分钟左右，然后做拉伸"},
|
||||
{"role": "assistant", "content": "30分钟加拉伸是很健康的安排"}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
)
|
||||
curl -s -o /dev/null -w "" -X POST "${HOST}/v2/conversation/add" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${API_KEY}" \
|
||||
-H "x-tdai-service-id: ${SERVICE_ID}" \
|
||||
-d "${BODY}" &
|
||||
pids+=($!)
|
||||
done
|
||||
|
||||
# Wait for all concurrent requests
|
||||
fail_count=0
|
||||
for pid in "${pids[@]}"; do
|
||||
wait "$pid" || fail_count=$((fail_count+1))
|
||||
done
|
||||
|
||||
if [[ $fail_count -eq 0 ]]; then
|
||||
log_ok "Round $round: $CONCURRENCY concurrent conversation/add OK"
|
||||
else
|
||||
log_fail "Round $round: $fail_count/$CONCURRENCY conversation/add failed"
|
||||
fi
|
||||
done
|
||||
|
||||
# ============================================================
|
||||
# 2. Conversation Query (L0 查询)
|
||||
# ============================================================
|
||||
log_info "Step 2: POST /v2/conversation/query"
|
||||
SESSION_Q="stress-${TS}-r1-c1"
|
||||
call_api "/v2/conversation/query" \
|
||||
"{\"instance_id\":\"${SERVICE_ID}\",\"session_id\":\"${SESSION_Q}\",\"limit\":10}" \
|
||||
"conversation/query session=${SESSION_Q}" > /dev/null || true
|
||||
|
||||
# ============================================================
|
||||
# 3. Conversation Search (L0 搜索)
|
||||
# ============================================================
|
||||
log_info "Step 3: POST /v2/conversation/search"
|
||||
call_api "/v2/conversation/search" \
|
||||
"{\"instance_id\":\"${SERVICE_ID}\",\"query\":\"跑步\",\"limit\":5}" \
|
||||
"conversation/search query=跑步" > /dev/null || true
|
||||
|
||||
# ============================================================
|
||||
# 4. Conversation Delete (L0 删除)
|
||||
# ============================================================
|
||||
log_info "Step 4: POST /v2/conversation/delete"
|
||||
call_api "/v2/conversation/delete" \
|
||||
"{\"instance_id\":\"${SERVICE_ID}\",\"session_id\":\"stress-${TS}-r${ROUNDS}-c${CONCURRENCY}\",\"ids\":[\"nonexistent-id\"]}" \
|
||||
"conversation/delete (nonexistent OK)" > /dev/null || true
|
||||
|
||||
# ============================================================
|
||||
# 5. Atomic Add (L1 写入)
|
||||
# ============================================================
|
||||
log_info "Step 5: POST /v2/atomic/add — concurrent"
|
||||
pids=()
|
||||
for i in $(seq 1 $CONCURRENCY); do
|
||||
BODY=$(cat <<EOF
|
||||
{
|
||||
"instance_id": "${SERVICE_ID}",
|
||||
"session_id": "stress-atomic-${TS}",
|
||||
"memories": [
|
||||
{"text": "用户喜欢在周末跑步 (thread=$i)", "type": "episodic", "priority": 0.8},
|
||||
{"text": "用户每次跑30分钟 (thread=$i)", "type": "episodic", "priority": 0.6}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
)
|
||||
curl -s -o /dev/null -w "" -X POST "${HOST}/v2/atomic/add" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${API_KEY}" \
|
||||
-H "x-tdai-service-id: ${SERVICE_ID}" \
|
||||
-d "${BODY}" &
|
||||
pids+=($!)
|
||||
done
|
||||
fail_count=0
|
||||
for pid in "${pids[@]}"; do
|
||||
wait "$pid" || fail_count=$((fail_count+1))
|
||||
done
|
||||
if [[ $fail_count -eq 0 ]]; then
|
||||
log_ok "atomic/add: $CONCURRENCY concurrent OK"
|
||||
else
|
||||
log_fail "atomic/add: $fail_count/$CONCURRENCY failed"
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# 6. Atomic Query (L1 查询)
|
||||
# ============================================================
|
||||
log_info "Step 6: POST /v2/atomic/query"
|
||||
call_api "/v2/atomic/query" \
|
||||
"{\"instance_id\":\"${SERVICE_ID}\",\"session_id\":\"stress-atomic-${TS}\",\"limit\":20}" \
|
||||
"atomic/query" > /dev/null || true
|
||||
|
||||
# ============================================================
|
||||
# 7. Atomic Search (L1 搜索)
|
||||
# ============================================================
|
||||
log_info "Step 7: POST /v2/atomic/search"
|
||||
call_api "/v2/atomic/search" \
|
||||
"{\"instance_id\":\"${SERVICE_ID}\",\"query\":\"跑步\",\"limit\":5}" \
|
||||
"atomic/search query=跑步" > /dev/null || true
|
||||
|
||||
# ============================================================
|
||||
# 8. Atomic Delete (L1 删除)
|
||||
# ============================================================
|
||||
log_info "Step 8: POST /v2/atomic/delete"
|
||||
call_api "/v2/atomic/delete" \
|
||||
"{\"instance_id\":\"${SERVICE_ID}\",\"ids\":[\"nonexistent-l1-id\"]}" \
|
||||
"atomic/delete (nonexistent OK)" > /dev/null || true
|
||||
|
||||
# ============================================================
|
||||
# 9. Scenario List (L2 列表)
|
||||
# ============================================================
|
||||
log_info "Step 9: POST /v2/scenario/ls"
|
||||
call_api "/v2/scenario/ls" \
|
||||
"{\"instance_id\":\"${SERVICE_ID}\"}" \
|
||||
"scenario/ls" > /dev/null || true
|
||||
|
||||
# ============================================================
|
||||
# 10. Scenario Read (L2 读取)
|
||||
# ============================================================
|
||||
log_info "Step 10: POST /v2/scenario/read"
|
||||
call_api "/v2/scenario/read" \
|
||||
"{\"instance_id\":\"${SERVICE_ID}\",\"path\":\"failover-test.md\"}" \
|
||||
"scenario/read path=failover-test.md" > /dev/null || true
|
||||
|
||||
# ============================================================
|
||||
# 11. Scenario Write (L2 写入)
|
||||
# ============================================================
|
||||
log_info "Step 11: POST /v2/scenario/write"
|
||||
call_api "/v2/scenario/write" \
|
||||
"{\"instance_id\":\"${SERVICE_ID}\",\"path\":\"stress-test-${TS}.md\",\"content\":\"# 压测场景\\n\\n这是压测写入的场景文件。\\n\\n## 用户偏好\\n- 周末跑步\"}" \
|
||||
"scenario/write" > /dev/null || true
|
||||
|
||||
# ============================================================
|
||||
# 12. Scenario Delete (L2 删除)
|
||||
# ============================================================
|
||||
log_info "Step 12: POST /v2/scenario/rm"
|
||||
call_api "/v2/scenario/rm" \
|
||||
"{\"instance_id\":\"${SERVICE_ID}\",\"path\":\"stress-test-${TS}.md\"}" \
|
||||
"scenario/rm (cleanup)" > /dev/null || true
|
||||
|
||||
# ============================================================
|
||||
# 13. Persona Read (L3 读取)
|
||||
# ============================================================
|
||||
log_info "Step 13: POST /v2/persona/read"
|
||||
call_api "/v2/persona/read" \
|
||||
"{\"instance_id\":\"${SERVICE_ID}\"}" \
|
||||
"persona/read" > /dev/null || true
|
||||
|
||||
# ============================================================
|
||||
# 14. Persona Write (L3 写入)
|
||||
# ============================================================
|
||||
log_info "Step 14: POST /v2/persona/write"
|
||||
call_api "/v2/persona/write" \
|
||||
"{\"instance_id\":\"${SERVICE_ID}\",\"content\":\"# 用户画像\\n\\n## 基本信息\\n- 喜欢跑步\\n- 周末锻炼\\n\"}" \
|
||||
"persona/write" > /dev/null || true
|
||||
|
||||
# ============================================================
|
||||
# 15. 高并发混合读写压测
|
||||
# ============================================================
|
||||
log_info "Step 15: Mixed concurrent read/write stress ($((CONCURRENCY*2)) requests)"
|
||||
pids=()
|
||||
for i in $(seq 1 $CONCURRENCY); do
|
||||
# Write
|
||||
curl -s -o /dev/null -X POST "${HOST}/v2/conversation/add" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${API_KEY}" \
|
||||
-H "x-tdai-service-id: ${SERVICE_ID}" \
|
||||
-d "{\"instance_id\":\"${SERVICE_ID}\",\"session_id\":\"mix-${TS}-${i}\",\"messages\":[{\"role\":\"user\",\"content\":\"混合压测消息 $i\"}]}" &
|
||||
pids+=($!)
|
||||
# Read
|
||||
curl -s -o /dev/null -X POST "${HOST}/v2/conversation/query" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${API_KEY}" \
|
||||
-H "x-tdai-service-id: ${SERVICE_ID}" \
|
||||
-d "{\"instance_id\":\"${SERVICE_ID}\",\"session_id\":\"mix-${TS}-${i}\",\"limit\":5}" &
|
||||
pids+=($!)
|
||||
done
|
||||
fail_count=0
|
||||
for pid in "${pids[@]}"; do
|
||||
wait "$pid" || fail_count=$((fail_count+1))
|
||||
done
|
||||
if [[ $fail_count -eq 0 ]]; then
|
||||
log_ok "Mixed stress: $((CONCURRENCY*2)) concurrent OK"
|
||||
else
|
||||
log_fail "Mixed stress: $fail_count/$((CONCURRENCY*2)) failed"
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# 16. Redis 状态校验
|
||||
# ============================================================
|
||||
log_info "Step 16: Health check + Redis state validation"
|
||||
HEALTH_RESP=$(curl -s "${HOST}/health")
|
||||
WORKER_TASKS=$(echo "$HEALTH_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); s=d['services']; print(f\"consumed={s['pipelineWorker']['tasksConsumed']} completed={s['pipelineWorker']['tasksCompleted']} failed={s['pipelineWorker']['tasksFailed']} deadLetter={s['pipelineWorker']['tasksDeadLettered']}\")" 2>/dev/null)
|
||||
SCANNER=$(echo "$HEALTH_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); s=d['services']['timerScanner']; print(f\"scans={s['scansCompleted']} enqueued={s['tasksEnqueued']} errors={s['scanErrors']}\")" 2>/dev/null)
|
||||
STATE=$(echo "$HEALTH_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['services']['stateBackend'])" 2>/dev/null)
|
||||
|
||||
echo " Worker: $WORKER_TASKS"
|
||||
echo " Scanner: $SCANNER"
|
||||
echo " Redis: $STATE"
|
||||
|
||||
if [[ "$STATE" == "connected" ]]; then
|
||||
log_ok "Redis state: connected"
|
||||
else
|
||||
log_fail "Redis state: $STATE"
|
||||
fi
|
||||
|
||||
TASK_FAILED=$(echo "$HEALTH_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['services']['pipelineWorker']['tasksFailed'])" 2>/dev/null)
|
||||
if [[ "$TASK_FAILED" == "0" ]]; then
|
||||
log_ok "No pipeline task failures"
|
||||
else
|
||||
log_fail "Pipeline task failures: $TASK_FAILED"
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# Summary
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo " 压测结果汇总"
|
||||
echo "=========================================="
|
||||
echo -e " ${GREEN}PASS: $PASS${NC}"
|
||||
echo -e " ${RED}FAIL: $FAIL${NC}"
|
||||
if [[ $FAIL -gt 0 ]]; then
|
||||
echo -e " 失败详情:${ERRORS}"
|
||||
fi
|
||||
echo "=========================================="
|
||||
TOTAL_MSGS=$((CONCURRENCY * ROUNDS * 4 + CONCURRENCY * 1))
|
||||
echo " 总写入消息数: ~$TOTAL_MSGS (L0) + $((CONCURRENCY*2)) (L1)"
|
||||
echo "=========================================="
|
||||
|
||||
exit $FAIL
|
||||
Executable
+233
@@ -0,0 +1,233 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# sync-hermes-llm-to-tdai.sh — 读取系统 Hermes 模型配置，同步到 TDAI Gateway
|
||||
#
|
||||
# 前置条件：
|
||||
# 1. memory-tencentdb-ctl.sh 存在
|
||||
# 2. ~/.hermes/config.yaml 存在且包含模型配置
|
||||
#
|
||||
# 用法：
|
||||
# bash sync-hermes-llm-to-tdai.sh [--restart] [--dry-run] [--hermes]
|
||||
#
|
||||
# 参数语义：
|
||||
# --restart 仅当 CTL 脚本的 `status` 显示 Gateway 当前为 RUNNING 时，
|
||||
# 才透传给 `memory-tencentdb-ctl.sh config llm` 执行重启；
|
||||
# 若当前未运行，则本脚本只同步配置，不拉起新进程。
|
||||
# --hermes 以 hermes 集成模式调用 CTL（影响 status 判定与配置落点）
|
||||
# --standalone 以 standalone 模式调用 CTL
|
||||
# --dry-run 透传给 CTL，预演写配置但不真正落盘
|
||||
#
|
||||
# 可选环境变量：
|
||||
# HERMES_HOME hermes 主目录（默认 ~/.hermes）
|
||||
# MEMORY_TENCENTDB_ROOT TDAI 统一根目录（默认 ~/.memory-tencentdb）
|
||||
#
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_NAME="sync-hermes-llm-to-tdai"
|
||||
USER_HOME="${HOME:-$(eval echo "~$(whoami)")}"
|
||||
|
||||
# ============================================================
|
||||
# 路径
|
||||
# ============================================================
|
||||
|
||||
HERMES_HOME="${HERMES_HOME:-$USER_HOME/.hermes}"
|
||||
HERMES_CONFIG="$HERMES_HOME/config.yaml"
|
||||
|
||||
MEMORY_TENCENTDB_ROOT="${MEMORY_TENCENTDB_ROOT:-$USER_HOME/.memory-tencentdb}"
|
||||
TDAI_INSTALL_DIR="${TDAI_INSTALL_DIR:-$MEMORY_TENCENTDB_ROOT/tdai-memory-openclaw-plugin}"
|
||||
|
||||
CTL_SCRIPT="$TDAI_INSTALL_DIR/scripts/memory-tencentdb-ctl.sh"
|
||||
|
||||
# ============================================================
|
||||
# 通用 helpers
|
||||
# ============================================================
|
||||
|
||||
log() { printf '[%s] %s\n' "$SCRIPT_NAME" "$*"; }
|
||||
warn() { printf '[%s:warn] %s\n' "$SCRIPT_NAME" "$*" >&2; }
|
||||
die() { printf '[%s:error] %s\n' "$SCRIPT_NAME" "$*" >&2; exit "${2:-1}"; }
|
||||
|
||||
should_restart_gateway() {
|
||||
local ctl_mode_args=()
|
||||
case "${SYNC_MODE:-standalone}" in
|
||||
hermes) ctl_mode_args+=("--hermes") ;;
|
||||
*) ctl_mode_args+=("--standalone") ;;
|
||||
esac
|
||||
|
||||
local status_out
|
||||
status_out="$(bash "$CTL_SCRIPT" "${ctl_mode_args[@]}" status 2>&1 || true)"
|
||||
|
||||
if printf '%s\n' "$status_out" | grep -Eq 'state[[:space:]]*:[[:space:]]*RUNNING'; then
|
||||
log "通过 ctl status 检测到 Gateway 正在运行，允许执行 --restart"
|
||||
return 0
|
||||
fi
|
||||
|
||||
warn "ctl status 未显示 Gateway 处于 RUNNING；本次仅同步配置，不执行 --restart，也不会拉起新进程。"
|
||||
return 1
|
||||
}
|
||||
|
||||
# ============================================================
|
||||
# Step 1: 检查 memory-tencentdb-ctl.sh 是否存在
|
||||
# ============================================================
|
||||
|
||||
if [[ ! -f "$CTL_SCRIPT" ]]; then
|
||||
# 也尝试旧路径
|
||||
_LEGACY_CTL="$USER_HOME/tdai-memory-openclaw-plugin/scripts/memory-tencentdb-ctl.sh"
|
||||
if [[ -f "$_LEGACY_CTL" ]]; then
|
||||
CTL_SCRIPT="$_LEGACY_CTL"
|
||||
warn "使用旧路径: $CTL_SCRIPT"
|
||||
else
|
||||
die "memory-tencentdb-ctl.sh 不存在，TDAI 未安装。退出。" 1
|
||||
fi
|
||||
fi
|
||||
|
||||
log "找到 ctl 脚本: $CTL_SCRIPT"
|
||||
|
||||
# ============================================================
|
||||
# Step 2: 检查 hermes config.yaml
|
||||
# ============================================================
|
||||
|
||||
if [[ ! -f "$HERMES_CONFIG" ]]; then
|
||||
die "Hermes 配置文件不存在: $HERMES_CONFIG" 1
|
||||
fi
|
||||
|
||||
log "读取 Hermes 配置: $HERMES_CONFIG"
|
||||
|
||||
# ============================================================
|
||||
# Step 3: 从 hermes config.yaml 提取模型配置
|
||||
# ============================================================
|
||||
|
||||
need_cmd() {
|
||||
command -v "$1" >/dev/null 2>&1 || die "必需命令未找到: $1" 127
|
||||
}
|
||||
|
||||
need_cmd python3
|
||||
|
||||
# 使用 python3 安全解析 YAML，提取 model 段的 default / base_url / api_key
|
||||
read_hermes_model_config() {
|
||||
python3 - "$HERMES_CONFIG" <<'PYEOF'
|
||||
import sys, json
|
||||
|
||||
config_path = sys.argv[1]
|
||||
|
||||
# 尝试使用 PyYAML；如果不存在则手动逐行解析
|
||||
try:
|
||||
import yaml
|
||||
with open(config_path, "r", encoding="utf-8") as f:
|
||||
cfg = yaml.safe_load(f) or {}
|
||||
model_section = cfg.get("model", {})
|
||||
result = {
|
||||
"model": model_section.get("default", ""),
|
||||
"base_url": model_section.get("base_url", ""),
|
||||
"api_key": model_section.get("api_key", ""),
|
||||
}
|
||||
print(json.dumps(result))
|
||||
except ImportError:
|
||||
# PyYAML 不可用，用简单逐行解析提取 model 段
|
||||
import re
|
||||
with open(config_path, "r", encoding="utf-8") as f:
|
||||
lines = f.readlines()
|
||||
|
||||
in_model = False
|
||||
result = {"model": "", "base_url": "", "api_key": ""}
|
||||
for line in lines:
|
||||
stripped = line.rstrip("\n")
|
||||
# 检测 model: 顶层段
|
||||
if re.match(r"^model:\s*$", stripped) or re.match(r"^model:\s*#", stripped):
|
||||
in_model = True
|
||||
continue
|
||||
# 进入新的顶层段则退出
|
||||
if in_model and re.match(r"^[A-Za-z_]", stripped):
|
||||
break
|
||||
if in_model:
|
||||
m = re.match(r"^\s+default:\s*(.+)", stripped)
|
||||
if m:
|
||||
result["model"] = m.group(1).strip().strip("'\"")
|
||||
m = re.match(r"^\s+base_url:\s*(.+)", stripped)
|
||||
if m:
|
||||
result["base_url"] = m.group(1).strip().strip("'\"")
|
||||
m = re.match(r"^\s+api_key:\s*(.+)", stripped)
|
||||
if m:
|
||||
result["api_key"] = m.group(1).strip().strip("'\"")
|
||||
print(json.dumps(result))
|
||||
PYEOF
|
||||
}
|
||||
|
||||
MODEL_JSON="$(read_hermes_model_config)"
|
||||
|
||||
HERMES_MODEL="$(printf '%s' "$MODEL_JSON" | python3 -c 'import json,sys; print(json.load(sys.stdin)["model"])')"
|
||||
HERMES_BASE_URL="$(printf '%s' "$MODEL_JSON" | python3 -c 'import json,sys; print(json.load(sys.stdin)["base_url"])')"
|
||||
HERMES_API_KEY="$(printf '%s' "$MODEL_JSON" | python3 -c 'import json,sys; print(json.load(sys.stdin)["api_key"])')"
|
||||
|
||||
# 校验必需字段
|
||||
if [[ -z "$HERMES_MODEL" ]]; then
|
||||
die "Hermes 配置中 model.default 为空" 1
|
||||
fi
|
||||
if [[ -z "$HERMES_BASE_URL" ]]; then
|
||||
die "Hermes 配置中 model.base_url 为空" 1
|
||||
fi
|
||||
if [[ -z "$HERMES_API_KEY" ]]; then
|
||||
die "Hermes 配置中 model.api_key 为空" 1
|
||||
fi
|
||||
|
||||
log "Hermes 模型配置:"
|
||||
log " model = $HERMES_MODEL"
|
||||
log " base_url = $HERMES_BASE_URL"
|
||||
log " api_key = <${#HERMES_API_KEY} chars>"
|
||||
|
||||
# ============================================================
|
||||
# Step 4: 透传参数（--restart / --dry-run / --hermes）
|
||||
# ============================================================
|
||||
|
||||
CTL_EXTRA_ARGS=()
|
||||
SYNC_MODE="standalone"
|
||||
REQUEST_RESTART=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--restart)
|
||||
REQUEST_RESTART=1
|
||||
;;
|
||||
--hermes)
|
||||
SYNC_MODE="hermes"
|
||||
CTL_EXTRA_ARGS+=("$arg")
|
||||
;;
|
||||
--standalone)
|
||||
SYNC_MODE="standalone"
|
||||
CTL_EXTRA_ARGS+=("$arg")
|
||||
;;
|
||||
--dry-run)
|
||||
CTL_EXTRA_ARGS+=("$arg")
|
||||
;;
|
||||
*)
|
||||
warn "忽略未知参数: $arg"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ $REQUEST_RESTART -eq 1 ]]; then
|
||||
# 新语义：--restart 只在 Gateway 当前已运行时才生效。
|
||||
# 这里复用 CTL 的 status 输出作为唯一事实来源，避免本脚本
|
||||
# 自己维护另一套“端口监听/健康检查”判定逻辑。
|
||||
if should_restart_gateway; then
|
||||
CTL_EXTRA_ARGS+=("--restart")
|
||||
else
|
||||
log "已按新逻辑跳过 --restart。"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# Step 5: 调用 memory-tencentdb-ctl.sh config llm 同步配置
|
||||
# ============================================================
|
||||
|
||||
log "调用 memory-tencentdb-ctl.sh config llm 同步配置到 TDAI ..."
|
||||
|
||||
bash "$CTL_SCRIPT" "${CTL_EXTRA_ARGS[@]+"${CTL_EXTRA_ARGS[@]}"}" \
|
||||
config llm \
|
||||
--api-key "$HERMES_API_KEY" \
|
||||
--base-url "$HERMES_BASE_URL" \
|
||||
--model "$HERMES_MODEL"
|
||||
|
||||
log "同步完成。"
|
||||
log ""
|
||||
log "如需立即重启 Gateway 使配置生效，请重新运行并追加 --restart："
|
||||
log " bash $0 --restart"
|
||||
@@ -0,0 +1,61 @@
|
||||
#!/bin/bash
|
||||
# ============================================================
|
||||
# 验证 1: 冷启动循环 Bug 复现
|
||||
#
|
||||
# 条件: 手动构造 checkpoint 状态 → last_persona_at=0,
|
||||
# 但 persona.md 存在 + scene_blocks 存在
|
||||
# 预期: L3 反复触发但每次都跳过
|
||||
# ============================================================
|
||||
|
||||
set -e
|
||||
|
||||
EP="https://tdai.apigateway.cd.test.polaris"
|
||||
AUTH="Authorization: Bearer DQfp9PnHn+iKwON8+ipBfOCXx1ISlfXxSWWENu095ZIp"
|
||||
SID="X-TDAI-Service-ID: mem-rkgqhd5z"
|
||||
CT="Content-Type: application/json"
|
||||
SESSION="coldstart-verify-$(date +%s)"
|
||||
|
||||
echo "============================================================"
|
||||
echo "验证 1: 冷启动循环复现"
|
||||
echo "Session: $SESSION"
|
||||
echo "============================================================"
|
||||
echo
|
||||
|
||||
# Step 1: 写入足够多的对话触发 L1 → L2 → L3 pipeline
|
||||
echo "--- Step 1: 写入对话数据触发 pipeline ---"
|
||||
for i in $(seq 1 10); do
|
||||
curl -sk -X POST "$EP/v2/conversation/add" -H "$CT" -H "$AUTH" -H "$SID" \
|
||||
-d "{\"session_id\":\"$SESSION\",\"messages\":[{\"role\":\"user\",\"content\":\"第${i}轮对话：测试冷启动循环\",\"timestamp\":\"2026-05-18T11:${i}:00Z\"},{\"role\":\"assistant\",\"content\":\"收到第${i}轮\",\"timestamp\":\"2026-05-18T11:${i}:05Z\"}]}" > /dev/null
|
||||
echo " 写入第 $i 轮对话"
|
||||
done
|
||||
echo
|
||||
|
||||
# Step 2: 写入 persona 和 scenario (模拟已有数据)
|
||||
echo "--- Step 2: 写入 persona + scenario ---"
|
||||
curl -sk -X POST "$EP/v2/persona/write" -H "$CT" -H "$AUTH" -H "$SID" \
|
||||
-d '{"content": "# Persona\n\n这是测试persona，模拟已存在的状态。"}'
|
||||
echo
|
||||
curl -sk -X POST "$EP/v2/scenario/write" -H "$CT" -H "$AUTH" -H "$SID" \
|
||||
-d '{"path": "test-coldstart.md", "content": "# Test Scene\n\n场景文件存在。"}'
|
||||
echo
|
||||
echo
|
||||
|
||||
# Step 3: 等待 pipeline 触发
|
||||
echo "--- Step 3: 等待 60s 让 pipeline 执行 ---"
|
||||
echo " (观察容器日志看是否出现 'Trigger P2 (cold start)' 循环)"
|
||||
echo " 在另一个终端执行:"
|
||||
echo " sudo docker logs -f memory-test-fixed 2>&1 | grep -E 'Trigger P2|cold start|Persona generation skipped|last_persona_at'"
|
||||
echo
|
||||
sleep 5
|
||||
|
||||
# Step 4: 检查结果
|
||||
echo "--- Step 4: 读取 persona 验证 ---"
|
||||
curl -sk -X POST "$EP/v2/persona/read" -H "$CT" -H "$AUTH" -H "$SID" -d '{}'
|
||||
echo
|
||||
echo
|
||||
echo "============================================================"
|
||||
echo "如果日志中反复出现:"
|
||||
echo " [trigger] Trigger P2 (cold start): scenes_processed=N, total_processed=0"
|
||||
echo " [L3] Persona generation skipped (no changes)"
|
||||
echo "则 Bug 复现成功。"
|
||||
echo "============================================================"
|
||||
@@ -0,0 +1,148 @@
|
||||
#!/bin/bash
|
||||
# 常温测试脚本：每30秒通过LLM生成对话写入，持续验证 L0→L1→L3 有效性
|
||||
# Usage: bash warm-test-loop.sh [rounds] (default: infinite)
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
APIG_URL="${APIG_URL:-https://tdai.apigateway.cd.test.polaris/v2}"
|
||||
INSTANCE="${INSTANCE:-mem-0294jqv7}"
|
||||
API_KEY="${API_KEY:?Please set API_KEY env var}"
|
||||
LLM_URL="${LLM_URL:-https://tokenhub.tencentmaas.com/v1/chat/completions}"
|
||||
LLM_KEY="${LLM_KEY:?Please set LLM_KEY env var}"
|
||||
LLM_MODEL="${LLM_MODEL:-minimax-m2.7}"
|
||||
INTERVAL=30
|
||||
MAX_ROUNDS=${1:-0} # 0 = infinite
|
||||
|
||||
ROUND=0
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
echo "=========================================="
|
||||
echo " 常温测试 - 每${INTERVAL}秒一轮 (LLM生成对话)"
|
||||
echo " APIG: ${APIG_URL}"
|
||||
echo " Instance: ${INSTANCE}"
|
||||
echo " Started: $(date '+%Y-%m-%d %H:%M:%S')"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
cleanup() {
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo " 测试结束 ($(date '+%H:%M:%S'))"
|
||||
echo " 总轮次: ${ROUND}"
|
||||
echo " PASS: ${PASS}"
|
||||
echo " FAIL: ${FAIL}"
|
||||
echo "=========================================="
|
||||
exit 0
|
||||
}
|
||||
trap cleanup SIGINT SIGTERM
|
||||
|
||||
generate_conversation() {
|
||||
local round=$1
|
||||
# 用LLM生成随机对话（3轮user/assistant）
|
||||
local prompt="请生成一段模拟用户和AI助手的对话（3轮，共6条消息）。用户在自我介绍中包含：姓名、年龄、职业、工作地点、技术栈或专业领域、业余爱好。每轮对话要有具体细节（数字、地名、品牌等）。第${round}次生成请确保内容独特不重复。
|
||||
|
||||
严格按以下JSON格式输出，不要输出任何其他内容：
|
||||
[{\"role\":\"user\",\"content\":\"...\"},{\"role\":\"assistant\",\"content\":\"...\"},{\"role\":\"user\",\"content\":\"...\"},{\"role\":\"assistant\",\"content\":\"...\"},{\"role\":\"user\",\"content\":\"...\"},{\"role\":\"assistant\",\"content\":\"...\"}]"
|
||||
|
||||
local resp
|
||||
resp=$(curl -s --connect-timeout 15 --max-time 30 "${LLM_URL}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${LLM_KEY}" \
|
||||
-d "{\"model\":\"${LLM_MODEL}\",\"messages\":[{\"role\":\"user\",\"content\":$(echo "$prompt" | python3 -c "import sys,json; print(json.dumps(sys.stdin.read()))")}],\"max_tokens\":800,\"temperature\":0.9}" 2>/dev/null)
|
||||
|
||||
# 提取content并解析JSON array
|
||||
echo "$resp" | python3 -c "
|
||||
import sys,json,re
|
||||
try:
|
||||
d=json.load(sys.stdin)
|
||||
content=d['choices'][0]['message']['content']
|
||||
# 提取JSON数组
|
||||
match=re.search(r'\[[\s\S]*\]', content)
|
||||
if match:
|
||||
msgs=json.loads(match.group())
|
||||
if len(msgs)>=6:
|
||||
print(json.dumps(msgs[:6], ensure_ascii=False))
|
||||
else:
|
||||
print('ERROR:not_enough_msgs')
|
||||
else:
|
||||
print('ERROR:no_json_array')
|
||||
except Exception as e:
|
||||
print(f'ERROR:{e}')
|
||||
" 2>/dev/null
|
||||
}
|
||||
|
||||
while true; do
|
||||
ROUND=$((ROUND + 1))
|
||||
if [ "${MAX_ROUNDS}" -gt 0 ] && [ "${ROUND}" -gt "${MAX_ROUNDS}" ]; then
|
||||
break
|
||||
fi
|
||||
|
||||
TS=$(date +%s)
|
||||
SESSION="warm-${TS}-r${ROUND}"
|
||||
|
||||
echo "[Round ${ROUND}] $(date '+%H:%M:%S') session=${SESSION}"
|
||||
|
||||
# 1. LLM 生成对话
|
||||
echo " 生成对话..."
|
||||
MESSAGES=$(generate_conversation ${ROUND})
|
||||
|
||||
if [[ "$MESSAGES" == ERROR:* ]]; then
|
||||
echo " ❌ LLM生成失败: ${MESSAGES}"
|
||||
FAIL=$((FAIL + 1))
|
||||
sleep ${INTERVAL}
|
||||
continue
|
||||
fi
|
||||
|
||||
# 2. 写入 L0
|
||||
RESP=$(curl -sk -X POST "${APIG_URL}/conversation/add" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${API_KEY}" \
|
||||
-H "X-TDAI-Service-ID: ${INSTANCE}" \
|
||||
-d "{\"session_id\":\"${SESSION}\",\"messages\":${MESSAGES}}" 2>/dev/null)
|
||||
|
||||
L0_CODE=$(echo "$RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('code','err'))" 2>/dev/null || echo "parse_err")
|
||||
L0_COUNT=$(echo "$RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('data',{}).get('accepted_ids',[])))" 2>/dev/null || echo "0")
|
||||
|
||||
if [ "$L0_CODE" = "0" ]; then
|
||||
echo " ✓ L0写入: ${L0_COUNT}条"
|
||||
else
|
||||
echo " ❌ L0写入失败: code=${L0_CODE}"
|
||||
FAIL=$((FAIL + 1))
|
||||
sleep ${INTERVAL}
|
||||
continue
|
||||
fi
|
||||
|
||||
# 3. 验证 L1 total 递增
|
||||
L1_BEFORE=$(curl -sk -X POST "${APIG_URL}/atomic/query" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${API_KEY}" \
|
||||
-H "X-TDAI-Service-ID: ${INSTANCE}" \
|
||||
-d "{\"limit\":1}" 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('data',{}).get('total',0))" 2>/dev/null || echo "0")
|
||||
echo " L1 total(before): ${L1_BEFORE}"
|
||||
|
||||
# 4. 验证 Persona 存在
|
||||
PERSONA_LEN=$(curl -sk -X POST "${APIG_URL}/persona/read" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${API_KEY}" \
|
||||
-H "X-TDAI-Service-ID: ${INSTANCE}" \
|
||||
-d '{}' 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('data',{}).get('content','')))" 2>/dev/null || echo "0")
|
||||
echo " L3 Persona长度: ${PERSONA_LEN}"
|
||||
|
||||
# 5. Health check
|
||||
HEALTH=$(curl -sk "${APIG_URL}/../health" 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('status','?'))" 2>/dev/null || echo "unknown")
|
||||
|
||||
if [ "$L0_CODE" = "0" ] && [ "$L0_COUNT" -ge 4 ]; then
|
||||
PASS=$((PASS + 1))
|
||||
echo " ✓ PASS (L0=${L0_COUNT}, L1_total=${L1_BEFORE}, Persona=${PERSONA_LEN})"
|
||||
else
|
||||
FAIL=$((FAIL + 1))
|
||||
echo " ❌ FAIL"
|
||||
fi
|
||||
|
||||
echo " [累计] PASS=${PASS} FAIL=${FAIL} | 等待${INTERVAL}s..."
|
||||
echo ""
|
||||
sleep ${INTERVAL}
|
||||
done
|
||||
|
||||
cleanup
|
||||
Reference in New Issue
Block a user