xiaoshuai
a64ea2acb3
A durable memory layer for OpenCode. Structure:
ROOT.md total plan: goals, hard rules, directory registry,
and a scenario -> file routing table
<category>/
index.md one line per file, COMPILED from frontmatter
<slug>.md the memory itself
Two roots, same shape: <config>/memory/ (global) and
<project canonical>/.memory/ (project). Project wins on conflict; global
hard rules stay resident everywhere.
Why a plugin rather than a skill that tells the model to read a folder:
model-driven retrieval is unreliable — the model forgets, reads half the
files, or reads the wrong ones. Markdown is the human-facing format; the
plugin is the engine that parses, scores, and injects.
Measured, not assumed: FTS5's two built-in tokenizers both fail on this
corpus. On SQLite 3.53.4 with "部署脚本用 pnpm 而不是 npm...": unicode61
emits 7 terms for the whole sentence and matches 0 hits for "部署";
trigram needs >= 3 characters and also returns 0 for that two-character
word. So tokenization happens in the plugin (CJK bigrams, latin words)
and no SQL index is used — a thousand memories are fine to scan.
index.md and ROOT's two tables are compiled, never hand-edited: a
hand-maintained index drifts, a generated one cannot. Hit counters live
in .state.json so cross-device sync does not conflict on every recall.
Tools are named recall/remember/forget/inspect rather than
write/read/status: generic names were observed shadowing the harness's
own built-in tools.
24 lines
1.1 KiB
Markdown
24 lines
1.1 KiB
Markdown
---
|
||
id: pitfalls/sqlite-cjk-tokenizer.md
|
||
kind: fact
|
||
summary: unicode61 把整段中文当一个 token，trigram 漏掉两字词，必须自己分词。
|
||
highlight: 实测：unicode61 查"部署"=0 命中；trigram 查"部署"=0 命中（需≥3字符）
|
||
created: 1791512161994
|
||
updated: 1791512161994
|
||
load_when:
|
||
- SQLite
|
||
- FTS5
|
||
- 全文检索
|
||
- 分词
|
||
- 中文检索
|
||
---
|
||
|
||
在 SQLite 3.53.4 上实测，字符串「部署脚本用 pnpm 而不是 npm，因为 lockfile 不兼容」：
|
||
|
||
- `unicode61`（FTS5 默认）把 `部署脚本用` 当作**一个 token**，整句只产生 7 个词条。查「部署」0 命中，查「部署脚本」也 0 命中。子串永远匹配不上。
|
||
- `trigram` 要求查询 ≥3 字符。词表是「不兼」「而不」「m，因」这种三字滑窗。查「部署」0 命中，查「部署脚」1 命中。
|
||
|
||
中文里「部署」「配置」「依赖」「接口」这类**两字词占比极高**，trigram 漏掉的正好是最常用的一批。
|
||
|
||
解法：索引前自己切 CJK 字符二元组，拉丁走 porter 词干，FTS5 只当倒排索引用。
|