基线长什么样
先说系统本身，因为脱离系统谈优化没有意义。
大约 1GB 的企业资料 —— 产品手册、规格表、报价与服务政策 —— 存在 S3。用 LangChain 加载并切块，OpenAI 的嵌入模型向量化，存进跑在 Docker 里的 ChromaDB。销售提问时，同一个模型嵌入问题，在 ChromaDB 里检索最相似的 5 个片段，拼成上下文交给 LLM 作答。
整条链路只有三次外部 API 调用：两次嵌入，一次生成。其余全部本地，没有额外成本。端到端 2 到 3 秒。
第一版上线时，回答质量大概在 80%。这个数字听起来还行，但对销售来说意味着每五个问题错一个 —— 而一个在客户面前被当场戳穿的错误，足以让他再也不打开这个工具。
动作一：按章节切块，不按长度
最初用的是固定长度切块，500 token。问题很快暴露：答案在片段里，但支撑答案的上下文被切到了隔壁。
我先把长度加到 800 token、重叠 100 token，检索质量确实变好了。但再往上加就开始反噬 —— 片段太大，一个块里混了好几个主题，向量相似度变得不精确。
真正的解法是换切法：按文档的自然章节切。这些 SOP 和手册本来就有清晰的结构 —— 「初步响应」「升级条件」「处置步骤」。按这个结构切，一条完整流程就不会被拦腰截断。每个块还带上元数据：文档 ID、场景类型、严重级别、章节类型。
动作二：先过滤，再检索
纯向量检索有个天然弱点：它只认语义相似，不认业务约束。问「门打不开怎么办」，可能召回一段语义很像但属于另一个场景的处置流程。
加了元数据过滤之后，如果系统已经判断出场景类型，就先把候选集缩小到这个场景，再在里面做向量搜索。检索空间小了一个数量级，精度提升非常明显。
向量检索负责「像不像」，元数据过滤负责「该不该」。只用前者，迟早会召回一段语义完美但业务上完全不适用的内容。
动作三：让模型闭嘴
提示词里明确要求：只依据提供的上下文作答。上下文里没有的，不要推断、不要补全、不要「合理猜测」—— 直接说明资料未覆盖，并转人工。
这个改动降低了准确率的分子吗？短期看是的，有些原本蒙对的回答变成了「我不知道」。但对销售工具来说，「我不知道」是一个诚实的、可用的回答；一个自信的错误答案不是。信任一旦掉了，准确率再高也救不回来。
动作四：让失败可以定位
这是四个动作里最有用的一个，而它根本不是模型技术。
我们给每一次问答记录三样东西：问题、召回的片段、最终答案。销售可以一键标记「这个答案不对」。
然后排查规则只有一条 —— 先看召回：
- 如果正确的片段召回了，但答案还是错的 → 这是生成问题。改提示词，当天就能修复并验证。
- 如果正确的片段根本没召回 → 这是切块或索引问题。要调整切块策略并从 S3 重建索引。慢，但是根治。
在有这条规则之前，「答案不对」是一个没法行动的反馈，团队只能凭感觉调。有了之后，每一个失败案例都会被分到一个明确的修复路径上。几周时间，准确率从 80% 走到 90%。
怎么衡量
「90%」这个数字必须有口径，否则它就是个感觉。我们用了三个层次：
- 离线黄金集 —— 约 200 对场景与标准回答，每次改动后跑一遍对比。
- 召回精度 —— precision@k，单独衡量检索层，把它和生成层的问题分开看。
- 线上代理指标 —— 转人工率。这是最诚实的一个：它不关心我们怎么定义正确，只反映用户实际有没有被满足。
没做的事
也值得说说我们没有做什么。没有微调模型 —— 在这个问题上，微调解决不了检索召回错误。没有换更大的模型 —— 80% 到 90% 的差距里，几乎没有一点是模型能力不足造成的。
带走的规则
RAG 出问题时，先分清是召回错了还是生成错了。这两类问题的修复成本差一个数量级，而且修法完全不同。分不清就调，是在用最贵的方式碰运气。
What the baseline looked like
Start with the system itself, because optimisation talk means nothing without it.
About 1GB of company material — product manuals, spec sheets, pricing and service policy — sits in S3. LangChain loads and chunks it, OpenAI embeddings vectorise each chunk, and everything lands in ChromaDB running locally in Docker. When a rep asks a question, the same model embeds it, ChromaDB returns the five most similar chunks, and those become the context handed to the LLM.
Three external API calls in the whole path: two embeddings and one completion. Everything else is local and costs nothing extra. Two to three seconds end to end.
At launch, answer quality sat around 80%. That sounds tolerable, but to a salesperson it means one in five answers is wrong — and a single answer contradicted in front of a client is enough that they never open the tool again.
Move one: chunk by section, not by length
We started with fixed-size chunks at 500 tokens. The failure showed up quickly: the answer was in the chunk, but the context that justified it had been cut into the next one.
Raising it to 800 tokens with 100 tokens of overlap genuinely improved retrieval. Pushing further started to hurt — chunks grew large enough to cover several topics at once, and vector similarity got imprecise.
The real fix was changing the strategy: chunk on the document's natural sections. These SOPs and manuals already had clear structure — initial response, escalation criteria, resolution steps. Chunking on that structure means a complete procedure stops being cut in half. Each chunk also carries metadata: document ID, scenario type, severity, section type.
Move two: filter before you search
Pure vector retrieval has a built-in weakness: it understands semantic similarity, not business constraint. Ask "what do I do if the door won't open" and you can retrieve a procedure that reads almost identically but belongs to a different scenario entirely.
With metadata filtering, when the system already knows the scenario type it narrows the candidate set to that scenario first, then runs vector search inside it. The search space shrinks by an order of magnitude and precision improves sharply.
Vector search answers "is this similar". Metadata filtering answers "is this allowed". Use only the first and eventually you will retrieve something semantically perfect and operationally wrong.
Move three: tell the model to stop talking
The prompt states it explicitly: answer only from the provided context. If it isn't there, don't infer, don't fill in, don't make a reasonable guess — say the documentation doesn't cover it and escalate to a human.
Did that lower the numerator on accuracy? Short-term, yes — some answers that used to be lucky guesses became "I don't know". But for a sales tool, "I don't know" is an honest and usable answer. A confident wrong one is not. Once trust is gone, no accuracy number brings it back.
Move four: make failure locatable
This was the most useful of the four, and it isn't model work at all.
We logged three things for every query: the question, the chunks retrieved, and the answer. Reps could flag an answer as wrong in one click.
Then one rule for triage — look at retrieval first:
- If the right chunk came back and the answer is still wrong → generation problem. Fix the prompt, ship and verify the same day.
- If the right chunk never came back → chunking or indexing problem. Adjust the strategy and rebuild the index from S3. Slower, but it fixes the class of failure rather than the instance.
Before that rule, "the answer is wrong" was feedback nobody could act on, and the team tuned by intuition. After it, every failure was sorted onto a specific repair path. Over a few weeks accuracy moved from 80% to 90%.
How we measured it
"90%" needs a definition or it's just a feeling. We used three layers:
- An offline gold set — about 200 scenario-and-expected-answer pairs, re-run after every change.
- Retrieval precision — precision@k, measuring the retrieval layer on its own so it stays separable from generation.
- A production proxy — escalation rate. The most honest of the three: it doesn't care how we define correctness, only whether the user actually got what they needed.
What we didn't do
Worth naming the non-actions too. We didn't fine-tune — fine-tuning doesn't fix a retrieval miss. And we didn't reach for a bigger model: almost none of the gap between 80% and 90% was caused by model capability.
The rule I kept
When RAG misbehaves, first separate a retrieval failure from a generation failure. The two cost an order of magnitude apart to fix and the fixes are completely different. Tuning before you've separated them is an expensive way to guess.