语音系统没有加载动画
做过 Web 产品的人对延迟有一种错觉：慢一点没关系，转个圈就好。语音系统没有这个选项。模型慢一秒，对面听到的就是一秒沉默 —— 而在一部卡住的电梯里，沉默的含义是「没人管我」。
所以在这个项目里，延迟不是性能指标，是产品指标。第一版端到端要 5.5 到 7 秒才发出第一个音节。这个数字在内部演示里没人在意，在真实通话里是灾难。
先把账算清楚
最朴素的实现是一条串行流水线，每一步等上一步做完：
- 等乘客说完，把整段音频送去识别 —— 等 1 到 1.5 秒
- 把文本送给 LLM，等完整回答 —— 等 3 到 4 秒
- 把完整回答送去合成 —— 等 1 到 1.5 秒
- 开始播放
加起来 5.5 到 7 秒。值得注意的是，这里没有任何一步是慢的。每个服务都在它的正常区间内。问题不在某个环节，在于它们排成一队，而且每一步都在等一个「完整」的输入。
延迟很少是被某个慢环节造成的。它是被「等完整」这个习惯造成的。
五个改动
一、识别不等说完
Azure 的流式识别在乘客说话的同时就在出临时转写。等他说完最后一个字，最终文本基本已经就绪。这一步省下 0.5 到 1 秒 —— 但更重要的是，它把「等待」从串行链路里删掉了，而不只是缩短。
二、按句子切分模型输出
LLM 的流式接口会一个 token 一个 token 地吐。我们把 token 缓冲成句子：遇到句号、问号、感叹号就把这一句立刻送去合成，不等整段生成完。如果连续凑到 100–150 个字符还没遇到断句，也强制刷出去 —— 模型偶尔会写出很长的从句，不能让它把整条链路卡住。
结果是：乘客听到「请保持冷静，救援正在赶来」的时候，模型还在生成下一句。
三、合成也是流式
Azure TTS 同样支持边合成边输出。第一句的音频字节已经在回传路上时，第二句可能还没生成完。音频经 Twilio 的同一条 WebSocket 送回，并在流里完成从压缩格式到 mulaw 的转换。
四、VAD 调优（这是最难的一个）
VAD 负责判断乘客什么时候开始说、什么时候说完了。说完了才触发后面整条链路。Azure 内置的端点检测能区分「思考中的停顿」和「说完了的停顿」—— 在正常对话里够用。
但被困在电梯里的人不是正常说话。他们语速不稳、句子断开、因为紧张而停顿很久。阈值设得激进，系统会在他们说到一半时打断；设得保守，每句话后面都拖着一段死寂。我们最后是拿真实通话录音一轮轮调出来的，没有捷径。
如果要我只说一件这个项目里最花时间的事，是这个。它和模型完全无关。
五、连接保活
每通电话新建到 Azure 的连接，光 TCP 加 TLS 握手就是 100 到 200 毫秒。我们维持常驻的 WebSocket 连接池，并且每 5 到 10 分钟发一次保活请求，避免服务端冷启动带来的 3 到 5 秒延迟。SDK 对象也跨通话复用，不每通电话新建。
最后的数字
| 阶段 | 典型耗时 | 说明 |
|---|
| 识别（说完 → 最终文本） | 300–500ms | 大部分工作已在说话期间完成 |
| 模型首个 token | 500ms–1s | 不等完整回答 |
| 合成首块音频 | 200–400ms | 拿到第一句就开始 |
| 跨云往返 | 50–100ms | 同地理区域 |
| 端到端首字音频 | 约 2.3 秒 | 原为 5.5–7 秒 |
一个绕不开的问题：打断
如果 AI 正在说话，乘客开口了怎么办？
必须立刻停。识别层一旦在播放期间检测到人声，我们马上中止 TTS、清空音频缓冲、切回收听状态。不处理这件事，系统就会和一个被困的人抢话 —— 在应急场景里，这不只是体验问题。
2.3 秒够好吗？
不够。正常人类对话的响应间隔是 300 到 500 毫秒，2.3 秒依然明显慢。
但比较的对象不是人类对话，是原来的方案：按键菜单，或者排队等几分钟接通人工。在这个对照组下，2.3 秒的智能应答是压倒性的改善。
继续往下压的方向也清楚：用一个更小更快的模型生成第一句确认语（「我听到了，救援正在安排」），同时让主模型在后台准备正式回答；把最常见场景的开场白预先合成好缓存起来。
带走的规则
优化实时链路时，先别问「哪一步最慢」，先问「哪一步在等一个完整的输入」。把「等完整」换成「边到边处理」，通常比换更快的服务收益大一个量级 —— 而且不花钱。
A voice system has no loading spinner
People who build for the web carry a quiet assumption about latency: a little slow is fine, just show a spinner. Voice doesn't offer that. A second of model latency is a second of silence on the line — and inside a stuck elevator, silence means nobody is coming.
So on this project latency wasn't a performance metric. It was a product metric. The first version took 5.5 to 7 seconds to produce its first syllable. Nobody noticed that number in an internal demo. On a real call it was a disaster.
Do the arithmetic first
The naive implementation is a serial pipeline where each step waits for the last:
- Wait for the caller to finish, send the whole clip to recognition — 1 to 1.5 seconds
- Send the transcript to the LLM, wait for the complete answer — 3 to 4 seconds
- Send the complete answer to synthesis — 1 to 1.5 seconds
- Start playback
That's 5.5 to 7 seconds. What's worth noticing is that no individual step is slow. Every service is inside its normal range. The problem isn't any one stage — it's that they queue, and each one waits for a complete input.
Latency is rarely caused by one slow component. It's caused by the habit of waiting for something to be complete.
Five changes
1. Recognition doesn't wait for them to stop
Azure's streaming recognition emits partial transcripts while the caller is still speaking. By the time they say their last word, the final text is essentially ready. That saves half a second to a second — but more importantly it removes the wait from the serial chain rather than shortening it.
2. Cut the model's output into sentences
The LLM's streaming API emits token by token. We buffer tokens into sentences: on a period, question mark or exclamation mark, that sentence goes straight to synthesis without waiting for the rest. If we accumulate 100–150 characters without hitting a boundary, we flush anyway — the model occasionally writes a very long clause, and it can't be allowed to stall the whole chain.
The effect: the caller hears "Please stay calm, help is on the way" while the model is still generating the next sentence.
3. Synthesis streams too
Azure TTS produces audio as it goes. Bytes for the first sentence are already travelling back while the second is still being generated. The audio returns over the same Twilio WebSocket, converted from a compressed format to mulaw inside the stream.
4. Tuning VAD — the hard one
Voice activity detection decides when the caller starts and, critically, when they have finished. Finishing is what triggers the rest of the chain. Azure's built-in endpointing distinguishes a thinking pause from a finished pause, and in ordinary conversation that works.
But someone trapped in an elevator is not having an ordinary conversation. They speak in bursts, break sentences apart, and pause for a long time out of stress. Set the threshold aggressively and the system cuts them off mid-sentence. Set it patiently and every utterance trails dead air. We tuned it against real call recordings, round after round. There was no shortcut.
If I had to name the single most time-consuming thing in this project, it's this. It has nothing to do with the model.
5. Keep connections warm
Opening a fresh connection to Azure per call costs 100 to 200 milliseconds in TCP and TLS handshake alone. We hold a pool of persistent WebSocket connections and send a keepalive every five to ten minutes to avoid the three-to-five-second cold start on the provider's side. SDK objects are reused across calls rather than constructed per call.
Where it landed
| Stage | Typical | Note |
|---|
| Recognition (end of speech → final text) | 300–500ms | Most of the work already done while speaking |
| Model first token | 500ms–1s | We never wait for the full answer |
| Synthesis first audio chunk | 200–400ms | Starts on the first sentence |
| Cross-cloud round trip | 50–100ms | Same geographic region |
| End to end, first audio | ~2.3 seconds | Down from 5.5–7 seconds |
The problem you can't skip: barge-in
What happens when the AI is talking and the caller speaks?
It has to stop immediately. The moment recognition detects speech during playback, we abort TTS, clear the audio buffer and return to listening. Without that handling, the system talks over someone who is trapped — and in an emergency that isn't just a bad experience.
Is 2.3 seconds good enough?
No. Human conversation turns around in 300 to 500 milliseconds; 2.3 seconds is still clearly slow.
But the comparison isn't human conversation. It's the alternative that existed before: a keypad menu, or minutes in a queue waiting for an operator. Against that baseline, 2.3 seconds of intelligent response is an overwhelming improvement.
The path lower is clear too: use a smaller, faster model for the first acknowledgement ("I hear you, help is being dispatched") while the main model prepares the real answer in the background, and pre-synthesise the opening lines for the most common scenarios.
The rule I kept
When you optimise a real-time chain, don't start by asking which step is slowest. Ask which step is waiting for a complete input. Replacing "wait for the whole thing" with "process as it arrives" usually beats buying a faster service by an order of magnitude — and it costs nothing.