问题：状态住在进程里
一通应急电话的上下文是有价值的：乘客说过什么、系统判断出的场景类型、已经召回的 SOP 片段、当前进行到哪一步。如果这些东西只存在于处理这通电话的那个进程的内存里，那么这个进程崩溃的瞬间，上下文就没了。
后果不是「重试一下」。后果是一个被困在电梯里的人，要把刚才说过的话再说一遍。在这个场景下，这是不可接受的。
数据模型
我们用 AWS ElastiCache for Redis 做共享状态层。这里有个容易说错的细节：它是托管服务，不是我们自己在集群里跑的一个 Redis 容器。AWS 负责节点、打补丁、复制和故障转移。它和 worker 在同一个 VPC 里，延迟亚毫秒级。
键是 Twilio 给每通电话分配的 CallSid。值是一个 JSON 对象，装着让任何一台 worker 都能接手所需的全部信息：
| 字段 | 作用 |
|---|
conversation_thread | 完整对话历史，换 worker 后模型仍有上下文 |
call_state | pending / in_progress / incomplete / complete |
assigned_worker | 当前由谁处理 |
last_heartbeat | 最后一次心跳时间，用来判定 worker 是否已死 |
retrieved_sop_chunks | 已召回的 SOP 片段，接管时不必重新检索 |
scenario_type | 场景类型（困人、门故障、断电…），用于检索过滤 |
created_at / updated_at | 监控与 TTL |
已完成的通话设 TTL，大约 30 分钟后自动过期，避免 Redis 里堆满陈旧数据。读写用 SET / GET 加 JSON 序列化 —— 简单且够用。规模再大一级时，可以换成 HSET / HGET 只更新单个字段，而不是每次重写整个对象。
状态机
每通电话只有四个状态，转换规则很窄：
pending — 电话已接入，尚未分配 workerin_progress — worker 正在处理，持续写心跳incomplete — worker 超时或失败，等待重新分配complete — 通话已处理完毕
一次完整的接管
worker A 正在处理通话 ABC123，每隔几秒写一次心跳。然后它崩了：
- 心跳停止
- watchdog 扫描 Redis，找到状态为
in_progress 但心跳超过 10 秒的通话
- 命中
ABC123，把状态改为 incomplete，清空 assigned_worker
- ECS 检测到容器不健康，拉起替换
- worker B 领走这通
incomplete 的电话，从 Redis 读回完整对话
- 继续对话
乘客体验到的是一次短暂的停顿。不是从头再来。
心跳发现死亡比健康检查快得多。在应急场景里，十秒和一分钟不是同一个产品。
为什么这同时解决了扩容
这是我最满意的部分：因为 worker 完全无状态，任何一台都能处理任何一通电话，所以「加机器」和「替换崩掉的机器」在系统看来是同一件事。
不需要粘性会话，不需要把某通电话路由回特定实例，不需要在扩容时做状态迁移。ECS 基于 CPU 使用率和一个自定义的「活跃通话数」指标自动扩缩，新容器起来就能干活。
那 Redis 自己挂了呢
这是这套设计里唯一的强依赖，所以值得单独回答。
ElastiCache 配了一主一从。主节点故障时，AWS 自动把只读副本提升为主。应用通过统一的 ElastiCache 端点连接，不需要知道哪个节点是主 —— 切换对我们的代码是透明的。
再往上一个量级，正确答案是 Redis Cluster：数据分片，避免单点成为吞吐瓶颈。我们当时的规模还没到那一步，所以没有提前做。
两个容易说错的地方
在技术尽调里，这两句话会立刻暴露深度：
- 别说「我们跑了个 Redis 容器」。生产环境用托管服务。自己在编排集群里跑一个有状态的 Redis，意味着复制、故障转移、备份全都要自己扛 —— 而这正是你不想在凌晨三点处理的事。
- 别说「我们把数据存成字符串」。准确的说法是「JSON 序列化的通话上下文，用 SET / GET 读写」。前者听起来像没想过数据模型。
带走的规则
先问「这台机器现在可以被直接拔电源吗」。如果答案是否定的，说明有状态还住在进程里。把它搬出去，故障转移和水平扩容会同时变成免费的。
The problem: state lives in the process
The context of an emergency call is valuable: what the caller has said, the scenario the system inferred, which SOP chunks were retrieved, how far along the exchange is. If all of that exists only in the memory of the process handling that call, then the moment the process dies, the context dies with it.
The consequence isn't "retry". The consequence is that someone trapped in an elevator has to say it all again. In that setting, that is not acceptable.
The data model
We used AWS ElastiCache for Redis as the shared state layer. One detail that's easy to get wrong when describing this: it's a managed service, not a Redis container we ran in our own cluster. AWS owns the nodes, the patching, the replication and the failover. It sits in the same VPC as the workers, sub-millisecond away.
The key is the CallSid Twilio assigns to each call. The value is a JSON object holding everything any worker would need to take over:
| Field | Purpose |
|---|
conversation_thread | Full message history, so the model keeps context across a handover |
call_state | pending / in_progress / incomplete / complete |
assigned_worker | Who is handling it right now |
last_heartbeat | Last update from the worker — how we detect death |
retrieved_sop_chunks | RAG results already fetched, so a takeover doesn't re-retrieve |
scenario_type | Entrapment, door malfunction, power failure… used to filter retrieval |
created_at / updated_at | Monitoring and TTL |
Completed calls carry a TTL and expire after about thirty minutes, so Redis doesn't fill with stale context. Reads and writes are SET / GET with JSON serialisation — simple and sufficient. At the next order of magnitude you'd move to HSET / HGET to update individual fields instead of rewriting the whole object each time.
The state machine
A call has four states and the transitions are deliberately narrow:
pending — received, not yet assignedin_progress — a worker is handling it and writing heartbeatsincomplete — the worker timed out or failed; awaiting reassignmentcomplete — resolved
One full takeover
Worker A is handling call ABC123 and writing a heartbeat every few seconds. Then it crashes:
- Heartbeats stop
- A watchdog scans Redis for calls still
in_progress whose heartbeat is older than ten seconds
- It finds
ABC123, flips the state to incomplete and clears assigned_worker
- ECS notices the unhealthy container and starts a replacement
- Worker B claims the incomplete call and reads the full conversation back from Redis
- The conversation continues
What the caller experiences is a brief pause. Not a restart.
Heartbeats detect death far faster than health checks do. In an emergency, ten seconds and one minute are not the same product.
Why this also solved scaling
This is the part I'm most pleased with. Because workers hold no state, any worker can take any call — which means "add a machine" and "replace a machine that died" are the same operation as far as the system is concerned.
No sticky sessions. No routing a given call back to a specific instance. No state migration when scaling. ECS scales on CPU utilisation plus a custom active-call-count metric, and a new container is useful the moment it boots.
What if Redis itself goes down?
It's the one hard dependency in the design, so it deserves a direct answer.
ElastiCache runs a primary with a read replica. If the primary fails, AWS promotes the replica automatically. The application connects through the ElastiCache endpoint and never needs to know which node is primary — the failover is transparent to our code.
An order of magnitude further up, the right answer is Redis Cluster with data partitioning so a single node stops being the throughput ceiling. We weren't at that scale, so we didn't build it early.
Two things people get wrong describing this
In a technical review these two phrases immediately reveal depth — or the lack of it:
- Don't say "we ran Redis in a container." Production uses a managed service. Running stateful Redis inside your own orchestration cluster means owning replication, failover and backups yourself — exactly the work you don't want at three in the morning.
- Don't say "we stored it as a string." The accurate description is "JSON-serialised call context, read and written with SET/GET". The first version sounds like nobody thought about the data model.
The rule I kept
Ask whether you could pull the power cord on any one machine right now. If the answer is no, state is still living inside a process. Move it out, and failover and horizontal scaling both become free.