Why we rebuilt the Kafka wire protocol from scratch for AI agents

# Why we rebuilt the Kafka wire protocol from scratch for AI agents

Or: three architectural bets, one benchmark, and the thing we got wrong.

The polling problem

Most AI agent stacks in 2026 still poll. A cron job wakes up, hits a REST endpoint, checks if something changed, calls an LLM, maybe fires an action. Repeat every N seconds. It works in demos and fails in production for two reasons:

1. Missed events. If your polling interval is 30 seconds, you miss anything that happens and resolves inside that window. For fraud detection, incident response, or IoT telemetry, that's most of your signal.
2. Cost. Every poll that returns "nothing changed" is money and latency wasted.

The obvious fix is to make agents event-driven: subscribe to a message stream, react as events arrive. The obvious backbone is Kafka. So why not just use Kafka?

Bet #1: Kafka wire compatibility, different engine

Kafka's protocol is battle-tested and every language has a client. We wanted that adoption surface. But Kafka's internal design is optimized for a workload we don't have.

Kafka is tuned for big batches on few topics. Producers accumulate records, brokers write to segmented log files, consumers pull large batches. The whole storage layer assumes throughput per topic-partition dominates.

Our workload is the opposite: small messages fanned out across many topics. An agent pipeline typically has one topic per stage (observe, decide, act, retry, dead-letter), each carrying a few KB payloads. We want sub-millisecond fan-out to hundreds of consumer groups, not 500 MB/s per partition.

So we implemented the Kafka wire protocol on top of a different storage engine — one that keeps hot topics fully in-memory with async durability, and pages cold topics to disk. Producers and consumers written for Kafka work unchanged; the engine underneath is not Kafka.

Bet #2: Per-topic state, not shared state

Traditional agent frameworks hold state in the process running the agent — a Python dict, a Redis cache, a vector store. When the process dies, the state dies.

We push state to the topic itself. Each pipeline stage reads from its input topic, writes to its output topic, and any recovery state to a compacted state topic. Restart the runtime and the next consumer picks up exactly where the last one stopped — no external cache, no coordination service.

This costs us some flexibility (state schema is per-topic, not per-agent) but buys us horizontal scaling for free. Add a replica, it joins the consumer group, done.

Bet #3: MCP as the tool contract

Every agent framework has invented its own tool interface. This fragmentation is why tool ecosystems don't compound.

The Model Context Protocol has enough momentum that we bet the tool interface on it. Every "act" stage in a Pulse pipeline is an MCP tool call. Users install any MCP-compatible tool and it works.

The risk: if MCP loses to something else, we rewrite this layer. The upside: we get the entire MCP ecosystem for free the day it exists.

The benchmark

On a c7g.4xlarge (16 vCPU ARM, 32 GB RAM) running Linux, single node, no replication:

| Payload | Throughput | p99 latency | |---|---|---| | 512 B | 4.13M msg/s | 2.1 ms | | 4 KB | 890K msg/s | 3.8 ms | | 32 KB | 118K msg/s | 12 ms |

Same box running a mainstream Kafka broker with sensible defaults hits roughly 8× less throughput on small payloads, closing to parity on large ones — exactly what the design predicts.

What we got wrong

We initially made state topics optional. "Users can opt into durability if they need it." Six months in, every production user had turned it on and complained the docs made it sound like a niche feature. It should have been on by default with an opt-out for benchmark rigs.

Try it

Pulse is the self-hosted edition of this engine — free, no signup, no cloud. pulse run pipeline.yaml and you have a running observe→decide→act loop.

Bring your own LLM, plug in any MCP tools, and start reacting to events instead of polling for them.

→ Get started with Pulse → Read the architecture docs → See the full benchmark methodology