The Kafka wire protocol is not the hard part — the runtime is

# The Kafka wire protocol is not the hard part — the runtime is

When we tell people StreamFlow speaks the native Kafka protocol on port 9092, the follow-up question is usually about the protocol itself. "How hard was it to reimplement?"

Honestly? Not very. The Kafka protocol is well-documented, stable, and battle-tested by a decade of clients. Getting the bytes right is a matter of engineering discipline, not invention.

The hard part starts the moment the bytes arrive. This post is about that.

What "runtime" means here

By "runtime" I mean everything that happens between "a producer sent a record" and "a consumer got an answer that was correct, ordered, and durable". That is:

• Storage layout
• Replication
• Consumer coordination
• Partition assignment
• Failure recovery
• Backpressure
• Compaction
• Observability

Kafka got each of these right for its era. But the tradeoffs bake in assumptions about hardware, workloads, and operating budgets that have all shifted.

Three assumptions that have aged

1. "Disks are slow, so batch everything"

Kafka was designed when spinning disks made sequential writes the dominant cost. NVMe changed the math. A modern SSD does 500k random IOPS at sub-100μs latency. The design that avoided random I/O now leaves throughput on the table.

A modern runtime uses direct I/O, page-cache-bypass, and per-shard append logs that map cleanly to NVMe queues. The result is not "faster Kafka" — it is a different curve. Consistent p99, no page-cache cliff, and the ability to run 10x the partitions per node.

2. "The broker owns the state, the consumer is dumb"

Kafka assumed consumers were stateless workers. Their only state was an offset. Everything else — deduplication, ordering across topics, transactions — pushed to the broker.

Agentic consumers are not stateless. They have working memory, plans, budgets. A modern runtime lets consumers checkpoint task state alongside offsets, so a restart resumes the task, not just the position. This one change collapses a whole category of "how do I make my agent recover" code.

3. "Fan-out means more consumer groups"

On Kafka, if 3 different services need to see the same topic, that is 3 consumer groups and 3× the read amplification. Fine for cheap consumers. Painful when each consumer is an LLM call.

A modern runtime does content-aware fan-out: one read, N deliveries, with per-consumer backpressure. You pay for the disk read once and route N times. On agent workloads this is the single biggest cost savings.

What we do not change

What clients see on the wire is Kafka. Every existing producer, consumer, admin tool, Connect connector, and observability integration works unchanged. The runtime is different; the contract is identical.

This is deliberate. Adopting a new streaming platform is a two-year project. Swapping the runtime under a compatible wire protocol is a one-day rollout with a feature flag.

The lesson

Protocols are contracts. Runtimes are implementations. Contracts should be stable and boring — that is how you build ecosystems. Implementations should evolve with the hardware and workloads underneath.

The next decade of streaming is not going to be won by a new protocol. It is going to be won by the runtime that ships the best implementation of the protocol we already have — one designed for NVMe, stateful consumers, and expensive-per-message workloads.

See the benchmarks or try Pulse — same wire protocol, different runtime.