Event mesh vs message broker: when Kafka is the wrong shape

# Event mesh vs message broker: when Kafka is the wrong shape

Kafka won the last decade of streaming. It deserved to. But the shape of the workload is changing, and a shape that fit ETL pipelines and analytics is starting to strain under AI agents.

This post is about the difference between a message broker and an event mesh, why it matters for agentic workloads, and when you should reach for each.

What Kafka is optimized for

Kafka is a distributed log. Producers append, consumers read. Partitions give you ordering within a key. Consumer groups give you parallelism.

This shape is perfect for:

• ETL pipelines with predictable throughput
• Change data capture
• Analytics fan-out
• Anything where the downstream is stateless and horizontally scalable

The implicit assumption is that consumers are cheap and homogeneous. Add more of them, get more throughput. The broker does not care what they do.

What an agent workload actually looks like

An AI agent is not a stateless consumer. It is:

• Expensive per message (LLM tokens, tool calls, latency)
• Stateful within a task (working memory, plan, budget)
• Heterogeneous (different agents for different message types)
• Bursty (10 messages arrive, one triggers a 30-second plan)

On Kafka, this shape hurts. You end up doing routing in application code: read a partition, look at the message, dispatch to the right agent, handle backpressure yourself, re-inject failures into a DLQ topic, write a compactor to keep state small.

Everything works. It is just that half your code is now a router the broker was never meant to be.

What an event mesh adds

An event mesh is a broker plus a routing layer that understands the shape of an event, not just its key.

Concretely:

• Content-based routing — deliver payment.suspicious to the fraud agent and payment.refund to the finance agent, without a Kafka Streams job in between.
• Backpressure by consumer, not by partition — if the fraud agent is slow, hold its events, do not stall the refund agent that shares a topic.
• Task-aware delivery — deliver a batch of related events (same user, same session) to the same agent instance so it can keep working memory local.
• Cost-aware fan-out — when an event needs 3 agents to see it, do not require 3 consumer groups and 3× the storage; the mesh handles the fan-out with one copy.

None of these are impossible on Kafka. They are just not what Kafka is for. You end up building them, and the buildout is where the operational bill lives.

When Kafka is still the right answer

• Analytics and CDC pipelines
• Homogeneous, cheap consumers
• You already have a Kafka team and it is working
• The throughput dominates the routing complexity

Do not rip out Kafka to be trendy. It is a solved problem for a specific shape.

When to reach for an event mesh

• Multi-agent systems where different events go to different agents
• Workloads where per-message cost is high (LLM inference, external API calls)
• Situations where you are writing more router code than business logic
• When you want the wire protocol to still be Kafka-compatible so your existing tools work

The last point matters more than it sounds. Adopting a new streaming platform is a multi-quarter project. Adopting an event mesh that speaks the Kafka protocol on port 9092 is a one-day migration for most producers and consumers.

The practical test

Here is the test we use: what percentage of your streaming code is business logic vs plumbing?

If it is >60% plumbing (routing, dispatch, DLQs, retry logic, per-consumer backpressure), you are using a message broker to do an event mesh job. That is when the shape is wrong, and no amount of tuning fixes it.

StreamFlow speaks the native Kafka protocol on port 9092 but routes like an event mesh. See the benchmarks →