Benchmarks

    Real runs. Real disk. Reproducible.

    How StreamFlow performs across throughput and latency — measured on the same hardware, with the official Kafka tools, against Apache Kafka and Redpanda. No mocks, no in-memory fakes, no simulated transports.

    AWS · 3× m6id.2xlarge brokersKafka 3.7 perf-test1M records · 256 B · 16 part.acks=1 · RF=3 · warm
    Broker shootout · Kafka wire protocol

    StreamFlow vs Apache Kafka vs Redpanda

    The only fair comparison: three broker backends, the same official kafka-*-perf-test tools, the same AWS cluster, the Kafka API on :9092.

    Produce throughput

    records / second · higher is better
    StreamFlow
    491,642
    Apache Kafka
    487,092
    Redpanda
    488,998
    Honest reading — all three land at ~490K: the single load generator is the wall, not the brokers, so produce is parity, not a win. The point is StreamFlow is not slower than Kafka on Kafka's own wire.

    Consume throughput

    records / second · higher is better
    StreamFlow
    824,538
    Apache Kafka
    870,447
    Redpanda
    934,602
    Honest reading — StreamFlow (~824K) is on par with Kafka (~0.95×) and slightly below Redpanda. The gap is the Kafka-wire fetch transcoding each record (StreamFlow's on-disk format differs from the wire format) where Kafka and Redpanda serve via sendfile zero-copy. For raw throughput, use the native protocol below.
    Produce latency · p99
    9.0 ms
    All three brokers at parity (9.0 ms p99) under the same acks=1, RF=3 workload.
    Produce bandwidth
    120 MB/s
    StreamFlow 120.0 · Kafka 118.9 · Redpanda 119.4 MB/s — the client NIC ceiling, shared by all three.
    Native protocol · StreamFlow's own client

    Beyond the Kafka wire — zero Kafka in the path

    StreamFlow has its own binary transport. On the same AWS cluster, off the Kafka protocol entirely:

    Native produce · pipelined columnar
    4.13 M msg/s
    1.06 GB/s
    Native consume · full record delivery
    3.31 M msg/s
    847 MB/s
    Both directions are capped at ~1 GB/s by the single load generator's NIC, not by the brokers — a floor, not a ceiling. StreamFlow's native path is ~8× the Kafka-wire path on the same hardware. The Kafka protocol is a compatibility surface; it is not the fast path.
    Storage engine · local disk hot path

    What the engine writes to disk

    DirectIOBenchmark — multi-shard, group-commit, real disk writes. 4-vCPU host, Oracle JDK 25, FileChannel / Panama FFM backends.

    Single thread
    3.6–3.8 M evt/s
    One writer, FlatEventBatch → engine → disk.
    Multi-shard
    8.9–9.3 M ops/s
    Shard-per-core, group-commit. io_uring is available but trails on ≤4 vCPU (its SQPOLL pollers steal cores from the shards).
    Methodology

    Reproduce it yourself

    Every number above is from a real run. Here is exactly how — same tools, same config.

    HardwareAWS · 3× m6id.2xlarge brokers + 1× c6i.xlarge load generator
    ToolsApache Kafka 3.7.0 — kafka-producer-perf-test / kafka-consumer-perf-test
    Workload1,000,000 records · 256-byte payload · 16 partitions
    Durabilityacks=1 · replication factor RF=3 · warm run (JIT hot)
    Broker buildsStreamFlow (this release) · apache/kafka:3.7.0 (KRaft) · redpandadata/redpanda:v24.1.7
    Storage engineDirectIOBenchmark · 4-vCPU · Oracle JDK 25 · FileChannel / Panama FFM

    One line each — the shipped CLI wraps both benchmarks

    bash
    1# storage engine — native disk hot path, every backend, no network, no Kafka
2java -Xmx4g --enable-native-access=ALL-UNNAMED -cp streamflow.jar \
3  com.streamflow.app.cli.StreamFlowCli bench disk
4
5# Kafka wire — stock kafka-clients 3.7.0 against StreamFlow's Kafka port (or any broker)
6java -cp streamflow.jar com.streamflow.app.cli.StreamFlowCli \
7  bench kafka --bootstrap $NODE:9092

    bench disk is the same measurement as the DirectIOBenchmark JUnit test, packaged to run from the jar. bench kafka uses the exact canonical flags (acks=1, batch.size=65536, linger.ms=5). Run every benchmark twice and report the warm (JIT-hot) run.

    Same numbers the raw way — Apache perf tools + the repo

    bash
    1# produce — 1M records, 256B, acks=1, RF=3 topic
2kafka-producer-perf-test --topic bench --num-records 1000000 --record-size 256 \
3  --throughput -1 --producer-props bootstrap.servers=$NODE:9092 acks=1 \
4  batch.size=65536 linger.ms=5
5
6# consume — same 1M records
7kafka-consumer-perf-test --topic bench --messages 1000000 \
8  --bootstrap-server $NODE:9092
9
10# storage engine — native disk hot path from the StreamFlow repo
11mvn -q -Plarge-heap test -Dtest=DirectIOBenchmark -pl streamflow-core

    The CLI wrappers above and these raw commands measure the same paths — pick whichever you have on hand.

    What these numbers do and don't say

    The honest caveats

    Transparency is the point. If a benchmark can't be reproduced or is quietly favourable, it isn't worth publishing.

    → Produce is client-bound parity, not a win.
    All three brokers saturate the single load generator at ~490K rec/s. It shows StreamFlow keeps pace with Kafka on Kafka's wire — nothing more.
    → Consume is ~0.95× Kafka, below Redpanda.
    The Kafka-wire fetch transcodes each record; Kafka/Redpanda use sendfile. Real trade-off, stated plainly. The native path avoids it.
    → io_uring is not always faster.
    On ≤4 vCPU its SQPOLL poller threads steal cores from the shards — FileChannel / Panama win there. io_uring pays off on larger boxes.
    → Native throughput is NIC-capped.
    4.13M / 3.31M msg/s hit the single client's ~1 GB/s NIC, not a broker limit — a floor, add clients to go higher.
    → StreamFlow is not Kafka-based.
    The Kafka protocol is one compatibility surface. The engine is its own native, shard-per-core, io_uring-capable path — where the multi-million-events-per-second figures come from.
    See for yourself

    Numbers are cheap. Run it.

    Spin up a cluster, point the official Kafka tools at it, and reproduce the table above — or build a live app in the browser in 3 minutes.

    Try live — 3 minutesReproduction guide on GitHub