Your agents keep running when a server doesn't.
StreamFlow runs your pipelines across a multi-node cluster with quorum-replicated storage. A node can be killed mid-write and not one acknowledged event is lost — failover is measured in seconds, not maintenance windows.
A visual of the real behaviour: on leader loss a follower is promoted (epoch advances by one), and every event already acknowledged with quorum stays readable. The engine enforces this in CI — see Proof.
The single-node trap
One crashed process shouldn't cost you the data — or the SLA. Most agent platforms shine on a laptop and fall over the moment production asks for uptime, horizontal scale, or an audit trail. Teams end up duct-taping reverse proxies, failover scripts, Postgres replicas and a pub/sub bus — none designed together. Six months in, the duct tape is the product.
The node dies → in-flight events vanish, the SLA breaks, and recovery is a manual scramble through logs. Scale is capped at one box.
Quorum-replicated storage + automatic failover. The same pipelines you wrote on day one run partitioned, replicated and persistent — no rewrite.
Quorum-based durability
Acknowledged means committed to a majority — or it isn't acknowledged. Writes replicate across the cluster and commit on a majority quorum before the producer sees an ack. Kill a node mid-produce and every acked record is still there. This is the data-plane guarantee, verified by chaos tests that SIGKILL a broker under load.
RPO = 0 for quorum-acked writes under a hard node kill (acks=all).
A 3-node cluster keeps serving through any single node loss; scale the factor for more.
Crash mid-window → recover from checkpoint → resume. Duplicates 0, losses 0.
Honest about the two modes. Non-transactional failover is RPO = 0 (no acked data lost) with at-least-once delivery, de-duplicated by a per-producer sequence stamp. End-to-end exactly-once (transactional) is a separate, opt-in path. We say which is which — never “zero loss” where we mean “no duplicates”.
Two planes, both fault-tolerant
The control plane coordinates. The data plane replicates. They fail over independently, so a control-plane blip never touches committed events, and a data-plane node loss never confuses the cluster about who leads.
- Leadership
- Postgres advisory lock, active / standby — external arbitration, no extra ZooKeeper.
- Split-brain defence
- Every promotion advances a monotonic epoch; the data plane rejects writes carrying a stale epoch.
- Promotion time
- ~10 s (heartbeat × 2), 15 s SLO ceiling.
- Replication
- Raft over gRPC, majority-ack commit, replication factor ≥ 1.
- Partitioning
- Slot-routed by event key — same key, same owner — so scale is horizontal and state stays local.
- Durability
- Append-only WAL on an io_uring / Panama fast path; replay = recovery.
From laptop to cluster
One click up the production ladder. Every step reversible. HA and the event mesh are one coherent progression, not two products to evaluate. Each step preserves your agents, pipelines, schedules, revisions and audit history.
Zero setup. Great for development, demos, and a first handful of pipelines.
Active/standby with advisory-lock failover and one-paste load-balancer config. Production-grade uptime.
Raft-replicated, partitioned storage across nodes. Production at scale, N-1 durable.
The activation wizard — 5 stages, each retryable, each with rollback
- Step 1License
Verifies your plan supports HA — no surprises at runtime.
- Step 2Peer reachability
Probes every configured node, so network issues surface before takeover, not during it.
- Step 3Advisory lock
Acquires the cluster-wide Postgres lock that prevents split-brain.
- Step 4Load balancer
Generates ready-to-paste nginx / HAProxy / Traefik config.
- Step 5Data migration
H2 → Postgres in one click, with rollback — no manual SQL, no downtime.
Restart-safe. The wizard's state lives in the database, not in memory. If the cluster reboots mid-migration, it resumes at the exact stage it was running — you never lose half a day to a restart bug. Every action is audit-logged and RBAC-gated (HA_ACTIVATE · HA_MIGRATE · HA_ROLLBACK).
Upgrades & disaster recovery
Upgrade with no window. Recover from a checkpoint, not a backup tape.
Drain one node, upgrade, rejoin — the rest keep serving. No maintenance window, no “stop the world”. The epoch guard keeps a rejoining node from acting on stale leadership.
The Data-Plane switch cuts over live: both backends run during the window and only the swap of the active read pointer is atomic. Sub-5ms event latency, no loss in tests. Roll back the same way.
A standby takes over a failed agent and resumes from its last checkpoint — not from zero. A watchdog restarts crashed agents automatically.
Each event carries a monotonic per-producer sequence. After a leader kill the producer state rebuilds, and duplicate replays are dropped — de-dup without a second system.
How we prove it
Every durability claim above is a test that runs in CI. Not a slide. These are the named integration tests that fail the build if the guarantee regresses — the same discipline behind our published benchmarks.
Failover promotion is ~10–15 s (heartbeat-driven), not instantaneous — we quote the measured number, not a marketing one. Geo / multi-region is coordinated by vector clocks for cross-region ordering; ask us about the current multi-AZ staging status.
Production uptime without a DevOps team.
One binary: the connector ecosystem, the streaming compute, the agent runtime and the storage backend — one audit log, one credential vault, one HA wizard. No external orchestrator to assemble.