AI agent throughput calculator
Size an event-driven agent pipeline before you build it: how much concurrency a decision stage needs, when it saturates, what p95 latency your users will feel, and what the tokens cost per month.
Your workload
Sustained peak rate on the source stream, not the daily average.
Filters and rules usually absorb most events before any model is called.
Round-trip time of one decision call, including tool calls it triggers.
How many decisions the stage runs in parallel — often capped by provider rate limits.
Prompt plus completion, averaged across the stage.
Set to 0 for a local model running on hardware you already own.
What it takes
4,320,000 events per day reach the pipeline.
Saturated — the backlog grows by ~133 events every minute.
Model time plus queue wait — what the downstream action actually experiences.
Slots required to hold utilisation near 75% with headroom for spikes.
Decision stage only. Filtering more aggressively upstream is the cheapest lever.
Maximum sustainable decisions per second at the current concurrency and latency.
This configuration cannot keep up. Raise concurrency, filter more events before the model, or use a faster model for the first-pass decision.
Numbers are a queueing approximation for capacity planning, not a benchmark. Validate against a replayed production stream before committing.
Questions
How do I size the concurrency of an AI agent pipeline?
Multiply the decisions per second by the model latency in seconds, then add roughly 30% headroom. A stage handling 20 decisions/s at 900 ms needs about 24 concurrent slots to stay below saturation.
What happens when an agent stage is saturated?
Utilisation above 100% means the queue grows without bound: latency climbs every minute and, without a durable event log, events are eventually dropped. Backpressure on a replayable stream turns that failure into a delay you can recover from.
Why is queue wait worse than model latency?
Model latency is fixed per call, but queue wait grows non-linearly as utilisation approaches 100%. At 90% utilisation the wait is already about nine times the service time per slot, which is why sizing for peak, not average, matters.
Are these numbers exact?
No. This is a queueing approximation for capacity planning, not a benchmark. It assumes stable arrival rates and a single decision stage. Use it to find the order of magnitude and the saturation point, then validate with a replayed production stream.
Method
The decision stage is modelled as an M/M/c queue: random arrivals, variable service time, c concurrent slots. Utilisation is arrival rate divided by c/service time; queue wait grows non-linearly as it approaches 100%.
Free to use and free to cite. Related reading: how to build an event-driven agent. Built by the team behind Pulse, an open-source agent runtime.