https://blog.rabbitmq.com/posts/2021/07/rabbitmq-streams-overview/
RabbitMQ 3.9 introduces a new type of data structure: streams. Streams unlock a set of use cases that could have been tedious to implement with “traditional” queues. Let’s discover in this post how streams expand the capabilities of RabbitMQ.
What are RabbitMQ Streams
A RabbitMQ stream models an append-only log with non-destructive consuming semantics. This means that – contrary to traditional queues in RabbitMQ – consuming from a stream does not remove messages.
Streams in RabbitMQ are persisted and replicated. This translates to data safety and availability (in case of the loss of a node), as well as scaling (reading the same stream from different nodes.)
Streams can look a bit opinionated compared to the very versatile queues, but they come in handy for a set of use cases. They expand the capabilities of RabbitMQ in a very nice way.
What are streams good for
RabbitMQ Streams shine for the following use cases:
- large fan-outs: where many applications need to read the same messages (with traditional queues, that would require declaring a queue per application and delivering a copy of the same message to each of them)
- large backlogs: streams store messages on disk, not in-memory, so the only limit is the disk capacity
- replay & time-traveling: consumers can attach anywhere in a stream, using an absolute offset or a timestamp, and they can read and re-read the same data
- high throughput: streams are super fast compared to traditional queues, several orders of magnitude faster
And as streams ship as a core plugin in RabbitMQ 3.9, you can use them along all the already existing RabbitMQ features.
RabbitMQ Streams in a nutshell
Let’s get more specific about streams:
- streams provide at-least-once guarantees thanks to publisher confirms and message de-duplication on the publisher side.
- streams support server-side offset tracking, to let consumers restart where they left off.
- as streams have non-destructive semantics, they can grow a lot. RabbitMQ Streams can truncate streams automatically according to retention policies, based on size or age.
- streams are accessible through a dedicated, blazing fast binary protocol and through AMQP 0.9.1 & 1.0 ( fast).