Build persistent message queues with consumer groups, message acknowledgment, and historical replay using Redis Streams—ideal for event sourcing and multi-consumer workloads.
Unlike Pub/Sub (fire-and-forget) or Lists (items removed on read), Streams persist messages and support multiple independent consumers reading the same data with tracking of what each has processed.
A Stream is an append-only log. Each entry has:
1526919030474-0)Entries are ordered by ID, guaranteeing strict temporal ordering.
XADD orders * customer_id "123" product "widget" quantity "5"
The * tells Redis to auto-generate an ID based on the current timestamp. The command returns the generated ID (e.g., 1526919030474-0).
You can also specify an ID explicitly, but auto-generation is recommended for most use cases.
Read from the beginning:
XREAD STREAMS orders 0
Read only new messages (blocking):
XREAD BLOCK 5000 STREAMS orders $
The $ means "only messages arriving after this command." BLOCK 5000 waits up to 5 seconds for new messages.
Read a specific range:
XRANGE orders 1526919030474-0 + COUNT 100
This reads up to 100 messages starting from the specified ID.
Streams enable event sourcing, where application state is derived from an immutable sequence of events.
Instead of storing current state directly ("user has 100 points"), you store events ("user earned 10 points", "user spent 5 points", "user earned 95 points"). The current state is computed by replaying events.
Services append events:
XADD user:123:events * type "points_earned" amount "10"
XADD user:123:events * type "points_spent" amount "5"
A new service can bootstrap by reading the stream from the beginning:
XREAD STREAMS user:123:events 0
It processes each event to build its local state.
For scaling message processing, Redis implements Consumer Groups (similar to Kafka).
XGROUP CREATE orders order-processors $ MKSTREAM
This creates a consumer group named order-processors on the orders stream. The $ means start from new messages only; use 0 to process the entire history.
XREADGROUP GROUP order-processors worker-1 COUNT 10 STREAMS orders >
The > means "give me messages never delivered to this group." Each message is delivered to exactly one consumer in the group.
When a message is delivered, it enters the Pending Entries List (PEL). It remains pending until acknowledged:
XACK orders order-processors 1526919030474-0
This tells Redis the message was successfully processed.
If a consumer crashes, its pending messages remain in the PEL. Other consumers can claim them:
XPENDING orders order-processors
This shows pending messages and which consumer holds them.
XCLAIM orders order-processors worker-2 60000 1526919030474-0
This transfers ownership of a message to worker-2 if it has been pending for at least 60 seconds (60000 milliseconds).
XAUTOCLAIM (Redis 6.2+): Combines XPENDING scan and XCLAIM in one command:
XAUTOCLAIM orders order-processors worker-2 60000 0-0 COUNT 10
This automatically finds and claims up to 10 messages pending for over 60 seconds, simplifying recovery logic.
Streams can grow indefinitely. To limit size:
XTRIM orders MAXLEN 10000
Keep only the most recent 10,000 entries. Use MAXLEN ~ 10000 for approximate trimming (faster).
MINID trimming (Redis 6.2+): Trim by ID instead of count:
XTRIM orders MINID 1609459200000-0
Removes all entries with IDs older than the specified minimum. Useful for time-based retention policies.
XINFO STREAM orders
XINFO GROUPS orders
XINFO CONSUMERS orders order-processors
| Feature | Pub/Sub | Lists | Streams |
|---|---|---|---|
| Persistence | No | Yes | Yes |
| Consumer Groups | No | No | Yes |
| Message History | No | No | Yes |
| Acknowledgment | No | Manual | Built-in |
| Multiple Consumers | Broadcast | Competing | Both |