Historical case study of Twitter's Redis customizations—many of these innovations are now built into Redis core (quicklist, improved memory management).
Twitter faces unique challenges due to the "fan-out" nature of its product—a single tweet from a high-profile account must appear in millions of timelines within seconds. This document covers patterns Twitter developed, noting which are now available natively in Redis.
When a user with 10 million followers tweets, that tweet must appear in 10 million timelines. Twitter caches these timelines in Redis for instant retrieval. The scale is immense: over 30 billion timeline updates per day.
Standard Redis data structures present a trade-off between memory efficiency and CPU performance:
Listpack (formerly Ziplist): Memory-compact, perfect for small lists, but modifying requires rewriting the entire contiguous memory block. For timeline services handling millions of insertions, the latency penalty becomes unacceptable.
Linked List: Offers O(1) insertion but suffers from significant memory overhead due to pointer storage and memory fragmentation.
Twitter developed the Hybrid List as a compromise: chains of multiple small Listpacks linked together.
Standard Listpack: A single contiguous memory block [entry1|entry2|...|entryN]. Must rewrite the entire block on any modification.
Hybrid List: Multiple small listpacks linked together. Each listpack holds a few entries. Modifications only affect one small block, not the entire structure.
Benefits: - Retains high memory locality and compactness of Listpacks - Enables flexible linking capabilities of standard lists - Minimizes memory fragmentation - Avoids "stop-the-world" latency spikes from resizing large monolithic structures - Ensures predictability for real-time timeline rendering
Modern Redis: This exact data structure is now built into Redis as quicklist (since Redis 3.2). All Redis Lists use quicklist internally—a doubly-linked list of listpacks. Configure with
list-max-listpack-sizeto control how many entries each listpack holds before a new node is created. Twitter's innovation became a core Redis feature.
For scenarios requiring both point lookups AND range queries:
Traditional approach: - Hash for point lookups (find specific tweet ID) - Sorted Set for ranges (tweets between timestamps) - Memory duplication
B-Tree Set approach: - Single structure supporting both operation types - Hierarchical keys enable point lookups and range queries - Significant memory footprint reduction when caching billions of objects
Modern Redis: Sorted Sets now support lexicographic ordering when members share the same score. By setting all scores to 0, members sort alphabetically, enabling B-tree-like functionality: -
ZRANGEBYLEXfor prefix and range queries on string keys -ZLEXCOUNTfor counting items in a lexicographic range - Composite keys likeuser:123:tweet:456enable hierarchical queriesSee the Lexicographic Sorted Set Patterns document for implementation details.
Twitter employs a sophisticated proxy layer separating concerns:
Layer 1 - Control Path (slow): Handles cluster management, resharding, and node additions.
Layer 2 - Cluster Management System: Maintains global view mapping Keys to Buckets to Machines.
Layer 3 - Proxy Layer (Twemproxy/Custom): Hashes keys to specific shards. Clients connect here.
Layer 4 - Shards: Multiple Redis shard instances (Shard 1, Shard 2, ... Shard N).
Key benefits: - Client application is unaware of physical cluster topology - Data can be moved between nodes without client updates - Cluster state changes don't induce latency jitter - Philosophy: "Predictability over peak performance"
The standard Redis MONITOR command halts the event loop, degrading performance by 50%+ at high throughput.
Twitter implemented a Lock-Free Command Logger:
Flow: Redis Main Thread writes to a Lock-Free Ring Buffer. An Async Thread drains the buffer to disk.
The main Redis thread writes to a lock-free ring buffer. A separate thread drains this buffer to disk asynchronously.
Captured per request: - Command type - Key size - Client IP - Timestamp
Capabilities enabled: - Traffic auditing at 100k+ QPS - Hot key identification (viral tweets causing uneven load) - Abusive client detection in real-time - Zero impact on primary Redis thread latency
Each timeline entry contains: - Tweet ID (8 bytes) - User ID (8 bytes) - Metadata (4 bytes) - Total: 20 bytes per entry
With 800 entries per user timeline, each user's cached timeline consumes approximately 16KB. Across hundreds of millions of users, this amounts to terabytes of RAM.
Not all users deserve cache space. Active users have their timelines hot in Redis. Inactive users (those who haven't logged in recently) have their timelines evicted. When an inactive user returns, their timeline is rebuilt from persistent storage on demand.
This tiered approach ensures RAM is spent on users who will actually benefit from low-latency access.
| Metric | Value |
|---|---|
| Timeline updates/day | 30 billion |
| Commands/second | 40M - 100M |
| Cache as % of infrastructure | ~3% |
| Timeline IDs per user | 800 (capped) |
| Replication factor | 3x (fault tolerance) |
Twitter's engineering team has shared these patterns in technical talks including "Scaling Redis at Twitter" and the infrastructure blog at blog.x.com/engineering.