How Redis Solved Our Real-Time Analytics Bottleneck: A Case Study

The Problem: Scaling Real-Time Analytics

Our platform, a high-traffic e-commerce website, faced a critical challenge: our real-time analytics pipeline was buckling under the weight of millions of user interactions daily. The existing setup relied on a traditional relational database (PostgreSQL) to log events like page views, cart additions, and purchases. While PostgreSQL excelled at transactional consistency, it struggled with:

• High Latency: Queries for real-time dashboards took seconds to execute, making them useless for live decision-making.
• Write Contention: Concurrent writes during peak hours caused locking issues, slowing down the entire system.
• Storage Bottlenecks: Disk I/O became a limiting factor as the dataset grew.

We needed a solution that could handle:

• Sub-millisecond read/write latency.
• Horizontal scalability to accommodate spikes in traffic.
• A simple way to aggregate and serve real-time metrics.

Why Redis?

After evaluating several technologies (e.g., Apache Kafka, Elasticsearch), we chose Redis for its:

• In-Memory Performance: Data resides in RAM, enabling lightning-fast access.
• Rich Data Structures: Beyond key-value storage, Redis offers sorted sets, hyperloglogs, and streams—perfect for analytics.
• Persistence Options: Despite being in-memory, Redis supports snapshotting and append-only files (AOF) for durability.

Architectural Overview

Here’s how we integrated Redis into our stack:

[Client App] --> [Log Events] --> [Redis Streams]  
                          |  
                          v  
                  [Real-Time Dashboard]  
                          |  
                          v  
            [Periodic ETL to PostgreSQL for long-term storage]  

The Solution: Implementing Redis Streams and Sorted Sets

Step 1: Event Ingestion with Redis Streams

We replaced direct PostgreSQL writes with Redis Streams, a log-like data structure ideal for high-throughput event data.

# Sample Python code to push an event to Redis Stream  
import redis  
r = redis.Redis(host='localhost', port=6379)  

event_data = {  
    "user_id": "123",  
    "action": "view_product",  
    "product_id": "xyz",  
    "timestamp": "2023-10-01T12:34:56"  
}  

r.xadd("user_events", event_data)  

Advantages:

• Low-latency writes (sub-millisecond).
• Consumers (e.g., analytics workers) can process streams asynchronously.

Step 2: Real-Time Aggregations with Sorted Sets

For dashboards, we used Redis’s sorted sets to maintain leaderboards (e.g., "top viewed products").

# Increment a product's view count in a sorted set  
r.zincrby("product_views", 1, "product_xyz")  

# Fetch top 10 viewed products  
top_products = r.zrevrange("product_views", 0, 9, withscores=True)  

Performance Gain:

• Aggregations ran in ~1ms vs. ~500ms with SQL queries.
• Sorted sets automatically handle ranking logic.

Step 3: Hybrid Persistence

To avoid losing data during failures, we:

1. Configured Redis AOF (Append-Only File) for durability.
2. Ran nightly jobs to archive cold data to PostgreSQL.

Results: From Bottleneck to Breakthrough

• Latency: Dashboard queries dropped from 2s to 10ms.
• Scalability: Handled 10x more concurrent users during Black Friday.
• Developer Experience: Simplified codebase by replacing complex SQL with Redis commands.

Lessons Learned

1. Right Tool for the Job: Relational databases aren’t always the answer for real-time workloads.
2. Start Small: We piloted Redis for one dashboard before full migration.
3. Monitor Memory Usage: In-memory systems require careful capacity planning.

Practical Takeaways for Your Project

• Use Redis When: You need low-latency reads/writes or real-time aggregations.
• Avoid Redis When: You require complex joins or strict ACID transactions.
• Pro Tip: Combine Redis with a durable database (e.g., PostgreSQL) for the best of both worlds.

Conclusion

Redis transformed our real-time analytics from a bottleneck into a competitive advantage. By leveraging its in-memory speed and versatile data structures, we achieved sub-millisecond performance and scalability—all while keeping the solution simple and maintainable.

Next Steps:

• Explore RedisTimeSeries for time-based analytics.
• Experiment with RedisGears for serverless-like processing.

Have you faced similar challenges? Share your experiences in the comments!

---

This case study is part of our "Tech in Action" series, where we dissect real-world problems and their solutions. Stay tuned for more deep dives!