Everything You Need to Know About Redis Benchmarking

Redis has exploded in popularity as a lightning fast in-memory data store used by leading companies like Twitter, Snapchat, Stack Overflow and more. As Redis usage ramps up across critical applications, comprehensive benchmarking becomes essential to optimize performance and plan capacity.

This extensive 4500+ word guide covers everything developers and system administrators need to know about properly benchmarking Redis instances. Follow these best practices to analyze throughput, latency, memory use, networking, and more to boost Redis performance.

Why Redis Benchmarking Matters

Let‘s start by reviewing why Redis benchmarking is so important for application success:

Identify Performance Needs – Benchmarking reveals exactly how fast Redis needs to be to support app requirements. You can tailor instance sizing accordingly.
Compare Configurations – Easily test different Redis configurations like persistence strategies to pick optimal settings.
Find Bottlenecks – Locate problems like maxed CPU slowing Redis down or insufficient memory bandwidth.
Plan Capacity – Benchmarking helps predict Redis memory, compute, network resource needs for future data volumes.
Optimize Data Structures – Learn which Redis data structures are fastest for different use cases.
Set Performance Goals – Benchmarking determines practical latency and throughput goals based on reality, not hopes.
Monitor Trends – Continual benchmarking shows the impact of application and data changes over time.
Validate Improvements – Prove upgrades, new hardware, configuration changes, etc. really do speed up Redis through measurable results.

Without proper Redis benchmarking, you are just guessing about true performance. Why leave it to chance when benchmarks provide concrete facts?

Key Redis Metrics to Benchmark

When benchmarking Redis, these are the most critical performance metrics to focus on:

Throughput – The number of operations per second Redis sustains under load, such as queries, inserts, updates, etc. This should be tested for both peak and sustained periods.
Latency – How long individual operations take to complete, measured in milliseconds. Lower latency translates to faster responses.
Memory Use – Memory consumed for storing data, caching, etc. Critical for capacity planning as data volumes grow.
Network IO – Network utilization between app servers and Redis, indicating potential bottlenecks.
Disk Persistence – Impact of snapshotting, AOF logging on throughput and latency.
Replication Lag – For Redis replication, how long data takes to sync from primary to secondaries.

You need to dig into all these metrics to get a complete picture of Redis performance across the stack. Now let‘s explore useful benchmarking tools.

Benchmarking with redis-benchmark

The simplest Redis benchmarking tool is redis-benchmark which comes pre-installed with every Redis server download. It provides quick baseline benchmarking from the command line.

Just point redis-benchmark at your Redis server:

redis-benchmark -h my.redis.server -p 6379

This runs a test using default options of:

50 parallel clients
100K requests total
2 byte request payloads
Pipeline size of 1 (no pipelining)

Sample output:

PING_INLINE: 101336.77 requests per second
PING_BULK: 102880.98 requests per second  
SET: 96313.23 requests per second
GET: 104239.66 requests per second

To customize, use additional options like:

-n for number of requests
-d for payload size
-P for pipeline size
-c concurrent connections

For example:

redis-benchmark -n 1000000 -d 512 -P 10 -c 100

redis-benchmark is great for simple perceived throughput checks. For more robust testing, leverage tools like memtier_benchmark and RedisInsight detailed next.

Heavy Duty Benchmarking with memtier_benchmark

memtier_benchmark is an advanced, open source load testing tool specifically built for benchmarking Redis and Memcached. It enables more real-world Redis benchmarking scenarios.

To install on Linux:

git clone https://github.com/RedisLabs/memtier_benchmark.git
cd memtier_benchmark  
./build.sh

memtier_benchmark is driven via command line options or a YAML config file. Key parameters:

-s <Redis host/port> – Server to benchmark
-n <requests> – Total number of test requests
-c <clients> – Number of parallel clients
--ratio <sets>:<gets> – Mix of write vs. read commands

For example:

memtier_benchmark -s 127.0.0.1:6379 -n 1000000 -c 50 --ratio 1:10

This benchmarks 1 million requests on Redis at 127.0.0.1:6379 using 50 clients with a 10% SET to 90% GET ratio.

Sample output:

Totals:
    Connections: 50
    Requests: 1000000   
    SETs: 100000
    GETs: 900000

Load gen: 4901.543 requests per second
SET: 919.676 requests per second
GET: 3981.387 requests per second

Latency: 
    Avg: 5.225 msec
    Min: 0.028 msec
    Max: 274.997 msec
    Std Dev: 4.501 msec
    ...

But what really makes memtier_benchmark shine is the detailed latency histograms in the output:

Latency Histogram:
     <= 1 msec: 5.29%
    <= 2 msec: 5.98% 
     <= 5 msec: 17.76%
    <= 10 msec: 39.01%
    <= 20 msec: 63.10% 
    <= 50 msec: 93.07%
   <= 100 msec: 98.45%
  <= 250 msec: 99.99%
 <= 1000 msec: 100.00%

This shows the percentage of operations completing within defined latency buckets, highlighting outliers.

Other memtier_benchmark benefits:

Support for Redis Cluster benchmarking
Custom request distributions
Multithreaded clients
Redis latency spike testing
Auth benchmarking for password protected Redis

With all these features, memtier_benchmark provides extremely thorough Redis benchmarking.

GUI Redis Benchmarking with RedisInsight

RedisInsight is a free Redis GUI provided by Redis Labs that makes benchmarking a breeze through an intuitive interface. It runs on Linux, Windows, and MacOS.

RedisInsight allows fast benchmarking of:

Standalone Redis servers
Redis on Flash (ROF) databases
Redis Enterprise clusters
Redis Cloud database instances

To start, just add your Redis server in RedisInsight then click the "Benchmark" button. You can configure:

Redis commands to benchmark (SET, GET, INCR, LPUSH, etc)
Ratio of different operations
Number of clients
Threads per client
Payload size
Duration of test

Once configured, RedisInsight runs the benchmark and displays interactive result charts and graphs for:

Overall throughput
Operations per second
Latency percentiles
Memory and CPU usage
Network IO
And more

The click to configure GUI makes it simple to tweak test parameters and re-run benchmarks. Compare results across different configurations to pick optimal settings.

RedisInsight also provides health monitoring, alerts, slow log analytics, and other critical Redis management features – all for free.

Benchmarking Redis Memory Usage

In addition to throughput and latency, carefully benchmarking Redis memory utilization is key for capacity planning. This can be done in two ways:

The INFO MEMORY Redis command provides an overview of total memory consumption and breakdown between data structures:

 127.0.0.1:6379> info memory

 used_memory:503528976
 used_memory_human:480.03M
 used_memory_rss:504201216
 ...

 used_memory_dataset:420111104
 used_memory_dataset_perc:78.06% 

 allocator_allocated:504201216
 allocator_active:504360960
 allocator_resident:504201216
 ...

The open source redis-rdb-tools includes an rdbanalyze tool that generates incredibly detailed memory analysis reports from Redis RDB files:

 $ rdbanalyze memory-report.json 

 Memory usage:

 Database 0:
      Overhead: 
        Hashtable Overhead:  
            Number of buckets: 524288
            Bucket size: 8 bytes
            Number of items: 581471
            Hashtable size: 4194304 bytes
         ...
      Data size: 422742624 bytes (89.78%)

      Total size of keys/values: 471864385 bytes
      Total size of keys/values + overhead: 471865984 bytes

 Total memory usage: 471865984 bytes

This helps identify keys using excessive memory for optimization.

Match Production Environments

To get truly meaningful benchmark results, mimic real production environments as closely as possible:

Use production data – Seed Redis with a copy of live data to match memory patterns and data access.
Observe traffic – Check application monitoring and logs to model the ratio of different operations.
Test data limits – Push maximum sizes expected now and in the future.
Add failures – Inject node failures, network blips, and other real-world conditions.

The more your benchmarks reflect reality, the better basis you have for optimizing Redis and planning capacity.

Automate Benchmarking for Ongoing Analysis

Running Redis benchmarks manually is time consuming and inconsistent. The best practice is to automate benchmarking for efficiency and frequent analysis.

Effective approaches include:

Scripting benchmark tests using Python, Go, Ruby, etc.
Leverage RedisInsight REST API for automation
Create CI/CD pipelines in Jenkins, CircleCI, etc.
Integrate benchmarking with monitoring tools like Datadog

For automation:

Test dev/staging environments to avoid production impact
Run during controlled low traffic periods
Alert on performance regressions
Track trends over time with monitoring integrations
Build dashboards to easily visualize results

Automated benchmarking provides huge benefits for continuously optimizing Redis.

Using Results to Improve Performance

The entire point of benchmarking is to uncover opportunities to improve Redis performance and efficiency. Use results to:

Tune Memory Settings

Set maxmemory based on peak usage
Adjust eviction policies as appropriate
Enable memory optimization features like active defrag

Choose Optimal Data Structures

Select the most appropriate data structures for each use case
Set expire times to limit bloat from stale data
Prefer hashes, sets, and sorted sets over big keys

Reduce Latency

Configure Redis IO threads based on CPU cores
Set latency SLOs as targets
Identify slow commands with slowlog for optimization
Implement pipeline request batching

Scale Throughput

Right size Redis server compute and memory resources
Implement read replicas to spread read load
Distribute load across a Redis Cluster
Cache intensive queries and computations

Properly applying optimizations uncovered via benchmarking can massively improve Redis throughput, reduce latency, lower resource consumption, and boost application performance.

Conclusion

Comprehensive Redis benchmarking is a must for mission critical applications, especially as data volumes and complexity continue to grow. Tools like redis-benchmark, memtier_benchmark, and RedisInsight make benchmarking easy.

Be sure to mimic production environments, automate testing, and tune configurations based on findings. Following these Redis benchmarking best practices will ensure fast, efficient performance both now and in the future.