Skip to main content

Redis Shake

Redis Shake is a powerful and versatile tool for migrating and replicating data between Redis instances. It's designed for minimal downtime and ensures data consistency during the process. It supports various Redis versions and configurations, making it a suitable solution for diverse use cases.

Features​

  • Full Data Migration (Initial Sync): Copies all data from the source Redis instance to the target instance.
  • Incremental Data Synchronization (Replication): Continuously synchronizes data between instances, ensuring that the target instance remains up-to-date with the source. This minimizes downtime during transition.
  • Cross-Version Compatibility: Supports migrating data between different Redis versions (e.g., Redis 5 to Redis 7).
  • Redis Cluster Support: Seamlessly works with Redis Cluster setups, allowing you to migrate or replicate data across sharded environments.
  • Data Filtering: Allows you to selectively migrate or replicate data based on key patterns, data types, or other criteria using powerful filter configurations. This is often done to exclude specific data or to migrate only specific features.
  • Data Transformation (Lua Scripting): Provides the capability to transform data during migration using Lua scripting, enabling data modification on the fly.
  • High Performance: Optimized for high throughput and efficient data transfer.
  • Command-Line Interface (CLI): Easy-to-use CLI for managing and monitoring migration and replication tasks.
  • TLS/SSL Support: Provides secure communication between Redis instances using TLS/SSL encryption. Allows configuring certificates and private keys.
  • Authentication Support: Supports authentication using passwords for both source and destination Redis instances.
  • Detailed Logging and Monitoring: Provides comprehensive logging and metrics for monitoring the progress of migration and replication tasks.

Use Cases​

  • Redis Version Upgrades: Migrate data to a new Redis version without significant downtime.
  • Cloud Migration: Move data between on-premises Redis instances and cloud-based Redis services (e.g., AWS ElastiCache, Azure Cache for Redis, Google Cloud Memorystore).
  • Disaster Recovery and High Availability: Replicate data between Redis instances in different regions for disaster recovery purposes.
  • Data Sharding: Distribute data across multiple Redis instances for improved scalability.
  • Database Migration: Migrate data from other data sources to Redis (requires data transformation).
  • Test Environment Setup: Create a copy of a production Redis instance for testing purposes.

Installation​

Redis Shake is typically distributed as a pre-compiled binary or as source code.

Binary Installation (Example)​

  1. Download the appropriate binary for your operating system from the Redis Shake releases page (usually on GitHub).

  2. Extract the archive.

  3. Place the binary in a directory included in your system's PATH. For example:

    wget [URL_TO_REDIS_SHAKE_BINARY]
    tar -zxvf redis-shake-[version].tar.gz
    sudo mv redis-shake /usr/local/bin/

Compilation from Source​

  1. Clone the Redis Shake repository from GitHub.

  2. Build the binary using the provided build script (usually written in Go).

    git clone [URL_TO_REDIS_SHAKE_REPOSITORY]
    cd redis-shake
    go build .

Configuration​

Redis Shake is configured using a YAML file. The configuration file specifies the source and destination Redis instances, filtering rules, and other settings.

Here's an example configuration file (redis-shake.yml):

source:
  address: redis://:password@source-redis-host:6379
  type: redis
  # tls:
  #    enabled: true
  #    cert_file: /path/to/cert.pem
  #    key_file: /path/to/key.pem
  #    ca_file: /path/to/ca.pem

target:
  address: redis://:password@target-redis-host:6379
  type: redis
  # tls:
  #    enabled: true
  #    cert_file: /path/to/cert.pem
  #    key_file: /path/to/key.pem
  #    ca_file: /path/to/ca.pem

filter:
  key_regex:
    - "user:*" # Include keys starting with "user:"
  #  not_key_regex:
  #    - "temp:*" # Exclude keys starting with "temp:"

rdb:
  output: true # Enable full rdb sync
  file_path: /tmp/dump.rdb # Path for the output RDB file on the READER redis.

sync:
  batch_size: 1000
  parallel: 8

log:
  level: info #debug, info, warn, error
  file: redis-shake.log

Key Configuration Parameters:

  • source.address: The connection string for the source Redis instance. Include the scheme (redis://), password (if any), host, and port.
  • target.address: The connection string for the target Redis instance.
  • filter.key_regex: A list of regular expressions used to include keys that match the pattern. When specifying multiple regexes, they are treated as OR (i.e., a key is included if it matches ANY of them).
  • filter.not_key_regex: A list of regular expressions used to exclude keys that match the pattern. When specifying multiple regexes, they are treated as OR (i.e., a key is excluded if it matches ANY of them).
  • rdb: Configuration options for initial full sync from RDB files
  • sync : Configuration options for incremental replication.
  • sync.batch_size: The number of keys to process in each batch during replication.
  • sync.parallel: The number of parallel threads to use for replication.
  • log.level: The logging level (e.g., info, debug, warn, error).
  • tls: (Optional) Enable TLS encryption for secure client-server communication.

Usage​

To run Redis Shake, use the following command:

redis-shake -conf redis-shake.yml

This command will start Redis Shake using the configuration file redis-shake.yml.

Common Commands:

  • redis-shake -type=dump -conf redis-shake.yml: Performs a full data migration (RDB dump and restore).
  • redis-shake -type=sync -conf redis-shake.yml: Starts incremental data synchronization (replication).

Monitoring and Troubleshooting​

Redis Shake provides logging and metrics that can be used to monitor the progress of a migration or replication task.

  • Logs: Check the Redis Shake log file (specified in the configuration file) for errors and warnings.
  • Metrics: Redis Shake exposes metrics that can be monitored using Prometheus or other monitoring systems. (Details on how to enable and access metrics will typically be in the official documentation).

Common Troubleshooting Tips:

  • Connection Errors: Verify that the source and target Redis instances are accessible and that the connection strings are correct.
  • Authentication Errors: Double-check the Redis passwords.
  • Filtering Issues: Ensure that the key filtering rules are configured correctly.
  • Performance Issues: Adjust the batch_size and parallel parameters to optimize performance.
  • Memory Issues: If Redis Shake encounters memory issues, try reducing the batch_size.
  • Firewall Issues: Make sure there are no firewall rules blocking connections between redis-shake and the Redis instances.

Advanced Features​

  • Lua Scripting for Data Transformation: Use Lua scripts to modify data during migration.
  • Custom Filters: Implement custom filtering logic.
  • Integration with Monitoring Systems: Monitor Redis Shake using Prometheus or other systems.

Performance Tuning​

  • Adjust the sync.parallel parameter (number of threads)
  • Adjust sync.batch-size parameters based on the network bandwidth.
  • Make sure network latency between the Redis instances is low. Use instances in the same region/datacenter for optimal performance.
  • Make sure that both source and target Redis have enough resources.

Limitations​

  • Complex data transformations may impact performance.
  • Specific data types may require special handling.
  • Redis Shake GitHub Repository: [Link to Redis Shake GitHub Repository] (Replace with the actual URL)
  • Redis Shake Documentation: [Link to Redis Shake Documentation] (Replace with the actual URL if one exists)
  • Redis Official Website: https://redis.io/