Skip to content

Redis Cluster

Jump to bottom
Mark Paluch edited this page Jun 13, 2020 · 11 revisions

Lettuce supports Redis Cluster with:

  • Support of all CLUSTER commands

  • Command routing based on the hash slot of the commands' key

  • High-level abstraction for selected cluster commands

  • Execution of commands on multiple cluster nodes

  • MOVED and ASK redirection handling

  • Obtaining direct connections to cluster nodes by slot and host/port (since 3.3)

  • SSL and authentication (since 4.2)

  • Periodic and adaptive cluster topology updates

  • Publish/Subscribe

Connecting to a Redis Cluster requires one or more initial seed nodes. The full cluster topology view (partitions) is obtained on the first connection so you’re not required to specify all cluster nodes. Specifying multiple seed nodes helps to improve resiliency as Lettuce is able to connect the cluster even if a seed node is not available. Lettuce holds multiple connections, which are opened on demand. You are free to operate on these connections.

Connections can be bound to specific hosts or nodeIds. Connections bound to a nodeId will always stick to the nodeId, even if the nodeId is handled by a different host. Requests to unknown nodeId’s or host/ports that are not part of the cluster are rejected. Do not close the connections. Otherwise, unpredictable behavior will occur. Keep also in mind that the node connections are used by the cluster connection itself to perform cluster operations: If you block one connection all other users of the cluster connection might be affected.

Command routing

The concept of Redis Cluster bases on sharding. Every upstream node within the cluster handles one or more slots. Slots are the unit of sharding and calculated from the commands' key using CRC16 MOD 16384. Hash slots can also be specified using hash tags such as {user:1000}.foo.

Every request, which incorporates at least one key is routed based on its hash slot to the corresponding node. Commands without a key are executed on the default connection that points most likely to the first provided RedisURI. The same rule applies to commands operating on multiple keys but with the limitation that all keys have to be in the same slot. Commands operating on multiple slots will be terminated with a CROSSSLOT error.

Cross-slot command execution and cluster-wide execution for selected commands

Regular Redis Cluster commands are limited to single-slot keys operation – either single key commands or multi-key commands that share the same hash slot.

The cross slot limitation can be mitigated by using the advanced cluster API for a set of selected multi-key commands. Commands that operate on keys with different slots are decomposed into multiple commands. The single commands are fired in a fork/join fashion. The commands are issued concurrently to avoid synchronous chaining. Results are synchronized before the command is completed.

Following commands are supported for cross-slot command execution:

  • DEL: Delete the KEYs. Returns the number of keys that were removed.

  • EXISTS: Count the number of KEYs that exist across the upstream nodes being responsible for the particular key.

  • MGET: Get the values of all given KEYs. Returns the values in the order of the keys.

  • MSET: Set multiple key/value pairs for all given KEYs. Returns always OK.

  • TOUCH: Alters the last access time of all given KEYs. Returns the number of keys that were touched.

  • UNLINK: Delete the KEYs and reclaiming memory in a different thread. Returns the number of keys that were removed.

Following commands are executed on multiple cluster nodes operations:

  • CLIENT SETNAME: Set the client name on all known cluster node connections. Returns always OK.

  • KEYS: Return/Stream all keys that are stored on all upstreams.

  • DBSIZE: Return the number of keys that are stored on all upstreams.

  • FLUSHALL: Flush all data on the cluster upstreams. Returns always OK.

  • FLUSHDB: Flush all data on the cluster upstreams. Returns always OK.

  • RANDOMKEY: Return a random key from a random upstream.

  • SCAN: Scan the keyspace across the whole cluster according to ReadFrom settings.

  • SCRIPT FLUSH: Remove all the scripts from the script cache on all cluster nodes.

  • SCRIPT LOAD: Load the script into the Lua script cache on all nodes.

  • SCRIPT KILL: Kill the script currently in execution on all cluster nodes. This call does not fail even if no scripts are running.

  • SHUTDOWN: Synchronously save the dataset to disk and then shut down all nodes of the cluster.

Cross-slot command execution is available on the following APIs:

  • RedisAdvancedClusterCommands

  • RedisAdvancedClusterAsyncCommands

  • RedisAdvancedClusterReactiveCommands

Execution of commands on one or multiple cluster nodes

Sometimes commands have to be executed on multiple cluster nodes. The advanced cluster API allows to select a set of nodes (e.g. all upstreams, all replicas) and trigger a command on this set.

Example 1. Using NodeSelection API to read all keys from all replicas
RedisAdvancedClusterAsyncCommands<String, String> async = clusterClient.connect().async();
AsyncNodeSelection<String, String> replicas = connection.slaves();

AsyncExecutions<List<String>> executions = replicas.commands().keys("*");
executions.forEach(result -> result.thenAccept(keys -> System.out.println(keys)));

The commands are triggered concurrently. This API is currently only available for async commands. Commands are dispatched to the nodes within the selection, the result (CompletionStage) is available through AsyncExecutions.

A node selection can be either dynamic or static. A dynamic node selection updates its node set upon a cluster topology view refresh. Node selections can be constructed by the following presets:

  • upstreams

  • replicas (operate on connections with activated READONLY mode)

  • all nodes

A custom selection of nodes is available by implementing custom predicates or lambdas.

The particular results map to a cluster node (RedisClusterNode) that was involved in the node selection. You can obtain the set of involved RedisClusterNodes and all results as CompletableFuture from AsyncExecutions.

The node selection API is a technical preview and can change at any time. That approach allows powerful operations but it requires further feedback from the users. So feel free to contribute.

Refreshing the cluster topology view

The Redis Cluster configuration may change at runtime. New nodes can be added, the upstream for a specific slot can change. Lettuce handles MOVED and ASK redirects transparently but in case too many commands run into redirects, you should refresh the cluster topology view. The topology is bound to a RedisClusterClient instance. All cluster connections that are created by one RedisClusterClient instance share the same cluster topology view. The view can be updated in three ways:

  1. Either by calling RedisClusterClient.reloadPartitions

  2. Periodic updates in the background based on an interval

  3. Adaptive updates in the background based on persistent disconnects and MOVED/ASK redirections

By default, commands follow -ASK and -MOVED redirects up to 5 times until the command execution is considered to be failed. Background topology updating starts with the first connection obtained through RedisClusterClient.

Connection Count for a Redis Cluster Connection Object

With Standalone Redis, a single connection object correlates with a single transport connection. Redis Cluster works differently: A connection object with Redis Cluster consists of multiple transport connections. These are:

  • Default connection object (Used for key-less commands and for Pub/Sub message publication)

  • Connection per node (read/write connection to communicate with individual Cluster nodes)

  • When using ReadFrom: Read-only connection per read replica node (read-only connection to read data from read replicas)

Connections are allocated on demand and not up-front to start with a minimal set of connections. Formula to calculate the maximum number of transport connections for a single connection object:

1 + (N * 2)

Where N is the number of cluster nodes.

Apart of connection objects, RedisClusterClient uses additional connections for topology refresh. These are created on topology refresh and closed after obtaining the topology:

  • Set of connections for cluster topology refresh (a connection to each cluster node)

Client-options

Examples

Example 2. Connecting to a Redis Cluster
RedisURI redisUri = RedisURI.Builder.redis("localhost").withPassword("authentication").build();

RedisClusterClient clusterClient = RedisClusterClient.create(redisUri);
StatefulRedisClusterConnection<String, String> connection = clusterClient.connect();
RedisAdvancedClusterCommands<String, String> syncCommands = connection.sync();

...

connection.close();
clusterClient.shutdown();
Example 3. Connecting to a Redis Cluster with multiple seed nodes
RedisURI node1 = RedisURI.create("node1", 6379);
RedisURI node2 = RedisURI.create("node2", 6379);

RedisClusterClient clusterClient = RedisClusterClient.create(Arrays.asList(node1, node2));
StatefulRedisClusterConnection<String, String> connection = clusterClient.connect();
RedisAdvancedClusterCommands<String, String> syncCommands = connection.sync();

...

connection.close();
clusterClient.shutdown();
Example 4. Enabling periodic cluster topology view updates
RedisClusterClient clusterClient = RedisClusterClient.create(RedisURI.create("localhost", 6379));

ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                                .enablePeriodicRefresh(10, TimeUnit.MINUTES)
                                .build();

clusterClient.setOptions(ClusterClientOptions.builder()
                                .topologyRefreshOptions(topologyRefreshOptions)
                                .build());
...

clusterClient.shutdown();
Example 5. Enabling adaptive cluster topology view updates
RedisURI node1 = RedisURI.create("node1", 6379);
RedisURI node2 = RedisURI.create("node2", 6379);

RedisClusterClient clusterClient = RedisClusterClient.create(Arrays.asList(node1, node2));

ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                                .enableAdaptiveRefreshTrigger(RefreshTrigger.MOVED_REDIRECT, RefreshTrigger.PERSISTENT_RECONNECTS)
                                .adaptiveRefreshTriggersTimeout(30, TimeUnit.SECONDS)
                                .build();

clusterClient.setOptions(ClusterClientOptions.builder()
                                .topologyRefreshOptions(topologyRefreshOptions)
                                .build());
...

clusterClient.shutdown();
Example 6. Obtaining a node connection
RedisURI node1 = RedisURI.create("node1", 6379);
RedisURI node2 = RedisURI.create("node2", 6379);

RedisClusterClient clusterClient = RedisClusterClient.create(Arrays.asList(node1, node2));
StatefulRedisClusterConnection<String, String> connection = clusterClient.connect();

RedisClusterCommands<String, String> node1 = connection.getConnection("host", 7379).sync();

...
// do not close node1

connection.close();
clusterClient.shutdown();

Lettuce documentation was moved to https://redis.github.io/lettuce/overview/

Intro

Getting started

HA and Sharding

Advanced usage

Integration and Extension

Internals

Clone this wiki locally