Redis 集群(Redis Cluster)是 Redis 官方提供的分布式解决方案,用于解决单节点 Redis 内存限制、并发瓶颈和单点故障问题,支持数据分片存储和自动故障转移,可扩展性强
1.基本概念

Master1 和 Slave11 和 Slave12 保存的是同样的数据. 占总数据的 1/3
Master2 和 Slave21 和 Slave22 保存的是同样的数据. 占总数据的 1/3
Master3 和 Slave31 和 Slave32 保存的是同样的数据. 占总数据的 1/3
每个 Slave 都是对应 Master 的备份(当 Master 挂了, 对应的 Slave 会补位成 Master).
每个红框部分都可以称为是⼀个 分⽚ (Sharding).
如果全量数据进⼀步增加, 只要再增加更多的分⽚, 即可解决.
2.数据分片算法
1.哈希求余
例如, N 为 3. 给定 key 为 hello, 对 hello 计算 hash 值(⽐如使⽤ md5 算法), 得到的结果为
bc4b2a76b9719d91 , 再把这个结果 % 3, 结果为 0, 那么就把 hello 这个 key 放到 0 号分⽚上.
当然, 实际⼯作中涉及到的系统, 计算 hash 的⽅式不⼀定是 md5, 但是思想是⼀致的.



2.一致性哈希算法

第二步, 假设当前存在三个分片, 就把分片放到圆环的某个位置上.



3.哈希槽分区算法(Redis 使用)
hash_slot = crc16(key) % 16384
其中 crc16 也是⼀种 hash 算法.
16384 其实是 16 * 1024, 也就是 2^14.



3.集群搭建(基于 docker)

1.创建目录和配置
创建 redis-cluster 目录。内部创建两个文件
redis-cluster/
├── docker-compose.yml
└── generate.sh

for port in $(seq 1 9); \
do \
mkdir -p redis${port}/
touch redis${port}/redis.conf
cat << EOF > redis${port}/redis.conf
port 6379
bind 0.0.0.0
protected-mode no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-announce-ip 172.30.0.10${port}
cluster-announce-port 6379
cluster-announce-bus-port 16379
EOF
done
# 注意 cluster-announce-ip 的值有变化.
for port in $(seq 10 11); \
do \
mkdir -p redis${port}/
touch redis${port}/redis.conf
cat << EOF > redis${port}/redis.conf
port 6379
bind 0.0.0.0
protected-mode no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-announce-ip 172.30.0.1${port}
cluster-announce-port 6379
cluster-announce-bus-port 16379
EOF
done

执行命令
bash generate.sh
生成目录如下:
redis-cluster/
├── docker-compose.yml
├── generate.sh
├── redis1
│ └── redis.conf
├── redis10
│ └── redis.conf
├── redis11
│ └── redis.conf
├── redis2
│ └── redis.conf
├── redis3
│ └── redis.conf
├── redis4
│ └── redis.conf
├── redis5
│ └── redis.conf
├── redis6
│ └── redis.conf
├── redis7
│ └── redis.conf
├── redis8
│ └── redis.conf
└── redis9└── redis.conf
其中 redis.conf 每个都不同. 以 redis1 为例:
区别在于每个配置中配置的 cluster-announce-ip 是不同的, 其他部分都相同.
port 6379
bind 0.0.0.0
protected-mode no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-announce-ip 172.30.0.101
cluster-announce-port 6379
cluster-announce-bus-port 16379


2.编写 docker-compose.yml
version: '3.7'
networks:mynet:ipam:config:- subnet: 172.30.0.0/24
services:redis1:image: 'redis:5.0.9'container_name: redis1restart: alwaysvolumes:- ./redis1/:/etc/redis/ports:- 6371:6379- 16371:16379command:redis-server /etc/redis/redis.confnetworks:mynet:ipv4_address: 172.30.0.101
等等等等
3.启动容器
启动之前,一定要把之前已经运行的 redis 啥的都干掉!!
否则就可能因为端口冲突等原因,导致现在的启动失败!!
docker-compose up -d
4.构建集群
redis-cli --cluster create 172.30.0.101:6379 172.30.0.102:6379
172.30.0.103:6379 172.30.0.104:6379 172.30.0.105:6379 172.30.0.106:6379
172.30.0.107:6379 172.30.0.108:6379 172.30.0.109:6379 --cluster-replicas 2
-cluster create 表⽰建⽴集群. 后⾯填写每个节点的 ip 和地址.
-cluster-replicas 2 表⽰每个主节点需要两个从节点备份

>>> Performing hash slots allocation on 9 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.30.0.105:6379 to 172.30.0.101:6379
Adding replica 172.30.0.106:6379 to 172.30.0.101:6379
Adding replica 172.30.0.107:6379 to 172.30.0.102:6379
Adding replica 172.30.0.108:6379 to 172.30.0.102:6379
..........
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
此时, 使⽤客⼾端连上集群中的任何⼀个节点, 都相当于连上了整个集群.
• 客⼾端后⾯要加上 -c 选项, 否则如果 key 没有落到当前节点上, 是不能操作的. -c 会⾃动把请求
重定向到对应节点.
• 使⽤ cluster nodes 可以查看到整个集群的情况
# redis-cli -h 172.30.0.101 -p 6379 -c
172.30.0.101:6379> CLUSTER nodes
5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 172.30.0.102:6379@16379 master - 0
1682595474000 2 connected 5461-10922
2e5dc211288784ba55d554a377b87bfe2b5398db 172.30.0.105:6379@16379 slave
e4f37f8f0ea0dafc584349999795716613910e51 0 1682595475158 5 connected
0a889103b35db2a6e82e8c09904bbef310cff3b1 172.30.0.108:6379@16379 slave
5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 0 1682595475560 8 connected
3584840ac704c3ee016f3bdcca3f7ebe6f6e8e80 172.30.0.107:6379@16379 slave
5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 0 1682595474656 7 connected
85025819223f12615046c54d89f510e9cd0444a1 172.30.0.104:6379@16379 slave
b3c0a96f6a206088ecea639147b6fcf903afe872 0 1682595475560 4 connected
e4f37f8f0ea0dafc584349999795716613910e51 172.30.0.101:6379@16379 myself,master
- 0 1682595475000 1 connected 0-5460
00ba82bed6abeb015116d51d1af7fcb1609d03ad 172.30.0.109:6379@16379 slave
b3c0a96f6a206088ecea639147b6fcf903afe872 0 1682595475000 9 connected
29f05d98982bd3df05d0222091e4b8ef9569f424 172.30.0.106:6379@16379 slave
e4f37f8f0ea0dafc584349999795716613910e51 0 1682595474000 6 connected
b3c0a96f6a206088ecea639147b6fcf903afe872 172.30.0.103:6379@16379 master - 0
1682595475659 3 connected 10923-16383
172.30.0.101:6379> set k1 1
-> Redirected to slot [12706] located at 172.30.0.103:6379
OK
172.30.0.103:6379> get k1
"1"
4.主节点宕机
docker start redis1
172.30.0.102:6379> CLUSTER NODES
e4f37f8f0ea0dafc584349999795716613910e51 172.30.0.101:6379@16379 slave
2e5dc211288784ba55d554a377b87bfe2b5398db 0 1682603944000 10 connected
0a889103b35db2a6e82e8c09904bbef310cff3b1 172.30.0.108:6379@16379 slave
5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 0 1682603944628 8 connected
29f05d98982bd3df05d0222091e4b8ef9569f424 172.30.0.106:6379@16379 slave
2e5dc211288784ba55d554a377b87bfe2b5398db 0 1682603943625 10 connected
00ba82bed6abeb015116d51d1af7fcb1609d03ad 172.30.0.109:6379@16379 slave
b3c0a96f6a206088ecea639147b6fcf903afe872 0 1682603944126 9 connected
5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 172.30.0.102:6379@16379 myself,master
- 0 1682603944000 2 connected 5461-10922
b3c0a96f6a206088ecea639147b6fcf903afe872 172.30.0.103:6379@16379 master - 0
1682603945030 3 connected 10923-16383
2e5dc211288784ba55d554a377b87bfe2b5398db 172.30.0.105:6379@16379 master - 0
1682603944025 10 connected 0-5460
3584840ac704c3ee016f3bdcca3f7ebe6f6e8e80 172.30.0.107:6379@16379 slave
5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 0 1682603945131 7 connected
85025819223f12615046c54d89f510e9cd0444a1 172.30.0.104:6379@16379 slave
b3c0a96f6a206088ecea639147b6fcf903afe872 0 1682603944628 4 connected
可以使⽤ cluster failover 进⾏集群恢复. 也就是把 101 重新设定成 master. (登录到 101 上执⾏)
处理流程
1.故障判定

核⼼原则是保证每个 slots 都能正常⼯作(存取数据)

2.故障迁移

5.集群扩容
redis-cli --cluster add-node 172.30.0.110:6379 172.30.0.101:6379
此处会先打印出当前集群每个机器的情况,并且要求用户输入要移动多少个 slots
add-node 后的第⼀组地址是新节点的地址. 第⼆组地址是集群中的任意节点地址
执⾏结果
>>> Adding node 172.30.0.110:6379 to cluster 172.30.0.101:6379
>>> Performing Cluster Check (using node 172.30.0.101:6379)
M: 00d319e23ef76a4d51e74600c42ee2a371ae81f6 172.30.0.101:6379
slots:[0-5460] (5461 slots) master
2 additional replica(s)
S: e34911c57d7605903de84ec05b3deac611aaef7e 172.30.0.105:6379
slots: (0 slots) slave
replicates 00d319e23ef76a4d51e74600c42ee2a371ae81f6
S: 6cf48cc11d0171b6ab1b418808473167acd7986e 172.30.0.106:6379slots: (0 slots) slavereplicates 00d319e23ef76a4d51e74600c42ee2a371ae81f6
S: fd18c7f164b09ec563f4573ec9d6466e6769221e 172.30.0.108:6379slots: (0 slots) slavereplicates b3f2ba758318f4bd54031c98c01d7a6155ff43d3
M: 579282abe81b3f20ffd17d5a1956cdca3b0e71b0 172.30.0.103:6379slots:[10923-16383] (5461 slots) master2 additional replica(s)
S: e9ea79b1326ea5a75a1701d5c12a0f6081c1d043 172.30.0.109:6379slots: (0 slots) slavereplicates 579282abe81b3f20ffd17d5a1956cdca3b0e71b0
S: 628d1ec9ceef6760b9038c4fbc83ee92430062ac 172.30.0.107:6379slots: (0 slots) slavereplicates b3f2ba758318f4bd54031c98c01d7a6155ff43d3
M: b3f2ba758318f4bd54031c98c01d7a6155ff43d3 172.30.0.102:6379slots:[5461-10922] (5462 slots) master2 additional replica(s)
S: 2a248acb47f0036655397897f9800c70ea22514f 172.30.0.104:6379slots: (0 slots) slavereplicates 579282abe81b3f20ffd17d5a1956cdca3b0e71b0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 172.30.0.110:6379 to make it join the cluster.
[OK] New node added correctly
2.重新分配 slots(槽位)
redis-cli --cluster reshard 172.30.0.101:6379
reshard 后的地址是集群中的任意节点地址.
另外, 注意单词拼写, 是 reshard (重新切分), 不是 reshared (重新分享) , 不要多写个 e
执⾏之后, 会进⼊交互式操作, redis 会提⽰⽤⼾输⼊以下内容:
• 多少个 slots 要进⾏ reshard ? (此处我们填写 4096)
• 哪个节点来接收这些 slots ? (此处我们填写 172.30.0.110 这个节点的集群节点 id)
• 这些 slots 从哪些节点搬运过来? (此处我们填写 all, 表⽰从其他所有的节点都进⾏搬运)
执⾏结果如下
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? 522a1bd88a1a9084e6919fa88f4bf1c3655ad837
Please enter all the source node IDs.Type 'all' to use all the nodes as source nodes for the hash slots.Type 'done' once you entered all the source nodes IDs.
Source node #1: all
确定之后, 会初步打印出搬运⽅案, 让⽤⼾确认.
之后就会进⾏集群的 key 搬运⼯作. 这个过程涉及到数据搬运. 可能需要消耗⼀定的时间.

3.给新的主节点添加从节点
redis-cli --cluster add-node 172.30.0.111:6379 172.30.0.101:6379
--clusterslave --cluster-master-id [172.30.1.110 节点的 nodeId]
6.集群缩容(了解)
1.删除从节点
此处删除的节点 nodeId 是 111 节点的 id.
# redis-cli --cluster del-node [集群中任⼀节点ip:port] [要删除的从机节点 nodeId]
redis-cli --cluster del-node 172.30.0.101:6379
03f4a97806a0d3de2299cc16e6a3559f0c832bc1
>>> Removing node 03f4a97806a0d3de2299cc16e6a3559f0c832bc1 from cluster
172.30.0.101:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
2.重新分配 slots
redis-cli --cluster reshard 172.30.0.101:6379
执⾏后仍然进⼊交互式操作

第⼀次重分配: 分配给 101 1365 个 slots
接收 slots 的 nodeId 填写 101 的 nodeId. Source Node 填写 110 的 nodeId
How many slots do you want to move (from 1 to 16384)? 1365
What is the receiving node ID? 3397c6364b43dd8a8d49057ad37be57760d3a81f
Please enter all the source node IDs.Type 'all' to use all the nodes as source nodes for the hash slots.Type 'done' once you entered all the source nodes IDs.
Source node #1: 7c343b7e3f82f2e601ac6b9eba9f846b3065c600
Source node #2: done
第⼆次重分配: 分配给 102 1365 个 slots 同上
第三次重分配: 分配给 103 1366 个 slots 同上
接收 slots 的 nodeId 填写 103 的 nodeId. Source Node 填写 110 的 nodeId
此时查看集群状态, 可以看到 110 节点已经不再持有 slots 了.
127.0.0.1:6379> CLUSTER NODES
26e98f947b99b3a2a5da5a7c3ed3875ae9cf366c 172.30.0.103:6379@16379 master - 0
1683187990000 13 connected 6826 10923-16383
ee283a923b77fda2e370ff4463cbb22047f261f7 172.30.0.108:6379@16379 slave
98736357a53c85aaebb31fa5ad286ab36b862426 0 1683187991544 12 connected
98736357a53c85aaebb31fa5ad286ab36b862426 172.30.0.102:6379@16379 master - 0
1683187991544 12 connected 5461-6825 6827-10922
b951da5c3573eec5db2926e022302188a8b686a6 172.30.0.107:6379@16379 slave
98736357a53c85aaebb31fa5ad286ab36b862426 0 1683187991645 12 connected
df193127f99c59999e14518ff7235faeb7b7e8e7 172.30.0.104:6379@16379 slave
26e98f947b99b3a2a5da5a7c3ed3875ae9cf366c 0 1683187990000 13 connected
7c343b7e3f82f2e601ac6b9eba9f846b3065c600 172.30.0.110:6379@16379 master - 0
1683187991000 10 connected
3d148089278f9022a62a5936022e9c086fb7a062 172.30.0.105:6379@16379 slave
3397c6364b43dd8a8d49057ad37be57760d3a81f 0 1683187990540 11 connected
6de544df01337d4bbdb2f4f69f4c408314761392 172.30.0.109:6379@16379 myself,slave
26e98f947b99b3a2a5da5a7c3ed3875ae9cf366c 0 1683187990000 9 connected
3397c6364b43dd8a8d49057ad37be57760d3a81f 172.30.0.101:6379@16379 master - 0
1683187990640 11 connected 0-5460
3b47ebc9868bbda018d8a5b926256861bb800cdc 172.30.0.106:6379@16379 slave
26e98f947b99b3a2a5da5a7c3ed3875ae9cf366c 0 1683187991544 13 connected
3.删除主节点
# redis-cli --cluster del-node [集群中任⼀节点ip:port] [要删除的从机节点 nodeId]
redis-cli --cluster del-node 172.30.0.101:6379
7c343b7e3f82f2e601ac6b9eba9f846b3065c600
>>> Removing node 7c343b7e3f82f2e601ac6b9eba9f846b3065c600 from cluster
172.30.0.101:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.


