pika3.5.5 哨兵模式主从切换后从节点无法同步主节点数据 #2905
Labels
☢️ Bug
Something isn't working
Comments
Title: pika3.5.5 The slave node cannot synchronize the master node data after the master-slave switch in sentry mode. |
|
后续看一下情况 |
Let’s take a look at the situation later |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is this a regression?
Yes
Description
pika366哨兵模式有三个节点A、B、C，统一升级到355版本后，C为主节点，C挂掉后哨兵拉起B作为新主，之后BC之间正常同步数据，A无法从B同步数据
Please provide a link to a minimal reproduction of the bug
No response
Screenshots or videos
节点A info信息
节点B info信息
节点C info信息
节点A error log日志
节点B error log日志
节点C无error log
Please provide the version you discovered this bug in (check about page for version information)
Anything else?
部分节点A的pika运行日志：
path : ./conf/pika.conf
-----------Pika server----------
pika_version: 3.5.5
pika_git_sha:0
pika_build_compile_date: 2024-09-20 08:41:17
redis_version: 3.5.5
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240920 17:20:29.963819 1742395 base_conf.cc:336] 1 port 9221
I20240920 17:20:29.963891 1742395 base_conf.cc:336] 2 thread-num 1
I20240920 17:20:29.963897 1742395 base_conf.cc:336] 3 thread-pool-size 12
I20240920 17:20:29.963903 1742395 base_conf.cc:336] 4 sync-thread-num 6
I20240920 17:20:29.963908 1742395 base_conf.cc:336] 5 log-path ./log/
I20240920 17:20:29.963914 1742395 base_conf.cc:336] 6 db-path ./db/
I20240920 17:20:29.963920 1742395 base_conf.cc:336] 7 write-buffer-size 268435456
I20240920 17:20:29.963927 1742395 base_conf.cc:336] 8 arena-block-size 33554432
I20240920 17:20:29.963932 1742395 base_conf.cc:336] 9 timeout 60
I20240920 17:20:29.963937 1742395 base_conf.cc:336] 10 requirepass Dareway@2024
I20240920 17:20:29.963943 1742395 base_conf.cc:336] 11 masterauth Dareway@2024
I20240920 17:20:29.963948 1742395 base_conf.cc:336] 12 userpass Dareway@2024
I20240920 17:20:29.963954 1742395 base_conf.cc:336] 13 userblacklist
I20240920 17:20:29.963960 1742395 base_conf.cc:336] 14 instance-mode classic
I20240920 17:20:29.963965 1742395 base_conf.cc:336] 15 databases 1
I20240920 17:20:29.963971 1742395 base_conf.cc:336] 16 default-slot-num 1024
I20240920 17:20:29.963976 1742395 base_conf.cc:336] 17 replication-num 0
I20240920 17:20:29.963982 1742395 base_conf.cc:336] 18 consensus-level 0
I20240920 17:20:29.963989 1742395 base_conf.cc:336] 19 dump-prefix
I20240920 17:20:29.963994 1742395 base_conf.cc:336] 20 dump-path ./dump/
I20240920 17:20:29.963999 1742395 base_conf.cc:336] 21 dump-expire 0
I20240920 17:20:29.964005 1742395 base_conf.cc:336] 22 pidfile ./pika.pid
I20240920 17:20:29.964010 1742395 base_conf.cc:336] 23 maxclients 20000
I20240920 17:20:29.964016 1742395 base_conf.cc:336] 24 target-file-size-base 20971520
I20240920 17:20:29.964021 1742395 base_conf.cc:336] 25 expire-logs-days 7
I20240920 17:20:29.964027 1742395 base_conf.cc:336] 26 expire-logs-nums 10
I20240920 17:20:29.964033 1742395 base_conf.cc:336] 27 root-connection-num 2
I20240920 17:20:29.964038 1742395 base_conf.cc:336] 28 slowlog-write-errorlog no
I20240920 17:20:29.964044 1742395 base_conf.cc:336] 29 slowlog-log-slower-than 10000
I20240920 17:20:29.964051 1742395 base_conf.cc:336] 30 slowlog-max-len 128
I20240920 17:20:29.964056 1742395 base_conf.cc:336] 31 db-sync-path ./dbsync/
I20240920 17:20:29.964061 1742395 base_conf.cc:336] 32 db-sync-speed 1024
I20240920 17:20:29.964066 1742395 base_conf.cc:336] 33 slave-priority 100
I20240920 17:20:29.964072 1742395 base_conf.cc:336] 34 sync-window-size 9000
I20240920 17:20:29.964078 1742395 base_conf.cc:336] 35 max-conn-rbuf-size 268435456
I20240920 17:20:29.964083 1742395 base_conf.cc:336] 36 write-binlog yes
I20240920 17:20:29.964089 1742395 base_conf.cc:336] 37 binlog-file-size 104857600
I20240920 17:20:29.964095 1742395 base_conf.cc:336] 38 max-cache-statistic-keys 0
I20240920 17:20:29.964100 1742395 base_conf.cc:336] 39 small-compaction-threshold 5000
I20240920 17:20:29.964107 1742395 base_conf.cc:336] 40 max-write-buffer-size 10737418240
I20240920 17:20:29.964111 1742395 base_conf.cc:336] 41 max-write-buffer-number 2
I20240920 17:20:29.964118 1742395 base_conf.cc:336] 42 max-client-response-size 1073741824
I20240920 17:20:29.964123 1742395 base_conf.cc:336] 43 compression snappy
I20240920 17:20:29.964128 1742395 base_conf.cc:336] 44 max-background-flushes 1
I20240920 17:20:29.964134 1742395 base_conf.cc:336] 45 max-background-compactions 2
I20240920 17:20:29.964140 1742395 base_conf.cc:336] 46 max-cache-files 5000
I20240920 17:20:29.964145 1742395 base_conf.cc:336] 47 max-bytes-for-level-multiplier 10
I20240920 17:20:29.964151 1742395 base_conf.cc:336] 48 slaveof B:9331
I20240920 17:20:29.964156 1742395 base_conf.cc:336] 49 admin-cmd-list info, monitor, ping
I20240920 17:20:29.964162 1742395 base_conf.cc:336] 50 replication-id ac78114eef217ce1037ccdc77783fd49e6a3bd0bde0931c3d0
I20240920 17:20:29.964193 1742395 base_conf.cc:336] 51 run-id c18e33d335cc558d4a724a3ccf8f8ff6a315252d
I20240920 17:20:29.964200 1742395 base_conf.cc:336] 52 internal-used-unfinished-full-sync
I20240920 17:20:29.964219 1742395 pika.cc:209] Server at: ./conf/pika.conf
I20240920 17:20:29.964485 1742395 net_interfaces.cc:104] Using Networker Interface: eno145
I20240920 17:20:29.966964 1742395 net_interfaces.cc:148] got ip A
I20240920 17:20:29.966989 1742395 pika_server.cc:156] host: A port: 9221
I20240920 17:20:29.967003 1742395 pika_server.cc:70] Worker queue limit is 20100
W20240920 17:20:29.967011 1742395 pika_server.cc:71] 0.0.0.0
I20240920 17:20:29.967927 1742395 pika_server.cc:1665] Dump file is not exist,path: ./dump/
I20240920 17:20:29.968278 1742395 pika_binlog.cc:98] Binlog: Find the exist file.
W20240920 17:20:29.968418 1742395 pika_binlog_reader.cc:207] Read BadRecord record, will decode failed, this record may dbsync padded record, not processed here
E20240920 17:20:29.968431 1742395 pika_binlog_transverter.cc:167] Binlog Item type error, expect type:1 actualy type: 12032
W20240920 17:20:29.968492 1742395 pika_stable_log.cc:187] Binlog item decode failed
I20240920 17:20:30.127966 1742395 pika_db.cc:49] db0 DB Success
I20240920 17:20:30.128170 1742583 pika_cache_load_thread.cc:186] PikaCacheLoadThread::ThreadMain Start
I20240920 17:20:30.130649 1742395 net_util.cc:121] TimerTaskThread Starting...
I20240920 17:20:30.131000 1742395 pika_server.cc:213] Pika Server going to start
I20240920 17:20:30.131016 1742395 rsync_server.cc:48] start RsyncServer ...
I20240920 17:20:30.131359 1742395 rsync_server.cc:60] RsyncServer started ...
I20240920 17:20:30.131649 1742614 pika_repl_client.cc:199] Try Send Meta Sync Request to Master (B:9331)
W20240920 17:20:30.132860 1742585 pika_repl_client_conn.cc:136] Meta Sync Failed: replicationid on both sides of the connection are inconsistent
W20240920 17:20:30.132918 1742585 pika_server.cc:684] Sync error, set repl_state to PIKA_REPL_ERROR
I20240920 17:20:30.132951 1742584 client_thread.cc:359] received kNotiClose
I20240920 17:20:30.133020 1742584 pika_repl_client_thread.cc:20] ReplClient Close conn, fd=78, ip_port=B:11331
I20240920 17:20:30.534910 1742612 dispatch_thread.cc:151] accept new conn 75:C:48458:0
I20240920 17:20:30.534950 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0
I20240920 17:20:30.534984 1742612 dispatch_thread.cc:151] accept new conn 76:C:48460:0
I20240920 17:20:30.535092 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0
I20240920 17:20:30.543035 1742612 dispatch_thread.cc:151] accept new conn 77:A:45752:0
I20240920 17:20:30.543071 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0
I20240920 17:20:30.543100 1742612 dispatch_thread.cc:151] accept new conn 78:A:45754:0
I20240920 17:20:30.543112 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0
I20240920 17:20:30.543141 1742612 dispatch_thread.cc:151] accept new conn 79:A:37028:0
I20240920 17:20:30.543151 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0
I20240920 17:20:30.543171 1742612 dispatch_thread.cc:151] accept new conn 80:A:37030:0
I20240920 17:20:30.543181 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0
I20240920 17:20:30.756268 1742612 dispatch_thread.cc:151] accept new conn 81:B:52700:0
I20240920 17:20:30.756398 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0
I20240920 17:20:30.756438 1742612 dispatch_thread.cc:151] accept new conn 82:B:52702:0
I20240920 17:20:30.756458 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0
I20240920 17:20:39.646625 1742612 dispatch_thread.cc:151] accept new conn 83:A:45822:0
I20240920 17:20:39.646687 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0
I20240920 17:20:56.832079 1742608 pika_command.cc:1061] command: info keyspace 1
I20240920 17:21:05.428328 1742609 pika_command.cc:1061] command: info keyspace 1
I20240920 17:21:08.944960 1742608 pika_command.cc:1061] command: info keyspace 1
I20240920 17:22:18.149905 1742610 worker_thread.cc:262] connection fd: 83, ip_port: A:45822, name: , is_reply: 0, close: 0 keepalive timeout, the keepalive_timeout_ is 60
I20240920 17:22:32.365897 1742612 dispatch_thread.cc:151] accept new conn 83:A:46476:0
I20240920 17:22:32.365960 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0
I20240920 17:22:45.480271 1742608 pika_command.cc:1061] command: info keyspace 1
I20240920 17:22:48.168867 1742609 pika_command.cc:1061] command: info keyspace 1
I20240920 17:24:12.161893 1742610 worker_thread.cc:262] connection fd: 83, ip_port: A:46476, name: , is_reply: 0, close: 0 keepalive timeout, the keepalive_timeout_ is 60
I20240920 17:28:05.465426 1742612 dispatch_thread.cc:151] accept new conn 83:A:47582:0
I20240920 17:28:05.465494 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0
I20240920 17:28:17.772475 1742604 pika_server.cc:697] Remove Master Success, ip_port: B:9331
I20240920 17:28:32.256162 1742603 base_conf.cc:345] ret OK
I20240920 17:28:32.256547 1752638 pika_server.cc:1798] clear cache start...
I20240920 17:28:32.256592 1742599 base_conf.cc:345] ret OK
I20240920 17:28:32.256605 1752638 pika_server.cc:1802] clear cache finish
部分节点A的pika运行日志2：
1445394:X 23 Sep 2024 16:44:05.087 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 16:45:36.006 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 16:46:06.024 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 16:46:36.034 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 16:47:06.105 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 16:47:32.467 # +sdown master mymaster1 B 9331
1445394:X 23 Sep 2024 16:47:32.598 * Sentinel new configuration saved on disk
1445394:X 23 Sep 2024 16:47:32.598 # +new-epoch 4
1445394:X 23 Sep 2024 16:47:32.607 * Sentinel new configuration saved on disk
1445394:X 23 Sep 2024 16:47:32.607 # +vote-for-leader 461094c8c4e9f902d556fef115b45051fb816393 4
1445394:X 23 Sep 2024 16:47:33.579 # +odown master mymaster1 B 9331 #quorum 2/2
1445394:X 23 Sep 2024 16:47:33.579 # Next failover delay: I will not start a failover before Mon Sep 23 16:53:33 2024
1445394:X 23 Sep 2024 16:47:36.078 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 16:47:39.435 # +config-update-from sentinel 461094c8c4e9f902d556fef115b45051fb816393 A 26380 @ mymaster1 B 9331
1445394:X 23 Sep 2024 16:47:39.435 # +switch-master mymaster1 B 9331 C 9221
1445394:X 23 Sep 2024 16:47:39.435 * +slave slave A:9221 A 9221 @ mymaster1 C 9221
1445394:X 23 Sep 2024 16:47:39.435 * +slave slave B:9331 B 9331 @ mymaster1 C 9221
1445394:X 23 Sep 2024 16:47:39.437 * Sentinel new configuration saved on disk
1445394:X 23 Sep 2024 16:48:06.080 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 16:48:09.459 # +sdown slave B:9331 B 9331 @ mymaster1 C 9221
1445394:X 23 Sep 2024 16:48:36.056 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 16:49:06.084 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 16:49:36.072 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 16:50:06.106 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 16:51:37.034 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 16:52:07.058 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 16:52:37.060 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 16:53:07.105 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 16:53:37.096 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 16:54:07.106 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 16:54:37.054 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 16:55:07.089 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 16:56:38.042 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 16:57:08.081 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 16:57:38.045 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 16:58:08.075 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 16:58:38.026 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 16:59:08.071 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 16:59:38.012 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 17:00:08.027 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 17:01:39.056 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 17:02:09.084 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 17:02:39.043 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 17:03:09.045 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 17:03:39.073 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 17:04:09.089 # -tilt #tilt mode exited
1445394:X 23 Sep 2024 17:04:39.039 # +tilt #tilt mode entered
1445394:X 23 Sep 2024 17:05:09.067 # -tilt #tilt mode exited
The text was updated successfully, but these errors were encountered: