使用TiKV-CDC实现rawkv集群的两地三中心

一、背景

业务的连续性和稳定性对企业来说至关重要，因此对数据库侧的容灾能力要求也越来越高，传统的高可用架构已无法满足高要求的容灾要求，两地三中心架构已成为主流架构。

对于rawkv集群来说，需要使用TiKV-CDC来实现在两个集群间的数据同步。

二、环境准备

安装介质

tikv-cdc***：

https://github.com/tikv/migration/releases/tag/cdc-v1.1.1

（注：TiKV-CDC是专门同步rawkv集群开发的，跟TiCDC不同）

tikv-cdc使用手册：https://tikv.org/docs/6.5/concepts/explore-tikv-features/cdc/cdc-cn/

三、集群拓扑

两地三中心架构下，总共有3套集群，分别为：主生产集群、同城应急集群、异地灾备集群。

主生产集群承担读写业务，通过TiKV-CDC同步数据至同城应急集群、异地灾备集群，同步链路为2条独立的链路，互不影响。

TiKV-CDC同步链路可以定期进行切换演练，验证灾备环境的可用性。

同城应急集群可以承担读流量。

故障情况说明

单个AZ级故障：

1、集群中同城A机房2个AZ中任意单AZ级故障，集群不受影响。

2、集群中同城B机房AZ故障，集群不受影响。

区域级故障：

1、同城A、B机房同时发生故障，异地C机房的灾备集群可以直接承载流量。

集群级故障：

1、主生产集群发生问题后，可以切换至同城应急集群继续提供服务。

读写分离：

1、同城应急库可以根据不同业务类型提供读能力。

四、创建两地三中心同步链路

4.1 备份主集群数据

rawkv备份使用的工具为tikv-br

备份流水索引集群 $ mkdir -p /data/br $ ./tikv-br backup raw --gcttl=600m --pd "http://PDIP:2379" --storage "local:///data/br" --dst-api-version v2 --log-file="/tmp/tikv-br.log"

4.2 还原数据至灾备集群

将备份数据scp到灾备集群上

$ ./tikv-br restore raw --pd "http://PDIP:2379" --storage "local:///data/br" --log-file /tmp/restore.log"

4.3 创建同步链路

在主集群的tikv-cdc机器上创建同步链路changefeed-id

（PD_MASTER_IP为主集群的PD地址，PD_SLAVE_IP为灾备集群的PD地址）

$ cd /home/tidb/tidb_deploy/cdc-8300/bin ./tikv-cdc cli changefeed create --pd "http://PD_MASTER_IP:2379" --sink-uri tikv://PD_SLAVE_IP:2379 --start-ts <前边记录的BackupTs> --changefeed-id test-cdc

4.4 查看同步链路状态

$ ./tikv-cdc cli changefeed --pd http://PD_MASTER_IP:2379 list

五、主备集群切换

5.1 主集群切换到灾备集群

5.1.1 主集群停止TICDC同步，检查主集群grpc流量为0

观察tikv-details下的grpc流量，若为0，则说明业务已经停止。

5.1.2 执行remove_cdc.sh脚本

remove_cdc.sh需要在主集群的tikv-cdc节点执行。

sh remove_cdc.sh

remove_cdc.sh脚本如下：

注意：CHANGEFEED_ID,pd_master这几个变量需要按照实际情况修改。

#!/bin/bash export PATH=/home/ap/cdacs/.tiup/bin:$PATH TIME=`(date "+%Y-%m-%d_%H-%M-%S")` #CHANGEFEED_ID,pdipdport are variables CHANGEFEED_ID=test-cdc-bak ##pd_master is up stream cdc pd info. pd_master=IP:2379 ticdc_dir=`ps -ef |grep tikv-[c]dc |awk -F" " {print $(NF)}|awk -F"log" {print $1}` ##new add 20230106 cdcstate=`${ticdc_dir}bin/tikv-cdc cli changefeed query -s --changefeed-id $CHANGEFEED_ID --pd http://$pd_master |grep -E "state" |awk -F: {print $2} |awk -F, {print $1} |sed s/[^a-z]//g` logpath="${ticdc_dir}log" scripts_log="$logpath/remove_changefeed_$TIME.log" cdc_change(){ echo "cdc changefeed remove." ${ticdc_dir}bin/tikv-cdc cli changefeed remove --changefeed-id $CHANGEFEED_ID --pd=http://$pd_master if [ $? -eq 0 ]; then echo "’$TIME‘ remove changefeed-id successed!!!" sleep 5 ${ticdc_dir}bin/tikv-cdc cli changefeed --pd http://$pd_master list else echo "‘$TIME’ remove changefeed-id failed!!!" fi } tikvcdcstate() { echo "tikvcdc replication state check." if [ $cdcstate = "normal" ];then echo "tikvcdc replication state is normal." else echo "tikvcdc replication state is $cdcstate." exit 1 fi } tikvcdcreplication() { tsoa=`${ticdc_dir}bin/tikv-cdc cli tso query --pd=http://$pd_master` echo "tikv cdc replication progress check." checknums=10 seconds=2 nums=0 while true do tsob=`${ticdc_dir}bin/tikv-cdc cli changefeed query -s --changefeed-id $CHANGEFEED-ID --pd http://$pd_master |grep -E "tso" |awk -F" " {print $2}|awk -F"," {print $1}` let nums++ if [ $tsob -gt $tsoa ] && [ $nums -le $checknums ];then echo "tikvcdc replication is done,this is $nums times replication check." break elif [ $tsob -lt $tsoa ] && [ $nums -lt $checknums ];then echo "tikvcdc replication is not done,this is $nums times replication check." sleep $seconds else echo "tikvcdc replication is not done,this is $nums times and last replication check." break fi done } main() { tikvcdcstate tikvcdcreplication cdc_change } main| tee ${scripts_log}5.1.3 观察日志输出

检查日志${scripts_log}有无报错，若无报错，则说明执行成功。

5.1.4 灾备集群启动TICDC同步

执行create_cdc.sh脚本，create_cdc.sh需要在tikv-cdc节点执行。

sh create_cdc.sh

create_cdc.sh脚本如下：

注意：CHANGEFEED_ID,pd_master,pd_slave这几个变量需要按照实际情况修改。

#!/bin/bash export PATH=/home/ap/cdacs/.tiup/bin:$PATH TIME=`(date "+%Y-%m-%d_%H-%M-%S")` #CHANGEFEED_ID,pd_master,pd_slave are variables CHANGEFEED_ID=xxx-cdc-bak ##pd_master is up stream cdc pd info. pd_master=IP:2379 #pd_slave is down stream cdc pd info. pd_slave=IP:2379 ticdc_dir=`ps -ef |grep tikv-[c]dc |awk -F" " {print $(NF)}|awk -F"log" {print $1}` logpath="${ticdc_dir}log" scripts_log="$logpath/create_changefeed_$TIME.log" tikvcdcstate() { echo "tikvcdc replication state check." if [ $cdcstate = "normal" ];then echo "tikvcdc replication state is normal." else echo "tikvcdc replication state is $cdcstate." exit 1 fi } cdc_change(){ echo "cdc changefeed-id create." .${ticdc_dir}bin/tikv-cdc cli changefeed create --pd "http://$pd_master" --sink-uri tikv://$pd_slave --changefeed-id $CHANGEFEED_ID if [ $? -eq 0 ]; then echo "$TIME Create changefeed-id successed!!!" cdcstate=`.${ticdc_dir}bin/tikv-cdc cli changefeed --pd "http://$pd_master" list |grep -E "state" |awk -F: {print $2} |awk -F, {print $1} |sed s/\"//g` tikvcdcstate else echo "$TIME Create changefeed-id failed!!!" exit 1 fi } cdc_change | tee ${scripts_log}5.1.5 观察日志输出

观察日志${scripts_log}有无报错，若无报错，则说明执行成功。

5.2 灾备集群回切到主集群

5.2.1 灾备集群停止TICDC同步，检查灾备集群grpc流量为0

观察tikv-details下的grpc流量，若为0，则说明业务已经停止。

5.2.2 执行remove_cdc.sh脚本

remove_cdc.sh需要在tikv-cdc节点执行

sh remove_cdc.sh

remove_cdc.sh脚本如下：

注意：CLUSTER_NAME,CHANGEFEED_ID,pdipdport这几个变量需要按照实际情况修改。

#!/bin/bash export PATH=/home/ap/cdacs/.tiup/bin:$PATH TIME=`(date "+%Y-%m-%d_%H-%M-%S")` #CHANGEFEED_ID,pdipdport are variables CHANGEFEED_ID=test-cdc-bak ##pd_master is up stream cdc pd info. pd_master=172.16.11.115:2379 ticdc_dir=`ps -ef |grep tikv-[c]dc |awk -F" " {print $(NF)}|awk -F"log" {print $1}` ##new add 20230106 cdcstate=`${ticdc_dir}bin/tikv-cdc cli changefeed query -s --changefeed-id $CHANGEFEED_ID --pd http://$pd_master |grep -E "state" |awk -F: {print $2} |awk -F, {print $1} |sed s/[^a-z]//g` logpath="${ticdc_dir}log" scripts_log="$logpath/remove_changefeed_$TIME.log" cdc_change(){ echo "cdc changefeed remove." ${ticdc_dir}bin/tikv-cdc cli changefeed remove --changefeed-id $CHANGEFEED_ID --pd=http://$pd_master if [ $? -eq 0 ]; then echo "’$TIME‘ remove changefeed-id successed!!!" sleep 5 ${ticdc_dir}bin/tikv-cdc cli changefeed --pd http://$pd_master list else echo "‘$TIME’ remove changefeed-id failed!!!" fi } ##new add 20230106 tikvcdcstate() { echo "tikvcdc replication state check." if [ $cdcstate = "normal" ];then echo "tikvcdc replication state is normal." else echo "tikvcdc replication state is $cdcstate." exit 1 fi } ##new add 20230106 tikvcdcreplication() { tsoa=`${ticdc_dir}bin/tikv-cdc cli tso query --pd=http://$pd_master` echo "tikv cdc replication progress check." checknums=10 seconds=2 nums=0 while true do tsob=`${ticdc_dir}bin/tikv-cdc cli changefeed query -s --changefeed-id $CHANGEFEED-ID --pd http://$pd_master |grep -E "tso" |awk -F" " {print $2}|awk -F"," {print $1}` let nums++ if [ $tsob -gt $tsoa ] && [ $nums -le $checknums ];then echo "tikvcdc replication is done,this is $nums times replication check." break elif [ $tsob -lt $tsoa ] && [ $nums -lt $checknums ];then echo "tikvcdc replication is not done,this is $nums times replication check." sleep $seconds else echo "tikvcdc replication is not done,this is $nums times and last replication check." break fi done } main() { tikvcdcstate tikvcdcreplication cdc_change } main| tee ${scripts_log}5.2.3 观察日志输出

检查$ticdc_dir/log/$logfile有无报错，若无报错，则说明执行成功。

5.2.4 主集群启动TICDC同步

执行create_cdc.sh脚本

create_cdc.sh需要在tikv-cdc节点执行

sh create_cdc.sh

create_cdc.sh脚本如下：

注意：CLUSTER_NAME,CHANGEFEED_ID,pd_master,pd_slave这几个变量需要按照实际情况修改。

#!/bin/bash set -e export PATH=/home/cdacs/.tiup/bin:$PATH TIME=`(date "+%Y-%m-%d_%H-%M-%S")` #CHANGEFEED_ID,pd_master,pd_slave are variables CHANGEFEED_ID=test-cdc-bak ##pd_master is up stream cdc pd info. pd_master=IP:2379 #pd_slave is down stream cdc pd info. pd_slave=IP:2379 ticdc_dir=`ps -ef |grep tikv-[c]dc |awk -F" " {print $(NF)}|awk -F"log" {print $1}` logpath="${ticdc_dir}log" scripts_log="$logpath/create_changefeed_$TIME.log" #cdcstate=`${ticdc_dir}bin/tikv-cdc cli changefeed --pd "http://$pd_master" list |grep -E "state" |awk -F: {print $2}|awk -F, {print $1}|sed s/[^a-z]//g` ##new add 20230106 tikvcdcstate() { cdcstate=`${ticdc_dir}bin/tikv-cdc cli changefeed query -s --changefeed-id $CHANGEFEED_ID --pd http://$pd_master |grep -E "state" |awk -F: {print $2} |awk -F, {print $1} |sed s/[^a-z]//g` echo "tikvcdc replication state check." if [ $cdcstate = normal ];then echo "tikvcdc replication state is normal." else echo "tikvcdc replication state is $cdcstate." exit 1 fi } cdc_change(){ echo "cdc changefeed-id start create." ${ticdc_dir}bin/tikv-cdc cli changefeed create --pd "http://$pd_master" --sink-uri tikv://$pd_slave --changefeed-id $CHANGEFEED_ID if [ $? -eq 0 ]; then echo "$TIME Create changefeed-id successed!!!" tikvcdcstate else echo "$TIME Create changefeed-id failed!!!" exit 1 fi } cdc_change | tee ${scripts_log}5.2.5 观察日志输出

观察日志${scripts_log}有无报错，若无报错，则说明执行成功。

六、总结

1、两地三中心架构给业务连续性提供了强有力的保障；

2、定期进行灾备切换演练。