环境描述

Cluster version:    v4.0.11

部署情况

节点名称部署情况Node11tidb+1tikv+1pdNode21tidb+1tikv+1pd+监控组件Node31tidb+1tikv+1pdNode44tikvNode54tikv

数据大小

主要任务

升级版本到v6.5.1

替换老旧机器Node1-Node3，并优化部署结构

升级测试

配置文件是否需要编辑

以源配置文件减配为3tidb+3pd+3tikv安装v4.0.11然后升级到v6.5.1。

这样可以验证配置文件是否有需要修改变更之处

监控迁移测试

生产系统从没有遇到过监控组件的迁移测试，为了保证过程安全可控，在测试环境进行监控组件迁移测试。

主要命令：

tiup cluster scale-in tidb-test -N XXX.XXX.XXX.XXX:9093,XXX.XXX.XXX.XXX:3000,XXX.XXX.XXX.XXX:9090 tiup cluster check ./single-monitor.yaml --user root -p （check因为tiup版本不同，方式有所不同） tiup cluster scale-out tidb-test single-monitor.yaml --user root -p

遇到问题：迁移后部分数据不显示

迁移前显示内容：

迁移后显示内容：

原因和解决方法：测试环境，新的监控组件所在服务器时间比正常晚了5分钟左右，时间同步正常

带数据升级测试

按照使用方提供的标准从生产环境导入部分数据，然后程序接入测试环境，做一次升级测试，并观察是否有不适应的地方

生产机器资源调整

tikv server通过扩容缩容的方式迁移

主要命令：

tiup cluster check ./scale-out20230301.yaml --user root -p tiup cluster scale-out <cluster-name>scale-out20230301.yaml --user root -p tiup cluster scale-in<cluster-name> --node XXX.XXX.XXX.60:20160,XXX.XXX.XXX.61:20160,XXX.XXX.XXX.62:20160

回收过程监控变化：

遇到问题：display没有Tombstone，监控还是有Tombstone

解决方法：

curl -X DELETE {pd_leader_ip}:2379/pd/api/v1/stores/remove-tombstone

tidb server通过扩容缩容的方式迁移

由于tidb是无状态节点，直接扩缩容

主要命令：

tiup cluster check ./tidb-scale-out2023031701.yaml --user root -p tiup cluster scale-out<cluster-name> tidb-scale-out2023031701.yaml --user root -p tiup cluster scale-in <cluster-name> --node XXX.XXX.XXX.XXX:3306

pd server通过扩容缩容的方式迁移

pd server 迁移官网注意提示：

主要命令：

tiup cluster check ./pd-scale-out2023031601.yaml --user root -p tiup cluster scale-out<cluster-name> pd-scale-out2023031601.yaml --user root -p tiup cluster scale-in <cluster-name> --node XXX.XXX.XXX.60:2379

原来leader 缩容：

我这里是通过reload的方式让主切换到新的机器上去的，可以先尝试reload，如果没有自动切换，再尝试手动切换leader

监控组件迁移

步骤和测试场景一样

调整后部署情况

节点名称部署情况Node44tikvNode54tikvNode64tikvNode74tikvNode84tikvNode91tidb+1pdNode101tidb+1pd+监控组件Node111tidb+1pd

生产升级

先升级 TiUP和cluster 版本（建议 tiup 版本不低于 1.11.0）

tiup update --self tiup --version tiup update cluster tiup cluster --version

检查当前集群的健康状况

tiup cluster check <cluster-name> --cluster

其他确认项

变更配置文件，根据测试修改需要变更的参数 检查当前集群没有 DDL 和 Backup 在执行 表记录数据情况统计，方便升级完成验证

执行不停机升级升级

tiup cluster upgrade <cluster-name> v6.5.1

[09:52:11][tidb@Node1 ~]$ tiup cluster check <cluster-name>--cluster [09:52:11]tiup is checking updates for component cluster ... [09:52:11]Starting component`cluster`: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster check <cluster-name>--cluster [09:52:13]Run command on XXX.XXX.XXX.68(sudo:false): /tmp/tiup/bin/insight [09:52:13]Run command on XXX.XXX.XXX.67(sudo:false): /tmp/tiup/bin/insight [09:52:13]Run command on XXX.XXX.XXX.98(sudo:false): /tmp/tiup/bin/insight [09:52:13]Run command on XXX.XXX.XXX.99(sudo:false): /tmp/tiup/bin/insight [09:52:13]Run command on XXX.XXX.XXX.97(sudo:false): /tmp/tiup/bin/insight [09:52:14]Run command on XXX.XXX.XXX.103(sudo:false): /tmp/tiup/bin/insight [09:52:15]Run command on XXX.XXX.XXX.102(sudo:false): /tmp/tiup/bin/insight [09:52:15]Run command on XXX.XXX.XXX.101(sudo:false): /tmp/tiup/bin/insight [09:52:17]Run command on XXX.XXX.XXX.97(sudo:false): cat /etc/security/limits.conf [09:52:17]Run command on XXX.XXX.XXX.98(sudo:false): cat /etc/security/limits.conf [09:52:17]Run command on XXX.XXX.XXX.99(sudo:false): cat /etc/security/limits.conf [09:52:17]Run command on XXX.XXX.XXX.97(sudo:true): sysctl -a [09:52:17]Run command on XXX.XXX.XXX.98(sudo:true): sysctl -a [09:52:17]Run command on XXX.XXX.XXX.67(sudo:false): cat /etc/security/limits.conf [09:52:17]Run command on XXX.XXX.XXX.99(sudo:true): sysctl -a [09:52:17]Run command on XXX.XXX.XXX.68(sudo:false): cat /etc/security/limits.conf [09:52:18]Run command on XXX.XXX.XXX.67(sudo:true): sysctl -a [09:52:18]Run command on XXX.XXX.XXX.68(sudo:true): sysctl -a [09:52:19]Run command on XXX.XXX.XXX.101(sudo:false): cat /etc/security/limits.conf [09:52:20]Run command on XXX.XXX.XXX.102(sudo:false): cat /etc/security/limits.conf [09:52:20]Run command on XXX.XXX.XXX.103(sudo:false): cat /etc/security/limits.conf [09:52:20]Run command on XXX.XXX.XXX.101(sudo:true): sysctl -a [09:52:20]Run command on XXX.XXX.XXX.103(sudo:true): sysctl -a [09:52:20]Run command on XXX.XXX.XXX.102(sudo:true): sysctl -a [09:52:22]Node Check Result Message [09:52:22]---- ----- ------ ------- [09:52:22]XXX.XXX.XXX.103 os-version Pass OS is CentOS Linux 7 (Core) 7.9.2009 [09:52:22]XXX.XXX.XXX.103 cpu-governor Warn Unable to determine current CPU frequency governor policy [09:52:22]XXX.XXX.XXX.103 thp Pass THP is disabled [09:52:22]XXX.XXX.XXX.103 timezone Pass time zone is the same as the first PD machine: Asia/Shanghai [09:52:22]XXX.XXX.XXX.103 permission Pass /tikv3/tidb-deploy/tikv-20162 is writable [09:52:22]XXX.XXX.XXX.103 permission Pass /tikv2/tidb-data/tikv-20161 is writable [09:52:22]XXX.XXX.XXX.103 permission Pass /tikv4/tidb-data/tikv-20163 is writable [09:52:22]XXX.XXX.XXX.103 permission Pass /tikv3/tidb-data/tikv-20162 is writable [09:52:22]XXX.XXX.XXX.103 permission Pass /tikv1/tidb-deploy/tikv-20160 is writable [09:52:22]XXX.XXX.XXX.103 permission Pass /tikv1/tidb-data/tikv-20160 is writable [09:52:22]XXX.XXX.XXX.103 permission Pass /tikv2/tidb-deploy/tikv-20161 is writable [09:52:22]XXX.XXX.XXX.103 permission Pass /tikv4/tidb-deploy/tikv-20163 is writable [09:52:22]XXX.XXX.XXX.103 network Pass network speed of bond0 is 20000MB [09:52:22]XXX.XXX.XXX.103 network Pass network speed of em1 is 10000MB [09:52:22]XXX.XXX.XXX.103 network Pass network speed of em2 is 10000MB [09:52:22]XXX.XXX.XXX.103 selinux Pass SELinux is disabled [09:52:22]XXX.XXX.XXX.103 command Pass numactl: policy: default [09:52:22]XXX.XXX.XXX.103 cpu-cores Pass number of CPU cores / threads: 72 [09:52:22]XXX.XXX.XXX.103 memory Pass memory size is 131072MB [09:52:22]XXX.XXX.XXX.97 permission Pass /tidb-deploy/pd-2379 is writable [09:52:22]XXX.XXX.XXX.97 permission Pass /tidb-data/pd-2379 is writable [09:52:22]XXX.XXX.XXX.97 permission Pass /tidb-deploy/tidb-3306 is writable [09:52:22]XXX.XXX.XXX.97 os-version Pass OS is CentOS Linux 7 (Core) 7.9.2009 [09:52:22]XXX.XXX.XXX.97 cpu-cores Pass number of CPU cores / threads: 56 [09:52:22]XXX.XXX.XXX.97 memory Pass memory size is 131072MB [09:52:22]XXX.XXX.XXX.97 network Pass network speed of p2p2 is 1000MB [09:52:22]XXX.XXX.XXX.97 network Pass network speed of bond0 is 20000MB [09:52:22]XXX.XXX.XXX.97 network Pass network speed of em1 is 10000MB [09:52:22]XXX.XXX.XXX.97 network Pass network speed of em2 is 10000MB [09:52:22]XXX.XXX.XXX.97 network Pass network speed of p2p1 is 1000MB [09:52:22]XXX.XXX.XXX.97 thp Pass THP is disabled [09:52:22]XXX.XXX.XXX.97 command Pass numactl: policy: default [09:52:22]XXX.XXX.XXX.97 cpu-governor Warn Unable to determine current CPU frequency governor policy [09:52:22]XXX.XXX.XXX.97 selinux Pass SELinux is disabled [09:52:22]XXX.XXX.XXX.98 timezone Pass time zone is the same as the first PD machine: Asia/Shanghai [09:52:22]XXX.XXX.XXX.98 os-version Pass OS is CentOS Linux 7 (Core) 7.9.2009 [09:52:22]XXX.XXX.XXX.98 selinux Pass SELinux is disabled [09:52:22]XXX.XXX.XXX.98 permission Pass /tidb-data/alertmanager-9093 is writable [09:52:22]XXX.XXX.XXX.98 permission Pass /tidb-deploy/pd-2379 is writable [09:52:22]XXX.XXX.XXX.98 permission Pass /tidb-data/pd-2379 is writable [09:52:22]XXX.XXX.XXX.98 permission Pass /tidb-deploy/tidb-3306 is writable [09:52:22]XXX.XXX.XXX.98 permission Pass /tidb-deploy/prometheus-9090 is writable [09:52:22]XXX.XXX.XXX.98 permission Pass /tidb-deploy/grafana-3000 is writable [09:52:22]XXX.XXX.XXX.98 permission Pass /tidb-deploy/alertmanager-9093 is writable [09:52:22]XXX.XXX.XXX.98 permission Pass /tidb-data/prometheus-9090 is writable [09:52:22]XXX.XXX.XXX.98 cpu-cores Pass number of CPU cores / threads: 56 [09:52:22]XXX.XXX.XXX.98 cpu-governor Warn Unable to determine current CPU frequency governor policy [09:52:22]XXX.XXX.XXX.98 memory Pass memory size is 131072MB [09:52:22]XXX.XXX.XXX.98 network Pass network speed of em2 is 10000MB [09:52:22]XXX.XXX.XXX.98 network Pass network speed of p1p1 is 1000MB [09:52:22]XXX.XXX.XXX.98 network Pass network speed of p1p2 is 1000MB [09:52:22]XXX.XXX.XXX.98 network Pass network speed of p2p1 is 1000MB [09:52:22]XXX.XXX.XXX.98 network Pass network speed of p2p2 is 1000MB [09:52:22]XXX.XXX.XXX.98 network Pass network speed of bond0 is 20000MB [09:52:22]XXX.XXX.XXX.98 network Pass network speed of em1 is 10000MB [09:52:22]XXX.XXX.XXX.98 thp Pass THP is disabled [09:52:22]XXX.XXX.XXX.98 command Pass numactl: policy: default [09:52:22]XXX.XXX.XXX.99 os-version Pass OS is CentOS Linux 7 (Core) 7.9.2009 [09:52:22]XXX.XXX.XXX.99 memory Pass memory size is 131072MB [09:52:22]XXX.XXX.XXX.99 selinux Pass SELinux is disabled [09:52:22]XXX.XXX.XXX.99 timezone Pass time zone is the same as the first PD machine: Asia/Shanghai [09:52:22]XXX.XXX.XXX.99 permission Pass /tidb-deploy/pd-2379 is writable [09:52:22]XXX.XXX.XXX.99 permission Pass /tidb-data/pd-2379 is writable [09:52:22]XXX.XXX.XXX.99 permission Pass /tidb-deploy/tidb-3306 is writable [09:52:22]XXX.XXX.XXX.99 cpu-cores Pass number of CPU cores / threads: 56 [09:52:22]XXX.XXX.XXX.99 cpu-governor Warn Unable to determine current CPU frequency governor policy [09:52:22]XXX.XXX.XXX.99 network Pass network speed of p2p1 is 1000MB [09:52:22]XXX.XXX.XXX.99 network Pass network speed of p2p2 is 1000MB [09:52:22]XXX.XXX.XXX.99 network Pass network speed of bond0 is 20000MB [09:52:22]XXX.XXX.XXX.99 network Pass network speed of em1 is 10000MB [09:52:22]XXX.XXX.XXX.99 network Pass network speed of em2 is 10000MB [09:52:22]XXX.XXX.XXX.99 network Pass network speed of p1p1 is 1000MB [09:52:22]XXX.XXX.XXX.99 network Pass network speed of p1p2 is 1000MB [09:52:22]XXX.XXX.XXX.99 thp Pass THP is disabled [09:52:22]XXX.XXX.XXX.99 command Pass numactl: policy: default [09:52:22]XXX.XXX.XXX.67 os-version Pass OS is CentOS Linux 7 (Core) 7.7.1908 [09:52:22]XXX.XXX.XXX.67 memory Pass memory size is 131072MB [09:52:22]XXX.XXX.XXX.67 network Pass network speed of em2 is 1000MB [09:52:22]XXX.XXX.XXX.67 network Pass network speed of em3 is 1000MB [09:52:22]XXX.XXX.XXX.67 network Pass network speed of em4 is 1000MB [09:52:22]XXX.XXX.XXX.67 network Pass network speed of p1p1 is 1000MB [09:52:22]XXX.XXX.XXX.67 network Pass network speed of bond0 is 1000MB [09:52:22]XXX.XXX.XXX.67 network Pass network speed of em1 is 1000MB [09:52:22]XXX.XXX.XXX.67 network Pass network speed of p1p4 is 1000MB [09:52:22]XXX.XXX.XXX.67 network Pass network speed of p1p2 is 1000MB [09:52:22]XXX.XXX.XXX.67 network Pass network speed of p1p3 is 1000MB [09:52:22]XXX.XXX.XXX.67 selinux Pass SELinux is disabled [09:52:22]XXX.XXX.XXX.67 command Pass numactl: policy: default [09:52:22]XXX.XXX.XXX.67 timezone Pass time zone is the same as the first PD machine: Asia/Shanghai [09:52:22]XXX.XXX.XXX.67 permission Pass /tikv1/tidb-deploy/tikv-20160 is writable [09:52:22]XXX.XXX.XXX.67 permission Pass /tikv1/tidb-data/tikv-20160 is writable [09:52:22]XXX.XXX.XXX.67 permission Pass /tikv2/tidb-data/tikv-20161 is writable [09:52:22]XXX.XXX.XXX.67 permission Pass /tikv3/tidb-deploy/tikv-20162 is writable [09:52:22]XXX.XXX.XXX.67 permission Pass /tikv4/tidb-deploy/tikv-20163 is writable [09:52:22]XXX.XXX.XXX.67 permission Pass /tikv3/tidb-data/tikv-20162 is writable [09:52:22]XXX.XXX.XXX.67 permission Pass /tikv4/tidb-data/tikv-20163 is writable [09:52:22]XXX.XXX.XXX.67 permission Pass /tikv2/tidb-deploy/tikv-20161 is writable [09:52:22]XXX.XXX.XXX.67 thp Pass THP is disabled [09:52:22]XXX.XXX.XXX.67 cpu-cores Pass number of CPU cores / threads: 56 [09:52:22]XXX.XXX.XXX.67 cpu-governor Warn Unable to determine current CPU frequency governor policy [09:52:22]XXX.XXX.XXX.68 network Pass network speed of p1p2 is 1000MB [09:52:22]XXX.XXX.XXX.68 network Pass network speed of bond0 is 1000MB [09:52:22]XXX.XXX.XXX.68 network Pass network speed of em1 is 1000MB [09:52:22]XXX.XXX.XXX.68 network Pass network speed of em2 is 1000MB [09:52:22]XXX.XXX.XXX.68 network Pass network speed of em3 is 1000MB [09:52:22]XXX.XXX.XXX.68 network Pass network speed of em4 is 1000MB [09:52:22]XXX.XXX.XXX.68 network Pass network speed of p1p1 is 1000MB [09:52:22]XXX.XXX.XXX.68 network Pass network speed of p1p3 is 1000MB [09:52:22]XXX.XXX.XXX.68 network Pass network speed of p1p4 is 1000MB [09:52:22]XXX.XXX.XXX.68 command Pass numactl: policy: default [09:52:22]XXX.XXX.XXX.68 timezone Pass time zone is the same as the first PD machine: Asia/Shanghai [09:52:22]XXX.XXX.XXX.68 permission Pass /tikv2/tidb-data/tikv-20161 is writable [09:52:22]XXX.XXX.XXX.68 permission Pass /tikv3/tidb-data/tikv-20162 is writable [09:52:22]XXX.XXX.XXX.68 permission Pass /tikv1/tidb-data/tikv-20160 is writable [09:52:22]XXX.XXX.XXX.68 permission Pass /tikv4/tidb-deploy/tikv-20163 is writable [09:52:22]XXX.XXX.XXX.68 permission Pass /tikv4/tidb-data/tikv-20163 is writable [09:52:22]XXX.XXX.XXX.68 permission Pass /tikv2/tidb-deploy/tikv-20161 is writable [09:52:22]XXX.XXX.XXX.68 permission Pass /tikv3/tidb-deploy/tikv-20162 is writable [09:52:22]XXX.XXX.XXX.68 permission Pass /tikv1/tidb-deploy/tikv-20160 is writable [09:52:22]XXX.XXX.XXX.68 os-version Pass OS is CentOS Linux 7 (Core) 7.7.1908 [09:52:22]XXX.XXX.XXX.68 cpu-governor Warn Unable to determine current CPU frequency governor policy [09:52:22]XXX.XXX.XXX.68 memory Pass memory size is 131072MB [09:52:22]XXX.XXX.XXX.68 cpu-cores Pass number of CPU cores / threads: 56 [09:52:22]XXX.XXX.XXX.68 selinux Pass SELinux is disabled [09:52:22]XXX.XXX.XXX.68 thp Pass THP is disabled [09:52:22]XXX.XXX.XXX.101 permission Pass /tikv1/tidb-deploy/tikv-20160 is writable [09:52:22]XXX.XXX.XXX.101 permission Pass /tikv2/tidb-deploy/tikv-20161 is writable [09:52:22]XXX.XXX.XXX.101 permission Pass /tikv3/tidb-deploy/tikv-20162 is writable [09:52:22]XXX.XXX.XXX.101 permission Pass /tikv4/tidb-deploy/tikv-20163 is writable [09:52:22]XXX.XXX.XXX.101 permission Pass /tikv1/tidb-data/tikv-20160 is writable [09:52:22]XXX.XXX.XXX.101 permission Pass /tikv4/tidb-data/tikv-20163 is writable [09:52:22]XXX.XXX.XXX.101 permission Pass /tikv2/tidb-data/tikv-20161 is writable [09:52:22]XXX.XXX.XXX.101 permission Pass /tikv3/tidb-data/tikv-20162 is writable [09:52:22]XXX.XXX.XXX.101 os-version Pass OS is CentOS Linux 7 (Core) 7.9.2009 [09:52:22]XXX.XXX.XXX.101 cpu-governor Warn Unable to determine current CPU frequency governor policy [09:52:22]XXX.XXX.XXX.101 network Pass network speed of em3 is 1000MB [09:52:22]XXX.XXX.XXX.101 network Pass network speed of em4 is 1000MB [09:52:22]XXX.XXX.XXX.101 network Pass network speed of p1p2 is 1000MB [09:52:22]XXX.XXX.XXX.101 network Pass network speed of p2p4 is 10000MB [09:52:22]XXX.XXX.XXX.101 network Pass network speed of bond0 is 20000MB [09:52:22]XXX.XXX.XXX.101 network Pass network speed of em2 is 10000MB [09:52:22]XXX.XXX.XXX.101 network Pass network speed of p1p1 is 1000MB [09:52:22]XXX.XXX.XXX.101 network Pass network speed of p2p2 is 10000MB [09:52:22]XXX.XXX.XXX.101 network Pass network speed of em1 is 10000MB [09:52:22]XXX.XXX.XXX.101 selinux Pass SELinux is disabled [09:52:22]XXX.XXX.XXX.101 command Pass numactl: policy: default [09:52:22]XXX.XXX.XXX.101 timezone Pass time zone is the same as the first PD machine: Asia/Shanghai [09:52:22]XXX.XXX.XXX.101 memory Pass memory size is 131072MB [09:52:22]XXX.XXX.XXX.101 thp Pass THP is disabled [09:52:22]XXX.XXX.XXX.101 cpu-cores Pass number of CPU cores / threads: 56 [09:52:22]XXX.XXX.XXX.102 timezone Pass time zone is the same as the first PD machine: Asia/Shanghai [09:52:22]XXX.XXX.XXX.102 os-version Pass OS is CentOS Linux 7 (Core) 7.9.2009 [09:52:22]XXX.XXX.XXX.102 cpu-cores Pass number of CPU cores / threads: 56 [09:52:22]XXX.XXX.XXX.102 selinux Pass SELinux is disabled [09:52:22]XXX.XXX.XXX.102 thp Pass THP is disabled [09:52:22]XXX.XXX.XXX.102 command Pass numactl: policy: default [09:52:22]XXX.XXX.XXX.102 permission Pass /tikv1/tidb-deploy/tikv-20160 is writable [09:52:22]XXX.XXX.XXX.102 permission Pass /tikv4/tidb-data/tikv-20163 is writable [09:52:22]XXX.XXX.XXX.102 permission Pass /tikv3/tidb-data/tikv-20162 is writable [09:52:22]XXX.XXX.XXX.102 permission Pass /tikv2/tidb-data/tikv-20161 is writable [09:52:22]XXX.XXX.XXX.102 permission Pass /tikv1/tidb-data/tikv-20160 is writable [09:52:22]XXX.XXX.XXX.102 permission Pass /tikv3/tidb-deploy/tikv-20162 is writable [09:52:22]XXX.XXX.XXX.102 permission Pass /tikv2/tidb-deploy/tikv-20161 is writable [09:52:22]XXX.XXX.XXX.102 permission Pass /tikv4/tidb-deploy/tikv-20163 is writable [09:52:22]XXX.XXX.XXX.102 cpu-governor Warn Unable to determine current CPU frequency governor policy [09:52:22]XXX.XXX.XXX.102 memory Pass memory size is 131072MB [09:52:22]XXX.XXX.XXX.102 network Pass network speed of p2p3 is 10000MB [09:52:22]XXX.XXX.XXX.102 network Pass network speed of bond0 is 20000MB [09:52:22]XXX.XXX.XXX.102 network Pass network speed of em1 is 10000MB [09:52:22]XXX.XXX.XXX.102 network Pass network speed of em2 is 10000MB [09:52:22]XXX.XXX.XXX.102 network Pass network speed of em4 is 1000MB [09:52:22]XXX.XXX.XXX.102 network Pass network speed of p1p2 is 1000MB [09:52:22]XXX.XXX.XXX.102 network Pass network speed of em3 is 1000MB [09:52:22]XXX.XXX.XXX.102 network Pass network speed of p1p1 is 1000MB [09:52:22]XXX.XXX.XXX.102 network Pass network speed of p2p4 is 10000MB [09:52:22]Checking region status of the cluster<cluster-name>... [09:52:22]All regions are healthy. [09:53:35][tidb@Node1 ~]$ tiup cluster upgrade <cluster-name>v6.5.1 [09:53:35]tiup is checking updates for component cluster ... [09:53:35]Starting component`cluster`: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster upgrade <cluster-name>v6.5.1 [09:53:35]Before the upgrade, it is recommended to read the upgrade guide at https://docs.pingcap.com/tidb/stable/upgrade-tidb-using-tiup and finish the preparation steps. [09:53:35]This operation will upgrade tidb v4.0.11 cluster<cluster-name>- SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/<cluster-name>/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/<cluster-name>/ssh/id_rsa.pub [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.67 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.98 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.67 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.99 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.67 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.67 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.97 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.101 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.101 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.68 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.68 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.102 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.68 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.102 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.102 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.101 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.103 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.102 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.68 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.103 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.101 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.97 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.103 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.103 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.99 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.98 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.98 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.98 [09:53:59]+ [Parallel] - UserSSH: user=tidb, host=XXX.XXX.XXX.98 [09:53:59]+ [ Serial ] - Download: component=grafana, versi